12 Replies Latest reply on Jun 15, 2020 2:50 PM by LJ LongWing

    Character replacements in ARS which cause issues in eBonding.

    Jaffer Mahsoob
      Share This:

      Here's one that comes up from time to time, when eBonding MS Word specific characters are in a message, it causes problems.  I think I have them all here, let me know if I missed anything or if there's a better way to do it...

       

       

      REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE($Field$, "‒", "-"), "–", "-"), "—", "-"), "―", "-"), "‖", "||"), "‗", "_"), "‘", "'"), "’", "'"), "‚", "'"), "‛", "'"), "“","""),"”","""),"„","""),"‟","""),"…","..."),"•","*"),"․",".")

       

      Basically, I was looking at the demoronizer.pl script, demoroniser - correct moronic and gratuitously incompatible Microsoft HTML, it had everything I could think of in it...  When the source/target is ServiceNow, it's easy enough to basically change the regular expression matches to work with the javascript replace(/\u2013|\u2014/g, '-'); function, but I needed it in AR Workflow too.  If there's a better way to do it, I'm open to suggestions...

       

      This isn't specific to ServiceNow eBonding, it just happens to be the main target/source that I've been working with recently...  I've noticed the same thing with other endpoints as well...

       

      -Jaffer

        • 1. Re: Character replacements in ARS which cause issues in eBonding.
          LJ LongWing

          Jaffer,

          I've seen similar attempts in the past, I've unfortunately never found a universal solution.  While Remedy doesn't support RegEx in it's fullest extent, I'm inclined to do something like a 'find everything not on my keyboard and replace with a ?' or something similar....your approach to convert the MS specific characters with the appropriate ascii version is better than my approach because it doesn't replace everything with an unknown value but with the appropriate generic version....I unfortunately don't know of a better way unless you wanted to step out of Remedy into either the DB or scripting worlds.

          • 2. Re: Character replacements in ARS which cause issues in eBonding.
            Sinisa Mikor

            Would using filter API make any difference?  (For better or for worse.)

             

            P. S.  Which model of keyboard did you have in mind -- Space-cadet, perhaps?

            • 3. Re: Character replacements in ARS which cause issues in eBonding.
              Mark Walters

              Just in case it helps - there is a regexp plugin - Regular Expression (Regex) AR API Filter Plug-in (v4.1)

              3 of 3 people found this helpful
              • 4. Re: Character replacements in ARS which cause issues in eBonding.
                LJ LongWing

                Well...as Mark points out, there is an old, but tried and true RegEx plugin that does exactly what you would want, assuming you can come up with the Regex to do what you want....and that you are able/willing to modify your system to push all of the appropriate data through the plugin to ensure it gets scrubbed properly

                • 5. Re: Character replacements in ARS which cause issues in eBonding.
                  Jason Miller

                  Going to the DB is probably the quickest and closest to "keeping it native". I have used a SQL Set Fields action in a filter to perform a SHA1 hash (

                  https://communities.bmc.com/community/bmcdn/bmc_it_service_support/blog/2013/07/24/the-pulse-what-s-trending-in-remedy-i…  ).

                  You could do the same thing with regex that is appropriate for your RDBMS.

                  2 of 2 people found this helpful
                  • 6. Re: Character replacements in ARS which cause issues in eBonding.
                    Sinisa Mikor

                    Does "quickest" refers to time necessary to set up workflow itself, or its execution time?  Haven't tested it myself, but I'd assume that passing parameters to database server where they are processed by database procedure and passing its output back to AR System server would take longer than calling API Filter Plug-in, which is running on AR System server, with the same parameters and collecting its output.  Or is it situational (database method being faster for processing values stored in database, plugin being faster for processing transactional values)?

                    • 7. Re: Character replacements in ARS which cause issues in eBonding.
                      Jason Miller

                      Very good distinction you are pointing out. I meant quickest as in creating the solution without regards to performance.

                       

                      AR Server is in constant communications with the DB so performance-wise I make the assumption you already have a solid, well-performing connection from AR to DB, and that the DB resources are sufficiently sized. I agree that a plug-in that would keep the regex processing local to the AR server vs. sending the load to the DB server could be beneficial.

                       

                      Which choice really depends on how frequently this is going to be triggered, the environment and its resources, how well the plug-in is written, how well is the plug-in supported/maintained, etc. Every case is different.

                      1 of 1 people found this helpful
                      • 8. Re: Character replacements in ARS which cause issues in eBonding.
                        Jaffer Mahsoob

                        These characters are the automatic replacements that MS Word and MS Outlook are throwing in documents and emails in order to make them more visually appealing, I guess.  I don't think there are any use cases where folks were actually typing the characters...  Either way when text is copied from Word or Outlook and then used in a web service it throws an error...  I was looking for the most reasonable way to remove the possibility to have the error occur in the most scalable way...

                         

                        Thanks,

                        Jaffer

                        • 9. Re: Character replacements in ARS which cause issues in eBonding.
                          Sinisa Mikor

                          If PERL script you mention works as advertised, PERL itself is (or can be) available on your AR System server, and performance is acceptable, there may be no reason not to step out into scripting world, as LJ puts it.  In case you need guidance on setting this up, you could use Set Fields action to populate demoronized character field with output of $PROCESS$ <PERL> <demoronizer.pl> applied to original content (Run Process action would disregard demoronized content and only populate error keywords as appropriate); in order to process entire content, it may be necessary to dump it temporarily to server and refer to name of temporary file during that Set Fields action instead of supplying original content's field name.

                          2 of 2 people found this helpful
                          • 10. Re: Character replacements in ARS which cause issues in eBonding.
                            Jaffer Mahsoob

                            I was really only using the script as a great example of how to clean up these characters, I had been fixing one at a time as they occurred, and I came across that awesome script that had all of them, made it really easy.  I think calling an external Perl script for each field that might have the goofy characters seems like asking for hard to trace workflow issues... 

                             

                            The replace/replace/replace statement I gave initially works, it would be awesome if I could craft it in a way that the same statement could be reused each time instead of putting that statement repeatedly in many different pieces of workflow.  That's why I was asking the question, seems like there must be a better way to do it...

                            -Jaffer

                            • 11. Re: Character replacements in ARS which cause issues in eBonding.
                              Sinisa Mikor

                              In that case, have you considered isolating replacement in separate workflow element (active link or filter) and using it as a sort of procedure with reference to field containing original content and reference to field containing corrected content?  (If defined properly, this workflow element could be reused on multiple forms.)  For example, your main workflow could contain qualification to recognize suspect content and in that case prepare necessary references (or copy content into field used in replacement action, if you decide to avoid references) and call guide to perform alterations; table could also be used to simplify processing multiple fields at once (e. g. its condition could detect which fields out of set of fields to verify might need correction, and call guide would easily process all rows of resulting table).

                              • 12. Re: Character replacements in ARS which cause issues in eBonding.
                                LJ LongWing

                                Jaffer,

                                If that's your goal...that's simple.  Put that into a filter that fires on Service.  Have the service take the field and do the replace as appropriate....then anywhere you want to do the replacing, make a service call, use the field you want 'cleaned' as the input and where you want cleaned (likely same field) as the output value....Remedy will call it like a function...you build it once and use it as many times as you would like

                                3 of 3 people found this helpful