SRP_Run_Command with multi-byte command characters

MattCrozier

A customer has found a regression that appears to have come about since SRP Utilities v2.1.6 . I've tracked it down a difference in the way SRP_Run_Command() works, and suspect the changes made for "Fixed bug in SRP_Run_Command where output would not go into variable if it was too long"

We're using the ImageMagick DOS programs to create thumbnails. This now fails when processing an image that contains unicode multi-byte characters in the filename. As an example, when processing "F:\media\飞蛾.jpg" via SRP_Run_Command(), the response is an error "convert.exe: unable to open image 'F:\media\飞蛾.jpg': No such file or directory"

A simpler test would be a DOS COPY command. From v2.1.6, the command "COPY F:\Media\æ.jpg C:\TEMP" just returns "The system cannot find the file specified." - whereas before you'd get "1 file(s) copied."

It appears that the multi-byte characters are being interpreted as single-byte?

This still occurs in SRP Utilities version 2.2.0.3 32bit. However 64bit seems ok!

Any chance of a 32bit fix for this OI9? :)
Cheers, M@

KevinFournier

Try 2.2.1.2. The problem was actually the Win32 API. I was calling CreateProcessA, which is supposed to convert everything to Unicode and call CreateProcessW. Apparently the 64-bit version does this, but not the 32-bit version. This version of SRP Utilities does the Unicode conversion manually and calls CreateProcessW directly.

MattCrozier

Brilliant! - thanks for this. Yes, that fixes it.

In testing this I found that it also fixed another quirk in that previous versions (even as far back as v1.6) would return multiple results for the command "dir f:\media\飞蛾.jpg /b", depending on what else was in that directory. It should only return one result, of course.

But curiously, the 64bit version (still at v2.2.0.3) still returns multiple results for this command. Eg,

run evalv "@ans = 'VAR'` call srp_run_command( 'dir f:\media\飞蛾.jpg /b', @ans)"
73.jpg
c.jpg
‘.jpg
??.jpg

This doesn't affect us at the moment, but it's probably worth noting the difference ;).

Cheers, M@

KevinFournier

Here is 2.2.1.2 64-bit if you want to test if this quirk is resolved in OI10.

MattCrozier

Hmm?, slightly different but not quite there..

_run evalv "@ans = 'VAR'` Call SRP_Run_Command( 'dir f:\media\飞蛾.jpg /b', @ans)"
73.jpg
c.jpg
�.jpg
??.jpg

The �.jpg filename is for the 飞蛾.jpg requested.
??.jpg is for æ.jpg

Actually, it doesn't recognise æ.jpg directly:

_run evalv "@ans = 'VAR'` Call SRP_Run_Command( 'dir f:\media\æ.jpg /b', @ans)"
File Not Found

Works in 32bit !?
Thanks, M@

KevinFournier

These 2.2.1.3 RDKs should do the trick. Commands like Dir and Copy aren't actually EXE's, so they have to be routed through CMD.EXE, which was outputing everything as ASCII. A simple flag cleared that up. Sadly, though, other commands like IPCONFIG only output in ASCII, so I also had to add code to dynamically determine when output was Unicode or when it was ASCII. It should all look correct now.

MattCrozier

Ah, thanks for the explanation Kevin!
This is looking even better in 32bit with the correct unicode characters coming back in the output.

The 64bit is still having problems though:

_run evalv "@ans = 'VAR'` Call SRP_Run_Command( 'dir f:\media\飞蛾.jpg /b', @ans)"
73.jpg
c.jpg
æ.jpg
飞蛾.jpg

Specifically for our case, I'm trying to extract the metadata of an image, eg for æ.jpg I get

_run evalv "@ans = 'VAR'` c= 'C:\Program Files (x86)\Vernon Systems\ImageMagick\identify.exe -ping -verbose ': quote( 'F:\media\æ.jpg')` Call SRP_Run_Command( c, @ans)"
identify.exe: UnableToOpenBlob 'F:\media\�.jpg': No such file or directory @ error/blob.c/OpenBlob/2691.

It seems the problem now is with translation to the command line as well? I've tried this again with version 2.2.0.3 and it does recognise the file, even though the characters on output aren't right

_run evalv "@ans = 'VAR'` c= 'C:\Program Files (x86)\Vernon Systems\ImageMagick\identify.exe -ping -verbose ': quote( 'F:\media\æ.jpg')` Call SRP_Run_Command( c, @ans)"
Image: F:\media\æ.jpg
Format: JPEG (Joint Photographic Experts Group JFIF format)
Mime type: image/jpeg
Class: DirectClass
Geometry: 468x813+0+0
Resolution: 300x300

So there seems to be a slight regression somewhere between 2.2.0.3 and 2.2.1.3 for 64bit.

Cheers, M@

KevinFournier

I updated the DLL stub to use LPWSTR to force OI to do all the encoding/decoding. Version 2.2.1.4 should work.

MattCrozier

Thanks for persisting with this Kevin ;).

Ok, that works for æ.jpg - but 飞蛾.jpg is still being interpreted as multiple files at DOS. If I try to extract the metadata for this image, it gives me the metadata for 73.jpg as well (but not the other two!?). Perhaps there's just something very odd about the encoding of those chinese characters.
And just confirming the 32bit is all ok.

Cheers, M@

KevinFournier

Give 2.2.1.5 a try. I confirmed that in OI10, the unicode is preserved both going into SRP_Run_Command and coming out, which can be confirmed using the DIR command. If this doesn't work, then I suspect that ImageMagick isn't handling the encoding correctly.

MattCrozier

Hmm, I'm getting results similar to prior versions. v2.2.1.4 seems to behave the best out of all of these patches, certainly for 32bit. I think it must be a very rare 64bit quirk that's not worth worrying about - but I'd say 2.2.1.4 would be the better one going forward ;).

Thanks for your time on this :)
Cheers, M@

KevinFournier

Can you to be more specific as to why 2.2.1.4 is better?

MattCrozier

The output comes back to OI with the correct characters:

In 2.2.1.4, 64bit

_run evalv "c= 'dir F:\media\æ.jpg /b'` @ans = 'VAR'` Call SRP_Run_Command( c, @ans)"
æ.jpg

_run evalv "c= 'dir F:\media\飞蛾.jpg /b'` @ans = 'VAR'` Call SRP_Run_Command( c, @ans)"
73.jpg
c.jpg
æ.jpg
??.jpg

In 2.2.1.4, 32bit (perfect :)

_run evalv "c= 'dir F:\media\æ.jpg /b'` @ans = 'VAR'` Call SRP_Run_Command( c, @ans)"
æ.jpg
.
_run evalv "c= 'dir F:\media\飞蛾.jpg /b'` @ans = 'VAR'` Call SRP_Run_Command( c, @ans)"
飞蛾.jpg

But in 2.2.1.5, 64bit

_run evalv "c= 'dir F:\media\æ.jpg /b'` @ans = 'VAR'` Call SRP_Run_Command( c, @ans)"
‘.jpg

_run evalv "c= 'dir F:\media\飞蛾.jpg /b'` @ans = 'VAR'` Call SRP_Run_Command( c, @ans)"
73.jpg
c.jpg
‘.jpg
??.jpg

and 2.2.1.5, 32bit

_run evalv "c= 'dir F:\media\æ.jpg /b'` @ans = 'VAR'` Call SRP_Run_Command( c, @ans)"
‘.jpg
.
_run evalv "c= 'dir F:\media\飞蛾.jpg /b'` @ans = 'VAR'` Call SRP_Run_Command( c, @ans)"
??.jpg

MattCrozier

Oh dear, we've now discovered a different problem with v2.2.1.5 (and 4) 32bit - and this is quite weird! We're calling ImageMagic via SRP_Run_Command() just to return the dimensions of a thumbnail image, eg '75x114'. It seems that in cases where the width of the image is less than 100 pixels, the return value comes back as unicode characters.

Eg, 2.2.1.4 32bit

_run evalv "@ans = 'VAR'` c= 'C:\Program Files (x86)\Vernon Systems\ImageMagick\identify.exe -ping -format %G ': quote( 'C:\T\1993.1313.bmp')` call srp_run_command( c, @ans)"
㔷ㅸ㐱

Using 2.1.6, this returns correctly as '75x114'

If I physically rotate the image 90 degrees, SRP_Run_Command() correctly returns '114x75'
Rotate it again and it comes back as '㔷ㅸ㐱' instead of '75x114'

This return value should just be a simple text string. Why the difference? Could ImageMagick be setting some kind of DOS return code that v2.2.1.5 is sensitive to??

Hope you can help. I've already deployed this and might have to revert back to v2.1.6
Cheers, M@.

KevinFournier

I downloaded ImageMagick. Email me the image and all the steps (including the commands to rotate), and I'll see if I can't figure out what's happening.

KevinFournier

Try SRP Utilities 2.2.2.0. I think I've finally settled on the best solution for handling Unicode.

MattCrozier

Yes!! This works for all the tests above, 32 and 64 bit, and in our application.
Thanks very much Kevin :)