[ncl-talk] Writing binary characters while invoking WriteByteOrder

[Kay Shelton]

Thanks very much for the simple workaround, Dave. It seems to work just
fine.

Kay

On 9 July 2016 at 00:20, Dave Allured - NOAA Affiliate <
dave.allured at noaa.gov> wrote:

> Sorry, there is a bug in the code that I just posted.  Please use this
> version instead, for correct handling of 8-bit character sets.
>
>   f = "test8.bin"
>   str = "abcdef"
>   ubytes  = toubyte (tochar (str))
>   ushorts = (256H * ubytes(0::2)) + ubytes(1::2)
>   setfileoption ("bin", "WriteByteOrder", "BigEndian")
>   fbinrecwrite (f, 0, ushorts)
>
> --Dave
>
>
> On Fri, Jul 8, 2016 at 4:41 PM, Dave Allured - NOAA Affiliate <
> dave.allured at noaa.gov> wrote:
>
>> Thanks, Dennis.
>>
>> Kay, here is a workaround to write characters in big endian mode.  This
>> uses unsigned shorts to circumvent the current problem with single-byte
>> data types.
>>
>>   f = "test8.bin"
>>   str = "abcdef"
>>   ubytes  = toubyte (tochar (str))
>>   ushorts = toushort (256 * ubytes(0::2) + ubytes(1::2))
>>   setfileoption ("bin", "WriteByteOrder", "BigEndian")
>>   fbinrecwrite (f, 0, ushorts)
>>
>> This trick requires an even number of characters in the string to be
>> written.  HTH.
>>
>> --Dave
>>
>>
>> On Fri, Jul 8, 2016 at 4:08 PM, Dennis Shea <shea at ucar.edu> wrote:
>>
>>> FYI: A JIRA ticket has been opened: NCL_2462
>>>
>>>
>>> On Fri, Jul 8, 2016 at 3:10 PM, Dave Allured - NOAA Affiliate <
>>> dave.allured at noaa.gov> wrote:
>>>
>>>> Kay and NCL,
>>>>
>>>> I can confirm that fbinrecwrite fails to correctly write an array of
>>>> characters in big endian mode.  I also found the same failure for types
>>>> byte and ubyte.  So I think this is a bug in NCL that needs fixing.
>>>>
>>>> Here is a reduced test case.  Use any of x1, x2, or x3 in the write
>>>> statement.  I get valid little-endian output files when the setfileoption
>>>> statement is commented out.  I get the same good and bad results on both
>>>> Linux and Mac systems; both are little-endian X86 type systems.
>>>>
>>>>   f = "test7.bin"
>>>>   x1 = tochar ("name7")
>>>>   x2 = tobyte (x1)
>>>>   x3 = toubyte (x1)
>>>>   setfileoption ("bin", "WriteByteOrder", "BigEndian")
>>>>   fbinrecwrite (f, 0, x1)
>>>>
>>>> --Dave
>>>>
>>>>
>>>> On Thu, Jul 7, 2016 at 4:31 AM, Kay Shelton <kay.shelton at gmail.com>
>>>> wrote:
>>>>
>>>>> Hello ncl-talkers,
>>>>>
>>>>> I have a little bit of a conundrum with trying to write a binary file
>>>>> for a colleague that is destined to be used in TELEMAC-2D. The binary file
>>>>> that I am trying to write needs to include characters, integers and floats,
>>>>> and will be used on a machine that is Big Endian. I am creating the file on
>>>>> a Linux machine that is Little Endian, hence when I write the binary file I
>>>>> am using "setfileoption("bin","WriteByteOrder","BigEndian")". This all
>>>>> works fine for integers and floats, but the characters are not correctly
>>>>> represented in the output binary.
>>>>>
>>>>> I have written a short piece of code that demonstrates the problem
>>>>> succinctly:
>>>>>
>>>>> --------------------------------------------------------
>>>>> begin
>>>>>
>>>>> endianness = isbigendian()
>>>>> words = "blah_de_blah_de_blah"
>>>>> words_char = new(strlen(words),character)
>>>>> words_char = tochar(" ")
>>>>> words_char(0:strlen(words)-1) = tochar(words)
>>>>> outfil = "temp.bin"
>>>>>
>>>>> if (.not.endianness) then
>>>>>   print("Endianness is LittleEndian")
>>>>>
>>>>> ; run and look at results, then uncomment the next two lines and re-run
>>>>> ;  setfileoption("bin","ReadByteOrder","BigEndian")
>>>>> ;  setfileoption("bin","WriteByteOrder","BigEndian")
>>>>>
>>>>>   if (fileexists(outfil)) then
>>>>>     system("rm -f "+outfil)
>>>>>   end if
>>>>>   fbinrecwrite (outfil,0, (/ words_char /))
>>>>>   new_words = fbinrecread(outfil, 0, -1, "character")
>>>>>   print("WORDS (original): $$ "+words+" $$")
>>>>>   print("WORDS (original, char to str): $$ "+tostring(words_char)+"
>>>>> $$")
>>>>>   print("NEW_WORDS (char to str): $$ "+tostring(new_words)+" $$")
>>>>> else
>>>>>   print("Already BigEndian")
>>>>> end if
>>>>>
>>>>> end
>>>>> --------------------------------------------------------
>>>>>
>>>>> I have tested this on two completely different Linux machines running
>>>>> NCL 6.3.0 (both 64-bit Ubuntu) and also on one of them using NCL 6.2.1. All
>>>>> give the same result.
>>>>>
>>>>> Running the script first time with the two setfileoption lines
>>>>> commented out (i.e. native LittleEndian binary):
>>>>> (0)     Endianness is LittleEndian
>>>>> (0)     WORDS (original): $$ blah_de_blah_de_blah $$
>>>>> (0)     WORDS (original, char to str): $$ blah_de_blah_de_blah $$
>>>>> (0)     NEW_WORDS (char to str): $$ blah_de_blah_de_blah $$
>>>>>
>>>>> Running with the two setfileoption lines un-commented:
>>>>> (0)     Endianness is LittleEndian
>>>>> (0)     WORDS (original): $$ blah_de_blah_de_blah $$
>>>>> (0)     WORDS (original, char to str): $$ blah_de_blah_de_blah $$
>>>>> (0)     NEW_WORDS (char to str): $$  $$
>>>>>
>>>>> As you can see with the BigEndian option the characters are not
>>>>> correctly written to the file.
>>>>>
>>>>> Is this correct behaviour? Have I missed something I should have
>>>>> included to get this to work? (This is entirely possible and I apologise
>>>>> for wasting your time if this is a simple user-error.)
>>>>>
>>>>> I have searched the ncl-talk archives and have not found this problem
>>>>> encountered by others, so it may well be an issue exclusive to my machines.
>>>>> If it is, I can work around it by wrapping in some fortran to handle the
>>>>> reading and writing of the binary correctly. I would prefer to avoid such a
>>>>> work around as the size of the arrays in the binary files I have to read
>>>>> and write will vary, and I would prefer not to have to hard-code the array
>>>>> sizes when NCL makes calls to the fortran code.
>>>>>
>>>>> [Note: on the machine running NCL 6.2.1 I also get the following
>>>>> warning message on both attempts:
>>>>> warning:fbinrecwrite: end of file reached before record number,
>>>>> writing record as last record in file]
>>>>>
>>>>> Many thanks for your help,
>>>>> Kay
>>>>>
>>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20160711/6acb182a/attachment.html