[ncl-talk] Writing binary characters while invoking WriteByteOrder

[Dave Allured - NOAA Affiliate]

Sorry, there is a bug in the code that I just posted.  Please use this
version instead, for correct handling of 8-bit character sets.

  f = "test8.bin"
  str = "abcdef"
  ubytes  = toubyte (tochar (str))
  ushorts = (256H * ubytes(0::2)) + ubytes(1::2)
  setfileoption ("bin", "WriteByteOrder", "BigEndian")
  fbinrecwrite (f, 0, ushorts)

--Dave


On Fri, Jul 8, 2016 at 4:41 PM, Dave Allured - NOAA Affiliate <
dave.allured at noaa.gov> wrote:

> Thanks, Dennis.
>
> Kay, here is a workaround to write characters in big endian mode.  This
> uses unsigned shorts to circumvent the current problem with single-byte
> data types.
>
>   f = "test8.bin"
>   str = "abcdef"
>   ubytes  = toubyte (tochar (str))
>   ushorts = toushort (256 * ubytes(0::2) + ubytes(1::2))
>   setfileoption ("bin", "WriteByteOrder", "BigEndian")
>   fbinrecwrite (f, 0, ushorts)
>
> This trick requires an even number of characters in the string to be
> written.  HTH.
>
> --Dave
>
>
> On Fri, Jul 8, 2016 at 4:08 PM, Dennis Shea <shea at ucar.edu> wrote:
>
>> FYI: A JIRA ticket has been opened: NCL_2462
>>
>>
>> On Fri, Jul 8, 2016 at 3:10 PM, Dave Allured - NOAA Affiliate <
>> dave.allured at noaa.gov> wrote:
>>
>>> Kay and NCL,
>>>
>>> I can confirm that fbinrecwrite fails to correctly write an array of
>>> characters in big endian mode.  I also found the same failure for types
>>> byte and ubyte.  So I think this is a bug in NCL that needs fixing.
>>>
>>> Here is a reduced test case.  Use any of x1, x2, or x3 in the write
>>> statement.  I get valid little-endian output files when the setfileoption
>>> statement is commented out.  I get the same good and bad results on both
>>> Linux and Mac systems; both are little-endian X86 type systems.
>>>
>>>   f = "test7.bin"
>>>   x1 = tochar ("name7")
>>>   x2 = tobyte (x1)
>>>   x3 = toubyte (x1)
>>>   setfileoption ("bin", "WriteByteOrder", "BigEndian")
>>>   fbinrecwrite (f, 0, x1)
>>>
>>> --Dave
>>>
>>>
>>> On Thu, Jul 7, 2016 at 4:31 AM, Kay Shelton <kay.shelton at gmail.com>
>>> wrote:
>>>
>>>> Hello ncl-talkers,
>>>>
>>>> I have a little bit of a conundrum with trying to write a binary file
>>>> for a colleague that is destined to be used in TELEMAC-2D. The binary file
>>>> that I am trying to write needs to include characters, integers and floats,
>>>> and will be used on a machine that is Big Endian. I am creating the file on
>>>> a Linux machine that is Little Endian, hence when I write the binary file I
>>>> am using "setfileoption("bin","WriteByteOrder","BigEndian")". This all
>>>> works fine for integers and floats, but the characters are not correctly
>>>> represented in the output binary.
>>>>
>>>> I have written a short piece of code that demonstrates the problem
>>>> succinctly:
>>>>
>>>> --------------------------------------------------------
>>>> begin
>>>>
>>>> endianness = isbigendian()
>>>> words = "blah_de_blah_de_blah"
>>>> words_char = new(strlen(words),character)
>>>> words_char = tochar(" ")
>>>> words_char(0:strlen(words)-1) = tochar(words)
>>>> outfil = "temp.bin"
>>>>
>>>> if (.not.endianness) then
>>>>   print("Endianness is LittleEndian")
>>>>
>>>> ; run and look at results, then uncomment the next two lines and re-run
>>>> ;  setfileoption("bin","ReadByteOrder","BigEndian")
>>>> ;  setfileoption("bin","WriteByteOrder","BigEndian")
>>>>
>>>>   if (fileexists(outfil)) then
>>>>     system("rm -f "+outfil)
>>>>   end if
>>>>   fbinrecwrite (outfil,0, (/ words_char /))
>>>>   new_words = fbinrecread(outfil, 0, -1, "character")
>>>>   print("WORDS (original): $$ "+words+" $$")
>>>>   print("WORDS (original, char to str): $$ "+tostring(words_char)+" $$")
>>>>   print("NEW_WORDS (char to str): $$ "+tostring(new_words)+" $$")
>>>> else
>>>>   print("Already BigEndian")
>>>> end if
>>>>
>>>> end
>>>> --------------------------------------------------------
>>>>
>>>> I have tested this on two completely different Linux machines running
>>>> NCL 6.3.0 (both 64-bit Ubuntu) and also on one of them using NCL 6.2.1. All
>>>> give the same result.
>>>>
>>>> Running the script first time with the two setfileoption lines
>>>> commented out (i.e. native LittleEndian binary):
>>>> (0)     Endianness is LittleEndian
>>>> (0)     WORDS (original): $$ blah_de_blah_de_blah $$
>>>> (0)     WORDS (original, char to str): $$ blah_de_blah_de_blah $$
>>>> (0)     NEW_WORDS (char to str): $$ blah_de_blah_de_blah $$
>>>>
>>>> Running with the two setfileoption lines un-commented:
>>>> (0)     Endianness is LittleEndian
>>>> (0)     WORDS (original): $$ blah_de_blah_de_blah $$
>>>> (0)     WORDS (original, char to str): $$ blah_de_blah_de_blah $$
>>>> (0)     NEW_WORDS (char to str): $$  $$
>>>>
>>>> As you can see with the BigEndian option the characters are not
>>>> correctly written to the file.
>>>>
>>>> Is this correct behaviour? Have I missed something I should have
>>>> included to get this to work? (This is entirely possible and I apologise
>>>> for wasting your time if this is a simple user-error.)
>>>>
>>>> I have searched the ncl-talk archives and have not found this problem
>>>> encountered by others, so it may well be an issue exclusive to my machines.
>>>> If it is, I can work around it by wrapping in some fortran to handle the
>>>> reading and writing of the binary correctly. I would prefer to avoid such a
>>>> work around as the size of the arrays in the binary files I have to read
>>>> and write will vary, and I would prefer not to have to hard-code the array
>>>> sizes when NCL makes calls to the fortran code.
>>>>
>>>> [Note: on the machine running NCL 6.2.1 I also get the following
>>>> warning message on both attempts:
>>>> warning:fbinrecwrite: end of file reached before record number, writing
>>>> record as last record in file]
>>>>
>>>> Many thanks for your help,
>>>> Kay
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20160708/4aa288cc/attachment.html