[ncl-talk] help reading binary file

[Dave Allured - NOAA Affiliate]

Laura,

1.  Please notice that your array size 43200 * 21600 * 4-byte floats
exceeds the conventional 2 Gbyte limit for normal fortran unformatted
sequential record size.  This may be the cause of "0 or less than zero
fortran control word".  Some compilers have schemes for larger record size,
but they are not standardized and can be troublesome.

2.  There are several solutions.  I recommend keeping the ordinary
fortran unformatted sequential format because it seems you are most
familiar with it.  But loop over Y, and write your array in pieces
sequentially, one Y subscript at a time.  This will write 21600 records
with reasonable record size.

3.  Check your output file before attempting to read with NCL.  The size of
this new file should be exactly 21600 * (43200 + 1 + 1) * 4 bytes in size,
including two standard fortran control words for each record.

4.  In NCL, pre-dimension your input array.  Then read this new binary
sequential file one record at a time, looping over Y, the same as it was
written.  Be sure to use the fbinrecread function for reading binary
sequential format, multiple records.  Untested:

nx = 43200
ny = 21600
oro_usgs = new ( (/ nx, ny /), float)
do y = 0, ny-1
   oro_usgs(:,y) = fbinrecread (fili, y, nx, "float")
end do

5.  If you are still having trouble, try writing and reading only a single
record (one Y subscript).  This will make a small test file and simplify
debugging.

6.  Unless you are constrained in some way by other issues in your fortran
program, please avoid reading and writing this file as big endian.  Modern
Macs are little endian machines, and they do well with their native
formats.  Reverse endian may be adding confusion needlessly.

7.  Consider using "od" or another binary viewing program to inspect the
contents of your file.  See this write-up for more details on investigating
binary formats:

   http://www.ncl.ucar.edu/Applications/r-binary.shtml

--Dave


On Mon, Nov 23, 2015 at 5:55 PM, Laura Fowler <laura at ucar.edu> wrote:

> Hi Dennis:
> Unfortunately, reading the fortran binary file still does not work. I
> rewrote the binary files as flat binary files, or
> open(unit=21,file='output.oro_usgs.bin',form='unformatted')
>
> so in my ncl script,  I have:
> oro_usgs = fbinrecread(file1,0,-1,"float")
>
> since I do not specify the length of the array oro_usgs. Since I had
> to use the option -fconvert=big-endian to read some data, I also tried
> to use the option:
>
> setfileoption("bin","ReadByteOrder","BigEndian")
>
> but that did not help either. I am really puzzled as what to do next
> since I can read that new file with a simple fortran unformatted read.
> What else do you suggest?
>
> Thanks,
> Laura
>
>
> On Fri, Nov 20, 2015 at 5:53 PM, Dennis Shea <shea at ucar.edu> wrote:
> > fortran binary writes can create 'flat' binary files (access='direct')
> > or access='sequential'. The latter is the default.
> >
> > To read 'flat' files use: fbinread
> >
> > To read 'sequential' (which you explicitly specified) use: fbinrecread
> >
> > ===
> > Note: fortran precedes *each* sequential record by a  record separator.
> >
> > Good luck
> >
> >
> > On Fri, Nov 20, 2015 at 5:25 PM, Laura Fowler <laura at ucar.edu> wrote:
> >> Hi:
> >>
> >> I created a binary file using a fortran sourcecode on my iMac. Using
> >> gfortran, I used the option -fconvert=big-endian tom compie the
> >> sourcecode. I used the open statement:
> >>
> >> open(unit=21,file='oro_usgs.bin',form='unformatted',access='sequential')
> >>
> >> to write the data out. I want to open and read the file in an ncl
> >> script to later plot the data. My script looks like that:
> >>
> >> setfileoption("bin","ReadByteOrder","BigEndian")
> >> file1    = "oro_usgs.bin"
> >> recl     = 43200*21600
> >> oro_usgs = fbinread(file1,recl,"float")
> >> print(max(oro_usgs))
> >> printVarSummary(oro_usgs)
> >>
> >> I get the error:
> >>  See http://www.ncl.ucar.edu/ for more details.
> >>  fatal:fbinread: 0 or less than zero fortran control word, FILE NOT
> >> SEQUENTIAL ACCESS!
> >>  fatal:["Execute.c":8128]:Execute: Error occurred at or near line 17
> >> in file plot.ncl
> >>
> >> without the setfileoption, I get the error:
> >>
> >> See http://www.ncl.ucar.edu/ for more details.
> >> fatal:fbinread: requested variable size exceeds record size
> >> fatal:["Execute.c":8128]:Execute: Error occurred at or near line 17 in
> >> file plot.ncl
> >>
> >> I do not know what to do next and help would be greatly appreciated.
> Thanks,
> >>
> >> Laura
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20151123/496f81cf/attachment.html