[ncl-talk] binary file

[Dave Allured - NOAA Affiliate]

Oops, I see a mistake.  Earlier I said to use *fbinrecread*, but you used
*fbindirread*.  These do two different things.  Please use only
*fbinrecread*, including in instruction (1) that I just sent.


On Fri, Oct 9, 2020 at 4:11 PM Dave Allured - NOAA Affiliate <
dave.allured at noaa.gov> wrote:

> Debasish,
>
> (1)  Since you already know the dimensions of the 2-D array, save yourself
> some trouble and read the file directly into a new 2-D array by inserting
> the actual dimensions into the third argument of *fbindirread*.  Replace
> "-1" with (/nlat,mlon/), and remove the *onedtond* conversion.
>
> (2)  You have a plain binary file with no metadata.  NCL does not know the
> grid coordinates until you add them by one method or another.  This is why
> we prefer Netcdf files with metadata conveniently included.
>
> For one way to add coordinates, please see example 2 on the *Read binary
> data* example page.  In particular, notice the way that *fspan* is used
> to generate linear coordinate arrays.  You do not need to write a new
> Netcdf file like the example shows, but add lat and lon coordinate arrays
> before going to NCL plot functions.  Do you know what the coordinates are
> supposed to be?
>
> Use *printVarSummary* and *printMinMax* to evaluate your data array at
> different processing stages.  Use *printMinMax* to check that your data
> to plot is in the expected numeric range.  Also, missing value handling can
> be added if you think you might have missing values in the array.
>
>
> On Fri, Oct 9, 2020 at 3:21 PM Debasish Hazra <debasish.hazra5 at gmail.com>
> wrote:
>
>> Thanks Dave. When I use fbinrecread as shown below , I am still getting
>> warning and generated figure does not show any value.
>> setfileoption("bin","readbyteorder","littleendian")
>>  d1 = fbindirread("./Test/modelout.dat",0,-1,"float")
>>
>>            printVarSummary(d1)
>>            print(d1(0:100))
>>             nlat  = 384
>>            mlon  = 384
>>            UNDEF = -999.0
>>
>>            N1d   = dimsizes(d1)
>>             M1d   = N1d-(nlat*mlon)
>>            print("N1d="+N1d+"  M1d="+M1d)
>>
>>            x     = onedtond(d1(1:), (/nlat,mlon/))
>>            x at _FillValue = UNDEF                     ;
>>            printVarSummary(x)
>>           printMinMax(x,0)
>>
>>     Variable: d1
>> Type: float
>> Total Size: 589832 bytes
>>             147458 values
>> Number of Dimensions: 1
>> Dimensions and sizes:   [147458]
>> Coordinates:
>>
>> Variable: d1 (subsection)
>> Type: float
>> Total Size: 404 bytes
>>             101 values
>> Number of Dimensions: 1
>> Dimensions and sizes:   [101]
>> Coordinates:
>> (0)     8.265195e-40
>> (1)      0
>> .....[SNIP]
>>
>>
>> *(0)     N1d=147458  M1d=2warning:onedtond : output dimension sizes have
>> fewer elements than input, some data not copied*
>> Variable: x
>> Type: float
>> Total Size: 589824 bytes
>>             147456 values
>> Number of Dimensions: 2
>> Dimensions and sizes:   [384] x [384]
>> Coordinates:
>> Number Of Attributes: 1
>>   _FillValue :  -999
>> (0)     min=0   max=6.12579e-07,
>>
>> On Fri, Oct 9, 2020 at 2:55 PM Dave Allured - NOAA Affiliate <
>> dave.allured at noaa.gov> wrote:
>>
>>> That looks like fortran unformatted sequential format, not flat binary.
>>> Confirm by seeing if the first four bytes are 00 00 09 00, the same as the
>>> last four bytes that you showed.  This would be the record size as a
>>> little-endian 4-byte integer, which would be correct for this format.  This
>>> would also explain the 8-byte difference.
>>>
>>> Read in NCL with fbinrecread, not cbinread.  Please see the function
>>> docs and follow the prototype.
>>>
>>>
>>> On Fri, Oct 9, 2020 at 12:34 PM Rick Brownrigg via ncl-talk <
>>> ncl-talk at mailman.ucar.edu> wrote:
>>>
>>>> Well I don't think you want to read it as double, particularly not if
>>>> you are certain the dimensions are 384x384. I'm wondering if you can safely
>>>> ignore that last 8 bytes (and ignore the warning).  A quick plot would
>>>> reveal whether the data seem reasonable or definitely misaligned.
>>>>
>>>> RB
>>>>
>>>> On Fri, Oct 9, 2020 at 12:14 PM Debasish Hazra <
>>>> debasish.hazra5 at gmail.com> wrote:
>>>>
>>>>> Thanks Rick.Should I read it as double?
>>>>>
>>>>> On Fri, Oct 9, 2020 at 1:39 PM Rick Brownrigg <brownrig at ucar.edu>
>>>>> wrote:
>>>>>
>>>>>> Well, the math works out: 4x384x384=589824.  Further, the last 8
>>>>>> bytes of that file do not look like valid floating point values. They are:
>>>>>>
>>>>>>  00 00 00 00 00 00 09 00
>>>>>>
>>>>>> Do you have reason to believe the data is not being read correctly,
>>>>>> or are you mostly concerned about the warning message?
>>>>>>
>>>>>> BTW - the data are mostly zeros.
>>>>>>
>>>>>> Rick
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 9, 2020 at 11:09 AM Debasish Hazra via ncl-talk <
>>>>>> ncl-talk at mailman.ucar.edu> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>> I am trying to read binary file (attached in the email) through NCL
>>>>>>> and file size is 589832 bytes. I am reading it as float but it coming out
>>>>>>> to be 589824. I am not sure what needs to be done to read it.
>>>>>>>
>>>>>>>  setfileoption("bin","readbyteorder","littleendian")
>>>>>>>  d1 = cbinread("./Test/modelout.dat",(/4,384,384/),"float")
>>>>>>>
>>>>>>> warning:cbinread: The size implied by the dimension arrays is less
>>>>>>> that the size of the file.
>>>>>>>  Only the first 589824 contiguous bytes of the file will be read in.
>>>>>>> Note dimensions and values may not be aligned properly
>>>>>>>
>>>>>>> Any help on this is appreciated.
>>>>>>> Thanks
>>>>>>> Debasish
>>>>>>>
>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20201009/a5955997/attachment.html>