[ncl-talk] binary file read
Debasish Hazra
debasish.hazra5 at gmail.com
Thu Mar 30 14:29:03 MDT 2017
Thanks Gus. Mary and myself both tried "endian" options, and presently
trying with
"setfileoption("bin","readbyteorder","bigendian") option which seems to
produce reasonable minimum and maximum of data values. However, as Mary
mentioned large number of values are constant whcih is bit strange.
You mentioned about "double" and I think input is in "double precision
floating point data and it is 8 bytes".
Thanks.
Debasish
On Thu, Mar 30, 2017 at 4:06 PM, Gus Correa <gus at ldeo.columbia.edu> wrote:
> Hi Mary, Debasish
>
> Could it be a little-endian vs. big-endian issue?
> I don't know IDL (I should! My boss uses it! :) )
> but their "read_binary" default endianness is "native" (like NCL).
> I.e., the endianness of the data on the file depends on the
> machine it was created (and data_type=5 is indeed double precision).
>
> Maybe using setfileoption('bin',"ReadByteOrder","BigEndian"),
> and trying also "LittleEndian" if not lucky with "Big"
> (who knows where the file was written ....),
> then cbinread/fbindirread with datatype "double" would help?
> Just a guess, and you probably tried the endianness thing already ...
>
> Best,
> Gus Correa
>
> On 03/30/2017 02:54 PM, Mary Haley wrote:
> > Hi Debasish,
> >
> > Dennis guess that maybe the "read_binary" function in IDL was meant to
> > read files created by "write_binary" but I didn't see a function with
> > that name. However, is it possible that this is some kind of special IDL
> > file and not a flat C binary file?
> >
> > In your IDL script, you have:
> >
> > fdata=read_binary('viirs_meandbdi_gridded_statis2013'+
> day+'.dat',data_type=5,data_dims=[1440,720])
> >
> > If you read the documentation for "read_binary", it states that
> > "data_type=5" is double.
> >
> > In your NCL script, you are reading the data as an unsigned integer.
> >
> > I tried reading your data as a double, but I get what looks like
> > nonsensical values:
> >
> > min=-1.642556686681977e+308 max=6.633924105807938e+307
> >
> > You are right that the unsigned integer values look reasonable, but only
> > after you multiply them by 1e-9.
> >
> > When I look at your unsigned values, I see that
> > 517,484
> > of your values are equal to the same number: 6.3615e-05, while only
> > 1,831
> > values are equal to something else.
> > This seems a bit suspicious to me, and is likely the source of the
> > problem.
> >
> > I modified your script to plot red markers where the values are all
> > equal to 6.3615e-05, and black markers everywhere else. Does this look
> > correct?
> >
> > I have a feeling that there's something more to the "read_binary"
> > function that we need to know in order to read the file correctly. As I
> > think I mentioned before: perhaps each byte of data represents something
> > different, and you need to use something like dim_gbits to pick off
> values.
> >
> > In your IDL script, is there anything you have to do additionally to the
> > data before you plot it? Can you check the IDL script to see if you are
> > getting a lot of values equal to the same constant value that NCL is?
> >
> > --Mary
> >
> >
> >
> > On Thu, Mar 30, 2017 at 8:36 AM, Debasish Hazra
> > <debasish.hazra5 at gmail.com <mailto:debasish.hazra5 at gmail.com>> wrote:
> >
> > Mary,
> >
> > Thanks.Taking your suggestion and reading that as 2 * 720 * 1440 and
> > assuming input as C binary file, I am getting min=1.4e-08
> > max=4.29371 , which is reasonble. Attached is the new script. Any
> > suggestions.
> >
> > Debasish
> >
> > On Wed, Mar 29, 2017 at 5:28 PM, Mary Haley <haley at ucar.edu
> > <mailto:haley at ucar.edu>> wrote:
> >
> > Hi Debasish,
> >
> > Kevin and I took a look at this. For starters, there *is* an
> > error message coming out of your script:
> >
> > warning:cbinread: The size implied by the dimension arrays is
> > greater that the size of the file.
> > The default _FillValue for the specified type will be filled in.
> > Note dimensions and values may not be aligned properly
> >
> > If you look at the size of the file, it doesn't match with the
> > dimensions you're requesting:
> >
> > Size of file = 8294400 bytes
> >
> > Size of dimensions = 5 * 720 * 1440 * 4 (for a uint) = 20736000
> >
> > If this is truly a C binary file, it looks like it only has 2 *
> > 720 * 1440 * 4 bytes.
> >
> > This doesn't really change the results, however, because you
> > still get two strange looking plots.
> >
> > We tried several different things:
> >
> > 1) reading the data as ubyte, int, and ushort
> > 2) reversing the array to 1440 x 720 x 2
> > 3) reading the data as little endian
> > 4) plotting the data as a simple contour plot to take out the
> > map component.
> >
> > Nothing we did produced more information about the file, or
> > produced better plots.
> >
> > Is there some documentation on this file to understand how it
> > was written? For example, are you sure the "uint" type is
> > correct? Are you sure the dimension sizes are correct? Why are
> > the values so large? Is it possible that this is "packed" data,
> > and that you need to use a function like dim_gbits to pick off
> > individual bits of information?
> >
> > If you can find a C or Fortran code that was used to create this
> > file, then it should be fairly straightforward to figure out how
> > to read it.
> >
> > --Mary
> >
> >
> >
> >
> >
> >
> > On Wed, Mar 29, 2017 at 2:18 PM, Debasish Hazra
> > <debasish.hazra5 at gmail.com <mailto:debasish.hazra5 at gmail.com>>
> > wrote:
> >
> > Hi,
> >
> > I am trying to read a binary file with the attached code,
> > but getting all empty fields in the figure with no apparent
> > error message. Uploaded the data file in the ftp server
> > "viirs_meandbdi_gridded_statis2015048.dat". Any help with
> > this is appreciated.
> >
> > Thanks.
> > Debasish
> >
> > On Wed, Mar 22, 2017 at 10:33 AM, Debasish Hazra
> > <debasish.hazra5 at gmail.com
> > <mailto:debasish.hazra5 at gmail.com>> wrote:
> >
> > Hi,
> >
> > I am trying to read a binary file with the attached
> > code, but getting all empty fields in the figure with
> > no apparent error message. Uploaded the data file in
> > the ftp server
> > "viirs_meandbdi_gridded_statis2015002.dat". Any help
> > with this is appreciated.
> >
> > Thanks.
> > Debasish.
> >
> >
> >
> >
> > _______________________________________________
> > ncl-talk mailing list
> > ncl-talk at ucar.edu <mailto:ncl-talk at ucar.edu>
> > List instructions, subscriber options, unsubscribe:
> > http://mailman.ucar.edu/mailman/listinfo/ncl-talk
> > <http://mailman.ucar.edu/mailman/listinfo/ncl-talk>
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > ncl-talk mailing list
> > ncl-talk at ucar.edu
> > List instructions, subscriber options, unsubscribe:
> > http://mailman.ucar.edu/mailman/listinfo/ncl-talk
> >
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20170330/7cef2556/attachment.html
More information about the ncl-talk
mailing list