<div dir="ltr">I want to add that funny _FillValue is probably the result of writing "-999.9" somewhere in source code, then converting it from single to double precision in a separate operation. This is a common mistake. This is NOT the same value as "-999.9d" or equivalent in NCL, IDL, or many other other languages.<br><br>ncl 0> a = todouble (-999.9)<br>ncl 1> b = -999.9d<br><br>ncl 2> print (sprintf ("%20.13f", a)) <br>(0) -999.9000244140625<br>ncl 3> print (sprintf ("%20.13f", b)) <br>(0) -999.9000000000000<div class="gmail_extra"><br></div><div class="gmail_extra">I agree with Dave Brown's observations and suggested method to read. He sets the _FillValue with the maximum necessary number of significant digits for type double. Otherwise NCL would not understand all the missing values.</div><div class="gmail_extra"><br></div><div class="gmail_extra">--Dave A.</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 30, 2017 at 3:55 PM, David Brown <span dir="ltr"><<a href="mailto:dbrown@ucar.edu" target="_blank">dbrown@ucar.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">My guess is that this file is a 2D 720 x 1440 array of doubles in<br>
little endian format. There is no specific IDL formatting.<br>
The file size is 8294400 which is exactly equal to 720 x 1440 x 8. It<br>
contains may NaNs (not a number) and it also has what is presumably a<br>
_FillValue with the value -999.9000244140625.<br>
<br>
Here's how I would read it:<br>
<br>
setfileoption("bin","<wbr>ReadByteOrder","LittleEndian")<br>
d1 = cbinread("viirs_meandbdi_<wbr>gridded_statis2015002.dat",-1,<wbr>"double")<br>
<br>
Make an array of all the non-nan values (otherwise printMinMax will<br>
return NaN for both min and max)<br>
d1ind = ind(.not. isnan_ieee(d1))<br>
d1x = d1(d1ind)<br>
printMinMax(d1x,0)<br>
output: (0) min=-999.9000244140625 max=5.164013057067244<br>
<br>
If you scroll through the variable d1x values you will see that the<br>
min value is clearly an outlier and therefore is most likely a fill<br>
value.<br>
So set the _FillValue<br>
d1x@_FillValue = -999.9000244140625<br>
Now<br>
printMinMax(d1x,0)<br>
output: (0) min=0.2350846065940295 max=5.164013057067244<br>
<br>
Hopefully these are reasonable values.<br>
<br>
Now set the _FillValue for the original data and turn the NaNs into _FillValue<br>
d1@_FillValue = d1x@_FillValue<br>
d1 = where(isnan_ieee(d1),d1@_<wbr>FillValue, d1)<br>
ncl 73> printMinMax(d1,0)<br>
(0) min=0.2350846065940295 max=5.164013057067244<br>
<br>
But note out of the whole array there are not very many valid values:<br>
<br>
ncl 74> printVarSummary(d1)<br>
Variable: d1<br>
Type: double<br>
Total Size: 8294400 bytes<br>
1036800 values<br>
Number of Dimensions: 1<br>
Dimensions and sizes: [1036800]<br>
Coordinates:<br>
Number Of Attributes: 1<br>
_FillValue : -999.9000244140625<br>
<br>
ncl 75> print(num(.not. ismissing(d1)))<br>
(0) 1820<br>
<br>
Nevertheless I believe this is the correct interpretation of this dataset.<br>
-dave<br><br>
<br>
On Thu, Mar 30, 2017 at 2:29 PM, Debasish Hazra<br>
<<a href="mailto:debasish.hazra5@gmail.com">debasish.hazra5@gmail.com</a>> wrote:<br>
> Thanks Gus. Mary and myself both tried "endian" options, and presently<br>
> trying with<br>
><br>
> "setfileoption("bin","<wbr>readbyteorder","bigendian") option which seems to<br>
> produce reasonable minimum and maximum of data values. However, as Mary<br>
> mentioned large number of values are constant whcih is bit strange.<br>
><br>
> You mentioned about "double" and I think input is in "double precision<br>
> floating point data and it is 8 bytes".<br>
><br>
> Thanks.<br>
> Debasish<br>
><br>
> On Thu, Mar 30, 2017 at 4:06 PM, Gus Correa <<a href="mailto:gus@ldeo.columbia.edu">gus@ldeo.columbia.edu</a>> wrote:<br>
>><br>
>> Hi Mary, Debasish<br>
>><br>
>> Could it be a little-endian vs. big-endian issue?<br>
>> I don't know IDL (I should! My boss uses it! :) )<br>
>> but their "read_binary" default endianness is "native" (like NCL).<br>
>> I.e., the endianness of the data on the file depends on the<br>
>> machine it was created (and data_type=5 is indeed double precision).<br>
>><br>
>> Maybe using setfileoption('bin',"<wbr>ReadByteOrder","BigEndian"),<br>
>> and trying also "LittleEndian" if not lucky with "Big"<br>
>> (who knows where the file was written ....),<br>
>> then cbinread/fbindirread with datatype "double" would help?<br>
>> Just a guess, and you probably tried the endianness thing already ...<br>
>><br>
>> Best,<br>
>> Gus Correa<br>
>><br>
>> On 03/30/2017 02:54 PM, Mary Haley wrote:<br>
>> > Hi Debasish,<br>
>> ><br>
>> > Dennis guess that maybe the "read_binary" function in IDL was meant to<br>
>> > read files created by "write_binary" but I didn't see a function with<br>
>> > that name. However, is it possible that this is some kind of special IDL<br>
>> > file and not a flat C binary file?<br>
>> ><br>
>> > In your IDL script, you have:<br>
>> ><br>
>> ><br>
>> > fdata=read_binary('viirs_<wbr>meandbdi_gridded_statis2013'+<wbr>day+'.dat',data_type=5,data_<wbr>dims=[1440,720])<br>
>> ><br>
>> > If you read the documentation for "read_binary", it states that<br>
>> > "data_type=5" is double.<br>
>> ><br>
>> > In your NCL script, you are reading the data as an unsigned integer.<br>
>> ><br>
>> > I tried reading your data as a double, but I get what looks like<br>
>> > nonsensical values:<br>
>> ><br>
>> > min=-1.642556686681977e+308 max=6.633924105807938e+307<br>
>> ><br>
>> > You are right that the unsigned integer values look reasonable, but only<br>
>> > after you multiply them by 1e-9.<br>
>> ><br>
>> > When I look at your unsigned values, I see that<br>
>> > 517,484<br>
>> > of your values are equal to the same number: 6.3615e-05, while only<br>
>> > 1,831<br>
>> > values are equal to something else.<br>
>> > This seems a bit suspicious to me, and is likely the source of the<br>
>> > problem.<br>
>> ><br>
>> > I modified your script to plot red markers where the values are all<br>
>> > equal to 6.3615e-05, and black markers everywhere else. Does this look<br>
>> > correct?<br>
>> ><br>
>> > I have a feeling that there's something more to the "read_binary"<br>
>> > function that we need to know in order to read the file correctly. As I<br>
>> > think I mentioned before: perhaps each byte of data represents something<br>
>> > different, and you need to use something like dim_gbits to pick off<br>
>> > values.<br>
>> ><br>
>> > In your IDL script, is there anything you have to do additionally to the<br>
>> > data before you plot it? Can you check the IDL script to see if you are<br>
>> > getting a lot of values equal to the same constant value that NCL is?<br>
>> ><br>
>> > --Mary<br>
>> ><br>
>> ><br>
>> ><br>
>> > On Thu, Mar 30, 2017 at 8:36 AM, Debasish Hazra<br>
>> > <<a href="mailto:debasish.hazra5@gmail.com">debasish.hazra5@gmail.com</a> <mailto:<a href="mailto:debasish.hazra5@gmail.com">debasish.hazra5@gmail.<wbr>com</a>>> wrote:<br>
>> ><br>
>> > Mary,<br>
>> ><br>
>> > Thanks.Taking your suggestion and reading that as 2 * 720 * 1440 and<br>
>> > assuming input as C binary file, I am getting min=1.4e-08<br>
>> > max=4.29371 , which is reasonble. Attached is the new script. Any<br>
>> > suggestions.<br>
>> ><br>
>> > Debasish<br>
>> ><br>
>> > On Wed, Mar 29, 2017 at 5:28 PM, Mary Haley <<a href="mailto:haley@ucar.edu">haley@ucar.edu</a><br>
>> > <mailto:<a href="mailto:haley@ucar.edu">haley@ucar.edu</a>>> wrote:<br>
>> ><br>
>> > Hi Debasish,<br>
>> ><br>
>> > Kevin and I took a look at this. For starters, there *is* an<br>
>> > error message coming out of your script:<br>
>> ><br>
>> > warning:cbinread: The size implied by the dimension arrays is<br>
>> > greater that the size of the file.<br>
>> > The default _FillValue for the specified type will be filled in.<br>
>> > Note dimensions and values may not be aligned properly<br>
>> ><br>
>> > If you look at the size of the file, it doesn't match with the<br>
>> > dimensions you're requesting:<br>
>> ><br>
>> > Size of file = 8294400 bytes<br>
>> ><br>
>> > Size of dimensions = 5 * 720 * 1440 * 4 (for a uint) = 20736000<br>
>> ><br>
>> > If this is truly a C binary file, it looks like it only has 2 *<br>
>> > 720 * 1440 * 4 bytes.<br>
>> ><br>
>> > This doesn't really change the results, however, because you<br>
>> > still get two strange looking plots.<br>
>> ><br>
>> > We tried several different things:<br>
>> ><br>
>> > 1) reading the data as ubyte, int, and ushort<br>
>> > 2) reversing the array to 1440 x 720 x 2<br>
>> > 3) reading the data as little endian<br>
>> > 4) plotting the data as a simple contour plot to take out the<br>
>> > map component.<br>
>> ><br>
>> > Nothing we did produced more information about the file, or<br>
>> > produced better plots.<br>
>> ><br>
>> > Is there some documentation on this file to understand how it<br>
>> > was written? For example, are you sure the "uint" type is<br>
>> > correct? Are you sure the dimension sizes are correct? Why are<br>
>> > the values so large? Is it possible that this is "packed" data,<br>
>> > and that you need to use a function like dim_gbits to pick off<br>
>> > individual bits of information?<br>
>> ><br>
>> > If you can find a C or Fortran code that was used to create this<br>
>> > file, then it should be fairly straightforward to figure out how<br>
>> > to read it.<br>
>> ><br>
>> > --Mary<br>
>> ><br>
>> ><br>
>> > On Wed, Mar 29, 2017 at 2:18 PM, Debasish Hazra<br>
>> > <<a href="mailto:debasish.hazra5@gmail.com">debasish.hazra5@gmail.com</a> <mailto:<a href="mailto:debasish.hazra5@gmail.com">debasish.hazra5@gmail.<wbr>com</a>>><br>
>> > wrote:<br>
>> ><br>
>> > Hi,<br>
>> ><br>
>> > I am trying to read a binary file with the attached code,<br>
>> > but getting all empty fields in the figure with no apparent<br>
>> > error message. Uploaded the data file in the ftp server<br>
>> > "viirs_meandbdi_gridded_<wbr>statis2015048.dat". Any help with<br>
>> > this is appreciated.<br>
>> ><br>
>> > Thanks.<br>
>> > Debasish<br>
>> ><br>
>> > On Wed, Mar 22, 2017 at 10:33 AM, Debasish Hazra<br>
>> > <<a href="mailto:debasish.hazra5@gmail.com">debasish.hazra5@gmail.com</a><br>
>> > <mailto:<a href="mailto:debasish.hazra5@gmail.com">debasish.hazra5@gmail.<wbr>com</a>>> wrote:<br>
>> ><br>
>> > Hi,<br>
>> ><br>
>> > I am trying to read a binary file with the attached<br>
>> > code, but getting all empty fields in the figure with<br>
>> > no apparent error message. Uploaded the data file in<br>
>> > the ftp server<br>
>> > "viirs_meandbdi_gridded_<wbr>statis2015002.dat". Any help<br>
>> > with this is appreciated.<br>
>> ><br>
>> > Thanks.<br>
>> > Debasish.<br></blockquote></div></div></div>