[ncl-talk] writing out ushort and uint to netcdf4

[David Brown]

Hi Archana,
After looking at your script and the files, I can see that you are
right that the ushort and uint values in the "_FillValue" and
"valid_range" attributes of the output NetCDF4 file are not being
printed correctly by ncl_filedump. However, I am also pretty sure
that, as you stated, the values are actually being written correctly
into the NetCDF4 file. I have 2 reasons for my confidence:
1) Using the Unidata ncdump tool, ncdump -h give the correct (i.e. the
same as the original HDF) numbers for these attributes.
2) Reading the file into NCL I get the correct values using:
printVarSummary(f->QUFLAG)
while I get incorrect values using:
printFileVarSummary(f,"QUFLAG")

There are separate printing routines for printVarSummary and
printFileVarSummary. And the printFileVarSummary routine seems to be
the one used for the ncl_filedump. So I think the problem is in the
code called by printFileVarSummary. Thanks for pointing this out. I
would recommend using ncdump as a check anytime you suspect an issue
with NCL NetCDF output.

I will create a ticket for this issue.
 -dave




On Wed, Nov 4, 2015 at 5:30 PM, Archana Dayalu <adayalu at seas.harvard.edu> wrote:
> Hi there,
> I am trying to write out some MODIS09A1 variables (QA flags) to netcdf4 (not
> the classic model). The QA flag and SQA flag are in uint and ushort format
> respectively. I am using NCLv6.2.1. I've correctly set my file option to
> NetCDF4 prior to creating the ncdf file.
> The problem is, in its native hdf format and when I read it in to ncl I get
> the following:
>
> ;;;;;;;;;;;;;;;;;;;;; QAFLAG ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
> Variable: QAFLAG
> Type: uint
> Total Size: 645120000 bytes
>             161280000 values
> Number of Dimensions: 2
> Dimensions and sizes: [YDim_MOD_Grid_500m_Surface_Reflectance | 9600] x
> [XDim_MOD_Grid_500m_Surface_Reflectance | 16800]
> Coordinates:
> Number Of Attributes: 7
>   _FillValue : 4294967295
>   long_name : Surface_reflectance_500m_quality_control_flags
>   units : bit field
>   valid_range : ( 0, 4294966531 )
>
>
> ;;;;;;;;;;;;;;;;;;;;;;;;; SQAFLAG ;;;;;;;;;;;;;;;;;;;;;;;
> Variable: SQAFLAG
> Type: ushort
> Total Size: 322560000 bytes
>             161280000 values
> Number of Dimensions: 2
> Dimensions and sizes: [YDim_MOD_Grid_500m_Surface_Reflectance | 9600] x
> [XDim_MOD_Grid_500m_Surface_Reflectance | 16800]
> Coordinates:
> Number Of Attributes: 7
>   _FillValue : 65535
>   long_name : Surface_reflectance_500m_state_flags
>   units : bit field
>   valid_range : ( 0, 57343 )
>
> ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
>
>
> But, when I write out these variables to netcdf4 and then view them using
> ncl_filedump, they are output as QAFLAG as float (and I don't specify this
> anywhere) and SQAFLAG as something else (doesn't look like float?):
>
> ;;;;;;;;;;;;;;;;;;;;; QAFLAG ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
>     Variable: QAFLAG
>     Type: uint
>     Total Size: 161280000 values
>                 645120000 bytes
>     Number of Dimensions: 2
>     Dimensions and sizes: [ 9600 <YDim> x 16800 <XDim> ]
>     Coordinates:
>         Number of Attributes:        8
>             missing_value : -1
>             _FillValue : -1
>             long_name : sur_refl_qc_500m
>             units : bit field
>             valid_range : ( 0, -765 )
>
> ;;;;;;;;;;;;;;;;;;;;; SQAFLAG ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
> Variable: SQAFLAG
>     Type: ushort
>     Total Size: 161280000 values
>                 322560000 bytes
>     Number of Dimensions: 2
>     Dimensions and sizes: [ 9600 <YDim> x 16800 <XDim> ]
>     Coordinates:
>         Number of Attributes:        8
>             missing_value :
>             _FillValue :
>             long_name : sur_refl_state_500m
>             units : bit field
>             valid_range : ( 0, -8193 )
>             QA_bitmap_index :
>
>
> ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
>
> However, when I read these converted .nc files *back* into NCL, it's all
> fine ... and looks like the output from the native hdf file. Could you
> explain why this is happening? Does it have something to do with the data
> compression? I just want to make sure it is not impacting the contents of
> the variables.
>
> (On a side note, the reason I am going through all this is I am eventually
> reading these objects into R to QC filter my MODIS reflectance data. Is
> there pre-existing NCL code that allows for application of QC criteria
> within ncl? It might be faster and would avoid all this ... )
>
> Thanks for any help!
>
> Regards,
> Archana
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>