[ncl-talk] satellite hdf5 error for offset and scaling_factor

[Dennis Shea]

[1] short2flt_hdf uses the formula:     x_float = scale_factor*(x_short -
offset)
[2] short2flt uses the formula:            x_float = x_short*scale_factor
+ offset


netCDF conventions (COARDS, CF [Climate & Forecast]) *require* use of the
attribute names: scale_factor and add_offset. Further, they *require* use
of formula [2]. The netCDF conventions require use of the attributes
_FillValue or missing_value to identify values that should be ignored. The
type of the _FillValue/missing_value  attribute should match the type of
the variable with which it is associated. For example, a short variable
should have a _FillValue that is type short.

To my knowledge, HDF-4/5, have no common conventions. Likely, it is left up
to each project to define assorted information. This often makes dealing
with HDF variables that must be converted to a new numeric  a nuisance!! It
becomes the user's responsibility to determine how to get the correct
numbers.

NCL's function [1] and [2] try to accommodate 'aliases' for scale_factor
and add_offset. Based on past experience, the following are allowed.

 sNames = (/"scale", "SCALE", "Scale", "_scale", "scale_factor",
"factor"    \
           ,"Scale_factor", "Slope" , "slope", "ScaleFactor",
"Scale_Factor" \
           ,"SCALING_FACTOR" /)

Note: there is no 'Scaling_Factor'. Thus, the resulting data are not scaled
properly.
---

Also, looking at the 'printVarSummary' output shows no _FillValue or
missing_value attribute. Only the following:

   Undef_Code :    32767

As noted in NCL's documentation **only** _FillValue is recognized. User's
must make the appropriate changes.

---
Attached is a simple scrip. No graphics.

Output is as follows:

Variable: ts         <===== original variable
Type: *short*
Total Size: 5972192 bytes
            2986096 values
Number of Dimensions: 2
Dimensions and sizes:    [DIM_000 | 1544] x [DIM_001 | 1934]
Coordinates:
Number Of Attributes: 11
  Data_Type :    STD_I16LE
  Maximum :    305.7448
  Minimum :    245.3125
  Number_of_Bits_per_Pixel :    10
  Number_of_Columns :    1934
  Number_of_Lines :    1544
  *Offset* :    220
  *Scaling_Factor* :    0.1270772
  *Undef_Code* :    32767
  *Unit* :    K
  *Variable_Long_Name* :    Land_Surface_Temperature

(0)    min=199   max=32767           <=== ts min and max

(0)    typeof(ts at Undef_Code)    =integer
(0)    typeof(ts at Scaling_Factor)=float
(0)    typeof(ts at Offset)        =float
(0)    ---

Variable: TS
Type: float
Total Size: 11944384 bytes
            2986096 values
Number of Dimensions: 2
Dimensions and sizes:    [DIM_000 | 1544] x [DIM_001 | 1934]
Coordinates:
Number Of Attributes: 1
  _FillValue :    32767

(0)    min=245.288   max=305.777













On Sun, Jul 30, 2017 at 2:06 PM, Jin-Young Kim <ruwiejin at gmail.com> wrote:

> Dear Shea
>
> I've tried to read hdf5 format with short2flt_hdf for COMS satellite.
>
> But it looks that short2flt_hdf doesn't control offset and scaling_factor
> well.
>
> Would you look into the error ?
>
> Here is my error and tried script and original sample data.
>
> In addition, if it solve then i would like to try to read multiple file
> more than 1024 number.
>
> Would you give me any guidance ?
>
> Best regards,
> Jin
>
> --
> *Jin-Young Kim, **Ph.D *
>
> *한국에너지기술연구원 *
> *KIER(Korea Institute of Energy Research)*
> *이메일: jinyoung.kim at kier.re.kr <jinyoung.kim at kier.re.kr>*
> *사무실: +82-42-860-3734 <+82%2042-860-3734>*
> *휴대폰: +82-10-2990-7595 <+82%2010-2990-7595>*
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20170730/9bdd0ed0/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: coms_mi_le2.ncl
Type: application/octet-stream
Size: 786 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20170730/9bdd0ed0/attachment.obj