[ncl-talk] hdf_to_netcdf
Dennis Shea
shea at ucar.edu
Wed Aug 15 14:58:49 MDT 2018
https://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/NclFormatSupport.shtml#HDF-EOS
.he2 <===> .hdfeos ... they both mean the same thing
===
.*hdf* are the original format coding. HDF variable names can contain
characters not recognized by NCL.
.he2 will automatically this situation.
====
.he2 recognizes the
*SwathName*
StructMetadata_0 : GROUP=SwathStructure
GROUP=SWATH_1
*SwathName="mod04"*
[sni]
HDF files can can contain multiple swaths whch will contian the same
variable names.
.he2 will automatically append the swath name to the variable to
differentiate
Deep_Blue_Aerosol_Optical_Depth_550_Land_mod04
Deep_Blue_Aerosol_Optical_Depth_550_Land_mod05
Deep_Blue_Aerosol_Optical_Depth_550_Land_mod06
.hdf requires the user to specify the full path [including group_name] to
access the variable.
For this data use .he2
On Wed, Aug 15, 2018 at 12:02 AM, Kunal Bali <kunal.bali9 at gmail.com> wrote:
> yes sir/ma'am, I did look at the file summary. and that's what I want know
> why it's giving two different variable names.
>
> whats the difference between .hdf and .he2. if we can get the variable
> name from .hdf then why are we using .he2? what is the use of .he2 over
> .hdf?
>
> could you please provide some information on that.
>
> regards
> Kunal Bali
>
>
>
>
>
>
> On Wed, Aug 15, 2018 at 2:28 AM Dennis Shea <shea at ucar.edu> wrote:
>
>> Did you look at the script?
>>
>> I added the following specifically to help you. Using .hdf and .he2
>> yields slightly different variable names. The .he2 appends _mod04 to each
>> variable name.
>>
>>
>> *;*vName= "Deep_Blue_Aerosol_Optical_Depth_550_Land_mod04" ; *".hdf.he2"
>> *
>> vName= "Deep_Blue_Aerosol_Optical_Depth_550_Land" ; "
>> *.hdf*"
>> ....
>> pthi = diri + fili
>> *;;*pthi = pthi + ".he2" ; use .he2 extension
>>
>> Please do a
>>
>> *%> ncl_filedump* MOD04_L2.A2017060.0510.006.2017060225118*.hdf *| less
>>
>> The *.hdf *variable name is: Deep_Blue_Aerosol_Optical_Depth_550_Land
>>
>> and
>>
>> *%> ncl_filedump* MOD04_L2.A2017060.0510.006.2017060225118*.hdf.he2* |
>> less
>>
>> The *.hdf.he2* variable name is: Deep_Blue_Aerosol_Optical_
>> Depth_550_Land_mod04
>>
>> ====
>>
>>
>> On Tue, Aug 14, 2018 at 12:07 AM, Kunal Bali <kunal.bali9 at gmail.com>
>> wrote:
>>
>>> First of all thanks a lot, Sir. You have been spending so much time on
>>> my work. I appreciate your time.
>>>
>>> ........
>>>
>>> I have one query regarding this script.
>>> Here you use the variable name
>>> vName= "Deep_Blue_Aerosol_Optical_Depth_550_Land"
>>>
>>> not
>>>
>>> vName= "Deep_Blue_Aerosol_Optical_Depth_550_Land_*mod04*"
>>>
>>> why is it so?
>>>
>>> Also, the output is coming with the values -9999
>>>
>>> AOT at 0.55 micron for land with all quality data (Quality flag=1,2,3)
>>> (None) : min=-9999 max=-9999
>>>
>>>
>>>
>>>
>>> ---
>>> Kunal Bali
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Aug 14, 2018 at 5:42 AM Dennis Shea <shea at ucar.edu> wrote:
>>>
>>>> You must do more debugging on your own. It takes time to look at your
>>>> issues.
>>>>
>>>> ===
>>>> re: "But the files have the given variable then what would be the
>>>> reason for this warning?"
>>>>
>>>> NCL is telling you the issue:
>>>> Warning:File ./MOD04_L2.A2017087.0450.006.2017087134018.hdf *dimension
>>>> sizes do not conform to others in list;* skipping file
>>>>
>>>> What should you do?
>>>>
>>>> Well, of course, look at the *dimension sizes* on each file.
>>>> ncl-talk always tells people to "look at the data" ... assume nothing."
>>>>
>>>> Please see: the attachments:
>>>>
>>>> *%> ncl* MOD04_dimsizes.ncl >&! out.MOD04_dimsizes
>>>>
>>>> Some files have soil_moisture with dimension sizes: (/*203*,135/)
>>>> while others are (/*204*,135/)
>>>>
>>>> *addfiles
>>>> <https://www.ncl.ucar.edu/Document/Functions/Built-in/addfiles.shtml>*
>>>> expects *consistent *dimension sizes. *addfiles* ["*join*"] across 60
>>>> files would expect *all*
>>>>
>>>> *all* (60, 203, 135)
>>>> or
>>>> *all* (60, 204, 135)
>>>>
>>>> It can *not* do
>>>>
>>>> (60, *203 or 204*, 135)
>>>>
>>>>
>>>> =======================
>>>> I am attaching a script that makes netCDF files. One netCDF per input
>>>> HDF file.
>>>>
>>>> %> ncl MOD04_to_netCDF.ncl
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Aug 13, 2018 at 1:56 AM, Kunal Bali <kunal.bali9 at gmail.com>
>>>> wrote:
>>>>
>>>>> Sir,
>>>>>
>>>>> sorry for sending this offline message.
>>>>>
>>>>>
>>>>> After running your script for the attached file list. I still gets the
>>>>> same warning
>>>>>
>>>>> warning:File ./MOD04_L2.A2017087.0450.006.2017087134018.hdf dimension
>>>>> sizes do not conform to others in list; skipping file
>>>>> warning:File ./MOD04_L2.A2017082.0435.006.2017082135321.hdf dimension
>>>>> sizes do not conform to others in list; skipping file
>>>>> warning:File ./MOD04_L2.A2017078.0500.006.2017081195531.hdf dimension
>>>>> sizes do not conform to others in list; skipping file
>>>>> warning:File ./MOD04_L2.A2017074.0520.006.2017074182316.hdf dimension
>>>>> sizes do not conform to others in list; skipping file
>>>>> warning:File ./MOD04_L2.A2017069.0505.006.2017069135117.hdf dimension
>>>>> sizes do not conform to others in list; skipping file
>>>>> warning:File ./MOD04_L2.A2017060.0510.006.2017060225118.hdf dimension
>>>>> sizes do not conform to others in list; skipping file
>>>>> warning:*A** valid instance of variable
>>>>> Deep_Blue_Aerosol_Optical_Depth_550_Land_mod04 was not found in one or more
>>>>> elements of the file list*
>>>>>
>>>>> But the files have the given variable then what would be the reason
>>>>> for this warning?
>>>>>
>>>>> and also, when I run the same script for the given skipping file
>>>>> (separately in the new directory) then it it gives warning with one
>>>>> skipping file.
>>>>>
>>>>> warning:File ./MOD04_L2.A2017073.0440.006.2017073134018.hdf dimension
>>>>> sizes do not conform to others in list; skipping file
>>>>> warning:A valid instance of variable Deep_Blue_Aerosol_Optical_Depth_550_Land_mod04
>>>>> was not found in one or more elements of the file list
>>>>>
>>>>>
>>>>> Could you please provide some thoughts on that as well.
>>>>>
>>>>>
>>>>> and sorry for sending this offline mail.
>>>>>
>>>>>
>>>>>
>>>>> ---
>>>>> Kunal Bali
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [image: Mailtrack]
>>>>> <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality1&> Sent
>>>>> with Mailtrack
>>>>> <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality1&>
>>>>>
>>>>> On Thu, Aug 9, 2018 at 10:14 AM Kunal Bali <kunal.bali9 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Dear Sir,
>>>>>> Thanks for the explanation and spending some time on this query.
>>>>>> Thank You so much.
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---
>>>>>> Kunal Bali
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 9, 2018 at 6:00 AM Dennis Shea <shea at ucar.edu> wrote:
>>>>>>
>>>>>>> *This is offline from ncl-talk*
>>>>>>>
>>>>>>> I had a few MOD04_L2 files so I created a basic script with many
>>>>>>> print statements.
>>>>>>> It shows multiple ways to define 'time.'
>>>>>>>
>>>>>>> Please run on your MOD04_L2.
>>>>>>> Look carefully at the output. In particular, the various 'time'
>>>>>>> variables.
>>>>>>> I suppose you use 'time_file' used rather that 'time_scan'
>>>>>>>
>>>>>>> This is *swath* data so while the latitude/longitude *dimension
>>>>>>> sizes *on each file are the same, it is not likely that the actual
>>>>>>> latitude and longitude values are the same. Hence, there is a 'time'
>>>>>>> associated with the lat/lon arrays.
>>>>>>>
>>>>>>> ============
>>>>>>>
>>>>>>> Good Luck
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Aug 8, 2018 at 9:20 AM, Dennis Shea <shea at ucar.edu> wrote:
>>>>>>>
>>>>>>>> Your 'summary.txt' contains:
>>>>>>>>
>>>>>>>> double* Scan_Start_Time* ( Cell_Along_Swath_mod04,
>>>>>>>> Cell_Across_Swath_mod04 )
>>>>>>>> valid_range : ( 0, 3155800000 )
>>>>>>>> _FillValue : -999
>>>>>>>> long_name : TAI Time at Start of Scan replicated across
>>>>>>>> the swath
>>>>>>>> units : Seconds since 1993-1-1 00:00:00.0 0
>>>>>>>> scale_factor : 1
>>>>>>>> add_offset : 0
>>>>>>>> Parameter_Type : MODIS Input
>>>>>>>> Cell_Along_Swath_Sampling : ( 1, 2021, 10 )
>>>>>>>> Cell_Across_Swath_Sampling : ( 1, 1345, 10 )
>>>>>>>> Geolocation_Pointer : Internal geolocation arrays
>>>>>>>> hdf_name : Scan_Start_Time
>>>>>>>>
>>>>>>>> This 'time' and it is *two-dimensional*.
>>>>>>>>
>>>>>>>> -----
>>>>>>>>
>>>>>>>> The 'time' you are trying to create is *one-dimensional *and is
>>>>>>>> based strictly on a file name.
>>>>>>>>
>>>>>>>> ----
>>>>>>>> Also, the following may be if interest
>>>>>>>>
>>>>>>>> https://www.ncl.ucar.edu/Applications/HDF.shtml
>>>>>>>> hdf4sds_5.ncl
>>>>>>>> <https://www.ncl.ucar.edu/Applications/Scripts/hdf4sds_5.ncl>
>>>>>>>>
>>>>>>>> ---
>>>>>>>> These files are HDF-EOS format:
>>>>>>>>
>>>>>>>> *HDFEOSVersion : HDFEOS_V2.19*
>>>>>>>> For fun, append a *.he2* file extension:
>>>>>>>>
>>>>>>>> %>* ncl_filedump *MOD04_L2.A2017202.0705.061.2017316182248.hdf
>>>>>>>> *.he2* | less
>>>>>>>>
>>>>>>>> It makes the file look like a netCDF file dump.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Aug 8, 2018 at 3:50 AM, Kunal Bali <kunal.bali9 at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Dear NCL user,
>>>>>>>>>
>>>>>>>>> I am converting hdf to netcdf using attached ncl script
>>>>>>>>> (extracting aerosol product). But it's showing some errors and values are
>>>>>>>>> also coming from +01. to -0.1.
>>>>>>>>> Summary of the file is also attached herewith.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> the errors/warning are:
>>>>>>>>>
>>>>>>>>> warning:File ./MOD04_L2.A2017209.0710.061.2017317003637.hdf
>>>>>>>>> dimension sizes do not conform to others in list; skipping file
>>>>>>>>> warning:File ./MOD04_L2.A2017209.0525.061.2017317002757.hdf
>>>>>>>>> dimension sizes do not conform to others in list; skipping file
>>>>>>>>> warning:File ./MOD04_L2.A2017205.0550.061.2017316194801.hdf
>>>>>>>>> dimension sizes do not conform to others in list; skipping file
>>>>>>>>> warning:File ./MOD04_L2.A2017200.0355.061.2017316192232.hdf
>>>>>>>>> dimension sizes do not conform to others in list; skipping file
>>>>>>>>> warning:File ./MOD04_L2.A2017196.0600.061.2017316164649.hdf
>>>>>>>>> dimension sizes do not conform to others in list; skipping file
>>>>>>>>> warning:File ./MOD04_L2.A2017195.0340.061.2017316165535.hdf
>>>>>>>>> dimension sizes do not conform to others in list; skipping file
>>>>>>>>> warning:File ./MOD04_L2.A2017192.0625.061.2017316024320.hdf
>>>>>>>>> dimension sizes do not conform to others in list; skipping file
>>>>>>>>> warning:File ./MOD04_L2.A2017191.0405.061.2017316013818.hdf
>>>>>>>>> dimension sizes do not conform to others in list; skipping file
>>>>>>>>> warning:File ./MOD04_L2.A2017188.0645.061.2017316002323.hdf
>>>>>>>>> dimension sizes do not conform to others in list; skipping file
>>>>>>>>> warning:File ./MOD04_L2.A2017187.0610.061.2017315234444.hdf
>>>>>>>>> dimension sizes do not conform to others in list; skipping file
>>>>>>>>> warning:File ./MOD04_L2.A2017187.0425.061.2017315234452.hdf
>>>>>>>>> dimension sizes do not conform to others in list; skipping file
>>>>>>>>> warning:File ./MOD04_L2.A2017184.0710.061.2017315214958.hdf
>>>>>>>>> dimension sizes do not conform to others in list; skipping file
>>>>>>>>> warning:File ./MOD04_L2.A2017183.0450.061.2017315215216.hdf
>>>>>>>>> dimension sizes do not conform to others in list; skipping file
>>>>>>>>> warning:A valid instance of variable Deep_Blue_Aerosol_Optical_Depth_550_Land_mod04
>>>>>>>>> was not found in one or more elements of the file list
>>>>>>>>> fatal:Coordinate variables must be the same dimension as their
>>>>>>>>> dimension
>>>>>>>>> fatal:No coordinate variable exists for dimension (time) in
>>>>>>>>> variable (var)
>>>>>>>>> fatal:["Execute.c":8640]:Execute: Error occurred at or near line
>>>>>>>>> 35 in file script_01.ncl
>>>>>>>>>
>>>>>>>>> fatal:["Execute.c":8640]:Execute: Error occurred at or near line
>>>>>>>>> 69 in file script_01.ncl
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Variable: od55_1km
>>>>>>>>> Type: short
>>>>>>>>> Total Size: 6303150 bytes
>>>>>>>>> 3151575 values
>>>>>>>>> Number of Dimensions: 3
>>>>>>>>> Dimensions and sizes: [time | 115] x [Cell_Along_Swath_mod04 |
>>>>>>>>> 203] x [Cell_Across_Swath_mod04 | 135]
>>>>>>>>> Coordinates:
>>>>>>>>> Number Of Attributes: 12
>>>>>>>>> coordinates : Latitude_mod04, Longitude_mod04
>>>>>>>>> hdfeos_name : Deep_Blue_Aerosol_Optical_Depth_550_Land
>>>>>>>>> _FillValue : -9999
>>>>>>>>> Geolocation_Pointer : Internal geolocation arrays
>>>>>>>>> Cell_Across_Swath_Sampling : ( 1, 1354, 10 )
>>>>>>>>> Cell_Along_Swath_Sampling : ( 1, 2021, 10 )
>>>>>>>>> Parameter_Type : Output
>>>>>>>>> add_offset : 0
>>>>>>>>> scale_factor : 0.001000000047497451
>>>>>>>>> units : None
>>>>>>>>> long_name : AOT at 0.55 micron for land with all quality
>>>>>>>>> data (Quality flag=1,2,3)
>>>>>>>>> valid_range : ( 0, 5000 )
>>>>>>>>> fatal:Dimension sizes of left hand side do not match right hand
>>>>>>>>> side
>>>>>>>>> fatal:["Execute.c":8640]:Execute: Error occurred at or near line
>>>>>>>>> 110 in file script_01.ncl
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> could you please provide some information on this.
>>>>>>>>>
>>>>>>>>> Thank You
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>> Kunal Bali
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [image: Mailtrack]
>>>>>>>>> <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality1&> Sent
>>>>>>>>> with Mailtrack
>>>>>>>>> <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=signaturevirality1&>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> ncl-talk mailing list
>>>>>>>>> ncl-talk at ucar.edu
>>>>>>>>> List instructions, subscriber options, unsubscribe:
>>>>>>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180815/c30e60a2/attachment.html>
More information about the ncl-talk
mailing list