[ncl-talk] set_date_time
Dennis Shea
shea at ucar.edu
Sun Oct 8 22:41:09 MDT 2017
Long answer BUT I suggest you read this carefully!
----------------------------------------------------------------------
As is suggested (ad nauseum), always look at your data:
%> ncl_filedump MAIACTAOT.h00v02.20003650620.*hdf * | less
[snip]
filename: MAIACTAOT.h00v02.20003650620
path: MAIACTAOT.h00v02.20003650620.hdf
file global attributes:
*HDFEOSVersion : HDFEOS_V2.12 <=== HDF-EOS file*
StructMetadata_0 : GROUP=SwathStructure
END_GROUP=SwathStructure
[snip]
----
It is a HDF-EOS file!!!
When a user sees HDFEOS [ because he/she looked at the file; :-) ], the
file extension *.he2* or equivalently, *.hdfeos* should be appended to the
file. This can be done programatically or on the command line. The file
need not be explicitly changed!
%> ncl_filedump MAIACTAOT.h00v02.20003650620.hdf*.he2* | less
NCL looks for the file MAIACTAOT.h00v02.20003650620.hdf*.he2* and*, *if it
does not find the file, it *'backs-off' *the last file extension [ here,*
.he2 *] and reads the file
*.*
*Why* should a user do this?
%> ncl_filedump MAIACTAOT.h00v02.20003650620.hdf*.he2 *| less
(a) If you look at the original file*, *you will see a rather complicated
file structure*. *It includes multiple *groups. *When the .he2 extenson is
used, NCL 'flattens' the file structure. It make it look like a standard
netCDF3 file. In this case the variables have appended extensions named:
_grid5km and _grid1km. A very nice feature.
(b) An even nicer feature of the .he2/.hdfeos extension is that NCL *creates
value-addedarrays *containing the lat/lon arrays.
double GridLat_grid5km ( YDim_grid5km, XDim_grid5km )
double GridLat_grid5km ( YDim_grid5km, XDim_grid5km )
double GridLat_grid1km ( YDim_grid1km, XDim_grid1km )
double GridLat_grid1km ( YDim_grid1km, XDim_grid1km )
(c) NCL *creates *CF Convention attributes like 'coordinates' for assorted
variable.
------
One of the less public NCL-developers (Dave Brown), deserves considerable
credit for NCL's IO capabilities. In particular, with respect to GRIB and
HDF files. Also, 'addfile' and 'addfiles'
As I say ... "A user need-not-fear any grib/hdf/netCDF files if using NCL!'
------
The 2nd part of this answer is 'How to add a 'time' (record) dimension?'
See attachment:
%> mkdir MAIACTAOT_TIME <== will contain created file
%> ncl create_time.MAIACTAO.ncl
-----
The keys are:
;---https://www.ncl.ucar.edu/Document/Functions/Built-in/addfiles.shtml
f = addfiles(pthi, "r")
*ListSetType (f, "join") ; add 'record' (time
dimension) *
;--- Create a time variable
refTime = "hours since 2000-01-01 00:00:00" ; arbitrary
time = *time_create_MAIACTAOT* (fili, refTime) ; local function
An file dump of the created file in directory:
%> cd /MAIACTAOT_TIME
%> ncl_filedump maiactaot.h00v02.20003650620-20003660700.nc
or
%> ncdump -h maiactaot.h00v02.20003650620-20003660700.nc
-++++++++
Internal NCAR users, the files are at:
/glade/p/ncldev/data/hdf
MAIACTAOT.h00v02.20003650620.hdf
MAIACTAOT.h00v02.20003660525.hdf
MAIACTAOT.h00v02.20003660700.hdf
Cheers
D
On Fri, Oct 6, 2017 at 9:18 PM, Kunal Bali <kunal.bali9 at gmail.com> wrote:
> Thanks Mery for providing this information.
> I think the problem is with CDO.
>
> But is there any function to convert filenames to date-time. and then
> create time variable in the netcdf file.
> such as :
> MAIACTAOT.h00v02.*2000364**07**15*.nc --> 2000=year, 364=day, 07=hr,
> 15=min
> *2000-01-29 07:15:00*
>
> could you please give some information on that.
> thank you
>
> regards
> Kunal Bali
>
>
>
>
>
> On Fri, Oct 6, 2017 at 2:16 AM, Guido Cioni <guidocioni at gmail.com> wrote:
>
>> I had to do something similar last week. Converting from hdf to netcdf
>> and then merge into a single file with time information. It was modis data.
>> I don't have time to explain the script so I'm copying here hoping that it
>> could help.
>>
>> for f in *.nc; do
>> year=${f:9:4}
>> day_of_year=${f:13:3}
>> month=`date -d "${year}-01-01 +${day_of_year} days -1 day" "+%m"`
>> day=`date -d "${year}-01-01 +${day_of_year} days -1 day" "+%d"`
>> echo ${year}-${month}-${day}
>> cdo -b F64 setdate,${year}-${month}-${day} ${f} ${f%.nc}_2.nc
>> done
>>
>>
>> Cheers
>>
>> Il 5 ott 2017 9:56 PM, "Marston Johnston" <shejo284 at gmail.com> ha
>> scritto:
>>
>>> It seems like you choose a fairly complicated way to solve the problem.
>>>
>>> May I suggest reading in the MAIACTAOT.h00v02.***.nc files in
>>> chronological order and then writing new netcdf files in ncl where you can
>>> create a time array with the correct dtg.
>>>
>>> It’s fairly quick and easy in ncl.
>>>
>>> I’ve had such problems before and found it easiest to with the above
>>> method.
>>>
>>>
>>>
>>> Hope this helps,
>>>
>>> /M
>>>
>>>
>>>
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>
>>> Marston S. Ward, PhD
>>>
>>> Department of Earth Sciences
>>>
>>> University of Gothenburg, Sweden
>>>
>>> Email: marston.johnston at gu.se
>>>
>>> SkypeID: marston.johnston
>>>
>>> Phone: +46-31-7864901 <+46%2031%20786%2049%2001>
>>>
>>> Only the fruitful thing is true!
>>>
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>
>>>
>>>
>>>
>>>
>>> *From: *ncl-talk <ncl-talk-bounces at ucar.edu> on behalf of Kunal Bali <
>>> kunal.bali9 at gmail.com>
>>> *Date: *Thursday, 5 October 2017 at 20:40
>>> *To: *"ncl-talk at ucar.edu" <ncl-talk at ucar.edu>
>>> *Subject: *[ncl-talk] set_date_time
>>>
>>>
>>>
>>> Dear NCL users,
>>>
>>>
>>>
>>> I have one question regarding date-time setting from filename. This
>>> question is not actually related to NCL, sorry in advance.
>>>
>>>
>>>
>>> I have 4 .nc files named given below. The time is not created in these
>>> files during conversion from hdf to netcdf in NCL.
>>>
>>>
>>>
>>> MAIACTAOT.h00v02.*20003640715*.nc --> 2000=year, 364=day, 07=hr,
>>> 15=min
>>>
>>> MAIACTAOT.h00v02.20003650620.nc
>>>
>>> MAIACTAOT.h00v02.20003660525.nc
>>>
>>> MAIACTAOT.h00v02.20003660700.nc
>>>
>>>
>>>
>>> So, I did try to create the date-time with shell scripting (given below)
>>>
>>> but it is creating wrong date as the files are of December month, but
>>> it is generating January month date also. I don't know why.
>>>
>>>
>>>
>>> *2000-01-01 05:25:00*
>>>
>>> *2000-01-01 07:00:00*
>>>
>>> 2000-12-30 07:15:00
>>>
>>> 2000-12-31 06:20:00
>>>
>>>
>>>
>>>
>>>
>>> the shell script, which I am using
>>>
>>>
>>>
>>> #######################################
>>>
>>>
>>>
>>> for n in MAIACTAOT.h00v02.20*.nc;do
>>>
>>> ls $n > text
>>>
>>> export xx=`cut -c18-28 text`
>>>
>>> export YYYY=`cut -c18-21 text`
>>>
>>> export JJJ=`cut -c22-24 text`
>>>
>>> export HH=`cut -c25-26 text`
>>>
>>> export MM=`cut -c27-28 text`
>>>
>>> mm=`date -d "$JJJ days $YYYY-01-01" +"%m"`
>>>
>>> dd=`date -d "$JJJ days $YYYY-01-01" +"%d"`
>>>
>>> mydate=${YYYY}-${mm}-${dd},${HH}:${MM}:00
>>>
>>> echo $mydate
>>>
>>>
>>>
>>> echo "cdo settaxis,$mydate $n re$n"
>>>
>>> cdo settaxis,$mydate $n re$n
>>>
>>>
>>>
>>> done
>>>
>>>
>>>
>>> cdo mergetime reMAI*.nc final.nc
>>>
>>> exit
>>>
>>> #################################
>>>
>>>
>>>
>>>
>>>
>>> Any suggestion regarding this issue would be very helpful.
>>>
>>>
>>>
>>> Thank You
>>>
>>>
>>> Kunal Bali
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________ ncl-talk mailing list
>>> ncl-talk at ucar.edu List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> ncl-talk at ucar.edu
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20171008/87c9f0c7/attachment.html>
-------------- next part --------------
netcdf maiactaot.h00v02.20003650620-20003660700 {
dimensions:
time = UNLIMITED ; // (3 currently)
YDim_grid1km = 1200 ;
XDim_grid1km = 1200 ;
variables:
double time(time) ;
time:units = "hours since 2000-01-01 00:00:00" ;
time:calendar = "standard" ;
double GridLat_grid1km(YDim_grid1km, XDim_grid1km) ;
GridLat_grid1km:projection = "Albers Conical Equal_Area" ;
GridLat_grid1km:corners = 32.038387901543, 34.9985549202943, 24.5580030404694, 21.8673735745303 ;
GridLat_grid1km:long_name = "latitude" ;
GridLat_grid1km:units = "degrees_north" ;
double GridLon_grid1km(YDim_grid1km, XDim_grid1km) ;
GridLon_grid1km:projection = "Albers Conical Equal_Area" ;
GridLon_grid1km:corners = 64.23634233372, 76.8178669755363, 79.4747290253718, 68.077149355049 ;
GridLon_grid1km:long_name = "longitude" ;
GridLon_grid1km:units = "degrees_east" ;
short AOT_QA_grid1km(YDim_grid1km, XDim_grid1km) ;
AOT_QA_grid1km:coordinates = "GridLat_grid1km, GridLon_grid1km" ;
AOT_QA_grid1km:hdfeos_name = "AOT_QA" ;
AOT_QA_grid1km:projection = "Albers Conical Equal_Area" ;
AOT_QA_grid1km:unsigned = 1 ;
AOT_QA_grid1km:valid_range = 0s, 255s ;
AOT_QA_grid1km:_FillValue = 0s ;
AOT_QA_grid1km:data_description = "Bits\tDefinition\n",
"0-2 Cloud Mask\n",
" 00 --- Undefined\n",
" 001 --- Clear\n",
" 010 --- Possible Cloudy\n",
" 011 --- Cloudy \n",
" 101 --- Cloud shadow\n",
" 110 --- Fire hotspot\n",
" 111 --- Water Sediments\n",
"3-4 Land Water Snow/ice Mask\n",
" 00 --- Land\n",
" 01 --- Water\n",
" 10 --- Snow\n",
" 11 --- Ice\n",
"5-7 Adjacency Mask\n",
" 000 --- Normal condition\n",
" 001 --- Adjacent to cloud\n",
" 010 --- Surrounded by more than 8 cloudy pixels\n",
" 011 --- Single cloudy pixel\n",
" 100 --- Adjacent to snow\n",
" 101 --- Snow was previously detected for this pixel\n",
"8-11 QA for AOT retrieval over Water \n",
" 0000 --- Best \n",
" 0001 --- Water Sediments are detected\n",
" 0010 --- AC over water done, but AOT>0.5\n",
" 0011 --- There is 1 neighbor cloud\n",
" 0100 --- There is >1 neighbor clouds\n",
" 0101 --- no retrieval (cloudy, or whatever)\n",
" 0110 --- no retrievals near detected or previously snow\n",
" 0111 --- no retrievals for altituide more than 3.5km\n",
" 1000 --- no retrieval due to sun glint\n",
" 1001 --- retrieved AOT is very low (<0.05) due to low Coxmunk angle\n",
" 1010 --- AOT within +-2km from the coastline is replaced by nearby AOT\n",
"12 Glint mask\n",
" 0 --- glint is not detected\n",
" 1 --- glint is detected\n",
"13-14 Aerosol model\n",
" 00 --- background model\n",
" 01 --- Smoke model\n",
" 10 --- Dust model\n",
"15 Reserved" ;
AOT_QA_grid1km:unit = "None" ;
AOT_QA_grid1km:long_name = "AOT_QA" ;
byte AOT_MODEL_grid1km(YDim_grid1km, XDim_grid1km) ;
AOT_MODEL_grid1km:coordinates = "GridLat_grid1km, GridLon_grid1km" ;
AOT_MODEL_grid1km:hdfeos_name = "AOT_MODEL" ;
AOT_MODEL_grid1km:projection = "Albers Conical Equal_Area" ;
AOT_MODEL_grid1km:unsigned = 1 ;
AOT_MODEL_grid1km:valid_range = 0b, 100b ;
AOT_MODEL_grid1km:_FillValue = -1b ;
AOT_MODEL_grid1km:unit = "None" ;
AOT_MODEL_grid1km:long_name = "AOT model used in retrieval" ;
short Optical_Depth_047_grid1km(time, YDim_grid1km, XDim_grid1km) ;
Optical_Depth_047_grid1km:coordinates = "GridLat_grid1km, GridLon_grid1km" ;
Optical_Depth_047_grid1km:hdfeos_name = "Optical_Depth_047" ;
Optical_Depth_047_grid1km:projection = "Albers Conical Equal_Area" ;
Optical_Depth_047_grid1km:valid_range = -100s, 5000s ;
Optical_Depth_047_grid1km:_FillValue = -28672s ;
Optical_Depth_047_grid1km:unit = "None" ;
Optical_Depth_047_grid1km:add_offset = 0. ;
Optical_Depth_047_grid1km:scale_factor = 0.001 ;
Optical_Depth_047_grid1km:long_name = "AOT at 0.47 micron" ;
short Optical_Depth_055_grid1km(time, YDim_grid1km, XDim_grid1km) ;
Optical_Depth_055_grid1km:coordinates = "GridLat_grid1km, GridLon_grid1km" ;
Optical_Depth_055_grid1km:hdfeos_name = "Optical_Depth_055" ;
Optical_Depth_055_grid1km:projection = "Albers Conical Equal_Area" ;
Optical_Depth_055_grid1km:valid_range = -100s, 5000s ;
Optical_Depth_055_grid1km:_FillValue = -28672s ;
Optical_Depth_055_grid1km:unit = "None" ;
Optical_Depth_055_grid1km:add_offset = 0. ;
Optical_Depth_055_grid1km:scale_factor = 0.001 ;
Optical_Depth_055_grid1km:long_name = "AOT at 0.55 micron" ;
short Column_WV_grid1km(time, YDim_grid1km, XDim_grid1km) ;
Column_WV_grid1km:coordinates = "GridLat_grid1km, GridLon_grid1km" ;
Column_WV_grid1km:hdfeos_name = "Column_WV" ;
Column_WV_grid1km:projection = "Albers Conical Equal_Area" ;
Column_WV_grid1km:valid_range = 0s, 30000s ;
Column_WV_grid1km:_FillValue = -28672s ;
Column_WV_grid1km:unit = "cm" ;
Column_WV_grid1km:add_offset = 0. ;
Column_WV_grid1km:scale_factor = 0.001 ;
Column_WV_grid1km:long_name = "Column Water Vapor" ;
// global attributes:
:creation_date = "Sun Oct 8 22:13:54 MDT 2017" ;
:Conventions = "None" ;
:NCL_version = "Any version of NCL >= 5.2.0 (4/2010)" ;
:source_file = "MAIACTAOT HDF-EOS files" ;
:title = "MAIACTAOT HDF-EOS: time dimension added to selected variables" ;
}
-------------- next part --------------
A non-text attachment was scrubbed...
Name: create_time.MAIACTAO.ncl
Type: application/octet-stream
Size: 5229 bytes
Desc: not available
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20171008/87c9f0c7/attachment.obj>
More information about the ncl-talk
mailing list