[Met_help] [rt.rap.ucar.edu #89253] History for RE: ENTLN netcdf files: still having problems with time
John Halley Gotway via RT
met_help at ucar.edu
Tue Jul 9 12:06:56 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
I really don't know what the problem is at this point. Pcp_combine still won't recognize the time of the netcdf file for a reason unknown to me:
sc1login1:/raid1/sport/people/casejl/MET/gridData/ENTLN/ITER5:=> pcp_combine -sum 00000000_000000 01 20180330_000000 06 ./observed_APCP_06 -pcpdir ./temp -field 'name="FlashRate"; level="R1";\' -v 3 -compress 0
DEBUG 2: Performing sum command: Init/In_Accum/Valid/Out_Accum Times = 00000000_000000/010000/20180330_000000/060000
DEBUG 2: Searching for 6 files with accumulation times of 010000 to sum to a total accumulation time of 060000.
WARNING:
WARNING: NcCfFile::open() -> could not determine the valid time, using 0.
WARNING:
WARNING:
WARNING: NcCfFile::open() -> could not determine the valid time, using 0.
WARNING:
WARNING:
WARNING: NcCfFile::open() -> could not determine the valid time, using 0.
WARNING:
WARNING:
WARNING: NcCfFile::open() -> could not determine the valid time, using 0.
WARNING:
WARNING:
WARNING: NcCfFile::open() -> could not determine the valid time, using 0.
WARNING:
WARNING:
WARNING: NcCfFile::open() -> could not determine the valid time, using 0.
WARNING:
ERROR :
ERROR : sum_data_files() -> Cannot find a file with a valid time of 20180330_000000 and accumulation time of 010000 matching the regular expression ".*"
ERROR :
sc1login1:/raid1/sport/people/casejl/MET/gridData/ENTLN/ITER5/temp:=> ncdump -h entlnFlashRate_20180329_19Z.nc
netcdf entlnFlashRate_20180329_19Z {
dimensions:
lon = 425 ;
lat = 350 ;
time = UNLIMITED ; // (1 currently)
variables:
float lat(lat) ;
lat:units = "degrees_north" ;
lat:long_name = "latitude" ;
float lon(lon) ;
lon:units = "degrees_east" ;
lon:long_name = "longitude" ;
double Time(time) ;
Time:units = "seconds since 2018-03-29 19:00:00" ;
Time:calendar = "standard" ;
Time:long_name = "time" ;
float FlashRate(time, lat, lon) ;
FlashRate:units = "Flashes per km2 per 5 min" ;
FlashRate:long_name = "" ;
FlashRate:coordinates = "lon lat time" ;
// global attributes:
:Description = "" ;
:Conventions = "CF-1.7" ;
:title = "" ;
:institution = "" ;
:source = "{0}" ;
:references = "" ;
:history = "Created Fri Mar 8 19:46:57 2019" ;
:comment = "" ;
}
Any other ideas anyone?
-Jon
From: GATLIN, PATRICK N. (MSFC-ST11) <patrick.gatlin at nasa.gov>
Sent: Friday, March 8, 2019 1:50 PM
To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov>
Subject: RE: ENTLN netcdf files: still having problems with time
Now with the Y-M-D H:M:S
Cheers,
Patrick
From: GATLIN, PATRICK N. (MSFC-ST11)
Sent: Friday, March 8, 2019 12:43 PM
To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov<mailto:jonathan.case-1 at nasa.gov>>
Subject: RE: ENTLN netcdf files: still having problems with time
I changed the time string. Give this a try.
Cheers,
Patrick
From: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov<mailto:jonathan.case-1 at nasa.gov>>
Sent: Friday, March 8, 2019 11:40 AM
To: GATLIN, PATRICK N. (MSFC-ST11) <patrick.gatlin at nasa.gov<mailto:patrick.gatlin at nasa.gov>>
Subject: RE: ENTLN netcdf files: still having problems with time
Patrick,
I saw the following at this Unidata site: https://www.unidata.ucar.edu/software/netcdf/docs/BestPractices.html
"Calendar Date/Time
Time as a fundamental unit means a time interval, measured in seconds. A Calendar date/time is a specific instance in real, physical time. Dates are specified as an interval from some reference time e.g. "days elapsed since Greenwich mean noon on 1 January 4713 BCE". The reference time implies a system of counting time called a calendar (e.g. Gregorian calendar) and a textual representation (e.g. ISO 8601<http://www.cl.cam.ac.uk/~mgk25/iso-time.html>).
There are two strategies for storing a date/time into a netCDF variable. One is to encode it as a numeric value and a unit that includes the reference time, e.g. "seconds since 2001-1-1 0:0:0" or"days since 2001-1-1 0:0:0" . The other is to store it as a String using a standard encoding and Calendar. The former is more compact if you have more than one date, and makes it easier to compute intervals between two dates.
Unidata's udunits<https://www.unidata.ucar.edu/software/udunits/> package provides a convenient way to implement the first strategy. It uses the ISO 8601 encoding and a hybrid Gregorian/Julian calendar, but udunits does not support use of other Calendars or encodings for the reference time. However the ncdump "-T" option can display numeric times that use udunits (and optionally climate calendars) as ISO 8601 strings that are easy for humans to interpret.
* If your data uses real, physical time that is well represented using the Gregorian/Julian calendar, encode it as an interval from a reference time, and add a units attribute which uses a udunits-compatible time unit. If the data assumes one of the non-standard calendars mentioned in the CF Conventions, specify that with a Calendar attribute. Readers can then use the udunits package to manipulate or format the date values, and the ncdump utility can display them with either numeric or string representation.
* If your data uses a calendar not supported by the CF Conventions, make it compatible with existing date manipulation packages if possible (for example, java.text.SimpleDateFormat).
* Add multiple sets of time encodings if necessary to allow different readers to work as well as possible.\"
Alternatively in the WRF forecast files, I see the following:
dimensions:
Time = UNLIMITED (1 currently)
DateStrLen = 19;
.
.
.
variables:
char Times(Time, DateStrLen)
.
.
.
Times =
"2019-03-07_21:00:00" ;
.
.
.
So see about replacing the "T" with a simple space between the data and HH:MM:SS.
If that still doesn't work, then we can try perhaps the WRF convention of using a date/time string variable.
-Jon
From: GATLIN, PATRICK N. (MSFC-ST11) <patrick.gatlin at nasa.gov<mailto:patrick.gatlin at nasa.gov>>
Sent: Friday, March 8, 2019 11:31 AM
To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov<mailto:jonathan.case-1 at nasa.gov>>
Subject: RE: ENTLN netcdf files: still having problems with time
That format is CF compatible. Just tell me what it should be and I'll hard code it.
Cheers,
Patrick
From: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov<mailto:jonathan.case-1 at nasa.gov>>
Sent: Friday, March 8, 2019 11:24 AM
To: GATLIN, PATRICK N. (MSFC-ST11) <patrick.gatlin at nasa.gov<mailto:patrick.gatlin at nasa.gov>>
Subject: ENTLN netcdf files: still having problems with time
Patrick,
Sorry for the long delay in testing out your latest iteration of the ENTLN netcdf files.
I tried running them through the MET pcp_combine and the plot_data_plane utilities, but I'm still getting errors about the time variable:
sc1login1:/raid1/sport/people/casejl/MET/gridData/ENTLN/ITER4:=> pcp_combine -sum 00000000_000000 01 20180330_000000 06 ./observed_APCP_06 -pcpdir ./temp -field \'name=\"FlashRate\"; level=\"R1\";\' -v 3 -compress 0
WARNING:
WARNING: NcCfFile::open() -> could not determine the valid time, using 0.
WARNING:
ERROR :
ERROR : yyerror() -> syntax error in file "/tmp/config_36913_0_.temp"
ERROR :
ERROR : line = 2
ERROR :
ERROR : column = 1
ERROR :
ERROR : text = "(nul)"
ERROR :
ERROR :
ERROR : (nul)
ERROR : (nul)
ERROR :
-bash: ': command not found
sc1login1:/raid1/sport/people/casejl/MET/gridData/ENTLN/ITER4:=> plot_data_plane ./temp/entln_met_2018033000.nc test.ps 'name="FlashRate"; level="(0,*,*)";' -v 5
DEBUG 1: Opening data file: ./temp/entln_met_2018033000.nc
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> could not extract valid time from the "time" variable.
DEBUG 4: NcCfFile::open() -> could not extract valid time from file name.
WARNING:
WARNING: NcCfFile::open() -> could not determine the valid time, using 0.
WARNING:
DEBUG 4: NcCfFile::open() -> could not extract init time from the "forecast_reference_time" variable.
DEBUG 4: NcCfFile::open() -> could not extract init time from file name.
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> could not extract valid time from the "time" variable.
DEBUG 4: NcCfFile::open() -> could not extract valid time from file name.
WARNING:
WARNING: NcCfFile::open() -> could not determine the valid time, using 0.
WARNING:
DEBUG 4: NcCfFile::open() -> could not extract init time from the "forecast_reference_time" variable.
DEBUG 4: NcCfFile::open() -> could not extract init time from file name.
When I ran ncdump on a sample file, I see that the time looks a bit peculiar to me:
netcdf entln_met_2018033000 {
dimensions:
lon = 425 ;
lat = 350 ;
time = UNLIMITED ; // (1 currently)
variables:
float lat(lat) ;
lat:units = "degrees_north" ;
lat:long_name = "latitude" ;
float lon(lon) ;
lon:units = "degrees_east" ;
lon:long_name = "longitude" ;
double Time(time) ;
Time:units = "seconds since 2018-03-30T00:00:00" ;
Time:calendar = "standard" ;
Time:long_name = "time" ;
float FlashRate(time, lat, lon) ;
FlashRate:units = "Flashes per km2 per 5 min" ;
FlashRate:long_name = "" ;
FlashRate:coordinates = "lon lat time" ;
Do you know what the "T" is within the time string?
-Jon
*****************************************************
Jonathan Case (Jon)
ENSCO, Inc./NASA SPoRT Center
National Space Science and Technology Center
320 Sparkman Drive, Room 3000B
Huntsville, AL 35805
Jonathan.Case-1 at nasa.gov<mailto:Jonathan.Case-1 at nasa.gov> (alt: case.jonathan at ensco.com<mailto:case.jonathan at ensco.com>)
*****************************************************
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: RE: ENTLN netcdf files: still having problems with time
From: John Halley Gotway
Time: Fri Mar 08 14:03:11 2019
Jon,
Please try running:
pcp_combine -sum 00000000_000000 01 20180330_000000 06
./observed_APCP_06
-pcpdir ./temp -field 'name="FlashRate"; level="(0,*,*)";' -v 3
-compress 0
Notice that I changed the level string to define how to index into
this
NetCDF data.
I also wanted to mention that we've been enhancing pcp_combine for
met-8.1
to support a new "-derive" option. With that you could specify that
you
want the sum, min, max, range, mean, standard deviation, or valid data
count from a set of input data files. We're also enhancing it so that
you
can process multiple input fields in the same call.
For example... if you're using GRIB data... you could compute the
daily
min/max value of 2-m temp and 10-m winds like this:
pcp_combine -derive min,max todays_data_f*.grib min_max_for_today.nc \
-field 'name="TMP"; level="Z2";' \
-field 'name="UGRD"; level="Z10";' \
-field 'name="VGRD"; level="Z10";'
Do you think this sort of functionality will be useful to you?
Thanks,
John
On Fri, Mar 8, 2019 at 12:57 PM Case, Jonathan[ENSCO INC] via RT <
met_help at ucar.edu> wrote:
>
> Fri Mar 08 12:57:40 2019: Request 89253 was acted upon.
> Transaction: Ticket created by jonathan.case-1 at nasa.gov
> Queue: met_help
> Subject: RE: ENTLN netcdf files: still having problems with
time
> Owner: Nobody
> Requestors: jonathan.case-1 at nasa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89253 >
>
>
> I really don't know what the problem is at this point. Pcp_combine
still
> won't recognize the time of the netcdf file for a reason unknown to
me:
>
> sc1login1:/raid1/sport/people/casejl/MET/gridData/ENTLN/ITER5:=>
> pcp_combine -sum 00000000_000000 01 20180330_000000 06
./observed_APCP_06
> -pcpdir ./temp -field 'name="FlashRate"; level="R1";\' -v 3
-compress 0
> DEBUG 2: Performing sum command: Init/In_Accum/Valid/Out_Accum Times
=
> 00000000_000000/010000/20180330_000000/060000
> DEBUG 2: Searching for 6 files with accumulation times of 010000 to
sum to
> a total accumulation time of 060000.
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> WARNING:
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> WARNING:
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> WARNING:
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> WARNING:
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> WARNING:
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> WARNING:
> ERROR :
> ERROR : sum_data_files() -> Cannot find a file with a valid time of
> 20180330_000000 and accumulation time of 010000 matching the regular
> expression ".*"
> ERROR :
>
>
>
sc1login1:/raid1/sport/people/casejl/MET/gridData/ENTLN/ITER5/temp:=>
> ncdump -h entlnFlashRate_20180329_19Z.nc
> netcdf entlnFlashRate_20180329_19Z {
> dimensions:
> lon = 425 ;
> lat = 350 ;
> time = UNLIMITED ; // (1 currently)
> variables:
> float lat(lat) ;
> lat:units = "degrees_north" ;
> lat:long_name = "latitude" ;
> float lon(lon) ;
> lon:units = "degrees_east" ;
> lon:long_name = "longitude" ;
> double Time(time) ;
> Time:units = "seconds since 2018-03-29 19:00:00" ;
> Time:calendar = "standard" ;
> Time:long_name = "time" ;
> float FlashRate(time, lat, lon) ;
> FlashRate:units = "Flashes per km2 per 5 min" ;
> FlashRate:long_name = "" ;
> FlashRate:coordinates = "lon lat time" ;
>
> // global attributes:
> :Description = "" ;
> :Conventions = "CF-1.7" ;
> :title = "" ;
> :institution = "" ;
> :source = "{0}" ;
> :references = "" ;
> :history = "Created Fri Mar 8 19:46:57 2019" ;
> :comment = "" ;
> }
>
>
> Any other ideas anyone?
> -Jon
>
>
> From: GATLIN, PATRICK N. (MSFC-ST11) <patrick.gatlin at nasa.gov>
> Sent: Friday, March 8, 2019 1:50 PM
> To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov>
> Subject: RE: ENTLN netcdf files: still having problems with time
>
> Now with the Y-M-D H:M:S
>
> Cheers,
> Patrick
>
> From: GATLIN, PATRICK N. (MSFC-ST11)
> Sent: Friday, March 8, 2019 12:43 PM
> To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov
> <mailto:jonathan.case-1 at nasa.gov>>
> Subject: RE: ENTLN netcdf files: still having problems with time
>
> I changed the time string. Give this a try.
>
> Cheers,
> Patrick
>
> From: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
1 at nasa.gov
> <mailto:jonathan.case-1 at nasa.gov>>
> Sent: Friday, March 8, 2019 11:40 AM
> To: GATLIN, PATRICK N. (MSFC-ST11) <patrick.gatlin at nasa.gov<mailto:
> patrick.gatlin at nasa.gov>>
> Subject: RE: ENTLN netcdf files: still having problems with time
>
> Patrick,
>
> I saw the following at this Unidata site:
> https://www.unidata.ucar.edu/software/netcdf/docs/BestPractices.html
>
> "Calendar Date/Time
>
> Time as a fundamental unit means a time interval, measured in
seconds. A
> Calendar date/time is a specific instance in real, physical time.
Dates are
> specified as an interval from some reference time e.g. "days elapsed
since
> Greenwich mean noon on 1 January 4713 BCE". The reference time
implies a
> system of counting time called a calendar (e.g. Gregorian calendar)
and a
> textual representation (e.g. ISO 8601<
> http://www.cl.cam.ac.uk/~mgk25/iso-time.html>).
>
> There are two strategies for storing a date/time into a netCDF
variable.
> One is to encode it as a numeric value and a unit that includes the
> reference time, e.g. "seconds since 2001-1-1 0:0:0" or"days since
2001-1-1
> 0:0:0" . The other is to store it as a String using a standard
encoding and
> Calendar. The former is more compact if you have more than one date,
and
> makes it easier to compute intervals between two dates.
>
> Unidata's udunits<https://www.unidata.ucar.edu/software/udunits/>
package
> provides a convenient way to implement the first strategy. It uses
the ISO
> 8601 encoding and a hybrid Gregorian/Julian calendar, but udunits
does not
> support use of other Calendars or encodings for the reference time.
However
> the ncdump "-T" option can display numeric times that use udunits
(and
> optionally climate calendars) as ISO 8601 strings that are easy for
humans
> to interpret.
>
> * If your data uses real, physical time that is well represented
using
> the Gregorian/Julian calendar, encode it as an interval from a
reference
> time, and add a units attribute which uses a udunits-compatible time
unit.
> If the data assumes one of the non-standard calendars mentioned in
the CF
> Conventions, specify that with a Calendar attribute. Readers can
then use
> the udunits package to manipulate or format the date values, and the
ncdump
> utility can display them with either numeric or string
representation.
> * If your data uses a calendar not supported by the CF
Conventions,
> make it compatible with existing date manipulation packages if
possible
> (for example, java.text.SimpleDateFormat).
> * Add multiple sets of time encodings if necessary to allow
different
> readers to work as well as possible.\"
> Alternatively in the WRF forecast files, I see the following:
> dimensions:
> Time = UNLIMITED (1 currently)
> DateStrLen = 19;
> .
> .
> .
>
> variables:
> char Times(Time, DateStrLen)
> .
> .
> .
> Times =
> "2019-03-07_21:00:00" ;
> .
> .
> .
>
> So see about replacing the "T" with a simple space between the data
and
> HH:MM:SS.
> If that still doesn't work, then we can try perhaps the WRF
convention of
> using a date/time string variable.
>
> -Jon
>
> From: GATLIN, PATRICK N. (MSFC-ST11)
<patrick.gatlin at nasa.gov<mailto:
> patrick.gatlin at nasa.gov>>
> Sent: Friday, March 8, 2019 11:31 AM
> To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov
> <mailto:jonathan.case-1 at nasa.gov>>
> Subject: RE: ENTLN netcdf files: still having problems with time
>
> That format is CF compatible. Just tell me what it should be and
I'll hard
> code it.
>
> Cheers,
> Patrick
>
> From: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
1 at nasa.gov
> <mailto:jonathan.case-1 at nasa.gov>>
> Sent: Friday, March 8, 2019 11:24 AM
> To: GATLIN, PATRICK N. (MSFC-ST11) <patrick.gatlin at nasa.gov<mailto:
> patrick.gatlin at nasa.gov>>
> Subject: ENTLN netcdf files: still having problems with time
>
> Patrick,
>
> Sorry for the long delay in testing out your latest iteration of the
ENTLN
> netcdf files.
> I tried running them through the MET pcp_combine and the
plot_data_plane
> utilities, but I'm still getting errors about the time variable:
>
> sc1login1:/raid1/sport/people/casejl/MET/gridData/ENTLN/ITER4:=>
> pcp_combine -sum 00000000_000000 01 20180330_000000 06
./observed_APCP_06
> -pcpdir ./temp -field \'name=\"FlashRate\"; level=\"R1\";\' -v 3
-compress 0
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> WARNING:
> ERROR :
> ERROR : yyerror() -> syntax error in file
"/tmp/config_36913_0_.temp"
> ERROR :
> ERROR : line = 2
> ERROR :
> ERROR : column = 1
> ERROR :
> ERROR : text = "(nul)"
> ERROR :
> ERROR :
> ERROR : (nul)
> ERROR : (nul)
> ERROR :
> -bash: ': command not found
> sc1login1:/raid1/sport/people/casejl/MET/gridData/ENTLN/ITER4:=>
> plot_data_plane ./temp/entln_met_2018033000.nc test.ps
'name="FlashRate";
> level="(0,*,*)";' -v 5
> DEBUG 1: Opening data file: ./temp/entln_met_2018033000.nc
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
> DEBUG 4: NcCfFile::open() -> could not extract valid time from the
"time"
> variable.
> DEBUG 4: NcCfFile::open() -> could not extract valid time from file
name.
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> WARNING:
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> type "FileType_NcCF".
> DEBUG 4: NcCfFile::open() -> could not extract valid time from the
"time"
> variable.
> DEBUG 4: NcCfFile::open() -> could not extract valid time from file
name.
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> WARNING:
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
>
>
> When I ran ncdump on a sample file, I see that the time looks a bit
> peculiar to me:
> netcdf entln_met_2018033000 {
> dimensions:
> lon = 425 ;
> lat = 350 ;
> time = UNLIMITED ; // (1 currently)
> variables:
> float lat(lat) ;
> lat:units = "degrees_north" ;
> lat:long_name = "latitude" ;
> float lon(lon) ;
> lon:units = "degrees_east" ;
> lon:long_name = "longitude" ;
> double Time(time) ;
> Time:units = "seconds since 2018-03-30T00:00:00" ;
> Time:calendar = "standard" ;
> Time:long_name = "time" ;
> float FlashRate(time, lat, lon) ;
> FlashRate:units = "Flashes per km2 per 5 min" ;
> FlashRate:long_name = "" ;
> FlashRate:coordinates = "lon lat time" ;
>
>
> Do you know what the "T" is within the time string?
> -Jon
>
> *****************************************************
> Jonathan Case (Jon)
> ENSCO, Inc./NASA SPoRT Center
> National Space Science and Technology Center
> 320 Sparkman Drive, Room 3000B
> Huntsville, AL 35805
> Jonathan.Case-1 at nasa.gov<mailto:Jonathan.Case-1 at nasa.gov> (alt:
> case.jonathan at ensco.com<mailto:case.jonathan at ensco.com>)
> *****************************************************
>
>
>
------------------------------------------------
More information about the Met_help
mailing list