[Met_help] [rt.rap.ucar.edu #88228] History for using netcdf files with unique field in pcp_combine
Julie Prestopnik via RT
met_help at ucar.edu
Mon Feb 11 10:09:38 MST 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Dear MET Help,
I'm trying to do something fairly unique in MET by validating derived total lightning flash rates from an NWP model against observed gridded lightning flash rates.
I'm "faking" MET into thinking the model flash rates are simply accumulated precip (already named as APCP in GRIB2 files), whereas the observed lightning files are identified as "FlashRate" in hourly netcdf files.
For starters, I'm just trying to work with the netcdf files to see if I can combine them into "accumulated" fields using pcp_combine, but I'm having trouble getting it to work.
What I'd like to do is combine/accumulate the data into various time intervals [much like precipitation], and then output the pcp_combine netcdf file with the traditional APCP name.
Is there a way I can do this using pcp_combine? I'm still on MET v6.0. I have an ncdump of the lightning file below that might be helpful.
Thanks!
-Jon
Ncdump of sample lightning netcdf file:
:=> ncdump -h ./precip_tmp/entln_met_2018033001.nc
netcdf entln_met_2018033001 {
dimensions:
lon = 425 ;
lat = 350 ;
time = UNLIMITED ; // (1 currently)
variables:
float lat(lat) ;
lat:units = "degrees_north" ;
lat:long_name = "latitude" ;
float lon(lon) ;
lon:units = "degrees east" ;
lon:long_name = "longitude" ;
double Time(time) ;
Time:units = "seconds since " ;
Time:calendar = "standard" ;
Time:long_name = "time" ;
float FlashRate(time, lat, lon) ;
FlashRate:units = "Flashes per km2 per 5 min" ;
FlashRate:long_name = "" ;
FlashRate:coordinates = "lon lat time" ;
// global attributes:
:Description = "" ;
:Conventions = "CF-1.7" ;
:title = "" ;
:institution = "" ;
:source = "{0}" ;
:references = "" ;
:history = "Created Mon Dec 17 22:23:33 2018" ;
:comment = "" ;
}
*****************************************************
Jonathan Case (Jon)
ENSCO, Inc./NASA SPoRT Center
National Space Science and Technology Center
320 Sparkman Drive, Room 3000B
Huntsville, AL 35805
Jonathan.Case-1 at nasa.gov<mailto:Jonathan.Case-1 at nasa.gov> (alt: case.jonathan at ensco.com<mailto:case.jonathan at ensco.com>)
*****************************************************
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: using netcdf files with unique field in pcp_combine
From: Julie Prestopnik
Time: Tue Dec 18 12:26:26 2018
Hi Jon. Unfortunately, our MET expert is out of the office for a
while due
to the holidays. I do not know the answers to your questions, however
I
can try to help.
For starters, I'm just trying to work with the netcdf files to see if
I can
> combine them into "accumulated" fields using pcp_combine, but I'm
having
> trouble getting it to work.
>
What kind of trouble are you having? If you'd like you could post the
data
to our ftp server ( following the instructions listed here:
http://www.dtcenter.org/met/users/support/met_help.php#ftp) and send
me the
command that you are running and I can see if I can help.
Thanks,
Julie
On Tue, Dec 18, 2018 at 11:31 AM Case, Jonathan[ENSCO INC] via RT <
met_help at ucar.edu> wrote:
>
> Tue Dec 18 11:31:09 2018: Request 88228 was acted upon.
> Transaction: Ticket created by jonathan.case-1 at nasa.gov
> Queue: met_help
> Subject: using netcdf files with unique field in pcp_combine
> Owner: Nobody
> Requestors: jonathan.case-1 at nasa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228 >
>
>
> Dear MET Help,
>
> I'm trying to do something fairly unique in MET by validating
derived
> total lightning flash rates from an NWP model against observed
gridded
> lightning flash rates.
> I'm "faking" MET into thinking the model flash rates are simply
> accumulated precip (already named as APCP in GRIB2 files), whereas
the
> observed lightning files are identified as "FlashRate" in hourly
netcdf
> files.
>
> For starters, I'm just trying to work with the netcdf files to see
if I
> can combine them into "accumulated" fields using pcp_combine, but
I'm
> having trouble getting it to work.
> What I'd like to do is combine/accumulate the data into various time
> intervals [much like precipitation], and then output the pcp_combine
netcdf
> file with the traditional APCP name.
>
> Is there a way I can do this using pcp_combine? I'm still on MET
v6.0. I
> have an ncdump of the lightning file below that might be helpful.
>
> Thanks!
> -Jon
>
>
> Ncdump of sample lightning netcdf file:
> :=> ncdump -h ./precip_tmp/entln_met_2018033001.nc
> netcdf entln_met_2018033001 {
> dimensions:
> lon = 425 ;
> lat = 350 ;
> time = UNLIMITED ; // (1 currently)
> variables:
> float lat(lat) ;
> lat:units = "degrees_north" ;
> lat:long_name = "latitude" ;
> float lon(lon) ;
> lon:units = "degrees east" ;
> lon:long_name = "longitude" ;
> double Time(time) ;
> Time:units = "seconds since " ;
> Time:calendar = "standard" ;
> Time:long_name = "time" ;
> float FlashRate(time, lat, lon) ;
> FlashRate:units = "Flashes per km2 per 5 min" ;
> FlashRate:long_name = "" ;
> FlashRate:coordinates = "lon lat time" ;
>
> // global attributes:
> :Description = "" ;
> :Conventions = "CF-1.7" ;
> :title = "" ;
> :institution = "" ;
> :source = "{0}" ;
> :references = "" ;
> :history = "Created Mon Dec 17 22:23:33 2018" ;
> :comment = "" ;
> }
>
>
>
>
> *****************************************************
> Jonathan Case (Jon)
> ENSCO, Inc./NASA SPoRT Center
> National Space Science and Technology Center
> 320 Sparkman Drive, Room 3000B
> Huntsville, AL 35805
> Jonathan.Case-1 at nasa.gov<mailto:Jonathan.Case-1 at nasa.gov> (alt:
> case.jonathan at ensco.com<mailto:case.jonathan at ensco.com>)
> *****************************************************
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #88228] using netcdf files with unique field in pcp_combine
From: Case, Jonathan[ENSCO INC]
Time: Tue Dec 18 13:38:12 2018
Sorry Julie,
I should have also included the pcp_combine error I'm getting:
sc1login1:/raid1/sport/people/casejl/MET/gridData:=> pcp_combine -sum
00000000_000000 01 20180330_060000 06 ./observed_APCP_06 -pcpdir
./precip_tmp -field 'name="FlashRate"; level="R1";' -v 3 -compress 0
DEBUG 2: Performing sum command: Init/In_Accum/Valid/Out_Accum Times =
00000000_000000/010000/20180330_060000/060000
DEBUG 2: Searching for 6 files with accumulation times of 010000 to
sum to a total accumulation time of 060000.
WARNING:
WARNING: NcCfFile::open() -> could not determine the valid time, using
0.
WARNING:
terminate called after throwing an instance of
'netCDF::exceptions::NcBadId'
what(): NcBadId: Not a netcdf id
file: ncDim.cpp line:80
Aborted
I need to know what the "NcBadId" means so I can persue a possible
solution.
Thx!
-Jon
-----Original Message-----
From: Julie Prestopnik via RT <met_help at ucar.edu>
Sent: Tuesday, December 18, 2018 1:26 PM
To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov>
Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files with unique
field in pcp_combine
Hi Jon. Unfortunately, our MET expert is out of the office for a
while due to the holidays. I do not know the answers to your
questions, however I can try to help.
For starters, I'm just trying to work with the netcdf files to see if
I can
> combine them into "accumulated" fields using pcp_combine, but I'm
> having trouble getting it to work.
>
What kind of trouble are you having? If you'd like you could post the
data to our ftp server ( following the instructions listed here:
http://www.dtcenter.org/met/users/support/met_help.php#ftp) and send
me the command that you are running and I can see if I can help.
Thanks,
Julie
On Tue, Dec 18, 2018 at 11:31 AM Case, Jonathan[ENSCO INC] via RT <
met_help at ucar.edu> wrote:
>
> Tue Dec 18 11:31:09 2018: Request 88228 was acted upon.
> Transaction: Ticket created by jonathan.case-1 at nasa.gov
> Queue: met_help
> Subject: using netcdf files with unique field in pcp_combine
> Owner: Nobody
> Requestors: jonathan.case-1 at nasa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228
> >
>
>
> Dear MET Help,
>
> I'm trying to do something fairly unique in MET by validating
derived
> total lightning flash rates from an NWP model against observed
gridded
> lightning flash rates.
> I'm "faking" MET into thinking the model flash rates are simply
> accumulated precip (already named as APCP in GRIB2 files), whereas
the
> observed lightning files are identified as "FlashRate" in hourly
> netcdf files.
>
> For starters, I'm just trying to work with the netcdf files to see
if
> I can combine them into "accumulated" fields using pcp_combine, but
> I'm having trouble getting it to work.
> What I'd like to do is combine/accumulate the data into various time
> intervals [much like precipitation], and then output the pcp_combine
> netcdf file with the traditional APCP name.
>
> Is there a way I can do this using pcp_combine? I'm still on MET
> v6.0. I have an ncdump of the lightning file below that might be
helpful.
>
> Thanks!
> -Jon
>
>
> Ncdump of sample lightning netcdf file:
> :=> ncdump -h ./precip_tmp/entln_met_2018033001.nc
> netcdf entln_met_2018033001 {
> dimensions:
> lon = 425 ;
> lat = 350 ;
> time = UNLIMITED ; // (1 currently)
> variables:
> float lat(lat) ;
> lat:units = "degrees_north" ;
> lat:long_name = "latitude" ;
> float lon(lon) ;
> lon:units = "degrees east" ;
> lon:long_name = "longitude" ;
> double Time(time) ;
> Time:units = "seconds since " ;
> Time:calendar = "standard" ;
> Time:long_name = "time" ;
> float FlashRate(time, lat, lon) ;
> FlashRate:units = "Flashes per km2 per 5 min" ;
> FlashRate:long_name = "" ;
> FlashRate:coordinates = "lon lat time" ;
>
> // global attributes:
> :Description = "" ;
> :Conventions = "CF-1.7" ;
> :title = "" ;
> :institution = "" ;
> :source = "{0}" ;
> :references = "" ;
> :history = "Created Mon Dec 17 22:23:33 2018" ;
> :comment = "" ;
> }
>
>
>
>
> *****************************************************
> Jonathan Case (Jon)
> ENSCO, Inc./NASA SPoRT Center
> National Space Science and Technology Center
> 320 Sparkman Drive, Room 3000B
> Huntsville, AL 35805
> Jonathan.Case-1 at nasa.gov<mailto:Jonathan.Case-1 at nasa.gov> (alt:
> case.jonathan at ensco.com<mailto:case.jonathan at ensco.com>)
> *****************************************************
>
>
>
------------------------------------------------
Subject: using netcdf files with unique field in pcp_combine
From: Howard Soh
Time: Tue Dec 18 15:36:44 2018
The warning is caused by the variable name "Time" (vs. "time").
MET is looking for "time" variable as CF NetCDF..
double Time(time) ;
Time:units = "seconds since " ;
Time:calendar = "standard" ;
Time:long_name = "time" ;
And time:unit attribute does not have the reference time.
time:units = "seconds since YYYY-MM-DD" ;
OR
time:units = "seconds since YYYY-MM-DD HH:MM:SS" ;
If the verbose level is increased ("-v 4" or "-v 5"), it will display
more
useful information.
Cheers,
Howard
On Tue, Dec 18, 2018 at 1:38 PM Case, Jonathan[ENSCO INC] via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228 >
>
> Sorry Julie,
>
> I should have also included the pcp_combine error I'm getting:
>
> sc1login1:/raid1/sport/people/casejl/MET/gridData:=> pcp_combine
-sum
> 00000000_000000 01 20180330_060000 06 ./observed_APCP_06 -pcpdir
> ./precip_tmp -field 'name="FlashRate"; level="R1";' -v 3 -compress 0
> DEBUG 2: Performing sum command: Init/In_Accum/Valid/Out_Accum Times
=
> 00000000_000000/010000/20180330_060000/060000
> DEBUG 2: Searching for 6 files with accumulation times of 010000 to
sum to
> a total accumulation time of 060000.
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> WARNING:
> terminate called after throwing an instance of
> 'netCDF::exceptions::NcBadId'
> what(): NcBadId: Not a netcdf id
> file: ncDim.cpp line:80
> Aborted
>
>
> I need to know what the "NcBadId" means so I can persue a possible
> solution.
>
> Thx!
> -Jon
>
>
> -----Original Message-----
> From: Julie Prestopnik via RT <met_help at ucar.edu>
> Sent: Tuesday, December 18, 2018 1:26 PM
> To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov>
> Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files with unique
> field in pcp_combine
>
> Hi Jon. Unfortunately, our MET expert is out of the office for a
while
> due to the holidays. I do not know the answers to your questions,
however
> I can try to help.
>
> For starters, I'm just trying to work with the netcdf files to see
if I can
> > combine them into "accumulated" fields using pcp_combine, but I'm
> > having trouble getting it to work.
> >
>
> What kind of trouble are you having? If you'd like you could post
the
> data to our ftp server ( following the instructions listed here:
> http://www.dtcenter.org/met/users/support/met_help.php#ftp) and send
me
> the command that you are running and I can see if I can help.
>
> Thanks,
> Julie
>
> On Tue, Dec 18, 2018 at 11:31 AM Case, Jonathan[ENSCO INC] via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Dec 18 11:31:09 2018: Request 88228 was acted upon.
> > Transaction: Ticket created by jonathan.case-1 at nasa.gov
> > Queue: met_help
> > Subject: using netcdf files with unique field in pcp_combine
> > Owner: Nobody
> > Requestors: jonathan.case-1 at nasa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228
> > >
> >
> >
> > Dear MET Help,
> >
> > I'm trying to do something fairly unique in MET by validating
derived
> > total lightning flash rates from an NWP model against observed
gridded
> > lightning flash rates.
> > I'm "faking" MET into thinking the model flash rates are simply
> > accumulated precip (already named as APCP in GRIB2 files), whereas
the
> > observed lightning files are identified as "FlashRate" in hourly
> > netcdf files.
> >
> > For starters, I'm just trying to work with the netcdf files to see
if
> > I can combine them into "accumulated" fields using pcp_combine,
but
> > I'm having trouble getting it to work.
> > What I'd like to do is combine/accumulate the data into various
time
> > intervals [much like precipitation], and then output the
pcp_combine
> > netcdf file with the traditional APCP name.
> >
> > Is there a way I can do this using pcp_combine? I'm still on MET
> > v6.0. I have an ncdump of the lightning file below that might be
> helpful.
> >
> > Thanks!
> > -Jon
> >
> >
> > Ncdump of sample lightning netcdf file:
> > :=> ncdump -h ./precip_tmp/entln_met_2018033001.nc
> > netcdf entln_met_2018033001 {
> > dimensions:
> > lon = 425 ;
> > lat = 350 ;
> > time = UNLIMITED ; // (1 currently)
> > variables:
> > float lat(lat) ;
> > lat:units = "degrees_north" ;
> > lat:long_name = "latitude" ;
> > float lon(lon) ;
> > lon:units = "degrees east" ;
> > lon:long_name = "longitude" ;
> > double Time(time) ;
> > Time:units = "seconds since " ;
> > Time:calendar = "standard" ;
> > Time:long_name = "time" ;
> > float FlashRate(time, lat, lon) ;
> > FlashRate:units = "Flashes per km2 per 5 min" ;
> > FlashRate:long_name = "" ;
> > FlashRate:coordinates = "lon lat time" ;
> >
> > // global attributes:
> > :Description = "" ;
> > :Conventions = "CF-1.7" ;
> > :title = "" ;
> > :institution = "" ;
> > :source = "{0}" ;
> > :references = "" ;
> > :history = "Created Mon Dec 17 22:23:33 2018" ;
> > :comment = "" ;
> > }
> >
> >
> >
> >
> > *****************************************************
> > Jonathan Case (Jon)
> > ENSCO, Inc./NASA SPoRT Center
> > National Space Science and Technology Center
> > 320 Sparkman Drive, Room 3000B
> > Huntsville, AL 35805
> > Jonathan.Case-1 at nasa.gov<mailto:Jonathan.Case-1 at nasa.gov> (alt:
> > case.jonathan at ensco.com<mailto:case.jonathan at ensco.com>)
> > *****************************************************
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: using netcdf files with unique field in pcp_combine
From: Julie Prestopnik
Time: Wed Dec 19 10:44:52 2018
Hi Jonathan. Thank you for including the error that you are getting.
The warning is caused by the variable name "Time" (vs. "time"). MET
is
looking for a "time" variable, as specified by CF compliant files.
double Time(time) ;
Time:units = "seconds since " ;
Time:calendar = "standard" ;
Time:long_name = "time" ;
And time:unit attribute does not have the reference time.
time:units = "seconds since YYYY-MM-DD" ;
OR
time:units = "seconds since YYYY-MM-DD HH:MM:SS" ;
Could you please modify your file?
Also, if the verbosity level is increased ("-v 4" or "-v 5"), MET will
display more useful information, which may be useful in helping with
the
error. Could you please rerun with a higher verbosity level as well?
Thanks,
Julie
On Tue, Dec 18, 2018 at 1:38 PM Case, Jonathan[ENSCO INC] via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228 >
>
> Sorry Julie,
>
> I should have also included the pcp_combine error I'm getting:
>
> sc1login1:/raid1/sport/people/casejl/MET/gridData:=> pcp_combine
-sum
> 00000000_000000 01 20180330_060000 06 ./observed_APCP_06 -pcpdir
> ./precip_tmp -field 'name="FlashRate"; level="R1";' -v 3 -compress 0
> DEBUG 2: Performing sum command: Init/In_Accum/Valid/Out_Accum Times
=
> 00000000_000000/010000/20180330_060000/060000
> DEBUG 2: Searching for 6 files with accumulation times of 010000 to
sum to
> a total accumulation time of 060000.
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> WARNING:
> terminate called after throwing an instance of
> 'netCDF::exceptions::NcBadId'
> what(): NcBadId: Not a netcdf id
> file: ncDim.cpp line:80
> Aborted
>
>
> I need to know what the "NcBadId" means so I can persue a possible
> solution.
>
> Thx!
> -Jon
>
>
> -----Original Message-----
> From: Julie Prestopnik via RT <met_help at ucar.edu>
> Sent: Tuesday, December 18, 2018 1:26 PM
> To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov>
> Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files with unique
> field in pcp_combine
>
> Hi Jon. Unfortunately, our MET expert is out of the office for a
while
> due to the holidays. I do not know the answers to your questions,
however
> I can try to help.
>
> For starters, I'm just trying to work with the netcdf files to see
if I can
> > combine them into "accumulated" fields using pcp_combine, but I'm
> > having trouble getting it to work.
> >
>
> What kind of trouble are you having? If you'd like you could post
the
> data to our ftp server ( following the instructions listed here:
> http://www.dtcenter.org/met/users/support/met_help.php#ftp) and send
me
> the command that you are running and I can see if I can help.
>
> Thanks,
> Julie
>
> On Tue, Dec 18, 2018 at 11:31 AM Case, Jonathan[ENSCO INC] via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Dec 18 11:31:09 2018: Request 88228 was acted upon.
> > Transaction: Ticket created by jonathan.case-1 at nasa.gov
> > Queue: met_help
> > Subject: using netcdf files with unique field in pcp_combine
> > Owner: Nobody
> > Requestors: jonathan.case-1 at nasa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228
> > >
> >
> >
> > Dear MET Help,
> >
> > I'm trying to do something fairly unique in MET by validating
derived
> > total lightning flash rates from an NWP model against observed
gridded
> > lightning flash rates.
> > I'm "faking" MET into thinking the model flash rates are simply
> > accumulated precip (already named as APCP in GRIB2 files), whereas
the
> > observed lightning files are identified as "FlashRate" in hourly
> > netcdf files.
> >
> > For starters, I'm just trying to work with the netcdf files to see
if
> > I can combine them into "accumulated" fields using pcp_combine,
but
> > I'm having trouble getting it to work.
> > What I'd like to do is combine/accumulate the data into various
time
> > intervals [much like precipitation], and then output the
pcp_combine
> > netcdf file with the traditional APCP name.
> >
> > Is there a way I can do this using pcp_combine? I'm still on MET
> > v6.0. I have an ncdump of the lightning file below that might be
> helpful.
> >
> > Thanks!
> > -Jon
> >
> >
> > Ncdump of sample lightning netcdf file:
> > :=> ncdump -h ./precip_tmp/entln_met_2018033001.nc
> > netcdf entln_met_2018033001 {
> > dimensions:
> > lon = 425 ;
> > lat = 350 ;
> > time = UNLIMITED ; // (1 currently)
> > variables:
> > float lat(lat) ;
> > lat:units = "degrees_north" ;
> > lat:long_name = "latitude" ;
> > float lon(lon) ;
> > lon:units = "degrees east" ;
> > lon:long_name = "longitude" ;
> > double Time(time) ;
> > Time:units = "seconds since " ;
> > Time:calendar = "standard" ;
> > Time:long_name = "time" ;
> > float FlashRate(time, lat, lon) ;
> > FlashRate:units = "Flashes per km2 per 5 min" ;
> > FlashRate:long_name = "" ;
> > FlashRate:coordinates = "lon lat time" ;
> >
> > // global attributes:
> > :Description = "" ;
> > :Conventions = "CF-1.7" ;
> > :title = "" ;
> > :institution = "" ;
> > :source = "{0}" ;
> > :references = "" ;
> > :history = "Created Mon Dec 17 22:23:33 2018" ;
> > :comment = "" ;
> > }
> >
> >
> >
> >
> > *****************************************************
> > Jonathan Case (Jon)
> > ENSCO, Inc./NASA SPoRT Center
> > National Space Science and Technology Center
> > 320 Sparkman Drive, Room 3000B
> > Huntsville, AL 35805
> > Jonathan.Case-1 at nasa.gov<mailto:Jonathan.Case-1 at nasa.gov> (alt:
> > case.jonathan at ensco.com<mailto:case.jonathan at ensco.com>)
> > *****************************************************
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #88228] using netcdf files with unique field in pcp_combine
From: Case, Jonathan[ENSCO INC]
Time: Thu Dec 20 12:21:46 2018
Hello again,
I ran pcp_combine with -v5 log level. It appears that the time is
just a warning, whereas the program crashes in response to the
NcBadId, which I'm not familiar with.
Any insight on that? (see below)
Sincerely,
Jonathan
pcp_combine -sum 00000000_000000 01 20180330_060000 06
./observed_APCP_06 -pcpdir ./precip_tmp -field 'name="FlashRate";
level="R1";' -v 5 -compress 0
DEBUG 2: Performing sum command: Init/In_Accum/Valid/Out_Accum Times =
00000000_000000/010000/20180330_060000/060000
DEBUG 2: Searching for 6 files with accumulation times of 010000 to
sum to a total accumulation time of 060000.
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> could not extract valid time from the
"time" variable.
DEBUG 4: NcCfFile::open() -> could not extract valid time from file
name.
WARNING:
WARNING: NcCfFile::open() -> could not determine the valid time, using
0.
WARNING:
DEBUG 4: NcCfFile::open() -> could not extract init time from the
"forecast_reference_time" variable.
DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
terminate called after throwing an instance of
'netCDF::exceptions::NcBadId'
what(): NcBadId: Not a netcdf id
file: ncDim.cpp line:80
Aborted
-----Original Message-----
From: Howard Soh via RT <met_help at ucar.edu>
Sent: Tuesday, December 18, 2018 4:37 PM
To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov>
Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files with unique
field in pcp_combine
The warning is caused by the variable name "Time" (vs. "time").
MET is looking for "time" variable as CF NetCDF..
double Time(time) ;
Time:units = "seconds since " ;
Time:calendar = "standard" ;
Time:long_name = "time" ;
And time:unit attribute does not have the reference time.
time:units = "seconds since YYYY-MM-DD" ;
OR
time:units = "seconds since YYYY-MM-DD HH:MM:SS" ;
If the verbose level is increased ("-v 4" or "-v 5"), it will display
more useful information.
Cheers,
Howard
On Tue, Dec 18, 2018 at 1:38 PM Case, Jonathan[ENSCO INC] via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228 >
>
> Sorry Julie,
>
> I should have also included the pcp_combine error I'm getting:
>
> sc1login1:/raid1/sport/people/casejl/MET/gridData:=> pcp_combine
-sum
> 00000000_000000 01 20180330_060000 06 ./observed_APCP_06 -pcpdir
> ./precip_tmp -field 'name="FlashRate"; level="R1";' -v 3 -compress 0
> DEBUG 2: Performing sum command: Init/In_Accum/Valid/Out_Accum Times
=
> 00000000_000000/010000/20180330_060000/060000
> DEBUG 2: Searching for 6 files with accumulation times of 010000 to
> sum to a total accumulation time of 060000.
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> WARNING:
> terminate called after throwing an instance of
> 'netCDF::exceptions::NcBadId'
> what(): NcBadId: Not a netcdf id
> file: ncDim.cpp line:80
> Aborted
>
>
> I need to know what the "NcBadId" means so I can persue a possible
> solution.
>
> Thx!
> -Jon
>
>
> -----Original Message-----
> From: Julie Prestopnik via RT <met_help at ucar.edu>
> Sent: Tuesday, December 18, 2018 1:26 PM
> To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov>
> Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files with unique
> field in pcp_combine
>
> Hi Jon. Unfortunately, our MET expert is out of the office for a
> while due to the holidays. I do not know the answers to your
> questions, however I can try to help.
>
> For starters, I'm just trying to work with the netcdf files to see
if
> I can
> > combine them into "accumulated" fields using pcp_combine, but I'm
> > having trouble getting it to work.
> >
>
> What kind of trouble are you having? If you'd like you could post
the
> data to our ftp server ( following the instructions listed here:
> http://www.dtcenter.org/met/users/support/met_help.php#ftp) and send
> me the command that you are running and I can see if I can help.
>
> Thanks,
> Julie
>
> On Tue, Dec 18, 2018 at 11:31 AM Case, Jonathan[ENSCO INC] via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Dec 18 11:31:09 2018: Request 88228 was acted upon.
> > Transaction: Ticket created by jonathan.case-1 at nasa.gov
> > Queue: met_help
> > Subject: using netcdf files with unique field in pcp_combine
> > Owner: Nobody
> > Requestors: jonathan.case-1 at nasa.gov
> > Status: new
> > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228
> > >
> >
> >
> > Dear MET Help,
> >
> > I'm trying to do something fairly unique in MET by validating
> > derived total lightning flash rates from an NWP model against
> > observed gridded lightning flash rates.
> > I'm "faking" MET into thinking the model flash rates are simply
> > accumulated precip (already named as APCP in GRIB2 files), whereas
> > the observed lightning files are identified as "FlashRate" in
hourly
> > netcdf files.
> >
> > For starters, I'm just trying to work with the netcdf files to see
> > if I can combine them into "accumulated" fields using pcp_combine,
> > but I'm having trouble getting it to work.
> > What I'd like to do is combine/accumulate the data into various
time
> > intervals [much like precipitation], and then output the
pcp_combine
> > netcdf file with the traditional APCP name.
> >
> > Is there a way I can do this using pcp_combine? I'm still on MET
> > v6.0. I have an ncdump of the lightning file below that might be
> helpful.
> >
> > Thanks!
> > -Jon
> >
> >
> > Ncdump of sample lightning netcdf file:
> > :=> ncdump -h ./precip_tmp/entln_met_2018033001.nc
> > netcdf entln_met_2018033001 {
> > dimensions:
> > lon = 425 ;
> > lat = 350 ;
> > time = UNLIMITED ; // (1 currently)
> > variables:
> > float lat(lat) ;
> > lat:units = "degrees_north" ;
> > lat:long_name = "latitude" ;
> > float lon(lon) ;
> > lon:units = "degrees east" ;
> > lon:long_name = "longitude" ;
> > double Time(time) ;
> > Time:units = "seconds since " ;
> > Time:calendar = "standard" ;
> > Time:long_name = "time" ;
> > float FlashRate(time, lat, lon) ;
> > FlashRate:units = "Flashes per km2 per 5 min" ;
> > FlashRate:long_name = "" ;
> > FlashRate:coordinates = "lon lat time" ;
> >
> > // global attributes:
> > :Description = "" ;
> > :Conventions = "CF-1.7" ;
> > :title = "" ;
> > :institution = "" ;
> > :source = "{0}" ;
> > :references = "" ;
> > :history = "Created Mon Dec 17 22:23:33 2018" ;
> > :comment = "" ;
> > }
> >
> >
> >
> >
> > *****************************************************
> > Jonathan Case (Jon)
> > ENSCO, Inc./NASA SPoRT Center
> > National Space Science and Technology Center
> > 320 Sparkman Drive, Room 3000B
> > Huntsville, AL 35805
> > Jonathan.Case-1 at nasa.gov<mailto:Jonathan.Case-1 at nasa.gov> (alt:
> > case.jonathan at ensco.com<mailto:case.jonathan at ensco.com>)
> > *****************************************************
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: using netcdf files with unique field in pcp_combine
From: Howard Soh
Time: Thu Dec 20 13:41:49 2018
The error message is from the NetCDF library. The line:80 means it
failed to get the dimension size because the dimension ID is not
valid. So it could be a bad NetCDF file at ./precip_tmp.
Would you post the data and configuration files as Julie suggested
before?
What kind of trouble are you having? If you'd like you could
post the data to our ftp server ( following the instructions
listed here:
http://www.dtcenter.org/met/users/support/met_help.php#ftp)
and send me the command that you are running and I can see if I can
help.
If you won't post the data, would you do these?
Option 1:
- Run plot_data_plane for FlashRate variable with each NetCDF files at
./precip_tmp
- Check if all of them have the same problem
Option 2:
- check if the NetCDF files at ./precip_tmp have the same header.
- keep "entln_met_2018033001" at ./precip_tmp
- move other NetCDF files to somewhere else
- run pcp_combined command
- check if the same problem happens.
- If this works, add NetCDF file one by one.
Cheers,
Howard
On Thu Dec 20 12:21:46 2018, jonathan.case-1 at nasa.gov wrote:
> Hello again,
>
> I ran pcp_combine with -v5 log level. It appears that the time is
> just a warning, whereas the program crashes in response to the
> NcBadId, which I'm not familiar with.
> Any insight on that? (see below)
>
> Sincerely,
> Jonathan
>
> pcp_combine -sum 00000000_000000 01 20180330_060000 06
> ./observed_APCP_06 -pcpdir ./precip_tmp -field 'name="FlashRate";
> level="R1";' -v 5 -compress 0
> DEBUG 2: Performing sum command: Init/In_Accum/Valid/Out_Accum Times
=
> 00000000_000000/010000/20180330_060000/060000
> DEBUG 2: Searching for 6 files with accumulation times of 010000 to
> sum to a total accumulation time of 060000.
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
> DEBUG 4: NcCfFile::open() -> could not extract valid time from the
> "time" variable.
> DEBUG 4: NcCfFile::open() -> could not extract valid time from file
> name.
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using
> 0.
> WARNING:
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
> name.
> terminate called after throwing an instance of
> 'netCDF::exceptions::NcBadId'
> what(): NcBadId: Not a netcdf id
> file: ncDim.cpp line:80
> Aborted
>
>
>
> -----Original Message-----
> From: Howard Soh via RT <met_help at ucar.edu>
> Sent: Tuesday, December 18, 2018 4:37 PM
> To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov>
> Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files with unique
> field in pcp_combine
>
> The warning is caused by the variable name "Time" (vs. "time").
> MET is looking for "time" variable as CF NetCDF..
>
> double Time(time) ;
> Time:units = "seconds since " ;
> Time:calendar = "standard" ;
> Time:long_name = "time" ;
>
> And time:unit attribute does not have the reference time.
> time:units = "seconds since YYYY-MM-DD" ;
> OR
> time:units = "seconds since YYYY-MM-DD HH:MM:SS" ;
>
> If the verbose level is increased ("-v 4" or "-v 5"), it will
display
> more useful information.
>
> Cheers,
> Howard
>
> On Tue, Dec 18, 2018 at 1:38 PM Case, Jonathan[ENSCO INC] via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228 >
> >
> > Sorry Julie,
> >
> > I should have also included the pcp_combine error I'm getting:
> >
> > sc1login1:/raid1/sport/people/casejl/MET/gridData:=> pcp_combine
-sum
> > 00000000_000000 01 20180330_060000 06 ./observed_APCP_06 -pcpdir
> > ./precip_tmp -field 'name="FlashRate"; level="R1";' -v 3
-compress 0
> > DEBUG 2: Performing sum command: Init/In_Accum/Valid/Out_Accum
Times
> > =
> > 00000000_000000/010000/20180330_060000/060000
> > DEBUG 2: Searching for 6 files with accumulation times of 010000
to
> > sum to a total accumulation time of 060000.
> > WARNING:
> > WARNING: NcCfFile::open() -> could not determine the valid time,
> > using 0.
> > WARNING:
> > terminate called after throwing an instance of
> > 'netCDF::exceptions::NcBadId'
> > what(): NcBadId: Not a netcdf id
> > file: ncDim.cpp line:80
> > Aborted
> >
> >
> > I need to know what the "NcBadId" means so I can persue a possible
> > solution.
> >
> > Thx!
> > -Jon
> >
> >
> > -----Original Message-----
> > From: Julie Prestopnik via RT <met_help at ucar.edu>
> > Sent: Tuesday, December 18, 2018 1:26 PM
> > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
1 at nasa.gov>
> > Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files with
unique
> > field in pcp_combine
> >
> > Hi Jon. Unfortunately, our MET expert is out of the office for a
> > while due to the holidays. I do not know the answers to your
> > questions, however I can try to help.
> >
> > For starters, I'm just trying to work with the netcdf files to see
if
> > I can
> > > combine them into "accumulated" fields using pcp_combine, but
I'm
> > > having trouble getting it to work.
> > >
> >
> > What kind of trouble are you having? If you'd like you could post
> > the
> > data to our ftp server ( following the instructions listed here:
> > http://www.dtcenter.org/met/users/support/met_help.php#ftp) and
send
> > me the command that you are running and I can see if I can help.
> >
> > Thanks,
> > Julie
> >
> > On Tue, Dec 18, 2018 at 11:31 AM Case, Jonathan[ENSCO INC] via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Dec 18 11:31:09 2018: Request 88228 was acted upon.
> > > Transaction: Ticket created by jonathan.case-1 at nasa.gov
> > > Queue: met_help
> > > Subject: using netcdf files with unique field in
pcp_combine
> > > Owner: Nobody
> > > Requestors: jonathan.case-1 at nasa.gov
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228
> > > >
> > >
> > >
> > > Dear MET Help,
> > >
> > > I'm trying to do something fairly unique in MET by validating
> > > derived total lightning flash rates from an NWP model against
> > > observed gridded lightning flash rates.
> > > I'm "faking" MET into thinking the model flash rates are simply
> > > accumulated precip (already named as APCP in GRIB2 files),
whereas
> > > the observed lightning files are identified as "FlashRate" in
> > > hourly
> > > netcdf files.
> > >
> > > For starters, I'm just trying to work with the netcdf files to
see
> > > if I can combine them into "accumulated" fields using
pcp_combine,
> > > but I'm having trouble getting it to work.
> > > What I'd like to do is combine/accumulate the data into various
> > > time
> > > intervals [much like precipitation], and then output the
> > > pcp_combine
> > > netcdf file with the traditional APCP name.
> > >
> > > Is there a way I can do this using pcp_combine? I'm still on
MET
> > > v6.0. I have an ncdump of the lightning file below that might
be
> > helpful.
> > >
> > > Thanks!
> > > -Jon
> > >
> > >
> > > Ncdump of sample lightning netcdf file:
> > > :=> ncdump -h ./precip_tmp/entln_met_2018033001.nc
> > > netcdf entln_met_2018033001 {
> > > dimensions:
> > > lon = 425 ;
> > > lat = 350 ;
> > > time = UNLIMITED ; // (1 currently)
> > > variables:
> > > float lat(lat) ;
> > > lat:units = "degrees_north" ;
> > > lat:long_name = "latitude" ;
> > > float lon(lon) ;
> > > lon:units = "degrees east" ;
> > > lon:long_name = "longitude" ;
> > > double Time(time) ;
> > > Time:units = "seconds since " ;
> > > Time:calendar = "standard" ;
> > > Time:long_name = "time" ;
> > > float FlashRate(time, lat, lon) ;
> > > FlashRate:units = "Flashes per km2 per 5 min" ;
> > > FlashRate:long_name = "" ;
> > > FlashRate:coordinates = "lon lat time" ;
> > >
> > > // global attributes:
> > > :Description = "" ;
> > > :Conventions = "CF-1.7" ;
> > > :title = "" ;
> > > :institution = "" ;
> > > :source = "{0}" ;
> > > :references = "" ;
> > > :history = "Created Mon Dec 17 22:23:33 2018" ;
> > > :comment = "" ;
> > > }
> > >
> > >
> > >
> > >
> > > *****************************************************
> > > Jonathan Case (Jon)
> > > ENSCO, Inc./NASA SPoRT Center
> > > National Space Science and Technology Center
> > > 320 Sparkman Drive, Room 3000B
> > > Huntsville, AL 35805
> > > Jonathan.Case-1 at nasa.gov<mailto:Jonathan.Case-1 at nasa.gov> (alt:
> > > case.jonathan at ensco.com<mailto:case.jonathan at ensco.com>)
> > > *****************************************************
> > >
> > >
> > >
> >
> >
> >
> >
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #88228] using netcdf files with unique field in pcp_combine
From: Case, Jonathan[ENSCO INC]
Time: Fri Dec 21 10:39:36 2018
Hello again Howard,
I tried running plot_data_place, but I immediately get a cryptic
error:
sc1login1:/raid1/sport/people/casejl/MET/gridData/precip_tmp:=>
plot_data_plane entln_met_2018033001.nc test FlashRate -v 5
ERROR :
ERROR : yyerror() -> syntax error in file "/tmp/config_20992_0_.temp"
ERROR :
ERROR : line = 2
ERROR :
ERROR : column = 1
ERROR :
ERROR : text = "(nul)"
ERROR :
ERROR :
ERROR : (nul)
ERROR : (nul)
ERROR :
I uploaded the sample netcdf files to our public-facing server at:
https://geo.nsstc.nasa.gov/SPoRT/outgoing/jlc/MET/
(file is flashrate.tar, which will unpack in to the precip_tmp sub-
folder). Please keep the data among yourself because as you're likely
aware, ENTLN lightning data are not freely open to the public.
All I really need to know is how we need to set up these netcdf files
correctly so that pcp_combine can operate on them. Once that hurdle
is cleared, then I'm confident that I can output the pcp_combine
intermediate netcdf output files for use within grid_stat.
Thanks for your continued help.
-Jonathan
-----Original Message-----
From: Howard Soh via RT <met_help at ucar.edu>
Sent: Thursday, December 20, 2018 2:42 PM
To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov>
Subject: [rt.rap.ucar.edu #88228] using netcdf files with unique field
in pcp_combine
The error message is from the NetCDF library. The line:80 means it
failed to get the dimension size because the dimension ID is not
valid. So it could be a bad NetCDF file at ./precip_tmp.
Would you post the data and configuration files as Julie suggested
before?
What kind of trouble are you having? If you'd like you could
post the data to our ftp server ( following the instructions
listed here:
http://www.dtcenter.org/met/users/support/met_help.php#ftp)
and send me the command that you are running and I can see if I can
help.
If you won't post the data, would you do these?
Option 1:
- Run plot_data_plane for FlashRate variable with each NetCDF files at
./precip_tmp
- Check if all of them have the same problem
Option 2:
- check if the NetCDF files at ./precip_tmp have the same header.
- keep "entln_met_2018033001" at ./precip_tmp
- move other NetCDF files to somewhere else
- run pcp_combined command
- check if the same problem happens.
- If this works, add NetCDF file one by one.
Cheers,
Howard
On Thu Dec 20 12:21:46 2018, jonathan.case-1 at nasa.gov wrote:
> Hello again,
>
> I ran pcp_combine with -v5 log level. It appears that the time is
> just a warning, whereas the program crashes in response to the
> NcBadId, which I'm not familiar with.
> Any insight on that? (see below)
>
> Sincerely,
> Jonathan
>
> pcp_combine -sum 00000000_000000 01 20180330_060000 06
> ./observed_APCP_06 -pcpdir ./precip_tmp -field 'name="FlashRate";
> level="R1";' -v 5 -compress 0 DEBUG 2: Performing sum command:
> Init/In_Accum/Valid/Out_Accum Times =
> 00000000_000000/010000/20180330_060000/060000
> DEBUG 2: Searching for 6 files with accumulation times of 010000 to
> sum to a total accumulation time of 060000.
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
> DEBUG 4: NcCfFile::open() -> could not extract valid time from the
> "time" variable.
> DEBUG 4: NcCfFile::open() -> could not extract valid time from file
> name.
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using
> 0.
> WARNING:
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
> name.
> terminate called after throwing an instance of
> 'netCDF::exceptions::NcBadId'
> what(): NcBadId: Not a netcdf id
> file: ncDim.cpp line:80
> Aborted
>
>
>
> -----Original Message-----
> From: Howard Soh via RT <met_help at ucar.edu>
> Sent: Tuesday, December 18, 2018 4:37 PM
> To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov>
> Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files with unique
> field in pcp_combine
>
> The warning is caused by the variable name "Time" (vs. "time").
> MET is looking for "time" variable as CF NetCDF..
>
> double Time(time) ;
> Time:units = "seconds since " ;
> Time:calendar = "standard" ;
> Time:long_name = "time" ;
>
> And time:unit attribute does not have the reference time.
> time:units = "seconds since YYYY-MM-DD" ;
> OR
> time:units = "seconds since YYYY-MM-DD HH:MM:SS" ;
>
> If the verbose level is increased ("-v 4" or "-v 5"), it will
display
> more useful information.
>
> Cheers,
> Howard
>
> On Tue, Dec 18, 2018 at 1:38 PM Case, Jonathan[ENSCO INC] via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228 >
> >
> > Sorry Julie,
> >
> > I should have also included the pcp_combine error I'm getting:
> >
> > sc1login1:/raid1/sport/people/casejl/MET/gridData:=> pcp_combine
> > -sum
> > 00000000_000000 01 20180330_060000 06 ./observed_APCP_06 -pcpdir
> > ./precip_tmp -field 'name="FlashRate"; level="R1";' -v 3 -compress
0
> > DEBUG 2: Performing sum command: Init/In_Accum/Valid/Out_Accum
Times
> > =
> > 00000000_000000/010000/20180330_060000/060000
> > DEBUG 2: Searching for 6 files with accumulation times of 010000
to
> > sum to a total accumulation time of 060000.
> > WARNING:
> > WARNING: NcCfFile::open() -> could not determine the valid time,
> > using 0.
> > WARNING:
> > terminate called after throwing an instance of
> > 'netCDF::exceptions::NcBadId'
> > what(): NcBadId: Not a netcdf id
> > file: ncDim.cpp line:80
> > Aborted
> >
> >
> > I need to know what the "NcBadId" means so I can persue a possible
> > solution.
> >
> > Thx!
> > -Jon
> >
> >
> > -----Original Message-----
> > From: Julie Prestopnik via RT <met_help at ucar.edu>
> > Sent: Tuesday, December 18, 2018 1:26 PM
> > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
1 at nasa.gov>
> > Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files with
> > unique field in pcp_combine
> >
> > Hi Jon. Unfortunately, our MET expert is out of the office for a
> > while due to the holidays. I do not know the answers to your
> > questions, however I can try to help.
> >
> > For starters, I'm just trying to work with the netcdf files to see
> > if I can
> > > combine them into "accumulated" fields using pcp_combine, but
I'm
> > > having trouble getting it to work.
> > >
> >
> > What kind of trouble are you having? If you'd like you could post
> > the data to our ftp server ( following the instructions listed
here:
> > http://www.dtcenter.org/met/users/support/met_help.php#ftp) and
send
> > me the command that you are running and I can see if I can help.
> >
> > Thanks,
> > Julie
> >
> > On Tue, Dec 18, 2018 at 11:31 AM Case, Jonathan[ENSCO INC] via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Dec 18 11:31:09 2018: Request 88228 was acted upon.
> > > Transaction: Ticket created by jonathan.case-1 at nasa.gov
> > > Queue: met_help
> > > Subject: using netcdf files with unique field in
pcp_combine
> > > Owner: Nobody
> > > Requestors: jonathan.case-1 at nasa.gov
> > > Status: new
> > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228
> > > >
> > >
> > >
> > > Dear MET Help,
> > >
> > > I'm trying to do something fairly unique in MET by validating
> > > derived total lightning flash rates from an NWP model against
> > > observed gridded lightning flash rates.
> > > I'm "faking" MET into thinking the model flash rates are simply
> > > accumulated precip (already named as APCP in GRIB2 files),
whereas
> > > the observed lightning files are identified as "FlashRate" in
> > > hourly netcdf files.
> > >
> > > For starters, I'm just trying to work with the netcdf files to
see
> > > if I can combine them into "accumulated" fields using
pcp_combine,
> > > but I'm having trouble getting it to work.
> > > What I'd like to do is combine/accumulate the data into various
> > > time intervals [much like precipitation], and then output the
> > > pcp_combine netcdf file with the traditional APCP name.
> > >
> > > Is there a way I can do this using pcp_combine? I'm still on
MET
> > > v6.0. I have an ncdump of the lightning file below that might
be
> > helpful.
> > >
> > > Thanks!
> > > -Jon
> > >
> > >
> > > Ncdump of sample lightning netcdf file:
> > > :=> ncdump -h ./precip_tmp/entln_met_2018033001.nc
> > > netcdf entln_met_2018033001 {
> > > dimensions:
> > > lon = 425 ;
> > > lat = 350 ;
> > > time = UNLIMITED ; // (1 currently)
> > > variables:
> > > float lat(lat) ;
> > > lat:units = "degrees_north" ;
> > > lat:long_name = "latitude" ;
> > > float lon(lon) ;
> > > lon:units = "degrees east" ;
> > > lon:long_name = "longitude" ;
> > > double Time(time) ;
> > > Time:units = "seconds since " ;
> > > Time:calendar = "standard" ;
> > > Time:long_name = "time" ;
> > > float FlashRate(time, lat, lon) ;
> > > FlashRate:units = "Flashes per km2 per 5 min" ;
> > > FlashRate:long_name = "" ;
> > > FlashRate:coordinates = "lon lat time" ;
> > >
> > > // global attributes:
> > > :Description = "" ;
> > > :Conventions = "CF-1.7" ;
> > > :title = "" ;
> > > :institution = "" ;
> > > :source = "{0}" ;
> > > :references = "" ;
> > > :history = "Created Mon Dec 17 22:23:33 2018" ;
> > > :comment = "" ;
> > > }
> > >
> > >
> > >
> > >
> > > *****************************************************
> > > Jonathan Case (Jon)
> > > ENSCO, Inc./NASA SPoRT Center
> > > National Space Science and Technology Center
> > > 320 Sparkman Drive, Room 3000B
> > > Huntsville, AL 35805
> > > Jonathan.Case-1 at nasa.gov<mailto:Jonathan.Case-1 at nasa.gov> (alt:
> > > case.jonathan at ensco.com<mailto:case.jonathan at ensco.com>)
> > > *****************************************************
> > >
> > >
> > >
> >
> >
> >
> >
------------------------------------------------
Subject: using netcdf files with unique field in pcp_combine
From: Howard Soh
Time: Fri Dec 21 11:31:38 2018
Thank you on shared your data.
I downloaded both files. flashrate.tar contains the softlinks only.
Would you post the real NetCDF files?
precip_tmp$ ls -l
total 0
lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:19 entln_met_2018033001.nc ->
../entlnFlashRate_20180330_01Z.nc
lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:19 entln_met_2018033002.nc ->
../entlnFlashRate_20180330_02Z.nc
lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:19 entln_met_2018033003.nc ->
../entlnFlashRate_20180330_03Z.nc
lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:19 entln_met_2018033004.nc ->
../entlnFlashRate_20180330_04Z.nc
lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:20 entln_met_2018033005.nc ->
../entlnFlashRate_20180330_05Z.nc
lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:20 entln_met_2018033006.nc ->
../entlnFlashRate_20180330_06Z.nc
Cheers,
Howard
On Fri Dec 21 10:39:36 2018, jonathan.case-1 at nasa.gov wrote:
> Hello again Howard,
>
> I tried running plot_data_place, but I immediately get a cryptic
> error:
> sc1login1:/raid1/sport/people/casejl/MET/gridData/precip_tmp:=>
> plot_data_plane entln_met_2018033001.nc test FlashRate -v 5
> ERROR :
> ERROR : yyerror() -> syntax error in file
"/tmp/config_20992_0_.temp"
> ERROR :
> ERROR : line = 2
> ERROR :
> ERROR : column = 1
> ERROR :
> ERROR : text = "(nul)"
> ERROR :
> ERROR :
> ERROR : (nul)
> ERROR : (nul)
> ERROR :
>
> I uploaded the sample netcdf files to our public-facing server at:
> https://geo.nsstc.nasa.gov/SPoRT/outgoing/jlc/MET/
> (file is flashrate.tar, which will unpack in to the precip_tmp sub-
> folder). Please keep the data among yourself because as you're
likely
> aware, ENTLN lightning data are not freely open to the public.
>
> All I really need to know is how we need to set up these netcdf
files
> correctly so that pcp_combine can operate on them. Once that hurdle
> is cleared, then I'm confident that I can output the pcp_combine
> intermediate netcdf output files for use within grid_stat.
>
> Thanks for your continued help.
> -Jonathan
>
> -----Original Message-----
> From: Howard Soh via RT <met_help at ucar.edu>
> Sent: Thursday, December 20, 2018 2:42 PM
> To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov>
> Subject: [rt.rap.ucar.edu #88228] using netcdf files with unique
field
> in pcp_combine
>
> The error message is from the NetCDF library. The line:80 means it
> failed to get the dimension size because the dimension ID is not
> valid. So it could be a bad NetCDF file at ./precip_tmp.
>
> Would you post the data and configuration files as Julie suggested
> before?
>
> What kind of trouble are you having? If you'd like you could
> post the data to our ftp server ( following the instructions
> listed here:
> http://www.dtcenter.org/met/users/support/met_help.php#ftp)
> and send me the command that you are running and I can see if I can
> help.
>
> If you won't post the data, would you do these?
>
> Option 1:
>
> - Run plot_data_plane for FlashRate variable with each NetCDF files
at
> ./precip_tmp
> - Check if all of them have the same problem
>
> Option 2:
>
> - check if the NetCDF files at ./precip_tmp have the same header.
> - keep "entln_met_2018033001" at ./precip_tmp
> - move other NetCDF files to somewhere else
> - run pcp_combined command
> - check if the same problem happens.
> - If this works, add NetCDF file one by one.
>
> Cheers,
> Howard
>
> On Thu Dec 20 12:21:46 2018, jonathan.case-1 at nasa.gov wrote:
> > Hello again,
> >
> > I ran pcp_combine with -v5 log level. It appears that the time is
> > just a warning, whereas the program crashes in response to the
> > NcBadId, which I'm not familiar with.
> > Any insight on that? (see below)
> >
> > Sincerely,
> > Jonathan
> >
> > pcp_combine -sum 00000000_000000 01 20180330_060000 06
> > ./observed_APCP_06 -pcpdir ./precip_tmp -field 'name="FlashRate";
> > level="R1";' -v 5 -compress 0 DEBUG 2: Performing sum command:
> > Init/In_Accum/Valid/Out_Accum Times =
> > 00000000_000000/010000/20180330_060000/060000
> > DEBUG 2: Searching for 6 files with accumulation times of 010000
to
> > sum to a total accumulation time of 060000.
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_NcCF".
> > DEBUG 4: NcCfFile::open() -> could not extract valid time from
the
> > "time" variable.
> > DEBUG 4: NcCfFile::open() -> could not extract valid time from
file
> > name.
> > WARNING:
> > WARNING: NcCfFile::open() -> could not determine the valid time,
> > using
> > 0.
> > WARNING:
> > DEBUG 4: NcCfFile::open() -> could not extract init time from the
> > "forecast_reference_time" variable.
> > DEBUG 4: NcCfFile::open() -> could not extract init time from
file
> > name.
> > terminate called after throwing an instance of
> > 'netCDF::exceptions::NcBadId'
> > what(): NcBadId: Not a netcdf id
> > file: ncDim.cpp line:80
> > Aborted
> >
> >
> >
> > -----Original Message-----
> > From: Howard Soh via RT <met_help at ucar.edu>
> > Sent: Tuesday, December 18, 2018 4:37 PM
> > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
1 at nasa.gov>
> > Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files with
unique
> > field in pcp_combine
> >
> > The warning is caused by the variable name "Time" (vs. "time").
> > MET is looking for "time" variable as CF NetCDF..
> >
> > double Time(time) ;
> > Time:units = "seconds since " ;
> > Time:calendar = "standard" ;
> > Time:long_name = "time" ;
> >
> > And time:unit attribute does not have the reference time.
> > time:units = "seconds since YYYY-MM-DD" ;
> > OR
> > time:units = "seconds since YYYY-MM-DD HH:MM:SS" ;
> >
> > If the verbose level is increased ("-v 4" or "-v 5"), it will
display
> > more useful information.
> >
> > Cheers,
> > Howard
> >
> > On Tue, Dec 18, 2018 at 1:38 PM Case, Jonathan[ENSCO INC] via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228 >
> > >
> > > Sorry Julie,
> > >
> > > I should have also included the pcp_combine error I'm getting:
> > >
> > > sc1login1:/raid1/sport/people/casejl/MET/gridData:=> pcp_combine
> > > -sum
> > > 00000000_000000 01 20180330_060000 06 ./observed_APCP_06
-pcpdir
> > > ./precip_tmp -field 'name="FlashRate"; level="R1";' -v 3
-compress
> > > 0
> > > DEBUG 2: Performing sum command: Init/In_Accum/Valid/Out_Accum
> > > Times
> > > =
> > > 00000000_000000/010000/20180330_060000/060000
> > > DEBUG 2: Searching for 6 files with accumulation times of
010000
> > > to
> > > sum to a total accumulation time of 060000.
> > > WARNING:
> > > WARNING: NcCfFile::open() -> could not determine the valid time,
> > > using 0.
> > > WARNING:
> > > terminate called after throwing an instance of
> > > 'netCDF::exceptions::NcBadId'
> > > what(): NcBadId: Not a netcdf id
> > > file: ncDim.cpp line:80
> > > Aborted
> > >
> > >
> > > I need to know what the "NcBadId" means so I can persue a
possible
> > > solution.
> > >
> > > Thx!
> > > -Jon
> > >
> > >
> > > -----Original Message-----
> > > From: Julie Prestopnik via RT <met_help at ucar.edu>
> > > Sent: Tuesday, December 18, 2018 1:26 PM
> > > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
> > > 1 at nasa.gov>
> > > Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files with
> > > unique field in pcp_combine
> > >
> > > Hi Jon. Unfortunately, our MET expert is out of the office for
a
> > > while due to the holidays. I do not know the answers to your
> > > questions, however I can try to help.
> > >
> > > For starters, I'm just trying to work with the netcdf files to
see
> > > if I can
> > > > combine them into "accumulated" fields using pcp_combine, but
I'm
> > > > having trouble getting it to work.
> > > >
> > >
> > > What kind of trouble are you having? If you'd like you could
post
> > > the data to our ftp server ( following the instructions listed
> > > here:
> > > http://www.dtcenter.org/met/users/support/met_help.php#ftp) and
> > > send
> > > me the command that you are running and I can see if I can help.
> > >
> > > Thanks,
> > > Julie
> > >
> > > On Tue, Dec 18, 2018 at 11:31 AM Case, Jonathan[ENSCO INC] via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Tue Dec 18 11:31:09 2018: Request 88228 was acted upon.
> > > > Transaction: Ticket created by jonathan.case-1 at nasa.gov
> > > > Queue: met_help
> > > > Subject: using netcdf files with unique field in
pcp_combine
> > > > Owner: Nobody
> > > > Requestors: jonathan.case-1 at nasa.gov
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228
> > > > >
> > > >
> > > >
> > > > Dear MET Help,
> > > >
> > > > I'm trying to do something fairly unique in MET by validating
> > > > derived total lightning flash rates from an NWP model against
> > > > observed gridded lightning flash rates.
> > > > I'm "faking" MET into thinking the model flash rates are
simply
> > > > accumulated precip (already named as APCP in GRIB2 files),
> > > > whereas
> > > > the observed lightning files are identified as "FlashRate" in
> > > > hourly netcdf files.
> > > >
> > > > For starters, I'm just trying to work with the netcdf files to
> > > > see
> > > > if I can combine them into "accumulated" fields using
> > > > pcp_combine,
> > > > but I'm having trouble getting it to work.
> > > > What I'd like to do is combine/accumulate the data into
various
> > > > time intervals [much like precipitation], and then output the
> > > > pcp_combine netcdf file with the traditional APCP name.
> > > >
> > > > Is there a way I can do this using pcp_combine? I'm still on
MET
> > > > v6.0. I have an ncdump of the lightning file below that might
be
> > > helpful.
> > > >
> > > > Thanks!
> > > > -Jon
> > > >
> > > >
> > > > Ncdump of sample lightning netcdf file:
> > > > :=> ncdump -h ./precip_tmp/entln_met_2018033001.nc
> > > > netcdf entln_met_2018033001 {
> > > > dimensions:
> > > > lon = 425 ;
> > > > lat = 350 ;
> > > > time = UNLIMITED ; // (1 currently)
> > > > variables:
> > > > float lat(lat) ;
> > > > lat:units = "degrees_north" ;
> > > > lat:long_name = "latitude" ;
> > > > float lon(lon) ;
> > > > lon:units = "degrees east" ;
> > > > lon:long_name = "longitude" ;
> > > > double Time(time) ;
> > > > Time:units = "seconds since " ;
> > > > Time:calendar = "standard" ;
> > > > Time:long_name = "time" ;
> > > > float FlashRate(time, lat, lon) ;
> > > > FlashRate:units = "Flashes per km2 per 5 min"
;
> > > > FlashRate:long_name = "" ;
> > > > FlashRate:coordinates = "lon lat time" ;
> > > >
> > > > // global attributes:
> > > > :Description = "" ;
> > > > :Conventions = "CF-1.7" ;
> > > > :title = "" ;
> > > > :institution = "" ;
> > > > :source = "{0}" ;
> > > > :references = "" ;
> > > > :history = "Created Mon Dec 17 22:23:33 2018"
;
> > > > :comment = "" ;
> > > > }
> > > >
> > > >
> > > >
> > > >
> > > > *****************************************************
> > > > Jonathan Case (Jon)
> > > > ENSCO, Inc./NASA SPoRT Center
> > > > National Space Science and Technology Center
> > > > 320 Sparkman Drive, Room 3000B
> > > > Huntsville, AL 35805
> > > > Jonathan.Case-1 at nasa.gov<mailto:Jonathan.Case-1 at nasa.gov>
(alt:
> > > > case.jonathan at ensco.com<mailto:case.jonathan at ensco.com>)
> > > > *****************************************************
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #88228] using netcdf files with unique field in pcp_combine
From: Case, Jonathan[ENSCO INC]
Time: Fri Dec 21 13:06:33 2018
Hi Howard,
You don't need the MRMSTEST file. So please delete.
I copied in the original files and re-tarred. Try working with the
flashrate.tar files now.
Thx,
Jonathan
-----Original Message-----
From: Howard Soh via RT <met_help at ucar.edu>
Sent: Friday, December 21, 2018 12:32 PM
To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov>
Subject: [rt.rap.ucar.edu #88228] using netcdf files with unique field
in pcp_combine
Thank you on shared your data.
I downloaded both files. flashrate.tar contains the softlinks only.
Would you post the real NetCDF files?
precip_tmp$ ls -l
total 0
lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:19 entln_met_2018033001.nc ->
../entlnFlashRate_20180330_01Z.nc lrwxrwxrwx 1 hsoh rap 33 Dec 18
11:19 entln_met_2018033002.nc -> ../entlnFlashRate_20180330_02Z.nc
lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:19 entln_met_2018033003.nc ->
../entlnFlashRate_20180330_03Z.nc lrwxrwxrwx 1 hsoh rap 33 Dec 18
11:19 entln_met_2018033004.nc -> ../entlnFlashRate_20180330_04Z.nc
lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:20 entln_met_2018033005.nc ->
../entlnFlashRate_20180330_05Z.nc lrwxrwxrwx 1 hsoh rap 33 Dec 18
11:20 entln_met_2018033006.nc -> ../entlnFlashRate_20180330_06Z.nc
Cheers,
Howard
On Fri Dec 21 10:39:36 2018, jonathan.case-1 at nasa.gov wrote:
> Hello again Howard,
>
> I tried running plot_data_place, but I immediately get a cryptic
> error:
> sc1login1:/raid1/sport/people/casejl/MET/gridData/precip_tmp:=>
> plot_data_plane entln_met_2018033001.nc test FlashRate -v 5 ERROR :
> ERROR : yyerror() -> syntax error in file
"/tmp/config_20992_0_.temp"
> ERROR :
> ERROR : line = 2
> ERROR :
> ERROR : column = 1
> ERROR :
> ERROR : text = "(nul)"
> ERROR :
> ERROR :
> ERROR : (nul)
> ERROR : (nul)
> ERROR :
>
> I uploaded the sample netcdf files to our public-facing server at:
> https://geo.nsstc.nasa.gov/SPoRT/outgoing/jlc/MET/
> (file is flashrate.tar, which will unpack in to the precip_tmp sub-
> folder). Please keep the data among yourself because as you're
likely
> aware, ENTLN lightning data are not freely open to the public.
>
> All I really need to know is how we need to set up these netcdf
files
> correctly so that pcp_combine can operate on them. Once that hurdle
> is cleared, then I'm confident that I can output the pcp_combine
> intermediate netcdf output files for use within grid_stat.
>
> Thanks for your continued help.
> -Jonathan
>
> -----Original Message-----
> From: Howard Soh via RT <met_help at ucar.edu>
> Sent: Thursday, December 20, 2018 2:42 PM
> To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov>
> Subject: [rt.rap.ucar.edu #88228] using netcdf files with unique
field
> in pcp_combine
>
> The error message is from the NetCDF library. The line:80 means it
> failed to get the dimension size because the dimension ID is not
> valid. So it could be a bad NetCDF file at ./precip_tmp.
>
> Would you post the data and configuration files as Julie suggested
> before?
>
> What kind of trouble are you having? If you'd like you could post
the
> data to our ftp server ( following the instructions listed here:
> http://www.dtcenter.org/met/users/support/met_help.php#ftp)
> and send me the command that you are running and I can see if I can
> help.
>
> If you won't post the data, would you do these?
>
> Option 1:
>
> - Run plot_data_plane for FlashRate variable with each NetCDF files
at
> ./precip_tmp
> - Check if all of them have the same problem
>
> Option 2:
>
> - check if the NetCDF files at ./precip_tmp have the same header.
> - keep "entln_met_2018033001" at ./precip_tmp
> - move other NetCDF files to somewhere else
> - run pcp_combined command
> - check if the same problem happens.
> - If this works, add NetCDF file one by one.
>
> Cheers,
> Howard
>
> On Thu Dec 20 12:21:46 2018, jonathan.case-1 at nasa.gov wrote:
> > Hello again,
> >
> > I ran pcp_combine with -v5 log level. It appears that the time is
> > just a warning, whereas the program crashes in response to the
> > NcBadId, which I'm not familiar with.
> > Any insight on that? (see below)
> >
> > Sincerely,
> > Jonathan
> >
> > pcp_combine -sum 00000000_000000 01 20180330_060000 06
> > ./observed_APCP_06 -pcpdir ./precip_tmp -field 'name="FlashRate";
> > level="R1";' -v 5 -compress 0 DEBUG 2: Performing sum command:
> > Init/In_Accum/Valid/Out_Accum Times =
> > 00000000_000000/010000/20180330_060000/060000
> > DEBUG 2: Searching for 6 files with accumulation times of 010000
to
> > sum to a total accumulation time of 060000.
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
> > new Met2dDataFile object of type "FileType_NcCF".
> > DEBUG 4: NcCfFile::open() -> could not extract valid time from
the
> > "time" variable.
> > DEBUG 4: NcCfFile::open() -> could not extract valid time from
file
> > name.
> > WARNING:
> > WARNING: NcCfFile::open() -> could not determine the valid time,
> > using 0.
> > WARNING:
> > DEBUG 4: NcCfFile::open() -> could not extract init time from the
> > "forecast_reference_time" variable.
> > DEBUG 4: NcCfFile::open() -> could not extract init time from
file
> > name.
> > terminate called after throwing an instance of
> > 'netCDF::exceptions::NcBadId'
> > what(): NcBadId: Not a netcdf id
> > file: ncDim.cpp line:80
> > Aborted
> >
> >
> >
> > -----Original Message-----
> > From: Howard Soh via RT <met_help at ucar.edu>
> > Sent: Tuesday, December 18, 2018 4:37 PM
> > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
1 at nasa.gov>
> > Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files with
> > unique field in pcp_combine
> >
> > The warning is caused by the variable name "Time" (vs. "time").
> > MET is looking for "time" variable as CF NetCDF..
> >
> > double Time(time) ;
> > Time:units = "seconds since " ;
> > Time:calendar = "standard" ;
> > Time:long_name = "time" ;
> >
> > And time:unit attribute does not have the reference time.
> > time:units = "seconds since YYYY-MM-DD" ;
> > OR
> > time:units = "seconds since YYYY-MM-DD HH:MM:SS" ;
> >
> > If the verbose level is increased ("-v 4" or "-v 5"), it will
> > display more useful information.
> >
> > Cheers,
> > Howard
> >
> > On Tue, Dec 18, 2018 at 1:38 PM Case, Jonathan[ENSCO INC] via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228 >
> > >
> > > Sorry Julie,
> > >
> > > I should have also included the pcp_combine error I'm getting:
> > >
> > > sc1login1:/raid1/sport/people/casejl/MET/gridData:=> pcp_combine
> > > -sum
> > > 00000000_000000 01 20180330_060000 06 ./observed_APCP_06
-pcpdir
> > > ./precip_tmp -field 'name="FlashRate"; level="R1";' -v 3
-compress
> > > 0
> > > DEBUG 2: Performing sum command: Init/In_Accum/Valid/Out_Accum
> > > Times =
> > > 00000000_000000/010000/20180330_060000/060000
> > > DEBUG 2: Searching for 6 files with accumulation times of
010000
> > > to sum to a total accumulation time of 060000.
> > > WARNING:
> > > WARNING: NcCfFile::open() -> could not determine the valid time,
> > > using 0.
> > > WARNING:
> > > terminate called after throwing an instance of
> > > 'netCDF::exceptions::NcBadId'
> > > what(): NcBadId: Not a netcdf id
> > > file: ncDim.cpp line:80
> > > Aborted
> > >
> > >
> > > I need to know what the "NcBadId" means so I can persue a
possible
> > > solution.
> > >
> > > Thx!
> > > -Jon
> > >
> > >
> > > -----Original Message-----
> > > From: Julie Prestopnik via RT <met_help at ucar.edu>
> > > Sent: Tuesday, December 18, 2018 1:26 PM
> > > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
> > > 1 at nasa.gov>
> > > Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files with
> > > unique field in pcp_combine
> > >
> > > Hi Jon. Unfortunately, our MET expert is out of the office for
a
> > > while due to the holidays. I do not know the answers to your
> > > questions, however I can try to help.
> > >
> > > For starters, I'm just trying to work with the netcdf files to
see
> > > if I can
> > > > combine them into "accumulated" fields using pcp_combine, but
I'm
> > > > having trouble getting it to work.
> > > >
> > >
> > > What kind of trouble are you having? If you'd like you could
post
> > > the data to our ftp server ( following the instructions listed
> > > here:
> > > http://www.dtcenter.org/met/users/support/met_help.php#ftp) and
> > > send
> > > me the command that you are running and I can see if I can help.
> > >
> > > Thanks,
> > > Julie
> > >
> > > On Tue, Dec 18, 2018 at 11:31 AM Case, Jonathan[ENSCO INC] via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Tue Dec 18 11:31:09 2018: Request 88228 was acted upon.
> > > > Transaction: Ticket created by jonathan.case-1 at nasa.gov
> > > > Queue: met_help
> > > > Subject: using netcdf files with unique field in
pcp_combine
> > > > Owner: Nobody
> > > > Requestors: jonathan.case-1 at nasa.gov
> > > > Status: new
> > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228
> > > > >
> > > >
> > > >
> > > > Dear MET Help,
> > > >
> > > > I'm trying to do something fairly unique in MET by validating
> > > > derived total lightning flash rates from an NWP model against
> > > > observed gridded lightning flash rates.
> > > > I'm "faking" MET into thinking the model flash rates are
simply
> > > > accumulated precip (already named as APCP in GRIB2 files),
> > > > whereas
> > > > the observed lightning files are identified as "FlashRate" in
> > > > hourly netcdf files.
> > > >
> > > > For starters, I'm just trying to work with the netcdf files to
> > > > see
> > > > if I can combine them into "accumulated" fields using
> > > > pcp_combine,
> > > > but I'm having trouble getting it to work.
> > > > What I'd like to do is combine/accumulate the data into
various
> > > > time intervals [much like precipitation], and then output the
> > > > pcp_combine netcdf file with the traditional APCP name.
> > > >
> > > > Is there a way I can do this using pcp_combine? I'm still on
MET
> > > > v6.0. I have an ncdump of the lightning file below that might
be
> > > helpful.
> > > >
> > > > Thanks!
> > > > -Jon
> > > >
> > > >
> > > > Ncdump of sample lightning netcdf file:
> > > > :=> ncdump -h ./precip_tmp/entln_met_2018033001.nc
> > > > netcdf entln_met_2018033001 {
> > > > dimensions:
> > > > lon = 425 ;
> > > > lat = 350 ;
> > > > time = UNLIMITED ; // (1 currently)
> > > > variables:
> > > > float lat(lat) ;
> > > > lat:units = "degrees_north" ;
> > > > lat:long_name = "latitude" ;
> > > > float lon(lon) ;
> > > > lon:units = "degrees east" ;
> > > > lon:long_name = "longitude" ;
> > > > double Time(time) ;
> > > > Time:units = "seconds since " ;
> > > > Time:calendar = "standard" ;
> > > > Time:long_name = "time" ;
> > > > float FlashRate(time, lat, lon) ;
> > > > FlashRate:units = "Flashes per km2 per 5 min"
;
> > > > FlashRate:long_name = "" ;
> > > > FlashRate:coordinates = "lon lat time" ;
> > > >
> > > > // global attributes:
> > > > :Description = "" ;
> > > > :Conventions = "CF-1.7" ;
> > > > :title = "" ;
> > > > :institution = "" ;
> > > > :source = "{0}" ;
> > > > :references = "" ;
> > > > :history = "Created Mon Dec 17 22:23:33 2018"
;
> > > > :comment = "" ;
> > > > }
> > > >
> > > >
> > > >
> > > >
> > > > *****************************************************
> > > > Jonathan Case (Jon)
> > > > ENSCO, Inc./NASA SPoRT Center
> > > > National Space Science and Technology Center
> > > > 320 Sparkman Drive, Room 3000B
> > > > Huntsville, AL 35805
> > > > Jonathan.Case-1 at nasa.gov<mailto:Jonathan.Case-1 at nasa.gov>
(alt:
> > > > case.jonathan at ensco.com<mailto:case.jonathan at ensco.com>)
> > > > *****************************************************
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
------------------------------------------------
Subject: using netcdf files with unique field in pcp_combine
From: Howard Soh
Time: Fri Dec 21 14:33:12 2018
I appreciate your posting and I deleted MRMSTEST files.
The problem is related with "Time" variable and the NetCDF files are
not CF compliant.
The following commands will resolve Bad ID issue if NCO is installed.
ncatted -O -a coordinates,FlashRate,m,c,"Time lon lat"
entlnFlashRate_20180330_01Z.nc
ncatted -O -a units,lon,m,c,"degrees_east"
entlnFlashRate_20180330_01Z.nc
ncatted -O -a units,lat,m,c,"degrees_north"
entlnFlashRate_20180330_01Z.nc
I also recommend to correct the Time unit (missing reference time).
Two issues with Bad ID:
Issue 1: time VS Time variable
FlashRate:coordinates = "lon lat time" ;
The coordinates attribute of FlashRate variable is "lon lat time".
This implies the coordinate variables are "lon", "lat", and "time".
But "time" variable does not exist. So a quick solution is changing
the attribute:
ncatted -O -a coordinates,FlashRate,m,c,"Time lon lat"
entlnFlashRate_20180330_01Z.nc
Issue 2: "degrees east" and "degrees north" are not CF compliant.
The recommended unit of latitude is degrees_north. Also acceptable are
degree_north, degree_N, degrees_N, degreeN, and degreesN.
A quick solution is changing them:
ncatted -O -a units,lon,m,c,"degrees_east"
entlnFlashRate_20180330_01Z.nc
ncatted -O -a units,lat,m,c,"degrees_north"
entlnFlashRate_20180330_01Z.nc
So please let us know if this does not solve the NetCDF problem.
Have a nice weekend!
Cheers,
Howard
On Fri Dec 21 13:06:33 2018, jonathan.case-1 at nasa.gov wrote:
> Hi Howard,
>
> You don't need the MRMSTEST file. So please delete.
> I copied in the original files and re-tarred. Try working with the
> flashrate.tar files now.
>
> Thx,
> Jonathan
>
> -----Original Message-----
> From: Howard Soh via RT <met_help at ucar.edu>
> Sent: Friday, December 21, 2018 12:32 PM
> To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov>
> Subject: [rt.rap.ucar.edu #88228] using netcdf files with unique
field
> in pcp_combine
>
> Thank you on shared your data.
> I downloaded both files. flashrate.tar contains the softlinks only.
> Would you post the real NetCDF files?
>
> precip_tmp$ ls -l
> total 0
> lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:19 entln_met_2018033001.nc ->
> ../entlnFlashRate_20180330_01Z.nc lrwxrwxrwx 1 hsoh rap 33 Dec 18
> 11:19 entln_met_2018033002.nc -> ../entlnFlashRate_20180330_02Z.nc
> lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:19 entln_met_2018033003.nc ->
> ../entlnFlashRate_20180330_03Z.nc lrwxrwxrwx 1 hsoh rap 33 Dec 18
> 11:19 entln_met_2018033004.nc -> ../entlnFlashRate_20180330_04Z.nc
> lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:20 entln_met_2018033005.nc ->
> ../entlnFlashRate_20180330_05Z.nc lrwxrwxrwx 1 hsoh rap 33 Dec 18
> 11:20 entln_met_2018033006.nc -> ../entlnFlashRate_20180330_06Z.nc
>
>
> Cheers,
> Howard
>
> On Fri Dec 21 10:39:36 2018, jonathan.case-1 at nasa.gov wrote:
> > Hello again Howard,
> >
> > I tried running plot_data_place, but I immediately get a cryptic
> > error:
> > sc1login1:/raid1/sport/people/casejl/MET/gridData/precip_tmp:=>
> > plot_data_plane entln_met_2018033001.nc test FlashRate -v 5 ERROR
:
> > ERROR : yyerror() -> syntax error in file
> > "/tmp/config_20992_0_.temp"
> > ERROR :
> > ERROR : line = 2
> > ERROR :
> > ERROR : column = 1
> > ERROR :
> > ERROR : text = "(nul)"
> > ERROR :
> > ERROR :
> > ERROR : (nul)
> > ERROR : (nul)
> > ERROR :
> >
> > I uploaded the sample netcdf files to our public-facing server at:
> > https://geo.nsstc.nasa.gov/SPoRT/outgoing/jlc/MET/
> > (file is flashrate.tar, which will unpack in to the precip_tmp
sub-
> > folder). Please keep the data among yourself because as you're
> > likely
> > aware, ENTLN lightning data are not freely open to the public.
> >
> > All I really need to know is how we need to set up these netcdf
files
> > correctly so that pcp_combine can operate on them. Once that
hurdle
> > is cleared, then I'm confident that I can output the pcp_combine
> > intermediate netcdf output files for use within grid_stat.
> >
> > Thanks for your continued help.
> > -Jonathan
> >
> > -----Original Message-----
> > From: Howard Soh via RT <met_help at ucar.edu>
> > Sent: Thursday, December 20, 2018 2:42 PM
> > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
1 at nasa.gov>
> > Subject: [rt.rap.ucar.edu #88228] using netcdf files with unique
> > field
> > in pcp_combine
> >
> > The error message is from the NetCDF library. The line:80 means it
> > failed to get the dimension size because the dimension ID is not
> > valid. So it could be a bad NetCDF file at ./precip_tmp.
> >
> > Would you post the data and configuration files as Julie suggested
> > before?
> >
> > What kind of trouble are you having? If you'd like you could post
> > the
> > data to our ftp server ( following the instructions listed here:
> > http://www.dtcenter.org/met/users/support/met_help.php#ftp)
> > and send me the command that you are running and I can see if I
can
> > help.
> >
> > If you won't post the data, would you do these?
> >
> > Option 1:
> >
> > - Run plot_data_plane for FlashRate variable with each NetCDF
files
> > at
> > ./precip_tmp
> > - Check if all of them have the same problem
> >
> > Option 2:
> >
> > - check if the NetCDF files at ./precip_tmp have the same header.
> > - keep "entln_met_2018033001" at ./precip_tmp
> > - move other NetCDF files to somewhere else
> > - run pcp_combined command
> > - check if the same problem happens.
> > - If this works, add NetCDF file one by one.
> >
> > Cheers,
> > Howard
> >
> > On Thu Dec 20 12:21:46 2018, jonathan.case-1 at nasa.gov wrote:
> > > Hello again,
> > >
> > > I ran pcp_combine with -v5 log level. It appears that the time
is
> > > just a warning, whereas the program crashes in response to the
> > > NcBadId, which I'm not familiar with.
> > > Any insight on that? (see below)
> > >
> > > Sincerely,
> > > Jonathan
> > >
> > > pcp_combine -sum 00000000_000000 01 20180330_060000 06
> > > ./observed_APCP_06 -pcpdir ./precip_tmp -field
> > > 'name="FlashRate";
> > > level="R1";' -v 5 -compress 0 DEBUG 2: Performing sum command:
> > > Init/In_Accum/Valid/Out_Accum Times =
> > > 00000000_000000/010000/20180330_060000/060000
> > > DEBUG 2: Searching for 6 files with accumulation times of
010000
> > > to
> > > sum to a total accumulation time of 060000.
> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> > > new Met2dDataFile object of type "FileType_NcCF".
> > > DEBUG 4: NcCfFile::open() -> could not extract valid time from
> > > the
> > > "time" variable.
> > > DEBUG 4: NcCfFile::open() -> could not extract valid time from
> > > file
> > > name.
> > > WARNING:
> > > WARNING: NcCfFile::open() -> could not determine the valid
time,
> > > using 0.
> > > WARNING:
> > > DEBUG 4: NcCfFile::open() -> could not extract init time from
the
> > > "forecast_reference_time" variable.
> > > DEBUG 4: NcCfFile::open() -> could not extract init time from
> > > file
> > > name.
> > > terminate called after throwing an instance of
> > > 'netCDF::exceptions::NcBadId'
> > > what(): NcBadId: Not a netcdf id
> > > file: ncDim.cpp line:80
> > > Aborted
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Howard Soh via RT <met_help at ucar.edu>
> > > Sent: Tuesday, December 18, 2018 4:37 PM
> > > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
> > > 1 at nasa.gov>
> > > Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files with
> > > unique field in pcp_combine
> > >
> > > The warning is caused by the variable name "Time" (vs. "time").
> > > MET is looking for "time" variable as CF NetCDF..
> > >
> > > double Time(time) ;
> > > Time:units = "seconds since " ;
> > > Time:calendar = "standard" ;
> > > Time:long_name = "time" ;
> > >
> > > And time:unit attribute does not have the reference time.
> > > time:units = "seconds since YYYY-MM-DD" ;
> > > OR
> > > time:units = "seconds since YYYY-MM-DD HH:MM:SS"
;
> > >
> > > If the verbose level is increased ("-v 4" or "-v 5"), it will
> > > display more useful information.
> > >
> > > Cheers,
> > > Howard
> > >
> > > On Tue, Dec 18, 2018 at 1:38 PM Case, Jonathan[ENSCO INC] via RT
<
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228
>
> > > >
> > > > Sorry Julie,
> > > >
> > > > I should have also included the pcp_combine error I'm getting:
> > > >
> > > > sc1login1:/raid1/sport/people/casejl/MET/gridData:=>
pcp_combine
> > > > -sum
> > > > 00000000_000000 01 20180330_060000 06 ./observed_APCP_06
> > > > -pcpdir
> > > > ./precip_tmp -field 'name="FlashRate"; level="R1";' -v 3
> > > > -compress
> > > > 0
> > > > DEBUG 2: Performing sum command: Init/In_Accum/Valid/Out_Accum
> > > > Times =
> > > > 00000000_000000/010000/20180330_060000/060000
> > > > DEBUG 2: Searching for 6 files with accumulation times of
010000
> > > > to sum to a total accumulation time of 060000.
> > > > WARNING:
> > > > WARNING: NcCfFile::open() -> could not determine the valid
time,
> > > > using 0.
> > > > WARNING:
> > > > terminate called after throwing an instance of
> > > > 'netCDF::exceptions::NcBadId'
> > > > what(): NcBadId: Not a netcdf id
> > > > file: ncDim.cpp line:80
> > > > Aborted
> > > >
> > > >
> > > > I need to know what the "NcBadId" means so I can persue a
> > > > possible
> > > > solution.
> > > >
> > > > Thx!
> > > > -Jon
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Julie Prestopnik via RT <met_help at ucar.edu>
> > > > Sent: Tuesday, December 18, 2018 1:26 PM
> > > > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
> > > > 1 at nasa.gov>
> > > > Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files
with
> > > > unique field in pcp_combine
> > > >
> > > > Hi Jon. Unfortunately, our MET expert is out of the office
for a
> > > > while due to the holidays. I do not know the answers to your
> > > > questions, however I can try to help.
> > > >
> > > > For starters, I'm just trying to work with the netcdf files to
> > > > see
> > > > if I can
> > > > > combine them into "accumulated" fields using pcp_combine,
but
> > > > > I'm
> > > > > having trouble getting it to work.
> > > > >
> > > >
> > > > What kind of trouble are you having? If you'd like you could
> > > > post
> > > > the data to our ftp server ( following the instructions listed
> > > > here:
> > > > http://www.dtcenter.org/met/users/support/met_help.php#ftp)
and
> > > > send
> > > > me the command that you are running and I can see if I can
help.
> > > >
> > > > Thanks,
> > > > Julie
> > > >
> > > > On Tue, Dec 18, 2018 at 11:31 AM Case, Jonathan[ENSCO INC] via
RT
> > > > <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Tue Dec 18 11:31:09 2018: Request 88228 was acted upon.
> > > > > Transaction: Ticket created by jonathan.case-1 at nasa.gov
> > > > > Queue: met_help
> > > > > Subject: using netcdf files with unique field in
> > > > > pcp_combine
> > > > > Owner: Nobody
> > > > > Requestors: jonathan.case-1 at nasa.gov
> > > > > Status: new
> > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228
> > > > > >
> > > > >
> > > > >
> > > > > Dear MET Help,
> > > > >
> > > > > I'm trying to do something fairly unique in MET by
validating
> > > > > derived total lightning flash rates from an NWP model
against
> > > > > observed gridded lightning flash rates.
> > > > > I'm "faking" MET into thinking the model flash rates are
simply
> > > > > accumulated precip (already named as APCP in GRIB2 files),
> > > > > whereas
> > > > > the observed lightning files are identified as "FlashRate"
in
> > > > > hourly netcdf files.
> > > > >
> > > > > For starters, I'm just trying to work with the netcdf files
to
> > > > > see
> > > > > if I can combine them into "accumulated" fields using
> > > > > pcp_combine,
> > > > > but I'm having trouble getting it to work.
> > > > > What I'd like to do is combine/accumulate the data into
various
> > > > > time intervals [much like precipitation], and then output
the
> > > > > pcp_combine netcdf file with the traditional APCP name.
> > > > >
> > > > > Is there a way I can do this using pcp_combine? I'm still
on
> > > > > MET
> > > > > v6.0. I have an ncdump of the lightning file below that
might
> > > > > be
> > > > helpful.
> > > > >
> > > > > Thanks!
> > > > > -Jon
> > > > >
> > > > >
> > > > > Ncdump of sample lightning netcdf file:
> > > > > :=> ncdump -h ./precip_tmp/entln_met_2018033001.nc
> > > > > netcdf entln_met_2018033001 {
> > > > > dimensions:
> > > > > lon = 425 ;
> > > > > lat = 350 ;
> > > > > time = UNLIMITED ; // (1 currently)
> > > > > variables:
> > > > > float lat(lat) ;
> > > > > lat:units = "degrees_north" ;
> > > > > lat:long_name = "latitude" ;
> > > > > float lon(lon) ;
> > > > > lon:units = "degrees east" ;
> > > > > lon:long_name = "longitude" ;
> > > > > double Time(time) ;
> > > > > Time:units = "seconds since " ;
> > > > > Time:calendar = "standard" ;
> > > > > Time:long_name = "time" ;
> > > > > float FlashRate(time, lat, lon) ;
> > > > > FlashRate:units = "Flashes per km2 per 5
min" ;
> > > > > FlashRate:long_name = "" ;
> > > > > FlashRate:coordinates = "lon lat time" ;
> > > > >
> > > > > // global attributes:
> > > > > :Description = "" ;
> > > > > :Conventions = "CF-1.7" ;
> > > > > :title = "" ;
> > > > > :institution = "" ;
> > > > > :source = "{0}" ;
> > > > > :references = "" ;
> > > > > :history = "Created Mon Dec 17 22:23:33
2018" ;
> > > > > :comment = "" ;
> > > > > }
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *****************************************************
> > > > > Jonathan Case (Jon)
> > > > > ENSCO, Inc./NASA SPoRT Center
> > > > > National Space Science and Technology Center
> > > > > 320 Sparkman Drive, Room 3000B
> > > > > Huntsville, AL 35805
> > > > > Jonathan.Case-1 at nasa.gov<mailto:Jonathan.Case-1 at nasa.gov>
(alt:
> > > > > case.jonathan at ensco.com<mailto:case.jonathan at ensco.com>)
> > > > > *****************************************************
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
------------------------------------------------
Subject: using netcdf files with unique field in pcp_combine
From: Howard Soh
Time: Fri Dec 21 14:46:18 2018
The following command will show if the NetCDF file is properly handled
by MET:
plot_data_plane entln_met_2018033001.nc test.ps 'name="FlashRate";
level="(0,*,*)";' -v 5
Cheers,
Howard
On Fri Dec 21 14:33:12 2018, hsoh wrote:
> I appreciate your posting and I deleted MRMSTEST files.
>
> The problem is related with "Time" variable and the NetCDF files are
> not CF compliant.
>
> The following commands will resolve Bad ID issue if NCO is
installed.
>
> ncatted -O -a coordinates,FlashRate,m,c,"Time lon lat"
> entlnFlashRate_20180330_01Z.nc
> ncatted -O -a units,lon,m,c,"degrees_east"
> entlnFlashRate_20180330_01Z.nc
> ncatted -O -a units,lat,m,c,"degrees_north"
> entlnFlashRate_20180330_01Z.nc
>
> I also recommend to correct the Time unit (missing reference time).
>
> Two issues with Bad ID:
>
> Issue 1: time VS Time variable
>
> FlashRate:coordinates = "lon lat time" ;
>
> The coordinates attribute of FlashRate variable is "lon lat time".
> This implies the coordinate variables are "lon", "lat", and "time".
> But "time" variable does not exist. So a quick solution is changing
> the attribute:
>
> ncatted -O -a coordinates,FlashRate,m,c,"Time lon lat"
> entlnFlashRate_20180330_01Z.nc
>
> Issue 2: "degrees east" and "degrees north" are not CF compliant.
> The recommended unit of latitude is degrees_north. Also acceptable
are
> degree_north, degree_N, degrees_N, degreeN, and degreesN.
>
> A quick solution is changing them:
>
> ncatted -O -a units,lon,m,c,"degrees_east"
> entlnFlashRate_20180330_01Z.nc
> ncatted -O -a units,lat,m,c,"degrees_north"
> entlnFlashRate_20180330_01Z.nc
>
> So please let us know if this does not solve the NetCDF problem.
>
> Have a nice weekend!
>
> Cheers,
> Howard
>
> On Fri Dec 21 13:06:33 2018, jonathan.case-1 at nasa.gov wrote:
> > Hi Howard,
> >
> > You don't need the MRMSTEST file. So please delete.
> > I copied in the original files and re-tarred. Try working with the
> > flashrate.tar files now.
> >
> > Thx,
> > Jonathan
> >
> > -----Original Message-----
> > From: Howard Soh via RT <met_help at ucar.edu>
> > Sent: Friday, December 21, 2018 12:32 PM
> > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
1 at nasa.gov>
> > Subject: [rt.rap.ucar.edu #88228] using netcdf files with unique
> > field
> > in pcp_combine
> >
> > Thank you on shared your data.
> > I downloaded both files. flashrate.tar contains the softlinks
only.
> > Would you post the real NetCDF files?
> >
> > precip_tmp$ ls -l
> > total 0
> > lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:19 entln_met_2018033001.nc ->
> > ../entlnFlashRate_20180330_01Z.nc lrwxrwxrwx 1 hsoh rap 33 Dec 18
> > 11:19 entln_met_2018033002.nc -> ../entlnFlashRate_20180330_02Z.nc
> > lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:19 entln_met_2018033003.nc ->
> > ../entlnFlashRate_20180330_03Z.nc lrwxrwxrwx 1 hsoh rap 33 Dec 18
> > 11:19 entln_met_2018033004.nc -> ../entlnFlashRate_20180330_04Z.nc
> > lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:20 entln_met_2018033005.nc ->
> > ../entlnFlashRate_20180330_05Z.nc lrwxrwxrwx 1 hsoh rap 33 Dec 18
> > 11:20 entln_met_2018033006.nc -> ../entlnFlashRate_20180330_06Z.nc
> >
> >
> > Cheers,
> > Howard
> >
> > On Fri Dec 21 10:39:36 2018, jonathan.case-1 at nasa.gov wrote:
> > > Hello again Howard,
> > >
> > > I tried running plot_data_place, but I immediately get a cryptic
> > > error:
> > > sc1login1:/raid1/sport/people/casejl/MET/gridData/precip_tmp:=>
> > > plot_data_plane entln_met_2018033001.nc test FlashRate -v 5
ERROR
> > > :
> > > ERROR : yyerror() -> syntax error in file
> > > "/tmp/config_20992_0_.temp"
> > > ERROR :
> > > ERROR : line = 2
> > > ERROR :
> > > ERROR : column = 1
> > > ERROR :
> > > ERROR : text = "(nul)"
> > > ERROR :
> > > ERROR :
> > > ERROR : (nul)
> > > ERROR : (nul)
> > > ERROR :
> > >
> > > I uploaded the sample netcdf files to our public-facing server
at:
> > > https://geo.nsstc.nasa.gov/SPoRT/outgoing/jlc/MET/
> > > (file is flashrate.tar, which will unpack in to the precip_tmp
> > > sub-
> > > folder). Please keep the data among yourself because as you're
> > > likely
> > > aware, ENTLN lightning data are not freely open to the public.
> > >
> > > All I really need to know is how we need to set up these netcdf
> > > files
> > > correctly so that pcp_combine can operate on them. Once that
> > > hurdle
> > > is cleared, then I'm confident that I can output the pcp_combine
> > > intermediate netcdf output files for use within grid_stat.
> > >
> > > Thanks for your continued help.
> > > -Jonathan
> > >
> > > -----Original Message-----
> > > From: Howard Soh via RT <met_help at ucar.edu>
> > > Sent: Thursday, December 20, 2018 2:42 PM
> > > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
> > > 1 at nasa.gov>
> > > Subject: [rt.rap.ucar.edu #88228] using netcdf files with
unique
> > > field
> > > in pcp_combine
> > >
> > > The error message is from the NetCDF library. The line:80 means
it
> > > failed to get the dimension size because the dimension ID is not
> > > valid. So it could be a bad NetCDF file at ./precip_tmp.
> > >
> > > Would you post the data and configuration files as Julie
suggested
> > > before?
> > >
> > > What kind of trouble are you having? If you'd like you could
post
> > > the
> > > data to our ftp server ( following the instructions listed
here:
> > > http://www.dtcenter.org/met/users/support/met_help.php#ftp)
> > > and send me the command that you are running and I can see if I
can
> > > help.
> > >
> > > If you won't post the data, would you do these?
> > >
> > > Option 1:
> > >
> > > - Run plot_data_plane for FlashRate variable with each NetCDF
files
> > > at
> > > ./precip_tmp
> > > - Check if all of them have the same problem
> > >
> > > Option 2:
> > >
> > > - check if the NetCDF files at ./precip_tmp have the same
header.
> > > - keep "entln_met_2018033001" at ./precip_tmp
> > > - move other NetCDF files to somewhere else
> > > - run pcp_combined command
> > > - check if the same problem happens.
> > > - If this works, add NetCDF file one by one.
> > >
> > > Cheers,
> > > Howard
> > >
> > > On Thu Dec 20 12:21:46 2018, jonathan.case-1 at nasa.gov wrote:
> > > > Hello again,
> > > >
> > > > I ran pcp_combine with -v5 log level. It appears that the
time
> > > > is
> > > > just a warning, whereas the program crashes in response to the
> > > > NcBadId, which I'm not familiar with.
> > > > Any insight on that? (see below)
> > > >
> > > > Sincerely,
> > > > Jonathan
> > > >
> > > > pcp_combine -sum 00000000_000000 01 20180330_060000 06
> > > > ./observed_APCP_06 -pcpdir ./precip_tmp -field
> > > > 'name="FlashRate";
> > > > level="R1";' -v 5 -compress 0 DEBUG 2: Performing sum command:
> > > > Init/In_Accum/Valid/Out_Accum Times =
> > > > 00000000_000000/010000/20180330_060000/060000
> > > > DEBUG 2: Searching for 6 files with accumulation times of
> > > > 010000
> > > > to
> > > > sum to a total accumulation time of 060000.
> > > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
> > > > created
> > > > new Met2dDataFile object of type "FileType_NcCF".
> > > > DEBUG 4: NcCfFile::open() -> could not extract valid time
from
> > > > the
> > > > "time" variable.
> > > > DEBUG 4: NcCfFile::open() -> could not extract valid time
from
> > > > file
> > > > name.
> > > > WARNING:
> > > > WARNING: NcCfFile::open() -> could not determine the valid
> > > > time,
> > > > using 0.
> > > > WARNING:
> > > > DEBUG 4: NcCfFile::open() -> could not extract init time
from
> > > > the
> > > > "forecast_reference_time" variable.
> > > > DEBUG 4: NcCfFile::open() -> could not extract init time
from
> > > > file
> > > > name.
> > > > terminate called after throwing an instance of
> > > > 'netCDF::exceptions::NcBadId'
> > > > what(): NcBadId: Not a netcdf id
> > > > file: ncDim.cpp line:80
> > > > Aborted
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Howard Soh via RT <met_help at ucar.edu>
> > > > Sent: Tuesday, December 18, 2018 4:37 PM
> > > > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
> > > > 1 at nasa.gov>
> > > > Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files
with
> > > > unique field in pcp_combine
> > > >
> > > > The warning is caused by the variable name "Time" (vs.
"time").
> > > > MET is looking for "time" variable as CF NetCDF..
> > > >
> > > > double Time(time) ;
> > > > Time:units = "seconds since " ;
> > > > Time:calendar = "standard" ;
> > > > Time:long_name = "time" ;
> > > >
> > > > And time:unit attribute does not have the reference time.
> > > > time:units = "seconds since YYYY-MM-DD" ;
> > > > OR
> > > > time:units = "seconds since YYYY-MM-DD
HH:MM:SS"
> > > > ;
> > > >
> > > > If the verbose level is increased ("-v 4" or "-v 5"), it will
> > > > display more useful information.
> > > >
> > > > Cheers,
> > > > Howard
> > > >
> > > > On Tue, Dec 18, 2018 at 1:38 PM Case, Jonathan[ENSCO INC] via
RT
> > > > <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228 >
> > > > >
> > > > > Sorry Julie,
> > > > >
> > > > > I should have also included the pcp_combine error I'm
getting:
> > > > >
> > > > > sc1login1:/raid1/sport/people/casejl/MET/gridData:=>
> > > > > pcp_combine
> > > > > -sum
> > > > > 00000000_000000 01 20180330_060000 06 ./observed_APCP_06
> > > > > -pcpdir
> > > > > ./precip_tmp -field 'name="FlashRate"; level="R1";' -v 3
> > > > > -compress
> > > > > 0
> > > > > DEBUG 2: Performing sum command:
Init/In_Accum/Valid/Out_Accum
> > > > > Times =
> > > > > 00000000_000000/010000/20180330_060000/060000
> > > > > DEBUG 2: Searching for 6 files with accumulation times of
> > > > > 010000
> > > > > to sum to a total accumulation time of 060000.
> > > > > WARNING:
> > > > > WARNING: NcCfFile::open() -> could not determine the valid
> > > > > time,
> > > > > using 0.
> > > > > WARNING:
> > > > > terminate called after throwing an instance of
> > > > > 'netCDF::exceptions::NcBadId'
> > > > > what(): NcBadId: Not a netcdf id
> > > > > file: ncDim.cpp line:80
> > > > > Aborted
> > > > >
> > > > >
> > > > > I need to know what the "NcBadId" means so I can persue a
> > > > > possible
> > > > > solution.
> > > > >
> > > > > Thx!
> > > > > -Jon
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Julie Prestopnik via RT <met_help at ucar.edu>
> > > > > Sent: Tuesday, December 18, 2018 1:26 PM
> > > > > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
> > > > > 1 at nasa.gov>
> > > > > Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files
with
> > > > > unique field in pcp_combine
> > > > >
> > > > > Hi Jon. Unfortunately, our MET expert is out of the office
for
> > > > > a
> > > > > while due to the holidays. I do not know the answers to
your
> > > > > questions, however I can try to help.
> > > > >
> > > > > For starters, I'm just trying to work with the netcdf files
to
> > > > > see
> > > > > if I can
> > > > > > combine them into "accumulated" fields using pcp_combine,
but
> > > > > > I'm
> > > > > > having trouble getting it to work.
> > > > > >
> > > > >
> > > > > What kind of trouble are you having? If you'd like you
could
> > > > > post
> > > > > the data to our ftp server ( following the instructions
listed
> > > > > here:
> > > > > http://www.dtcenter.org/met/users/support/met_help.php#ftp)
and
> > > > > send
> > > > > me the command that you are running and I can see if I can
> > > > > help.
> > > > >
> > > > > Thanks,
> > > > > Julie
> > > > >
> > > > > On Tue, Dec 18, 2018 at 11:31 AM Case, Jonathan[ENSCO INC]
via
> > > > > RT
> > > > > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Tue Dec 18 11:31:09 2018: Request 88228 was acted upon.
> > > > > > Transaction: Ticket created by jonathan.case-1 at nasa.gov
> > > > > > Queue: met_help
> > > > > > Subject: using netcdf files with unique field in
> > > > > > pcp_combine
> > > > > > Owner: Nobody
> > > > > > Requestors: jonathan.case-1 at nasa.gov
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228
> > > > > > >
> > > > > >
> > > > > >
> > > > > > Dear MET Help,
> > > > > >
> > > > > > I'm trying to do something fairly unique in MET by
validating
> > > > > > derived total lightning flash rates from an NWP model
against
> > > > > > observed gridded lightning flash rates.
> > > > > > I'm "faking" MET into thinking the model flash rates are
> > > > > > simply
> > > > > > accumulated precip (already named as APCP in GRIB2 files),
> > > > > > whereas
> > > > > > the observed lightning files are identified as "FlashRate"
in
> > > > > > hourly netcdf files.
> > > > > >
> > > > > > For starters, I'm just trying to work with the netcdf
files
> > > > > > to
> > > > > > see
> > > > > > if I can combine them into "accumulated" fields using
> > > > > > pcp_combine,
> > > > > > but I'm having trouble getting it to work.
> > > > > > What I'd like to do is combine/accumulate the data into
> > > > > > various
> > > > > > time intervals [much like precipitation], and then output
the
> > > > > > pcp_combine netcdf file with the traditional APCP name.
> > > > > >
> > > > > > Is there a way I can do this using pcp_combine? I'm still
on
> > > > > > MET
> > > > > > v6.0. I have an ncdump of the lightning file below that
> > > > > > might
> > > > > > be
> > > > > helpful.
> > > > > >
> > > > > > Thanks!
> > > > > > -Jon
> > > > > >
> > > > > >
> > > > > > Ncdump of sample lightning netcdf file:
> > > > > > :=> ncdump -h ./precip_tmp/entln_met_2018033001.nc
> > > > > > netcdf entln_met_2018033001 {
> > > > > > dimensions:
> > > > > > lon = 425 ;
> > > > > > lat = 350 ;
> > > > > > time = UNLIMITED ; // (1 currently)
> > > > > > variables:
> > > > > > float lat(lat) ;
> > > > > > lat:units = "degrees_north" ;
> > > > > > lat:long_name = "latitude" ;
> > > > > > float lon(lon) ;
> > > > > > lon:units = "degrees east" ;
> > > > > > lon:long_name = "longitude" ;
> > > > > > double Time(time) ;
> > > > > > Time:units = "seconds since " ;
> > > > > > Time:calendar = "standard" ;
> > > > > > Time:long_name = "time" ;
> > > > > > float FlashRate(time, lat, lon) ;
> > > > > > FlashRate:units = "Flashes per km2 per 5
min"
> > > > > > ;
> > > > > > FlashRate:long_name = "" ;
> > > > > > FlashRate:coordinates = "lon lat time" ;
> > > > > >
> > > > > > // global attributes:
> > > > > > :Description = "" ;
> > > > > > :Conventions = "CF-1.7" ;
> > > > > > :title = "" ;
> > > > > > :institution = "" ;
> > > > > > :source = "{0}" ;
> > > > > > :references = "" ;
> > > > > > :history = "Created Mon Dec 17 22:23:33
2018"
> > > > > > ;
> > > > > > :comment = "" ;
> > > > > > }
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > *****************************************************
> > > > > > Jonathan Case (Jon)
> > > > > > ENSCO, Inc./NASA SPoRT Center
> > > > > > National Space Science and Technology Center
> > > > > > 320 Sparkman Drive, Room 3000B
> > > > > > Huntsville, AL 35805
> > > > > > Jonathan.Case-1 at nasa.gov<mailto:Jonathan.Case-1 at nasa.gov>
> > > > > > (alt:
> > > > > > case.jonathan at ensco.com<mailto:case.jonathan at ensco.com>)
> > > > > > *****************************************************
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #88228] using netcdf files with unique field in pcp_combine
From: Case, Jonathan[ENSCO INC]
Time: Mon Dec 31 08:54:54 2018
Thanks for the information Howard. We will on getting the lightning
flash netcdf files CF-compliant and addressing the issues you raised
below.
Sincerely,
Jonathan
-----Original Message-----
From: Howard Soh via RT <met_help at ucar.edu>
Sent: Friday, December 21, 2018 3:33 PM
To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov>
Subject: [rt.rap.ucar.edu #88228] using netcdf files with unique field
in pcp_combine
I appreciate your posting and I deleted MRMSTEST files.
The problem is related with "Time" variable and the NetCDF files are
not CF compliant.
The following commands will resolve Bad ID issue if NCO is installed.
ncatted -O -a coordinates,FlashRate,m,c,"Time lon lat"
entlnFlashRate_20180330_01Z.nc ncatted -O -a
units,lon,m,c,"degrees_east" entlnFlashRate_20180330_01Z.nc ncatted -O
-a units,lat,m,c,"degrees_north" entlnFlashRate_20180330_01Z.nc
I also recommend to correct the Time unit (missing reference time).
Two issues with Bad ID:
Issue 1: time VS Time variable
FlashRate:coordinates = "lon lat time" ;
The coordinates attribute of FlashRate variable is "lon lat time".
This implies the coordinate variables are "lon", "lat", and "time".
But "time" variable does not exist. So a quick solution is changing
the attribute:
ncatted -O -a coordinates,FlashRate,m,c,"Time lon lat"
entlnFlashRate_20180330_01Z.nc
Issue 2: "degrees east" and "degrees north" are not CF compliant.
The recommended unit of latitude is degrees_north. Also acceptable are
degree_north, degree_N, degrees_N, degreeN, and degreesN.
A quick solution is changing them:
ncatted -O -a units,lon,m,c,"degrees_east"
entlnFlashRate_20180330_01Z.nc ncatted -O -a
units,lat,m,c,"degrees_north" entlnFlashRate_20180330_01Z.nc
So please let us know if this does not solve the NetCDF problem.
Have a nice weekend!
Cheers,
Howard
On Fri Dec 21 13:06:33 2018, jonathan.case-1 at nasa.gov wrote:
> Hi Howard,
>
> You don't need the MRMSTEST file. So please delete.
> I copied in the original files and re-tarred. Try working with the
> flashrate.tar files now.
>
> Thx,
> Jonathan
>
> -----Original Message-----
> From: Howard Soh via RT <met_help at ucar.edu>
> Sent: Friday, December 21, 2018 12:32 PM
> To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov>
> Subject: [rt.rap.ucar.edu #88228] using netcdf files with unique
field
> in pcp_combine
>
> Thank you on shared your data.
> I downloaded both files. flashrate.tar contains the softlinks only.
> Would you post the real NetCDF files?
>
> precip_tmp$ ls -l
> total 0
> lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:19 entln_met_2018033001.nc ->
> ../entlnFlashRate_20180330_01Z.nc lrwxrwxrwx 1 hsoh rap 33 Dec 18
> 11:19 entln_met_2018033002.nc -> ../entlnFlashRate_20180330_02Z.nc
> lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:19 entln_met_2018033003.nc ->
> ../entlnFlashRate_20180330_03Z.nc lrwxrwxrwx 1 hsoh rap 33 Dec 18
> 11:19 entln_met_2018033004.nc -> ../entlnFlashRate_20180330_04Z.nc
> lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:20 entln_met_2018033005.nc ->
> ../entlnFlashRate_20180330_05Z.nc lrwxrwxrwx 1 hsoh rap 33 Dec 18
> 11:20 entln_met_2018033006.nc -> ../entlnFlashRate_20180330_06Z.nc
>
>
> Cheers,
> Howard
>
> On Fri Dec 21 10:39:36 2018, jonathan.case-1 at nasa.gov wrote:
> > Hello again Howard,
> >
> > I tried running plot_data_place, but I immediately get a cryptic
> > error:
> > sc1login1:/raid1/sport/people/casejl/MET/gridData/precip_tmp:=>
> > plot_data_plane entln_met_2018033001.nc test FlashRate -v 5 ERROR
:
> > ERROR : yyerror() -> syntax error in file
> > "/tmp/config_20992_0_.temp"
> > ERROR :
> > ERROR : line = 2
> > ERROR :
> > ERROR : column = 1
> > ERROR :
> > ERROR : text = "(nul)"
> > ERROR :
> > ERROR :
> > ERROR : (nul)
> > ERROR : (nul)
> > ERROR :
> >
> > I uploaded the sample netcdf files to our public-facing server at:
> > https://geo.nsstc.nasa.gov/SPoRT/outgoing/jlc/MET/
> > (file is flashrate.tar, which will unpack in to the precip_tmp
sub-
> > folder). Please keep the data among yourself because as you're
> > likely aware, ENTLN lightning data are not freely open to the
> > public.
> >
> > All I really need to know is how we need to set up these netcdf
> > files correctly so that pcp_combine can operate on them. Once
that
> > hurdle is cleared, then I'm confident that I can output the
> > pcp_combine intermediate netcdf output files for use within
grid_stat.
> >
> > Thanks for your continued help.
> > -Jonathan
> >
> > -----Original Message-----
> > From: Howard Soh via RT <met_help at ucar.edu>
> > Sent: Thursday, December 20, 2018 2:42 PM
> > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
1 at nasa.gov>
> > Subject: [rt.rap.ucar.edu #88228] using netcdf files with unique
> > field in pcp_combine
> >
> > The error message is from the NetCDF library. The line:80 means it
> > failed to get the dimension size because the dimension ID is not
> > valid. So it could be a bad NetCDF file at ./precip_tmp.
> >
> > Would you post the data and configuration files as Julie suggested
> > before?
> >
> > What kind of trouble are you having? If you'd like you could post
> > the data to our ftp server ( following the instructions listed
> > here:
> > http://www.dtcenter.org/met/users/support/met_help.php#ftp)
> > and send me the command that you are running and I can see if I
can
> > help.
> >
> > If you won't post the data, would you do these?
> >
> > Option 1:
> >
> > - Run plot_data_plane for FlashRate variable with each NetCDF
files
> > at ./precip_tmp
> > - Check if all of them have the same problem
> >
> > Option 2:
> >
> > - check if the NetCDF files at ./precip_tmp have the same header.
> > - keep "entln_met_2018033001" at ./precip_tmp
> > - move other NetCDF files to somewhere else
> > - run pcp_combined command
> > - check if the same problem happens.
> > - If this works, add NetCDF file one by one.
> >
> > Cheers,
> > Howard
> >
> > On Thu Dec 20 12:21:46 2018, jonathan.case-1 at nasa.gov wrote:
> > > Hello again,
> > >
> > > I ran pcp_combine with -v5 log level. It appears that the time
is
> > > just a warning, whereas the program crashes in response to the
> > > NcBadId, which I'm not familiar with.
> > > Any insight on that? (see below)
> > >
> > > Sincerely,
> > > Jonathan
> > >
> > > pcp_combine -sum 00000000_000000 01 20180330_060000 06
> > > ./observed_APCP_06 -pcpdir ./precip_tmp -field
> > > 'name="FlashRate"; level="R1";' -v 5 -compress 0 DEBUG 2:
> > > Performing sum command:
> > > Init/In_Accum/Valid/Out_Accum Times =
> > > 00000000_000000/010000/20180330_060000/060000
> > > DEBUG 2: Searching for 6 files with accumulation times of
010000
> > > to sum to a total accumulation time of 060000.
> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> > > new Met2dDataFile object of type "FileType_NcCF".
> > > DEBUG 4: NcCfFile::open() -> could not extract valid time from
> > > the "time" variable.
> > > DEBUG 4: NcCfFile::open() -> could not extract valid time from
> > > file name.
> > > WARNING:
> > > WARNING: NcCfFile::open() -> could not determine the valid
time,
> > > using 0.
> > > WARNING:
> > > DEBUG 4: NcCfFile::open() -> could not extract init time from
> > > the "forecast_reference_time" variable.
> > > DEBUG 4: NcCfFile::open() -> could not extract init time from
> > > file name.
> > > terminate called after throwing an instance of
> > > 'netCDF::exceptions::NcBadId'
> > > what(): NcBadId: Not a netcdf id
> > > file: ncDim.cpp line:80
> > > Aborted
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Howard Soh via RT <met_help at ucar.edu>
> > > Sent: Tuesday, December 18, 2018 4:37 PM
> > > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
> > > 1 at nasa.gov>
> > > Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files with
> > > unique field in pcp_combine
> > >
> > > The warning is caused by the variable name "Time" (vs. "time").
> > > MET is looking for "time" variable as CF NetCDF..
> > >
> > > double Time(time) ;
> > > Time:units = "seconds since " ;
> > > Time:calendar = "standard" ;
> > > Time:long_name = "time" ;
> > >
> > > And time:unit attribute does not have the reference time.
> > > time:units = "seconds since YYYY-MM-DD" ;
> > > OR
> > > time:units = "seconds since YYYY-MM-DD HH:MM:SS"
;
> > >
> > > If the verbose level is increased ("-v 4" or "-v 5"), it will
> > > display more useful information.
> > >
> > > Cheers,
> > > Howard
> > >
> > > On Tue, Dec 18, 2018 at 1:38 PM Case, Jonathan[ENSCO INC] via RT
<
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228
>
> > > >
> > > > Sorry Julie,
> > > >
> > > > I should have also included the pcp_combine error I'm getting:
> > > >
> > > > sc1login1:/raid1/sport/people/casejl/MET/gridData:=>
pcp_combine
> > > > -sum
> > > > 00000000_000000 01 20180330_060000 06 ./observed_APCP_06
> > > > -pcpdir ./precip_tmp -field 'name="FlashRate"; level="R1";' -v
3
> > > > -compress
> > > > 0
> > > > DEBUG 2: Performing sum command: Init/In_Accum/Valid/Out_Accum
> > > > Times =
> > > > 00000000_000000/010000/20180330_060000/060000
> > > > DEBUG 2: Searching for 6 files with accumulation times of
> > > > 010000 to sum to a total accumulation time of 060000.
> > > > WARNING:
> > > > WARNING: NcCfFile::open() -> could not determine the valid
time,
> > > > using 0.
> > > > WARNING:
> > > > terminate called after throwing an instance of
> > > > 'netCDF::exceptions::NcBadId'
> > > > what(): NcBadId: Not a netcdf id
> > > > file: ncDim.cpp line:80
> > > > Aborted
> > > >
> > > >
> > > > I need to know what the "NcBadId" means so I can persue a
> > > > possible solution.
> > > >
> > > > Thx!
> > > > -Jon
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Julie Prestopnik via RT <met_help at ucar.edu>
> > > > Sent: Tuesday, December 18, 2018 1:26 PM
> > > > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
> > > > 1 at nasa.gov>
> > > > Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files
with
> > > > unique field in pcp_combine
> > > >
> > > > Hi Jon. Unfortunately, our MET expert is out of the office
for
> > > > a while due to the holidays. I do not know the answers to
your
> > > > questions, however I can try to help.
> > > >
> > > > For starters, I'm just trying to work with the netcdf files to
> > > > see if I can
> > > > > combine them into "accumulated" fields using pcp_combine,
but
> > > > > I'm having trouble getting it to work.
> > > > >
> > > >
> > > > What kind of trouble are you having? If you'd like you could
> > > > post the data to our ftp server ( following the instructions
> > > > listed
> > > > here:
> > > > http://www.dtcenter.org/met/users/support/met_help.php#ftp)
and
> > > > send me the command that you are running and I can see if I
can
> > > > help.
> > > >
> > > > Thanks,
> > > > Julie
> > > >
> > > > On Tue, Dec 18, 2018 at 11:31 AM Case, Jonathan[ENSCO INC] via
> > > > RT < met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Tue Dec 18 11:31:09 2018: Request 88228 was acted upon.
> > > > > Transaction: Ticket created by jonathan.case-1 at nasa.gov
> > > > > Queue: met_help
> > > > > Subject: using netcdf files with unique field in
> > > > > pcp_combine
> > > > > Owner: Nobody
> > > > > Requestors: jonathan.case-1 at nasa.gov
> > > > > Status: new
> > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228
> > > > > >
> > > > >
> > > > >
> > > > > Dear MET Help,
> > > > >
> > > > > I'm trying to do something fairly unique in MET by
validating
> > > > > derived total lightning flash rates from an NWP model
against
> > > > > observed gridded lightning flash rates.
> > > > > I'm "faking" MET into thinking the model flash rates are
> > > > > simply accumulated precip (already named as APCP in GRIB2
> > > > > files), whereas the observed lightning files are identified
as
> > > > > "FlashRate" in hourly netcdf files.
> > > > >
> > > > > For starters, I'm just trying to work with the netcdf files
to
> > > > > see if I can combine them into "accumulated" fields using
> > > > > pcp_combine, but I'm having trouble getting it to work.
> > > > > What I'd like to do is combine/accumulate the data into
> > > > > various time intervals [much like precipitation], and then
> > > > > output the pcp_combine netcdf file with the traditional APCP
name.
> > > > >
> > > > > Is there a way I can do this using pcp_combine? I'm still
on
> > > > > MET v6.0. I have an ncdump of the lightning file below that
> > > > > might be
> > > > helpful.
> > > > >
> > > > > Thanks!
> > > > > -Jon
> > > > >
> > > > >
> > > > > Ncdump of sample lightning netcdf file:
> > > > > :=> ncdump -h ./precip_tmp/entln_met_2018033001.nc
> > > > > netcdf entln_met_2018033001 {
> > > > > dimensions:
> > > > > lon = 425 ;
> > > > > lat = 350 ;
> > > > > time = UNLIMITED ; // (1 currently)
> > > > > variables:
> > > > > float lat(lat) ;
> > > > > lat:units = "degrees_north" ;
> > > > > lat:long_name = "latitude" ;
> > > > > float lon(lon) ;
> > > > > lon:units = "degrees east" ;
> > > > > lon:long_name = "longitude" ;
> > > > > double Time(time) ;
> > > > > Time:units = "seconds since " ;
> > > > > Time:calendar = "standard" ;
> > > > > Time:long_name = "time" ;
> > > > > float FlashRate(time, lat, lon) ;
> > > > > FlashRate:units = "Flashes per km2 per 5
min" ;
> > > > > FlashRate:long_name = "" ;
> > > > > FlashRate:coordinates = "lon lat time" ;
> > > > >
> > > > > // global attributes:
> > > > > :Description = "" ;
> > > > > :Conventions = "CF-1.7" ;
> > > > > :title = "" ;
> > > > > :institution = "" ;
> > > > > :source = "{0}" ;
> > > > > :references = "" ;
> > > > > :history = "Created Mon Dec 17 22:23:33
2018" ;
> > > > > :comment = "" ; }
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *****************************************************
> > > > > Jonathan Case (Jon)
> > > > > ENSCO, Inc./NASA SPoRT Center
> > > > > National Space Science and Technology Center
> > > > > 320 Sparkman Drive, Room 3000B
> > > > > Huntsville, AL 35805
> > > > > Jonathan.Case-1 at nasa.gov<mailto:Jonathan.Case-1 at nasa.gov>
(alt:
> > > > > case.jonathan at ensco.com<mailto:case.jonathan at ensco.com>)
> > > > > *****************************************************
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
------------------------------------------------
Subject: Recall: [rt.rap.ucar.edu #88228] using netcdf files with unique field in pcp_combine
From: Case, Jonathan[ENSCO INC]
Time: Mon Dec 31 08:55:08 2018
Case, Jonathan (MSFC-ST11)[ENSCO INC] would like to recall the
message, "[rt.rap.ucar.edu #88228] using netcdf files with unique
field in pcp_combine".
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #88228] using netcdf files with unique field in pcp_combine
From: Case, Jonathan[ENSCO INC]
Time: Mon Dec 31 08:55:21 2018
Thanks for the information Howard. We will work on getting the
lightning flash netcdf files CF-compliant and addressing the issues
you raised below.
Sincerely,
Jonathan
-----Original Message-----
From: Howard Soh via RT <met_help at ucar.edu>
Sent: Friday, December 21, 2018 3:33 PM
To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov>
Subject: [rt.rap.ucar.edu #88228] using netcdf files with unique field
in pcp_combine
I appreciate your posting and I deleted MRMSTEST files.
The problem is related with "Time" variable and the NetCDF files are
not CF compliant.
The following commands will resolve Bad ID issue if NCO is installed.
ncatted -O -a coordinates,FlashRate,m,c,"Time lon lat"
entlnFlashRate_20180330_01Z.nc ncatted -O -a
units,lon,m,c,"degrees_east" entlnFlashRate_20180330_01Z.nc ncatted -O
-a units,lat,m,c,"degrees_north" entlnFlashRate_20180330_01Z.nc
I also recommend to correct the Time unit (missing reference time).
Two issues with Bad ID:
Issue 1: time VS Time variable
FlashRate:coordinates = "lon lat time" ;
The coordinates attribute of FlashRate variable is "lon lat time".
This implies the coordinate variables are "lon", "lat", and "time".
But "time" variable does not exist. So a quick solution is changing
the attribute:
ncatted -O -a coordinates,FlashRate,m,c,"Time lon lat"
entlnFlashRate_20180330_01Z.nc
Issue 2: "degrees east" and "degrees north" are not CF compliant.
The recommended unit of latitude is degrees_north. Also acceptable are
degree_north, degree_N, degrees_N, degreeN, and degreesN.
A quick solution is changing them:
ncatted -O -a units,lon,m,c,"degrees_east"
entlnFlashRate_20180330_01Z.nc ncatted -O -a
units,lat,m,c,"degrees_north" entlnFlashRate_20180330_01Z.nc
So please let us know if this does not solve the NetCDF problem.
Have a nice weekend!
Cheers,
Howard
On Fri Dec 21 13:06:33 2018, jonathan.case-1 at nasa.gov wrote:
> Hi Howard,
>
> You don't need the MRMSTEST file. So please delete.
> I copied in the original files and re-tarred. Try working with the
> flashrate.tar files now.
>
> Thx,
> Jonathan
>
> -----Original Message-----
> From: Howard Soh via RT <met_help at ucar.edu>
> Sent: Friday, December 21, 2018 12:32 PM
> To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov>
> Subject: [rt.rap.ucar.edu #88228] using netcdf files with unique
field
> in pcp_combine
>
> Thank you on shared your data.
> I downloaded both files. flashrate.tar contains the softlinks only.
> Would you post the real NetCDF files?
>
> precip_tmp$ ls -l
> total 0
> lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:19 entln_met_2018033001.nc ->
> ../entlnFlashRate_20180330_01Z.nc lrwxrwxrwx 1 hsoh rap 33 Dec 18
> 11:19 entln_met_2018033002.nc -> ../entlnFlashRate_20180330_02Z.nc
> lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:19 entln_met_2018033003.nc ->
> ../entlnFlashRate_20180330_03Z.nc lrwxrwxrwx 1 hsoh rap 33 Dec 18
> 11:19 entln_met_2018033004.nc -> ../entlnFlashRate_20180330_04Z.nc
> lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:20 entln_met_2018033005.nc ->
> ../entlnFlashRate_20180330_05Z.nc lrwxrwxrwx 1 hsoh rap 33 Dec 18
> 11:20 entln_met_2018033006.nc -> ../entlnFlashRate_20180330_06Z.nc
>
>
> Cheers,
> Howard
>
> On Fri Dec 21 10:39:36 2018, jonathan.case-1 at nasa.gov wrote:
> > Hello again Howard,
> >
> > I tried running plot_data_place, but I immediately get a cryptic
> > error:
> > sc1login1:/raid1/sport/people/casejl/MET/gridData/precip_tmp:=>
> > plot_data_plane entln_met_2018033001.nc test FlashRate -v 5 ERROR
:
> > ERROR : yyerror() -> syntax error in file
> > "/tmp/config_20992_0_.temp"
> > ERROR :
> > ERROR : line = 2
> > ERROR :
> > ERROR : column = 1
> > ERROR :
> > ERROR : text = "(nul)"
> > ERROR :
> > ERROR :
> > ERROR : (nul)
> > ERROR : (nul)
> > ERROR :
> >
> > I uploaded the sample netcdf files to our public-facing server at:
> > https://geo.nsstc.nasa.gov/SPoRT/outgoing/jlc/MET/
> > (file is flashrate.tar, which will unpack in to the precip_tmp
sub-
> > folder). Please keep the data among yourself because as you're
> > likely aware, ENTLN lightning data are not freely open to the
> > public.
> >
> > All I really need to know is how we need to set up these netcdf
> > files correctly so that pcp_combine can operate on them. Once
that
> > hurdle is cleared, then I'm confident that I can output the
> > pcp_combine intermediate netcdf output files for use within
grid_stat.
> >
> > Thanks for your continued help.
> > -Jonathan
> >
> > -----Original Message-----
> > From: Howard Soh via RT <met_help at ucar.edu>
> > Sent: Thursday, December 20, 2018 2:42 PM
> > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
1 at nasa.gov>
> > Subject: [rt.rap.ucar.edu #88228] using netcdf files with unique
> > field in pcp_combine
> >
> > The error message is from the NetCDF library. The line:80 means it
> > failed to get the dimension size because the dimension ID is not
> > valid. So it could be a bad NetCDF file at ./precip_tmp.
> >
> > Would you post the data and configuration files as Julie suggested
> > before?
> >
> > What kind of trouble are you having? If you'd like you could post
> > the data to our ftp server ( following the instructions listed
> > here:
> > http://www.dtcenter.org/met/users/support/met_help.php#ftp)
> > and send me the command that you are running and I can see if I
can
> > help.
> >
> > If you won't post the data, would you do these?
> >
> > Option 1:
> >
> > - Run plot_data_plane for FlashRate variable with each NetCDF
files
> > at ./precip_tmp
> > - Check if all of them have the same problem
> >
> > Option 2:
> >
> > - check if the NetCDF files at ./precip_tmp have the same header.
> > - keep "entln_met_2018033001" at ./precip_tmp
> > - move other NetCDF files to somewhere else
> > - run pcp_combined command
> > - check if the same problem happens.
> > - If this works, add NetCDF file one by one.
> >
> > Cheers,
> > Howard
> >
> > On Thu Dec 20 12:21:46 2018, jonathan.case-1 at nasa.gov wrote:
> > > Hello again,
> > >
> > > I ran pcp_combine with -v5 log level. It appears that the time
is
> > > just a warning, whereas the program crashes in response to the
> > > NcBadId, which I'm not familiar with.
> > > Any insight on that? (see below)
> > >
> > > Sincerely,
> > > Jonathan
> > >
> > > pcp_combine -sum 00000000_000000 01 20180330_060000 06
> > > ./observed_APCP_06 -pcpdir ./precip_tmp -field
> > > 'name="FlashRate"; level="R1";' -v 5 -compress 0 DEBUG 2:
> > > Performing sum command:
> > > Init/In_Accum/Valid/Out_Accum Times =
> > > 00000000_000000/010000/20180330_060000/060000
> > > DEBUG 2: Searching for 6 files with accumulation times of
010000
> > > to sum to a total accumulation time of 060000.
> > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> > > new Met2dDataFile object of type "FileType_NcCF".
> > > DEBUG 4: NcCfFile::open() -> could not extract valid time from
> > > the "time" variable.
> > > DEBUG 4: NcCfFile::open() -> could not extract valid time from
> > > file name.
> > > WARNING:
> > > WARNING: NcCfFile::open() -> could not determine the valid
time,
> > > using 0.
> > > WARNING:
> > > DEBUG 4: NcCfFile::open() -> could not extract init time from
> > > the "forecast_reference_time" variable.
> > > DEBUG 4: NcCfFile::open() -> could not extract init time from
> > > file name.
> > > terminate called after throwing an instance of
> > > 'netCDF::exceptions::NcBadId'
> > > what(): NcBadId: Not a netcdf id
> > > file: ncDim.cpp line:80
> > > Aborted
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Howard Soh via RT <met_help at ucar.edu>
> > > Sent: Tuesday, December 18, 2018 4:37 PM
> > > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
> > > 1 at nasa.gov>
> > > Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files with
> > > unique field in pcp_combine
> > >
> > > The warning is caused by the variable name "Time" (vs. "time").
> > > MET is looking for "time" variable as CF NetCDF..
> > >
> > > double Time(time) ;
> > > Time:units = "seconds since " ;
> > > Time:calendar = "standard" ;
> > > Time:long_name = "time" ;
> > >
> > > And time:unit attribute does not have the reference time.
> > > time:units = "seconds since YYYY-MM-DD" ;
> > > OR
> > > time:units = "seconds since YYYY-MM-DD HH:MM:SS"
;
> > >
> > > If the verbose level is increased ("-v 4" or "-v 5"), it will
> > > display more useful information.
> > >
> > > Cheers,
> > > Howard
> > >
> > > On Tue, Dec 18, 2018 at 1:38 PM Case, Jonathan[ENSCO INC] via RT
<
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228
>
> > > >
> > > > Sorry Julie,
> > > >
> > > > I should have also included the pcp_combine error I'm getting:
> > > >
> > > > sc1login1:/raid1/sport/people/casejl/MET/gridData:=>
pcp_combine
> > > > -sum
> > > > 00000000_000000 01 20180330_060000 06 ./observed_APCP_06
> > > > -pcpdir ./precip_tmp -field 'name="FlashRate"; level="R1";' -v
3
> > > > -compress
> > > > 0
> > > > DEBUG 2: Performing sum command: Init/In_Accum/Valid/Out_Accum
> > > > Times =
> > > > 00000000_000000/010000/20180330_060000/060000
> > > > DEBUG 2: Searching for 6 files with accumulation times of
> > > > 010000 to sum to a total accumulation time of 060000.
> > > > WARNING:
> > > > WARNING: NcCfFile::open() -> could not determine the valid
time,
> > > > using 0.
> > > > WARNING:
> > > > terminate called after throwing an instance of
> > > > 'netCDF::exceptions::NcBadId'
> > > > what(): NcBadId: Not a netcdf id
> > > > file: ncDim.cpp line:80
> > > > Aborted
> > > >
> > > >
> > > > I need to know what the "NcBadId" means so I can persue a
> > > > possible solution.
> > > >
> > > > Thx!
> > > > -Jon
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Julie Prestopnik via RT <met_help at ucar.edu>
> > > > Sent: Tuesday, December 18, 2018 1:26 PM
> > > > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
> > > > 1 at nasa.gov>
> > > > Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files
with
> > > > unique field in pcp_combine
> > > >
> > > > Hi Jon. Unfortunately, our MET expert is out of the office
for
> > > > a while due to the holidays. I do not know the answers to
your
> > > > questions, however I can try to help.
> > > >
> > > > For starters, I'm just trying to work with the netcdf files to
> > > > see if I can
> > > > > combine them into "accumulated" fields using pcp_combine,
but
> > > > > I'm having trouble getting it to work.
> > > > >
> > > >
> > > > What kind of trouble are you having? If you'd like you could
> > > > post the data to our ftp server ( following the instructions
> > > > listed
> > > > here:
> > > > http://www.dtcenter.org/met/users/support/met_help.php#ftp)
and
> > > > send me the command that you are running and I can see if I
can
> > > > help.
> > > >
> > > > Thanks,
> > > > Julie
> > > >
> > > > On Tue, Dec 18, 2018 at 11:31 AM Case, Jonathan[ENSCO INC] via
> > > > RT < met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Tue Dec 18 11:31:09 2018: Request 88228 was acted upon.
> > > > > Transaction: Ticket created by jonathan.case-1 at nasa.gov
> > > > > Queue: met_help
> > > > > Subject: using netcdf files with unique field in
> > > > > pcp_combine
> > > > > Owner: Nobody
> > > > > Requestors: jonathan.case-1 at nasa.gov
> > > > > Status: new
> > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228
> > > > > >
> > > > >
> > > > >
> > > > > Dear MET Help,
> > > > >
> > > > > I'm trying to do something fairly unique in MET by
validating
> > > > > derived total lightning flash rates from an NWP model
against
> > > > > observed gridded lightning flash rates.
> > > > > I'm "faking" MET into thinking the model flash rates are
> > > > > simply accumulated precip (already named as APCP in GRIB2
> > > > > files), whereas the observed lightning files are identified
as
> > > > > "FlashRate" in hourly netcdf files.
> > > > >
> > > > > For starters, I'm just trying to work with the netcdf files
to
> > > > > see if I can combine them into "accumulated" fields using
> > > > > pcp_combine, but I'm having trouble getting it to work.
> > > > > What I'd like to do is combine/accumulate the data into
> > > > > various time intervals [much like precipitation], and then
> > > > > output the pcp_combine netcdf file with the traditional APCP
name.
> > > > >
> > > > > Is there a way I can do this using pcp_combine? I'm still
on
> > > > > MET v6.0. I have an ncdump of the lightning file below that
> > > > > might be
> > > > helpful.
> > > > >
> > > > > Thanks!
> > > > > -Jon
> > > > >
> > > > >
> > > > > Ncdump of sample lightning netcdf file:
> > > > > :=> ncdump -h ./precip_tmp/entln_met_2018033001.nc
> > > > > netcdf entln_met_2018033001 {
> > > > > dimensions:
> > > > > lon = 425 ;
> > > > > lat = 350 ;
> > > > > time = UNLIMITED ; // (1 currently)
> > > > > variables:
> > > > > float lat(lat) ;
> > > > > lat:units = "degrees_north" ;
> > > > > lat:long_name = "latitude" ;
> > > > > float lon(lon) ;
> > > > > lon:units = "degrees east" ;
> > > > > lon:long_name = "longitude" ;
> > > > > double Time(time) ;
> > > > > Time:units = "seconds since " ;
> > > > > Time:calendar = "standard" ;
> > > > > Time:long_name = "time" ;
> > > > > float FlashRate(time, lat, lon) ;
> > > > > FlashRate:units = "Flashes per km2 per 5
min" ;
> > > > > FlashRate:long_name = "" ;
> > > > > FlashRate:coordinates = "lon lat time" ;
> > > > >
> > > > > // global attributes:
> > > > > :Description = "" ;
> > > > > :Conventions = "CF-1.7" ;
> > > > > :title = "" ;
> > > > > :institution = "" ;
> > > > > :source = "{0}" ;
> > > > > :references = "" ;
> > > > > :history = "Created Mon Dec 17 22:23:33
2018" ;
> > > > > :comment = "" ; }
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *****************************************************
> > > > > Jonathan Case (Jon)
> > > > > ENSCO, Inc./NASA SPoRT Center
> > > > > National Space Science and Technology Center
> > > > > 320 Sparkman Drive, Room 3000B
> > > > > Huntsville, AL 35805
> > > > > Jonathan.Case-1 at nasa.gov<mailto:Jonathan.Case-1 at nasa.gov>
(alt:
> > > > > case.jonathan at ensco.com<mailto:case.jonathan at ensco.com>)
> > > > > *****************************************************
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
------------------------------------------------
Subject: using netcdf files with unique field in pcp_combine
From: John Halley Gotway
Time: Thu Jan 03 14:38:04 2019
Hi Jon,
This is John Halley Gotway. I was out of the office the last couple
of
weeks, but I saw you had a met-help question about validating
lightning
with MET. Just wondering how it's going. Do you have any remaning
questions?
Thanks,
John
On Mon, Dec 31, 2018 at 8:55 AM Case, Jonathan[ENSCO INC] via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228 >
>
> Thanks for the information Howard. We will work on getting the
lightning
> flash netcdf files CF-compliant and addressing the issues you raised
below.
> Sincerely,
> Jonathan
>
> -----Original Message-----
> From: Howard Soh via RT <met_help at ucar.edu>
> Sent: Friday, December 21, 2018 3:33 PM
> To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-1 at nasa.gov>
> Subject: [rt.rap.ucar.edu #88228] using netcdf files with unique
field in
> pcp_combine
>
> I appreciate your posting and I deleted MRMSTEST files.
>
> The problem is related with "Time" variable and the NetCDF files are
not
> CF compliant.
>
> The following commands will resolve Bad ID issue if NCO is
installed.
>
> ncatted -O -a coordinates,FlashRate,m,c,"Time lon lat"
> entlnFlashRate_20180330_01Z.nc ncatted -O -a
units,lon,m,c,"degrees_east"
> entlnFlashRate_20180330_01Z.nc ncatted -O -a
units,lat,m,c,"degrees_north"
> entlnFlashRate_20180330_01Z.nc
>
> I also recommend to correct the Time unit (missing reference time).
>
> Two issues with Bad ID:
>
> Issue 1: time VS Time variable
>
> FlashRate:coordinates = "lon lat time" ;
>
> The coordinates attribute of FlashRate variable is "lon lat time".
This
> implies the coordinate variables are "lon", "lat", and "time". But
"time"
> variable does not exist. So a quick solution is changing the
attribute:
>
> ncatted -O -a coordinates,FlashRate,m,c,"Time lon lat"
> entlnFlashRate_20180330_01Z.nc
>
> Issue 2: "degrees east" and "degrees north" are not CF compliant.
> The recommended unit of latitude is degrees_north. Also acceptable
are
> degree_north, degree_N, degrees_N, degreeN, and degreesN.
>
> A quick solution is changing them:
>
> ncatted -O -a units,lon,m,c,"degrees_east"
entlnFlashRate_20180330_01Z.nc
> ncatted -O -a units,lat,m,c,"degrees_north"
entlnFlashRate_20180330_01Z.nc
>
> So please let us know if this does not solve the NetCDF problem.
>
> Have a nice weekend!
>
> Cheers,
> Howard
>
> On Fri Dec 21 13:06:33 2018, jonathan.case-1 at nasa.gov wrote:
> > Hi Howard,
> >
> > You don't need the MRMSTEST file. So please delete.
> > I copied in the original files and re-tarred. Try working with the
> > flashrate.tar files now.
> >
> > Thx,
> > Jonathan
> >
> > -----Original Message-----
> > From: Howard Soh via RT <met_help at ucar.edu>
> > Sent: Friday, December 21, 2018 12:32 PM
> > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
1 at nasa.gov>
> > Subject: [rt.rap.ucar.edu #88228] using netcdf files with unique
field
> > in pcp_combine
> >
> > Thank you on shared your data.
> > I downloaded both files. flashrate.tar contains the softlinks
only.
> > Would you post the real NetCDF files?
> >
> > precip_tmp$ ls -l
> > total 0
> > lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:19 entln_met_2018033001.nc ->
> > ../entlnFlashRate_20180330_01Z.nc lrwxrwxrwx 1 hsoh rap 33 Dec 18
> > 11:19 entln_met_2018033002.nc -> ../entlnFlashRate_20180330_02Z.nc
> > lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:19 entln_met_2018033003.nc ->
> > ../entlnFlashRate_20180330_03Z.nc lrwxrwxrwx 1 hsoh rap 33 Dec 18
> > 11:19 entln_met_2018033004.nc -> ../entlnFlashRate_20180330_04Z.nc
> > lrwxrwxrwx 1 hsoh rap 33 Dec 18 11:20 entln_met_2018033005.nc ->
> > ../entlnFlashRate_20180330_05Z.nc lrwxrwxrwx 1 hsoh rap 33 Dec 18
> > 11:20 entln_met_2018033006.nc -> ../entlnFlashRate_20180330_06Z.nc
> >
> >
> > Cheers,
> > Howard
> >
> > On Fri Dec 21 10:39:36 2018, jonathan.case-1 at nasa.gov wrote:
> > > Hello again Howard,
> > >
> > > I tried running plot_data_place, but I immediately get a cryptic
> > > error:
> > > sc1login1:/raid1/sport/people/casejl/MET/gridData/precip_tmp:=>
> > > plot_data_plane entln_met_2018033001.nc test FlashRate -v 5
ERROR :
> > > ERROR : yyerror() -> syntax error in file
> > > "/tmp/config_20992_0_.temp"
> > > ERROR :
> > > ERROR : line = 2
> > > ERROR :
> > > ERROR : column = 1
> > > ERROR :
> > > ERROR : text = "(nul)"
> > > ERROR :
> > > ERROR :
> > > ERROR : (nul)
> > > ERROR : (nul)
> > > ERROR :
> > >
> > > I uploaded the sample netcdf files to our public-facing server
at:
> > > https://geo.nsstc.nasa.gov/SPoRT/outgoing/jlc/MET/
> > > (file is flashrate.tar, which will unpack in to the precip_tmp
sub-
> > > folder). Please keep the data among yourself because as you're
> > > likely aware, ENTLN lightning data are not freely open to the
> > > public.
> > >
> > > All I really need to know is how we need to set up these netcdf
> > > files correctly so that pcp_combine can operate on them. Once
that
> > > hurdle is cleared, then I'm confident that I can output the
> > > pcp_combine intermediate netcdf output files for use within
grid_stat.
> > >
> > > Thanks for your continued help.
> > > -Jonathan
> > >
> > > -----Original Message-----
> > > From: Howard Soh via RT <met_help at ucar.edu>
> > > Sent: Thursday, December 20, 2018 2:42 PM
> > > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
1 at nasa.gov>
> > > Subject: [rt.rap.ucar.edu #88228] using netcdf files with
unique
> > > field in pcp_combine
> > >
> > > The error message is from the NetCDF library. The line:80 means
it
> > > failed to get the dimension size because the dimension ID is not
> > > valid. So it could be a bad NetCDF file at ./precip_tmp.
> > >
> > > Would you post the data and configuration files as Julie
suggested
> > > before?
> > >
> > > What kind of trouble are you having? If you'd like you could
post
> > > the data to our ftp server ( following the instructions listed
> > > here:
> > > http://www.dtcenter.org/met/users/support/met_help.php#ftp)
> > > and send me the command that you are running and I can see if I
can
> > > help.
> > >
> > > If you won't post the data, would you do these?
> > >
> > > Option 1:
> > >
> > > - Run plot_data_plane for FlashRate variable with each NetCDF
files
> > > at ./precip_tmp
> > > - Check if all of them have the same problem
> > >
> > > Option 2:
> > >
> > > - check if the NetCDF files at ./precip_tmp have the same
header.
> > > - keep "entln_met_2018033001" at ./precip_tmp
> > > - move other NetCDF files to somewhere else
> > > - run pcp_combined command
> > > - check if the same problem happens.
> > > - If this works, add NetCDF file one by one.
> > >
> > > Cheers,
> > > Howard
> > >
> > > On Thu Dec 20 12:21:46 2018, jonathan.case-1 at nasa.gov wrote:
> > > > Hello again,
> > > >
> > > > I ran pcp_combine with -v5 log level. It appears that the
time is
> > > > just a warning, whereas the program crashes in response to the
> > > > NcBadId, which I'm not familiar with.
> > > > Any insight on that? (see below)
> > > >
> > > > Sincerely,
> > > > Jonathan
> > > >
> > > > pcp_combine -sum 00000000_000000 01 20180330_060000 06
> > > > ./observed_APCP_06 -pcpdir ./precip_tmp -field
> > > > 'name="FlashRate"; level="R1";' -v 5 -compress 0 DEBUG 2:
> > > > Performing sum command:
> > > > Init/In_Accum/Valid/Out_Accum Times =
> > > > 00000000_000000/010000/20180330_060000/060000
> > > > DEBUG 2: Searching for 6 files with accumulation times of
010000
> > > > to sum to a total accumulation time of 060000.
> > > > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> > > > new Met2dDataFile object of type "FileType_NcCF".
> > > > DEBUG 4: NcCfFile::open() -> could not extract valid time
from
> > > > the "time" variable.
> > > > DEBUG 4: NcCfFile::open() -> could not extract valid time
from
> > > > file name.
> > > > WARNING:
> > > > WARNING: NcCfFile::open() -> could not determine the valid
time,
> > > > using 0.
> > > > WARNING:
> > > > DEBUG 4: NcCfFile::open() -> could not extract init time
from
> > > > the "forecast_reference_time" variable.
> > > > DEBUG 4: NcCfFile::open() -> could not extract init time
from
> > > > file name.
> > > > terminate called after throwing an instance of
> > > > 'netCDF::exceptions::NcBadId'
> > > > what(): NcBadId: Not a netcdf id
> > > > file: ncDim.cpp line:80
> > > > Aborted
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Howard Soh via RT <met_help at ucar.edu>
> > > > Sent: Tuesday, December 18, 2018 4:37 PM
> > > > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
> > > > 1 at nasa.gov>
> > > > Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files
with
> > > > unique field in pcp_combine
> > > >
> > > > The warning is caused by the variable name "Time" (vs.
"time").
> > > > MET is looking for "time" variable as CF NetCDF..
> > > >
> > > > double Time(time) ;
> > > > Time:units = "seconds since " ;
> > > > Time:calendar = "standard" ;
> > > > Time:long_name = "time" ;
> > > >
> > > > And time:unit attribute does not have the reference time.
> > > > time:units = "seconds since YYYY-MM-DD" ;
> > > > OR
> > > > time:units = "seconds since YYYY-MM-DD
HH:MM:SS" ;
> > > >
> > > > If the verbose level is increased ("-v 4" or "-v 5"), it will
> > > > display more useful information.
> > > >
> > > > Cheers,
> > > > Howard
> > > >
> > > > On Tue, Dec 18, 2018 at 1:38 PM Case, Jonathan[ENSCO INC] via
RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228 >
> > > > >
> > > > > Sorry Julie,
> > > > >
> > > > > I should have also included the pcp_combine error I'm
getting:
> > > > >
> > > > > sc1login1:/raid1/sport/people/casejl/MET/gridData:=>
pcp_combine
> > > > > -sum
> > > > > 00000000_000000 01 20180330_060000 06 ./observed_APCP_06
> > > > > -pcpdir ./precip_tmp -field 'name="FlashRate"; level="R1";'
-v 3
> > > > > -compress
> > > > > 0
> > > > > DEBUG 2: Performing sum command:
Init/In_Accum/Valid/Out_Accum
> > > > > Times =
> > > > > 00000000_000000/010000/20180330_060000/060000
> > > > > DEBUG 2: Searching for 6 files with accumulation times of
> > > > > 010000 to sum to a total accumulation time of 060000.
> > > > > WARNING:
> > > > > WARNING: NcCfFile::open() -> could not determine the valid
time,
> > > > > using 0.
> > > > > WARNING:
> > > > > terminate called after throwing an instance of
> > > > > 'netCDF::exceptions::NcBadId'
> > > > > what(): NcBadId: Not a netcdf id
> > > > > file: ncDim.cpp line:80
> > > > > Aborted
> > > > >
> > > > >
> > > > > I need to know what the "NcBadId" means so I can persue a
> > > > > possible solution.
> > > > >
> > > > > Thx!
> > > > > -Jon
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Julie Prestopnik via RT <met_help at ucar.edu>
> > > > > Sent: Tuesday, December 18, 2018 1:26 PM
> > > > > To: Case, Jonathan (MSFC-ST11)[ENSCO INC] <jonathan.case-
> > > > > 1 at nasa.gov>
> > > > > Subject: Re: [rt.rap.ucar.edu #88228] using netcdf files
with
> > > > > unique field in pcp_combine
> > > > >
> > > > > Hi Jon. Unfortunately, our MET expert is out of the office
for
> > > > > a while due to the holidays. I do not know the answers to
your
> > > > > questions, however I can try to help.
> > > > >
> > > > > For starters, I'm just trying to work with the netcdf files
to
> > > > > see if I can
> > > > > > combine them into "accumulated" fields using pcp_combine,
but
> > > > > > I'm having trouble getting it to work.
> > > > > >
> > > > >
> > > > > What kind of trouble are you having? If you'd like you
could
> > > > > post the data to our ftp server ( following the instructions
> > > > > listed
> > > > > here:
> > > > > http://www.dtcenter.org/met/users/support/met_help.php#ftp)
and
> > > > > send me the command that you are running and I can see if I
can
> > > > > help.
> > > > >
> > > > > Thanks,
> > > > > Julie
> > > > >
> > > > > On Tue, Dec 18, 2018 at 11:31 AM Case, Jonathan[ENSCO INC]
via
> > > > > RT < met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Tue Dec 18 11:31:09 2018: Request 88228 was acted upon.
> > > > > > Transaction: Ticket created by jonathan.case-1 at nasa.gov
> > > > > > Queue: met_help
> > > > > > Subject: using netcdf files with unique field in
> > > > > > pcp_combine
> > > > > > Owner: Nobody
> > > > > > Requestors: jonathan.case-1 at nasa.gov
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=88228
> > > > > > >
> > > > > >
> > > > > >
> > > > > > Dear MET Help,
> > > > > >
> > > > > > I'm trying to do something fairly unique in MET by
validating
> > > > > > derived total lightning flash rates from an NWP model
against
> > > > > > observed gridded lightning flash rates.
> > > > > > I'm "faking" MET into thinking the model flash rates are
> > > > > > simply accumulated precip (already named as APCP in GRIB2
> > > > > > files), whereas the observed lightning files are
identified as
> > > > > > "FlashRate" in hourly netcdf files.
> > > > > >
> > > > > > For starters, I'm just trying to work with the netcdf
files to
> > > > > > see if I can combine them into "accumulated" fields using
> > > > > > pcp_combine, but I'm having trouble getting it to work.
> > > > > > What I'd like to do is combine/accumulate the data into
> > > > > > various time intervals [much like precipitation], and then
> > > > > > output the pcp_combine netcdf file with the traditional
APCP
> name.
> > > > > >
> > > > > > Is there a way I can do this using pcp_combine? I'm still
on
> > > > > > MET v6.0. I have an ncdump of the lightning file below
that
> > > > > > might be
> > > > > helpful.
> > > > > >
> > > > > > Thanks!
> > > > > > -Jon
> > > > > >
> > > > > >
> > > > > > Ncdump of sample lightning netcdf file:
> > > > > > :=> ncdump -h ./precip_tmp/entln_met_2018033001.nc
> > > > > > netcdf entln_met_2018033001 {
> > > > > > dimensions:
> > > > > > lon = 425 ;
> > > > > > lat = 350 ;
> > > > > > time = UNLIMITED ; // (1 currently)
> > > > > > variables:
> > > > > > float lat(lat) ;
> > > > > > lat:units = "degrees_north" ;
> > > > > > lat:long_name = "latitude" ;
> > > > > > float lon(lon) ;
> > > > > > lon:units = "degrees east" ;
> > > > > > lon:long_name = "longitude" ;
> > > > > > double Time(time) ;
> > > > > > Time:units = "seconds since " ;
> > > > > > Time:calendar = "standard" ;
> > > > > > Time:long_name = "time" ;
> > > > > > float FlashRate(time, lat, lon) ;
> > > > > > FlashRate:units = "Flashes per km2 per 5
min" ;
> > > > > > FlashRate:long_name = "" ;
> > > > > > FlashRate:coordinates = "lon lat time" ;
> > > > > >
> > > > > > // global attributes:
> > > > > > :Description = "" ;
> > > > > > :Conventions = "CF-1.7" ;
> > > > > > :title = "" ;
> > > > > > :institution = "" ;
> > > > > > :source = "{0}" ;
> > > > > > :references = "" ;
> > > > > > :history = "Created Mon Dec 17 22:23:33
2018" ;
> > > > > > :comment = "" ; }
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > *****************************************************
> > > > > > Jonathan Case (Jon)
> > > > > > ENSCO, Inc./NASA SPoRT Center
> > > > > > National Space Science and Technology Center
> > > > > > 320 Sparkman Drive, Room 3000B
> > > > > > Huntsville, AL 35805
> > > > > > Jonathan.Case-1 at nasa.gov<mailto:Jonathan.Case-1 at nasa.gov>
(alt:
> > > > > > case.jonathan at ensco.com<mailto:case.jonathan at ensco.com>)
> > > > > > *****************************************************
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
>
>
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list