[Met_help] [rt.rap.ucar.edu #83514] History for RFC netcdf input into regrid_data_plane
John Halley Gotway via RT
met_help at ucar.edu
Tue Jul 9 12:03:58 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
To whom it may concern,
I have been working with ABRFC Mosaic QPF in netcdf format and using it as
input into regrid_data_plane. Using the simple script below, I get
following results (pasted below script). I was'nt sure how severe the
ERRORS were and wanted to double check with the help desk. I did notice in
the MET FAQ that to "use gridded NetCDF files in MET, the files need to
look like the output of the pcp_combine tool" with some requirements
listed.
I have sent the script and input files via the anon ftp site in diehl_data.
The script name is rfc_test.sh, input files are rfc_2018010512f006.nc and
urma_co.
NOTE: I get the following from ncdump:
float QPF_SFC(ProjectionHr, YCells, XCells) ;
QPF_SFC:long_name = "Total precipitation" ;
QPF_SFC:units = "inch" ;
QPF_SFC:coordinates = "longitude latitude" ;
QPF_SFC:ReferenceTime = 1515153600 ;
QPF_SFC:ReferenceTimeString = "2018-01-05 12:00:00 00:00" ;
QPF_SFC:gridType = "SCALAR" ;
QPF_SFC:level = "SFC" ;
Note that there is no name defined, only long_name. I assume name is
QPF_SFC.
Thanks in advance for your help....Brad Diehl
#!/bin/sh
###
############
source /u/Brad.Diehl/met_env/tut_env
regrid_data_plane rfc_2018010512f006.nc \
urma_co \
re_grid_out.nc \
-method BUDGET \
-width 2 \
-v 2 \
-field 'name="QPF_SFC"; level="(*,*)";'
DEBUG 1: Reading data file: rfc_2018010512f006.nc
WARNING:
WARNING: NcCfFile::open() -> could not determine the valid time, using 0.
WARNING:
DEBUG 2: Input grid: Projection: Lambert Conformal Nx: 2145 Ny: 1377
Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000 Alpha: 6509.082 Cone:
0.423 Bx: 1088.0030 By: 5483.6617
DEBUG 2: Output grid: Projection: Lambert Conformal Nx: 2145 Ny: 1377
Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000 Alpha: 6509.082 Cone:
0.423 Bx: 1088.0030 By: 5483.6617
DEBUG 2: Interpolation options: method = BUDGET, width = 2, vld_thresh = 0.5
ERROR :
ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &) ->
needed 3 arguments for variable QPF_SFC, got 2
ERROR :
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: RFC netcdf input into regrid_data_plane
From: John Halley Gotway
Time: Tue Jan 09 08:01:59 2018
Hello Brad,
I see you're having trouble passing a NetCDF file to the
regrid_data_plane
tool. The problem is how you've specified the "level" value in the
command
line argument. You specified "(*,*)" (i.e. 2 dimensions) but that
variable
actually has 3 dimensions:
QPF_SFC(ProjectionHr, YCells, XCells)
That's what this error message is telling you:
ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &)
->
needed 3 arguments for variable QPF_SFC, got 2
Please try using this:
-field 'name="QPF_SFC"; level="(0,*,*)";'
That'll process the first 2D slice of data from that 3D variable.
Thanks,
John
On Mon, Jan 8, 2018 at 10:00 AM, Brad Diehl - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> Mon Jan 08 10:00:01 2018: Request 83514 was acted upon.
> Transaction: Ticket created by brad.diehl at noaa.gov
> Queue: met_help
> Subject: RFC netcdf input into regrid_data_plane
> Owner: Nobody
> Requestors: brad.diehl at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
>
>
> To whom it may concern,
>
> I have been working with ABRFC Mosaic QPF in netcdf format and using
it as
> input into regrid_data_plane. Using the simple script below, I get
> following results (pasted below script). I was'nt sure how severe
the
> ERRORS were and wanted to double check with the help desk. I did
notice in
> the MET FAQ that to "use gridded NetCDF files in MET, the files need
to
> look like the output of the pcp_combine tool" with some requirements
> listed.
>
> I have sent the script and input files via the anon ftp site in
diehl_data.
> The script name is rfc_test.sh, input files are
rfc_2018010512f006.nc and
> urma_co.
>
> NOTE: I get the following from ncdump:
>
>
> float QPF_SFC(ProjectionHr, YCells, XCells) ;
> QPF_SFC:long_name = "Total precipitation" ;
> QPF_SFC:units = "inch" ;
> QPF_SFC:coordinates = "longitude latitude" ;
> QPF_SFC:ReferenceTime = 1515153600 ;
> QPF_SFC:ReferenceTimeString = "2018-01-05 12:00:00 00:00" ;
> QPF_SFC:gridType = "SCALAR" ;
> QPF_SFC:level = "SFC" ;
>
> Note that there is no name defined, only long_name. I assume name
is
> QPF_SFC.
>
>
>
> Thanks in advance for your help....Brad Diehl
>
> #!/bin/sh
> ###
> ############
>
> source /u/Brad.Diehl/met_env/tut_env
>
> regrid_data_plane rfc_2018010512f006.nc \
> urma_co \
> re_grid_out.nc \
> -method BUDGET \
> -width 2 \
> -v 2 \
> -field 'name="QPF_SFC"; level="(*,*)";'
>
>
>
> DEBUG 1: Reading data file: rfc_2018010512f006.nc
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> WARNING:
> DEBUG 2: Input grid: Projection: Lambert Conformal Nx: 2145 Ny: 1377
> Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000 Alpha: 6509.082
Cone:
> 0.423 Bx: 1088.0030 By: 5483.6617
> DEBUG 2: Output grid: Projection: Lambert Conformal Nx: 2145 Ny:
1377
> Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000 Alpha: 6509.082
Cone:
> 0.423 Bx: 1088.0030 By: 5483.6617
> DEBUG 2: Interpolation options: method = BUDGET, width = 2,
vld_thresh =
> 0.5
> ERROR :
> ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &)
->
> needed 3 arguments for variable QPF_SFC, got 2
> ERROR :
>
>
------------------------------------------------
Subject: RFC netcdf input into regrid_data_plane
From: Brad Diehl - NOAA Federal
Time: Wed Jan 10 07:00:38 2018
Once again, thanks for the help, I'll have to be more aware of the
dimensions moving forward. And the solution of using -field
'name="QPF_SFC"; level="(0,*,*)";' worked just fine. Many thanks
again.
However, I continue to get the warning below from regrid_data_plane.
Not
sure if its a show stopper but does'nt appear to be. Is there anything
I
can do? If not, I'll consider this ticket closed.
WARNING:
WARNING: NcCfFile::open() -> could not determine the valid time, using
0.
WARNING:
On Tue, Jan 9, 2018 at 3:01 PM, John Halley Gotway via RT
<met_help at ucar.edu
> wrote:
> Hello Brad,
>
> I see you're having trouble passing a NetCDF file to the
regrid_data_plane
> tool. The problem is how you've specified the "level" value in the
command
> line argument. You specified "(*,*)" (i.e. 2 dimensions) but that
variable
> actually has 3 dimensions:
> QPF_SFC(ProjectionHr, YCells, XCells)
>
> That's what this error message is telling you:
> ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane
&) ->
> needed 3 arguments for variable QPF_SFC, got 2
>
> Please try using this:
> -field 'name="QPF_SFC"; level="(0,*,*)";'
>
> That'll process the first 2D slice of data from that 3D variable.
>
> Thanks,
> John
>
> On Mon, Jan 8, 2018 at 10:00 AM, Brad Diehl - NOAA Federal via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Jan 08 10:00:01 2018: Request 83514 was acted upon.
> > Transaction: Ticket created by brad.diehl at noaa.gov
> > Queue: met_help
> > Subject: RFC netcdf input into regrid_data_plane
> > Owner: Nobody
> > Requestors: brad.diehl at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
> >
> >
> > To whom it may concern,
> >
> > I have been working with ABRFC Mosaic QPF in netcdf format and
using it
> as
> > input into regrid_data_plane. Using the simple script below, I get
> > following results (pasted below script). I was'nt sure how severe
the
> > ERRORS were and wanted to double check with the help desk. I did
notice
> in
> > the MET FAQ that to "use gridded NetCDF files in MET, the files
need to
> > look like the output of the pcp_combine tool" with some
requirements
> > listed.
> >
> > I have sent the script and input files via the anon ftp site in
> diehl_data.
> > The script name is rfc_test.sh, input files are
rfc_2018010512f006.nc
> and
> > urma_co.
> >
> > NOTE: I get the following from ncdump:
> >
> >
> > float QPF_SFC(ProjectionHr, YCells, XCells) ;
> > QPF_SFC:long_name = "Total precipitation" ;
> > QPF_SFC:units = "inch" ;
> > QPF_SFC:coordinates = "longitude latitude" ;
> > QPF_SFC:ReferenceTime = 1515153600 ;
> > QPF_SFC:ReferenceTimeString = "2018-01-05 12:00:00 00:00"
;
> > QPF_SFC:gridType = "SCALAR" ;
> > QPF_SFC:level = "SFC" ;
> >
> > Note that there is no name defined, only long_name. I assume name
is
> > QPF_SFC.
> >
> >
> >
> > Thanks in advance for your help....Brad Diehl
> >
> > #!/bin/sh
> > ###
> > ############
> >
> > source /u/Brad.Diehl/met_env/tut_env
> >
> > regrid_data_plane rfc_2018010512f006.nc \
> > urma_co \
> > re_grid_out.nc \
> > -method BUDGET \
> > -width 2 \
> > -v 2 \
> > -field 'name="QPF_SFC"; level="(*,*)";'
> >
> >
> >
> > DEBUG 1: Reading data file: rfc_2018010512f006.nc
> > WARNING:
> > WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> > WARNING:
> > DEBUG 2: Input grid: Projection: Lambert Conformal Nx: 2145 Ny:
1377
> > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000 Alpha: 6509.082
Cone:
> > 0.423 Bx: 1088.0030 By: 5483.6617
> > DEBUG 2: Output grid: Projection: Lambert Conformal Nx: 2145 Ny:
1377
> > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000 Alpha: 6509.082
Cone:
> > 0.423 Bx: 1088.0030 By: 5483.6617
> > DEBUG 2: Interpolation options: method = BUDGET, width = 2,
vld_thresh =
> > 0.5
> > ERROR :
> > ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane
&) ->
> > needed 3 arguments for variable QPF_SFC, got 2
> > ERROR :
> >
> >
>
>
------------------------------------------------
Subject: RFC netcdf input into regrid_data_plane
From: John Halley Gotway
Time: Wed Jan 10 10:22:36 2018
Brad,
I took a look at your data. This file is a NetCDF file which
purportedly
follows the CF convention. However, MET is not correctly parsing the
timing info. Running though the debugger, I find that it's expecting
the
timing variable to simply be named "time". To test this out, I used
the
ncrename utility to rename the "ProjectionHr" variable and dimension
to
"time":
ncrename -v ProjectionHr,time -d ProjectionHr,time -o
rfc_2018010512f006_MOD.nc rfc_2018010512f006.nc
Rerunning reveals the next warning:
WARNING: parse_cf_time_string() -> unexpected NetCDF CF convention
time
unit "2018-01-05 12:00:00 00:00"
For the CF convention, the "units" attribute for time should look like
this
"[TIME UNITS] since [REFERENCE TIME]". To test, I used the ncatted
utility
to modify your sample file:
ncatted -a units,time,o,c,"hours since 2018-01-05 12:00:00"
rfc_2018010512f006_MOD.nc
And finally that allows MET run without warning:
met-6.1/bin/plot_data_plane rfc_2018010512f006_MOD.nc QPF_SFC.ps
'name="QPF_SFC"; level="(0,*,*)";'
And the resulting plot is attached.
In all, there are 3 issues:
(1) MET expects the time variable to be named "time". I'm not sure
whether
this is actually a CF requirement. If not, please let me know, and we
can
take a closer look at the CF convention to see what MET *should* be
doing.
(2) The time "units" attribute must be named "[TIME UNITS] since
[REFERENCE
TIME]". I'm pretty confident that this a CF requirement.
(3) If this is analysis data, then we're all set. But for forecast
data,
you'd need to define the forecast initialization time in the CF
compliant
way. That's done by defining a variable named
"forecast_reference_time"...
looking something like this:
double forecast_reference_time ;
forecast_reference_time:long_name = "reference time" ;
forecast_reference_time:units = "seconds since 1970-01-01" ;
Of course the units can be defined as seconds, minutes, hour, days,
and so
on.
Thanks,
John
On Wed, Jan 10, 2018 at 7:00 AM, Brad Diehl - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
>
> Once again, thanks for the help, I'll have to be more aware of the
> dimensions moving forward. And the solution of using -field
> 'name="QPF_SFC"; level="(0,*,*)";' worked just fine. Many thanks
again.
>
>
> However, I continue to get the warning below from regrid_data_plane.
Not
> sure if its a show stopper but does'nt appear to be. Is there
anything I
> can do? If not, I'll consider this ticket closed.
>
> WARNING:
> WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> WARNING:
>
> On Tue, Jan 9, 2018 at 3:01 PM, John Halley Gotway via RT <
> met_help at ucar.edu
> > wrote:
>
> > Hello Brad,
> >
> > I see you're having trouble passing a NetCDF file to the
> regrid_data_plane
> > tool. The problem is how you've specified the "level" value in
the
> command
> > line argument. You specified "(*,*)" (i.e. 2 dimensions) but that
> variable
> > actually has 3 dimensions:
> > QPF_SFC(ProjectionHr, YCells, XCells)
> >
> > That's what this error message is telling you:
> > ERROR : NcCfFile::getData(NcVar *, const LongArray &,
DataPlane &) ->
> > needed 3 arguments for variable QPF_SFC, got 2
> >
> > Please try using this:
> > -field 'name="QPF_SFC"; level="(0,*,*)";'
> >
> > That'll process the first 2D slice of data from that 3D variable.
> >
> > Thanks,
> > John
> >
> > On Mon, Jan 8, 2018 at 10:00 AM, Brad Diehl - NOAA Federal via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Mon Jan 08 10:00:01 2018: Request 83514 was acted upon.
> > > Transaction: Ticket created by brad.diehl at noaa.gov
> > > Queue: met_help
> > > Subject: RFC netcdf input into regrid_data_plane
> > > Owner: Nobody
> > > Requestors: brad.diehl at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514
> >
> > >
> > >
> > > To whom it may concern,
> > >
> > > I have been working with ABRFC Mosaic QPF in netcdf format and
using it
> > as
> > > input into regrid_data_plane. Using the simple script below, I
get
> > > following results (pasted below script). I was'nt sure how
severe the
> > > ERRORS were and wanted to double check with the help desk. I did
notice
> > in
> > > the MET FAQ that to "use gridded NetCDF files in MET, the files
need to
> > > look like the output of the pcp_combine tool" with some
requirements
> > > listed.
> > >
> > > I have sent the script and input files via the anon ftp site in
> > diehl_data.
> > > The script name is rfc_test.sh, input files are
rfc_2018010512f006.nc
> > and
> > > urma_co.
> > >
> > > NOTE: I get the following from ncdump:
> > >
> > >
> > > float QPF_SFC(ProjectionHr, YCells, XCells) ;
> > > QPF_SFC:long_name = "Total precipitation" ;
> > > QPF_SFC:units = "inch" ;
> > > QPF_SFC:coordinates = "longitude latitude" ;
> > > QPF_SFC:ReferenceTime = 1515153600 ;
> > > QPF_SFC:ReferenceTimeString = "2018-01-05 12:00:00
00:00" ;
> > > QPF_SFC:gridType = "SCALAR" ;
> > > QPF_SFC:level = "SFC" ;
> > >
> > > Note that there is no name defined, only long_name. I assume
name is
> > > QPF_SFC.
> > >
> > >
> > >
> > > Thanks in advance for your help....Brad Diehl
> > >
> > > #!/bin/sh
> > > ###
> > > ############
> > >
> > > source /u/Brad.Diehl/met_env/tut_env
> > >
> > > regrid_data_plane rfc_2018010512f006.nc \
> > > urma_co \
> > > re_grid_out.nc \
> > > -method BUDGET \
> > > -width 2 \
> > > -v 2 \
> > > -field 'name="QPF_SFC"; level="(*,*)";'
> > >
> > >
> > >
> > > DEBUG 1: Reading data file: rfc_2018010512f006.nc
> > > WARNING:
> > > WARNING: NcCfFile::open() -> could not determine the valid time,
using
> 0.
> > > WARNING:
> > > DEBUG 2: Input grid: Projection: Lambert Conformal Nx: 2145 Ny:
1377
> > > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000 Alpha:
6509.082 Cone:
> > > 0.423 Bx: 1088.0030 By: 5483.6617
> > > DEBUG 2: Output grid: Projection: Lambert Conformal Nx: 2145 Ny:
1377
> > > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000 Alpha:
6509.082 Cone:
> > > 0.423 Bx: 1088.0030 By: 5483.6617
> > > DEBUG 2: Interpolation options: method = BUDGET, width = 2,
vld_thresh
> =
> > > 0.5
> > > ERROR :
> > > ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane
&) ->
> > > needed 3 arguments for variable QPF_SFC, got 2
> > > ERROR :
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: RFC netcdf input into regrid_data_plane
From: Brad Diehl - NOAA Federal
Time: Wed Jan 10 11:15:49 2018
Thanks for the reply and recommendations. This data set has been a
challenge in different ways, and this just adds to it. But if we use
the
NCO utilities that you mentioned we can manipulate it, and it is
forecast
data so some further processing will me necessary as you mentioned.
I'll
likely be back in touch, but thanks for all the effort thus
far.....Brad
Diehl
On Wed, Jan 10, 2018 at 5:22 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Brad,
>
> I took a look at your data. This file is a NetCDF file which
purportedly
> follows the CF convention. However, MET is not correctly parsing
the
> timing info. Running though the debugger, I find that it's
expecting the
> timing variable to simply be named "time". To test this out, I used
the
> ncrename utility to rename the "ProjectionHr" variable and dimension
to
> "time":
>
> ncrename -v ProjectionHr,time -d ProjectionHr,time -o
> rfc_2018010512f006_MOD.nc rfc_2018010512f006.nc
>
>
> Rerunning reveals the next warning:
>
>
> WARNING: parse_cf_time_string() -> unexpected NetCDF CF convention
time
> unit "2018-01-05 12:00:00 00:00"
>
>
> For the CF convention, the "units" attribute for time should look
like this
> "[TIME UNITS] since [REFERENCE TIME]". To test, I used the ncatted
utility
> to modify your sample file:
>
>
> ncatted -a units,time,o,c,"hours since 2018-01-05 12:00:00"
> rfc_2018010512f006_MOD.nc
>
>
> And finally that allows MET run without warning:
>
>
> met-6.1/bin/plot_data_plane rfc_2018010512f006_MOD.nc QPF_SFC.ps
> 'name="QPF_SFC"; level="(0,*,*)";'
>
>
> And the resulting plot is attached.
>
>
> In all, there are 3 issues:
>
>
> (1) MET expects the time variable to be named "time". I'm not sure
whether
> this is actually a CF requirement. If not, please let me know, and
we can
> take a closer look at the CF convention to see what MET *should* be
doing.
>
>
> (2) The time "units" attribute must be named "[TIME UNITS] since
[REFERENCE
> TIME]". I'm pretty confident that this a CF requirement.
>
>
> (3) If this is analysis data, then we're all set. But for forecast
data,
> you'd need to define the forecast initialization time in the CF
compliant
> way. That's done by defining a variable named
"forecast_reference_time"...
> looking something like this:
>
>
> double forecast_reference_time ;
>
> forecast_reference_time:long_name = "reference time" ;
>
> forecast_reference_time:units = "seconds since 1970-01-01" ;
>
>
> Of course the units can be defined as seconds, minutes, hour, days,
and so
> on.
>
>
> Thanks,
> John
>
> On Wed, Jan 10, 2018 at 7:00 AM, Brad Diehl - NOAA Federal via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
> >
> > Once again, thanks for the help, I'll have to be more aware of the
> > dimensions moving forward. And the solution of using -field
> > 'name="QPF_SFC"; level="(0,*,*)";' worked just fine. Many thanks
again.
> >
> >
> > However, I continue to get the warning below from
regrid_data_plane. Not
> > sure if its a show stopper but does'nt appear to be. Is there
anything I
> > can do? If not, I'll consider this ticket closed.
> >
> > WARNING:
> > WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> > WARNING:
> >
> > On Tue, Jan 9, 2018 at 3:01 PM, John Halley Gotway via RT <
> > met_help at ucar.edu
> > > wrote:
> >
> > > Hello Brad,
> > >
> > > I see you're having trouble passing a NetCDF file to the
> > regrid_data_plane
> > > tool. The problem is how you've specified the "level" value in
the
> > command
> > > line argument. You specified "(*,*)" (i.e. 2 dimensions) but
that
> > variable
> > > actually has 3 dimensions:
> > > QPF_SFC(ProjectionHr, YCells, XCells)
> > >
> > > That's what this error message is telling you:
> > > ERROR : NcCfFile::getData(NcVar *, const LongArray &,
DataPlane &)
> ->
> > > needed 3 arguments for variable QPF_SFC, got 2
> > >
> > > Please try using this:
> > > -field 'name="QPF_SFC"; level="(0,*,*)";'
> > >
> > > That'll process the first 2D slice of data from that 3D
variable.
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Jan 8, 2018 at 10:00 AM, Brad Diehl - NOAA Federal via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Mon Jan 08 10:00:01 2018: Request 83514 was acted upon.
> > > > Transaction: Ticket created by brad.diehl at noaa.gov
> > > > Queue: met_help
> > > > Subject: RFC netcdf input into regrid_data_plane
> > > > Owner: Nobody
> > > > Requestors: brad.diehl at noaa.gov
> > > > Status: new
> > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=83514
> > >
> > > >
> > > >
> > > > To whom it may concern,
> > > >
> > > > I have been working with ABRFC Mosaic QPF in netcdf format and
using
> it
> > > as
> > > > input into regrid_data_plane. Using the simple script below, I
get
> > > > following results (pasted below script). I was'nt sure how
severe the
> > > > ERRORS were and wanted to double check with the help desk. I
did
> notice
> > > in
> > > > the MET FAQ that to "use gridded NetCDF files in MET, the
files need
> to
> > > > look like the output of the pcp_combine tool" with some
requirements
> > > > listed.
> > > >
> > > > I have sent the script and input files via the anon ftp site
in
> > > diehl_data.
> > > > The script name is rfc_test.sh, input files are
> rfc_2018010512f006.nc
> > > and
> > > > urma_co.
> > > >
> > > > NOTE: I get the following from ncdump:
> > > >
> > > >
> > > > float QPF_SFC(ProjectionHr, YCells, XCells) ;
> > > > QPF_SFC:long_name = "Total precipitation" ;
> > > > QPF_SFC:units = "inch" ;
> > > > QPF_SFC:coordinates = "longitude latitude" ;
> > > > QPF_SFC:ReferenceTime = 1515153600 ;
> > > > QPF_SFC:ReferenceTimeString = "2018-01-05 12:00:00
00:00" ;
> > > > QPF_SFC:gridType = "SCALAR" ;
> > > > QPF_SFC:level = "SFC" ;
> > > >
> > > > Note that there is no name defined, only long_name. I assume
name is
> > > > QPF_SFC.
> > > >
> > > >
> > > >
> > > > Thanks in advance for your help....Brad Diehl
> > > >
> > > > #!/bin/sh
> > > > ###
> > > > ############
> > > >
> > > > source /u/Brad.Diehl/met_env/tut_env
> > > >
> > > > regrid_data_plane rfc_2018010512f006.nc \
> > > > urma_co \
> > > > re_grid_out.nc \
> > > > -method BUDGET \
> > > > -width 2 \
> > > > -v 2 \
> > > > -field 'name="QPF_SFC"; level="(*,*)";'
> > > >
> > > >
> > > >
> > > > DEBUG 1: Reading data file: rfc_2018010512f006.nc
> > > > WARNING:
> > > > WARNING: NcCfFile::open() -> could not determine the valid
time,
> using
> > 0.
> > > > WARNING:
> > > > DEBUG 2: Input grid: Projection: Lambert Conformal Nx: 2145
Ny: 1377
> > > > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000 Alpha:
6509.082
> Cone:
> > > > 0.423 Bx: 1088.0030 By: 5483.6617
> > > > DEBUG 2: Output grid: Projection: Lambert Conformal Nx: 2145
Ny: 1377
> > > > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000 Alpha:
6509.082
> Cone:
> > > > 0.423 Bx: 1088.0030 By: 5483.6617
> > > > DEBUG 2: Interpolation options: method = BUDGET, width = 2,
> vld_thresh
> > =
> > > > 0.5
> > > > ERROR :
> > > > ERROR : NcCfFile::getData(NcVar *, const LongArray &,
DataPlane &)
> ->
> > > > needed 3 arguments for variable QPF_SFC, got 2
> > > > ERROR :
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: RFC netcdf input into regrid_data_plane
From: Brad Diehl - NOAA Federal
Time: Thu Jan 11 08:51:20 2018
Using the recommended NCO utilities, I did the following two steps as
suggested:
ncrename -v ProjectionHr,time -d ProjectionHr,time -o
rfc_2018010512f006_MOD.nc rfc_2018010512f006.nc
ncatted -a units,time,o,c,"hours since 2018-01-05 12:00:00"
rfc_2018010512f006_MOD.nc
And MET/regrid_data_plane ran without warning.
For forecast data, the following was recommended:
(3) If this is analysis data, then we're all set. But for forecast
data,
you'd need to define the forecast initialization time in the CF
compliant
way. That's done by defining a variable named
"forecast_reference_time"...
looking something like this:
double forecast_reference_time ;
forecast_reference_time:long_name = "reference time" ;
forecast_reference_time:units = "seconds since 1970-01-01" ;
However, if I look at forecast data via ncdump that we've processed
through
regrid_data_plane, I never see forecast_reference_time referred to.
For
example, see below for hrrr (gfs had same result):
Brad.Diehl at g14a1:
/mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
ncdump hrrrconc2p518010512qpf06f006 | grep time
APCP_06:init_time = "20180105_120000" ;
APCP_06:init_time_ut = "1515153600" ;
APCP_06:valid_time = "20180105_180000" ;
APCP_06:valid_time_ut = "1515175200" ;
APCP_06:accum_time = "060000" ;
APCP_06:accum_time_sec = 21600 ;
Brad.Diehl at g14a1:
/mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
ncdump hrrrconc2p518010512qpf06f006 | grep forecast
Brad.Diehl at g14a1:
/mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
So, not sure how to proceed. Any additional help would be
appreciated........Brad Diehl
On Wed, Jan 10, 2018 at 5:22 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Brad,
>
> I took a look at your data. This file is a NetCDF file which
purportedly
> follows the CF convention. However, MET is not correctly parsing
the
> timing info. Running though the debugger, I find that it's
expecting the
> timing variable to simply be named "time". To test this out, I used
the
> ncrename utility to rename the "ProjectionHr" variable and dimension
to
> "time":
>
> ncrename -v ProjectionHr,time -d ProjectionHr,time -o
> rfc_2018010512f006_MOD.nc rfc_2018010512f006.nc
>
>
> Rerunning reveals the next warning:
>
>
> WARNING: parse_cf_time_string() -> unexpected NetCDF CF convention
time
> unit "2018-01-05 12:00:00 00:00"
>
>
> For the CF convention, the "units" attribute for time should look
like this
> "[TIME UNITS] since [REFERENCE TIME]". To test, I used the ncatted
utility
> to modify your sample file:
>
>
> ncatted -a units,time,o,c,"hours since 2018-01-05 12:00:00"
> rfc_2018010512f006_MOD.nc
>
>
> And finally that allows MET run without warning:
>
>
> met-6.1/bin/plot_data_plane rfc_2018010512f006_MOD.nc QPF_SFC.ps
> 'name="QPF_SFC"; level="(0,*,*)";'
>
>
> And the resulting plot is attached.
>
>
> In all, there are 3 issues:
>
>
> (1) MET expects the time variable to be named "time". I'm not sure
whether
> this is actually a CF requirement. If not, please let me know, and
we can
> take a closer look at the CF convention to see what MET *should* be
doing.
>
>
> (2) The time "units" attribute must be named "[TIME UNITS] since
[REFERENCE
> TIME]". I'm pretty confident that this a CF requirement.
>
>
> (3) If this is analysis data, then we're all set. But for forecast
data,
> you'd need to define the forecast initialization time in the CF
compliant
> way. That's done by defining a variable named
"forecast_reference_time"...
> looking something like this:
>
>
> double forecast_reference_time ;
>
> forecast_reference_time:long_name = "reference time" ;
>
> forecast_reference_time:units = "seconds since 1970-01-01" ;
>
>
> Of course the units can be defined as seconds, minutes, hour, days,
and so
> on.
>
>
> Thanks,
> John
>
> On Wed, Jan 10, 2018 at 7:00 AM, Brad Diehl - NOAA Federal via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
> >
> > Once again, thanks for the help, I'll have to be more aware of the
> > dimensions moving forward. And the solution of using -field
> > 'name="QPF_SFC"; level="(0,*,*)";' worked just fine. Many thanks
again.
> >
> >
> > However, I continue to get the warning below from
regrid_data_plane. Not
> > sure if its a show stopper but does'nt appear to be. Is there
anything I
> > can do? If not, I'll consider this ticket closed.
> >
> > WARNING:
> > WARNING: NcCfFile::open() -> could not determine the valid time,
using 0.
> > WARNING:
> >
> > On Tue, Jan 9, 2018 at 3:01 PM, John Halley Gotway via RT <
> > met_help at ucar.edu
> > > wrote:
> >
> > > Hello Brad,
> > >
> > > I see you're having trouble passing a NetCDF file to the
> > regrid_data_plane
> > > tool. The problem is how you've specified the "level" value in
the
> > command
> > > line argument. You specified "(*,*)" (i.e. 2 dimensions) but
that
> > variable
> > > actually has 3 dimensions:
> > > QPF_SFC(ProjectionHr, YCells, XCells)
> > >
> > > That's what this error message is telling you:
> > > ERROR : NcCfFile::getData(NcVar *, const LongArray &,
DataPlane &)
> ->
> > > needed 3 arguments for variable QPF_SFC, got 2
> > >
> > > Please try using this:
> > > -field 'name="QPF_SFC"; level="(0,*,*)";'
> > >
> > > That'll process the first 2D slice of data from that 3D
variable.
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Jan 8, 2018 at 10:00 AM, Brad Diehl - NOAA Federal via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Mon Jan 08 10:00:01 2018: Request 83514 was acted upon.
> > > > Transaction: Ticket created by brad.diehl at noaa.gov
> > > > Queue: met_help
> > > > Subject: RFC netcdf input into regrid_data_plane
> > > > Owner: Nobody
> > > > Requestors: brad.diehl at noaa.gov
> > > > Status: new
> > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=83514
> > >
> > > >
> > > >
> > > > To whom it may concern,
> > > >
> > > > I have been working with ABRFC Mosaic QPF in netcdf format and
using
> it
> > > as
> > > > input into regrid_data_plane. Using the simple script below, I
get
> > > > following results (pasted below script). I was'nt sure how
severe the
> > > > ERRORS were and wanted to double check with the help desk. I
did
> notice
> > > in
> > > > the MET FAQ that to "use gridded NetCDF files in MET, the
files need
> to
> > > > look like the output of the pcp_combine tool" with some
requirements
> > > > listed.
> > > >
> > > > I have sent the script and input files via the anon ftp site
in
> > > diehl_data.
> > > > The script name is rfc_test.sh, input files are
> rfc_2018010512f006.nc
> > > and
> > > > urma_co.
> > > >
> > > > NOTE: I get the following from ncdump:
> > > >
> > > >
> > > > float QPF_SFC(ProjectionHr, YCells, XCells) ;
> > > > QPF_SFC:long_name = "Total precipitation" ;
> > > > QPF_SFC:units = "inch" ;
> > > > QPF_SFC:coordinates = "longitude latitude" ;
> > > > QPF_SFC:ReferenceTime = 1515153600 ;
> > > > QPF_SFC:ReferenceTimeString = "2018-01-05 12:00:00
00:00" ;
> > > > QPF_SFC:gridType = "SCALAR" ;
> > > > QPF_SFC:level = "SFC" ;
> > > >
> > > > Note that there is no name defined, only long_name. I assume
name is
> > > > QPF_SFC.
> > > >
> > > >
> > > >
> > > > Thanks in advance for your help....Brad Diehl
> > > >
> > > > #!/bin/sh
> > > > ###
> > > > ############
> > > >
> > > > source /u/Brad.Diehl/met_env/tut_env
> > > >
> > > > regrid_data_plane rfc_2018010512f006.nc \
> > > > urma_co \
> > > > re_grid_out.nc \
> > > > -method BUDGET \
> > > > -width 2 \
> > > > -v 2 \
> > > > -field 'name="QPF_SFC"; level="(*,*)";'
> > > >
> > > >
> > > >
> > > > DEBUG 1: Reading data file: rfc_2018010512f006.nc
> > > > WARNING:
> > > > WARNING: NcCfFile::open() -> could not determine the valid
time,
> using
> > 0.
> > > > WARNING:
> > > > DEBUG 2: Input grid: Projection: Lambert Conformal Nx: 2145
Ny: 1377
> > > > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000 Alpha:
6509.082
> Cone:
> > > > 0.423 Bx: 1088.0030 By: 5483.6617
> > > > DEBUG 2: Output grid: Projection: Lambert Conformal Nx: 2145
Ny: 1377
> > > > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000 Alpha:
6509.082
> Cone:
> > > > 0.423 Bx: 1088.0030 By: 5483.6617
> > > > DEBUG 2: Interpolation options: method = BUDGET, width = 2,
> vld_thresh
> > =
> > > > 0.5
> > > > ERROR :
> > > > ERROR : NcCfFile::getData(NcVar *, const LongArray &,
DataPlane &)
> ->
> > > > needed 3 arguments for variable QPF_SFC, got 2
> > > > ERROR :
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: RFC netcdf input into regrid_data_plane
From: John Halley Gotway
Time: Thu Jan 11 10:56:21 2018
Brad,
The NetCDF input you’re passing to regrid data plane is cf-compliant
but
the NetCDF output from it is not.
We do plan to eventually change the output format to be cg compliant
but it
currently isn’t.
If you rerun regrid data plane with a higher verbosity level (-v 4 or
-v 5
maybe), you should see a DEBUG message telling you that it can’t
determine
the init time from the input netcdf file... so it just sets the init
and
valid times to the same time.
The cf way of defining the init time is adding the
forecast_reference_time
variable to define it. Unfortunately I don’t have a quick NCO command
to
add a new variable.
But I could use ncdump/ncgen to do it for our sample case if that
would be
helpful.
Thanks
John
On Thu, Jan 11, 2018 at 9:51 AM Brad Diehl - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
>
> Using the recommended NCO utilities, I did the following two steps
as
> suggested:
>
> ncrename -v ProjectionHr,time -d ProjectionHr,time -o
> rfc_2018010512f006_MOD.nc rfc_2018010512f006.nc
>
> ncatted -a units,time,o,c,"hours since 2018-01-05 12:00:00"
> rfc_2018010512f006_MOD.nc
>
> And MET/regrid_data_plane ran without warning.
>
> For forecast data, the following was recommended:
>
> (3) If this is analysis data, then we're all set. But for forecast
data,
> you'd need to define the forecast initialization time in the CF
compliant
> way. That's done by defining a variable named
"forecast_reference_time"...
> looking something like this:
>
>
> double forecast_reference_time ;
>
> forecast_reference_time:long_name = "reference time" ;
>
> forecast_reference_time:units = "seconds since 1970-01-01" ;
>
> However, if I look at forecast data via ncdump that we've processed
through
> regrid_data_plane, I never see forecast_reference_time referred to.
For
> example, see below for hrrr (gfs had same result):
>
> Brad.Diehl at g14a1:
> /mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
> ncdump hrrrconc2p518010512qpf06f006 | grep time
> APCP_06:init_time = "20180105_120000" ;
> APCP_06:init_time_ut = "1515153600" ;
> APCP_06:valid_time = "20180105_180000" ;
> APCP_06:valid_time_ut = "1515175200" ;
> APCP_06:accum_time = "060000" ;
> APCP_06:accum_time_sec = 21600 ;
>
> Brad.Diehl at g14a1:
> /mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
> ncdump hrrrconc2p518010512qpf06f006 | grep forecast
> Brad.Diehl at g14a1:
> /mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
>
> So, not sure how to proceed. Any additional help would be
> appreciated........Brad Diehl
>
>
> On Wed, Jan 10, 2018 at 5:22 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Brad,
> >
> > I took a look at your data. This file is a NetCDF file which
purportedly
> > follows the CF convention. However, MET is not correctly parsing
the
> > timing info. Running though the debugger, I find that it's
expecting the
> > timing variable to simply be named "time". To test this out, I
used the
> > ncrename utility to rename the "ProjectionHr" variable and
dimension to
> > "time":
> >
> > ncrename -v ProjectionHr,time -d ProjectionHr,time -o
> > rfc_2018010512f006_MOD.nc rfc_2018010512f006.nc
> >
> >
> > Rerunning reveals the next warning:
> >
> >
> > WARNING: parse_cf_time_string() -> unexpected NetCDF CF convention
time
> > unit "2018-01-05 12:00:00 00:00"
> >
> >
> > For the CF convention, the "units" attribute for time should look
like
> this
> > "[TIME UNITS] since [REFERENCE TIME]". To test, I used the
ncatted
> utility
> > to modify your sample file:
> >
> >
> > ncatted -a units,time,o,c,"hours since 2018-01-05 12:00:00"
> > rfc_2018010512f006_MOD.nc
> >
> >
> > And finally that allows MET run without warning:
> >
> >
> > met-6.1/bin/plot_data_plane rfc_2018010512f006_MOD.nc QPF_SFC.ps
> > 'name="QPF_SFC"; level="(0,*,*)";'
> >
> >
> > And the resulting plot is attached.
> >
> >
> > In all, there are 3 issues:
> >
> >
> > (1) MET expects the time variable to be named "time". I'm not
sure
> whether
> > this is actually a CF requirement. If not, please let me know,
and we
> can
> > take a closer look at the CF convention to see what MET *should*
be
> doing.
> >
> >
> > (2) The time "units" attribute must be named "[TIME UNITS] since
> [REFERENCE
> > TIME]". I'm pretty confident that this a CF requirement.
> >
> >
> > (3) If this is analysis data, then we're all set. But for
forecast data,
> > you'd need to define the forecast initialization time in the CF
compliant
> > way. That's done by defining a variable named
> "forecast_reference_time"...
> > looking something like this:
> >
> >
> > double forecast_reference_time ;
> >
> > forecast_reference_time:long_name = "reference time" ;
> >
> > forecast_reference_time:units = "seconds since 1970-01-01" ;
> >
> >
> > Of course the units can be defined as seconds, minutes, hour,
days, and
> so
> > on.
> >
> >
> > Thanks,
> > John
> >
> > On Wed, Jan 10, 2018 at 7:00 AM, Brad Diehl - NOAA Federal via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
> > >
> > > Once again, thanks for the help, I'll have to be more aware of
the
> > > dimensions moving forward. And the solution of using -field
> > > 'name="QPF_SFC"; level="(0,*,*)";' worked just fine. Many thanks
again.
> > >
> > >
> > > However, I continue to get the warning below from
regrid_data_plane.
> Not
> > > sure if its a show stopper but does'nt appear to be. Is there
anything
> I
> > > can do? If not, I'll consider this ticket closed.
> > >
> > > WARNING:
> > > WARNING: NcCfFile::open() -> could not determine the valid time,
using
> 0.
> > > WARNING:
> > >
> > > On Tue, Jan 9, 2018 at 3:01 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu
> > > > wrote:
> > >
> > > > Hello Brad,
> > > >
> > > > I see you're having trouble passing a NetCDF file to the
> > > regrid_data_plane
> > > > tool. The problem is how you've specified the "level" value
in the
> > > command
> > > > line argument. You specified "(*,*)" (i.e. 2 dimensions) but
that
> > > variable
> > > > actually has 3 dimensions:
> > > > QPF_SFC(ProjectionHr, YCells, XCells)
> > > >
> > > > That's what this error message is telling you:
> > > > ERROR : NcCfFile::getData(NcVar *, const LongArray &,
DataPlane
> &)
> > ->
> > > > needed 3 arguments for variable QPF_SFC, got 2
> > > >
> > > > Please try using this:
> > > > -field 'name="QPF_SFC"; level="(0,*,*)";'
> > > >
> > > > That'll process the first 2D slice of data from that 3D
variable.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Mon, Jan 8, 2018 at 10:00 AM, Brad Diehl - NOAA Federal via
RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Mon Jan 08 10:00:01 2018: Request 83514 was acted upon.
> > > > > Transaction: Ticket created by brad.diehl at noaa.gov
> > > > > Queue: met_help
> > > > > Subject: RFC netcdf input into regrid_data_plane
> > > > > Owner: Nobody
> > > > > Requestors: brad.diehl at noaa.gov
> > > > > Status: new
> > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> > Ticket/Display.html?id=83514
> > > >
> > > > >
> > > > >
> > > > > To whom it may concern,
> > > > >
> > > > > I have been working with ABRFC Mosaic QPF in netcdf format
and
> using
> > it
> > > > as
> > > > > input into regrid_data_plane. Using the simple script below,
I get
> > > > > following results (pasted below script). I was'nt sure how
severe
> the
> > > > > ERRORS were and wanted to double check with the help desk. I
did
> > notice
> > > > in
> > > > > the MET FAQ that to "use gridded NetCDF files in MET, the
files
> need
> > to
> > > > > look like the output of the pcp_combine tool" with some
> requirements
> > > > > listed.
> > > > >
> > > > > I have sent the script and input files via the anon ftp site
in
> > > > diehl_data.
> > > > > The script name is rfc_test.sh, input files are
> > rfc_2018010512f006.nc
> > > > and
> > > > > urma_co.
> > > > >
> > > > > NOTE: I get the following from ncdump:
> > > > >
> > > > >
> > > > > float QPF_SFC(ProjectionHr, YCells, XCells) ;
> > > > > QPF_SFC:long_name = "Total precipitation" ;
> > > > > QPF_SFC:units = "inch" ;
> > > > > QPF_SFC:coordinates = "longitude latitude" ;
> > > > > QPF_SFC:ReferenceTime = 1515153600 ;
> > > > > QPF_SFC:ReferenceTimeString = "2018-01-05 12:00:00
00:00" ;
> > > > > QPF_SFC:gridType = "SCALAR" ;
> > > > > QPF_SFC:level = "SFC" ;
> > > > >
> > > > > Note that there is no name defined, only long_name. I assume
name
> is
> > > > > QPF_SFC.
> > > > >
> > > > >
> > > > >
> > > > > Thanks in advance for your help....Brad Diehl
> > > > >
> > > > > #!/bin/sh
> > > > > ###
> > > > > ############
> > > > >
> > > > > source /u/Brad.Diehl/met_env/tut_env
> > > > >
> > > > > regrid_data_plane rfc_2018010512f006.nc \
> > > > > urma_co \
> > > > > re_grid_out.nc \
> > > > > -method BUDGET \
> > > > > -width 2 \
> > > > > -v 2 \
> > > > > -field 'name="QPF_SFC"; level="(*,*)";'
> > > > >
> > > > >
> > > > >
> > > > > DEBUG 1: Reading data file: rfc_2018010512f006.nc
> > > > > WARNING:
> > > > > WARNING: NcCfFile::open() -> could not determine the valid
time,
> > using
> > > 0.
> > > > > WARNING:
> > > > > DEBUG 2: Input grid: Projection: Lambert Conformal Nx: 2145
Ny:
> 1377
> > > > > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000 Alpha:
6509.082
> > Cone:
> > > > > 0.423 Bx: 1088.0030 By: 5483.6617
> > > > > DEBUG 2: Output grid: Projection: Lambert Conformal Nx: 2145
Ny:
> 1377
> > > > > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000 Alpha:
6509.082
> > Cone:
> > > > > 0.423 Bx: 1088.0030 By: 5483.6617
> > > > > DEBUG 2: Interpolation options: method = BUDGET, width = 2,
> > vld_thresh
> > > =
> > > > > 0.5
> > > > > ERROR :
> > > > > ERROR : NcCfFile::getData(NcVar *, const LongArray &,
DataPlane &)
> > ->
> > > > > needed 3 arguments for variable QPF_SFC, got 2
> > > > > ERROR :
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: RFC netcdf input into regrid_data_plane
From: Brad Diehl - NOAA Federal
Time: Thu Jan 11 11:22:43 2018
I was going to ask the netcdf guru here how to go about it, but if you
know
what to do and have the time sure use ncdump/ncgen to do it for our
sample
case.
BTW, one of our developers is having some issues with trying to
convert the
rfc data into geotiff files, he has done it successfully with all the
other
models.
He chatted:
and so my variable that I'm trying to read
"*QPF_SFC_0\,\*\,\*"?*
"*QPF_SFC_0\,\*\,\*"*
is coming out of MET
Can you send them a note and ask them why the variable name is such?
I know why it is, I think it's not parsing things properly
But I can't read the variable
If he get s.t. to work I'll let you know ASAP.
On Thu, Jan 11, 2018 at 5:56 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Brad,
>
> The NetCDF input you’re passing to regrid data plane is cf-compliant
but
> the NetCDF output from it is not.
>
> We do plan to eventually change the output format to be cg compliant
but it
> currently isn’t.
>
> If you rerun regrid data plane with a higher verbosity level (-v 4
or -v 5
> maybe), you should see a DEBUG message telling you that it can’t
determine
> the init time from the input netcdf file... so it just sets the init
and
> valid times to the same time.
>
> The cf way of defining the init time is adding the
forecast_reference_time
> variable to define it. Unfortunately I don’t have a quick NCO
command to
> add a new variable.
>
> But I could use ncdump/ncgen to do it for our sample case if that
would be
> helpful.
>
> Thanks
> John
>
> On Thu, Jan 11, 2018 at 9:51 AM Brad Diehl - NOAA Federal via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
> >
> > Using the recommended NCO utilities, I did the following two steps
as
> > suggested:
> >
> > ncrename -v ProjectionHr,time -d ProjectionHr,time -o
> > rfc_2018010512f006_MOD.nc rfc_2018010512f006.nc
> >
> > ncatted -a units,time,o,c,"hours since 2018-01-05 12:00:00"
> > rfc_2018010512f006_MOD.nc
> >
> > And MET/regrid_data_plane ran without warning.
> >
> > For forecast data, the following was recommended:
> >
> > (3) If this is analysis data, then we're all set. But for
forecast data,
> > you'd need to define the forecast initialization time in the CF
compliant
> > way. That's done by defining a variable named
> "forecast_reference_time"...
> > looking something like this:
> >
> >
> > double forecast_reference_time ;
> >
> > forecast_reference_time:long_name = "reference time" ;
> >
> > forecast_reference_time:units = "seconds since 1970-01-01" ;
> >
> > However, if I look at forecast data via ncdump that we've
processed
> through
> > regrid_data_plane, I never see forecast_reference_time referred
to. For
> > example, see below for hrrr (gfs had same result):
> >
> > Brad.Diehl at g14a1:
> > /mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
> > ncdump hrrrconc2p518010512qpf06f006 | grep time
> > APCP_06:init_time = "20180105_120000" ;
> > APCP_06:init_time_ut = "1515153600" ;
> > APCP_06:valid_time = "20180105_180000" ;
> > APCP_06:valid_time_ut = "1515175200" ;
> > APCP_06:accum_time = "060000" ;
> > APCP_06:accum_time_sec = 21600 ;
> >
> > Brad.Diehl at g14a1:
> > /mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
> > ncdump hrrrconc2p518010512qpf06f006 | grep forecast
> > Brad.Diehl at g14a1:
> > /mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
> >
> > So, not sure how to proceed. Any additional help would be
> > appreciated........Brad Diehl
> >
> >
> > On Wed, Jan 10, 2018 at 5:22 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Brad,
> > >
> > > I took a look at your data. This file is a NetCDF file which
> purportedly
> > > follows the CF convention. However, MET is not correctly
parsing the
> > > timing info. Running though the debugger, I find that it's
expecting
> the
> > > timing variable to simply be named "time". To test this out, I
used
> the
> > > ncrename utility to rename the "ProjectionHr" variable and
dimension to
> > > "time":
> > >
> > > ncrename -v ProjectionHr,time -d ProjectionHr,time -o
> > > rfc_2018010512f006_MOD.nc rfc_2018010512f006.nc
> > >
> > >
> > > Rerunning reveals the next warning:
> > >
> > >
> > > WARNING: parse_cf_time_string() -> unexpected NetCDF CF
convention time
> > > unit "2018-01-05 12:00:00 00:00"
> > >
> > >
> > > For the CF convention, the "units" attribute for time should
look like
> > this
> > > "[TIME UNITS] since [REFERENCE TIME]". To test, I used the
ncatted
> > utility
> > > to modify your sample file:
> > >
> > >
> > > ncatted -a units,time,o,c,"hours since 2018-01-05 12:00:00"
> > > rfc_2018010512f006_MOD.nc
> > >
> > >
> > > And finally that allows MET run without warning:
> > >
> > >
> > > met-6.1/bin/plot_data_plane rfc_2018010512f006_MOD.nc QPF_SFC.ps
> > > 'name="QPF_SFC"; level="(0,*,*)";'
> > >
> > >
> > > And the resulting plot is attached.
> > >
> > >
> > > In all, there are 3 issues:
> > >
> > >
> > > (1) MET expects the time variable to be named "time". I'm not
sure
> > whether
> > > this is actually a CF requirement. If not, please let me know,
and we
> > can
> > > take a closer look at the CF convention to see what MET *should*
be
> > doing.
> > >
> > >
> > > (2) The time "units" attribute must be named "[TIME UNITS] since
> > [REFERENCE
> > > TIME]". I'm pretty confident that this a CF requirement.
> > >
> > >
> > > (3) If this is analysis data, then we're all set. But for
forecast
> data,
> > > you'd need to define the forecast initialization time in the CF
> compliant
> > > way. That's done by defining a variable named
> > "forecast_reference_time"...
> > > looking something like this:
> > >
> > >
> > > double forecast_reference_time ;
> > >
> > > forecast_reference_time:long_name = "reference time" ;
> > >
> > > forecast_reference_time:units = "seconds since 1970-01-01" ;
> > >
> > >
> > > Of course the units can be defined as seconds, minutes, hour,
days, and
> > so
> > > on.
> > >
> > >
> > > Thanks,
> > > John
> > >
> > > On Wed, Jan 10, 2018 at 7:00 AM, Brad Diehl - NOAA Federal via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514
>
> > > >
> > > > Once again, thanks for the help, I'll have to be more aware of
the
> > > > dimensions moving forward. And the solution of using -field
> > > > 'name="QPF_SFC"; level="(0,*,*)";' worked just fine. Many
thanks
> again.
> > > >
> > > >
> > > > However, I continue to get the warning below from
regrid_data_plane.
> > Not
> > > > sure if its a show stopper but does'nt appear to be. Is there
> anything
> > I
> > > > can do? If not, I'll consider this ticket closed.
> > > >
> > > > WARNING:
> > > > WARNING: NcCfFile::open() -> could not determine the valid
time,
> using
> > 0.
> > > > WARNING:
> > > >
> > > > On Tue, Jan 9, 2018 at 3:01 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu
> > > > > wrote:
> > > >
> > > > > Hello Brad,
> > > > >
> > > > > I see you're having trouble passing a NetCDF file to the
> > > > regrid_data_plane
> > > > > tool. The problem is how you've specified the "level" value
in the
> > > > command
> > > > > line argument. You specified "(*,*)" (i.e. 2 dimensions)
but that
> > > > variable
> > > > > actually has 3 dimensions:
> > > > > QPF_SFC(ProjectionHr, YCells, XCells)
> > > > >
> > > > > That's what this error message is telling you:
> > > > > ERROR : NcCfFile::getData(NcVar *, const LongArray &,
DataPlane
> > &)
> > > ->
> > > > > needed 3 arguments for variable QPF_SFC, got 2
> > > > >
> > > > > Please try using this:
> > > > > -field 'name="QPF_SFC"; level="(0,*,*)";'
> > > > >
> > > > > That'll process the first 2D slice of data from that 3D
variable.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Mon, Jan 8, 2018 at 10:00 AM, Brad Diehl - NOAA Federal
via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Mon Jan 08 10:00:01 2018: Request 83514 was acted upon.
> > > > > > Transaction: Ticket created by brad.diehl at noaa.gov
> > > > > > Queue: met_help
> > > > > > Subject: RFC netcdf input into regrid_data_plane
> > > > > > Owner: Nobody
> > > > > > Requestors: brad.diehl at noaa.gov
> > > > > > Status: new
> > > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> > > Ticket/Display.html?id=83514
> > > > >
> > > > > >
> > > > > >
> > > > > > To whom it may concern,
> > > > > >
> > > > > > I have been working with ABRFC Mosaic QPF in netcdf format
and
> > using
> > > it
> > > > > as
> > > > > > input into regrid_data_plane. Using the simple script
below, I
> get
> > > > > > following results (pasted below script). I was'nt sure how
severe
> > the
> > > > > > ERRORS were and wanted to double check with the help desk.
I did
> > > notice
> > > > > in
> > > > > > the MET FAQ that to "use gridded NetCDF files in MET, the
files
> > need
> > > to
> > > > > > look like the output of the pcp_combine tool" with some
> > requirements
> > > > > > listed.
> > > > > >
> > > > > > I have sent the script and input files via the anon ftp
site in
> > > > > diehl_data.
> > > > > > The script name is rfc_test.sh, input files are
> > > rfc_2018010512f006.nc
> > > > > and
> > > > > > urma_co.
> > > > > >
> > > > > > NOTE: I get the following from ncdump:
> > > > > >
> > > > > >
> > > > > > float QPF_SFC(ProjectionHr, YCells, XCells) ;
> > > > > > QPF_SFC:long_name = "Total precipitation" ;
> > > > > > QPF_SFC:units = "inch" ;
> > > > > > QPF_SFC:coordinates = "longitude latitude" ;
> > > > > > QPF_SFC:ReferenceTime = 1515153600 ;
> > > > > > QPF_SFC:ReferenceTimeString = "2018-01-05 12:00:00
> 00:00" ;
> > > > > > QPF_SFC:gridType = "SCALAR" ;
> > > > > > QPF_SFC:level = "SFC" ;
> > > > > >
> > > > > > Note that there is no name defined, only long_name. I
assume name
> > is
> > > > > > QPF_SFC.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Thanks in advance for your help....Brad Diehl
> > > > > >
> > > > > > #!/bin/sh
> > > > > > ###
> > > > > > ############
> > > > > >
> > > > > > source /u/Brad.Diehl/met_env/tut_env
> > > > > >
> > > > > > regrid_data_plane rfc_2018010512f006.nc \
> > > > > > urma_co \
> > > > > > re_grid_out.nc \
> > > > > > -method BUDGET \
> > > > > > -width 2 \
> > > > > > -v 2 \
> > > > > > -field 'name="QPF_SFC"; level="(*,*)";'
> > > > > >
> > > > > >
> > > > > >
> > > > > > DEBUG 1: Reading data file: rfc_2018010512f006.nc
> > > > > > WARNING:
> > > > > > WARNING: NcCfFile::open() -> could not determine the valid
time,
> > > using
> > > > 0.
> > > > > > WARNING:
> > > > > > DEBUG 2: Input grid: Projection: Lambert Conformal Nx:
2145 Ny:
> > 1377
> > > > > > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000 Alpha:
6509.082
> > > Cone:
> > > > > > 0.423 Bx: 1088.0030 By: 5483.6617
> > > > > > DEBUG 2: Output grid: Projection: Lambert Conformal Nx:
2145 Ny:
> > 1377
> > > > > > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000 Alpha:
6509.082
> > > Cone:
> > > > > > 0.423 Bx: 1088.0030 By: 5483.6617
> > > > > > DEBUG 2: Interpolation options: method = BUDGET, width =
2,
> > > vld_thresh
> > > > =
> > > > > > 0.5
> > > > > > ERROR :
> > > > > > ERROR : NcCfFile::getData(NcVar *, const LongArray &,
DataPlane
> &)
> > > ->
> > > > > > needed 3 arguments for variable QPF_SFC, got 2
> > > > > > ERROR :
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: RFC netcdf input into regrid_data_plane
From: John Halley Gotway
Time: Thu Jan 11 15:26:40 2018
Brad,
OK, I ran the following steps...
(1) Run ncdump to dump data to ascii:
ncdump rfc_2018010512f006_MOD.nc > rfc_2018010512f006_MOD.ncdump
(2) Edit the ascii output by adding:
double forecast_reference_time ;
forecast_reference_time:long_name = "reference_time" ;
forecast_reference_time:units = "hours since 2018-01-
05
12:00:00" ;
...
forecast_reference_time = 0 ;
(3) Run ncgen to generate the NetCDF file from the modified ncdump
output:
ncgen rfc_2018010512f006_MOD.ncdump -o rfc_2018010512f006_MOD2.nc
(4) Run plot_data_plane again at verbosity level 4 (which lists the
timing
info MET parsed from the file):
met-6.1/bin/plot_data_plane rfc_2018010512f006_MOD.nc QPF_SFC.ps
'name="QPF_SFC"; level="(0,*,*)";' -v 4
DEBUG 4: Data plane information:
DEBUG 4: plane min: 0
DEBUG 4: plane max: 0.854724
DEBUG 4: valid time: 20180105_160000
DEBUG 4: lead time: 040000
DEBUG 4: init time: 20180105_120000
DEBUG 4: accum time: 000000
These steps demonstrate that adding the forecast_reference_time
variable
does define the model initialization time for MET. While using
ncdump/ncgen is fine for testing, it's not good for systematic use.
Instead, I'd suggest talking to your NetCDF guy about preferred
methods for
updating the contents of your NetCDF files.
I copied all of these files back up to the ftp site:
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/diehl_data/for_brad
Yes, the regrid_data_plane tool is writing an output variable named
"QPF_SFC_0\,\*\,\*".
Those embedded commas are a pain... and in fact, we've updated MET 6.1
to
replace all instances of commas with underscores. Also note that you
do
have control over the output variable name from regrid_data_plane.
The
"-name" command line argument is used to name that variable however
you'd
like. Feel free to name it whatever is convenient for downstream
applications.
Hope that helps.
John
On Thu, Jan 11, 2018 at 11:22 AM, Brad Diehl - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
>
> I was going to ask the netcdf guru here how to go about it, but if
you know
> what to do and have the time sure use ncdump/ncgen to do it for our
sample
> case.
>
> BTW, one of our developers is having some issues with trying to
convert the
> rfc data into geotiff files, he has done it successfully with all
the other
> models.
>
> He chatted:
>
> and so my variable that I'm trying to read
> "*QPF_SFC_0\,\*\,\*"?*
> "*QPF_SFC_0\,\*\,\*"*
> is coming out of MET
>
> Can you send them a note and ask them why the variable name is such?
> I know why it is, I think it's not parsing things properly
> But I can't read the variable
>
> If he get s.t. to work I'll let you know ASAP.
>
> On Thu, Jan 11, 2018 at 5:56 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Brad,
> >
> > The NetCDF input you’re passing to regrid data plane is cf-
compliant but
> > the NetCDF output from it is not.
> >
> > We do plan to eventually change the output format to be cg
compliant but
> it
> > currently isn’t.
> >
> > If you rerun regrid data plane with a higher verbosity level (-v 4
or -v
> 5
> > maybe), you should see a DEBUG message telling you that it can’t
> determine
> > the init time from the input netcdf file... so it just sets the
init and
> > valid times to the same time.
> >
> > The cf way of defining the init time is adding the
> forecast_reference_time
> > variable to define it. Unfortunately I don’t have a quick NCO
command to
> > add a new variable.
> >
> > But I could use ncdump/ncgen to do it for our sample case if that
would
> be
> > helpful.
> >
> > Thanks
> > John
> >
> > On Thu, Jan 11, 2018 at 9:51 AM Brad Diehl - NOAA Federal via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
> > >
> > > Using the recommended NCO utilities, I did the following two
steps as
> > > suggested:
> > >
> > > ncrename -v ProjectionHr,time -d ProjectionHr,time -o
> > > rfc_2018010512f006_MOD.nc rfc_2018010512f006.nc
> > >
> > > ncatted -a units,time,o,c,"hours since 2018-01-05 12:00:00"
> > > rfc_2018010512f006_MOD.nc
> > >
> > > And MET/regrid_data_plane ran without warning.
> > >
> > > For forecast data, the following was recommended:
> > >
> > > (3) If this is analysis data, then we're all set. But for
forecast
> data,
> > > you'd need to define the forecast initialization time in the CF
> compliant
> > > way. That's done by defining a variable named
> > "forecast_reference_time"...
> > > looking something like this:
> > >
> > >
> > > double forecast_reference_time ;
> > >
> > > forecast_reference_time:long_name = "reference time" ;
> > >
> > > forecast_reference_time:units = "seconds since 1970-01-01" ;
> > >
> > > However, if I look at forecast data via ncdump that we've
processed
> > through
> > > regrid_data_plane, I never see forecast_reference_time referred
to. For
> > > example, see below for hrrr (gfs had same result):
> > >
> > > Brad.Diehl at g14a1:
> > > /mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
> > > ncdump hrrrconc2p518010512qpf06f006 | grep time
> > > APCP_06:init_time = "20180105_120000" ;
> > > APCP_06:init_time_ut = "1515153600" ;
> > > APCP_06:valid_time = "20180105_180000" ;
> > > APCP_06:valid_time_ut = "1515175200" ;
> > > APCP_06:accum_time = "060000" ;
> > > APCP_06:accum_time_sec = 21600 ;
> > >
> > > Brad.Diehl at g14a1:
> > > /mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
> > > ncdump hrrrconc2p518010512qpf06f006 | grep forecast
> > > Brad.Diehl at g14a1:
> > > /mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
> > >
> > > So, not sure how to proceed. Any additional help would be
> > > appreciated........Brad Diehl
> > >
> > >
> > > On Wed, Jan 10, 2018 at 5:22 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Brad,
> > > >
> > > > I took a look at your data. This file is a NetCDF file which
> > purportedly
> > > > follows the CF convention. However, MET is not correctly
parsing the
> > > > timing info. Running though the debugger, I find that it's
expecting
> > the
> > > > timing variable to simply be named "time". To test this out,
I used
> > the
> > > > ncrename utility to rename the "ProjectionHr" variable and
dimension
> to
> > > > "time":
> > > >
> > > > ncrename -v ProjectionHr,time -d ProjectionHr,time -o
> > > > rfc_2018010512f006_MOD.nc rfc_2018010512f006.nc
> > > >
> > > >
> > > > Rerunning reveals the next warning:
> > > >
> > > >
> > > > WARNING: parse_cf_time_string() -> unexpected NetCDF CF
convention
> time
> > > > unit "2018-01-05 12:00:00 00:00"
> > > >
> > > >
> > > > For the CF convention, the "units" attribute for time should
look
> like
> > > this
> > > > "[TIME UNITS] since [REFERENCE TIME]". To test, I used the
ncatted
> > > utility
> > > > to modify your sample file:
> > > >
> > > >
> > > > ncatted -a units,time,o,c,"hours since 2018-01-05 12:00:00"
> > > > rfc_2018010512f006_MOD.nc
> > > >
> > > >
> > > > And finally that allows MET run without warning:
> > > >
> > > >
> > > > met-6.1/bin/plot_data_plane rfc_2018010512f006_MOD.nc
QPF_SFC.ps
> > > > 'name="QPF_SFC"; level="(0,*,*)";'
> > > >
> > > >
> > > > And the resulting plot is attached.
> > > >
> > > >
> > > > In all, there are 3 issues:
> > > >
> > > >
> > > > (1) MET expects the time variable to be named "time". I'm not
sure
> > > whether
> > > > this is actually a CF requirement. If not, please let me
know, and
> we
> > > can
> > > > take a closer look at the CF convention to see what MET
*should* be
> > > doing.
> > > >
> > > >
> > > > (2) The time "units" attribute must be named "[TIME UNITS]
since
> > > [REFERENCE
> > > > TIME]". I'm pretty confident that this a CF requirement.
> > > >
> > > >
> > > > (3) If this is analysis data, then we're all set. But for
forecast
> > data,
> > > > you'd need to define the forecast initialization time in the
CF
> > compliant
> > > > way. That's done by defining a variable named
> > > "forecast_reference_time"...
> > > > looking something like this:
> > > >
> > > >
> > > > double forecast_reference_time ;
> > > >
> > > > forecast_reference_time:long_name = "reference time" ;
> > > >
> > > > forecast_reference_time:units = "seconds since 1970-01-01" ;
> > > >
> > > >
> > > > Of course the units can be defined as seconds, minutes, hour,
days,
> and
> > > so
> > > > on.
> > > >
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Wed, Jan 10, 2018 at 7:00 AM, Brad Diehl - NOAA Federal via
RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
> > > > >
> > > > > Once again, thanks for the help, I'll have to be more aware
of the
> > > > > dimensions moving forward. And the solution of using -field
> > > > > 'name="QPF_SFC"; level="(0,*,*)";' worked just fine. Many
thanks
> > again.
> > > > >
> > > > >
> > > > > However, I continue to get the warning below from
> regrid_data_plane.
> > > Not
> > > > > sure if its a show stopper but does'nt appear to be. Is
there
> > anything
> > > I
> > > > > can do? If not, I'll consider this ticket closed.
> > > > >
> > > > > WARNING:
> > > > > WARNING: NcCfFile::open() -> could not determine the valid
time,
> > using
> > > 0.
> > > > > WARNING:
> > > > >
> > > > > On Tue, Jan 9, 2018 at 3:01 PM, John Halley Gotway via RT <
> > > > > met_help at ucar.edu
> > > > > > wrote:
> > > > >
> > > > > > Hello Brad,
> > > > > >
> > > > > > I see you're having trouble passing a NetCDF file to the
> > > > > regrid_data_plane
> > > > > > tool. The problem is how you've specified the "level"
value in
> the
> > > > > command
> > > > > > line argument. You specified "(*,*)" (i.e. 2 dimensions)
but
> that
> > > > > variable
> > > > > > actually has 3 dimensions:
> > > > > > QPF_SFC(ProjectionHr, YCells, XCells)
> > > > > >
> > > > > > That's what this error message is telling you:
> > > > > > ERROR : NcCfFile::getData(NcVar *, const LongArray &,
> DataPlane
> > > &)
> > > > ->
> > > > > > needed 3 arguments for variable QPF_SFC, got 2
> > > > > >
> > > > > > Please try using this:
> > > > > > -field 'name="QPF_SFC"; level="(0,*,*)";'
> > > > > >
> > > > > > That'll process the first 2D slice of data from that 3D
variable.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Mon, Jan 8, 2018 at 10:00 AM, Brad Diehl - NOAA Federal
via
> RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > Mon Jan 08 10:00:01 2018: Request 83514 was acted upon.
> > > > > > > Transaction: Ticket created by brad.diehl at noaa.gov
> > > > > > > Queue: met_help
> > > > > > > Subject: RFC netcdf input into regrid_data_plane
> > > > > > > Owner: Nobody
> > > > > > > Requestors: brad.diehl at noaa.gov
> > > > > > > Status: new
> > > > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> > > > Ticket/Display.html?id=83514
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > To whom it may concern,
> > > > > > >
> > > > > > > I have been working with ABRFC Mosaic QPF in netcdf
format and
> > > using
> > > > it
> > > > > > as
> > > > > > > input into regrid_data_plane. Using the simple script
below, I
> > get
> > > > > > > following results (pasted below script). I was'nt sure
how
> severe
> > > the
> > > > > > > ERRORS were and wanted to double check with the help
desk. I
> did
> > > > notice
> > > > > > in
> > > > > > > the MET FAQ that to "use gridded NetCDF files in MET,
the files
> > > need
> > > > to
> > > > > > > look like the output of the pcp_combine tool" with some
> > > requirements
> > > > > > > listed.
> > > > > > >
> > > > > > > I have sent the script and input files via the anon ftp
site in
> > > > > > diehl_data.
> > > > > > > The script name is rfc_test.sh, input files are
> > > > rfc_2018010512f006.nc
> > > > > > and
> > > > > > > urma_co.
> > > > > > >
> > > > > > > NOTE: I get the following from ncdump:
> > > > > > >
> > > > > > >
> > > > > > > float QPF_SFC(ProjectionHr, YCells, XCells) ;
> > > > > > > QPF_SFC:long_name = "Total precipitation" ;
> > > > > > > QPF_SFC:units = "inch" ;
> > > > > > > QPF_SFC:coordinates = "longitude latitude" ;
> > > > > > > QPF_SFC:ReferenceTime = 1515153600 ;
> > > > > > > QPF_SFC:ReferenceTimeString = "2018-01-05
12:00:00
> > 00:00" ;
> > > > > > > QPF_SFC:gridType = "SCALAR" ;
> > > > > > > QPF_SFC:level = "SFC" ;
> > > > > > >
> > > > > > > Note that there is no name defined, only long_name. I
assume
> name
> > > is
> > > > > > > QPF_SFC.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Thanks in advance for your help....Brad Diehl
> > > > > > >
> > > > > > > #!/bin/sh
> > > > > > > ###
> > > > > > > ############
> > > > > > >
> > > > > > > source /u/Brad.Diehl/met_env/tut_env
> > > > > > >
> > > > > > > regrid_data_plane rfc_2018010512f006.nc \
> > > > > > > urma_co \
> > > > > > > re_grid_out.nc \
> > > > > > > -method BUDGET \
> > > > > > > -width 2 \
> > > > > > > -v 2 \
> > > > > > > -field 'name="QPF_SFC"; level="(*,*)";'
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > DEBUG 1: Reading data file: rfc_2018010512f006.nc
> > > > > > > WARNING:
> > > > > > > WARNING: NcCfFile::open() -> could not determine the
valid
> time,
> > > > using
> > > > > 0.
> > > > > > > WARNING:
> > > > > > > DEBUG 2: Input grid: Projection: Lambert Conformal Nx:
2145 Ny:
> > > 1377
> > > > > > > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000 Alpha:
> 6509.082
> > > > Cone:
> > > > > > > 0.423 Bx: 1088.0030 By: 5483.6617
> > > > > > > DEBUG 2: Output grid: Projection: Lambert Conformal Nx:
2145
> Ny:
> > > 1377
> > > > > > > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000 Alpha:
> 6509.082
> > > > Cone:
> > > > > > > 0.423 Bx: 1088.0030 By: 5483.6617
> > > > > > > DEBUG 2: Interpolation options: method = BUDGET, width =
2,
> > > > vld_thresh
> > > > > =
> > > > > > > 0.5
> > > > > > > ERROR :
> > > > > > > ERROR : NcCfFile::getData(NcVar *, const LongArray &,
> DataPlane
> > &)
> > > > ->
> > > > > > > needed 3 arguments for variable QPF_SFC, got 2
> > > > > > > ERROR :
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: RFC netcdf input into regrid_data_plane
From: Brad Diehl - NOAA Federal
Time: Fri Jan 12 07:16:23 2018
John, as always want to thank you for your efforts. After some
internal
discussion here, we did'nt realize that a lot of pre-processing of the
RFC
data might have to be done before being used in MET. Our team is
getting
together next week to discuss all this, and in the end we will likely
have
to give some recommendations to the RFC team on how to make the data
compliant for our needs.
Thanks again, Brad
On Thu, Jan 11, 2018 at 10:26 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Brad,
>
> OK, I ran the following steps...
>
> (1) Run ncdump to dump data to ascii:
>
> ncdump rfc_2018010512f006_MOD.nc > rfc_2018010512f006_MOD.ncdump
>
>
> (2) Edit the ascii output by adding:
>
>
> double forecast_reference_time ;
>
> forecast_reference_time:long_name = "reference_time"
;
>
> forecast_reference_time:units = "hours since 2018-
01-05
> 12:00:00" ;
>
> ...
>
> forecast_reference_time = 0 ;
>
>
> (3) Run ncgen to generate the NetCDF file from the modified ncdump
output:
>
>
> ncgen rfc_2018010512f006_MOD.ncdump -o rfc_2018010512f006_MOD2.nc
>
>
> (4) Run plot_data_plane again at verbosity level 4 (which lists the
timing
> info MET parsed from the file):
>
>
> met-6.1/bin/plot_data_plane rfc_2018010512f006_MOD.nc QPF_SFC.ps
> 'name="QPF_SFC"; level="(0,*,*)";' -v 4
>
>
> DEBUG 4: Data plane information:
>
> DEBUG 4: plane min: 0
>
> DEBUG 4: plane max: 0.854724
>
> DEBUG 4: valid time: 20180105_160000
>
> DEBUG 4: lead time: 040000
>
> DEBUG 4: init time: 20180105_120000
>
> DEBUG 4: accum time: 000000
>
>
> These steps demonstrate that adding the forecast_reference_time
variable
> does define the model initialization time for MET. While using
> ncdump/ncgen is fine for testing, it's not good for systematic use.
> Instead, I'd suggest talking to your NetCDF guy about preferred
methods for
> updating the contents of your NetCDF files.
>
>
> I copied all of these files back up to the ftp site:
>
> ftp://ftp.rap.ucar.edu/incoming/irap/met_help/diehl_data/for_brad
>
>
> Yes, the regrid_data_plane tool is writing an output variable named
> "QPF_SFC_0\,\*\,\*".
> Those embedded commas are a pain... and in fact, we've updated MET
6.1 to
> replace all instances of commas with underscores. Also note that
you do
> have control over the output variable name from regrid_data_plane.
The
> "-name" command line argument is used to name that variable however
you'd
> like. Feel free to name it whatever is convenient for downstream
> applications.
>
>
> Hope that helps.
>
>
> John
>
> On Thu, Jan 11, 2018 at 11:22 AM, Brad Diehl - NOAA Federal via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
> >
> > I was going to ask the netcdf guru here how to go about it, but if
you
> know
> > what to do and have the time sure use ncdump/ncgen to do it for
our
> sample
> > case.
> >
> > BTW, one of our developers is having some issues with trying to
convert
> the
> > rfc data into geotiff files, he has done it successfully with all
the
> other
> > models.
> >
> > He chatted:
> >
> > and so my variable that I'm trying to read
> > "*QPF_SFC_0\,\*\,\*"?*
> > "*QPF_SFC_0\,\*\,\*"*
> > is coming out of MET
> >
> > Can you send them a note and ask them why the variable name is
such?
> > I know why it is, I think it's not parsing things properly
> > But I can't read the variable
> >
> > If he get s.t. to work I'll let you know ASAP.
> >
> > On Thu, Jan 11, 2018 at 5:56 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Brad,
> > >
> > > The NetCDF input you’re passing to regrid data plane is cf-
compliant
> but
> > > the NetCDF output from it is not.
> > >
> > > We do plan to eventually change the output format to be cg
compliant
> but
> > it
> > > currently isn’t.
> > >
> > > If you rerun regrid data plane with a higher verbosity level (-v
4 or
> -v
> > 5
> > > maybe), you should see a DEBUG message telling you that it can’t
> > determine
> > > the init time from the input netcdf file... so it just sets the
init
> and
> > > valid times to the same time.
> > >
> > > The cf way of defining the init time is adding the
> > forecast_reference_time
> > > variable to define it. Unfortunately I don’t have a quick NCO
command
> to
> > > add a new variable.
> > >
> > > But I could use ncdump/ncgen to do it for our sample case if
that would
> > be
> > > helpful.
> > >
> > > Thanks
> > > John
> > >
> > > On Thu, Jan 11, 2018 at 9:51 AM Brad Diehl - NOAA Federal via RT
<
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514
>
> > > >
> > > > Using the recommended NCO utilities, I did the following two
steps as
> > > > suggested:
> > > >
> > > > ncrename -v ProjectionHr,time -d ProjectionHr,time -o
> > > > rfc_2018010512f006_MOD.nc rfc_2018010512f006.nc
> > > >
> > > > ncatted -a units,time,o,c,"hours since 2018-01-05 12:00:00"
> > > > rfc_2018010512f006_MOD.nc
> > > >
> > > > And MET/regrid_data_plane ran without warning.
> > > >
> > > > For forecast data, the following was recommended:
> > > >
> > > > (3) If this is analysis data, then we're all set. But for
forecast
> > data,
> > > > you'd need to define the forecast initialization time in the
CF
> > compliant
> > > > way. That's done by defining a variable named
> > > "forecast_reference_time"...
> > > > looking something like this:
> > > >
> > > >
> > > > double forecast_reference_time ;
> > > >
> > > > forecast_reference_time:long_name = "reference time" ;
> > > >
> > > > forecast_reference_time:units = "seconds since 1970-01-01" ;
> > > >
> > > > However, if I look at forecast data via ncdump that we've
processed
> > > through
> > > > regrid_data_plane, I never see forecast_reference_time
referred to.
> For
> > > > example, see below for hrrr (gfs had same result):
> > > >
> > > > Brad.Diehl at g14a1:
> > > > /mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
> > > > ncdump hrrrconc2p518010512qpf06f006 | grep time
> > > > APCP_06:init_time = "20180105_120000" ;
> > > > APCP_06:init_time_ut = "1515153600" ;
> > > > APCP_06:valid_time = "20180105_180000" ;
> > > > APCP_06:valid_time_ut = "1515175200" ;
> > > > APCP_06:accum_time = "060000" ;
> > > > APCP_06:accum_time_sec = 21600 ;
> > > >
> > > > Brad.Diehl at g14a1:
> > > > /mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
> > > > ncdump hrrrconc2p518010512qpf06f006 | grep forecast
> > > > Brad.Diehl at g14a1:
> > > > /mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
> > > >
> > > > So, not sure how to proceed. Any additional help would be
> > > > appreciated........Brad Diehl
> > > >
> > > >
> > > > On Wed, Jan 10, 2018 at 5:22 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Brad,
> > > > >
> > > > > I took a look at your data. This file is a NetCDF file
which
> > > purportedly
> > > > > follows the CF convention. However, MET is not correctly
parsing
> the
> > > > > timing info. Running though the debugger, I find that it's
> expecting
> > > the
> > > > > timing variable to simply be named "time". To test this
out, I
> used
> > > the
> > > > > ncrename utility to rename the "ProjectionHr" variable and
> dimension
> > to
> > > > > "time":
> > > > >
> > > > > ncrename -v ProjectionHr,time -d ProjectionHr,time -o
> > > > > rfc_2018010512f006_MOD.nc rfc_2018010512f006.nc
> > > > >
> > > > >
> > > > > Rerunning reveals the next warning:
> > > > >
> > > > >
> > > > > WARNING: parse_cf_time_string() -> unexpected NetCDF CF
convention
> > time
> > > > > unit "2018-01-05 12:00:00 00:00"
> > > > >
> > > > >
> > > > > For the CF convention, the "units" attribute for time should
look
> > like
> > > > this
> > > > > "[TIME UNITS] since [REFERENCE TIME]". To test, I used the
ncatted
> > > > utility
> > > > > to modify your sample file:
> > > > >
> > > > >
> > > > > ncatted -a units,time,o,c,"hours since 2018-01-05 12:00:00"
> > > > > rfc_2018010512f006_MOD.nc
> > > > >
> > > > >
> > > > > And finally that allows MET run without warning:
> > > > >
> > > > >
> > > > > met-6.1/bin/plot_data_plane rfc_2018010512f006_MOD.nc
QPF_SFC.ps
> > > > > 'name="QPF_SFC"; level="(0,*,*)";'
> > > > >
> > > > >
> > > > > And the resulting plot is attached.
> > > > >
> > > > >
> > > > > In all, there are 3 issues:
> > > > >
> > > > >
> > > > > (1) MET expects the time variable to be named "time". I'm
not sure
> > > > whether
> > > > > this is actually a CF requirement. If not, please let me
know, and
> > we
> > > > can
> > > > > take a closer look at the CF convention to see what MET
*should* be
> > > > doing.
> > > > >
> > > > >
> > > > > (2) The time "units" attribute must be named "[TIME UNITS]
since
> > > > [REFERENCE
> > > > > TIME]". I'm pretty confident that this a CF requirement.
> > > > >
> > > > >
> > > > > (3) If this is analysis data, then we're all set. But for
forecast
> > > data,
> > > > > you'd need to define the forecast initialization time in the
CF
> > > compliant
> > > > > way. That's done by defining a variable named
> > > > "forecast_reference_time"...
> > > > > looking something like this:
> > > > >
> > > > >
> > > > > double forecast_reference_time ;
> > > > >
> > > > > forecast_reference_time:long_name = "reference time" ;
> > > > >
> > > > > forecast_reference_time:units = "seconds since 1970-01-01" ;
> > > > >
> > > > >
> > > > > Of course the units can be defined as seconds, minutes,
hour, days,
> > and
> > > > so
> > > > > on.
> > > > >
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Wed, Jan 10, 2018 at 7:00 AM, Brad Diehl - NOAA Federal
via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
> > > > > >
> > > > > > Once again, thanks for the help, I'll have to be more
aware of
> the
> > > > > > dimensions moving forward. And the solution of using
-field
> > > > > > 'name="QPF_SFC"; level="(0,*,*)";' worked just fine. Many
thanks
> > > again.
> > > > > >
> > > > > >
> > > > > > However, I continue to get the warning below from
> > regrid_data_plane.
> > > > Not
> > > > > > sure if its a show stopper but does'nt appear to be. Is
there
> > > anything
> > > > I
> > > > > > can do? If not, I'll consider this ticket closed.
> > > > > >
> > > > > > WARNING:
> > > > > > WARNING: NcCfFile::open() -> could not determine the valid
time,
> > > using
> > > > 0.
> > > > > > WARNING:
> > > > > >
> > > > > > On Tue, Jan 9, 2018 at 3:01 PM, John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu
> > > > > > > wrote:
> > > > > >
> > > > > > > Hello Brad,
> > > > > > >
> > > > > > > I see you're having trouble passing a NetCDF file to the
> > > > > > regrid_data_plane
> > > > > > > tool. The problem is how you've specified the "level"
value in
> > the
> > > > > > command
> > > > > > > line argument. You specified "(*,*)" (i.e. 2
dimensions) but
> > that
> > > > > > variable
> > > > > > > actually has 3 dimensions:
> > > > > > > QPF_SFC(ProjectionHr, YCells, XCells)
> > > > > > >
> > > > > > > That's what this error message is telling you:
> > > > > > > ERROR : NcCfFile::getData(NcVar *, const LongArray
&,
> > DataPlane
> > > > &)
> > > > > ->
> > > > > > > needed 3 arguments for variable QPF_SFC, got 2
> > > > > > >
> > > > > > > Please try using this:
> > > > > > > -field 'name="QPF_SFC"; level="(0,*,*)";'
> > > > > > >
> > > > > > > That'll process the first 2D slice of data from that 3D
> variable.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Mon, Jan 8, 2018 at 10:00 AM, Brad Diehl - NOAA
Federal via
> > RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Mon Jan 08 10:00:01 2018: Request 83514 was acted
upon.
> > > > > > > > Transaction: Ticket created by brad.diehl at noaa.gov
> > > > > > > > Queue: met_help
> > > > > > > > Subject: RFC netcdf input into regrid_data_plane
> > > > > > > > Owner: Nobody
> > > > > > > > Requestors: brad.diehl at noaa.gov
> > > > > > > > Status: new
> > > > > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> > > > > Ticket/Display.html?id=83514
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > To whom it may concern,
> > > > > > > >
> > > > > > > > I have been working with ABRFC Mosaic QPF in netcdf
format
> and
> > > > using
> > > > > it
> > > > > > > as
> > > > > > > > input into regrid_data_plane. Using the simple script
below,
> I
> > > get
> > > > > > > > following results (pasted below script). I was'nt sure
how
> > severe
> > > > the
> > > > > > > > ERRORS were and wanted to double check with the help
desk. I
> > did
> > > > > notice
> > > > > > > in
> > > > > > > > the MET FAQ that to "use gridded NetCDF files in MET,
the
> files
> > > > need
> > > > > to
> > > > > > > > look like the output of the pcp_combine tool" with
some
> > > > requirements
> > > > > > > > listed.
> > > > > > > >
> > > > > > > > I have sent the script and input files via the anon
ftp site
> in
> > > > > > > diehl_data.
> > > > > > > > The script name is rfc_test.sh, input files are
> > > > > rfc_2018010512f006.nc
> > > > > > > and
> > > > > > > > urma_co.
> > > > > > > >
> > > > > > > > NOTE: I get the following from ncdump:
> > > > > > > >
> > > > > > > >
> > > > > > > > float QPF_SFC(ProjectionHr, YCells, XCells) ;
> > > > > > > > QPF_SFC:long_name = "Total precipitation" ;
> > > > > > > > QPF_SFC:units = "inch" ;
> > > > > > > > QPF_SFC:coordinates = "longitude latitude" ;
> > > > > > > > QPF_SFC:ReferenceTime = 1515153600 ;
> > > > > > > > QPF_SFC:ReferenceTimeString = "2018-01-05
12:00:00
> > > 00:00" ;
> > > > > > > > QPF_SFC:gridType = "SCALAR" ;
> > > > > > > > QPF_SFC:level = "SFC" ;
> > > > > > > >
> > > > > > > > Note that there is no name defined, only long_name. I
assume
> > name
> > > > is
> > > > > > > > QPF_SFC.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks in advance for your help....Brad Diehl
> > > > > > > >
> > > > > > > > #!/bin/sh
> > > > > > > > ###
> > > > > > > > ############
> > > > > > > >
> > > > > > > > source /u/Brad.Diehl/met_env/tut_env
> > > > > > > >
> > > > > > > > regrid_data_plane rfc_2018010512f006.nc \
> > > > > > > > urma_co \
> > > > > > > > re_grid_out.nc \
> > > > > > > > -method BUDGET \
> > > > > > > > -width 2 \
> > > > > > > > -v 2 \
> > > > > > > > -field 'name="QPF_SFC"; level="(*,*)";'
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > DEBUG 1: Reading data file: rfc_2018010512f006.nc
> > > > > > > > WARNING:
> > > > > > > > WARNING: NcCfFile::open() -> could not determine the
valid
> > time,
> > > > > using
> > > > > > 0.
> > > > > > > > WARNING:
> > > > > > > > DEBUG 2: Input grid: Projection: Lambert Conformal Nx:
2145
> Ny:
> > > > 1377
> > > > > > > > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000
Alpha:
> > 6509.082
> > > > > Cone:
> > > > > > > > 0.423 Bx: 1088.0030 By: 5483.6617
> > > > > > > > DEBUG 2: Output grid: Projection: Lambert Conformal
Nx: 2145
> > Ny:
> > > > 1377
> > > > > > > > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000
Alpha:
> > 6509.082
> > > > > Cone:
> > > > > > > > 0.423 Bx: 1088.0030 By: 5483.6617
> > > > > > > > DEBUG 2: Interpolation options: method = BUDGET, width
= 2,
> > > > > vld_thresh
> > > > > > =
> > > > > > > > 0.5
> > > > > > > > ERROR :
> > > > > > > > ERROR : NcCfFile::getData(NcVar *, const LongArray &,
> > DataPlane
> > > &)
> > > > > ->
> > > > > > > > needed 3 arguments for variable QPF_SFC, got 2
> > > > > > > > ERROR :
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: RFC netcdf input into regrid_data_plane
From: John Halley Gotway
Time: Fri Jan 12 08:10:20 2018
Brad,
That makes sense. If they’re tagging this data as being CF-compliant,
it
looks to me like they’re close but not quite there yet.
There are some cf compliance checker tools out there, but I’ve never
actually used those.
And please let me know if there are cf compliance details that you
think
MET isn’t handling correctly. That’s always a possibility too!
Thanks
John
On Fri, Jan 12, 2018 at 7:16 AM Brad Diehl - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
>
> John, as always want to thank you for your efforts. After some
internal
> discussion here, we did'nt realize that a lot of pre-processing of
the RFC
> data might have to be done before being used in MET. Our team is
getting
> together next week to discuss all this, and in the end we will
likely have
> to give some recommendations to the RFC team on how to make the data
> compliant for our needs.
>
> Thanks again, Brad
>
>
> On Thu, Jan 11, 2018 at 10:26 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Brad,
> >
> > OK, I ran the following steps...
> >
> > (1) Run ncdump to dump data to ascii:
> >
> > ncdump rfc_2018010512f006_MOD.nc > rfc_2018010512f006_MOD.ncdump
> >
> >
> > (2) Edit the ascii output by adding:
> >
> >
> > double forecast_reference_time ;
> >
> > forecast_reference_time:long_name =
"reference_time" ;
> >
> > forecast_reference_time:units = "hours since 2018-
01-05
> > 12:00:00" ;
> >
> > ...
> >
> > forecast_reference_time = 0 ;
> >
> >
> > (3) Run ncgen to generate the NetCDF file from the modified ncdump
> output:
> >
> >
> > ncgen rfc_2018010512f006_MOD.ncdump -o rfc_2018010512f006_MOD2.nc
> >
> >
> > (4) Run plot_data_plane again at verbosity level 4 (which lists
the
> timing
> > info MET parsed from the file):
> >
> >
> > met-6.1/bin/plot_data_plane rfc_2018010512f006_MOD.nc QPF_SFC.ps
> > 'name="QPF_SFC"; level="(0,*,*)";' -v 4
> >
> >
> > DEBUG 4: Data plane information:
> >
> > DEBUG 4: plane min: 0
> >
> > DEBUG 4: plane max: 0.854724
> >
> > DEBUG 4: valid time: 20180105_160000
> >
> > DEBUG 4: lead time: 040000
> >
> > DEBUG 4: init time: 20180105_120000
> >
> > DEBUG 4: accum time: 000000
> >
> >
> > These steps demonstrate that adding the forecast_reference_time
variable
> > does define the model initialization time for MET. While using
> > ncdump/ncgen is fine for testing, it's not good for systematic
use.
> > Instead, I'd suggest talking to your NetCDF guy about preferred
methods
> for
> > updating the contents of your NetCDF files.
> >
> >
> > I copied all of these files back up to the ftp site:
> >
> >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/diehl_data/for_brad
> >
> >
> > Yes, the regrid_data_plane tool is writing an output variable
named
> > "QPF_SFC_0\,\*\,\*".
> > Those embedded commas are a pain... and in fact, we've updated MET
6.1 to
> > replace all instances of commas with underscores. Also note that
you do
> > have control over the output variable name from regrid_data_plane.
The
> > "-name" command line argument is used to name that variable
however you'd
> > like. Feel free to name it whatever is convenient for downstream
> > applications.
> >
> >
> > Hope that helps.
> >
> >
> > John
> >
> > On Thu, Jan 11, 2018 at 11:22 AM, Brad Diehl - NOAA Federal via RT
<
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
> > >
> > > I was going to ask the netcdf guru here how to go about it, but
if you
> > know
> > > what to do and have the time sure use ncdump/ncgen to do it for
our
> > sample
> > > case.
> > >
> > > BTW, one of our developers is having some issues with trying to
convert
> > the
> > > rfc data into geotiff files, he has done it successfully with
all the
> > other
> > > models.
> > >
> > > He chatted:
> > >
> > > and so my variable that I'm trying to read
> > > "*QPF_SFC_0\,\*\,\*"?*
> > > "*QPF_SFC_0\,\*\,\*"*
> > > is coming out of MET
> > >
> > > Can you send them a note and ask them why the variable name is
such?
> > > I know why it is, I think it's not parsing things properly
> > > But I can't read the variable
> > >
> > > If he get s.t. to work I'll let you know ASAP.
> > >
> > > On Thu, Jan 11, 2018 at 5:56 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Brad,
> > > >
> > > > The NetCDF input you’re passing to regrid data plane is cf-
compliant
> > but
> > > > the NetCDF output from it is not.
> > > >
> > > > We do plan to eventually change the output format to be cg
compliant
> > but
> > > it
> > > > currently isn’t.
> > > >
> > > > If you rerun regrid data plane with a higher verbosity level
(-v 4 or
> > -v
> > > 5
> > > > maybe), you should see a DEBUG message telling you that it
can’t
> > > determine
> > > > the init time from the input netcdf file... so it just sets
the init
> > and
> > > > valid times to the same time.
> > > >
> > > > The cf way of defining the init time is adding the
> > > forecast_reference_time
> > > > variable to define it. Unfortunately I don’t have a quick NCO
> command
> > to
> > > > add a new variable.
> > > >
> > > > But I could use ncdump/ncgen to do it for our sample case if
that
> would
> > > be
> > > > helpful.
> > > >
> > > > Thanks
> > > > John
> > > >
> > > > On Thu, Jan 11, 2018 at 9:51 AM Brad Diehl - NOAA Federal via
RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
> > > > >
> > > > > Using the recommended NCO utilities, I did the following two
steps
> as
> > > > > suggested:
> > > > >
> > > > > ncrename -v ProjectionHr,time -d ProjectionHr,time -o
> > > > > rfc_2018010512f006_MOD.nc rfc_2018010512f006.nc
> > > > >
> > > > > ncatted -a units,time,o,c,"hours since 2018-01-05 12:00:00"
> > > > > rfc_2018010512f006_MOD.nc
> > > > >
> > > > > And MET/regrid_data_plane ran without warning.
> > > > >
> > > > > For forecast data, the following was recommended:
> > > > >
> > > > > (3) If this is analysis data, then we're all set. But for
forecast
> > > data,
> > > > > you'd need to define the forecast initialization time in the
CF
> > > compliant
> > > > > way. That's done by defining a variable named
> > > > "forecast_reference_time"...
> > > > > looking something like this:
> > > > >
> > > > >
> > > > > double forecast_reference_time ;
> > > > >
> > > > > forecast_reference_time:long_name = "reference time" ;
> > > > >
> > > > > forecast_reference_time:units = "seconds since 1970-01-01" ;
> > > > >
> > > > > However, if I look at forecast data via ncdump that we've
processed
> > > > through
> > > > > regrid_data_plane, I never see forecast_reference_time
referred to.
> > For
> > > > > example, see below for hrrr (gfs had same result):
> > > > >
> > > > > Brad.Diehl at g14a1:
> > > > > /mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
> > > > > ncdump hrrrconc2p518010512qpf06f006 | grep time
> > > > > APCP_06:init_time = "20180105_120000" ;
> > > > > APCP_06:init_time_ut = "1515153600" ;
> > > > > APCP_06:valid_time = "20180105_180000" ;
> > > > > APCP_06:valid_time_ut = "1515175200" ;
> > > > > APCP_06:accum_time = "060000" ;
> > > > > APCP_06:accum_time_sec = 21600 ;
> > > > >
> > > > > Brad.Diehl at g14a1:
> > > > > /mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
> > > > > ncdump hrrrconc2p518010512qpf06f006 | grep forecast
> > > > > Brad.Diehl at g14a1:
> > > > > /mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
> > > > >
> > > > > So, not sure how to proceed. Any additional help would be
> > > > > appreciated........Brad Diehl
> > > > >
> > > > >
> > > > > On Wed, Jan 10, 2018 at 5:22 PM, John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > > Brad,
> > > > > >
> > > > > > I took a look at your data. This file is a NetCDF file
which
> > > > purportedly
> > > > > > follows the CF convention. However, MET is not correctly
parsing
> > the
> > > > > > timing info. Running though the debugger, I find that
it's
> > expecting
> > > > the
> > > > > > timing variable to simply be named "time". To test this
out, I
> > used
> > > > the
> > > > > > ncrename utility to rename the "ProjectionHr" variable and
> > dimension
> > > to
> > > > > > "time":
> > > > > >
> > > > > > ncrename -v ProjectionHr,time -d ProjectionHr,time -o
> > > > > > rfc_2018010512f006_MOD.nc rfc_2018010512f006.nc
> > > > > >
> > > > > >
> > > > > > Rerunning reveals the next warning:
> > > > > >
> > > > > >
> > > > > > WARNING: parse_cf_time_string() -> unexpected NetCDF CF
> convention
> > > time
> > > > > > unit "2018-01-05 12:00:00 00:00"
> > > > > >
> > > > > >
> > > > > > For the CF convention, the "units" attribute for time
should look
> > > like
> > > > > this
> > > > > > "[TIME UNITS] since [REFERENCE TIME]". To test, I used
the
> ncatted
> > > > > utility
> > > > > > to modify your sample file:
> > > > > >
> > > > > >
> > > > > > ncatted -a units,time,o,c,"hours since 2018-01-05
12:00:00"
> > > > > > rfc_2018010512f006_MOD.nc
> > > > > >
> > > > > >
> > > > > > And finally that allows MET run without warning:
> > > > > >
> > > > > >
> > > > > > met-6.1/bin/plot_data_plane rfc_2018010512f006_MOD.nc
QPF_SFC.ps
> > > > > > 'name="QPF_SFC"; level="(0,*,*)";'
> > > > > >
> > > > > >
> > > > > > And the resulting plot is attached.
> > > > > >
> > > > > >
> > > > > > In all, there are 3 issues:
> > > > > >
> > > > > >
> > > > > > (1) MET expects the time variable to be named "time". I'm
not
> sure
> > > > > whether
> > > > > > this is actually a CF requirement. If not, please let me
know,
> and
> > > we
> > > > > can
> > > > > > take a closer look at the CF convention to see what MET
*should*
> be
> > > > > doing.
> > > > > >
> > > > > >
> > > > > > (2) The time "units" attribute must be named "[TIME UNITS]
since
> > > > > [REFERENCE
> > > > > > TIME]". I'm pretty confident that this a CF requirement.
> > > > > >
> > > > > >
> > > > > > (3) If this is analysis data, then we're all set. But for
> forecast
> > > > data,
> > > > > > you'd need to define the forecast initialization time in
the CF
> > > > compliant
> > > > > > way. That's done by defining a variable named
> > > > > "forecast_reference_time"...
> > > > > > looking something like this:
> > > > > >
> > > > > >
> > > > > > double forecast_reference_time ;
> > > > > >
> > > > > > forecast_reference_time:long_name = "reference time" ;
> > > > > >
> > > > > > forecast_reference_time:units = "seconds since 1970-01-01"
;
> > > > > >
> > > > > >
> > > > > > Of course the units can be defined as seconds, minutes,
hour,
> days,
> > > and
> > > > > so
> > > > > > on.
> > > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Wed, Jan 10, 2018 at 7:00 AM, Brad Diehl - NOAA Federal
via
> RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514
> >
> > > > > > >
> > > > > > > Once again, thanks for the help, I'll have to be more
aware of
> > the
> > > > > > > dimensions moving forward. And the solution of using
-field
> > > > > > > 'name="QPF_SFC"; level="(0,*,*)";' worked just fine.
Many
> thanks
> > > > again.
> > > > > > >
> > > > > > >
> > > > > > > However, I continue to get the warning below from
> > > regrid_data_plane.
> > > > > Not
> > > > > > > sure if its a show stopper but does'nt appear to be. Is
there
> > > > anything
> > > > > I
> > > > > > > can do? If not, I'll consider this ticket closed.
> > > > > > >
> > > > > > > WARNING:
> > > > > > > WARNING: NcCfFile::open() -> could not determine the
valid
> time,
> > > > using
> > > > > 0.
> > > > > > > WARNING:
> > > > > > >
> > > > > > > On Tue, Jan 9, 2018 at 3:01 PM, John Halley Gotway via
RT <
> > > > > > > met_help at ucar.edu
> > > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Brad,
> > > > > > > >
> > > > > > > > I see you're having trouble passing a NetCDF file to
the
> > > > > > > regrid_data_plane
> > > > > > > > tool. The problem is how you've specified the "level"
value
> in
> > > the
> > > > > > > command
> > > > > > > > line argument. You specified "(*,*)" (i.e. 2
dimensions) but
> > > that
> > > > > > > variable
> > > > > > > > actually has 3 dimensions:
> > > > > > > > QPF_SFC(ProjectionHr, YCells, XCells)
> > > > > > > >
> > > > > > > > That's what this error message is telling you:
> > > > > > > > ERROR : NcCfFile::getData(NcVar *, const LongArray
&,
> > > DataPlane
> > > > > &)
> > > > > > ->
> > > > > > > > needed 3 arguments for variable QPF_SFC, got 2
> > > > > > > >
> > > > > > > > Please try using this:
> > > > > > > > -field 'name="QPF_SFC"; level="(0,*,*)";'
> > > > > > > >
> > > > > > > > That'll process the first 2D slice of data from that
3D
> > variable.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Mon, Jan 8, 2018 at 10:00 AM, Brad Diehl - NOAA
Federal
> via
> > > RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Mon Jan 08 10:00:01 2018: Request 83514 was acted
upon.
> > > > > > > > > Transaction: Ticket created by brad.diehl at noaa.gov
> > > > > > > > > Queue: met_help
> > > > > > > > > Subject: RFC netcdf input into
regrid_data_plane
> > > > > > > > > Owner: Nobody
> > > > > > > > > Requestors: brad.diehl at noaa.gov
> > > > > > > > > Status: new
> > > > > > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> > > > > > Ticket/Display.html?id=83514
> > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > To whom it may concern,
> > > > > > > > >
> > > > > > > > > I have been working with ABRFC Mosaic QPF in netcdf
format
> > and
> > > > > using
> > > > > > it
> > > > > > > > as
> > > > > > > > > input into regrid_data_plane. Using the simple
script
> below,
> > I
> > > > get
> > > > > > > > > following results (pasted below script). I was'nt
sure how
> > > severe
> > > > > the
> > > > > > > > > ERRORS were and wanted to double check with the help
desk.
> I
> > > did
> > > > > > notice
> > > > > > > > in
> > > > > > > > > the MET FAQ that to "use gridded NetCDF files in
MET, the
> > files
> > > > > need
> > > > > > to
> > > > > > > > > look like the output of the pcp_combine tool" with
some
> > > > > requirements
> > > > > > > > > listed.
> > > > > > > > >
> > > > > > > > > I have sent the script and input files via the anon
ftp
> site
> > in
> > > > > > > > diehl_data.
> > > > > > > > > The script name is rfc_test.sh, input files are
> > > > > > rfc_2018010512f006.nc
> > > > > > > > and
> > > > > > > > > urma_co.
> > > > > > > > >
> > > > > > > > > NOTE: I get the following from ncdump:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > float QPF_SFC(ProjectionHr, YCells, XCells) ;
> > > > > > > > > QPF_SFC:long_name = "Total precipitation" ;
> > > > > > > > > QPF_SFC:units = "inch" ;
> > > > > > > > > QPF_SFC:coordinates = "longitude latitude" ;
> > > > > > > > > QPF_SFC:ReferenceTime = 1515153600 ;
> > > > > > > > > QPF_SFC:ReferenceTimeString = "2018-01-05
12:00:00
> > > > 00:00" ;
> > > > > > > > > QPF_SFC:gridType = "SCALAR" ;
> > > > > > > > > QPF_SFC:level = "SFC" ;
> > > > > > > > >
> > > > > > > > > Note that there is no name defined, only long_name.
I
> assume
> > > name
> > > > > is
> > > > > > > > > QPF_SFC.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks in advance for your help....Brad Diehl
> > > > > > > > >
> > > > > > > > > #!/bin/sh
> > > > > > > > > ###
> > > > > > > > > ############
> > > > > > > > >
> > > > > > > > > source /u/Brad.Diehl/met_env/tut_env
> > > > > > > > >
> > > > > > > > > regrid_data_plane rfc_2018010512f006.nc \
> > > > > > > > > urma_co \
> > > > > > > > > re_grid_out.nc \
> > > > > > > > > -method BUDGET \
> > > > > > > > > -width 2 \
> > > > > > > > > -v 2 \
> > > > > > > > > -field 'name="QPF_SFC"; level="(*,*)";'
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > DEBUG 1: Reading data file: rfc_2018010512f006.nc
> > > > > > > > > WARNING:
> > > > > > > > > WARNING: NcCfFile::open() -> could not determine the
valid
> > > time,
> > > > > > using
> > > > > > > 0.
> > > > > > > > > WARNING:
> > > > > > > > > DEBUG 2: Input grid: Projection: Lambert Conformal
Nx: 2145
> > Ny:
> > > > > 1377
> > > > > > > > > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000
Alpha:
> > > 6509.082
> > > > > > Cone:
> > > > > > > > > 0.423 Bx: 1088.0030 By: 5483.6617
> > > > > > > > > DEBUG 2: Output grid: Projection: Lambert Conformal
Nx:
> 2145
> > > Ny:
> > > > > 1377
> > > > > > > > > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000
Alpha:
> > > 6509.082
> > > > > > Cone:
> > > > > > > > > 0.423 Bx: 1088.0030 By: 5483.6617
> > > > > > > > > DEBUG 2: Interpolation options: method = BUDGET,
width = 2,
> > > > > > vld_thresh
> > > > > > > =
> > > > > > > > > 0.5
> > > > > > > > > ERROR :
> > > > > > > > > ERROR : NcCfFile::getData(NcVar *, const LongArray
&,
> > > DataPlane
> > > > &)
> > > > > > ->
> > > > > > > > > needed 3 arguments for variable QPF_SFC, got 2
> > > > > > > > > ERROR :
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: RFC netcdf input into regrid_data_plane
From: Brad Diehl - NOAA Federal
Time: Fri Jan 12 08:16:56 2018
will do, thanks again, and I'll keep you up to date as we get more
information.....Brad
On Fri, Jan 12, 2018 at 3:10 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Brad,
>
> That makes sense. If they’re tagging this data as being CF-
compliant, it
> looks to me like they’re close but not quite there yet.
>
> There are some cf compliance checker tools out there, but I’ve never
> actually used those.
>
> And please let me know if there are cf compliance details that you
think
> MET isn’t handling correctly. That’s always a possibility too!
>
> Thanks
> John
>
> On Fri, Jan 12, 2018 at 7:16 AM Brad Diehl - NOAA Federal via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
> >
> > John, as always want to thank you for your efforts. After some
internal
> > discussion here, we did'nt realize that a lot of pre-processing of
the
> RFC
> > data might have to be done before being used in MET. Our team is
getting
> > together next week to discuss all this, and in the end we will
likely
> have
> > to give some recommendations to the RFC team on how to make the
data
> > compliant for our needs.
> >
> > Thanks again, Brad
> >
> >
> > On Thu, Jan 11, 2018 at 10:26 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Brad,
> > >
> > > OK, I ran the following steps...
> > >
> > > (1) Run ncdump to dump data to ascii:
> > >
> > > ncdump rfc_2018010512f006_MOD.nc > rfc_2018010512f006_MOD.ncdump
> > >
> > >
> > > (2) Edit the ascii output by adding:
> > >
> > >
> > > double forecast_reference_time ;
> > >
> > > forecast_reference_time:long_name =
"reference_time" ;
> > >
> > > forecast_reference_time:units = "hours since
2018-01-05
> > > 12:00:00" ;
> > >
> > > ...
> > >
> > > forecast_reference_time = 0 ;
> > >
> > >
> > > (3) Run ncgen to generate the NetCDF file from the modified
ncdump
> > output:
> > >
> > >
> > > ncgen rfc_2018010512f006_MOD.ncdump -o
rfc_2018010512f006_MOD2.nc
> > >
> > >
> > > (4) Run plot_data_plane again at verbosity level 4 (which lists
the
> > timing
> > > info MET parsed from the file):
> > >
> > >
> > > met-6.1/bin/plot_data_plane rfc_2018010512f006_MOD.nc QPF_SFC.ps
> > > 'name="QPF_SFC"; level="(0,*,*)";' -v 4
> > >
> > >
> > > DEBUG 4: Data plane information:
> > >
> > > DEBUG 4: plane min: 0
> > >
> > > DEBUG 4: plane max: 0.854724
> > >
> > > DEBUG 4: valid time: 20180105_160000
> > >
> > > DEBUG 4: lead time: 040000
> > >
> > > DEBUG 4: init time: 20180105_120000
> > >
> > > DEBUG 4: accum time: 000000
> > >
> > >
> > > These steps demonstrate that adding the forecast_reference_time
> variable
> > > does define the model initialization time for MET. While using
> > > ncdump/ncgen is fine for testing, it's not good for systematic
use.
> > > Instead, I'd suggest talking to your NetCDF guy about preferred
methods
> > for
> > > updating the contents of your NetCDF files.
> > >
> > >
> > > I copied all of these files back up to the ftp site:
> > >
> > >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/diehl_data/for_brad
> > >
> > >
> > > Yes, the regrid_data_plane tool is writing an output variable
named
> > > "QPF_SFC_0\,\*\,\*".
> > > Those embedded commas are a pain... and in fact, we've updated
MET 6.1
> to
> > > replace all instances of commas with underscores. Also note
that you
> do
> > > have control over the output variable name from
regrid_data_plane. The
> > > "-name" command line argument is used to name that variable
however
> you'd
> > > like. Feel free to name it whatever is convenient for
downstream
> > > applications.
> > >
> > >
> > > Hope that helps.
> > >
> > >
> > > John
> > >
> > > On Thu, Jan 11, 2018 at 11:22 AM, Brad Diehl - NOAA Federal via
RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514
>
> > > >
> > > > I was going to ask the netcdf guru here how to go about it,
but if
> you
> > > know
> > > > what to do and have the time sure use ncdump/ncgen to do it
for our
> > > sample
> > > > case.
> > > >
> > > > BTW, one of our developers is having some issues with trying
to
> convert
> > > the
> > > > rfc data into geotiff files, he has done it successfully with
all the
> > > other
> > > > models.
> > > >
> > > > He chatted:
> > > >
> > > > and so my variable that I'm trying to read
> > > > "*QPF_SFC_0\,\*\,\*"?*
> > > > "*QPF_SFC_0\,\*\,\*"*
> > > > is coming out of MET
> > > >
> > > > Can you send them a note and ask them why the variable name is
such?
> > > > I know why it is, I think it's not parsing things properly
> > > > But I can't read the variable
> > > >
> > > > If he get s.t. to work I'll let you know ASAP.
> > > >
> > > > On Thu, Jan 11, 2018 at 5:56 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Brad,
> > > > >
> > > > > The NetCDF input you’re passing to regrid data plane is
> cf-compliant
> > > but
> > > > > the NetCDF output from it is not.
> > > > >
> > > > > We do plan to eventually change the output format to be cg
> compliant
> > > but
> > > > it
> > > > > currently isn’t.
> > > > >
> > > > > If you rerun regrid data plane with a higher verbosity level
(-v 4
> or
> > > -v
> > > > 5
> > > > > maybe), you should see a DEBUG message telling you that it
can’t
> > > > determine
> > > > > the init time from the input netcdf file... so it just sets
the
> init
> > > and
> > > > > valid times to the same time.
> > > > >
> > > > > The cf way of defining the init time is adding the
> > > > forecast_reference_time
> > > > > variable to define it. Unfortunately I don’t have a quick
NCO
> > command
> > > to
> > > > > add a new variable.
> > > > >
> > > > > But I could use ncdump/ncgen to do it for our sample case if
that
> > would
> > > > be
> > > > > helpful.
> > > > >
> > > > > Thanks
> > > > > John
> > > > >
> > > > > On Thu, Jan 11, 2018 at 9:51 AM Brad Diehl - NOAA Federal
via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
> > > > > >
> > > > > > Using the recommended NCO utilities, I did the following
two
> steps
> > as
> > > > > > suggested:
> > > > > >
> > > > > > ncrename -v ProjectionHr,time -d ProjectionHr,time -o
> > > > > > rfc_2018010512f006_MOD.nc rfc_2018010512f006.nc
> > > > > >
> > > > > > ncatted -a units,time,o,c,"hours since 2018-01-05
12:00:00"
> > > > > > rfc_2018010512f006_MOD.nc
> > > > > >
> > > > > > And MET/regrid_data_plane ran without warning.
> > > > > >
> > > > > > For forecast data, the following was recommended:
> > > > > >
> > > > > > (3) If this is analysis data, then we're all set. But for
> forecast
> > > > data,
> > > > > > you'd need to define the forecast initialization time in
the CF
> > > > compliant
> > > > > > way. That's done by defining a variable named
> > > > > "forecast_reference_time"...
> > > > > > looking something like this:
> > > > > >
> > > > > >
> > > > > > double forecast_reference_time ;
> > > > > >
> > > > > > forecast_reference_time:long_name = "reference time" ;
> > > > > >
> > > > > > forecast_reference_time:units = "seconds since 1970-01-01"
;
> > > > > >
> > > > > > However, if I look at forecast data via ncdump that we've
> processed
> > > > > through
> > > > > > regrid_data_plane, I never see forecast_reference_time
referred
> to.
> > > For
> > > > > > example, see below for hrrr (gfs had same result):
> > > > > >
> > > > > > Brad.Diehl at g14a1:
> > > > > > /mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
> > > > > > ncdump hrrrconc2p518010512qpf06f006 | grep time
> > > > > > APCP_06:init_time = "20180105_120000" ;
> > > > > > APCP_06:init_time_ut = "1515153600" ;
> > > > > > APCP_06:valid_time = "20180105_180000" ;
> > > > > > APCP_06:valid_time_ut = "1515175200" ;
> > > > > > APCP_06:accum_time = "060000" ;
> > > > > > APCP_06:accum_time_sec = 21600 ;
> > > > > >
> > > > > > Brad.Diehl at g14a1:
> > > > > > /mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
> > > > > > ncdump hrrrconc2p518010512qpf06f006 | grep forecast
> > > > > > Brad.Diehl at g14a1:
> > > > > > /mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
> > > > > >
> > > > > > So, not sure how to proceed. Any additional help would be
> > > > > > appreciated........Brad Diehl
> > > > > >
> > > > > >
> > > > > > On Wed, Jan 10, 2018 at 5:22 PM, John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > > Brad,
> > > > > > >
> > > > > > > I took a look at your data. This file is a NetCDF file
which
> > > > > purportedly
> > > > > > > follows the CF convention. However, MET is not
correctly
> parsing
> > > the
> > > > > > > timing info. Running though the debugger, I find that
it's
> > > expecting
> > > > > the
> > > > > > > timing variable to simply be named "time". To test this
out, I
> > > used
> > > > > the
> > > > > > > ncrename utility to rename the "ProjectionHr" variable
and
> > > dimension
> > > > to
> > > > > > > "time":
> > > > > > >
> > > > > > > ncrename -v ProjectionHr,time -d ProjectionHr,time -o
> > > > > > > rfc_2018010512f006_MOD.nc rfc_2018010512f006.nc
> > > > > > >
> > > > > > >
> > > > > > > Rerunning reveals the next warning:
> > > > > > >
> > > > > > >
> > > > > > > WARNING: parse_cf_time_string() -> unexpected NetCDF CF
> > convention
> > > > time
> > > > > > > unit "2018-01-05 12:00:00 00:00"
> > > > > > >
> > > > > > >
> > > > > > > For the CF convention, the "units" attribute for time
should
> look
> > > > like
> > > > > > this
> > > > > > > "[TIME UNITS] since [REFERENCE TIME]". To test, I used
the
> > ncatted
> > > > > > utility
> > > > > > > to modify your sample file:
> > > > > > >
> > > > > > >
> > > > > > > ncatted -a units,time,o,c,"hours since 2018-01-05
12:00:00"
> > > > > > > rfc_2018010512f006_MOD.nc
> > > > > > >
> > > > > > >
> > > > > > > And finally that allows MET run without warning:
> > > > > > >
> > > > > > >
> > > > > > > met-6.1/bin/plot_data_plane rfc_2018010512f006_MOD.nc
> QPF_SFC.ps
> > > > > > > 'name="QPF_SFC"; level="(0,*,*)";'
> > > > > > >
> > > > > > >
> > > > > > > And the resulting plot is attached.
> > > > > > >
> > > > > > >
> > > > > > > In all, there are 3 issues:
> > > > > > >
> > > > > > >
> > > > > > > (1) MET expects the time variable to be named "time".
I'm not
> > sure
> > > > > > whether
> > > > > > > this is actually a CF requirement. If not, please let
me know,
> > and
> > > > we
> > > > > > can
> > > > > > > take a closer look at the CF convention to see what MET
> *should*
> > be
> > > > > > doing.
> > > > > > >
> > > > > > >
> > > > > > > (2) The time "units" attribute must be named "[TIME
UNITS]
> since
> > > > > > [REFERENCE
> > > > > > > TIME]". I'm pretty confident that this a CF
requirement.
> > > > > > >
> > > > > > >
> > > > > > > (3) If this is analysis data, then we're all set. But
for
> > forecast
> > > > > data,
> > > > > > > you'd need to define the forecast initialization time in
the CF
> > > > > compliant
> > > > > > > way. That's done by defining a variable named
> > > > > > "forecast_reference_time"...
> > > > > > > looking something like this:
> > > > > > >
> > > > > > >
> > > > > > > double forecast_reference_time ;
> > > > > > >
> > > > > > > forecast_reference_time:long_name = "reference time" ;
> > > > > > >
> > > > > > > forecast_reference_time:units = "seconds since 1970-01-
01" ;
> > > > > > >
> > > > > > >
> > > > > > > Of course the units can be defined as seconds, minutes,
hour,
> > days,
> > > > and
> > > > > > so
> > > > > > > on.
> > > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Wed, Jan 10, 2018 at 7:00 AM, Brad Diehl - NOAA
Federal via
> > RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=83514
> > >
> > > > > > > >
> > > > > > > > Once again, thanks for the help, I'll have to be more
aware
> of
> > > the
> > > > > > > > dimensions moving forward. And the solution of using
-field
> > > > > > > > 'name="QPF_SFC"; level="(0,*,*)";' worked just fine.
Many
> > thanks
> > > > > again.
> > > > > > > >
> > > > > > > >
> > > > > > > > However, I continue to get the warning below from
> > > > regrid_data_plane.
> > > > > > Not
> > > > > > > > sure if its a show stopper but does'nt appear to be.
Is there
> > > > > anything
> > > > > > I
> > > > > > > > can do? If not, I'll consider this ticket closed.
> > > > > > > >
> > > > > > > > WARNING:
> > > > > > > > WARNING: NcCfFile::open() -> could not determine the
valid
> > time,
> > > > > using
> > > > > > 0.
> > > > > > > > WARNING:
> > > > > > > >
> > > > > > > > On Tue, Jan 9, 2018 at 3:01 PM, John Halley Gotway via
RT <
> > > > > > > > met_help at ucar.edu
> > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello Brad,
> > > > > > > > >
> > > > > > > > > I see you're having trouble passing a NetCDF file to
the
> > > > > > > > regrid_data_plane
> > > > > > > > > tool. The problem is how you've specified the
"level"
> value
> > in
> > > > the
> > > > > > > > command
> > > > > > > > > line argument. You specified "(*,*)" (i.e. 2
dimensions)
> but
> > > > that
> > > > > > > > variable
> > > > > > > > > actually has 3 dimensions:
> > > > > > > > > QPF_SFC(ProjectionHr, YCells, XCells)
> > > > > > > > >
> > > > > > > > > That's what this error message is telling you:
> > > > > > > > > ERROR : NcCfFile::getData(NcVar *, const
LongArray &,
> > > > DataPlane
> > > > > > &)
> > > > > > > ->
> > > > > > > > > needed 3 arguments for variable QPF_SFC, got 2
> > > > > > > > >
> > > > > > > > > Please try using this:
> > > > > > > > > -field 'name="QPF_SFC"; level="(0,*,*)";'
> > > > > > > > >
> > > > > > > > > That'll process the first 2D slice of data from that
3D
> > > variable.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > > On Mon, Jan 8, 2018 at 10:00 AM, Brad Diehl - NOAA
Federal
> > via
> > > > RT <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Mon Jan 08 10:00:01 2018: Request 83514 was acted
upon.
> > > > > > > > > > Transaction: Ticket created by brad.diehl at noaa.gov
> > > > > > > > > > Queue: met_help
> > > > > > > > > > Subject: RFC netcdf input into
regrid_data_plane
> > > > > > > > > > Owner: Nobody
> > > > > > > > > > Requestors: brad.diehl at noaa.gov
> > > > > > > > > > Status: new
> > > > > > > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> > > > > > > Ticket/Display.html?id=83514
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > To whom it may concern,
> > > > > > > > > >
> > > > > > > > > > I have been working with ABRFC Mosaic QPF in
netcdf
> format
> > > and
> > > > > > using
> > > > > > > it
> > > > > > > > > as
> > > > > > > > > > input into regrid_data_plane. Using the simple
script
> > below,
> > > I
> > > > > get
> > > > > > > > > > following results (pasted below script). I was'nt
sure
> how
> > > > severe
> > > > > > the
> > > > > > > > > > ERRORS were and wanted to double check with the
help
> desk.
> > I
> > > > did
> > > > > > > notice
> > > > > > > > > in
> > > > > > > > > > the MET FAQ that to "use gridded NetCDF files in
MET, the
> > > files
> > > > > > need
> > > > > > > to
> > > > > > > > > > look like the output of the pcp_combine tool" with
some
> > > > > > requirements
> > > > > > > > > > listed.
> > > > > > > > > >
> > > > > > > > > > I have sent the script and input files via the
anon ftp
> > site
> > > in
> > > > > > > > > diehl_data.
> > > > > > > > > > The script name is rfc_test.sh, input files are
> > > > > > > rfc_2018010512f006.nc
> > > > > > > > > and
> > > > > > > > > > urma_co.
> > > > > > > > > >
> > > > > > > > > > NOTE: I get the following from ncdump:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > float QPF_SFC(ProjectionHr, YCells, XCells) ;
> > > > > > > > > > QPF_SFC:long_name = "Total precipitation"
;
> > > > > > > > > > QPF_SFC:units = "inch" ;
> > > > > > > > > > QPF_SFC:coordinates = "longitude latitude"
;
> > > > > > > > > > QPF_SFC:ReferenceTime = 1515153600 ;
> > > > > > > > > > QPF_SFC:ReferenceTimeString = "2018-01-05
> 12:00:00
> > > > > 00:00" ;
> > > > > > > > > > QPF_SFC:gridType = "SCALAR" ;
> > > > > > > > > > QPF_SFC:level = "SFC" ;
> > > > > > > > > >
> > > > > > > > > > Note that there is no name defined, only
long_name. I
> > assume
> > > > name
> > > > > > is
> > > > > > > > > > QPF_SFC.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks in advance for your help....Brad Diehl
> > > > > > > > > >
> > > > > > > > > > #!/bin/sh
> > > > > > > > > > ###
> > > > > > > > > > ############
> > > > > > > > > >
> > > > > > > > > > source /u/Brad.Diehl/met_env/tut_env
> > > > > > > > > >
> > > > > > > > > > regrid_data_plane rfc_2018010512f006.nc \
> > > > > > > > > > urma_co \
> > > > > > > > > > re_grid_out.nc \
> > > > > > > > > > -method BUDGET \
> > > > > > > > > > -width 2 \
> > > > > > > > > > -v 2 \
> > > > > > > > > > -field 'name="QPF_SFC"; level="(*,*)";'
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > DEBUG 1: Reading data file: rfc_2018010512f006.nc
> > > > > > > > > > WARNING:
> > > > > > > > > > WARNING: NcCfFile::open() -> could not determine
the
> valid
> > > > time,
> > > > > > > using
> > > > > > > > 0.
> > > > > > > > > > WARNING:
> > > > > > > > > > DEBUG 2: Input grid: Projection: Lambert Conformal
Nx:
> 2145
> > > Ny:
> > > > > > 1377
> > > > > > > > > > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000
Alpha:
> > > > 6509.082
> > > > > > > Cone:
> > > > > > > > > > 0.423 Bx: 1088.0030 By: 5483.6617
> > > > > > > > > > DEBUG 2: Output grid: Projection: Lambert
Conformal Nx:
> > 2145
> > > > Ny:
> > > > > > 1377
> > > > > > > > > > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000
Alpha:
> > > > 6509.082
> > > > > > > Cone:
> > > > > > > > > > 0.423 Bx: 1088.0030 By: 5483.6617
> > > > > > > > > > DEBUG 2: Interpolation options: method = BUDGET,
width =
> 2,
> > > > > > > vld_thresh
> > > > > > > > =
> > > > > > > > > > 0.5
> > > > > > > > > > ERROR :
> > > > > > > > > > ERROR : NcCfFile::getData(NcVar *, const
LongArray &,
> > > > DataPlane
> > > > > &)
> > > > > > > ->
> > > > > > > > > > needed 3 arguments for variable QPF_SFC, got 2
> > > > > > > > > > ERROR :
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: RFC netcdf input into regrid_data_plane
From: Brad Diehl - NOAA Federal
Time: Tue Jan 16 09:11:01 2018
John,
We had our mtg. today regarding the RFC data compliance issues. I'm
going
to write a script that will use the combination of the nco utilities
and
additional steps you recommended (adding forecast_reference_time,
etc.,)
that can post process the data we have piling up and use that script
as a
recommendation to what the RFC will have to do to make that data
compliant
for our verification purposes. So I'm quite sure I'll need some
additional
information from you as I go through this process, and will be in
touch.
Thanks again for all your previous efforts.
On Fri, Jan 12, 2018 at 3:16 PM, Brad Diehl - NOAA Federal <
brad.diehl at noaa.gov> wrote:
> will do, thanks again, and I'll keep you up to date as we get more
> information.....Brad
>
> On Fri, Jan 12, 2018 at 3:10 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Brad,
>>
>> That makes sense. If they’re tagging this data as being CF-
compliant, it
>> looks to me like they’re close but not quite there yet.
>>
>> There are some cf compliance checker tools out there, but I’ve
never
>> actually used those.
>>
>> And please let me know if there are cf compliance details that you
think
>> MET isn’t handling correctly. That’s always a possibility too!
>>
>> Thanks
>> John
>>
>> On Fri, Jan 12, 2018 at 7:16 AM Brad Diehl - NOAA Federal via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
>> >
>> > John, as always want to thank you for your efforts. After some
internal
>> > discussion here, we did'nt realize that a lot of pre-processing
of the
>> RFC
>> > data might have to be done before being used in MET. Our team is
getting
>> > together next week to discuss all this, and in the end we will
likely
>> have
>> > to give some recommendations to the RFC team on how to make the
data
>> > compliant for our needs.
>> >
>> > Thanks again, Brad
>> >
>> >
>> > On Thu, Jan 11, 2018 at 10:26 PM, John Halley Gotway via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > > Brad,
>> > >
>> > > OK, I ran the following steps...
>> > >
>> > > (1) Run ncdump to dump data to ascii:
>> > >
>> > > ncdump rfc_2018010512f006_MOD.nc >
rfc_2018010512f006_MOD.ncdump
>> > >
>> > >
>> > > (2) Edit the ascii output by adding:
>> > >
>> > >
>> > > double forecast_reference_time ;
>> > >
>> > > forecast_reference_time:long_name =
"reference_time"
>> ;
>> > >
>> > > forecast_reference_time:units = "hours since
>> 2018-01-05
>> > > 12:00:00" ;
>> > >
>> > > ...
>> > >
>> > > forecast_reference_time = 0 ;
>> > >
>> > >
>> > > (3) Run ncgen to generate the NetCDF file from the modified
ncdump
>> > output:
>> > >
>> > >
>> > > ncgen rfc_2018010512f006_MOD.ncdump -o
rfc_2018010512f006_MOD2.nc
>> > >
>> > >
>> > > (4) Run plot_data_plane again at verbosity level 4 (which lists
the
>> > timing
>> > > info MET parsed from the file):
>> > >
>> > >
>> > > met-6.1/bin/plot_data_plane rfc_2018010512f006_MOD.nc
QPF_SFC.ps
>> > > 'name="QPF_SFC"; level="(0,*,*)";' -v 4
>> > >
>> > >
>> > > DEBUG 4: Data plane information:
>> > >
>> > > DEBUG 4: plane min: 0
>> > >
>> > > DEBUG 4: plane max: 0.854724
>> > >
>> > > DEBUG 4: valid time: 20180105_160000
>> > >
>> > > DEBUG 4: lead time: 040000
>> > >
>> > > DEBUG 4: init time: 20180105_120000
>> > >
>> > > DEBUG 4: accum time: 000000
>> > >
>> > >
>> > > These steps demonstrate that adding the forecast_reference_time
>> variable
>> > > does define the model initialization time for MET. While using
>> > > ncdump/ncgen is fine for testing, it's not good for systematic
use.
>> > > Instead, I'd suggest talking to your NetCDF guy about preferred
>> methods
>> > for
>> > > updating the contents of your NetCDF files.
>> > >
>> > >
>> > > I copied all of these files back up to the ftp site:
>> > >
>> > >
ftp://ftp.rap.ucar.edu/incoming/irap/met_help/diehl_data/for_brad
>> > >
>> > >
>> > > Yes, the regrid_data_plane tool is writing an output variable
named
>> > > "QPF_SFC_0\,\*\,\*".
>> > > Those embedded commas are a pain... and in fact, we've updated
MET
>> 6.1 to
>> > > replace all instances of commas with underscores. Also note
that
>> you do
>> > > have control over the output variable name from
regrid_data_plane.
>> The
>> > > "-name" command line argument is used to name that variable
however
>> you'd
>> > > like. Feel free to name it whatever is convenient for
downstream
>> > > applications.
>> > >
>> > >
>> > > Hope that helps.
>> > >
>> > >
>> > > John
>> > >
>> > > On Thu, Jan 11, 2018 at 11:22 AM, Brad Diehl - NOAA Federal via
RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514
>
>> > > >
>> > > > I was going to ask the netcdf guru here how to go about it,
but if
>> you
>> > > know
>> > > > what to do and have the time sure use ncdump/ncgen to do it
for our
>> > > sample
>> > > > case.
>> > > >
>> > > > BTW, one of our developers is having some issues with trying
to
>> convert
>> > > the
>> > > > rfc data into geotiff files, he has done it successfully with
all
>> the
>> > > other
>> > > > models.
>> > > >
>> > > > He chatted:
>> > > >
>> > > > and so my variable that I'm trying to read
>> > > > "*QPF_SFC_0\,\*\,\*"?*
>> > > > "*QPF_SFC_0\,\*\,\*"*
>> > > > is coming out of MET
>> > > >
>> > > > Can you send them a note and ask them why the variable name
is such?
>> > > > I know why it is, I think it's not parsing things properly
>> > > > But I can't read the variable
>> > > >
>> > > > If he get s.t. to work I'll let you know ASAP.
>> > > >
>> > > > On Thu, Jan 11, 2018 at 5:56 PM, John Halley Gotway via RT <
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > > > Brad,
>> > > > >
>> > > > > The NetCDF input you’re passing to regrid data plane is
>> cf-compliant
>> > > but
>> > > > > the NetCDF output from it is not.
>> > > > >
>> > > > > We do plan to eventually change the output format to be cg
>> compliant
>> > > but
>> > > > it
>> > > > > currently isn’t.
>> > > > >
>> > > > > If you rerun regrid data plane with a higher verbosity
level (-v
>> 4 or
>> > > -v
>> > > > 5
>> > > > > maybe), you should see a DEBUG message telling you that it
can’t
>> > > > determine
>> > > > > the init time from the input netcdf file... so it just sets
the
>> init
>> > > and
>> > > > > valid times to the same time.
>> > > > >
>> > > > > The cf way of defining the init time is adding the
>> > > > forecast_reference_time
>> > > > > variable to define it. Unfortunately I don’t have a quick
NCO
>> > command
>> > > to
>> > > > > add a new variable.
>> > > > >
>> > > > > But I could use ncdump/ncgen to do it for our sample case
if that
>> > would
>> > > > be
>> > > > > helpful.
>> > > > >
>> > > > > Thanks
>> > > > > John
>> > > > >
>> > > > > On Thu, Jan 11, 2018 at 9:51 AM Brad Diehl - NOAA Federal
via RT <
>> > > > > met_help at ucar.edu> wrote:
>> > > > >
>> > > > > >
>> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
>> > > > > >
>> > > > > > Using the recommended NCO utilities, I did the following
two
>> steps
>> > as
>> > > > > > suggested:
>> > > > > >
>> > > > > > ncrename -v ProjectionHr,time -d ProjectionHr,time -o
>> > > > > > rfc_2018010512f006_MOD.nc rfc_2018010512f006.nc
>> > > > > >
>> > > > > > ncatted -a units,time,o,c,"hours since 2018-01-05
12:00:00"
>> > > > > > rfc_2018010512f006_MOD.nc
>> > > > > >
>> > > > > > And MET/regrid_data_plane ran without warning.
>> > > > > >
>> > > > > > For forecast data, the following was recommended:
>> > > > > >
>> > > > > > (3) If this is analysis data, then we're all set. But
for
>> forecast
>> > > > data,
>> > > > > > you'd need to define the forecast initialization time in
the CF
>> > > > compliant
>> > > > > > way. That's done by defining a variable named
>> > > > > "forecast_reference_time"...
>> > > > > > looking something like this:
>> > > > > >
>> > > > > >
>> > > > > > double forecast_reference_time ;
>> > > > > >
>> > > > > > forecast_reference_time:long_name = "reference time" ;
>> > > > > >
>> > > > > > forecast_reference_time:units = "seconds since 1970-01-
01" ;
>> > > > > >
>> > > > > > However, if I look at forecast data via ncdump that we've
>> processed
>> > > > > through
>> > > > > > regrid_data_plane, I never see forecast_reference_time
referred
>> to.
>> > > For
>> > > > > > example, see below for hrrr (gfs had same result):
>> > > > > >
>> > > > > > Brad.Diehl at g14a1:
>> > > > > > /mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
>> > > > > > ncdump hrrrconc2p518010512qpf06f006 | grep time
>> > > > > > APCP_06:init_time = "20180105_120000" ;
>> > > > > > APCP_06:init_time_ut = "1515153600" ;
>> > > > > > APCP_06:valid_time = "20180105_180000" ;
>> > > > > > APCP_06:valid_time_ut = "1515175200" ;
>> > > > > > APCP_06:accum_time = "060000" ;
>> > > > > > APCP_06:accum_time_sec = 21600 ;
>> > > > > >
>> > > > > > Brad.Diehl at g14a1:
>> > > > > > /mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
>> > > > > > ncdump hrrrconc2p518010512qpf06f006 | grep forecast
>> > > > > > Brad.Diehl at g14a1:
>> > > > > > /mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
>> > > > > >
>> > > > > > So, not sure how to proceed. Any additional help would be
>> > > > > > appreciated........Brad Diehl
>> > > > > >
>> > > > > >
>> > > > > > On Wed, Jan 10, 2018 at 5:22 PM, John Halley Gotway via
RT <
>> > > > > > met_help at ucar.edu> wrote:
>> > > > > >
>> > > > > > > Brad,
>> > > > > > >
>> > > > > > > I took a look at your data. This file is a NetCDF file
which
>> > > > > purportedly
>> > > > > > > follows the CF convention. However, MET is not
correctly
>> parsing
>> > > the
>> > > > > > > timing info. Running though the debugger, I find that
it's
>> > > expecting
>> > > > > the
>> > > > > > > timing variable to simply be named "time". To test
this out,
>> I
>> > > used
>> > > > > the
>> > > > > > > ncrename utility to rename the "ProjectionHr" variable
and
>> > > dimension
>> > > > to
>> > > > > > > "time":
>> > > > > > >
>> > > > > > > ncrename -v ProjectionHr,time -d ProjectionHr,time -o
>> > > > > > > rfc_2018010512f006_MOD.nc rfc_2018010512f006.nc
>> > > > > > >
>> > > > > > >
>> > > > > > > Rerunning reveals the next warning:
>> > > > > > >
>> > > > > > >
>> > > > > > > WARNING: parse_cf_time_string() -> unexpected NetCDF CF
>> > convention
>> > > > time
>> > > > > > > unit "2018-01-05 12:00:00 00:00"
>> > > > > > >
>> > > > > > >
>> > > > > > > For the CF convention, the "units" attribute for time
should
>> look
>> > > > like
>> > > > > > this
>> > > > > > > "[TIME UNITS] since [REFERENCE TIME]". To test, I used
the
>> > ncatted
>> > > > > > utility
>> > > > > > > to modify your sample file:
>> > > > > > >
>> > > > > > >
>> > > > > > > ncatted -a units,time,o,c,"hours since 2018-01-05
12:00:00"
>> > > > > > > rfc_2018010512f006_MOD.nc
>> > > > > > >
>> > > > > > >
>> > > > > > > And finally that allows MET run without warning:
>> > > > > > >
>> > > > > > >
>> > > > > > > met-6.1/bin/plot_data_plane rfc_2018010512f006_MOD.nc
>> QPF_SFC.ps
>> > > > > > > 'name="QPF_SFC"; level="(0,*,*)";'
>> > > > > > >
>> > > > > > >
>> > > > > > > And the resulting plot is attached.
>> > > > > > >
>> > > > > > >
>> > > > > > > In all, there are 3 issues:
>> > > > > > >
>> > > > > > >
>> > > > > > > (1) MET expects the time variable to be named "time".
I'm not
>> > sure
>> > > > > > whether
>> > > > > > > this is actually a CF requirement. If not, please let
me
>> know,
>> > and
>> > > > we
>> > > > > > can
>> > > > > > > take a closer look at the CF convention to see what MET
>> *should*
>> > be
>> > > > > > doing.
>> > > > > > >
>> > > > > > >
>> > > > > > > (2) The time "units" attribute must be named "[TIME
UNITS]
>> since
>> > > > > > [REFERENCE
>> > > > > > > TIME]". I'm pretty confident that this a CF
requirement.
>> > > > > > >
>> > > > > > >
>> > > > > > > (3) If this is analysis data, then we're all set. But
for
>> > forecast
>> > > > > data,
>> > > > > > > you'd need to define the forecast initialization time
in the
>> CF
>> > > > > compliant
>> > > > > > > way. That's done by defining a variable named
>> > > > > > "forecast_reference_time"...
>> > > > > > > looking something like this:
>> > > > > > >
>> > > > > > >
>> > > > > > > double forecast_reference_time ;
>> > > > > > >
>> > > > > > > forecast_reference_time:long_name = "reference time" ;
>> > > > > > >
>> > > > > > > forecast_reference_time:units = "seconds since 1970-01-
01" ;
>> > > > > > >
>> > > > > > >
>> > > > > > > Of course the units can be defined as seconds, minutes,
hour,
>> > days,
>> > > > and
>> > > > > > so
>> > > > > > > on.
>> > > > > > >
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > John
>> > > > > > >
>> > > > > > > On Wed, Jan 10, 2018 at 7:00 AM, Brad Diehl - NOAA
Federal via
>> > RT <
>> > > > > > > met_help at ucar.edu> wrote:
>> > > > > > >
>> > > > > > > >
>> > > > > > > > <URL: https://rt.rap.ucar.edu/rt/Tic
>> ket/Display.html?id=83514
>> > >
>> > > > > > > >
>> > > > > > > > Once again, thanks for the help, I'll have to be more
aware
>> of
>> > > the
>> > > > > > > > dimensions moving forward. And the solution of using
-field
>> > > > > > > > 'name="QPF_SFC"; level="(0,*,*)";' worked just fine.
Many
>> > thanks
>> > > > > again.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > However, I continue to get the warning below from
>> > > > regrid_data_plane.
>> > > > > > Not
>> > > > > > > > sure if its a show stopper but does'nt appear to be.
Is
>> there
>> > > > > anything
>> > > > > > I
>> > > > > > > > can do? If not, I'll consider this ticket closed.
>> > > > > > > >
>> > > > > > > > WARNING:
>> > > > > > > > WARNING: NcCfFile::open() -> could not determine the
valid
>> > time,
>> > > > > using
>> > > > > > 0.
>> > > > > > > > WARNING:
>> > > > > > > >
>> > > > > > > > On Tue, Jan 9, 2018 at 3:01 PM, John Halley Gotway
via RT <
>> > > > > > > > met_help at ucar.edu
>> > > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hello Brad,
>> > > > > > > > >
>> > > > > > > > > I see you're having trouble passing a NetCDF file
to the
>> > > > > > > > regrid_data_plane
>> > > > > > > > > tool. The problem is how you've specified the
"level"
>> value
>> > in
>> > > > the
>> > > > > > > > command
>> > > > > > > > > line argument. You specified "(*,*)" (i.e. 2
dimensions)
>> but
>> > > > that
>> > > > > > > > variable
>> > > > > > > > > actually has 3 dimensions:
>> > > > > > > > > QPF_SFC(ProjectionHr, YCells, XCells)
>> > > > > > > > >
>> > > > > > > > > That's what this error message is telling you:
>> > > > > > > > > ERROR : NcCfFile::getData(NcVar *, const
LongArray &,
>> > > > DataPlane
>> > > > > > &)
>> > > > > > > ->
>> > > > > > > > > needed 3 arguments for variable QPF_SFC, got 2
>> > > > > > > > >
>> > > > > > > > > Please try using this:
>> > > > > > > > > -field 'name="QPF_SFC"; level="(0,*,*)";'
>> > > > > > > > >
>> > > > > > > > > That'll process the first 2D slice of data from
that 3D
>> > > variable.
>> > > > > > > > >
>> > > > > > > > > Thanks,
>> > > > > > > > > John
>> > > > > > > > >
>> > > > > > > > > On Mon, Jan 8, 2018 at 10:00 AM, Brad Diehl - NOAA
Federal
>> > via
>> > > > RT <
>> > > > > > > > > met_help at ucar.edu> wrote:
>> > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Mon Jan 08 10:00:01 2018: Request 83514 was acted
upon.
>> > > > > > > > > > Transaction: Ticket created by
brad.diehl at noaa.gov
>> > > > > > > > > > Queue: met_help
>> > > > > > > > > > Subject: RFC netcdf input into
regrid_data_plane
>> > > > > > > > > > Owner: Nobody
>> > > > > > > > > > Requestors: brad.diehl at noaa.gov
>> > > > > > > > > > Status: new
>> > > > > > > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
>> > > > > > > Ticket/Display.html?id=83514
>> > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > To whom it may concern,
>> > > > > > > > > >
>> > > > > > > > > > I have been working with ABRFC Mosaic QPF in
netcdf
>> format
>> > > and
>> > > > > > using
>> > > > > > > it
>> > > > > > > > > as
>> > > > > > > > > > input into regrid_data_plane. Using the simple
script
>> > below,
>> > > I
>> > > > > get
>> > > > > > > > > > following results (pasted below script). I was'nt
sure
>> how
>> > > > severe
>> > > > > > the
>> > > > > > > > > > ERRORS were and wanted to double check with the
help
>> desk.
>> > I
>> > > > did
>> > > > > > > notice
>> > > > > > > > > in
>> > > > > > > > > > the MET FAQ that to "use gridded NetCDF files in
MET,
>> the
>> > > files
>> > > > > > need
>> > > > > > > to
>> > > > > > > > > > look like the output of the pcp_combine tool"
with some
>> > > > > > requirements
>> > > > > > > > > > listed.
>> > > > > > > > > >
>> > > > > > > > > > I have sent the script and input files via the
anon ftp
>> > site
>> > > in
>> > > > > > > > > diehl_data.
>> > > > > > > > > > The script name is rfc_test.sh, input files are
>> > > > > > > rfc_2018010512f006.nc
>> > > > > > > > > and
>> > > > > > > > > > urma_co.
>> > > > > > > > > >
>> > > > > > > > > > NOTE: I get the following from ncdump:
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > float QPF_SFC(ProjectionHr, YCells, XCells) ;
>> > > > > > > > > > QPF_SFC:long_name = "Total precipitation"
;
>> > > > > > > > > > QPF_SFC:units = "inch" ;
>> > > > > > > > > > QPF_SFC:coordinates = "longitude
latitude" ;
>> > > > > > > > > > QPF_SFC:ReferenceTime = 1515153600 ;
>> > > > > > > > > > QPF_SFC:ReferenceTimeString = "2018-01-05
>> 12:00:00
>> > > > > 00:00" ;
>> > > > > > > > > > QPF_SFC:gridType = "SCALAR" ;
>> > > > > > > > > > QPF_SFC:level = "SFC" ;
>> > > > > > > > > >
>> > > > > > > > > > Note that there is no name defined, only
long_name. I
>> > assume
>> > > > name
>> > > > > > is
>> > > > > > > > > > QPF_SFC.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > Thanks in advance for your help....Brad Diehl
>> > > > > > > > > >
>> > > > > > > > > > #!/bin/sh
>> > > > > > > > > > ###
>> > > > > > > > > > ############
>> > > > > > > > > >
>> > > > > > > > > > source /u/Brad.Diehl/met_env/tut_env
>> > > > > > > > > >
>> > > > > > > > > > regrid_data_plane rfc_2018010512f006.nc \
>> > > > > > > > > > urma_co \
>> > > > > > > > > > re_grid_out.nc \
>> > > > > > > > > > -method BUDGET \
>> > > > > > > > > > -width 2 \
>> > > > > > > > > > -v 2 \
>> > > > > > > > > > -field 'name="QPF_SFC";
level="(*,*)";'
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > DEBUG 1: Reading data file: rfc_2018010512f006.nc
>> > > > > > > > > > WARNING:
>> > > > > > > > > > WARNING: NcCfFile::open() -> could not determine
the
>> valid
>> > > > time,
>> > > > > > > using
>> > > > > > > > 0.
>> > > > > > > > > > WARNING:
>> > > > > > > > > > DEBUG 2: Input grid: Projection: Lambert
Conformal Nx:
>> 2145
>> > > Ny:
>> > > > > > 1377
>> > > > > > > > > > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000
Alpha:
>> > > > 6509.082
>> > > > > > > Cone:
>> > > > > > > > > > 0.423 Bx: 1088.0030 By: 5483.6617
>> > > > > > > > > > DEBUG 2: Output grid: Projection: Lambert
Conformal Nx:
>> > 2145
>> > > > Ny:
>> > > > > > 1377
>> > > > > > > > > > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient: 95.000
Alpha:
>> > > > 6509.082
>> > > > > > > Cone:
>> > > > > > > > > > 0.423 Bx: 1088.0030 By: 5483.6617
>> > > > > > > > > > DEBUG 2: Interpolation options: method = BUDGET,
width
>> = 2,
>> > > > > > > vld_thresh
>> > > > > > > > =
>> > > > > > > > > > 0.5
>> > > > > > > > > > ERROR :
>> > > > > > > > > > ERROR : NcCfFile::getData(NcVar *, const
LongArray &,
>> > > > DataPlane
>> > > > > &)
>> > > > > > > ->
>> > > > > > > > > > needed 3 arguments for variable QPF_SFC, got 2
>> > > > > > > > > > ERROR :
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> > >
>> >
>> >
>>
>>
>
------------------------------------------------
Subject: RFC netcdf input into regrid_data_plane
From: John Halley Gotway
Time: Tue Jan 16 09:57:09 2018
Brad,
OK, sounds good. I'll go ahead and resolve this issue for now. But
when
you have more questions, feel free to write a new question to
met_help.
Thanks,
John
On Tue, Jan 16, 2018 at 9:11 AM, Brad Diehl - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
>
> John,
>
> We had our mtg. today regarding the RFC data compliance issues. I'm
going
> to write a script that will use the combination of the nco utilities
and
> additional steps you recommended (adding forecast_reference_time,
etc.,)
> that can post process the data we have piling up and use that script
as a
> recommendation to what the RFC will have to do to make that data
compliant
> for our verification purposes. So I'm quite sure I'll need some
additional
> information from you as I go through this process, and will be in
touch.
> Thanks again for all your previous efforts.
>
>
>
> On Fri, Jan 12, 2018 at 3:16 PM, Brad Diehl - NOAA Federal <
> brad.diehl at noaa.gov> wrote:
>
> > will do, thanks again, and I'll keep you up to date as we get more
> > information.....Brad
> >
> > On Fri, Jan 12, 2018 at 3:10 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Brad,
> >>
> >> That makes sense. If they’re tagging this data as being CF-
compliant,
> it
> >> looks to me like they’re close but not quite there yet.
> >>
> >> There are some cf compliance checker tools out there, but I’ve
never
> >> actually used those.
> >>
> >> And please let me know if there are cf compliance details that
you think
> >> MET isn’t handling correctly. That’s always a possibility too!
> >>
> >> Thanks
> >> John
> >>
> >> On Fri, Jan 12, 2018 at 7:16 AM Brad Diehl - NOAA Federal via RT
<
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
> >> >
> >> > John, as always want to thank you for your efforts. After some
> internal
> >> > discussion here, we did'nt realize that a lot of pre-processing
of the
> >> RFC
> >> > data might have to be done before being used in MET. Our team
is
> getting
> >> > together next week to discuss all this, and in the end we will
likely
> >> have
> >> > to give some recommendations to the RFC team on how to make the
data
> >> > compliant for our needs.
> >> >
> >> > Thanks again, Brad
> >> >
> >> >
> >> > On Thu, Jan 11, 2018 at 10:26 PM, John Halley Gotway via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > > Brad,
> >> > >
> >> > > OK, I ran the following steps...
> >> > >
> >> > > (1) Run ncdump to dump data to ascii:
> >> > >
> >> > > ncdump rfc_2018010512f006_MOD.nc >
rfc_2018010512f006_MOD.ncdump
> >> > >
> >> > >
> >> > > (2) Edit the ascii output by adding:
> >> > >
> >> > >
> >> > > double forecast_reference_time ;
> >> > >
> >> > > forecast_reference_time:long_name =
> "reference_time"
> >> ;
> >> > >
> >> > > forecast_reference_time:units = "hours since
> >> 2018-01-05
> >> > > 12:00:00" ;
> >> > >
> >> > > ...
> >> > >
> >> > > forecast_reference_time = 0 ;
> >> > >
> >> > >
> >> > > (3) Run ncgen to generate the NetCDF file from the modified
ncdump
> >> > output:
> >> > >
> >> > >
> >> > > ncgen rfc_2018010512f006_MOD.ncdump -o
rfc_2018010512f006_MOD2.nc
> >> > >
> >> > >
> >> > > (4) Run plot_data_plane again at verbosity level 4 (which
lists the
> >> > timing
> >> > > info MET parsed from the file):
> >> > >
> >> > >
> >> > > met-6.1/bin/plot_data_plane rfc_2018010512f006_MOD.nc
QPF_SFC.ps
> >> > > 'name="QPF_SFC"; level="(0,*,*)";' -v 4
> >> > >
> >> > >
> >> > > DEBUG 4: Data plane information:
> >> > >
> >> > > DEBUG 4: plane min: 0
> >> > >
> >> > > DEBUG 4: plane max: 0.854724
> >> > >
> >> > > DEBUG 4: valid time: 20180105_160000
> >> > >
> >> > > DEBUG 4: lead time: 040000
> >> > >
> >> > > DEBUG 4: init time: 20180105_120000
> >> > >
> >> > > DEBUG 4: accum time: 000000
> >> > >
> >> > >
> >> > > These steps demonstrate that adding the
forecast_reference_time
> >> variable
> >> > > does define the model initialization time for MET. While
using
> >> > > ncdump/ncgen is fine for testing, it's not good for
systematic use.
> >> > > Instead, I'd suggest talking to your NetCDF guy about
preferred
> >> methods
> >> > for
> >> > > updating the contents of your NetCDF files.
> >> > >
> >> > >
> >> > > I copied all of these files back up to the ftp site:
> >> > >
> >> > > ftp://ftp.rap.ucar.edu/incoming/irap/met_help/diehl_
> data/for_brad
> >> > >
> >> > >
> >> > > Yes, the regrid_data_plane tool is writing an output variable
named
> >> > > "QPF_SFC_0\,\*\,\*".
> >> > > Those embedded commas are a pain... and in fact, we've
updated MET
> >> 6.1 to
> >> > > replace all instances of commas with underscores. Also note
that
> >> you do
> >> > > have control over the output variable name from
regrid_data_plane.
> >> The
> >> > > "-name" command line argument is used to name that variable
however
> >> you'd
> >> > > like. Feel free to name it whatever is convenient for
downstream
> >> > > applications.
> >> > >
> >> > >
> >> > > Hope that helps.
> >> > >
> >> > >
> >> > > John
> >> > >
> >> > > On Thu, Jan 11, 2018 at 11:22 AM, Brad Diehl - NOAA Federal
via RT <
> >> > > met_help at ucar.edu> wrote:
> >> > >
> >> > > >
> >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514 >
> >> > > >
> >> > > > I was going to ask the netcdf guru here how to go about it,
but if
> >> you
> >> > > know
> >> > > > what to do and have the time sure use ncdump/ncgen to do it
for
> our
> >> > > sample
> >> > > > case.
> >> > > >
> >> > > > BTW, one of our developers is having some issues with
trying to
> >> convert
> >> > > the
> >> > > > rfc data into geotiff files, he has done it successfully
with all
> >> the
> >> > > other
> >> > > > models.
> >> > > >
> >> > > > He chatted:
> >> > > >
> >> > > > and so my variable that I'm trying to read
> >> > > > "*QPF_SFC_0\,\*\,\*"?*
> >> > > > "*QPF_SFC_0\,\*\,\*"*
> >> > > > is coming out of MET
> >> > > >
> >> > > > Can you send them a note and ask them why the variable name
is
> such?
> >> > > > I know why it is, I think it's not parsing things properly
> >> > > > But I can't read the variable
> >> > > >
> >> > > > If he get s.t. to work I'll let you know ASAP.
> >> > > >
> >> > > > On Thu, Jan 11, 2018 at 5:56 PM, John Halley Gotway via RT
<
> >> > > > met_help at ucar.edu> wrote:
> >> > > >
> >> > > > > Brad,
> >> > > > >
> >> > > > > The NetCDF input you’re passing to regrid data plane is
> >> cf-compliant
> >> > > but
> >> > > > > the NetCDF output from it is not.
> >> > > > >
> >> > > > > We do plan to eventually change the output format to be
cg
> >> compliant
> >> > > but
> >> > > > it
> >> > > > > currently isn’t.
> >> > > > >
> >> > > > > If you rerun regrid data plane with a higher verbosity
level (-v
> >> 4 or
> >> > > -v
> >> > > > 5
> >> > > > > maybe), you should see a DEBUG message telling you that
it can’t
> >> > > > determine
> >> > > > > the init time from the input netcdf file... so it just
sets the
> >> init
> >> > > and
> >> > > > > valid times to the same time.
> >> > > > >
> >> > > > > The cf way of defining the init time is adding the
> >> > > > forecast_reference_time
> >> > > > > variable to define it. Unfortunately I don’t have a
quick NCO
> >> > command
> >> > > to
> >> > > > > add a new variable.
> >> > > > >
> >> > > > > But I could use ncdump/ncgen to do it for our sample case
if
> that
> >> > would
> >> > > > be
> >> > > > > helpful.
> >> > > > >
> >> > > > > Thanks
> >> > > > > John
> >> > > > >
> >> > > > > On Thu, Jan 11, 2018 at 9:51 AM Brad Diehl - NOAA Federal
via
> RT <
> >> > > > > met_help at ucar.edu> wrote:
> >> > > > >
> >> > > > > >
> >> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=83514
> >
> >> > > > > >
> >> > > > > > Using the recommended NCO utilities, I did the
following two
> >> steps
> >> > as
> >> > > > > > suggested:
> >> > > > > >
> >> > > > > > ncrename -v ProjectionHr,time -d ProjectionHr,time -o
> >> > > > > > rfc_2018010512f006_MOD.nc rfc_2018010512f006.nc
> >> > > > > >
> >> > > > > > ncatted -a units,time,o,c,"hours since 2018-01-05
12:00:00"
> >> > > > > > rfc_2018010512f006_MOD.nc
> >> > > > > >
> >> > > > > > And MET/regrid_data_plane ran without warning.
> >> > > > > >
> >> > > > > > For forecast data, the following was recommended:
> >> > > > > >
> >> > > > > > (3) If this is analysis data, then we're all set. But
for
> >> forecast
> >> > > > data,
> >> > > > > > you'd need to define the forecast initialization time
in the
> CF
> >> > > > compliant
> >> > > > > > way. That's done by defining a variable named
> >> > > > > "forecast_reference_time"...
> >> > > > > > looking something like this:
> >> > > > > >
> >> > > > > >
> >> > > > > > double forecast_reference_time ;
> >> > > > > >
> >> > > > > > forecast_reference_time:long_name = "reference time" ;
> >> > > > > >
> >> > > > > > forecast_reference_time:units = "seconds since 1970-01-
01" ;
> >> > > > > >
> >> > > > > > However, if I look at forecast data via ncdump that
we've
> >> processed
> >> > > > > through
> >> > > > > > regrid_data_plane, I never see forecast_reference_time
> referred
> >> to.
> >> > > For
> >> > > > > > example, see below for hrrr (gfs had same result):
> >> > > > > >
> >> > > > > > Brad.Diehl at g14a1:
> >> > > > > >
/mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
> >> > > > > > ncdump hrrrconc2p518010512qpf06f006 | grep time
> >> > > > > > APCP_06:init_time = "20180105_120000" ;
> >> > > > > > APCP_06:init_time_ut = "1515153600" ;
> >> > > > > > APCP_06:valid_time = "20180105_180000" ;
> >> > > > > > APCP_06:valid_time_ut = "1515175200" ;
> >> > > > > > APCP_06:accum_time = "060000" ;
> >> > > > > > APCP_06:accum_time_sec = 21600 ;
> >> > > > > >
> >> > > > > > Brad.Diehl at g14a1:
> >> > > > > >
/mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
> >> > > > > > ncdump hrrrconc2p518010512qpf06f006 | grep forecast
> >> > > > > > Brad.Diehl at g14a1:
> >> > > > > >
/mdlverif/noscrub/usr/Brad.Diehl/hrrr/co/grgb/1801/05/12$
> >> > > > > >
> >> > > > > > So, not sure how to proceed. Any additional help would
be
> >> > > > > > appreciated........Brad Diehl
> >> > > > > >
> >> > > > > >
> >> > > > > > On Wed, Jan 10, 2018 at 5:22 PM, John Halley Gotway via
RT <
> >> > > > > > met_help at ucar.edu> wrote:
> >> > > > > >
> >> > > > > > > Brad,
> >> > > > > > >
> >> > > > > > > I took a look at your data. This file is a NetCDF
file
> which
> >> > > > > purportedly
> >> > > > > > > follows the CF convention. However, MET is not
correctly
> >> parsing
> >> > > the
> >> > > > > > > timing info. Running though the debugger, I find
that it's
> >> > > expecting
> >> > > > > the
> >> > > > > > > timing variable to simply be named "time". To test
this
> out,
> >> I
> >> > > used
> >> > > > > the
> >> > > > > > > ncrename utility to rename the "ProjectionHr"
variable and
> >> > > dimension
> >> > > > to
> >> > > > > > > "time":
> >> > > > > > >
> >> > > > > > > ncrename -v ProjectionHr,time -d ProjectionHr,time -o
> >> > > > > > > rfc_2018010512f006_MOD.nc rfc_2018010512f006.nc
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > Rerunning reveals the next warning:
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > WARNING: parse_cf_time_string() -> unexpected NetCDF
CF
> >> > convention
> >> > > > time
> >> > > > > > > unit "2018-01-05 12:00:00 00:00"
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > For the CF convention, the "units" attribute for time
should
> >> look
> >> > > > like
> >> > > > > > this
> >> > > > > > > "[TIME UNITS] since [REFERENCE TIME]". To test, I
used the
> >> > ncatted
> >> > > > > > utility
> >> > > > > > > to modify your sample file:
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > ncatted -a units,time,o,c,"hours since 2018-01-05
12:00:00"
> >> > > > > > > rfc_2018010512f006_MOD.nc
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > And finally that allows MET run without warning:
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > met-6.1/bin/plot_data_plane rfc_2018010512f006_MOD.nc
> >> QPF_SFC.ps
> >> > > > > > > 'name="QPF_SFC"; level="(0,*,*)";'
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > And the resulting plot is attached.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > In all, there are 3 issues:
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > (1) MET expects the time variable to be named "time".
I'm
> not
> >> > sure
> >> > > > > > whether
> >> > > > > > > this is actually a CF requirement. If not, please
let me
> >> know,
> >> > and
> >> > > > we
> >> > > > > > can
> >> > > > > > > take a closer look at the CF convention to see what
MET
> >> *should*
> >> > be
> >> > > > > > doing.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > (2) The time "units" attribute must be named "[TIME
UNITS]
> >> since
> >> > > > > > [REFERENCE
> >> > > > > > > TIME]". I'm pretty confident that this a CF
requirement.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > (3) If this is analysis data, then we're all set.
But for
> >> > forecast
> >> > > > > data,
> >> > > > > > > you'd need to define the forecast initialization time
in the
> >> CF
> >> > > > > compliant
> >> > > > > > > way. That's done by defining a variable named
> >> > > > > > "forecast_reference_time"...
> >> > > > > > > looking something like this:
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > double forecast_reference_time ;
> >> > > > > > >
> >> > > > > > > forecast_reference_time:long_name = "reference time"
;
> >> > > > > > >
> >> > > > > > > forecast_reference_time:units = "seconds since 1970-
01-01" ;
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > Of course the units can be defined as seconds,
minutes,
> hour,
> >> > days,
> >> > > > and
> >> > > > > > so
> >> > > > > > > on.
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > Thanks,
> >> > > > > > > John
> >> > > > > > >
> >> > > > > > > On Wed, Jan 10, 2018 at 7:00 AM, Brad Diehl - NOAA
Federal
> via
> >> > RT <
> >> > > > > > > met_help at ucar.edu> wrote:
> >> > > > > > >
> >> > > > > > > >
> >> > > > > > > > <URL: https://rt.rap.ucar.edu/rt/Tic
> >> ket/Display.html?id=83514
> >> > >
> >> > > > > > > >
> >> > > > > > > > Once again, thanks for the help, I'll have to be
more
> aware
> >> of
> >> > > the
> >> > > > > > > > dimensions moving forward. And the solution of
using
> -field
> >> > > > > > > > 'name="QPF_SFC"; level="(0,*,*)";' worked just
fine. Many
> >> > thanks
> >> > > > > again.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > However, I continue to get the warning below from
> >> > > > regrid_data_plane.
> >> > > > > > Not
> >> > > > > > > > sure if its a show stopper but does'nt appear to
be. Is
> >> there
> >> > > > > anything
> >> > > > > > I
> >> > > > > > > > can do? If not, I'll consider this ticket closed.
> >> > > > > > > >
> >> > > > > > > > WARNING:
> >> > > > > > > > WARNING: NcCfFile::open() -> could not determine
the valid
> >> > time,
> >> > > > > using
> >> > > > > > 0.
> >> > > > > > > > WARNING:
> >> > > > > > > >
> >> > > > > > > > On Tue, Jan 9, 2018 at 3:01 PM, John Halley Gotway
via RT
> <
> >> > > > > > > > met_help at ucar.edu
> >> > > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Hello Brad,
> >> > > > > > > > >
> >> > > > > > > > > I see you're having trouble passing a NetCDF file
to the
> >> > > > > > > > regrid_data_plane
> >> > > > > > > > > tool. The problem is how you've specified the
"level"
> >> value
> >> > in
> >> > > > the
> >> > > > > > > > command
> >> > > > > > > > > line argument. You specified "(*,*)" (i.e. 2
> dimensions)
> >> but
> >> > > > that
> >> > > > > > > > variable
> >> > > > > > > > > actually has 3 dimensions:
> >> > > > > > > > > QPF_SFC(ProjectionHr, YCells, XCells)
> >> > > > > > > > >
> >> > > > > > > > > That's what this error message is telling you:
> >> > > > > > > > > ERROR : NcCfFile::getData(NcVar *, const
LongArray
> &,
> >> > > > DataPlane
> >> > > > > > &)
> >> > > > > > > ->
> >> > > > > > > > > needed 3 arguments for variable QPF_SFC, got 2
> >> > > > > > > > >
> >> > > > > > > > > Please try using this:
> >> > > > > > > > > -field 'name="QPF_SFC"; level="(0,*,*)";'
> >> > > > > > > > >
> >> > > > > > > > > That'll process the first 2D slice of data from
that 3D
> >> > > variable.
> >> > > > > > > > >
> >> > > > > > > > > Thanks,
> >> > > > > > > > > John
> >> > > > > > > > >
> >> > > > > > > > > On Mon, Jan 8, 2018 at 10:00 AM, Brad Diehl -
NOAA
> Federal
> >> > via
> >> > > > RT <
> >> > > > > > > > > met_help at ucar.edu> wrote:
> >> > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Mon Jan 08 10:00:01 2018: Request 83514 was
acted
> upon.
> >> > > > > > > > > > Transaction: Ticket created by
brad.diehl at noaa.gov
> >> > > > > > > > > > Queue: met_help
> >> > > > > > > > > > Subject: RFC netcdf input into
regrid_data_plane
> >> > > > > > > > > > Owner: Nobody
> >> > > > > > > > > > Requestors: brad.diehl at noaa.gov
> >> > > > > > > > > > Status: new
> >> > > > > > > > > > Ticket <URL: https://rt.rap.ucar.edu/rt/
> >> > > > > > > Ticket/Display.html?id=83514
> >> > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > To whom it may concern,
> >> > > > > > > > > >
> >> > > > > > > > > > I have been working with ABRFC Mosaic QPF in
netcdf
> >> format
> >> > > and
> >> > > > > > using
> >> > > > > > > it
> >> > > > > > > > > as
> >> > > > > > > > > > input into regrid_data_plane. Using the simple
script
> >> > below,
> >> > > I
> >> > > > > get
> >> > > > > > > > > > following results (pasted below script). I
was'nt sure
> >> how
> >> > > > severe
> >> > > > > > the
> >> > > > > > > > > > ERRORS were and wanted to double check with the
help
> >> desk.
> >> > I
> >> > > > did
> >> > > > > > > notice
> >> > > > > > > > > in
> >> > > > > > > > > > the MET FAQ that to "use gridded NetCDF files
in MET,
> >> the
> >> > > files
> >> > > > > > need
> >> > > > > > > to
> >> > > > > > > > > > look like the output of the pcp_combine tool"
with
> some
> >> > > > > > requirements
> >> > > > > > > > > > listed.
> >> > > > > > > > > >
> >> > > > > > > > > > I have sent the script and input files via the
anon
> ftp
> >> > site
> >> > > in
> >> > > > > > > > > diehl_data.
> >> > > > > > > > > > The script name is rfc_test.sh, input files are
> >> > > > > > > rfc_2018010512f006.nc
> >> > > > > > > > > and
> >> > > > > > > > > > urma_co.
> >> > > > > > > > > >
> >> > > > > > > > > > NOTE: I get the following from ncdump:
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > float QPF_SFC(ProjectionHr, YCells, XCells)
;
> >> > > > > > > > > > QPF_SFC:long_name = "Total
precipitation" ;
> >> > > > > > > > > > QPF_SFC:units = "inch" ;
> >> > > > > > > > > > QPF_SFC:coordinates = "longitude
latitude" ;
> >> > > > > > > > > > QPF_SFC:ReferenceTime = 1515153600 ;
> >> > > > > > > > > > QPF_SFC:ReferenceTimeString = "2018-01-
05
> >> 12:00:00
> >> > > > > 00:00" ;
> >> > > > > > > > > > QPF_SFC:gridType = "SCALAR" ;
> >> > > > > > > > > > QPF_SFC:level = "SFC" ;
> >> > > > > > > > > >
> >> > > > > > > > > > Note that there is no name defined, only
long_name. I
> >> > assume
> >> > > > name
> >> > > > > > is
> >> > > > > > > > > > QPF_SFC.
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks in advance for your help....Brad Diehl
> >> > > > > > > > > >
> >> > > > > > > > > > #!/bin/sh
> >> > > > > > > > > > ###
> >> > > > > > > > > > ############
> >> > > > > > > > > >
> >> > > > > > > > > > source /u/Brad.Diehl/met_env/tut_env
> >> > > > > > > > > >
> >> > > > > > > > > > regrid_data_plane rfc_2018010512f006.nc \
> >> > > > > > > > > > urma_co \
> >> > > > > > > > > > re_grid_out.nc \
> >> > > > > > > > > > -method BUDGET \
> >> > > > > > > > > > -width 2 \
> >> > > > > > > > > > -v 2 \
> >> > > > > > > > > > -field 'name="QPF_SFC";
level="(*,*)";'
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > DEBUG 1: Reading data file:
rfc_2018010512f006.nc
> >> > > > > > > > > > WARNING:
> >> > > > > > > > > > WARNING: NcCfFile::open() -> could not
determine the
> >> valid
> >> > > > time,
> >> > > > > > > using
> >> > > > > > > > 0.
> >> > > > > > > > > > WARNING:
> >> > > > > > > > > > DEBUG 2: Input grid: Projection: Lambert
Conformal Nx:
> >> 2145
> >> > > Ny:
> >> > > > > > 1377
> >> > > > > > > > > > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient:
95.000
> Alpha:
> >> > > > 6509.082
> >> > > > > > > Cone:
> >> > > > > > > > > > 0.423 Bx: 1088.0030 By: 5483.6617
> >> > > > > > > > > > DEBUG 2: Output grid: Projection: Lambert
Conformal
> Nx:
> >> > 2145
> >> > > > Ny:
> >> > > > > > 1377
> >> > > > > > > > > > Lat_LL: 20.192 Lon_LL: 121.554 Lon_orient:
95.000
> Alpha:
> >> > > > 6509.082
> >> > > > > > > Cone:
> >> > > > > > > > > > 0.423 Bx: 1088.0030 By: 5483.6617
> >> > > > > > > > > > DEBUG 2: Interpolation options: method =
BUDGET, width
> >> = 2,
> >> > > > > > > vld_thresh
> >> > > > > > > > =
> >> > > > > > > > > > 0.5
> >> > > > > > > > > > ERROR :
> >> > > > > > > > > > ERROR : NcCfFile::getData(NcVar *, const
LongArray &,
> >> > > > DataPlane
> >> > > > > &)
> >> > > > > > > ->
> >> > > > > > > > > > needed 3 arguments for variable QPF_SFC, got 2
> >> > > > > > > > > > ERROR :
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> >
> >> >
> >>
> >>
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list