[Met_help] [rt.rap.ucar.edu #97766] History for Need help with GridDiag
John Halley Gotway via RT
met_help at ucar.edu
Wed Feb 10 09:03:28 MST 2021
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello,
I'm trying to get GridDiag to work but struggling to understand some of the specifics of the netcdf headers that are required. I have attached two different files... the one works sometimes, the other doesn't work (see below).
I've attached the output I get as well as the input sample files and relevant config files. I have to say I haven't tried the METplus wrapper yet because I think I have more fundamental problems!
1. What does this "bnds" variable need to be?
2. There are two lead times in the glosea file but both the data sets are called "precipitation_amount"... that seems to be causing a problem? Maybe?
3. Running GridDiag with just the GPM file works. Running GridDiag with just the forecast works sometimes?
I think it does have to do with the code being unable to decode the time info correctly.
Any help gratefully received!
Thanks
Marion
DEBUG 1: Default Config File: /data/users/cfrd/MET_requirements/MET9.0/share/met/config/GridDiagConfig_default
DEBUG 1: User Config File: GridDiag.glosea
DEBUG 2: Read config files.
DEBUG 2: Enter get_mtddf.
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable "hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1970-01-01 00:00:00" as a reference time of 19700101_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> parsing units for the forecast_reference_time variable "hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1970-01-01 00:00:00" as a reference time of 19700101_000000 and 3600 second(s) per time step.
WARNING:
WARNING: get_nc_var(NcFile) --> The variable "bnds" does not exist!
WARNING:
DEBUG 2: Process configuration.
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_NcCF".
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_NcCF".
DEBUG 3: Use the grid defined by file "/scratch/fryl/GPM/gpm_2019060100_2019060200.nc".
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable "hours since 1970-01-01"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1970-01-01" as a reference time of 19700101_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> could not extract init time from the "forecast_reference_time" variable.
DEBUG 4: NcCfFile::open() -> could not extract init time from file name.
DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 3600 Ny: 1800 lat_ll: -89.950 lon_ll: 179.950 delta_lat: 0.100 delta_lon: 0.100
DEBUG 2: Processing masking regions.
DEBUG 2: Initializing precipitation_amount(0,*,*) histogram
DEBUG 2: Initializing precipitation_amount(*,*) histogram
DEBUG 2: Initializing precipitation_amount(0,*,*)_precipitation_amount(*,*) joint histogram
DEBUG 1: /scratch/frmm/MET/output/grid_diag/glosea//grid_diag_glosea_2019060100.nc
DEBUG 1: Length of configuration "data.field" = 2
DEBUG 1: Length of data file list = 2
DEBUG 1: Series defined by the data file list of length 2.
DEBUG 2: 100
DEBUG 2: Processing series entry 1: precipitation_amount: precipitation_amount(0,*,*)
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable "hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1970-01-01 00:00:00" as a reference time of 19700101_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> parsing units for the forecast_reference_time variable "hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1970-01-01 00:00:00" as a reference time of 19700101_000000 and 3600 second(s) per time step.
WARNING:
WARNING: get_nc_var(NcFile) --> The variable "bnds" does not exist!
WARNING:
DEBUG 4:
DEBUG 4: Data plane information:
DEBUG 4: plane min: 0
DEBUG 4: plane max: 262.128
DEBUG 4: valid time: 20190601_120000
DEBUG 4: lead time: 120000
DEBUG 4: init time: 20190601_000000
DEBUG 4: accum time: 000000
DEBUG 1: Regridding field precipitation_amount(0,*,*) to the verification grid.
DEBUG 2: Processing series entry 1: precipitation_amount: precipitation_amount(*,*)
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable "hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1970-01-01 00:00:00" as a reference time of 19700101_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> parsing units for the forecast_reference_time variable "hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1970-01-01 00:00:00" as a reference time of 19700101_000000 and 3600 second(s) per time step.
WARNING:
WARNING: get_nc_var(NcFile) --> The variable "bnds" does not exist!
WARNING:
WARNING:
WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) -> returns the first available time for "precipitation_amount" variable).
WARNING:
ERROR :
ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &) const -> needed 3 arguments for variable precipitation_amount, got 2
ERROR :
#########
FINISHED!
#########
Fri Dec 4 14:53:34 GMT 2020
--
Dr Marion Mittermaier Manager: Model diagnostics and novel verification
Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
Tel: +44 (0) 330 135 1604
E-mail: marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk> http://www.metoffice.gov.uk<http://www.metoffice.gov.uk/>
http://www.metoffice.gov.uk/research/people/marion-mittermaier
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Need help with GridDiag
From: John Halley Gotway
Time: Fri Dec 04 23:27:19 2020
Marion,
I see you have questions about the grid_diag tool in MET. Whenever
folks
send us data, I always start by running the plot_data_plane tool to
confirm
that MET can read that data. And sure enough, it can read both of the
files
you sent:
*plot_data_plane gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc> gpm.ps <http://gpm.ps>
'name="precipitation_amount"; level="(*,*)";' -v 4*
*plot_data_plane glosea_2019060100_001.nc
<http://glosea_2019060100_001.nc>
glosea.ps <http://glosea.ps> 'name="precipitation_amount";
level="(0,*,*)";' -v 4*
A screenshot of the resulting images is attached. So while MET can
read
them, I'm pretty suspicious of the range of values for the GPM data...
from
0 up to 9.96921e+36.
Perhaps there's an issue there? The glosea values from 0 to 262 seem
much
more reasonable for precip!
[image: Screen Shot 2020-12-04 at 10.52.01 PM.png]
Now on to your question about the grid_diag tool. First, I can confirm
that
I'm able to reproduce that exact same behavior you describe:
*grid_diag -data glosea_2019060100_001.nc
<http://glosea_2019060100_001.nc> -config GridDiag.glosea -out
grid_diag_out.nc <http://grid_diag_out.nc>*
ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &)
const
-> needed 3 arguments for variable precipitation_amount, got 2
Because of the common variable names, this could get confusing. But to
be
clear...
- glosea_2019060100_001.nc contains precipitation_amount(time,
latitude,
longitude)... that's 3 dimensions.
- gpm_2019060100_2019060200.nc contains precipitation_amount(latitude,
longitude)... that's 2 dimensions.
You listed two fields in the config file...
precipitation_amount(0,*,*) followed by precipitation_amount(*,*).
And I'm guessing you passed as input the glosea file followed by the
GPM
file. And you expected that Grid-Diag would read the first field from
the
first file and the second field from the second file.
But that's where the confusion lies. Grid-Diag attempts to read all
requested input fields from all specified input files. And that causes
the
error we're seeing. It tries to read the second field
"precipitation_amount(*,*)" from the first (Glosea) file, but it can't
because that one has "precipitation_amount" with 3 dimensions, not 2.
So the issue is in the processing logic of the tool. In general, it
was
designed to inventory data over one or more timesteps and derive 1 and
2
dimensional PDF's of the data it inventories. However, it expects to
read
all requested fields from all specified input files. Otherwise, you'll
get
an error. It just isn't set up to compare PDF's from different input
data
sources.
And if you have suggestions for how we could clarify this point in the
documentation, please let us know.
If you have more questions about the processing logic, I'll direct you
to
David Fillmore (cc'ed here). He's the engineer who knows the most
about
this tool. And hopefully he can correct anything I may have stated
incorrectly!
Thanks,
John Halley Gotway
On Fri, Dec 4, 2020 at 8:09 AM marion.mittermaier at metoffice.gov.uk via
RT <
met_help at ucar.edu> wrote:
>
> Fri Dec 04 08:09:16 2020: Request 97766 was acted upon.
> Transaction: Ticket created by marion.mittermaier at metoffice.gov.uk
> Queue: met_help
> Subject: Need help with GridDiag
> Owner: Nobody
> Requestors: marion.mittermaier at metoffice.gov.uk
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>
>
> Hello,
>
> I'm trying to get GridDiag to work but struggling to understand some
of
> the specifics of the netcdf headers that are required. I have
attached two
> different files... the one works sometimes, the other doesn't work
(see
> below).
>
> I've attached the output I get as well as the input sample files and
> relevant config files. I have to say I haven't tried the METplus
wrapper
> yet because I think I have more fundamental problems!
>
>
> 1. What does this "bnds" variable need to be?
> 2. There are two lead times in the glosea file but both the data
sets
> are called "precipitation_amount"... that seems to be causing a
problem?
> Maybe?
> 3. Running GridDiag with just the GPM file works. Running
GridDiag with
> just the forecast works sometimes?
>
> I think it does have to do with the code being unable to decode the
time
> info correctly.
>
> Any help gratefully received!
>
> Thanks
> Marion
>
> DEBUG 1: Default Config File:
>
/data/users/cfrd/MET_requirements/MET9.0/share/met/config/GridDiagConfig_default
> DEBUG 1: User Config File: GridDiag.glosea
> DEBUG 2: Read config files.
> DEBUG 2: Enter get_mtddf.
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours
> since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit
> string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> DEBUG 4: NcCfFile::open() -> parsing units for the
forecast_reference_time
> variable "hours since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit
> string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> WARNING:
> WARNING: get_nc_var(NcFile) --> The variable "bnds" does not exist!
> WARNING:
> DEBUG 2: Process configuration.
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> type "FileType_NcCF".
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> type "FileType_NcCF".
> DEBUG 3: Use the grid defined by file "/scratch/fryl/GPM/
> gpm_2019060100_2019060200.nc".
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours
> since 1970-01-01"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit
> string "hours since 1970-01-01" as a reference time of
19700101_000000 and
> 3600 second(s) per time step.
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
> DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 3600 Ny: 1800
lat_ll:
> -89.950 lon_ll: 179.950 delta_lat: 0.100 delta_lon: 0.100
> DEBUG 2: Processing masking regions.
> DEBUG 2: Initializing precipitation_amount(0,*,*) histogram
> DEBUG 2: Initializing precipitation_amount(*,*) histogram
> DEBUG 2: Initializing
> precipitation_amount(0,*,*)_precipitation_amount(*,*) joint
histogram
> DEBUG 1: /scratch/frmm/MET/output/grid_diag/glosea//
> grid_diag_glosea_2019060100.nc
> DEBUG 1: Length of configuration "data.field" = 2
> DEBUG 1: Length of data file list = 2
> DEBUG 1: Series defined by the data file list of length 2.
> DEBUG 2: 100
> DEBUG 2: Processing series entry 1: precipitation_amount:
> precipitation_amount(0,*,*)
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours
> since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit
> string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> DEBUG 4: NcCfFile::open() -> parsing units for the
forecast_reference_time
> variable "hours since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit
> string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> WARNING:
> WARNING: get_nc_var(NcFile) --> The variable "bnds" does not exist!
> WARNING:
> DEBUG 4:
> DEBUG 4: Data plane information:
> DEBUG 4: plane min: 0
> DEBUG 4: plane max: 262.128
> DEBUG 4: valid time: 20190601_120000
> DEBUG 4: lead time: 120000
> DEBUG 4: init time: 20190601_000000
> DEBUG 4: accum time: 000000
> DEBUG 1: Regridding field precipitation_amount(0,*,*) to the
verification
> grid.
> DEBUG 2: Processing series entry 1: precipitation_amount:
> precipitation_amount(*,*)
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours
> since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit
> string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> DEBUG 4: NcCfFile::open() -> parsing units for the
forecast_reference_time
> variable "hours since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit
> string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> WARNING:
> WARNING: get_nc_var(NcFile) --> The variable "bnds" does not exist!
> WARNING:
> WARNING:
> WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) ->
returns
> the first available time for "precipitation_amount" variable).
> WARNING:
> ERROR :
> ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &)
const
> -> needed 3 arguments for variable precipitation_amount, got 2
> ERROR :
> #########
> FINISHED!
> #########
> Fri Dec 4 14:53:34 GMT 2020
>
> --
> Dr Marion Mittermaier Manager: Model diagnostics and novel
verification
>
> Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> Tel: +44 (0) 330 135 1604
> E-mail: marion.mittermaier at metoffice.gov.uk<mailto:
> marion.mittermaier at metoffice.gov.uk> http://www.metoffice.gov.uk<
> http://www.metoffice.gov.uk/>
>
> http://www.metoffice.gov.uk/research/people/marion-mittermaier
>
>
>
------------------------------------------------
Subject: Need help with GridDiag
From: John Halley Gotway
Time: Fri Dec 04 23:30:27 2020
Marion,
It occurs to me that perhaps Grid-Diag could be made more flexible if
we
changed that error message to a debug message in this context. It
could
*ATTEMPT* to read all input fields from all input files... but when
it's
unable to, have it print a log message stating that rather than
erroring
out.
That would enable you to compute 2D pdf's using different variables
from
different input files.
Do you think that logic is preferable?
Thanks,
John
On Fri, Dec 4, 2020 at 11:27 PM The RT System itself via RT <
met_help at ucar.edu> wrote:
>
> Fri Dec 04 23:27:19 2020: Request 97766 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by RT_System
> Queue: met_help
> Subject: Need help with GridDiag
> Owner: johnhg
> Requestors: marion.mittermaier at metoffice.gov.uk
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>
>
> This transaction appears to have no content
>
------------------------------------------------
Subject: Need help with GridDiag
From: John Halley Gotway
Time: Fri Dec 04 23:41:26 2020
Marion,
Ugh, I should've just handed this ticket off to David in the first
place. I
did more testing and was able to get it to work. I just had to use
separate
"-data" command line options for each input data source.
So this command fails:
grid_diag -data glosea_2019060100_001.nc gpm_2019060100_2019060200.nc
-config GridDiag.glosea -out grid_diag_out.nc
But this command runs fine:
*grid_diag -data glosea_2019060100_001.nc
<http://glosea_2019060100_001.nc> -data gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc> -config GridDiag.glosea -out
grid_diag_out.nc <http://grid_diag_out.nc>*
In the second one, I've specified "-data" separately for each data
source.
Although, you'll see warning about the range of the data extending
beyond
the range of the bins defined in the config file.
Sorry to have such a rambling and confusing response!
So to summarize, I believe the logic is:
1. If "-data" is used once, all specified input fields are read from
all of
those data files.
2. If "-data" is used more than once, the number of "-data" entries
must
match the number of fields defined in the config file.
If not, you get an error message.
If so, only the i-th field is read from the files in the i-th
-data
source.
John
On Fri, Dec 4, 2020 at 11:30 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Marion,
>
> It occurs to me that perhaps Grid-Diag could be made more flexible
if we
> changed that error message to a debug message in this context. It
could
> *ATTEMPT* to read all input fields from all input files... but when
it's
> unable to, have it print a log message stating that rather than
erroring
> out.
>
> That would enable you to compute 2D pdf's using different variables
from
> different input files.
>
> Do you think that logic is preferable?
>
> Thanks,
> John
>
> On Fri, Dec 4, 2020 at 11:27 PM The RT System itself via RT <
> met_help at ucar.edu> wrote:
>
>>
>> Fri Dec 04 23:27:19 2020: Request 97766 was acted upon.
>> Transaction: Given to johnhg (John Halley Gotway) by RT_System
>> Queue: met_help
>> Subject: Need help with GridDiag
>> Owner: johnhg
>> Requestors: marion.mittermaier at metoffice.gov.uk
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>>
>>
>> This transaction appears to have no content
>>
>
------------------------------------------------
Subject: Need help with GridDiag
From: marion.mittermaier at metoffice.gov.uk
Time: Mon Dec 07 14:49:01 2020
Thanks John.
That is helpful. Just to follow up.
I tested to see if I could get it to work by making sure that the
standard_names and long_names are different. I also chose to only put
one lead time in the forecast file and used that with the GPM data.
But, as the output below shows, I’m thinking this isn’t the problem
per se.
It’s the dimensions… I’ve renamed the forecast fields as precip_amount
and only put one lead time into a forecast file but this file STILL
has 3 dimensions so the code still falls over. The forecast file would
need more “surgery” to get these to have the same dimensions but given
what it is I wouldn’t want to do this really.
DEBUG 1: Default Config File:
/data/users/cfrd/MET_requirements/MET9.0/share/met/config/GridDiagConfig_default
DEBUG 1: User Config File: GridDiag.glosea
DEBUG 2: Read config files.
DEBUG 2: Enter get_mtddf.
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit string "hours since 1970-01-01 00:00:00" as a reference time of
19700101_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> parsing units for the
forecast_reference_time variable "hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit string "hours since 1970-01-01 00:00:00" as a reference time of
19700101_000000 and 3600 second(s) per time step.
WARNING:
WARNING: get_nc_var(NcFile) --> The variable "bnds" does not exist!
WARNING:
DEBUG 2: Process configuration.
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object
of type "FileType_NcCF".
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object
of type "FileType_NcCF".
DEBUG 3: Use the grid defined by file
"/scratch/fryl/GPM/gpm_2019060100_2019060200.nc".
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours since 1970-01-01"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit string "hours since 1970-01-01" as a reference time of
19700101_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> could not extract init time from the
"forecast_reference_time" variable.
DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 3600 Ny: 1800
lat_ll: -89.950 lon_ll: 179.950 delta_lat: 0.100 delta_lon: 0.100
DEBUG 2: Processing masking regions.
DEBUG 2: Initializing precip_amount(*,*) histogram
DEBUG 2: Initializing precipitation_amount(*,*) histogram
DEBUG 2: Initializing precip_amount(*,*)_precipitation_amount(*,*)
joint histogram
DEBUG 1:
/scratch/frmm/MET/output/grid_diag/glosea//grid_diag_glosea_2019060100.nc
DEBUG 1: Length of configuration "data.field" = 2
DEBUG 1: Length of data file list = 2
DEBUG 1: Series defined by the data file list of length 2.
DEBUG 2: 100
DEBUG 2: Processing series entry 1: precip_amount: precip_amount(*,*)
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit string "hours since 1970-01-01 00:00:00" as a reference time of
19700101_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> parsing units for the
forecast_reference_time variable "hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit string "hours since 1970-01-01 00:00:00" as a reference time of
19700101_000000 and 3600 second(s) per time step.
WARNING:
WARNING: get_nc_var(NcFile) --> The variable "bnds" does not exist!
WARNING:
WARNING:
WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) ->
returns the first available time for "precip_amount" variable).
WARNING:
ERROR :
ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &)
const -> needed 3 arguments for variable precip_amount, got 2
ERROR :
#########
FINISHED!
#########
Mon Dec 7 11:29:49 GMT 2020
I guess the conclusion is that joint distributions between forecast
and observations (of the same variable) aren’t possible at this stage
without doing something significant to the forecast files (to reduce
dimensions) but I can run it sequentially and get the marginal
distributions which I can work with for the moment. Going forward I’m
not sure what to do though, other than make some changes to the
forecast files (i.e. lose the whole “forecast_period” part of the
netcdf header).
Marion
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 05 December 2020 06:27
To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
This email was received from an external source. Always check sender
details, links & attachments.
Marion,
I see you have questions about the grid_diag tool in MET. Whenever
folks send us data, I always start by running the plot_data_plane tool
to confirm that MET can read that data. And sure enough, it can read
both of the files you sent:
*plot_data_plane gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc> gpm.ps <http://gpm.ps>
'name="precipitation_amount"; level="(*,*)";' -v 4* *plot_data_plane
glosea_2019060100_001.nc <http://glosea_2019060100_001.nc> glosea.ps
<http://glosea.ps> 'name="precipitation_amount"; level="(0,*,*)";' -v
4*
A screenshot of the resulting images is attached. So while MET can
read them, I'm pretty suspicious of the range of values for the GPM
data... from
0 up to 9.96921e+36.
Perhaps there's an issue there? The glosea values from 0 to 262 seem
much more reasonable for precip!
[image: Screen Shot 2020-12-04 at 10.52.01 PM.png]
Now on to your question about the grid_diag tool. First, I can confirm
that I'm able to reproduce that exact same behavior you describe:
*grid_diag -data glosea_2019060100_001.nc
<http://glosea_2019060100_001.nc> -config GridDiag.glosea -out
grid_diag_out.nc <http://grid_diag_out.nc>*
ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &)
const
-> needed 3 arguments for variable precipitation_amount, got 2
Because of the common variable names, this could get confusing. But to
be clear...
- glosea_2019060100_001.nc contains precipitation_amount(time,
latitude, longitude)... that's 3 dimensions.
- gpm_2019060100_2019060200.nc contains precipitation_amount(latitude,
longitude)... that's 2 dimensions.
You listed two fields in the config file...
precipitation_amount(0,*,*) followed by precipitation_amount(*,*).
And I'm guessing you passed as input the glosea file followed by the
GPM file. And you expected that Grid-Diag would read the first field
from the first file and the second field from the second file.
But that's where the confusion lies. Grid-Diag attempts to read all
requested input fields from all specified input files. And that causes
the error we're seeing. It tries to read the second field
"precipitation_amount(*,*)" from the first (Glosea) file, but it can't
because that one has "precipitation_amount" with 3 dimensions, not 2.
So the issue is in the processing logic of the tool. In general, it
was designed to inventory data over one or more timesteps and derive 1
and 2 dimensional PDF's of the data it inventories. However, it
expects to read all requested fields from all specified input files.
Otherwise, you'll get an error. It just isn't set up to compare PDF's
from different input data sources.
And if you have suggestions for how we could clarify this point in the
documentation, please let us know.
If you have more questions about the processing logic, I'll direct you
to David Fillmore (cc'ed here). He's the engineer who knows the most
about this tool. And hopefully he can correct anything I may have
stated incorrectly!
Thanks,
John Halley Gotway
On Fri, Dec 4, 2020 at 8:09 AM
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>
via RT < met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
>
> Fri Dec 04 08:09:16 2020: Request 97766 was acted upon.
> Transaction: Ticket created by
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>
> Queue: met_help
> Subject: Need help with GridDiag
> Owner: Nobody
> Requestors:
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
> >
>
>
> Hello,
>
> I'm trying to get GridDiag to work but struggling to understand some
> of the specifics of the netcdf headers that are required. I have
> attached two different files... the one works sometimes, the other
> doesn't work (see below).
>
> I've attached the output I get as well as the input sample files and
> relevant config files. I have to say I haven't tried the METplus
> wrapper yet because I think I have more fundamental problems!
>
>
> 1. What does this "bnds" variable need to be?
> 2. There are two lead times in the glosea file but both the data
> sets are called "precipitation_amount"... that seems to be causing a
problem?
> Maybe?
> 3. Running GridDiag with just the GPM file works. Running
GridDiag
> with just the forecast works sometimes?
>
> I think it does have to do with the code being unable to decode the
> time info correctly.
>
> Any help gratefully received!
>
> Thanks
> Marion
>
> DEBUG 1: Default Config File:
>
/data/users/cfrd/MET_requirements/MET9.0/share/met/config/GridDiagConf
> ig_default DEBUG 1: User Config File: GridDiag.glosea DEBUG 2: Read
> config files.
> DEBUG 2: Enter get_mtddf.
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> "hours since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> DEBUG 4: NcCfFile::open() -> parsing units for the
> forecast_reference_time variable "hours since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> WARNING:
> WARNING: get_nc_var(NcFile) --> The variable "bnds" does not exist!
> WARNING:
> DEBUG 2: Process configuration.
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of type "FileType_NcCF".
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of type "FileType_NcCF".
> DEBUG 3: Use the grid defined by file "/scratch/fryl/GPM/
> gpm_2019060100_2019060200.nc".
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> "hours since 1970-01-01"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01" as a reference time of
> 19700101_000000 and
> 3600 second(s) per time step.
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
> DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 3600 Ny: 1800
lat_ll:
> -89.950 lon_ll: 179.950 delta_lat: 0.100 delta_lon: 0.100 DEBUG 2:
> Processing masking regions.
> DEBUG 2: Initializing precipitation_amount(0,*,*) histogram DEBUG 2:
> Initializing precipitation_amount(*,*) histogram DEBUG 2:
Initializing
> precipitation_amount(0,*,*)_precipitation_amount(*,*) joint
histogram
> DEBUG 1: /scratch/frmm/MET/output/grid_diag/glosea//
> grid_diag_glosea_2019060100.nc
> DEBUG 1: Length of configuration "data.field" = 2
> DEBUG 1: Length of data file list = 2
> DEBUG 1: Series defined by the data file list of length 2.
> DEBUG 2: 100
> DEBUG 2: Processing series entry 1: precipitation_amount:
> precipitation_amount(0,*,*)
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> "hours since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> DEBUG 4: NcCfFile::open() -> parsing units for the
> forecast_reference_time variable "hours since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> WARNING:
> WARNING: get_nc_var(NcFile) --> The variable "bnds" does not exist!
> WARNING:
> DEBUG 4:
> DEBUG 4: Data plane information:
> DEBUG 4: plane min: 0
> DEBUG 4: plane max: 262.128
> DEBUG 4: valid time: 20190601_120000
> DEBUG 4: lead time: 120000
> DEBUG 4: init time: 20190601_000000
> DEBUG 4: accum time: 000000
> DEBUG 1: Regridding field precipitation_amount(0,*,*) to the
> verification grid.
> DEBUG 2: Processing series entry 1: precipitation_amount:
> precipitation_amount(*,*)
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> "hours since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> DEBUG 4: NcCfFile::open() -> parsing units for the
> forecast_reference_time variable "hours since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> WARNING:
> WARNING: get_nc_var(NcFile) --> The variable "bnds" does not exist!
> WARNING:
> WARNING:
> WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) ->
> returns the first available time for "precipitation_amount"
variable).
> WARNING:
> ERROR :
> ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &)
> const
> -> needed 3 arguments for variable precipitation_amount, got 2
> ERROR :
> #########
> FINISHED!
> #########
> Fri Dec 4 14:53:34 GMT 2020
>
> --
> Dr Marion Mittermaier Manager: Model diagnostics and novel
verification
>
> Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> Tel: +44 (0) 330 135 1604
> E-mail:
marion.mittermaier at metoffice.gov.uk<mailto<mailto:marion.mittermaier at metoffice.gov.uk%3cmailto>:
>
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>>
http://www.metoffice.gov.uk<<http://www.metoffice.gov.uk%3c>
> http://www.metoffice.gov.uk/>
>
> http://www.metoffice.gov.uk/research/people/marion-mittermaier
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #97766] Need help with GridDiag
From: marion.mittermaier at metoffice.gov.uk
Time: Tue Dec 08 08:30:48 2020
Hi John,
one final question for GridDiag.
The bins are set up to be lower bound inclusive? i.e.
n_bins = 1000;
range = [0, 1000]; // kg/m^2
means that the bin is set up as [0,1) i.e. 0 <= x < 1?
I'm presuming so.... and the upper bound would be [999,1000) with no
bin for [1000,inf)? i.e. if there were a 1000 in my sample it would
NOT be counted? Or anything bigger.
That's what it looks like, i.e. you need to make sure that your range
encompasses all possible values. Not sure this is in the
documentation...? Would be useful to add.
Ta
Marion
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 05 December 2020 06:41
To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
This email was received from an external source. Always check sender
details, links & attachments.
Marion,
Ugh, I should've just handed this ticket off to David in the first
place. I did more testing and was able to get it to work. I just had
to use separate "-data" command line options for each input data
source.
So this command fails:
grid_diag -data glosea_2019060100_001.nc gpm_2019060100_2019060200.nc
-config GridDiag.glosea -out grid_diag_out.nc
But this command runs fine:
*grid_diag -data glosea_2019060100_001.nc
<http://glosea_2019060100_001.nc> -data gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc> -config GridDiag.glosea -out
grid_diag_out.nc <http://grid_diag_out.nc>*
In the second one, I've specified "-data" separately for each data
source.
Although, you'll see warning about the range of the data extending
beyond the range of the bins defined in the config file.
Sorry to have such a rambling and confusing response!
So to summarize, I believe the logic is:
1. If "-data" is used once, all specified input fields are read from
all of those data files.
2. If "-data" is used more than once, the number of "-data" entries
must match the number of fields defined in the config file.
If not, you get an error message.
If so, only the i-th field is read from the files in the i-th
-data source.
John
On Fri, Dec 4, 2020 at 11:30 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Marion,
>
> It occurs to me that perhaps Grid-Diag could be made more flexible
if
> we changed that error message to a debug message in this context. It
> could
> *ATTEMPT* to read all input fields from all input files... but when
> it's unable to, have it print a log message stating that rather than
> erroring out.
>
> That would enable you to compute 2D pdf's using different variables
> from different input files.
>
> Do you think that logic is preferable?
>
> Thanks,
> John
>
> On Fri, Dec 4, 2020 at 11:27 PM The RT System itself via RT <
> met_help at ucar.edu> wrote:
>
>>
>> Fri Dec 04 23:27:19 2020: Request 97766 was acted upon.
>> Transaction: Given to johnhg (John Halley Gotway) by RT_System
>> Queue: met_help
>> Subject: Need help with GridDiag
>> Owner: johnhg
>> Requestors: marion.mittermaier at metoffice.gov.uk
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
>> >
>>
>>
>> This transaction appears to have no content
>>
>
------------------------------------------------
Subject: Need help with GridDiag
From: John Halley Gotway
Time: Tue Dec 08 13:54:07 2020
Marion,
I'm still not convinced. Please take a very careful look at the
command
you're executing. I see all the log messages you sent, but not the
actual
command you ran. At that matters a lot!
This command fails on my machine with the SAME ERROR you're getting:
* grid_diag -data glosea_2019060100_001.nc
<http://glosea_2019060100_001.nc/> gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea
-out grid_diag_out.nc <http://grid_diag_out.nc/>*
ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &)
const -> needed 3 arguments for variable precipitation_amount, got 2
But this command runs WITHOUT ERROR:
* grid_diag -data glosea_2019060100_001.nc
<http://glosea_2019060100_001.nc/> -data gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea
-out grid_diag_out.nc <http://grid_diag_out.nc/>*
Do you see the difference?
In the first, I used "-data" once and listed 2 different file names.
In the second, I used "-data" twice, once for each of the input files.
Please try doing exactly the same thing.
Does that enable it to run without error? Note that you'll still get
warnings about the actual data values falling outside the defined
range of
the bins.
As for the bin endpoints, I think this is the relevant line of code:
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/met/src/libcode/vx_series_data/series_pdf.cc#L72
int k = floor((value - min) / delta);
This places the current "value" into the k-th bin where "min" is the
minimum PDF value and "delta" is the bin width.
I believe this is consistent with bins being closed on the left side
and
open on the right:
[0,1) i.e. 0 <= x < 1
John
On Tue, Dec 8, 2020 at 8:30 AM marion.mittermaier at metoffice.gov.uk via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>
> Hi John,
>
> one final question for GridDiag.
>
> The bins are set up to be lower bound inclusive? i.e.
> n_bins = 1000;
> range = [0, 1000]; // kg/m^2
>
> means that the bin is set up as [0,1) i.e. 0 <= x < 1?
>
> I'm presuming so.... and the upper bound would be [999,1000) with no
bin
> for [1000,inf)? i.e. if there were a 1000 in my sample it would NOT
be
> counted? Or anything bigger.
>
> That's what it looks like, i.e. you need to make sure that your
range
> encompasses all possible values. Not sure this is in the
documentation...?
> Would be useful to add.
>
> Ta
> Marion
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 05 December 2020 06:41
> To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
>
> This email was received from an external source. Always check
sender
> details, links & attachments.
>
> Marion,
>
> Ugh, I should've just handed this ticket off to David in the first
place.
> I did more testing and was able to get it to work. I just had to use
> separate "-data" command line options for each input data source.
>
> So this command fails:
> grid_diag -data glosea_2019060100_001.nc
gpm_2019060100_2019060200.nc
> -config GridDiag.glosea -out grid_diag_out.nc
>
> But this command runs fine:
> *grid_diag -data glosea_2019060100_001.nc <
> http://glosea_2019060100_001.nc> -data gpm_2019060100_2019060200.nc
<
> http://gpm_2019060100_2019060200.nc> -config GridDiag.glosea -out
> grid_diag_out.nc <http://grid_diag_out.nc>*
>
> In the second one, I've specified "-data" separately for each data
source.
> Although, you'll see warning about the range of the data extending
beyond
> the range of the bins defined in the config file.
>
> Sorry to have such a rambling and confusing response!
>
> So to summarize, I believe the logic is:
> 1. If "-data" is used once, all specified input fields are read from
all
> of those data files.
> 2. If "-data" is used more than once, the number of "-data" entries
must
> match the number of fields defined in the config file.
> If not, you get an error message.
> If so, only the i-th field is read from the files in the i-th
-data
> source.
>
> John
>
> On Fri, Dec 4, 2020 at 11:30 PM John Halley Gotway <johnhg at ucar.edu>
> wrote:
>
> > Marion,
> >
> > It occurs to me that perhaps Grid-Diag could be made more flexible
if
> > we changed that error message to a debug message in this context.
It
> > could
> > *ATTEMPT* to read all input fields from all input files... but
when
> > it's unable to, have it print a log message stating that rather
than
> > erroring out.
> >
> > That would enable you to compute 2D pdf's using different
variables
> > from different input files.
> >
> > Do you think that logic is preferable?
> >
> > Thanks,
> > John
> >
> > On Fri, Dec 4, 2020 at 11:27 PM The RT System itself via RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> Fri Dec 04 23:27:19 2020: Request 97766 was acted upon.
> >> Transaction: Given to johnhg (John Halley Gotway) by RT_System
> >> Queue: met_help
> >> Subject: Need help with GridDiag
> >> Owner: johnhg
> >> Requestors: marion.mittermaier at metoffice.gov.uk
> >> Status: new
> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
> >> >
> >>
> >>
> >> This transaction appears to have no content
> >>
> >
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list