[Met_help] [rt.rap.ucar.edu #97766] History for Need help with GridDiag
John Halley Gotway via RT
met_help at ucar.edu
Fri Mar 5 08:50:31 MST 2021
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello,
I'm trying to get GridDiag to work but struggling to understand some of the specifics of the netcdf headers that are required. I have attached two different files... the one works sometimes, the other doesn't work (see below).
I've attached the output I get as well as the input sample files and relevant config files. I have to say I haven't tried the METplus wrapper yet because I think I have more fundamental problems!
1. What does this "bnds" variable need to be?
2. There are two lead times in the glosea file but both the data sets are called "precipitation_amount"... that seems to be causing a problem? Maybe?
3. Running GridDiag with just the GPM file works. Running GridDiag with just the forecast works sometimes?
I think it does have to do with the code being unable to decode the time info correctly.
Any help gratefully received!
Thanks
Marion
DEBUG 1: Default Config File: /data/users/cfrd/MET_requirements/MET9.0/share/met/config/GridDiagConfig_default
DEBUG 1: User Config File: GridDiag.glosea
DEBUG 2: Read config files.
DEBUG 2: Enter get_mtddf.
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable "hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1970-01-01 00:00:00" as a reference time of 19700101_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> parsing units for the forecast_reference_time variable "hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1970-01-01 00:00:00" as a reference time of 19700101_000000 and 3600 second(s) per time step.
WARNING:
WARNING: get_nc_var(NcFile) --> The variable "bnds" does not exist!
WARNING:
DEBUG 2: Process configuration.
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_NcCF".
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_NcCF".
DEBUG 3: Use the grid defined by file "/scratch/fryl/GPM/gpm_2019060100_2019060200.nc".
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable "hours since 1970-01-01"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1970-01-01" as a reference time of 19700101_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> could not extract init time from the "forecast_reference_time" variable.
DEBUG 4: NcCfFile::open() -> could not extract init time from file name.
DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 3600 Ny: 1800 lat_ll: -89.950 lon_ll: 179.950 delta_lat: 0.100 delta_lon: 0.100
DEBUG 2: Processing masking regions.
DEBUG 2: Initializing precipitation_amount(0,*,*) histogram
DEBUG 2: Initializing precipitation_amount(*,*) histogram
DEBUG 2: Initializing precipitation_amount(0,*,*)_precipitation_amount(*,*) joint histogram
DEBUG 1: /scratch/frmm/MET/output/grid_diag/glosea//grid_diag_glosea_2019060100.nc
DEBUG 1: Length of configuration "data.field" = 2
DEBUG 1: Length of data file list = 2
DEBUG 1: Series defined by the data file list of length 2.
DEBUG 2: 100
DEBUG 2: Processing series entry 1: precipitation_amount: precipitation_amount(0,*,*)
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable "hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1970-01-01 00:00:00" as a reference time of 19700101_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> parsing units for the forecast_reference_time variable "hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1970-01-01 00:00:00" as a reference time of 19700101_000000 and 3600 second(s) per time step.
WARNING:
WARNING: get_nc_var(NcFile) --> The variable "bnds" does not exist!
WARNING:
DEBUG 4:
DEBUG 4: Data plane information:
DEBUG 4: plane min: 0
DEBUG 4: plane max: 262.128
DEBUG 4: valid time: 20190601_120000
DEBUG 4: lead time: 120000
DEBUG 4: init time: 20190601_000000
DEBUG 4: accum time: 000000
DEBUG 1: Regridding field precipitation_amount(0,*,*) to the verification grid.
DEBUG 2: Processing series entry 1: precipitation_amount: precipitation_amount(*,*)
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable "hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1970-01-01 00:00:00" as a reference time of 19700101_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> parsing units for the forecast_reference_time variable "hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 1970-01-01 00:00:00" as a reference time of 19700101_000000 and 3600 second(s) per time step.
WARNING:
WARNING: get_nc_var(NcFile) --> The variable "bnds" does not exist!
WARNING:
WARNING:
WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) -> returns the first available time for "precipitation_amount" variable).
WARNING:
ERROR :
ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &) const -> needed 3 arguments for variable precipitation_amount, got 2
ERROR :
#########
FINISHED!
#########
Fri Dec 4 14:53:34 GMT 2020
--
Dr Marion Mittermaier Manager: Model diagnostics and novel verification
Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
Tel: +44 (0) 330 135 1604
E-mail: marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk> http://www.metoffice.gov.uk<http://www.metoffice.gov.uk/>
http://www.metoffice.gov.uk/research/people/marion-mittermaier
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Need help with GridDiag
From: John Halley Gotway
Time: Fri Dec 04 23:27:19 2020
Marion,
I see you have questions about the grid_diag tool in MET. Whenever
folks
send us data, I always start by running the plot_data_plane tool to
confirm
that MET can read that data. And sure enough, it can read both of the
files
you sent:
*plot_data_plane gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc> gpm.ps <http://gpm.ps>
'name="precipitation_amount"; level="(*,*)";' -v 4*
*plot_data_plane glosea_2019060100_001.nc
<http://glosea_2019060100_001.nc>
glosea.ps <http://glosea.ps> 'name="precipitation_amount";
level="(0,*,*)";' -v 4*
A screenshot of the resulting images is attached. So while MET can
read
them, I'm pretty suspicious of the range of values for the GPM data...
from
0 up to 9.96921e+36.
Perhaps there's an issue there? The glosea values from 0 to 262 seem
much
more reasonable for precip!
[image: Screen Shot 2020-12-04 at 10.52.01 PM.png]
Now on to your question about the grid_diag tool. First, I can confirm
that
I'm able to reproduce that exact same behavior you describe:
*grid_diag -data glosea_2019060100_001.nc
<http://glosea_2019060100_001.nc> -config GridDiag.glosea -out
grid_diag_out.nc <http://grid_diag_out.nc>*
ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &)
const
-> needed 3 arguments for variable precipitation_amount, got 2
Because of the common variable names, this could get confusing. But to
be
clear...
- glosea_2019060100_001.nc contains precipitation_amount(time,
latitude,
longitude)... that's 3 dimensions.
- gpm_2019060100_2019060200.nc contains precipitation_amount(latitude,
longitude)... that's 2 dimensions.
You listed two fields in the config file...
precipitation_amount(0,*,*) followed by precipitation_amount(*,*).
And I'm guessing you passed as input the glosea file followed by the
GPM
file. And you expected that Grid-Diag would read the first field from
the
first file and the second field from the second file.
But that's where the confusion lies. Grid-Diag attempts to read all
requested input fields from all specified input files. And that causes
the
error we're seeing. It tries to read the second field
"precipitation_amount(*,*)" from the first (Glosea) file, but it can't
because that one has "precipitation_amount" with 3 dimensions, not 2.
So the issue is in the processing logic of the tool. In general, it
was
designed to inventory data over one or more timesteps and derive 1 and
2
dimensional PDF's of the data it inventories. However, it expects to
read
all requested fields from all specified input files. Otherwise, you'll
get
an error. It just isn't set up to compare PDF's from different input
data
sources.
And if you have suggestions for how we could clarify this point in the
documentation, please let us know.
If you have more questions about the processing logic, I'll direct you
to
David Fillmore (cc'ed here). He's the engineer who knows the most
about
this tool. And hopefully he can correct anything I may have stated
incorrectly!
Thanks,
John Halley Gotway
On Fri, Dec 4, 2020 at 8:09 AM marion.mittermaier at metoffice.gov.uk via
RT <
met_help at ucar.edu> wrote:
>
> Fri Dec 04 08:09:16 2020: Request 97766 was acted upon.
> Transaction: Ticket created by marion.mittermaier at metoffice.gov.uk
> Queue: met_help
> Subject: Need help with GridDiag
> Owner: Nobody
> Requestors: marion.mittermaier at metoffice.gov.uk
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>
>
> Hello,
>
> I'm trying to get GridDiag to work but struggling to understand some
of
> the specifics of the netcdf headers that are required. I have
attached two
> different files... the one works sometimes, the other doesn't work
(see
> below).
>
> I've attached the output I get as well as the input sample files and
> relevant config files. I have to say I haven't tried the METplus
wrapper
> yet because I think I have more fundamental problems!
>
>
> 1. What does this "bnds" variable need to be?
> 2. There are two lead times in the glosea file but both the data
sets
> are called "precipitation_amount"... that seems to be causing a
problem?
> Maybe?
> 3. Running GridDiag with just the GPM file works. Running
GridDiag with
> just the forecast works sometimes?
>
> I think it does have to do with the code being unable to decode the
time
> info correctly.
>
> Any help gratefully received!
>
> Thanks
> Marion
>
> DEBUG 1: Default Config File:
>
/data/users/cfrd/MET_requirements/MET9.0/share/met/config/GridDiagConfig_default
> DEBUG 1: User Config File: GridDiag.glosea
> DEBUG 2: Read config files.
> DEBUG 2: Enter get_mtddf.
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours
> since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit
> string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> DEBUG 4: NcCfFile::open() -> parsing units for the
forecast_reference_time
> variable "hours since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit
> string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> WARNING:
> WARNING: get_nc_var(NcFile) --> The variable "bnds" does not exist!
> WARNING:
> DEBUG 2: Process configuration.
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> type "FileType_NcCF".
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> type "FileType_NcCF".
> DEBUG 3: Use the grid defined by file "/scratch/fryl/GPM/
> gpm_2019060100_2019060200.nc".
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours
> since 1970-01-01"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit
> string "hours since 1970-01-01" as a reference time of
19700101_000000 and
> 3600 second(s) per time step.
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
> DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 3600 Ny: 1800
lat_ll:
> -89.950 lon_ll: 179.950 delta_lat: 0.100 delta_lon: 0.100
> DEBUG 2: Processing masking regions.
> DEBUG 2: Initializing precipitation_amount(0,*,*) histogram
> DEBUG 2: Initializing precipitation_amount(*,*) histogram
> DEBUG 2: Initializing
> precipitation_amount(0,*,*)_precipitation_amount(*,*) joint
histogram
> DEBUG 1: /scratch/frmm/MET/output/grid_diag/glosea//
> grid_diag_glosea_2019060100.nc
> DEBUG 1: Length of configuration "data.field" = 2
> DEBUG 1: Length of data file list = 2
> DEBUG 1: Series defined by the data file list of length 2.
> DEBUG 2: 100
> DEBUG 2: Processing series entry 1: precipitation_amount:
> precipitation_amount(0,*,*)
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours
> since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit
> string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> DEBUG 4: NcCfFile::open() -> parsing units for the
forecast_reference_time
> variable "hours since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit
> string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> WARNING:
> WARNING: get_nc_var(NcFile) --> The variable "bnds" does not exist!
> WARNING:
> DEBUG 4:
> DEBUG 4: Data plane information:
> DEBUG 4: plane min: 0
> DEBUG 4: plane max: 262.128
> DEBUG 4: valid time: 20190601_120000
> DEBUG 4: lead time: 120000
> DEBUG 4: init time: 20190601_000000
> DEBUG 4: accum time: 000000
> DEBUG 1: Regridding field precipitation_amount(0,*,*) to the
verification
> grid.
> DEBUG 2: Processing series entry 1: precipitation_amount:
> precipitation_amount(*,*)
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours
> since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit
> string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> DEBUG 4: NcCfFile::open() -> parsing units for the
forecast_reference_time
> variable "hours since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit
> string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> WARNING:
> WARNING: get_nc_var(NcFile) --> The variable "bnds" does not exist!
> WARNING:
> WARNING:
> WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) ->
returns
> the first available time for "precipitation_amount" variable).
> WARNING:
> ERROR :
> ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &)
const
> -> needed 3 arguments for variable precipitation_amount, got 2
> ERROR :
> #########
> FINISHED!
> #########
> Fri Dec 4 14:53:34 GMT 2020
>
> --
> Dr Marion Mittermaier Manager: Model diagnostics and novel
verification
>
> Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> Tel: +44 (0) 330 135 1604
> E-mail: marion.mittermaier at metoffice.gov.uk<mailto:
> marion.mittermaier at metoffice.gov.uk> http://www.metoffice.gov.uk<
> http://www.metoffice.gov.uk/>
>
> http://www.metoffice.gov.uk/research/people/marion-mittermaier
>
>
>
------------------------------------------------
Subject: Need help with GridDiag
From: John Halley Gotway
Time: Fri Dec 04 23:30:27 2020
Marion,
It occurs to me that perhaps Grid-Diag could be made more flexible if
we
changed that error message to a debug message in this context. It
could
*ATTEMPT* to read all input fields from all input files... but when
it's
unable to, have it print a log message stating that rather than
erroring
out.
That would enable you to compute 2D pdf's using different variables
from
different input files.
Do you think that logic is preferable?
Thanks,
John
On Fri, Dec 4, 2020 at 11:27 PM The RT System itself via RT <
met_help at ucar.edu> wrote:
>
> Fri Dec 04 23:27:19 2020: Request 97766 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by RT_System
> Queue: met_help
> Subject: Need help with GridDiag
> Owner: johnhg
> Requestors: marion.mittermaier at metoffice.gov.uk
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>
>
> This transaction appears to have no content
>
------------------------------------------------
Subject: Need help with GridDiag
From: John Halley Gotway
Time: Fri Dec 04 23:41:26 2020
Marion,
Ugh, I should've just handed this ticket off to David in the first
place. I
did more testing and was able to get it to work. I just had to use
separate
"-data" command line options for each input data source.
So this command fails:
grid_diag -data glosea_2019060100_001.nc gpm_2019060100_2019060200.nc
-config GridDiag.glosea -out grid_diag_out.nc
But this command runs fine:
*grid_diag -data glosea_2019060100_001.nc
<http://glosea_2019060100_001.nc> -data gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc> -config GridDiag.glosea -out
grid_diag_out.nc <http://grid_diag_out.nc>*
In the second one, I've specified "-data" separately for each data
source.
Although, you'll see warning about the range of the data extending
beyond
the range of the bins defined in the config file.
Sorry to have such a rambling and confusing response!
So to summarize, I believe the logic is:
1. If "-data" is used once, all specified input fields are read from
all of
those data files.
2. If "-data" is used more than once, the number of "-data" entries
must
match the number of fields defined in the config file.
If not, you get an error message.
If so, only the i-th field is read from the files in the i-th
-data
source.
John
On Fri, Dec 4, 2020 at 11:30 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Marion,
>
> It occurs to me that perhaps Grid-Diag could be made more flexible
if we
> changed that error message to a debug message in this context. It
could
> *ATTEMPT* to read all input fields from all input files... but when
it's
> unable to, have it print a log message stating that rather than
erroring
> out.
>
> That would enable you to compute 2D pdf's using different variables
from
> different input files.
>
> Do you think that logic is preferable?
>
> Thanks,
> John
>
> On Fri, Dec 4, 2020 at 11:27 PM The RT System itself via RT <
> met_help at ucar.edu> wrote:
>
>>
>> Fri Dec 04 23:27:19 2020: Request 97766 was acted upon.
>> Transaction: Given to johnhg (John Halley Gotway) by RT_System
>> Queue: met_help
>> Subject: Need help with GridDiag
>> Owner: johnhg
>> Requestors: marion.mittermaier at metoffice.gov.uk
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>>
>>
>> This transaction appears to have no content
>>
>
------------------------------------------------
Subject: Need help with GridDiag
From: marion.mittermaier at metoffice.gov.uk
Time: Mon Dec 07 14:49:01 2020
Thanks John.
That is helpful. Just to follow up.
I tested to see if I could get it to work by making sure that the
standard_names and long_names are different. I also chose to only put
one lead time in the forecast file and used that with the GPM data.
But, as the output below shows, I’m thinking this isn’t the problem
per se.
It’s the dimensions… I’ve renamed the forecast fields as precip_amount
and only put one lead time into a forecast file but this file STILL
has 3 dimensions so the code still falls over. The forecast file would
need more “surgery” to get these to have the same dimensions but given
what it is I wouldn’t want to do this really.
DEBUG 1: Default Config File:
/data/users/cfrd/MET_requirements/MET9.0/share/met/config/GridDiagConfig_default
DEBUG 1: User Config File: GridDiag.glosea
DEBUG 2: Read config files.
DEBUG 2: Enter get_mtddf.
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit string "hours since 1970-01-01 00:00:00" as a reference time of
19700101_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> parsing units for the
forecast_reference_time variable "hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit string "hours since 1970-01-01 00:00:00" as a reference time of
19700101_000000 and 3600 second(s) per time step.
WARNING:
WARNING: get_nc_var(NcFile) --> The variable "bnds" does not exist!
WARNING:
DEBUG 2: Process configuration.
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object
of type "FileType_NcCF".
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object
of type "FileType_NcCF".
DEBUG 3: Use the grid defined by file
"/scratch/fryl/GPM/gpm_2019060100_2019060200.nc".
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours since 1970-01-01"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit string "hours since 1970-01-01" as a reference time of
19700101_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> could not extract init time from the
"forecast_reference_time" variable.
DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 3600 Ny: 1800
lat_ll: -89.950 lon_ll: 179.950 delta_lat: 0.100 delta_lon: 0.100
DEBUG 2: Processing masking regions.
DEBUG 2: Initializing precip_amount(*,*) histogram
DEBUG 2: Initializing precipitation_amount(*,*) histogram
DEBUG 2: Initializing precip_amount(*,*)_precipitation_amount(*,*)
joint histogram
DEBUG 1:
/scratch/frmm/MET/output/grid_diag/glosea//grid_diag_glosea_2019060100.nc
DEBUG 1: Length of configuration "data.field" = 2
DEBUG 1: Length of data file list = 2
DEBUG 1: Series defined by the data file list of length 2.
DEBUG 2: 100
DEBUG 2: Processing series entry 1: precip_amount: precip_amount(*,*)
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit string "hours since 1970-01-01 00:00:00" as a reference time of
19700101_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> parsing units for the
forecast_reference_time variable "hours since 1970-01-01 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit string "hours since 1970-01-01 00:00:00" as a reference time of
19700101_000000 and 3600 second(s) per time step.
WARNING:
WARNING: get_nc_var(NcFile) --> The variable "bnds" does not exist!
WARNING:
WARNING:
WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) ->
returns the first available time for "precip_amount" variable).
WARNING:
ERROR :
ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &)
const -> needed 3 arguments for variable precip_amount, got 2
ERROR :
#########
FINISHED!
#########
Mon Dec 7 11:29:49 GMT 2020
I guess the conclusion is that joint distributions between forecast
and observations (of the same variable) aren’t possible at this stage
without doing something significant to the forecast files (to reduce
dimensions) but I can run it sequentially and get the marginal
distributions which I can work with for the moment. Going forward I’m
not sure what to do though, other than make some changes to the
forecast files (i.e. lose the whole “forecast_period” part of the
netcdf header).
Marion
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 05 December 2020 06:27
To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
This email was received from an external source. Always check sender
details, links & attachments.
Marion,
I see you have questions about the grid_diag tool in MET. Whenever
folks send us data, I always start by running the plot_data_plane tool
to confirm that MET can read that data. And sure enough, it can read
both of the files you sent:
*plot_data_plane gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc> gpm.ps <http://gpm.ps>
'name="precipitation_amount"; level="(*,*)";' -v 4* *plot_data_plane
glosea_2019060100_001.nc <http://glosea_2019060100_001.nc> glosea.ps
<http://glosea.ps> 'name="precipitation_amount"; level="(0,*,*)";' -v
4*
A screenshot of the resulting images is attached. So while MET can
read them, I'm pretty suspicious of the range of values for the GPM
data... from
0 up to 9.96921e+36.
Perhaps there's an issue there? The glosea values from 0 to 262 seem
much more reasonable for precip!
[image: Screen Shot 2020-12-04 at 10.52.01 PM.png]
Now on to your question about the grid_diag tool. First, I can confirm
that I'm able to reproduce that exact same behavior you describe:
*grid_diag -data glosea_2019060100_001.nc
<http://glosea_2019060100_001.nc> -config GridDiag.glosea -out
grid_diag_out.nc <http://grid_diag_out.nc>*
ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &)
const
-> needed 3 arguments for variable precipitation_amount, got 2
Because of the common variable names, this could get confusing. But to
be clear...
- glosea_2019060100_001.nc contains precipitation_amount(time,
latitude, longitude)... that's 3 dimensions.
- gpm_2019060100_2019060200.nc contains precipitation_amount(latitude,
longitude)... that's 2 dimensions.
You listed two fields in the config file...
precipitation_amount(0,*,*) followed by precipitation_amount(*,*).
And I'm guessing you passed as input the glosea file followed by the
GPM file. And you expected that Grid-Diag would read the first field
from the first file and the second field from the second file.
But that's where the confusion lies. Grid-Diag attempts to read all
requested input fields from all specified input files. And that causes
the error we're seeing. It tries to read the second field
"precipitation_amount(*,*)" from the first (Glosea) file, but it can't
because that one has "precipitation_amount" with 3 dimensions, not 2.
So the issue is in the processing logic of the tool. In general, it
was designed to inventory data over one or more timesteps and derive 1
and 2 dimensional PDF's of the data it inventories. However, it
expects to read all requested fields from all specified input files.
Otherwise, you'll get an error. It just isn't set up to compare PDF's
from different input data sources.
And if you have suggestions for how we could clarify this point in the
documentation, please let us know.
If you have more questions about the processing logic, I'll direct you
to David Fillmore (cc'ed here). He's the engineer who knows the most
about this tool. And hopefully he can correct anything I may have
stated incorrectly!
Thanks,
John Halley Gotway
On Fri, Dec 4, 2020 at 8:09 AM
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>
via RT < met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
>
> Fri Dec 04 08:09:16 2020: Request 97766 was acted upon.
> Transaction: Ticket created by
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>
> Queue: met_help
> Subject: Need help with GridDiag
> Owner: Nobody
> Requestors:
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
> >
>
>
> Hello,
>
> I'm trying to get GridDiag to work but struggling to understand some
> of the specifics of the netcdf headers that are required. I have
> attached two different files... the one works sometimes, the other
> doesn't work (see below).
>
> I've attached the output I get as well as the input sample files and
> relevant config files. I have to say I haven't tried the METplus
> wrapper yet because I think I have more fundamental problems!
>
>
> 1. What does this "bnds" variable need to be?
> 2. There are two lead times in the glosea file but both the data
> sets are called "precipitation_amount"... that seems to be causing a
problem?
> Maybe?
> 3. Running GridDiag with just the GPM file works. Running
GridDiag
> with just the forecast works sometimes?
>
> I think it does have to do with the code being unable to decode the
> time info correctly.
>
> Any help gratefully received!
>
> Thanks
> Marion
>
> DEBUG 1: Default Config File:
>
/data/users/cfrd/MET_requirements/MET9.0/share/met/config/GridDiagConf
> ig_default DEBUG 1: User Config File: GridDiag.glosea DEBUG 2: Read
> config files.
> DEBUG 2: Enter get_mtddf.
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> "hours since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> DEBUG 4: NcCfFile::open() -> parsing units for the
> forecast_reference_time variable "hours since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> WARNING:
> WARNING: get_nc_var(NcFile) --> The variable "bnds" does not exist!
> WARNING:
> DEBUG 2: Process configuration.
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of type "FileType_NcCF".
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of type "FileType_NcCF".
> DEBUG 3: Use the grid defined by file "/scratch/fryl/GPM/
> gpm_2019060100_2019060200.nc".
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> "hours since 1970-01-01"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01" as a reference time of
> 19700101_000000 and
> 3600 second(s) per time step.
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
> DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 3600 Ny: 1800
lat_ll:
> -89.950 lon_ll: 179.950 delta_lat: 0.100 delta_lon: 0.100 DEBUG 2:
> Processing masking regions.
> DEBUG 2: Initializing precipitation_amount(0,*,*) histogram DEBUG 2:
> Initializing precipitation_amount(*,*) histogram DEBUG 2:
Initializing
> precipitation_amount(0,*,*)_precipitation_amount(*,*) joint
histogram
> DEBUG 1: /scratch/frmm/MET/output/grid_diag/glosea//
> grid_diag_glosea_2019060100.nc
> DEBUG 1: Length of configuration "data.field" = 2
> DEBUG 1: Length of data file list = 2
> DEBUG 1: Series defined by the data file list of length 2.
> DEBUG 2: 100
> DEBUG 2: Processing series entry 1: precipitation_amount:
> precipitation_amount(0,*,*)
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> "hours since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> DEBUG 4: NcCfFile::open() -> parsing units for the
> forecast_reference_time variable "hours since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> WARNING:
> WARNING: get_nc_var(NcFile) --> The variable "bnds" does not exist!
> WARNING:
> DEBUG 4:
> DEBUG 4: Data plane information:
> DEBUG 4: plane min: 0
> DEBUG 4: plane max: 262.128
> DEBUG 4: valid time: 20190601_120000
> DEBUG 4: lead time: 120000
> DEBUG 4: init time: 20190601_000000
> DEBUG 4: accum time: 000000
> DEBUG 1: Regridding field precipitation_amount(0,*,*) to the
> verification grid.
> DEBUG 2: Processing series entry 1: precipitation_amount:
> precipitation_amount(*,*)
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> "hours since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> DEBUG 4: NcCfFile::open() -> parsing units for the
> forecast_reference_time variable "hours since 1970-01-01 00:00:00"
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01 00:00:00" as a reference time of
> 19700101_000000 and 3600 second(s) per time step.
> WARNING:
> WARNING: get_nc_var(NcFile) --> The variable "bnds" does not exist!
> WARNING:
> WARNING:
> WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) ->
> returns the first available time for "precipitation_amount"
variable).
> WARNING:
> ERROR :
> ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &)
> const
> -> needed 3 arguments for variable precipitation_amount, got 2
> ERROR :
> #########
> FINISHED!
> #########
> Fri Dec 4 14:53:34 GMT 2020
>
> --
> Dr Marion Mittermaier Manager: Model diagnostics and novel
verification
>
> Met Office FitzRoy Road Exeter EX1 3PB United Kingdom
> Tel: +44 (0) 330 135 1604
> E-mail:
marion.mittermaier at metoffice.gov.uk<mailto<mailto:marion.mittermaier at metoffice.gov.uk%3cmailto>:
>
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>>
http://www.metoffice.gov.uk<<http://www.metoffice.gov.uk%3c>
> http://www.metoffice.gov.uk/>
>
> http://www.metoffice.gov.uk/research/people/marion-mittermaier
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #97766] Need help with GridDiag
From: marion.mittermaier at metoffice.gov.uk
Time: Tue Dec 08 08:30:48 2020
Hi John,
one final question for GridDiag.
The bins are set up to be lower bound inclusive? i.e.
n_bins = 1000;
range = [0, 1000]; // kg/m^2
means that the bin is set up as [0,1) i.e. 0 <= x < 1?
I'm presuming so.... and the upper bound would be [999,1000) with no
bin for [1000,inf)? i.e. if there were a 1000 in my sample it would
NOT be counted? Or anything bigger.
That's what it looks like, i.e. you need to make sure that your range
encompasses all possible values. Not sure this is in the
documentation...? Would be useful to add.
Ta
Marion
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 05 December 2020 06:41
To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
This email was received from an external source. Always check sender
details, links & attachments.
Marion,
Ugh, I should've just handed this ticket off to David in the first
place. I did more testing and was able to get it to work. I just had
to use separate "-data" command line options for each input data
source.
So this command fails:
grid_diag -data glosea_2019060100_001.nc gpm_2019060100_2019060200.nc
-config GridDiag.glosea -out grid_diag_out.nc
But this command runs fine:
*grid_diag -data glosea_2019060100_001.nc
<http://glosea_2019060100_001.nc> -data gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc> -config GridDiag.glosea -out
grid_diag_out.nc <http://grid_diag_out.nc>*
In the second one, I've specified "-data" separately for each data
source.
Although, you'll see warning about the range of the data extending
beyond the range of the bins defined in the config file.
Sorry to have such a rambling and confusing response!
So to summarize, I believe the logic is:
1. If "-data" is used once, all specified input fields are read from
all of those data files.
2. If "-data" is used more than once, the number of "-data" entries
must match the number of fields defined in the config file.
If not, you get an error message.
If so, only the i-th field is read from the files in the i-th
-data source.
John
On Fri, Dec 4, 2020 at 11:30 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Marion,
>
> It occurs to me that perhaps Grid-Diag could be made more flexible
if
> we changed that error message to a debug message in this context. It
> could
> *ATTEMPT* to read all input fields from all input files... but when
> it's unable to, have it print a log message stating that rather than
> erroring out.
>
> That would enable you to compute 2D pdf's using different variables
> from different input files.
>
> Do you think that logic is preferable?
>
> Thanks,
> John
>
> On Fri, Dec 4, 2020 at 11:27 PM The RT System itself via RT <
> met_help at ucar.edu> wrote:
>
>>
>> Fri Dec 04 23:27:19 2020: Request 97766 was acted upon.
>> Transaction: Given to johnhg (John Halley Gotway) by RT_System
>> Queue: met_help
>> Subject: Need help with GridDiag
>> Owner: johnhg
>> Requestors: marion.mittermaier at metoffice.gov.uk
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
>> >
>>
>>
>> This transaction appears to have no content
>>
>
------------------------------------------------
Subject: Need help with GridDiag
From: John Halley Gotway
Time: Tue Dec 08 13:54:07 2020
Marion,
I'm still not convinced. Please take a very careful look at the
command
you're executing. I see all the log messages you sent, but not the
actual
command you ran. At that matters a lot!
This command fails on my machine with the SAME ERROR you're getting:
* grid_diag -data glosea_2019060100_001.nc
<http://glosea_2019060100_001.nc/> gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea
-out grid_diag_out.nc <http://grid_diag_out.nc/>*
ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &)
const -> needed 3 arguments for variable precipitation_amount, got 2
But this command runs WITHOUT ERROR:
* grid_diag -data glosea_2019060100_001.nc
<http://glosea_2019060100_001.nc/> -data gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea
-out grid_diag_out.nc <http://grid_diag_out.nc/>*
Do you see the difference?
In the first, I used "-data" once and listed 2 different file names.
In the second, I used "-data" twice, once for each of the input files.
Please try doing exactly the same thing.
Does that enable it to run without error? Note that you'll still get
warnings about the actual data values falling outside the defined
range of
the bins.
As for the bin endpoints, I think this is the relevant line of code:
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/met/src/libcode/vx_series_data/series_pdf.cc#L72
int k = floor((value - min) / delta);
This places the current "value" into the k-th bin where "min" is the
minimum PDF value and "delta" is the bin width.
I believe this is consistent with bins being closed on the left side
and
open on the right:
[0,1) i.e. 0 <= x < 1
John
On Tue, Dec 8, 2020 at 8:30 AM marion.mittermaier at metoffice.gov.uk via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>
> Hi John,
>
> one final question for GridDiag.
>
> The bins are set up to be lower bound inclusive? i.e.
> n_bins = 1000;
> range = [0, 1000]; // kg/m^2
>
> means that the bin is set up as [0,1) i.e. 0 <= x < 1?
>
> I'm presuming so.... and the upper bound would be [999,1000) with no
bin
> for [1000,inf)? i.e. if there were a 1000 in my sample it would NOT
be
> counted? Or anything bigger.
>
> That's what it looks like, i.e. you need to make sure that your
range
> encompasses all possible values. Not sure this is in the
documentation...?
> Would be useful to add.
>
> Ta
> Marion
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 05 December 2020 06:41
> To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
>
> This email was received from an external source. Always check
sender
> details, links & attachments.
>
> Marion,
>
> Ugh, I should've just handed this ticket off to David in the first
place.
> I did more testing and was able to get it to work. I just had to use
> separate "-data" command line options for each input data source.
>
> So this command fails:
> grid_diag -data glosea_2019060100_001.nc
gpm_2019060100_2019060200.nc
> -config GridDiag.glosea -out grid_diag_out.nc
>
> But this command runs fine:
> *grid_diag -data glosea_2019060100_001.nc <
> http://glosea_2019060100_001.nc> -data gpm_2019060100_2019060200.nc
<
> http://gpm_2019060100_2019060200.nc> -config GridDiag.glosea -out
> grid_diag_out.nc <http://grid_diag_out.nc>*
>
> In the second one, I've specified "-data" separately for each data
source.
> Although, you'll see warning about the range of the data extending
beyond
> the range of the bins defined in the config file.
>
> Sorry to have such a rambling and confusing response!
>
> So to summarize, I believe the logic is:
> 1. If "-data" is used once, all specified input fields are read from
all
> of those data files.
> 2. If "-data" is used more than once, the number of "-data" entries
must
> match the number of fields defined in the config file.
> If not, you get an error message.
> If so, only the i-th field is read from the files in the i-th
-data
> source.
>
> John
>
> On Fri, Dec 4, 2020 at 11:30 PM John Halley Gotway <johnhg at ucar.edu>
> wrote:
>
> > Marion,
> >
> > It occurs to me that perhaps Grid-Diag could be made more flexible
if
> > we changed that error message to a debug message in this context.
It
> > could
> > *ATTEMPT* to read all input fields from all input files... but
when
> > it's unable to, have it print a log message stating that rather
than
> > erroring out.
> >
> > That would enable you to compute 2D pdf's using different
variables
> > from different input files.
> >
> > Do you think that logic is preferable?
> >
> > Thanks,
> > John
> >
> > On Fri, Dec 4, 2020 at 11:27 PM The RT System itself via RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> Fri Dec 04 23:27:19 2020: Request 97766 was acted upon.
> >> Transaction: Given to johnhg (John Halley Gotway) by RT_System
> >> Queue: met_help
> >> Subject: Need help with GridDiag
> >> Owner: johnhg
> >> Requestors: marion.mittermaier at metoffice.gov.uk
> >> Status: new
> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
> >> >
> >>
> >>
> >> This transaction appears to have no content
> >>
> >
>
>
>
>
------------------------------------------------
Subject: Need help with GridDiag
From: marion.mittermaier at metoffice.gov.uk
Time: Mon Feb 15 04:02:49 2021
Hi John,
it's taken me a while to get back to this... I do see the difference
but I still don't get it to work. I can get either on their own to
work but not together.
I have split out the forecast days into separate files.
grid_diag -data
/scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc -data
/scratch/fryl/GPM/gpm_2019060100_2019060200.nc -config GridDiag.glosea
-out grid_diag_out.nc -v 4
This gives the following error (I think it's in regrid_data_plane).
Not sure why it is trying to regrid both? Or maybe I’m just not quite
following the messages….
vld497:/net/home/h02/frmm/WCSSP/India > grid_diag -data
/scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc -data
/scratch/fryl/GPM/gpm_2019060100_2019060200.nc
DEBUG 1: Default Config File:
/data/users/cfrd/MET_requirements/MET9.0/share/met/config/GridDiagConfig_default
DEBUG 1: User Config File: GridDiag.glosea
DEBUG 2: Read config files.
DEBUG 2: Enter get_mtddf.
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours since 1970-01-01"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit string "hours since 1970-01-01" as a reference time of
19700101_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> could not extract init time from the
"forecast_reference_time" variable.
DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
DEBUG 2: Process configuration.
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object
of type "FileType_NcCF".
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object
of type "FileType_NcCF".
DEBUG 3: Use the grid defined by file
"/scratch/fryl/WCSSPIndia/GPM/gpm_2019060100_2019060200.nc".
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours since 1970-01-01"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit string "hours since 1970-01-01" as a reference time of
19700101_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> could not extract init time from the
"forecast_reference_time" variable.
DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 360 lat_ll:
4.550 lon_ll: -64.550 delta_lat: 0.100 delta_lon: 0.100
DEBUG 2: Processing masking regions.
DEBUG 3: Processing poly mask: India.poly
DEBUG 4: parse_poly_mask() -> parsing poly mask "India.poly"
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_None".
DEBUG 4: parse_poly_mask() -> parsing poly mask file "India.poly"
DEBUG 2: Initializing precipitation_amount(*,*) histogram
DEBUG 2: Initializing Precipitation_Amount(*,*) histogram
DEBUG 2: Initializing
precipitation_amount(*,*)_Precipitation_Amount(*,*) joint histogram
DEBUG 1: grid_diag_out.nc
DEBUG 1: Length of configuration "data.field" = 2
DEBUG 1: Length of data file list = 1
DEBUG 1: Series defined by the data file list of length 1.
DEBUG 2: 1000
DEBUG 2: Processing series entry 1: precipitation_amount:
precipitation_amount(*,*)
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours since 1970-01-01"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit string "hours since 1970-01-01" as a reference time of
19700101_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> could not extract init time from the
"forecast_reference_time" variable.
DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
DEBUG 4: NcCfFile::getData() -> setting the unset init time to the
valid time of 20190601_115959.
DEBUG 4:
DEBUG 4: Data plane information:
DEBUG 4: plane min: 0
DEBUG 4: plane max: 9.96921e+36
DEBUG 4: valid time: 20190601_115959
DEBUG 4: lead time: 000000
DEBUG 4: init time: 20190601_115959
DEBUG 4: accum time: 000000
DEBUG 1: Regridding field precipitation_amount(*,*) to the
verification grid.
DEBUG 2: Processing series entry 1: Precipitation_Amount:
Precipitation_Amount(*,*)
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours since 1970-01-01"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit string "hours since 1970-01-01" as a reference time of
19700101_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> could not extract init time from the
"forecast_reference_time" variable.
DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
DEBUG 1: Regridding field precipitation_amount(*,*) to the
verification grid.
ERROR :
ERROR : DataPlane::two_to_one() -> range check error: (Nx, Ny) = (0,
0), (x, y) = (0, 0)
ERROR :
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 08 December 2020 20:54
To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
This email was received from an external source. Always check sender
details, links & attachments.
Marion,
I'm still not convinced. Please take a very careful look at the
command you're executing. I see all the log messages you sent, but not
the actual command you ran. At that matters a lot!
This command fails on my machine with the SAME ERROR you're getting:
* grid_diag -data glosea_2019060100_001.nc
<http://glosea_2019060100_001.nc/> gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea -out
grid_diag_out.nc <http://grid_diag_out.nc/>*
ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &)
const -> needed 3 arguments for variable precipitation_amount, got 2
But this command runs WITHOUT ERROR:
* grid_diag -data glosea_2019060100_001.nc
<http://glosea_2019060100_001.nc/> -data gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea -out
grid_diag_out.nc <http://grid_diag_out.nc/>*
Do you see the difference?
In the first, I used "-data" once and listed 2 different file names.
In the second, I used "-data" twice, once for each of the input files.
Please try doing exactly the same thing.
Does that enable it to run without error? Note that you'll still get
warnings about the actual data values falling outside the defined
range of the bins.
As for the bin endpoints, I think this is the relevant line of code:
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/met/src/libcode/vx_series_data/series_pdf.cc#L72
int k = floor((value - min) / delta);
This places the current "value" into the k-th bin where "min" is the
minimum PDF value and "delta" is the bin width.
I believe this is consistent with bins being closed on the left side
and open on the right:
[0,1) i.e. 0 <= x < 1
John
On Tue, Dec 8, 2020 at 8:30 AM
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>
via RT < met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>
> Hi John,
>
> one final question for GridDiag.
>
> The bins are set up to be lower bound inclusive? i.e.
> n_bins = 1000;
> range = [0, 1000]; // kg/m^2
>
> means that the bin is set up as [0,1) i.e. 0 <= x < 1?
>
> I'm presuming so.... and the upper bound would be [999,1000) with no
> bin for [1000,inf)? i.e. if there were a 1000 in my sample it would
> NOT be counted? Or anything bigger.
>
> That's what it looks like, i.e. you need to make sure that your
range
> encompasses all possible values. Not sure this is in the
documentation...?
> Would be useful to add.
>
> Ta
> Marion
>
> -----Original Message-----
> From: John Halley Gotway via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>>
> Sent: 05 December 2020 06:41
> To: Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>>
> Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
>
> This email was received from an external source. Always check
sender
> details, links & attachments.
>
> Marion,
>
> Ugh, I should've just handed this ticket off to David in the first
place.
> I did more testing and was able to get it to work. I just had to use
> separate "-data" command line options for each input data source.
>
> So this command fails:
> grid_diag -data glosea_2019060100_001.nc
gpm_2019060100_2019060200.nc
> -config GridDiag.glosea -out grid_diag_out.nc
>
> But this command runs fine:
> *grid_diag -data glosea_2019060100_001.nc <
> http://glosea_2019060100_001.nc> -data gpm_2019060100_2019060200.nc
<
> http://gpm_2019060100_2019060200.nc> -config GridDiag.glosea -out
> grid_diag_out.nc <http://grid_diag_out.nc>*
>
> In the second one, I've specified "-data" separately for each data
source.
> Although, you'll see warning about the range of the data extending
> beyond the range of the bins defined in the config file.
>
> Sorry to have such a rambling and confusing response!
>
> So to summarize, I believe the logic is:
> 1. If "-data" is used once, all specified input fields are read from
> all of those data files.
> 2. If "-data" is used more than once, the number of "-data" entries
> must match the number of fields defined in the config file.
> If not, you get an error message.
> If so, only the i-th field is read from the files in the i-th
> -data source.
>
> John
>
> On Fri, Dec 4, 2020 at 11:30 PM John Halley Gotway
<johnhg at ucar.edu<mailto:johnhg at ucar.edu>>
> wrote:
>
> > Marion,
> >
> > It occurs to me that perhaps Grid-Diag could be made more flexible
> > if we changed that error message to a debug message in this
context.
> > It could
> > *ATTEMPT* to read all input fields from all input files... but
when
> > it's unable to, have it print a log message stating that rather
than
> > erroring out.
> >
> > That would enable you to compute 2D pdf's using different
variables
> > from different input files.
> >
> > Do you think that logic is preferable?
> >
> > Thanks,
> > John
> >
> > On Fri, Dec 4, 2020 at 11:27 PM The RT System itself via RT <
> > met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
> >
> >>
> >> Fri Dec 04 23:27:19 2020: Request 97766 was acted upon.
> >> Transaction: Given to johnhg (John Halley Gotway) by RT_System
> >> Queue: met_help
> >> Subject: Need help with GridDiag
> >> Owner: johnhg
> >> Requestors:
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>
> >> Status: new
> >> Ticket <URL:
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
> >> >
> >>
> >>
> >> This transaction appears to have no content
> >>
> >
>
>
>
>
------------------------------------------------
Subject: Need help with GridDiag
From: marion.mittermaier at metoffice.gov.uk
Time: Mon Feb 15 04:42:42 2021
Hi all,
in the mean time I have got regrid_data_plane to work in stand-alone
model… running the following command (and I’ve tried in both
directions):
regrid_data_plane glosea_2019060100_d001_001.nc
gpm_2019060100_2019060200.nc grid_diag_out.nc -field 019060200.nc
grid_diag_out.nc -field ‘name="precipitation_amount"; level="(*,*)";'
<
DEBUG 1: Reading data file: glosea_2019060100_d001_001.nc
DEBUG 2: Input grid: Projection: Lat/Lon Nx: 432 Ny: 324 lat_ll:
-89.722 lon_ll: -0.417 delta_lat: 0.556 delta_lon: 0.833
DEBUG 2: Output grid: Projection: Lat/Lon Nx: 3600 Ny: 1800 lat_ll:
-89.950 lon_ll: 179.950 delta_lat: 0.100 delta_lon: 0.100
DEBUG 2: Interpolation options: method = NEAREST, width = 1, shape =
SQUARE, vld_thresh = 0.5
DEBUG 2: Range of input data (name="precipitation_amount";
level="(*,*)";) is 0 to 262.128.
DEBUG 2: Range of regridded data (name="precipitation_amount";
level="(*,*)";) is 0 to 262.128.
DEBUG 1: Writing output file: grid_diag_out.nc
Which looks sensible… so it’s NOT regrid_data_plane.
Marion
From: Mittermaier, Marion
Sent: 15 February 2021 11:02
To: 'met_help at ucar.edu' <met_help at ucar.edu>
Subject: RE: [rt.rap.ucar.edu #97766] Need help with GridDiag
Hi John,
it's taken me a while to get back to this... I do see the difference
but I still don't get it to work. I can get either on their own to
work but not together.
I have split out the forecast days into separate files.
grid_diag -data
/scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc -data
/scratch/fryl/GPM/gpm_2019060100_2019060200.nc -config GridDiag.glosea
-out grid_diag_out.nc -v 4
This gives the following error (I think it's in regrid_data_plane).
Not sure why it is trying to regrid both? Or maybe I’m just not quite
following the messages….
vld497:/net/home/h02/frmm/WCSSP/India > grid_diag -data
/scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc -data
/scratch/fryl/GPM/gpm_2019060100_2019060200.nc
DEBUG 1: Default Config File:
/data/users/cfrd/MET_requirements/MET9.0/share/met/config/GridDiagConfig_default
DEBUG 1: User Config File: GridDiag.glosea
DEBUG 2: Read config files.
DEBUG 2: Enter get_mtddf.
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours since 1970-01-01"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit string "hours since 1970-01-01" as a reference time of
19700101_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> could not extract init time from the
"forecast_reference_time" variable.
DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
DEBUG 2: Process configuration.
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object
of type "FileType_NcCF".
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object
of type "FileType_NcCF".
DEBUG 3: Use the grid defined by file
"/scratch/fryl/WCSSPIndia/GPM/gpm_2019060100_2019060200.nc".
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours since 1970-01-01"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit string "hours since 1970-01-01" as a reference time of
19700101_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> could not extract init time from the
"forecast_reference_time" variable.
DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 360 lat_ll:
4.550 lon_ll: -64.550 delta_lat: 0.100 delta_lon: 0.100
DEBUG 2: Processing masking regions.
DEBUG 3: Processing poly mask: India.poly
DEBUG 4: parse_poly_mask() -> parsing poly mask "India.poly"
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_None".
DEBUG 4: parse_poly_mask() -> parsing poly mask file "India.poly"
DEBUG 2: Initializing precipitation_amount(*,*) histogram
DEBUG 2: Initializing Precipitation_Amount(*,*) histogram
DEBUG 2: Initializing
precipitation_amount(*,*)_Precipitation_Amount(*,*) joint histogram
DEBUG 1: grid_diag_out.nc
DEBUG 1: Length of configuration "data.field" = 2
DEBUG 1: Length of data file list = 1
DEBUG 1: Series defined by the data file list of length 1.
DEBUG 2: 1000
DEBUG 2: Processing series entry 1: precipitation_amount:
precipitation_amount(*,*)
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours since 1970-01-01"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit string "hours since 1970-01-01" as a reference time of
19700101_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> could not extract init time from the
"forecast_reference_time" variable.
DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
DEBUG 4: NcCfFile::getData() -> setting the unset init time to the
valid time of 20190601_115959.
DEBUG 4:
DEBUG 4: Data plane information:
DEBUG 4: plane min: 0
DEBUG 4: plane max: 9.96921e+36
DEBUG 4: valid time: 20190601_115959
DEBUG 4: lead time: 000000
DEBUG 4: init time: 20190601_115959
DEBUG 4: accum time: 000000
DEBUG 1: Regridding field precipitation_amount(*,*) to the
verification grid.
DEBUG 2: Processing series entry 1: Precipitation_Amount:
Precipitation_Amount(*,*)
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours since 1970-01-01"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit string "hours since 1970-01-01" as a reference time of
19700101_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> could not extract init time from the
"forecast_reference_time" variable.
DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
DEBUG 1: Regridding field precipitation_amount(*,*) to the
verification grid.
ERROR :
ERROR : DataPlane::two_to_one() -> range check error: (Nx, Ny) = (0,
0), (x, y) = (0, 0)
ERROR :
-----Original Message-----
From: John Halley Gotway via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>>
Sent: 08 December 2020 20:54
To: Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>>
Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
This email was received from an external source. Always check sender
details, links & attachments.
Marion,
I'm still not convinced. Please take a very careful look at the
command you're executing. I see all the log messages you sent, but not
the actual command you ran. At that matters a lot!
This command fails on my machine with the SAME ERROR you're getting:
* grid_diag -data glosea_2019060100_001.nc
<http://glosea_2019060100_001.nc/> gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea -out
grid_diag_out.nc <http://grid_diag_out.nc/>*
ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &)
const -> needed 3 arguments for variable precipitation_amount, got 2
But this command runs WITHOUT ERROR:
* grid_diag -data glosea_2019060100_001.nc
<http://glosea_2019060100_001.nc/> -data gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea -out
grid_diag_out.nc <http://grid_diag_out.nc/>*
Do you see the difference?
In the first, I used "-data" once and listed 2 different file names.
In the second, I used "-data" twice, once for each of the input files.
Please try doing exactly the same thing.
Does that enable it to run without error? Note that you'll still get
warnings about the actual data values falling outside the defined
range of the bins.
As for the bin endpoints, I think this is the relevant line of code:
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/met/src/libcode/vx_series_data/series_pdf.cc#L72
int k = floor((value - min) / delta);
This places the current "value" into the k-th bin where "min" is the
minimum PDF value and "delta" is the bin width.
I believe this is consistent with bins being closed on the left side
and open on the right:
[0,1) i.e. 0 <= x < 1
John
On Tue, Dec 8, 2020 at 8:30 AM
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>
via RT < met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>
> Hi John,
>
> one final question for GridDiag.
>
> The bins are set up to be lower bound inclusive? i.e.
> n_bins = 1000;
> range = [0, 1000]; // kg/m^2
>
> means that the bin is set up as [0,1) i.e. 0 <= x < 1?
>
> I'm presuming so.... and the upper bound would be [999,1000) with no
> bin for [1000,inf)? i.e. if there were a 1000 in my sample it would
> NOT be counted? Or anything bigger.
>
> That's what it looks like, i.e. you need to make sure that your
range
> encompasses all possible values. Not sure this is in the
documentation...?
> Would be useful to add.
>
> Ta
> Marion
>
> -----Original Message-----
> From: John Halley Gotway via RT
<met_help at ucar.edu<mailto:met_help at ucar.edu>>
> Sent: 05 December 2020 06:41
> To: Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>>
> Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
>
> This email was received from an external source. Always check
sender
> details, links & attachments.
>
> Marion,
>
> Ugh, I should've just handed this ticket off to David in the first
place.
> I did more testing and was able to get it to work. I just had to use
> separate "-data" command line options for each input data source.
>
> So this command fails:
> grid_diag -data glosea_2019060100_001.nc
gpm_2019060100_2019060200.nc
> -config GridDiag.glosea -out grid_diag_out.nc
>
> But this command runs fine:
> *grid_diag -data glosea_2019060100_001.nc <
> http://glosea_2019060100_001.nc> -data gpm_2019060100_2019060200.nc
<
> http://gpm_2019060100_2019060200.nc> -config GridDiag.glosea -out
> grid_diag_out.nc <http://grid_diag_out.nc>*
>
> In the second one, I've specified "-data" separately for each data
source.
> Although, you'll see warning about the range of the data extending
> beyond the range of the bins defined in the config file.
>
> Sorry to have such a rambling and confusing response!
>
> So to summarize, I believe the logic is:
> 1. If "-data" is used once, all specified input fields are read from
> all of those data files.
> 2. If "-data" is used more than once, the number of "-data" entries
> must match the number of fields defined in the config file.
> If not, you get an error message.
> If so, only the i-th field is read from the files in the i-th
> -data source.
>
> John
>
> On Fri, Dec 4, 2020 at 11:30 PM John Halley Gotway
<johnhg at ucar.edu<mailto:johnhg at ucar.edu>>
> wrote:
>
> > Marion,
> >
> > It occurs to me that perhaps Grid-Diag could be made more flexible
> > if we changed that error message to a debug message in this
context.
> > It could
> > *ATTEMPT* to read all input fields from all input files... but
when
> > it's unable to, have it print a log message stating that rather
than
> > erroring out.
> >
> > That would enable you to compute 2D pdf's using different
variables
> > from different input files.
> >
> > Do you think that logic is preferable?
> >
> > Thanks,
> > John
> >
> > On Fri, Dec 4, 2020 at 11:27 PM The RT System itself via RT <
> > met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
> >
> >>
> >> Fri Dec 04 23:27:19 2020: Request 97766 was acted upon.
> >> Transaction: Given to johnhg (John Halley Gotway) by RT_System
> >> Queue: met_help
> >> Subject: Need help with GridDiag
> >> Owner: johnhg
> >> Requestors:
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>
> >> Status: new
> >> Ticket <URL:
> >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
> >> >
> >>
> >>
> >> This transaction appears to have no content
> >>
> >
>
>
>
>
------------------------------------------------
Subject: Need help with GridDiag
From: John Halley Gotway
Time: Mon Feb 15 10:37:59 2021
Marion,
Thanks for sending sample data. I am a little confused though. The
config
file references variables named "precipitation_amount" and
"Precipitation_Amount":
* name = "precipitation_amount";*
* name = "Precipitation_Amount"; *
But the actual data you sent does not:
* ncdump -h glosea_2019060100_d001_001.nc
<http://glosea_2019060100_d001_001.nc> | grep float float
precipitation_amount(latitude, longitude) ;*
* ncdump -h gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc> | grep float float
precipitation_amount(latitude, longitude) ;*
I do believe the NetCDF is case-sensitive. So I think that's
important. But
I decided to just try to get Grid-Diag working on the data you sent. I
did
run into an error... different from the one you sent:
*terminate called after throwing an instance of
'netCDF::exceptions::NcNameInUse' what(): NetCDF: String match to
name in
use*
Grid-Diag reads the first variable from the first "-data" source and
the
second variable from the second "-data" source. And writes data to the
output NetCDF file. But since those variable names are the same,
there's a
name conflict! It's trying to write the same variable names to the
output
NetCDF file which causes the NetCDF library to error out.
To get around this, I redefined the input variable names and levels in
the
config file using:
* set_attr_name = "glosea_precip";*
* set_attr_level = "sfc"; set_attr_name = "gpm_precip";*
* set_attr_level = "sfc";*
That works fine and produces the attached output file. I also attached
my
run command, config file, and resulting log file.
Here's some improvements we could make:
(1) Using set_attr_name and set_attr_level works great. But if you
look
closely at the log messages, the updated strings are NOT used there.
Seems
like they should be. Do you agree?
(2) The actual data values don't go much over 250... but you've
defined
bins all the way out to 1000. That'll leave you with at least 75% of
the
bins empty. Can you think of any enhancements to make the setting of
the
number and size of bins easier?
(3) The GPM data appears to use a value of 9.96921E36 as missing data,
but
no _FillValue or missing_value variable attribute defines that. So MET
processes considers all these pink values to be valid data. Doesn't
matter
in this case because you're only using data from India, which has all
valid
data.
[image: Screen Shot 2021-02-15 at 10.31.31 AM.png]
I'd recommend using _FillValue or missing_value to define bad data
values
in the GPM files:
https://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-
conventions.html#missing-data
And do the same for the glosea files as well.
Thanks,
John
On Mon, Feb 15, 2021 at 4:43 AM marion.mittermaier at metoffice.gov.uk
via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>
> Hi all,
>
> in the mean time I have got regrid_data_plane to work in stand-alone
> model… running the following command (and I’ve tried in both
directions):
>
> regrid_data_plane glosea_2019060100_d001_001.nc
> gpm_2019060100_2019060200.nc grid_diag_out.nc -field 019060200.nc
> grid_diag_out.nc -field ‘name="precipitation_amount";
level="(*,*)";'
> <
>
> DEBUG 1: Reading data file: glosea_2019060100_d001_001.nc
> DEBUG 2: Input grid: Projection: Lat/Lon Nx: 432 Ny: 324 lat_ll:
-89.722
> lon_ll: -0.417 delta_lat: 0.556 delta_lon: 0.833
> DEBUG 2: Output grid: Projection: Lat/Lon Nx: 3600 Ny: 1800 lat_ll:
> -89.950 lon_ll: 179.950 delta_lat: 0.100 delta_lon: 0.100
> DEBUG 2: Interpolation options: method = NEAREST, width = 1, shape =
> SQUARE, vld_thresh = 0.5
> DEBUG 2: Range of input data (name="precipitation_amount";
level="(*,*)";)
> is 0 to 262.128.
> DEBUG 2: Range of regridded data (name="precipitation_amount";
> level="(*,*)";) is 0 to 262.128.
> DEBUG 1: Writing output file: grid_diag_out.nc
>
> Which looks sensible… so it’s NOT regrid_data_plane.
>
> Marion
>
> From: Mittermaier, Marion
> Sent: 15 February 2021 11:02
> To: 'met_help at ucar.edu' <met_help at ucar.edu>
> Subject: RE: [rt.rap.ucar.edu #97766] Need help with GridDiag
>
>
> Hi John,
>
>
>
> it's taken me a while to get back to this... I do see the difference
but I
> still don't get it to work. I can get either on their own to work
but not
> together.
>
> I have split out the forecast days into separate files.
>
>
>
> grid_diag -data
/scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc
> -data /scratch/fryl/GPM/gpm_2019060100_2019060200.nc -config
> GridDiag.glosea -out grid_diag_out.nc -v 4
>
>
>
> This gives the following error (I think it's in regrid_data_plane).
Not
> sure why it is trying to regrid both? Or maybe I’m just not quite
following
> the messages….
>
>
>
> vld497:/net/home/h02/frmm/WCSSP/India > grid_diag -data
> /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc -data
> /scratch/fryl/GPM/gpm_2019060100_2019060200.nc
>
> DEBUG 1: Default Config File:
>
/data/users/cfrd/MET_requirements/MET9.0/share/met/config/GridDiagConfig_default
>
> DEBUG 1: User Config File: GridDiag.glosea
>
> DEBUG 2: Read config files.
>
> DEBUG 2: Enter get_mtddf.
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
>
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours
> since 1970-01-01"
>
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit
> string "hours since 1970-01-01" as a reference time of
19700101_000000 and
> 3600 second(s) per time step.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
>
> DEBUG 2: Process configuration.
>
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> type "FileType_NcCF".
>
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object of
> type "FileType_NcCF".
>
> DEBUG 3: Use the grid defined by file "/scratch/fryl/WCSSPIndia/GPM/
> gpm_2019060100_2019060200.nc".
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
>
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours
> since 1970-01-01"
>
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit
> string "hours since 1970-01-01" as a reference time of
19700101_000000 and
> 3600 second(s) per time step.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
>
> DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 360
lat_ll:
> 4.550 lon_ll: -64.550 delta_lat: 0.100 delta_lon: 0.100
>
> DEBUG 2: Processing masking regions.
>
> DEBUG 3: Processing poly mask: India.poly
>
> DEBUG 4: parse_poly_mask() -> parsing poly mask "India.poly"
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_None".
>
> DEBUG 4: parse_poly_mask() -> parsing poly mask file "India.poly"
>
> DEBUG 2: Initializing precipitation_amount(*,*) histogram
>
> DEBUG 2: Initializing Precipitation_Amount(*,*) histogram
>
> DEBUG 2: Initializing
precipitation_amount(*,*)_Precipitation_Amount(*,*)
> joint histogram
>
> DEBUG 1: grid_diag_out.nc
>
> DEBUG 1: Length of configuration "data.field" = 2
>
> DEBUG 1: Length of data file list = 1
>
> DEBUG 1: Series defined by the data file list of length 1.
>
> DEBUG 2: 1000
>
> DEBUG 2: Processing series entry 1: precipitation_amount:
> precipitation_amount(*,*)
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
>
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours
> since 1970-01-01"
>
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit
> string "hours since 1970-01-01" as a reference time of
19700101_000000 and
> 3600 second(s) per time step.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
>
> DEBUG 4: NcCfFile::getData() -> setting the unset init time to the
valid
> time of 20190601_115959.
>
> DEBUG 4:
>
> DEBUG 4: Data plane information:
>
> DEBUG 4: plane min: 0
>
> DEBUG 4: plane max: 9.96921e+36
>
> DEBUG 4: valid time: 20190601_115959
>
> DEBUG 4: lead time: 000000
>
> DEBUG 4: init time: 20190601_115959
>
> DEBUG 4: accum time: 000000
>
> DEBUG 1: Regridding field precipitation_amount(*,*) to the
verification
> grid.
>
> DEBUG 2: Processing series entry 1: Precipitation_Amount:
> Precipitation_Amount(*,*)
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
>
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
"hours
> since 1970-01-01"
>
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
unit
> string "hours since 1970-01-01" as a reference time of
19700101_000000 and
> 3600 second(s) per time step.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
>
> DEBUG 1: Regridding field precipitation_amount(*,*) to the
verification
> grid.
>
> ERROR :
>
> ERROR : DataPlane::two_to_one() -> range check error: (Nx, Ny) =
(0, 0),
> (x, y) = (0, 0)
>
> ERROR :
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu<mailto:
> met_help at ucar.edu>>
> Sent: 08 December 2020 20:54
> To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk<mailto:
> marion.mittermaier at metoffice.gov.uk>>
> Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
>
>
>
> This email was received from an external source. Always check
sender
> details, links & attachments.
>
>
>
> Marion,
>
>
>
> I'm still not convinced. Please take a very careful look at the
command
> you're executing. I see all the log messages you sent, but not the
actual
> command you ran. At that matters a lot!
>
>
>
> This command fails on my machine with the SAME ERROR you're getting:
>
> * grid_diag -data glosea_2019060100_001.nc
>
> <http://glosea_2019060100_001.nc/> gpm_2019060100_2019060200.nc <
> http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea -out
> grid_diag_out.nc <http://grid_diag_out.nc/>*
>
>
>
> ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane
&)
> const -> needed 3 arguments for variable precipitation_amount, got 2
>
>
>
> But this command runs WITHOUT ERROR:
>
> * grid_diag -data glosea_2019060100_001.nc
>
> <http://glosea_2019060100_001.nc/> -data
gpm_2019060100_2019060200.nc <
> http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea -out
> grid_diag_out.nc <http://grid_diag_out.nc/>*
>
>
>
> Do you see the difference?
>
> In the first, I used "-data" once and listed 2 different file names.
>
> In the second, I used "-data" twice, once for each of the input
files.
>
>
>
> Please try doing exactly the same thing.
>
>
>
> Does that enable it to run without error? Note that you'll still get
> warnings about the actual data values falling outside the defined
range of
> the bins.
>
>
>
> As for the bin endpoints, I think this is the relevant line of code:
>
>
>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc5981bb8/met/src/libcode/vx_series_data/series_pdf.cc#L72
>
>
>
> int k = floor((value - min) / delta);
>
>
>
> This places the current "value" into the k-th bin where "min" is the
> minimum PDF value and "delta" is the bin width.
>
> I believe this is consistent with bins being closed on the left side
and
> open on the right:
>
> [0,1) i.e. 0 <= x < 1
>
>
>
> John
>
>
>
> On Tue, Dec 8, 2020 at 8:30 AM
marion.mittermaier at metoffice.gov.uk<mailto:
> marion.mittermaier at metoffice.gov.uk> via RT <
met_help at ucar.edu<mailto:
> met_help at ucar.edu>> wrote:
>
>
>
> >
>
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>
> >
>
> > Hi John,
>
> >
>
> > one final question for GridDiag.
>
> >
>
> > The bins are set up to be lower bound inclusive? i.e.
>
> > n_bins = 1000;
>
> > range = [0, 1000]; // kg/m^2
>
> >
>
> > means that the bin is set up as [0,1) i.e. 0 <= x < 1?
>
> >
>
> > I'm presuming so.... and the upper bound would be [999,1000) with
no
>
> > bin for [1000,inf)? i.e. if there were a 1000 in my sample it
would
>
> > NOT be counted? Or anything bigger.
>
> >
>
> > That's what it looks like, i.e. you need to make sure that your
range
>
> > encompasses all possible values. Not sure this is in the
> documentation...?
>
> > Would be useful to add.
>
> >
>
> > Ta
>
> > Marion
>
> >
>
> > -----Original Message-----
>
> > From: John Halley Gotway via RT <met_help at ucar.edu<mailto:
> met_help at ucar.edu>>
>
> > Sent: 05 December 2020 06:41
>
> > To: Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk<mailto:
> marion.mittermaier at metoffice.gov.uk>>
>
> > Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
>
> >
>
> > This email was received from an external source. Always check
sender
>
> > details, links & attachments.
>
> >
>
> > Marion,
>
> >
>
> > Ugh, I should've just handed this ticket off to David in the first
place.
>
> > I did more testing and was able to get it to work. I just had to
use
>
> > separate "-data" command line options for each input data source.
>
> >
>
> > So this command fails:
>
> > grid_diag -data glosea_2019060100_001.nc
gpm_2019060100_2019060200.nc
>
> > -config GridDiag.glosea -out grid_diag_out.nc
>
> >
>
> > But this command runs fine:
>
> > *grid_diag -data glosea_2019060100_001.nc <
>
> > http://glosea_2019060100_001.nc> -data
gpm_2019060100_2019060200.nc <
>
> > http://gpm_2019060100_2019060200.nc> -config GridDiag.glosea -out
>
> > grid_diag_out.nc <http://grid_diag_out.nc>*
>
> >
>
> > In the second one, I've specified "-data" separately for each data
> source.
>
> > Although, you'll see warning about the range of the data extending
>
> > beyond the range of the bins defined in the config file.
>
> >
>
> > Sorry to have such a rambling and confusing response!
>
> >
>
> > So to summarize, I believe the logic is:
>
> > 1. If "-data" is used once, all specified input fields are read
from
>
> > all of those data files.
>
> > 2. If "-data" is used more than once, the number of "-data"
entries
>
> > must match the number of fields defined in the config file.
>
> > If not, you get an error message.
>
> > If so, only the i-th field is read from the files in the i-th
>
> > -data source.
>
> >
>
> > John
>
> >
>
> > On Fri, Dec 4, 2020 at 11:30 PM John Halley Gotway
<johnhg at ucar.edu
> <mailto:johnhg at ucar.edu>>
>
> > wrote:
>
> >
>
> > > Marion,
>
> > >
>
> > > It occurs to me that perhaps Grid-Diag could be made more
flexible
>
> > > if we changed that error message to a debug message in this
context.
>
> > > It could
>
> > > *ATTEMPT* to read all input fields from all input files... but
when
>
> > > it's unable to, have it print a log message stating that rather
than
>
> > > erroring out.
>
> > >
>
> > > That would enable you to compute 2D pdf's using different
variables
>
> > > from different input files.
>
> > >
>
> > > Do you think that logic is preferable?
>
> > >
>
> > > Thanks,
>
> > > John
>
> > >
>
> > > On Fri, Dec 4, 2020 at 11:27 PM The RT System itself via RT <
>
> > > met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
>
> > >
>
> > >>
>
> > >> Fri Dec 04 23:27:19 2020: Request 97766 was acted upon.
>
> > >> Transaction: Given to johnhg (John Halley Gotway) by RT_System
>
> > >> Queue: met_help
>
> > >> Subject: Need help with GridDiag
>
> > >> Owner: johnhg
>
> > >> Requestors: marion.mittermaier at metoffice.gov.uk<mailto:
> marion.mittermaier at metoffice.gov.uk>
>
> > >> Status: new
>
> > >> Ticket <URL:
>
> > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
>
> > >> >
>
> > >>
>
> > >>
>
> > >> This transaction appears to have no content
>
> > >>
>
> > >
>
> >
>
> >
>
> >
>
> >
>
>
>
>
------------------------------------------------
Subject: Need help with GridDiag
From: marion.mittermaier at metoffice.gov.uk
Time: Tue Feb 16 08:03:24 2021
Hi John,
for info, I can't get that to work in MET9.0. It gives the same
error. I presume you know that?!
terminate called after throwing an instance of
'netCDF::exceptions::NcNameInUse'
what(): NetCDF: String match to name in use
file: ncGroup.cpp line:1016
Abort
I have managed to get it to work with MET9.1, which is what you used
(but I wasn’t). 😊 Thankfully we have that installed.
I will go and check out the output.
Further to your points:
(1) Knowing about the set_attr_name / level is really useful. This
could happen a lot.
(2) The distribution doesn't go above 250 for this particular forecast
but when run for a whole month/months (as I have done for the obs and
fc’ss separately), values do go that high. Trying to get some sort of
automatic bin setting is tricky business.
(3) This was known to Rachel, our resident GPM guru, and has been
fixed for more recent data conversions. As you point out, it does not
make a difference for this particular application but it is something
that needs attention... an a classic example of CF compliance
checking.
Thank you! I shall see what "trouble" I can get into next 😉
Regards
Marion
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 15 February 2021 17:38
To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
Subject: Re: FW: [rt.rap.ucar.edu #97766] Need help with GridDiag
This email was received from an external source. Always check sender
details, links & attachments.
Marion,
Thanks for sending sample data. I am a little confused though. The
config file references variables named "precipitation_amount" and
"Precipitation_Amount":
* name = "precipitation_amount";*
* name = "Precipitation_Amount"; *
But the actual data you sent does not:
* ncdump -h glosea_2019060100_d001_001.nc
<http://glosea_2019060100_d001_001.nc> | grep float float
precipitation_amount(latitude, longitude) ;*
* ncdump -h gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc> | grep float float
precipitation_amount(latitude, longitude) ;*
I do believe the NetCDF is case-sensitive. So I think that's
important. But I decided to just try to get Grid-Diag working on the
data you sent. I did run into an error... different from the one you
sent:
*terminate called after throwing an instance of
'netCDF::exceptions::NcNameInUse' what(): NetCDF: String match to
name in
use*
Grid-Diag reads the first variable from the first "-data" source and
the second variable from the second "-data" source. And writes data to
the output NetCDF file. But since those variable names are the same,
there's a name conflict! It's trying to write the same variable names
to the output NetCDF file which causes the NetCDF library to error
out.
To get around this, I redefined the input variable names and levels in
the config file using:
* set_attr_name = "glosea_precip";*
* set_attr_level = "sfc"; set_attr_name = "gpm_precip";*
* set_attr_level = "sfc";*
That works fine and produces the attached output file. I also attached
my run command, config file, and resulting log file.
Here's some improvements we could make:
(1) Using set_attr_name and set_attr_level works great. But if you
look closely at the log messages, the updated strings are NOT used
there. Seems like they should be. Do you agree?
(2) The actual data values don't go much over 250... but you've
defined bins all the way out to 1000. That'll leave you with at least
75% of the bins empty. Can you think of any enhancements to make the
setting of the number and size of bins easier?
(3) The GPM data appears to use a value of 9.96921E36 as missing data,
but no _FillValue or missing_value variable attribute defines that. So
MET processes considers all these pink values to be valid data.
Doesn't matter in this case because you're only using data from India,
which has all valid data.
[image: Screen Shot 2021-02-15 at 10.31.31 AM.png] I'd recommend using
_FillValue or missing_value to define bad data values in the GPM
files:
https://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-
conventions.html#missing-data
And do the same for the glosea files as well.
Thanks,
John
On Mon, Feb 15, 2021 at 4:43 AM
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>
via RT < met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>
> Hi all,
>
> in the mean time I have got regrid_data_plane to work in stand-alone
> model… running the following command (and I’ve tried in both
directions):
>
> regrid_data_plane glosea_2019060100_d001_001.nc
> gpm_2019060100_2019060200.nc grid_diag_out.nc -field 019060200.nc
> grid_diag_out.nc -field ‘name="precipitation_amount";
level="(*,*)";'
> <
>
> DEBUG 1: Reading data file: glosea_2019060100_d001_001.nc DEBUG 2:
> Input grid: Projection: Lat/Lon Nx: 432 Ny: 324 lat_ll: -89.722
> lon_ll: -0.417 delta_lat: 0.556 delta_lon: 0.833 DEBUG 2: Output
grid:
> Projection: Lat/Lon Nx: 3600 Ny: 1800 lat_ll:
> -89.950 lon_ll: 179.950 delta_lat: 0.100 delta_lon: 0.100 DEBUG 2:
> Interpolation options: method = NEAREST, width = 1, shape = SQUARE,
> vld_thresh = 0.5 DEBUG 2: Range of input data
> (name="precipitation_amount"; level="(*,*)";) is 0 to 262.128.
> DEBUG 2: Range of regridded data (name="precipitation_amount";
> level="(*,*)";) is 0 to 262.128.
> DEBUG 1: Writing output file: grid_diag_out.nc
>
> Which looks sensible… so it’s NOT regrid_data_plane.
>
> Marion
>
> From: Mittermaier, Marion
> Sent: 15 February 2021 11:02
> To: 'met_help at ucar.edu'
<met_help at ucar.edu<mailto:met_help at ucar.edu>>
> Subject: RE: [rt.rap.ucar.edu #97766] Need help with GridDiag
>
>
> Hi John,
>
>
>
> it's taken me a while to get back to this... I do see the difference
> but I still don't get it to work. I can get either on their own to
> work but not together.
>
> I have split out the forecast days into separate files.
>
>
>
> grid_diag -data
> /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc
> -data /scratch/fryl/GPM/gpm_2019060100_2019060200.nc -config
> GridDiag.glosea -out grid_diag_out.nc -v 4
>
>
>
> This gives the following error (I think it's in regrid_data_plane).
> Not sure why it is trying to regrid both? Or maybe I’m just not
quite
> following the messages….
>
>
>
> vld497:/net/home/h02/frmm/WCSSP/India > grid_diag -data
> /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc -data
> /scratch/fryl/GPM/gpm_2019060100_2019060200.nc
>
> DEBUG 1: Default Config File:
>
/data/users/cfrd/MET_requirements/MET9.0/share/met/config/GridDiagConf
> ig_default
>
> DEBUG 1: User Config File: GridDiag.glosea
>
> DEBUG 2: Read config files.
>
> DEBUG 2: Enter get_mtddf.
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
>
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> "hours since 1970-01-01"
>
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01" as a reference time of
> 19700101_000000 and
> 3600 second(s) per time step.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
>
> DEBUG 2: Process configuration.
>
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of type "FileType_NcCF".
>
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of type "FileType_NcCF".
>
> DEBUG 3: Use the grid defined by file "/scratch/fryl/WCSSPIndia/GPM/
> gpm_2019060100_2019060200.nc".
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
>
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> "hours since 1970-01-01"
>
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01" as a reference time of
> 19700101_000000 and
> 3600 second(s) per time step.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
>
> DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 360
lat_ll:
> 4.550 lon_ll: -64.550 delta_lat: 0.100 delta_lon: 0.100
>
> DEBUG 2: Processing masking regions.
>
> DEBUG 3: Processing poly mask: India.poly
>
> DEBUG 4: parse_poly_mask() -> parsing poly mask "India.poly"
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_None".
>
> DEBUG 4: parse_poly_mask() -> parsing poly mask file "India.poly"
>
> DEBUG 2: Initializing precipitation_amount(*,*) histogram
>
> DEBUG 2: Initializing Precipitation_Amount(*,*) histogram
>
> DEBUG 2: Initializing
> precipitation_amount(*,*)_Precipitation_Amount(*,*)
> joint histogram
>
> DEBUG 1: grid_diag_out.nc
>
> DEBUG 1: Length of configuration "data.field" = 2
>
> DEBUG 1: Length of data file list = 1
>
> DEBUG 1: Series defined by the data file list of length 1.
>
> DEBUG 2: 1000
>
> DEBUG 2: Processing series entry 1: precipitation_amount:
> precipitation_amount(*,*)
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
>
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> "hours since 1970-01-01"
>
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01" as a reference time of
> 19700101_000000 and
> 3600 second(s) per time step.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
>
> DEBUG 4: NcCfFile::getData() -> setting the unset init time to the
> valid time of 20190601_115959.
>
> DEBUG 4:
>
> DEBUG 4: Data plane information:
>
> DEBUG 4: plane min: 0
>
> DEBUG 4: plane max: 9.96921e+36
>
> DEBUG 4: valid time: 20190601_115959
>
> DEBUG 4: lead time: 000000
>
> DEBUG 4: init time: 20190601_115959
>
> DEBUG 4: accum time: 000000
>
> DEBUG 1: Regridding field precipitation_amount(*,*) to the
> verification grid.
>
> DEBUG 2: Processing series entry 1: Precipitation_Amount:
> Precipitation_Amount(*,*)
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
>
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> "hours since 1970-01-01"
>
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01" as a reference time of
> 19700101_000000 and
> 3600 second(s) per time step.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
>
> DEBUG 1: Regridding field precipitation_amount(*,*) to the
> verification grid.
>
> ERROR :
>
> ERROR : DataPlane::two_to_one() -> range check error: (Nx, Ny) =
(0,
> 0), (x, y) = (0, 0)
>
> ERROR :
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu<mailto:
> met_help at ucar.edu<mailto:met_help at ucar.edu>>>
> Sent: 08 December 2020 20:54
> To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk<mailto:
>
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>>>
> Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
>
>
>
> This email was received from an external source. Always check
sender
> details, links & attachments.
>
>
>
> Marion,
>
>
>
> I'm still not convinced. Please take a very careful look at the
> command you're executing. I see all the log messages you sent, but
not
> the actual command you ran. At that matters a lot!
>
>
>
> This command fails on my machine with the SAME ERROR you're getting:
>
> * grid_diag -data glosea_2019060100_001.nc
>
> <http://glosea_2019060100_001.nc/> gpm_2019060100_2019060200.nc <
> http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea -out
> grid_diag_out.nc <http://grid_diag_out.nc/>*
>
>
>
> ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane
&)
> const -> needed 3 arguments for variable precipitation_amount, got 2
>
>
>
> But this command runs WITHOUT ERROR:
>
> * grid_diag -data glosea_2019060100_001.nc
>
> <http://glosea_2019060100_001.nc/> -data
gpm_2019060100_2019060200.nc
> < http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea -out
> grid_diag_out.nc <http://grid_diag_out.nc/>*
>
>
>
> Do you see the difference?
>
> In the first, I used "-data" once and listed 2 different file names.
>
> In the second, I used "-data" twice, once for each of the input
files.
>
>
>
> Please try doing exactly the same thing.
>
>
>
> Does that enable it to run without error? Note that you'll still get
> warnings about the actual data values falling outside the defined
> range of the bins.
>
>
>
> As for the bin endpoints, I think this is the relevant line of code:
>
>
>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc
> 5981bb8/met/src/libcode/vx_series_data/series_pdf.cc#L72
>
>
>
> int k = floor((value - min) / delta);
>
>
>
> This places the current "value" into the k-th bin where "min" is the
> minimum PDF value and "delta" is the bin width.
>
> I believe this is consistent with bins being closed on the left side
> and open on the right:
>
> [0,1) i.e. 0 <= x < 1
>
>
>
> John
>
>
>
> On Tue, Dec 8, 2020 at 8:30 AM
marion.mittermaier at metoffice.gov.uk<mailto<mailto:marion.mittermaier at metoffice.gov.uk%3cmailto>:
>
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>>
via RT < met_help at ucar.edu<mailto<mailto:met_help at ucar.edu%3cmailto>:
> met_help at ucar.edu<mailto:met_help at ucar.edu>>> wrote:
>
>
>
> >
>
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>
> >
>
> > Hi John,
>
> >
>
> > one final question for GridDiag.
>
> >
>
> > The bins are set up to be lower bound inclusive? i.e.
>
> > n_bins = 1000;
>
> > range = [0, 1000]; // kg/m^2
>
> >
>
> > means that the bin is set up as [0,1) i.e. 0 <= x < 1?
>
> >
>
> > I'm presuming so.... and the upper bound would be [999,1000) with
no
>
> > bin for [1000,inf)? i.e. if there were a 1000 in my sample it
would
>
> > NOT be counted? Or anything bigger.
>
> >
>
> > That's what it looks like, i.e. you need to make sure that your
> > range
>
> > encompasses all possible values. Not sure this is in the
> documentation...?
>
> > Would be useful to add.
>
> >
>
> > Ta
>
> > Marion
>
> >
>
> > -----Original Message-----
>
> > From: John Halley Gotway via RT <met_help at ucar.edu<mailto:
> met_help at ucar.edu<mailto:met_help at ucar.edu>>>
>
> > Sent: 05 December 2020 06:41
>
> > To: Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk<mailto:
>
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>>>
>
> > Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
>
> >
>
> > This email was received from an external source. Always check
sender
>
> > details, links & attachments.
>
> >
>
> > Marion,
>
> >
>
> > Ugh, I should've just handed this ticket off to David in the first
place.
>
> > I did more testing and was able to get it to work. I just had to
use
>
> > separate "-data" command line options for each input data source.
>
> >
>
> > So this command fails:
>
> > grid_diag -data glosea_2019060100_001.nc
> > gpm_2019060100_2019060200.nc
>
> > -config GridDiag.glosea -out grid_diag_out.nc
>
> >
>
> > But this command runs fine:
>
> > *grid_diag -data glosea_2019060100_001.nc <
>
> > http://glosea_2019060100_001.nc> -data
gpm_2019060100_2019060200.nc
> > <
>
> > http://gpm_2019060100_2019060200.nc> -config GridDiag.glosea -out
>
> > grid_diag_out.nc <http://grid_diag_out.nc>*
>
> >
>
> > In the second one, I've specified "-data" separately for each data
> source.
>
> > Although, you'll see warning about the range of the data extending
>
> > beyond the range of the bins defined in the config file.
>
> >
>
> > Sorry to have such a rambling and confusing response!
>
> >
>
> > So to summarize, I believe the logic is:
>
> > 1. If "-data" is used once, all specified input fields are read
from
>
> > all of those data files.
>
> > 2. If "-data" is used more than once, the number of "-data"
entries
>
> > must match the number of fields defined in the config file.
>
> > If not, you get an error message.
>
> > If so, only the i-th field is read from the files in the i-th
>
> > -data source.
>
> >
>
> > John
>
> >
>
> > On Fri, Dec 4, 2020 at 11:30 PM John Halley Gotway
<johnhg at ucar.edu
> <mailto:johnhg at ucar.edu>>
>
> > wrote:
>
> >
>
> > > Marion,
>
> > >
>
> > > It occurs to me that perhaps Grid-Diag could be made more
flexible
>
> > > if we changed that error message to a debug message in this
context.
>
> > > It could
>
> > > *ATTEMPT* to read all input fields from all input files... but
> > > when
>
> > > it's unable to, have it print a log message stating that rather
> > > than
>
> > > erroring out.
>
> > >
>
> > > That would enable you to compute 2D pdf's using different
> > > variables
>
> > > from different input files.
>
> > >
>
> > > Do you think that logic is preferable?
>
> > >
>
> > > Thanks,
>
> > > John
>
> > >
>
> > > On Fri, Dec 4, 2020 at 11:27 PM The RT System itself via RT <
>
> > >
met_help at ucar.edu<mailto:met_help at ucar.edu<mailto:met_help at ucar.edu%3cmailto:met_help at ucar.edu>>>
wrote:
>
> > >
>
> > >>
>
> > >> Fri Dec 04 23:27:19 2020: Request 97766 was acted upon.
>
> > >> Transaction: Given to johnhg (John Halley Gotway) by RT_System
>
> > >> Queue: met_help
>
> > >> Subject: Need help with GridDiag
>
> > >> Owner: johnhg
>
> > >> Requestors:
marion.mittermaier at metoffice.gov.uk<mailto<mailto:marion.mittermaier at metoffice.gov.uk%3cmailto>:
>
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>>
>
> > >> Status: new
>
> > >> Ticket <URL:
>
> > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
>
> > >> >
>
> > >>
>
> > >>
>
> > >> This transaction appears to have no content
>
> > >>
>
> > >
>
> >
>
> >
>
> >
>
> >
>
>
>
>
------------------------------------------------
Subject: Need help with GridDiag
From: marion.mittermaier at metoffice.gov.uk
Time: Wed Feb 17 04:19:42 2021
Hi John,
this all works now which is great. Thanks so much. One small question,
and this may be solved in the GridDiag wrapper github issue that I saw
George mention in the governance meeting yesterday is the cycling.
At the moment off the command line I can't get it to run off two file
lists, i.e. it doesn't seem to like -data fc_list -data obs_list. This
only worked with one file list. My question would then be can the list
be a mixed list? i.e. the forecast and observation files interspersed
in the same file list? I haven't tried that. I only used the single
file list for processing either the forecast or the observations.
Anyway, it's not a blocker right now and I plan to investigate what
George mentioned to see if that is what I think it is. At this stage
I'm writing out each file separately and I will then combine the joint
histograms later. It does mean more output files but I can cope with
that! Main thing is I am getting some output and I can hopefully get a
progress report written before the end of March 😉
Regards
Marion
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 15 February 2021 17:38
To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
Subject: Re: FW: [rt.rap.ucar.edu #97766] Need help with GridDiag
This email was received from an external source. Always check sender
details, links & attachments.
Marion,
Thanks for sending sample data. I am a little confused though. The
config file references variables named "precipitation_amount" and
"Precipitation_Amount":
* name = "precipitation_amount";*
* name = "Precipitation_Amount"; *
But the actual data you sent does not:
* ncdump -h glosea_2019060100_d001_001.nc
<http://glosea_2019060100_d001_001.nc> | grep float float
precipitation_amount(latitude, longitude) ;*
* ncdump -h gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc> | grep float float
precipitation_amount(latitude, longitude) ;*
I do believe the NetCDF is case-sensitive. So I think that's
important. But I decided to just try to get Grid-Diag working on the
data you sent. I did run into an error... different from the one you
sent:
*terminate called after throwing an instance of
'netCDF::exceptions::NcNameInUse' what(): NetCDF: String match to
name in
use*
Grid-Diag reads the first variable from the first "-data" source and
the second variable from the second "-data" source. And writes data to
the output NetCDF file. But since those variable names are the same,
there's a name conflict! It's trying to write the same variable names
to the output NetCDF file which causes the NetCDF library to error
out.
To get around this, I redefined the input variable names and levels in
the config file using:
* set_attr_name = "glosea_precip";*
* set_attr_level = "sfc"; set_attr_name = "gpm_precip";*
* set_attr_level = "sfc";*
That works fine and produces the attached output file. I also attached
my run command, config file, and resulting log file.
Here's some improvements we could make:
(1) Using set_attr_name and set_attr_level works great. But if you
look closely at the log messages, the updated strings are NOT used
there. Seems like they should be. Do you agree?
(2) The actual data values don't go much over 250... but you've
defined bins all the way out to 1000. That'll leave you with at least
75% of the bins empty. Can you think of any enhancements to make the
setting of the number and size of bins easier?
(3) The GPM data appears to use a value of 9.96921E36 as missing data,
but no _FillValue or missing_value variable attribute defines that. So
MET processes considers all these pink values to be valid data.
Doesn't matter in this case because you're only using data from India,
which has all valid data.
[image: Screen Shot 2021-02-15 at 10.31.31 AM.png] I'd recommend using
_FillValue or missing_value to define bad data values in the GPM
files:
https://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-
conventions.html#missing-data
And do the same for the glosea files as well.
Thanks,
John
On Mon, Feb 15, 2021 at 4:43 AM
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>
via RT < met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>
> Hi all,
>
> in the mean time I have got regrid_data_plane to work in stand-alone
> model… running the following command (and I’ve tried in both
directions):
>
> regrid_data_plane glosea_2019060100_d001_001.nc
> gpm_2019060100_2019060200.nc grid_diag_out.nc -field 019060200.nc
> grid_diag_out.nc -field ‘name="precipitation_amount";
level="(*,*)";'
> <
>
> DEBUG 1: Reading data file: glosea_2019060100_d001_001.nc DEBUG 2:
> Input grid: Projection: Lat/Lon Nx: 432 Ny: 324 lat_ll: -89.722
> lon_ll: -0.417 delta_lat: 0.556 delta_lon: 0.833 DEBUG 2: Output
grid:
> Projection: Lat/Lon Nx: 3600 Ny: 1800 lat_ll:
> -89.950 lon_ll: 179.950 delta_lat: 0.100 delta_lon: 0.100 DEBUG 2:
> Interpolation options: method = NEAREST, width = 1, shape = SQUARE,
> vld_thresh = 0.5 DEBUG 2: Range of input data
> (name="precipitation_amount"; level="(*,*)";) is 0 to 262.128.
> DEBUG 2: Range of regridded data (name="precipitation_amount";
> level="(*,*)";) is 0 to 262.128.
> DEBUG 1: Writing output file: grid_diag_out.nc
>
> Which looks sensible… so it’s NOT regrid_data_plane.
>
> Marion
>
> From: Mittermaier, Marion
> Sent: 15 February 2021 11:02
> To: 'met_help at ucar.edu'
<met_help at ucar.edu<mailto:met_help at ucar.edu>>
> Subject: RE: [rt.rap.ucar.edu #97766] Need help with GridDiag
>
>
> Hi John,
>
>
>
> it's taken me a while to get back to this... I do see the difference
> but I still don't get it to work. I can get either on their own to
> work but not together.
>
> I have split out the forecast days into separate files.
>
>
>
> grid_diag -data
> /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc
> -data /scratch/fryl/GPM/gpm_2019060100_2019060200.nc -config
> GridDiag.glosea -out grid_diag_out.nc -v 4
>
>
>
> This gives the following error (I think it's in regrid_data_plane).
> Not sure why it is trying to regrid both? Or maybe I’m just not
quite
> following the messages….
>
>
>
> vld497:/net/home/h02/frmm/WCSSP/India > grid_diag -data
> /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc -data
> /scratch/fryl/GPM/gpm_2019060100_2019060200.nc
>
> DEBUG 1: Default Config File:
>
/data/users/cfrd/MET_requirements/MET9.0/share/met/config/GridDiagConf
> ig_default
>
> DEBUG 1: User Config File: GridDiag.glosea
>
> DEBUG 2: Read config files.
>
> DEBUG 2: Enter get_mtddf.
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
>
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> "hours since 1970-01-01"
>
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01" as a reference time of
> 19700101_000000 and
> 3600 second(s) per time step.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
>
> DEBUG 2: Process configuration.
>
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of type "FileType_NcCF".
>
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of type "FileType_NcCF".
>
> DEBUG 3: Use the grid defined by file "/scratch/fryl/WCSSPIndia/GPM/
> gpm_2019060100_2019060200.nc".
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
>
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> "hours since 1970-01-01"
>
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01" as a reference time of
> 19700101_000000 and
> 3600 second(s) per time step.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
>
> DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 360
lat_ll:
> 4.550 lon_ll: -64.550 delta_lat: 0.100 delta_lon: 0.100
>
> DEBUG 2: Processing masking regions.
>
> DEBUG 3: Processing poly mask: India.poly
>
> DEBUG 4: parse_poly_mask() -> parsing poly mask "India.poly"
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_None".
>
> DEBUG 4: parse_poly_mask() -> parsing poly mask file "India.poly"
>
> DEBUG 2: Initializing precipitation_amount(*,*) histogram
>
> DEBUG 2: Initializing Precipitation_Amount(*,*) histogram
>
> DEBUG 2: Initializing
> precipitation_amount(*,*)_Precipitation_Amount(*,*)
> joint histogram
>
> DEBUG 1: grid_diag_out.nc
>
> DEBUG 1: Length of configuration "data.field" = 2
>
> DEBUG 1: Length of data file list = 1
>
> DEBUG 1: Series defined by the data file list of length 1.
>
> DEBUG 2: 1000
>
> DEBUG 2: Processing series entry 1: precipitation_amount:
> precipitation_amount(*,*)
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
>
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> "hours since 1970-01-01"
>
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01" as a reference time of
> 19700101_000000 and
> 3600 second(s) per time step.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
>
> DEBUG 4: NcCfFile::getData() -> setting the unset init time to the
> valid time of 20190601_115959.
>
> DEBUG 4:
>
> DEBUG 4: Data plane information:
>
> DEBUG 4: plane min: 0
>
> DEBUG 4: plane max: 9.96921e+36
>
> DEBUG 4: valid time: 20190601_115959
>
> DEBUG 4: lead time: 000000
>
> DEBUG 4: init time: 20190601_115959
>
> DEBUG 4: accum time: 000000
>
> DEBUG 1: Regridding field precipitation_amount(*,*) to the
> verification grid.
>
> DEBUG 2: Processing series entry 1: Precipitation_Amount:
> Precipitation_Amount(*,*)
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
>
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> "hours since 1970-01-01"
>
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01" as a reference time of
> 19700101_000000 and
> 3600 second(s) per time step.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
>
> DEBUG 1: Regridding field precipitation_amount(*,*) to the
> verification grid.
>
> ERROR :
>
> ERROR : DataPlane::two_to_one() -> range check error: (Nx, Ny) =
(0,
> 0), (x, y) = (0, 0)
>
> ERROR :
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu<mailto:
> met_help at ucar.edu<mailto:met_help at ucar.edu>>>
> Sent: 08 December 2020 20:54
> To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk<mailto:
>
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>>>
> Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
>
>
>
> This email was received from an external source. Always check
sender
> details, links & attachments.
>
>
>
> Marion,
>
>
>
> I'm still not convinced. Please take a very careful look at the
> command you're executing. I see all the log messages you sent, but
not
> the actual command you ran. At that matters a lot!
>
>
>
> This command fails on my machine with the SAME ERROR you're getting:
>
> * grid_diag -data glosea_2019060100_001.nc
>
> <http://glosea_2019060100_001.nc/> gpm_2019060100_2019060200.nc <
> http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea -out
> grid_diag_out.nc <http://grid_diag_out.nc/>*
>
>
>
> ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane
&)
> const -> needed 3 arguments for variable precipitation_amount, got 2
>
>
>
> But this command runs WITHOUT ERROR:
>
> * grid_diag -data glosea_2019060100_001.nc
>
> <http://glosea_2019060100_001.nc/> -data
gpm_2019060100_2019060200.nc
> < http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea -out
> grid_diag_out.nc <http://grid_diag_out.nc/>*
>
>
>
> Do you see the difference?
>
> In the first, I used "-data" once and listed 2 different file names.
>
> In the second, I used "-data" twice, once for each of the input
files.
>
>
>
> Please try doing exactly the same thing.
>
>
>
> Does that enable it to run without error? Note that you'll still get
> warnings about the actual data values falling outside the defined
> range of the bins.
>
>
>
> As for the bin endpoints, I think this is the relevant line of code:
>
>
>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc
> 5981bb8/met/src/libcode/vx_series_data/series_pdf.cc#L72
>
>
>
> int k = floor((value - min) / delta);
>
>
>
> This places the current "value" into the k-th bin where "min" is the
> minimum PDF value and "delta" is the bin width.
>
> I believe this is consistent with bins being closed on the left side
> and open on the right:
>
> [0,1) i.e. 0 <= x < 1
>
>
>
> John
>
>
>
> On Tue, Dec 8, 2020 at 8:30 AM
marion.mittermaier at metoffice.gov.uk<mailto<mailto:marion.mittermaier at metoffice.gov.uk%3cmailto>:
>
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>>
via RT < met_help at ucar.edu<mailto<mailto:met_help at ucar.edu%3cmailto>:
> met_help at ucar.edu<mailto:met_help at ucar.edu>>> wrote:
>
>
>
> >
>
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>
> >
>
> > Hi John,
>
> >
>
> > one final question for GridDiag.
>
> >
>
> > The bins are set up to be lower bound inclusive? i.e.
>
> > n_bins = 1000;
>
> > range = [0, 1000]; // kg/m^2
>
> >
>
> > means that the bin is set up as [0,1) i.e. 0 <= x < 1?
>
> >
>
> > I'm presuming so.... and the upper bound would be [999,1000) with
no
>
> > bin for [1000,inf)? i.e. if there were a 1000 in my sample it
would
>
> > NOT be counted? Or anything bigger.
>
> >
>
> > That's what it looks like, i.e. you need to make sure that your
> > range
>
> > encompasses all possible values. Not sure this is in the
> documentation...?
>
> > Would be useful to add.
>
> >
>
> > Ta
>
> > Marion
>
> >
>
> > -----Original Message-----
>
> > From: John Halley Gotway via RT <met_help at ucar.edu<mailto:
> met_help at ucar.edu<mailto:met_help at ucar.edu>>>
>
> > Sent: 05 December 2020 06:41
>
> > To: Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk<mailto:
>
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>>>
>
> > Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
>
> >
>
> > This email was received from an external source. Always check
sender
>
> > details, links & attachments.
>
> >
>
> > Marion,
>
> >
>
> > Ugh, I should've just handed this ticket off to David in the first
place.
>
> > I did more testing and was able to get it to work. I just had to
use
>
> > separate "-data" command line options for each input data source.
>
> >
>
> > So this command fails:
>
> > grid_diag -data glosea_2019060100_001.nc
> > gpm_2019060100_2019060200.nc
>
> > -config GridDiag.glosea -out grid_diag_out.nc
>
> >
>
> > But this command runs fine:
>
> > *grid_diag -data glosea_2019060100_001.nc <
>
> > http://glosea_2019060100_001.nc> -data
gpm_2019060100_2019060200.nc
> > <
>
> > http://gpm_2019060100_2019060200.nc> -config GridDiag.glosea -out
>
> > grid_diag_out.nc <http://grid_diag_out.nc>*
>
> >
>
> > In the second one, I've specified "-data" separately for each data
> source.
>
> > Although, you'll see warning about the range of the data extending
>
> > beyond the range of the bins defined in the config file.
>
> >
>
> > Sorry to have such a rambling and confusing response!
>
> >
>
> > So to summarize, I believe the logic is:
>
> > 1. If "-data" is used once, all specified input fields are read
from
>
> > all of those data files.
>
> > 2. If "-data" is used more than once, the number of "-data"
entries
>
> > must match the number of fields defined in the config file.
>
> > If not, you get an error message.
>
> > If so, only the i-th field is read from the files in the i-th
>
> > -data source.
>
> >
>
> > John
>
> >
>
> > On Fri, Dec 4, 2020 at 11:30 PM John Halley Gotway
<johnhg at ucar.edu
> <mailto:johnhg at ucar.edu>>
>
> > wrote:
>
> >
>
> > > Marion,
>
> > >
>
> > > It occurs to me that perhaps Grid-Diag could be made more
flexible
>
> > > if we changed that error message to a debug message in this
context.
>
> > > It could
>
> > > *ATTEMPT* to read all input fields from all input files... but
> > > when
>
> > > it's unable to, have it print a log message stating that rather
> > > than
>
> > > erroring out.
>
> > >
>
> > > That would enable you to compute 2D pdf's using different
> > > variables
>
> > > from different input files.
>
> > >
>
> > > Do you think that logic is preferable?
>
> > >
>
> > > Thanks,
>
> > > John
>
> > >
>
> > > On Fri, Dec 4, 2020 at 11:27 PM The RT System itself via RT <
>
> > >
met_help at ucar.edu<mailto:met_help at ucar.edu<mailto:met_help at ucar.edu%3cmailto:met_help at ucar.edu>>>
wrote:
>
> > >
>
> > >>
>
> > >> Fri Dec 04 23:27:19 2020: Request 97766 was acted upon.
>
> > >> Transaction: Given to johnhg (John Halley Gotway) by RT_System
>
> > >> Queue: met_help
>
> > >> Subject: Need help with GridDiag
>
> > >> Owner: johnhg
>
> > >> Requestors:
marion.mittermaier at metoffice.gov.uk<mailto<mailto:marion.mittermaier at metoffice.gov.uk%3cmailto>:
>
marion.mittermaier at metoffice.gov.uk<mailto:marion.mittermaier at metoffice.gov.uk>>
>
> > >> Status: new
>
> > >> Ticket <URL:
>
> > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
>
> > >> >
>
> > >>
>
> > >>
>
> > >> This transaction appears to have no content
>
> > >>
>
> > >
>
> >
>
> >
>
> >
>
> >
>
>
>
>
------------------------------------------------
Subject: Need help with GridDiag
From: John Halley Gotway
Time: Wed Feb 17 10:34:11 2021
Marion,
Glad to hear you've made progress. I see that you're not having any
luck
providing "-data file_list1 -data file_list2".
I just tested MET version 9.1.1 to see what trouble turns up, but
surprisingly, it all went smoothly. Here's how I tested... just
listing the
same input file 3 times in my fcst_files and obs_files lists:
*echo "glosea_2019060100_d001_001.nc
<http://glosea_2019060100_d001_001.nc>
glosea_2019060100_d001_001.nc <http://glosea_2019060100_d001_001.nc>
glosea_2019060100_d001_001.nc <http://glosea_2019060100_d001_001.nc>"
>
fcst_filesecho "gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc> gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc> gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc>" >
obs_files/Volumes/d1/projects/MET/MET_development/MET-
main_v9.1/met/bin/grid_diag
\-data fcst_files -data obs_files \-config GridDiag.glosea -out
grid_diag_out.nc <http://grid_diag_out.nc> -v 3 -log run_gd.log*
I've attached the log file in case that helps. I wonder perhaps if
there
were any fixes in met-9.1.1 that affected Grid-Diag? I don't see any
clear
explanation in the release notes
<https://dtcenter.github.io/MET/latest/Users_Guide/overview.html#version-
version-release-notes-release-date>
.
If you continue to have trouble with this, please follow up with more
details with which I can help you debug.
Thanks,
John
On Wed, Feb 17, 2021 at 4:20 AM marion.mittermaier at metoffice.gov.uk
via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>
> Hi John,
>
>
>
> this all works now which is great. Thanks so much. One small
question, and
> this may be solved in the GridDiag wrapper github issue that I saw
George
> mention in the governance meeting yesterday is the cycling.
>
>
>
> At the moment off the command line I can't get it to run off two
file
> lists, i.e. it doesn't seem to like -data fc_list -data obs_list.
This only
> worked with one file list. My question would then be can the list be
a
> mixed list? i.e. the forecast and observation files interspersed in
the
> same file list? I haven't tried that. I only used the single file
list for
> processing either the forecast or the observations. Anyway, it's not
a
> blocker right now and I plan to investigate what George mentioned to
see if
> that is what I think it is. At this stage I'm writing out each file
> separately and I will then combine the joint histograms later. It
does mean
> more output files but I can cope with that! Main thing is I am
getting some
> output and I can hopefully get a progress report written before the
end of
> March 😉
>
>
>
> Regards
>
> Marion
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 15 February 2021 17:38
> To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> Subject: Re: FW: [rt.rap.ucar.edu #97766] Need help with GridDiag
>
>
>
> This email was received from an external source. Always check
sender
> details, links & attachments.
>
>
>
> Marion,
>
>
>
> Thanks for sending sample data. I am a little confused though. The
config
> file references variables named "precipitation_amount" and
>
> "Precipitation_Amount":
>
> * name = "precipitation_amount";*
>
> * name = "Precipitation_Amount"; *
>
>
>
> But the actual data you sent does not:
>
>
>
>
>
> * ncdump -h glosea_2019060100_d001_001.nc
>
> <http://glosea_2019060100_d001_001.nc> | grep float float
>
> precipitation_amount(latitude, longitude) ;*
>
>
>
> * ncdump -h gpm_2019060100_2019060200.nc
>
> <http://gpm_2019060100_2019060200.nc> | grep float float
>
> precipitation_amount(latitude, longitude) ;*
>
>
>
> I do believe the NetCDF is case-sensitive. So I think that's
important.
> But I decided to just try to get Grid-Diag working on the data you
sent. I
> did run into an error... different from the one you sent:
>
>
>
>
>
> *terminate called after throwing an instance of
> 'netCDF::exceptions::NcNameInUse' what(): NetCDF: String match to
name in
>
> use*
>
>
>
> Grid-Diag reads the first variable from the first "-data" source and
the
> second variable from the second "-data" source. And writes data to
the
> output NetCDF file. But since those variable names are the same,
there's a
> name conflict! It's trying to write the same variable names to the
output
> NetCDF file which causes the NetCDF library to error out.
>
>
>
> To get around this, I redefined the input variable names and levels
in the
> config file using:
>
> * set_attr_name = "glosea_precip";*
>
>
>
> * set_attr_level = "sfc"; set_attr_name = "gpm_precip";*
>
> * set_attr_level = "sfc";*
>
>
>
> That works fine and produces the attached output file. I also
attached my
> run command, config file, and resulting log file.
>
>
>
> Here's some improvements we could make:
>
>
>
> (1) Using set_attr_name and set_attr_level works great. But if you
look
> closely at the log messages, the updated strings are NOT used there.
Seems
> like they should be. Do you agree?
>
>
>
> (2) The actual data values don't go much over 250... but you've
defined
> bins all the way out to 1000. That'll leave you with at least 75% of
the
> bins empty. Can you think of any enhancements to make the setting of
the
> number and size of bins easier?
>
>
>
> (3) The GPM data appears to use a value of 9.96921E36 as missing
data, but
> no _FillValue or missing_value variable attribute defines that. So
MET
> processes considers all these pink values to be valid data. Doesn't
matter
> in this case because you're only using data from India, which has
all valid
> data.
>
> [image: Screen Shot 2021-02-15 at 10.31.31 AM.png] I'd recommend
using
> _FillValue or missing_value to define bad data values in the GPM
files:
>
>
>
>
> https://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-
conventions.html#missing-data
>
> And do the same for the glosea files as well.
>
>
>
> Thanks,
>
> John
>
>
>
> On Mon, Feb 15, 2021 at 4:43 AM marion.mittermaier at metoffice.gov.uk
> <mailto:marion.mittermaier at metoffice.gov.uk> via RT <
met_help at ucar.edu
> <mailto:met_help at ucar.edu>> wrote:
>
>
>
> >
>
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>
> >
>
> > Hi all,
>
> >
>
> > in the mean time I have got regrid_data_plane to work in stand-
alone
>
> > model… running the following command (and I’ve tried in both
directions):
>
> >
>
> > regrid_data_plane glosea_2019060100_d001_001.nc
>
> > gpm_2019060100_2019060200.nc grid_diag_out.nc -field 019060200.nc
>
> > grid_diag_out.nc -field ‘name="precipitation_amount";
level="(*,*)";'
>
> > <
>
> >
>
> > DEBUG 1: Reading data file: glosea_2019060100_d001_001.nc DEBUG 2:
>
> > Input grid: Projection: Lat/Lon Nx: 432 Ny: 324 lat_ll: -89.722
>
> > lon_ll: -0.417 delta_lat: 0.556 delta_lon: 0.833 DEBUG 2: Output
grid:
>
> > Projection: Lat/Lon Nx: 3600 Ny: 1800 lat_ll:
>
> > -89.950 lon_ll: 179.950 delta_lat: 0.100 delta_lon: 0.100 DEBUG 2:
>
> > Interpolation options: method = NEAREST, width = 1, shape =
SQUARE,
>
> > vld_thresh = 0.5 DEBUG 2: Range of input data
>
> > (name="precipitation_amount"; level="(*,*)";) is 0 to 262.128.
>
> > DEBUG 2: Range of regridded data (name="precipitation_amount";
>
> > level="(*,*)";) is 0 to 262.128.
>
> > DEBUG 1: Writing output file: grid_diag_out.nc
>
> >
>
> > Which looks sensible… so it’s NOT regrid_data_plane.
>
> >
>
> > Marion
>
> >
>
> > From: Mittermaier, Marion
>
> > Sent: 15 February 2021 11:02
>
> > To: 'met_help at ucar.edu'
<met_help at ucar.edu<mailto:met_help at ucar.edu>>
>
> > Subject: RE: [rt.rap.ucar.edu #97766] Need help with GridDiag
>
> >
>
> >
>
> > Hi John,
>
> >
>
> >
>
> >
>
> > it's taken me a while to get back to this... I do see the
difference
>
> > but I still don't get it to work. I can get either on their own to
>
> > work but not together.
>
> >
>
> > I have split out the forecast days into separate files.
>
> >
>
> >
>
> >
>
> > grid_diag -data
>
> > /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc
>
> > -data /scratch/fryl/GPM/gpm_2019060100_2019060200.nc -config
>
> > GridDiag.glosea -out grid_diag_out.nc -v 4
>
> >
>
> >
>
> >
>
> > This gives the following error (I think it's in
regrid_data_plane).
>
> > Not sure why it is trying to regrid both? Or maybe I’m just not
quite
>
> > following the messages….
>
> >
>
> >
>
> >
>
> > vld497:/net/home/h02/frmm/WCSSP/India > grid_diag -data
>
> > /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc -data
>
> > /scratch/fryl/GPM/gpm_2019060100_2019060200.nc
>
> >
>
> > DEBUG 1: Default Config File:
>
> >
/data/users/cfrd/MET_requirements/MET9.0/share/met/config/GridDiagConf
>
> > ig_default
>
> >
>
> > DEBUG 1: User Config File: GridDiag.glosea
>
> >
>
> > DEBUG 2: Read config files.
>
> >
>
> > DEBUG 2: Enter get_mtddf.
>
> >
>
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
>
> > Met2dDataFile object of type "FileType_NcCF".
>
> >
>
> > DEBUG 4: NcCfFile::open() -> parsing units for the time variable
>
> > "hours since 1970-01-01"
>
> >
>
> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention
time
>
> > unit string "hours since 1970-01-01" as a reference time of
>
> > 19700101_000000 and
>
> > 3600 second(s) per time step.
>
> >
>
> > DEBUG 4: NcCfFile::open() -> could not extract init time from the
>
> > "forecast_reference_time" variable.
>
> >
>
> > DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
>
> >
>
> > DEBUG 2: Process configuration.
>
> >
>
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
>
> > of type "FileType_NcCF".
>
> >
>
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
>
> > of type "FileType_NcCF".
>
> >
>
> > DEBUG 3: Use the grid defined by file
"/scratch/fryl/WCSSPIndia/GPM/
>
> > gpm_2019060100_2019060200.nc".
>
> >
>
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
>
> > Met2dDataFile object of type "FileType_NcCF".
>
> >
>
> > DEBUG 4: NcCfFile::open() -> parsing units for the time variable
>
> > "hours since 1970-01-01"
>
> >
>
> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention
time
>
> > unit string "hours since 1970-01-01" as a reference time of
>
> > 19700101_000000 and
>
> > 3600 second(s) per time step.
>
> >
>
> > DEBUG 4: NcCfFile::open() -> could not extract init time from the
>
> > "forecast_reference_time" variable.
>
> >
>
> > DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
>
> >
>
> > DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 360
lat_ll:
>
> > 4.550 lon_ll: -64.550 delta_lat: 0.100 delta_lon: 0.100
>
> >
>
> > DEBUG 2: Processing masking regions.
>
> >
>
> > DEBUG 3: Processing poly mask: India.poly
>
> >
>
> > DEBUG 4: parse_poly_mask() -> parsing poly mask "India.poly"
>
> >
>
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
>
> > Met2dDataFile object of type "FileType_None".
>
> >
>
> > DEBUG 4: parse_poly_mask() -> parsing poly mask file "India.poly"
>
> >
>
> > DEBUG 2: Initializing precipitation_amount(*,*) histogram
>
> >
>
> > DEBUG 2: Initializing Precipitation_Amount(*,*) histogram
>
> >
>
> > DEBUG 2: Initializing
>
> > precipitation_amount(*,*)_Precipitation_Amount(*,*)
>
> > joint histogram
>
> >
>
> > DEBUG 1: grid_diag_out.nc
>
> >
>
> > DEBUG 1: Length of configuration "data.field" = 2
>
> >
>
> > DEBUG 1: Length of data file list = 1
>
> >
>
> > DEBUG 1: Series defined by the data file list of length 1.
>
> >
>
> > DEBUG 2: 1000
>
> >
>
> > DEBUG 2: Processing series entry 1: precipitation_amount:
>
> > precipitation_amount(*,*)
>
> >
>
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
>
> > Met2dDataFile object of type "FileType_NcCF".
>
> >
>
> > DEBUG 4: NcCfFile::open() -> parsing units for the time variable
>
> > "hours since 1970-01-01"
>
> >
>
> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention
time
>
> > unit string "hours since 1970-01-01" as a reference time of
>
> > 19700101_000000 and
>
> > 3600 second(s) per time step.
>
> >
>
> > DEBUG 4: NcCfFile::open() -> could not extract init time from the
>
> > "forecast_reference_time" variable.
>
> >
>
> > DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
>
> >
>
> > DEBUG 4: NcCfFile::getData() -> setting the unset init time to the
>
> > valid time of 20190601_115959.
>
> >
>
> > DEBUG 4:
>
> >
>
> > DEBUG 4: Data plane information:
>
> >
>
> > DEBUG 4: plane min: 0
>
> >
>
> > DEBUG 4: plane max: 9.96921e+36
>
> >
>
> > DEBUG 4: valid time: 20190601_115959
>
> >
>
> > DEBUG 4: lead time: 000000
>
> >
>
> > DEBUG 4: init time: 20190601_115959
>
> >
>
> > DEBUG 4: accum time: 000000
>
> >
>
> > DEBUG 1: Regridding field precipitation_amount(*,*) to the
>
> > verification grid.
>
> >
>
> > DEBUG 2: Processing series entry 1: Precipitation_Amount:
>
> > Precipitation_Amount(*,*)
>
> >
>
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
>
> > Met2dDataFile object of type "FileType_NcCF".
>
> >
>
> > DEBUG 4: NcCfFile::open() -> parsing units for the time variable
>
> > "hours since 1970-01-01"
>
> >
>
> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention
time
>
> > unit string "hours since 1970-01-01" as a reference time of
>
> > 19700101_000000 and
>
> > 3600 second(s) per time step.
>
> >
>
> > DEBUG 4: NcCfFile::open() -> could not extract init time from the
>
> > "forecast_reference_time" variable.
>
> >
>
> > DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
>
> >
>
> > DEBUG 1: Regridding field precipitation_amount(*,*) to the
>
> > verification grid.
>
> >
>
> > ERROR :
>
> >
>
> > ERROR : DataPlane::two_to_one() -> range check error: (Nx, Ny) =
(0,
>
> > 0), (x, y) = (0, 0)
>
> >
>
> > ERROR :
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> >
>
> > -----Original Message-----
>
> > From: John Halley Gotway via RT <met_help at ucar.edu<mailto:
>
> > met_help at ucar.edu<mailto:met_help at ucar.edu>>>
>
> > Sent: 08 December 2020 20:54
>
> > To: Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk<mailto:
>
> > marion.mittermaier at metoffice.gov.uk<mailto:
> marion.mittermaier at metoffice.gov.uk>>>
>
> > Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
>
> >
>
> >
>
> >
>
> > This email was received from an external source. Always check
sender
>
> > details, links & attachments.
>
> >
>
> >
>
> >
>
> > Marion,
>
> >
>
> >
>
> >
>
> > I'm still not convinced. Please take a very careful look at the
>
> > command you're executing. I see all the log messages you sent, but
not
>
> > the actual command you ran. At that matters a lot!
>
> >
>
> >
>
> >
>
> > This command fails on my machine with the SAME ERROR you're
getting:
>
> >
>
> > * grid_diag -data glosea_2019060100_001.nc
>
> >
>
> > <http://glosea_2019060100_001.nc/> gpm_2019060100_2019060200.nc <
>
> > http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea -out
>
> > grid_diag_out.nc <http://grid_diag_out.nc/>*
>
> >
>
> >
>
> >
>
> > ERROR : NcCfFile::getData(NcVar *, const LongArray &,
DataPlane &)
>
> > const -> needed 3 arguments for variable precipitation_amount, got
2
>
> >
>
> >
>
> >
>
> > But this command runs WITHOUT ERROR:
>
> >
>
> > * grid_diag -data glosea_2019060100_001.nc
>
> >
>
> > <http://glosea_2019060100_001.nc/> -data
gpm_2019060100_2019060200.nc
>
> > < http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea
-out
>
> > grid_diag_out.nc <http://grid_diag_out.nc/>*
>
> >
>
> >
>
> >
>
> > Do you see the difference?
>
> >
>
> > In the first, I used "-data" once and listed 2 different file
names.
>
> >
>
> > In the second, I used "-data" twice, once for each of the input
files.
>
> >
>
> >
>
> >
>
> > Please try doing exactly the same thing.
>
> >
>
> >
>
> >
>
> > Does that enable it to run without error? Note that you'll still
get
>
> > warnings about the actual data values falling outside the defined
>
> > range of the bins.
>
> >
>
> >
>
> >
>
> > As for the bin endpoints, I think this is the relevant line of
code:
>
> >
>
> >
>
> >
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc
>
> > 5981bb8/met/src/libcode/vx_series_data/series_pdf.cc#L72
>
> >
>
> >
>
> >
>
> > int k = floor((value - min) / delta);
>
> >
>
> >
>
> >
>
> > This places the current "value" into the k-th bin where "min" is
the
>
> > minimum PDF value and "delta" is the bin width.
>
> >
>
> > I believe this is consistent with bins being closed on the left
side
>
> > and open on the right:
>
> >
>
> > [0,1) i.e. 0 <= x < 1
>
> >
>
> >
>
> >
>
> > John
>
> >
>
> >
>
> >
>
> > On Tue, Dec 8, 2020 at 8:30 AM marion.mittermaier at metoffice.gov.uk
> <mailto<mailto:marion.mittermaier at metoffice.gov.uk%3cmailto>:
>
> > marion.mittermaier at metoffice.gov.uk<mailto:
> marion.mittermaier at metoffice.gov.uk>> via RT < met_help at ucar.edu
> <mailto<mailto:met_help at ucar.edu%3cmailto>:
>
> > met_help at ucar.edu<mailto:met_help at ucar.edu>>> wrote:
>
> >
>
> >
>
> >
>
> > >
>
> >
>
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>
> >
>
> > >
>
> >
>
> > > Hi John,
>
> >
>
> > >
>
> >
>
> > > one final question for GridDiag.
>
> >
>
> > >
>
> >
>
> > > The bins are set up to be lower bound inclusive? i.e.
>
> >
>
> > > n_bins = 1000;
>
> >
>
> > > range = [0, 1000]; // kg/m^2
>
> >
>
> > >
>
> >
>
> > > means that the bin is set up as [0,1) i.e. 0 <= x < 1?
>
> >
>
> > >
>
> >
>
> > > I'm presuming so.... and the upper bound would be [999,1000)
with no
>
> >
>
> > > bin for [1000,inf)? i.e. if there were a 1000 in my sample it
would
>
> >
>
> > > NOT be counted? Or anything bigger.
>
> >
>
> > >
>
> >
>
> > > That's what it looks like, i.e. you need to make sure that your
>
> > > range
>
> >
>
> > > encompasses all possible values. Not sure this is in the
>
> > documentation...?
>
> >
>
> > > Would be useful to add.
>
> >
>
> > >
>
> >
>
> > > Ta
>
> >
>
> > > Marion
>
> >
>
> > >
>
> >
>
> > > -----Original Message-----
>
> >
>
> > > From: John Halley Gotway via RT <met_help at ucar.edu<mailto:
>
> > met_help at ucar.edu<mailto:met_help at ucar.edu>>>
>
> >
>
> > > Sent: 05 December 2020 06:41
>
> >
>
> > > To: Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk<mailto:
>
> > marion.mittermaier at metoffice.gov.uk<mailto:
> marion.mittermaier at metoffice.gov.uk>>>
>
> >
>
> > > Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
>
> >
>
> > >
>
> >
>
> > > This email was received from an external source. Always check
sender
>
> >
>
> > > details, links & attachments.
>
> >
>
> > >
>
> >
>
> > > Marion,
>
> >
>
> > >
>
> >
>
> > > Ugh, I should've just handed this ticket off to David in the
first
> place.
>
> >
>
> > > I did more testing and was able to get it to work. I just had to
use
>
> >
>
> > > separate "-data" command line options for each input data
source.
>
> >
>
> > >
>
> >
>
> > > So this command fails:
>
> >
>
> > > grid_diag -data glosea_2019060100_001.nc
>
> > > gpm_2019060100_2019060200.nc
>
> >
>
> > > -config GridDiag.glosea -out grid_diag_out.nc
>
> >
>
> > >
>
> >
>
> > > But this command runs fine:
>
> >
>
> > > *grid_diag -data glosea_2019060100_001.nc <
>
> >
>
> > > http://glosea_2019060100_001.nc> -data
gpm_2019060100_2019060200.nc
>
> > > <
>
> >
>
> > > http://gpm_2019060100_2019060200.nc> -config GridDiag.glosea
-out
>
> >
>
> > > grid_diag_out.nc <http://grid_diag_out.nc>*
>
> >
>
> > >
>
> >
>
> > > In the second one, I've specified "-data" separately for each
data
>
> > source.
>
> >
>
> > > Although, you'll see warning about the range of the data
extending
>
> >
>
> > > beyond the range of the bins defined in the config file.
>
> >
>
> > >
>
> >
>
> > > Sorry to have such a rambling and confusing response!
>
> >
>
> > >
>
> >
>
> > > So to summarize, I believe the logic is:
>
> >
>
> > > 1. If "-data" is used once, all specified input fields are read
from
>
> >
>
> > > all of those data files.
>
> >
>
> > > 2. If "-data" is used more than once, the number of "-data"
entries
>
> >
>
> > > must match the number of fields defined in the config file.
>
> >
>
> > > If not, you get an error message.
>
> >
>
> > > If so, only the i-th field is read from the files in the i-
th
>
> >
>
> > > -data source.
>
> >
>
> > >
>
> >
>
> > > John
>
> >
>
> > >
>
> >
>
> > > On Fri, Dec 4, 2020 at 11:30 PM John Halley Gotway
<johnhg at ucar.edu
>
> > <mailto:johnhg at ucar.edu>>
>
> >
>
> > > wrote:
>
> >
>
> > >
>
> >
>
> > > > Marion,
>
> >
>
> > > >
>
> >
>
> > > > It occurs to me that perhaps Grid-Diag could be made more
flexible
>
> >
>
> > > > if we changed that error message to a debug message in this
context.
>
> >
>
> > > > It could
>
> >
>
> > > > *ATTEMPT* to read all input fields from all input files... but
>
> > > > when
>
> >
>
> > > > it's unable to, have it print a log message stating that
rather
>
> > > > than
>
> >
>
> > > > erroring out.
>
> >
>
> > > >
>
> >
>
> > > > That would enable you to compute 2D pdf's using different
>
> > > > variables
>
> >
>
> > > > from different input files.
>
> >
>
> > > >
>
> >
>
> > > > Do you think that logic is preferable?
>
> >
>
> > > >
>
> >
>
> > > > Thanks,
>
> >
>
> > > > John
>
> >
>
> > > >
>
> >
>
> > > > On Fri, Dec 4, 2020 at 11:27 PM The RT System itself via RT <
>
> >
>
> > > >
met_help at ucar.edu<mailto:met_help at ucar.edu<mailto:met_help at ucar.edu%
> 3cmailto:met_help at ucar.edu>>> wrote:
>
> >
>
> > > >
>
> >
>
> > > >>
>
> >
>
> > > >> Fri Dec 04 23:27:19 2020: Request 97766 was acted upon.
>
> >
>
> > > >> Transaction: Given to johnhg (John Halley Gotway) by
RT_System
>
> >
>
> > > >> Queue: met_help
>
> >
>
> > > >> Subject: Need help with GridDiag
>
> >
>
> > > >> Owner: johnhg
>
> >
>
> > > >> Requestors:
marion.mittermaier at metoffice.gov.uk<mailto<mailto:
> marion.mittermaier at metoffice.gov.uk%3cmailto>:
>
> > marion.mittermaier at metoffice.gov.uk<mailto:
> marion.mittermaier at metoffice.gov.uk>>
>
> >
>
> > > >> Status: new
>
> >
>
> > > >> Ticket <URL:
>
> >
>
> > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
>
> >
>
> > > >> >
>
> >
>
> > > >>
>
> >
>
> > > >>
>
> >
>
> > > >> This transaction appears to have no content
>
> >
>
> > > >>
>
> >
>
> > > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> > >
>
> >
>
> >
>
> >
>
> >
>
>
>
>
------------------------------------------------
Subject: RE: FW: [rt.rap.ucar.edu #97766] Need help with GridDiag
From: marion.mittermaier at metoffice.gov.uk
Time: Thu Mar 04 02:29:14 2021
Hi John,
I thought I had cracked it but something is still not quite right...
the joint distribution does appear when using the dual -data commands
with the changed attribute names, and it looks plausible BUT but the
marginal distributions are wrong.
If I run the GridDiag code with both data sets then the marginal
distributions are the same, which isn't right.
running together
glosea 171504, 18373, 11192, 7943, 6551, 4922, 4065, 3680,
3344, 2750, 2583, 1719, 2432, 1602, 1632, 1173, 1358, 899, 774,
685, 720,
625, 782, 650, 569, 502, 377, 385, 236, 286, 244, 237, 123, 116,
156,
112, 128, 100, 151, 88, 122, 173, 109, 63, 62, 53, 88, 40, 49, 46,
84,
29, 32, 48, 41, 29, 30, 32, 36, 32, 29, 26, 24, 18, 20, 22, 20,
24, 21,
16, 15, 21, 12, 20, 16, 15, 17, 16, 7, 13, 11, 15, 8, 12, 10, 11,
11, 3,
8, 9, 4, 6, 13, 7, 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4, 5, 50, 5,
7, 6, 5,
2, 2, 53, 4, 5, 3, 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1, 4, 1, 0, 1,
1, 2,
4, 3, 2, 3, 1, 2, 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1, 3, 2, 1,
2, 0,
0, 3, 2, 1, 2, 2, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0, 4, 0, 0,
1, 1,
0, 2, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0,
0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
gpm 171504, 18373, 11192, 7943, 6551, 4922, 4065, 3680,
3344, 2750, 2583, 1719, 2432, 1602, 1632, 1173, 1358, 899, 774,
685, 720,
625, 782, 650, 569, 502, 377, 385, 236, 286, 244, 237, 123, 116,
156,
112, 128, 100, 151, 88, 122, 173, 109, 63, 62, 53, 88, 40, 49, 46,
84,
29, 32, 48, 41, 29, 30, 32, 36, 32, 29, 26, 24, 18, 20, 22, 20,
24, 21,
16, 15, 21, 12, 20, 16, 15, 17, 16, 7, 13, 11, 15, 8, 12, 10, 11,
11, 3,
8, 9, 4, 6, 13, 7, 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4, 5, 50, 5,
7, 6, 5,
2, 2, 53, 4, 5, 3, 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1, 4, 1, 0, 1,
1, 2,
4, 3, 2, 3, 1, 2, 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1, 3, 2, 1,
2, 0,
0, 3, 2, 1, 2, 2, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0, 4, 0, 0,
1, 1,
0, 2, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0,
0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
But when I run them separately I get the following for gpm only (same
input files)
93193, 7954, 4758, 3206, 2364, 1889, 1554, 1287, 1081,
977, 824, 685, 673, 550, 520, 470, 490, 409, 391, 344, 340, 311,
278,
295, 246, 224, 195, 198, 188, 184, 175, 143, 123, 116, 116, 112,
88, 100,
111, 88, 82, 74, 69, 63, 62, 53, 48, 40, 49, 46, 36, 29, 32, 48,
41, 29,
30, 32, 36, 32, 29, 26, 24, 18, 20, 22, 20, 24, 21, 16, 15, 21,
12, 20,
16, 15, 17, 16, 7, 13, 11, 15, 8, 12, 10, 11, 11, 3, 8, 9, 4, 6,
13, 7,
6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4, 5, 2, 5, 7, 6, 5, 2, 2, 5, 4,
5, 3,
2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1, 4, 1, 0, 1, 1, 2, 4, 3, 2, 3,
1, 2,
0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1, 3, 2, 1, 2, 0, 0, 3, 2, 1,
2, 2,
0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0, 4, 0, 0, 1, 1, 0, 2, 0, 1,
0, 1,
1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
and for glosea only
78311, 10419, 6434, 4737, 4187, 3033, 2511, 2393,
2263, 1773, 1759, 1034, 1759, 1052, 1112, 703, 868, 490, 383, 341,
380,
314, 504, 355, 323, 278, 182, 187, 48, 102, 69, 94, 0, 0, 40, 0,
40, 0,
40, 0, 40, 99, 40, 0, 0, 0, 40, 0, 0, 0, 48, 0, 0, 0, 0, 0, 0, 0,
0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 48,
0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
Looking at the first bin 93193+78311 = 171504, bin 2 is
7953+10419=18373 etc. which are the values in bins 1 and 2 for both
the marginal distributions when I input both files together. I am
wondering whether the attribute changes are not pulled through to this
somehow? This is for the sample data you've got and using MET9.1. My
workaround for now is that I will have to run the code 3 times, once
for the joint distribution, which looks ok, and twice more running it
with just the single data stream to get the marginals for the two
datasets.
Regards
Marion
-----Original Message-----
From: John Halley Gotway via RT <met_help at ucar.edu>
Sent: 15 February 2021 17:38
To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
Subject: Re: FW: [rt.rap.ucar.edu #97766] Need help with GridDiag
This email was received from an external source. Always check sender
details, links & attachments.
Marion,
Thanks for sending sample data. I am a little confused though. The
config file references variables named "precipitation_amount" and
"Precipitation_Amount":
* name = "precipitation_amount";*
* name = "Precipitation_Amount"; *
But the actual data you sent does not:
* ncdump -h glosea_2019060100_d001_001.nc
<http://glosea_2019060100_d001_001.nc> | grep float float
precipitation_amount(latitude, longitude) ;*
* ncdump -h gpm_2019060100_2019060200.nc
<http://gpm_2019060100_2019060200.nc> | grep float float
precipitation_amount(latitude, longitude) ;*
I do believe the NetCDF is case-sensitive. So I think that's
important. But I decided to just try to get Grid-Diag working on the
data you sent. I did run into an error... different from the one you
sent:
*terminate called after throwing an instance of
'netCDF::exceptions::NcNameInUse' what(): NetCDF: String match to
name in
use*
Grid-Diag reads the first variable from the first "-data" source and
the second variable from the second "-data" source. And writes data to
the output NetCDF file. But since those variable names are the same,
there's a name conflict! It's trying to write the same variable names
to the output NetCDF file which causes the NetCDF library to error
out.
To get around this, I redefined the input variable names and levels in
the config file using:
* set_attr_name = "glosea_precip";*
* set_attr_level = "sfc"; set_attr_name = "gpm_precip";*
* set_attr_level = "sfc";*
That works fine and produces the attached output file. I also attached
my run command, config file, and resulting log file.
Here's some improvements we could make:
(1) Using set_attr_name and set_attr_level works great. But if you
look closely at the log messages, the updated strings are NOT used
there. Seems like they should be. Do you agree?
(2) The actual data values don't go much over 250... but you've
defined bins all the way out to 1000. That'll leave you with at least
75% of the bins empty. Can you think of any enhancements to make the
setting of the number and size of bins easier?
(3) The GPM data appears to use a value of 9.96921E36 as missing data,
but no _FillValue or missing_value variable attribute defines that. So
MET processes considers all these pink values to be valid data.
Doesn't matter in this case because you're only using data from India,
which has all valid data.
[image: Screen Shot 2021-02-15 at 10.31.31 AM.png] I'd recommend using
_FillValue or missing_value to define bad data values in the GPM
files:
https://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-
conventions.html#missing-data
And do the same for the glosea files as well.
Thanks,
John
On Mon, Feb 15, 2021 at 4:43 AM marion.mittermaier at metoffice.gov.uk
via RT < met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>
> Hi all,
>
> in the mean time I have got regrid_data_plane to work in stand-alone
> model… running the following command (and I’ve tried in both
directions):
>
> regrid_data_plane glosea_2019060100_d001_001.nc
> gpm_2019060100_2019060200.nc grid_diag_out.nc -field 019060200.nc
> grid_diag_out.nc -field ‘name="precipitation_amount";
level="(*,*)";'
> <
>
> DEBUG 1: Reading data file: glosea_2019060100_d001_001.nc DEBUG 2:
> Input grid: Projection: Lat/Lon Nx: 432 Ny: 324 lat_ll: -89.722
> lon_ll: -0.417 delta_lat: 0.556 delta_lon: 0.833 DEBUG 2: Output
grid:
> Projection: Lat/Lon Nx: 3600 Ny: 1800 lat_ll:
> -89.950 lon_ll: 179.950 delta_lat: 0.100 delta_lon: 0.100 DEBUG 2:
> Interpolation options: method = NEAREST, width = 1, shape = SQUARE,
> vld_thresh = 0.5 DEBUG 2: Range of input data
> (name="precipitation_amount"; level="(*,*)";) is 0 to 262.128.
> DEBUG 2: Range of regridded data (name="precipitation_amount";
> level="(*,*)";) is 0 to 262.128.
> DEBUG 1: Writing output file: grid_diag_out.nc
>
> Which looks sensible… so it’s NOT regrid_data_plane.
>
> Marion
>
> From: Mittermaier, Marion
> Sent: 15 February 2021 11:02
> To: 'met_help at ucar.edu' <met_help at ucar.edu>
> Subject: RE: [rt.rap.ucar.edu #97766] Need help with GridDiag
>
>
> Hi John,
>
>
>
> it's taken me a while to get back to this... I do see the difference
> but I still don't get it to work. I can get either on their own to
> work but not together.
>
> I have split out the forecast days into separate files.
>
>
>
> grid_diag -data
> /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc
> -data /scratch/fryl/GPM/gpm_2019060100_2019060200.nc -config
> GridDiag.glosea -out grid_diag_out.nc -v 4
>
>
>
> This gives the following error (I think it's in regrid_data_plane).
> Not sure why it is trying to regrid both? Or maybe I’m just not
quite
> following the messages….
>
>
>
> vld497:/net/home/h02/frmm/WCSSP/India > grid_diag -data
> /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc -data
> /scratch/fryl/GPM/gpm_2019060100_2019060200.nc
>
> DEBUG 1: Default Config File:
>
/data/users/cfrd/MET_requirements/MET9.0/share/met/config/GridDiagConf
> ig_default
>
> DEBUG 1: User Config File: GridDiag.glosea
>
> DEBUG 2: Read config files.
>
> DEBUG 2: Enter get_mtddf.
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
>
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> "hours since 1970-01-01"
>
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01" as a reference time of
> 19700101_000000 and
> 3600 second(s) per time step.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
>
> DEBUG 2: Process configuration.
>
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of type "FileType_NcCF".
>
> DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> of type "FileType_NcCF".
>
> DEBUG 3: Use the grid defined by file "/scratch/fryl/WCSSPIndia/GPM/
> gpm_2019060100_2019060200.nc".
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
>
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> "hours since 1970-01-01"
>
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01" as a reference time of
> 19700101_000000 and
> 3600 second(s) per time step.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
>
> DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 360
lat_ll:
> 4.550 lon_ll: -64.550 delta_lat: 0.100 delta_lon: 0.100
>
> DEBUG 2: Processing masking regions.
>
> DEBUG 3: Processing poly mask: India.poly
>
> DEBUG 4: parse_poly_mask() -> parsing poly mask "India.poly"
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_None".
>
> DEBUG 4: parse_poly_mask() -> parsing poly mask file "India.poly"
>
> DEBUG 2: Initializing precipitation_amount(*,*) histogram
>
> DEBUG 2: Initializing Precipitation_Amount(*,*) histogram
>
> DEBUG 2: Initializing
> precipitation_amount(*,*)_Precipitation_Amount(*,*)
> joint histogram
>
> DEBUG 1: grid_diag_out.nc
>
> DEBUG 1: Length of configuration "data.field" = 2
>
> DEBUG 1: Length of data file list = 1
>
> DEBUG 1: Series defined by the data file list of length 1.
>
> DEBUG 2: 1000
>
> DEBUG 2: Processing series entry 1: precipitation_amount:
> precipitation_amount(*,*)
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
>
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> "hours since 1970-01-01"
>
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01" as a reference time of
> 19700101_000000 and
> 3600 second(s) per time step.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
>
> DEBUG 4: NcCfFile::getData() -> setting the unset init time to the
> valid time of 20190601_115959.
>
> DEBUG 4:
>
> DEBUG 4: Data plane information:
>
> DEBUG 4: plane min: 0
>
> DEBUG 4: plane max: 9.96921e+36
>
> DEBUG 4: valid time: 20190601_115959
>
> DEBUG 4: lead time: 000000
>
> DEBUG 4: init time: 20190601_115959
>
> DEBUG 4: accum time: 000000
>
> DEBUG 1: Regridding field precipitation_amount(*,*) to the
> verification grid.
>
> DEBUG 2: Processing series entry 1: Precipitation_Amount:
> Precipitation_Amount(*,*)
>
> DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new
> Met2dDataFile object of type "FileType_NcCF".
>
> DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> "hours since 1970-01-01"
>
> DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time
> unit string "hours since 1970-01-01" as a reference time of
> 19700101_000000 and
> 3600 second(s) per time step.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from the
> "forecast_reference_time" variable.
>
> DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
>
> DEBUG 1: Regridding field precipitation_amount(*,*) to the
> verification grid.
>
> ERROR :
>
> ERROR : DataPlane::two_to_one() -> range check error: (Nx, Ny) =
(0,
> 0), (x, y) = (0, 0)
>
> ERROR :
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu<mailto:
> met_help at ucar.edu>>
> Sent: 08 December 2020 20:54
> To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk<mailto:
> marion.mittermaier at metoffice.gov.uk>>
> Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
>
>
>
> This email was received from an external source. Always check
sender
> details, links & attachments.
>
>
>
> Marion,
>
>
>
> I'm still not convinced. Please take a very careful look at the
> command you're executing. I see all the log messages you sent, but
not
> the actual command you ran. At that matters a lot!
>
>
>
> This command fails on my machine with the SAME ERROR you're getting:
>
> * grid_diag -data glosea_2019060100_001.nc
>
> <http://glosea_2019060100_001.nc/> gpm_2019060100_2019060200.nc <
> http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea -out
> grid_diag_out.nc <http://grid_diag_out.nc/>*
>
>
>
> ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane
&)
> const -> needed 3 arguments for variable precipitation_amount, got 2
>
>
>
> But this command runs WITHOUT ERROR:
>
> * grid_diag -data glosea_2019060100_001.nc
>
> <http://glosea_2019060100_001.nc/> -data
gpm_2019060100_2019060200.nc
> < http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea -out
> grid_diag_out.nc <http://grid_diag_out.nc/>*
>
>
>
> Do you see the difference?
>
> In the first, I used "-data" once and listed 2 different file names.
>
> In the second, I used "-data" twice, once for each of the input
files.
>
>
>
> Please try doing exactly the same thing.
>
>
>
> Does that enable it to run without error? Note that you'll still get
> warnings about the actual data values falling outside the defined
> range of the bins.
>
>
>
> As for the bin endpoints, I think this is the relevant line of code:
>
>
>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc
> 5981bb8/met/src/libcode/vx_series_data/series_pdf.cc#L72
>
>
>
> int k = floor((value - min) / delta);
>
>
>
> This places the current "value" into the k-th bin where "min" is the
> minimum PDF value and "delta" is the bin width.
>
> I believe this is consistent with bins being closed on the left side
> and open on the right:
>
> [0,1) i.e. 0 <= x < 1
>
>
>
> John
>
>
>
> On Tue, Dec 8, 2020 at 8:30 AM
marion.mittermaier at metoffice.gov.uk<mailto:
> marion.mittermaier at metoffice.gov.uk> via RT <
met_help at ucar.edu<mailto:
> met_help at ucar.edu>> wrote:
>
>
>
> >
>
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>
> >
>
> > Hi John,
>
> >
>
> > one final question for GridDiag.
>
> >
>
> > The bins are set up to be lower bound inclusive? i.e.
>
> > n_bins = 1000;
>
> > range = [0, 1000]; // kg/m^2
>
> >
>
> > means that the bin is set up as [0,1) i.e. 0 <= x < 1?
>
> >
>
> > I'm presuming so.... and the upper bound would be [999,1000) with
no
>
> > bin for [1000,inf)? i.e. if there were a 1000 in my sample it
would
>
> > NOT be counted? Or anything bigger.
>
> >
>
> > That's what it looks like, i.e. you need to make sure that your
> > range
>
> > encompasses all possible values. Not sure this is in the
> documentation...?
>
> > Would be useful to add.
>
> >
>
> > Ta
>
> > Marion
>
> >
>
> > -----Original Message-----
>
> > From: John Halley Gotway via RT <met_help at ucar.edu<mailto:
> met_help at ucar.edu>>
>
> > Sent: 05 December 2020 06:41
>
> > To: Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk<mailto:
> marion.mittermaier at metoffice.gov.uk>>
>
> > Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
>
> >
>
> > This email was received from an external source. Always check
sender
>
> > details, links & attachments.
>
> >
>
> > Marion,
>
> >
>
> > Ugh, I should've just handed this ticket off to David in the first
place.
>
> > I did more testing and was able to get it to work. I just had to
use
>
> > separate "-data" command line options for each input data source.
>
> >
>
> > So this command fails:
>
> > grid_diag -data glosea_2019060100_001.nc
> > gpm_2019060100_2019060200.nc
>
> > -config GridDiag.glosea -out grid_diag_out.nc
>
> >
>
> > But this command runs fine:
>
> > *grid_diag -data glosea_2019060100_001.nc <
>
> > http://glosea_2019060100_001.nc> -data
gpm_2019060100_2019060200.nc
> > <
>
> > http://gpm_2019060100_2019060200.nc> -config GridDiag.glosea -out
>
> > grid_diag_out.nc <http://grid_diag_out.nc>*
>
> >
>
> > In the second one, I've specified "-data" separately for each data
> source.
>
> > Although, you'll see warning about the range of the data extending
>
> > beyond the range of the bins defined in the config file.
>
> >
>
> > Sorry to have such a rambling and confusing response!
>
> >
>
> > So to summarize, I believe the logic is:
>
> > 1. If "-data" is used once, all specified input fields are read
from
>
> > all of those data files.
>
> > 2. If "-data" is used more than once, the number of "-data"
entries
>
> > must match the number of fields defined in the config file.
>
> > If not, you get an error message.
>
> > If so, only the i-th field is read from the files in the i-th
>
> > -data source.
>
> >
>
> > John
>
> >
>
> > On Fri, Dec 4, 2020 at 11:30 PM John Halley Gotway
<johnhg at ucar.edu
> <mailto:johnhg at ucar.edu>>
>
> > wrote:
>
> >
>
> > > Marion,
>
> > >
>
> > > It occurs to me that perhaps Grid-Diag could be made more
flexible
>
> > > if we changed that error message to a debug message in this
context.
>
> > > It could
>
> > > *ATTEMPT* to read all input fields from all input files... but
> > > when
>
> > > it's unable to, have it print a log message stating that rather
> > > than
>
> > > erroring out.
>
> > >
>
> > > That would enable you to compute 2D pdf's using different
> > > variables
>
> > > from different input files.
>
> > >
>
> > > Do you think that logic is preferable?
>
> > >
>
> > > Thanks,
>
> > > John
>
> > >
>
> > > On Fri, Dec 4, 2020 at 11:27 PM The RT System itself via RT <
>
> > > met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
>
> > >
>
> > >>
>
> > >> Fri Dec 04 23:27:19 2020: Request 97766 was acted upon.
>
> > >> Transaction: Given to johnhg (John Halley Gotway) by RT_System
>
> > >> Queue: met_help
>
> > >> Subject: Need help with GridDiag
>
> > >> Owner: johnhg
>
> > >> Requestors: marion.mittermaier at metoffice.gov.uk<mailto:
> marion.mittermaier at metoffice.gov.uk>
>
> > >> Status: new
>
> > >> Ticket <URL:
>
> > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
>
> > >> >
>
> > >>
>
> > >>
>
> > >> This transaction appears to have no content
>
> > >>
>
> > >
>
> >
>
> >
>
> >
>
> >
>
>
>
>
------------------------------------------------
Subject: Need help with GridDiag
From: John Halley Gotway
Time: Thu Mar 04 09:30:34 2021
Marion,
You're right. I looked closely at the numbers that grid_diag is
producing
and this is pretty obviously a bug. Thanks for finding this and
letting us
know!
I'll coordinate with David Fillmore, the engineer who originally
worked on
this to get this fixed for the met-10.0.0 release.
John
On Thu, Mar 4, 2021 at 2:29 AM marion.mittermaier at metoffice.gov.uk via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>
> Hi John,
>
> I thought I had cracked it but something is still not quite right...
the
> joint distribution does appear when using the dual -data commands
with the
> changed attribute names, and it looks plausible BUT but the marginal
> distributions are wrong.
>
> If I run the GridDiag code with both data sets then the marginal
> distributions are the same, which isn't right.
> running together
> glosea 171504, 18373, 11192, 7943, 6551, 4922, 4065, 3680,
> 3344, 2750, 2583, 1719, 2432, 1602, 1632, 1173, 1358, 899, 774,
685,
> 720,
> 625, 782, 650, 569, 502, 377, 385, 236, 286, 244, 237, 123, 116,
156,
> 112, 128, 100, 151, 88, 122, 173, 109, 63, 62, 53, 88, 40, 49,
46, 84,
> 29, 32, 48, 41, 29, 30, 32, 36, 32, 29, 26, 24, 18, 20, 22, 20,
24,
> 21,
> 16, 15, 21, 12, 20, 16, 15, 17, 16, 7, 13, 11, 15, 8, 12, 10,
11, 11,
> 3,
> 8, 9, 4, 6, 13, 7, 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4, 5, 50, 5,
7, 6,
> 5,
> 2, 2, 53, 4, 5, 3, 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1, 4, 1, 0,
1, 1,
> 2,
> 4, 3, 2, 3, 1, 2, 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1, 3, 2,
1, 2,
> 0,
> 0, 3, 2, 1, 2, 2, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0, 4, 0,
0, 1,
> 1,
> 0, 2, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0,
0, 0,
> 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
> 0,
> gpm 171504, 18373, 11192, 7943, 6551, 4922, 4065, 3680,
> 3344, 2750, 2583, 1719, 2432, 1602, 1632, 1173, 1358, 899, 774,
685,
> 720,
> 625, 782, 650, 569, 502, 377, 385, 236, 286, 244, 237, 123, 116,
156,
> 112, 128, 100, 151, 88, 122, 173, 109, 63, 62, 53, 88, 40, 49,
46, 84,
> 29, 32, 48, 41, 29, 30, 32, 36, 32, 29, 26, 24, 18, 20, 22, 20,
24,
> 21,
> 16, 15, 21, 12, 20, 16, 15, 17, 16, 7, 13, 11, 15, 8, 12, 10,
11, 11,
> 3,
> 8, 9, 4, 6, 13, 7, 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4, 5, 50, 5,
7, 6,
> 5,
> 2, 2, 53, 4, 5, 3, 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1, 4, 1, 0,
1, 1,
> 2,
> 4, 3, 2, 3, 1, 2, 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1, 3, 2,
1, 2,
> 0,
> 0, 3, 2, 1, 2, 2, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0, 4, 0,
0, 1,
> 1,
> 0, 2, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0,
0, 0,
> 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0,
>
> But when I run them separately I get the following for gpm only
(same
> input files)
> 93193, 7954, 4758, 3206, 2364, 1889, 1554, 1287, 1081,
> 977, 824, 685, 673, 550, 520, 470, 490, 409, 391, 344, 340, 311,
278,
> 295, 246, 224, 195, 198, 188, 184, 175, 143, 123, 116, 116, 112,
88,
> 100,
> 111, 88, 82, 74, 69, 63, 62, 53, 48, 40, 49, 46, 36, 29, 32, 48,
41,
> 29,
> 30, 32, 36, 32, 29, 26, 24, 18, 20, 22, 20, 24, 21, 16, 15, 21,
12,
> 20,
> 16, 15, 17, 16, 7, 13, 11, 15, 8, 12, 10, 11, 11, 3, 8, 9, 4, 6,
13,
> 7,
> 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4, 5, 2, 5, 7, 6, 5, 2, 2, 5,
4, 5,
> 3,
> 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1, 4, 1, 0, 1, 1, 2, 4, 3, 2,
3, 1,
> 2,
> 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1, 3, 2, 1, 2, 0, 0, 3, 2,
1, 2,
> 2,
> 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0, 4, 0, 0, 1, 1, 0, 2, 0,
1, 0,
> 1,
> 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0,
>
> and for glosea only
> 78311, 10419, 6434, 4737, 4187, 3033, 2511, 2393,
> 2263, 1773, 1759, 1034, 1759, 1052, 1112, 703, 868, 490, 383,
341,
> 380,
> 314, 504, 355, 323, 278, 182, 187, 48, 102, 69, 94, 0, 0, 40, 0,
40,
> 0,
> 40, 0, 40, 99, 40, 0, 0, 0, 40, 0, 0, 0, 48, 0, 0, 0, 0, 0, 0,
0, 0,
> 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
> 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
> 48,
> 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
> 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
> 0,
> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0,
>
> Looking at the first bin 93193+78311 = 171504, bin 2 is
7953+10419=18373
> etc. which are the values in bins 1 and 2 for both the marginal
> distributions when I input both files together. I am wondering
whether the
> attribute changes are not pulled through to this somehow? This is
for the
> sample data you've got and using MET9.1. My workaround for now is
that I
> will have to run the code 3 times, once for the joint distribution,
which
> looks ok, and twice more running it with just the single data stream
to get
> the marginals for the two datasets.
>
> Regards
> Marion
>
> -----Original Message-----
> From: John Halley Gotway via RT <met_help at ucar.edu>
> Sent: 15 February 2021 17:38
> To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> Subject: Re: FW: [rt.rap.ucar.edu #97766] Need help with GridDiag
>
> This email was received from an external source. Always check
sender
> details, links & attachments.
>
> Marion,
>
> Thanks for sending sample data. I am a little confused though. The
config
> file references variables named "precipitation_amount" and
> "Precipitation_Amount":
> * name = "precipitation_amount";*
> * name = "Precipitation_Amount"; *
>
> But the actual data you sent does not:
>
>
> * ncdump -h glosea_2019060100_d001_001.nc
> <http://glosea_2019060100_d001_001.nc> | grep float float
> precipitation_amount(latitude, longitude) ;*
>
> * ncdump -h gpm_2019060100_2019060200.nc
> <http://gpm_2019060100_2019060200.nc> | grep float float
> precipitation_amount(latitude, longitude) ;*
>
> I do believe the NetCDF is case-sensitive. So I think that's
important.
> But I decided to just try to get Grid-Diag working on the data you
sent. I
> did run into an error... different from the one you sent:
>
>
> *terminate called after throwing an instance of
> 'netCDF::exceptions::NcNameInUse' what(): NetCDF: String match to
name in
> use*
>
> Grid-Diag reads the first variable from the first "-data" source and
the
> second variable from the second "-data" source. And writes data to
the
> output NetCDF file. But since those variable names are the same,
there's a
> name conflict! It's trying to write the same variable names to the
output
> NetCDF file which causes the NetCDF library to error out.
>
> To get around this, I redefined the input variable names and levels
in the
> config file using:
> * set_attr_name = "glosea_precip";*
>
> * set_attr_level = "sfc"; set_attr_name = "gpm_precip";*
> * set_attr_level = "sfc";*
>
> That works fine and produces the attached output file. I also
attached my
> run command, config file, and resulting log file.
>
> Here's some improvements we could make:
>
> (1) Using set_attr_name and set_attr_level works great. But if you
look
> closely at the log messages, the updated strings are NOT used there.
Seems
> like they should be. Do you agree?
>
> (2) The actual data values don't go much over 250... but you've
defined
> bins all the way out to 1000. That'll leave you with at least 75% of
the
> bins empty. Can you think of any enhancements to make the setting of
the
> number and size of bins easier?
>
> (3) The GPM data appears to use a value of 9.96921E36 as missing
data, but
> no _FillValue or missing_value variable attribute defines that. So
MET
> processes considers all these pink values to be valid data. Doesn't
matter
> in this case because you're only using data from India, which has
all valid
> data.
> [image: Screen Shot 2021-02-15 at 10.31.31 AM.png] I'd recommend
using
> _FillValue or missing_value to define bad data values in the GPM
files:
>
>
> https://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-
conventions.html#missing-data
> And do the same for the glosea files as well.
>
> Thanks,
> John
>
> On Mon, Feb 15, 2021 at 4:43 AM marion.mittermaier at metoffice.gov.uk
via
> RT < met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
> >
> > Hi all,
> >
> > in the mean time I have got regrid_data_plane to work in stand-
alone
> > model… running the following command (and I’ve tried in both
directions):
> >
> > regrid_data_plane glosea_2019060100_d001_001.nc
> > gpm_2019060100_2019060200.nc grid_diag_out.nc -field 019060200.nc
> > grid_diag_out.nc -field ‘name="precipitation_amount";
level="(*,*)";'
> > <
> >
> > DEBUG 1: Reading data file: glosea_2019060100_d001_001.nc DEBUG 2:
> > Input grid: Projection: Lat/Lon Nx: 432 Ny: 324 lat_ll: -89.722
> > lon_ll: -0.417 delta_lat: 0.556 delta_lon: 0.833 DEBUG 2: Output
grid:
> > Projection: Lat/Lon Nx: 3600 Ny: 1800 lat_ll:
> > -89.950 lon_ll: 179.950 delta_lat: 0.100 delta_lon: 0.100 DEBUG 2:
> > Interpolation options: method = NEAREST, width = 1, shape =
SQUARE,
> > vld_thresh = 0.5 DEBUG 2: Range of input data
> > (name="precipitation_amount"; level="(*,*)";) is 0 to 262.128.
> > DEBUG 2: Range of regridded data (name="precipitation_amount";
> > level="(*,*)";) is 0 to 262.128.
> > DEBUG 1: Writing output file: grid_diag_out.nc
> >
> > Which looks sensible… so it’s NOT regrid_data_plane.
> >
> > Marion
> >
> > From: Mittermaier, Marion
> > Sent: 15 February 2021 11:02
> > To: 'met_help at ucar.edu' <met_help at ucar.edu>
> > Subject: RE: [rt.rap.ucar.edu #97766] Need help with GridDiag
> >
> >
> > Hi John,
> >
> >
> >
> > it's taken me a while to get back to this... I do see the
difference
> > but I still don't get it to work. I can get either on their own to
> > work but not together.
> >
> > I have split out the forecast days into separate files.
> >
> >
> >
> > grid_diag -data
> > /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc
> > -data /scratch/fryl/GPM/gpm_2019060100_2019060200.nc -config
> > GridDiag.glosea -out grid_diag_out.nc -v 4
> >
> >
> >
> > This gives the following error (I think it's in
regrid_data_plane).
> > Not sure why it is trying to regrid both? Or maybe I’m just not
quite
> > following the messages….
> >
> >
> >
> > vld497:/net/home/h02/frmm/WCSSP/India > grid_diag -data
> > /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc -data
> > /scratch/fryl/GPM/gpm_2019060100_2019060200.nc
> >
> > DEBUG 1: Default Config File:
> >
/data/users/cfrd/MET_requirements/MET9.0/share/met/config/GridDiagConf
> > ig_default
> >
> > DEBUG 1: User Config File: GridDiag.glosea
> >
> > DEBUG 2: Read config files.
> >
> > DEBUG 2: Enter get_mtddf.
> >
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_NcCF".
> >
> > DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> > "hours since 1970-01-01"
> >
> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention
time
> > unit string "hours since 1970-01-01" as a reference time of
> > 19700101_000000 and
> > 3600 second(s) per time step.
> >
> > DEBUG 4: NcCfFile::open() -> could not extract init time from the
> > "forecast_reference_time" variable.
> >
> > DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
> >
> > DEBUG 2: Process configuration.
> >
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> > of type "FileType_NcCF".
> >
> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> > of type "FileType_NcCF".
> >
> > DEBUG 3: Use the grid defined by file
"/scratch/fryl/WCSSPIndia/GPM/
> > gpm_2019060100_2019060200.nc".
> >
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_NcCF".
> >
> > DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> > "hours since 1970-01-01"
> >
> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention
time
> > unit string "hours since 1970-01-01" as a reference time of
> > 19700101_000000 and
> > 3600 second(s) per time step.
> >
> > DEBUG 4: NcCfFile::open() -> could not extract init time from the
> > "forecast_reference_time" variable.
> >
> > DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
> >
> > DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 360
lat_ll:
> > 4.550 lon_ll: -64.550 delta_lat: 0.100 delta_lon: 0.100
> >
> > DEBUG 2: Processing masking regions.
> >
> > DEBUG 3: Processing poly mask: India.poly
> >
> > DEBUG 4: parse_poly_mask() -> parsing poly mask "India.poly"
> >
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_None".
> >
> > DEBUG 4: parse_poly_mask() -> parsing poly mask file "India.poly"
> >
> > DEBUG 2: Initializing precipitation_amount(*,*) histogram
> >
> > DEBUG 2: Initializing Precipitation_Amount(*,*) histogram
> >
> > DEBUG 2: Initializing
> > precipitation_amount(*,*)_Precipitation_Amount(*,*)
> > joint histogram
> >
> > DEBUG 1: grid_diag_out.nc
> >
> > DEBUG 1: Length of configuration "data.field" = 2
> >
> > DEBUG 1: Length of data file list = 1
> >
> > DEBUG 1: Series defined by the data file list of length 1.
> >
> > DEBUG 2: 1000
> >
> > DEBUG 2: Processing series entry 1: precipitation_amount:
> > precipitation_amount(*,*)
> >
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_NcCF".
> >
> > DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> > "hours since 1970-01-01"
> >
> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention
time
> > unit string "hours since 1970-01-01" as a reference time of
> > 19700101_000000 and
> > 3600 second(s) per time step.
> >
> > DEBUG 4: NcCfFile::open() -> could not extract init time from the
> > "forecast_reference_time" variable.
> >
> > DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
> >
> > DEBUG 4: NcCfFile::getData() -> setting the unset init time to the
> > valid time of 20190601_115959.
> >
> > DEBUG 4:
> >
> > DEBUG 4: Data plane information:
> >
> > DEBUG 4: plane min: 0
> >
> > DEBUG 4: plane max: 9.96921e+36
> >
> > DEBUG 4: valid time: 20190601_115959
> >
> > DEBUG 4: lead time: 000000
> >
> > DEBUG 4: init time: 20190601_115959
> >
> > DEBUG 4: accum time: 000000
> >
> > DEBUG 1: Regridding field precipitation_amount(*,*) to the
> > verification grid.
> >
> > DEBUG 2: Processing series entry 1: Precipitation_Amount:
> > Precipitation_Amount(*,*)
> >
> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
> > Met2dDataFile object of type "FileType_NcCF".
> >
> > DEBUG 4: NcCfFile::open() -> parsing units for the time variable
> > "hours since 1970-01-01"
> >
> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention
time
> > unit string "hours since 1970-01-01" as a reference time of
> > 19700101_000000 and
> > 3600 second(s) per time step.
> >
> > DEBUG 4: NcCfFile::open() -> could not extract init time from the
> > "forecast_reference_time" variable.
> >
> > DEBUG 4: NcCfFile::open() -> could not extract init time from file
name.
> >
> > DEBUG 1: Regridding field precipitation_amount(*,*) to the
> > verification grid.
> >
> > ERROR :
> >
> > ERROR : DataPlane::two_to_one() -> range check error: (Nx, Ny) =
(0,
> > 0), (x, y) = (0, 0)
> >
> > ERROR :
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT <met_help at ucar.edu<mailto:
> > met_help at ucar.edu>>
> > Sent: 08 December 2020 20:54
> > To: Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk<mailto:
> > marion.mittermaier at metoffice.gov.uk>>
> > Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
> >
> >
> >
> > This email was received from an external source. Always check
sender
> > details, links & attachments.
> >
> >
> >
> > Marion,
> >
> >
> >
> > I'm still not convinced. Please take a very careful look at the
> > command you're executing. I see all the log messages you sent, but
not
> > the actual command you ran. At that matters a lot!
> >
> >
> >
> > This command fails on my machine with the SAME ERROR you're
getting:
> >
> > * grid_diag -data glosea_2019060100_001.nc
> >
> > <http://glosea_2019060100_001.nc/> gpm_2019060100_2019060200.nc <
> > http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea -out
> > grid_diag_out.nc <http://grid_diag_out.nc/>*
> >
> >
> >
> > ERROR : NcCfFile::getData(NcVar *, const LongArray &,
DataPlane &)
> > const -> needed 3 arguments for variable precipitation_amount, got
2
> >
> >
> >
> > But this command runs WITHOUT ERROR:
> >
> > * grid_diag -data glosea_2019060100_001.nc
> >
> > <http://glosea_2019060100_001.nc/> -data
gpm_2019060100_2019060200.nc
> > < http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea
-out
> > grid_diag_out.nc <http://grid_diag_out.nc/>*
> >
> >
> >
> > Do you see the difference?
> >
> > In the first, I used "-data" once and listed 2 different file
names.
> >
> > In the second, I used "-data" twice, once for each of the input
files.
> >
> >
> >
> > Please try doing exactly the same thing.
> >
> >
> >
> > Does that enable it to run without error? Note that you'll still
get
> > warnings about the actual data values falling outside the defined
> > range of the bins.
> >
> >
> >
> > As for the bin endpoints, I think this is the relevant line of
code:
> >
> >
> >
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc
> > 5981bb8/met/src/libcode/vx_series_data/series_pdf.cc#L72
> >
> >
> >
> > int k = floor((value - min) / delta);
> >
> >
> >
> > This places the current "value" into the k-th bin where "min" is
the
> > minimum PDF value and "delta" is the bin width.
> >
> > I believe this is consistent with bins being closed on the left
side
> > and open on the right:
> >
> > [0,1) i.e. 0 <= x < 1
> >
> >
> >
> > John
> >
> >
> >
> > On Tue, Dec 8, 2020 at 8:30 AM marion.mittermaier at metoffice.gov.uk
> <mailto:
> > marion.mittermaier at metoffice.gov.uk> via RT <
met_help at ucar.edu<mailto:
> > met_help at ucar.edu>> wrote:
> >
> >
> >
> > >
> >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
> >
> > >
> >
> > > Hi John,
> >
> > >
> >
> > > one final question for GridDiag.
> >
> > >
> >
> > > The bins are set up to be lower bound inclusive? i.e.
> >
> > > n_bins = 1000;
> >
> > > range = [0, 1000]; // kg/m^2
> >
> > >
> >
> > > means that the bin is set up as [0,1) i.e. 0 <= x < 1?
> >
> > >
> >
> > > I'm presuming so.... and the upper bound would be [999,1000)
with no
> >
> > > bin for [1000,inf)? i.e. if there were a 1000 in my sample it
would
> >
> > > NOT be counted? Or anything bigger.
> >
> > >
> >
> > > That's what it looks like, i.e. you need to make sure that your
> > > range
> >
> > > encompasses all possible values. Not sure this is in the
> > documentation...?
> >
> > > Would be useful to add.
> >
> > >
> >
> > > Ta
> >
> > > Marion
> >
> > >
> >
> > > -----Original Message-----
> >
> > > From: John Halley Gotway via RT <met_help at ucar.edu<mailto:
> > met_help at ucar.edu>>
> >
> > > Sent: 05 December 2020 06:41
> >
> > > To: Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk<mailto:
> > marion.mittermaier at metoffice.gov.uk>>
> >
> > > Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
> >
> > >
> >
> > > This email was received from an external source. Always check
sender
> >
> > > details, links & attachments.
> >
> > >
> >
> > > Marion,
> >
> > >
> >
> > > Ugh, I should've just handed this ticket off to David in the
first
> place.
> >
> > > I did more testing and was able to get it to work. I just had to
use
> >
> > > separate "-data" command line options for each input data
source.
> >
> > >
> >
> > > So this command fails:
> >
> > > grid_diag -data glosea_2019060100_001.nc
> > > gpm_2019060100_2019060200.nc
> >
> > > -config GridDiag.glosea -out grid_diag_out.nc
> >
> > >
> >
> > > But this command runs fine:
> >
> > > *grid_diag -data glosea_2019060100_001.nc <
> >
> > > http://glosea_2019060100_001.nc> -data
gpm_2019060100_2019060200.nc
> > > <
> >
> > > http://gpm_2019060100_2019060200.nc> -config GridDiag.glosea
-out
> >
> > > grid_diag_out.nc <http://grid_diag_out.nc>*
> >
> > >
> >
> > > In the second one, I've specified "-data" separately for each
data
> > source.
> >
> > > Although, you'll see warning about the range of the data
extending
> >
> > > beyond the range of the bins defined in the config file.
> >
> > >
> >
> > > Sorry to have such a rambling and confusing response!
> >
> > >
> >
> > > So to summarize, I believe the logic is:
> >
> > > 1. If "-data" is used once, all specified input fields are read
from
> >
> > > all of those data files.
> >
> > > 2. If "-data" is used more than once, the number of "-data"
entries
> >
> > > must match the number of fields defined in the config file.
> >
> > > If not, you get an error message.
> >
> > > If so, only the i-th field is read from the files in the i-
th
> >
> > > -data source.
> >
> > >
> >
> > > John
> >
> > >
> >
> > > On Fri, Dec 4, 2020 at 11:30 PM John Halley Gotway
<johnhg at ucar.edu
> > <mailto:johnhg at ucar.edu>>
> >
> > > wrote:
> >
> > >
> >
> > > > Marion,
> >
> > > >
> >
> > > > It occurs to me that perhaps Grid-Diag could be made more
flexible
> >
> > > > if we changed that error message to a debug message in this
context.
> >
> > > > It could
> >
> > > > *ATTEMPT* to read all input fields from all input files... but
> > > > when
> >
> > > > it's unable to, have it print a log message stating that
rather
> > > > than
> >
> > > > erroring out.
> >
> > > >
> >
> > > > That would enable you to compute 2D pdf's using different
> > > > variables
> >
> > > > from different input files.
> >
> > > >
> >
> > > > Do you think that logic is preferable?
> >
> > > >
> >
> > > > Thanks,
> >
> > > > John
> >
> > > >
> >
> > > > On Fri, Dec 4, 2020 at 11:27 PM The RT System itself via RT <
> >
> > > > met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
> >
> > > >
> >
> > > >>
> >
> > > >> Fri Dec 04 23:27:19 2020: Request 97766 was acted upon.
> >
> > > >> Transaction: Given to johnhg (John Halley Gotway) by
RT_System
> >
> > > >> Queue: met_help
> >
> > > >> Subject: Need help with GridDiag
> >
> > > >> Owner: johnhg
> >
> > > >> Requestors: marion.mittermaier at metoffice.gov.uk<mailto:
> > marion.mittermaier at metoffice.gov.uk>
> >
> > > >> Status: new
> >
> > > >> Ticket <URL:
> >
> > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
> >
> > > >> >
> >
> > > >>
> >
> > > >>
> >
> > > >> This transaction appears to have no content
> >
> > > >>
> >
> > > >
> >
> > >
> >
> > >
> >
> > >
> >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: Need help with GridDiag
From: John Halley Gotway
Time: Thu Mar 04 09:45:19 2021
Marion,
FYI, I'm pretty confident the bug is caused by the fact that we're
reading
the same variable named "precipitation_amount" from two different
input
data sources. When I change the variable name of the second one to
"precipitation_amount_MOD" then the problem goes away. Then 1D
histograms
are then correct.
So that would provide a short-term, temporary bandaid for now. You
could
nc_rename precipitaiton_amount to some other (unique) name prior to
running
grid_diag with met-9.1.
We should just need to update the code to make the string we're using
to
track the counts unique. And that seems very doable for met-10.0.0.
Thanks,
John
On Thu, Mar 4, 2021 at 9:30 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Marion,
>
> You're right. I looked closely at the numbers that grid_diag is
producing
> and this is pretty obviously a bug. Thanks for finding this and
letting us
> know!
>
> I'll coordinate with David Fillmore, the engineer who originally
worked on
> this to get this fixed for the met-10.0.0 release.
>
> John
>
> On Thu, Mar 4, 2021 at 2:29 AM marion.mittermaier at metoffice.gov.uk
via RT
> <met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>>
>> Hi John,
>>
>> I thought I had cracked it but something is still not quite
right... the
>> joint distribution does appear when using the dual -data commands
with the
>> changed attribute names, and it looks plausible BUT but the
marginal
>> distributions are wrong.
>>
>> If I run the GridDiag code with both data sets then the marginal
>> distributions are the same, which isn't right.
>> running together
>> glosea 171504, 18373, 11192, 7943, 6551, 4922, 4065, 3680,
>> 3344, 2750, 2583, 1719, 2432, 1602, 1632, 1173, 1358, 899, 774,
685,
>> 720,
>> 625, 782, 650, 569, 502, 377, 385, 236, 286, 244, 237, 123,
116, 156,
>> 112, 128, 100, 151, 88, 122, 173, 109, 63, 62, 53, 88, 40, 49,
46,
>> 84,
>> 29, 32, 48, 41, 29, 30, 32, 36, 32, 29, 26, 24, 18, 20, 22, 20,
24,
>> 21,
>> 16, 15, 21, 12, 20, 16, 15, 17, 16, 7, 13, 11, 15, 8, 12, 10,
11, 11,
>> 3,
>> 8, 9, 4, 6, 13, 7, 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4, 5, 50,
5, 7,
>> 6, 5,
>> 2, 2, 53, 4, 5, 3, 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1, 4, 1, 0,
1, 1,
>> 2,
>> 4, 3, 2, 3, 1, 2, 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1, 3, 2,
1, 2,
>> 0,
>> 0, 3, 2, 1, 2, 2, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0, 4, 0,
0, 1,
>> 1,
>> 0, 2, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0,
0, 0,
>> 0,
>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>> 0,
>> gpm 171504, 18373, 11192, 7943, 6551, 4922, 4065, 3680,
>> 3344, 2750, 2583, 1719, 2432, 1602, 1632, 1173, 1358, 899, 774,
685,
>> 720,
>> 625, 782, 650, 569, 502, 377, 385, 236, 286, 244, 237, 123,
116, 156,
>> 112, 128, 100, 151, 88, 122, 173, 109, 63, 62, 53, 88, 40, 49,
46,
>> 84,
>> 29, 32, 48, 41, 29, 30, 32, 36, 32, 29, 26, 24, 18, 20, 22, 20,
24,
>> 21,
>> 16, 15, 21, 12, 20, 16, 15, 17, 16, 7, 13, 11, 15, 8, 12, 10,
11, 11,
>> 3,
>> 8, 9, 4, 6, 13, 7, 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4, 5, 50,
5, 7,
>> 6, 5,
>> 2, 2, 53, 4, 5, 3, 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1, 4, 1, 0,
1, 1,
>> 2,
>> 4, 3, 2, 3, 1, 2, 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1, 3, 2,
1, 2,
>> 0,
>> 0, 3, 2, 1, 2, 2, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0, 4, 0,
0, 1,
>> 1,
>> 0, 2, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0,
0, 0,
>> 0,
>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>> 0,
>>
>> But when I run them separately I get the following for gpm only
(same
>> input files)
>> 93193, 7954, 4758, 3206, 2364, 1889, 1554, 1287, 1081,
>> 977, 824, 685, 673, 550, 520, 470, 490, 409, 391, 344, 340,
311, 278,
>> 295, 246, 224, 195, 198, 188, 184, 175, 143, 123, 116, 116,
112, 88,
>> 100,
>> 111, 88, 82, 74, 69, 63, 62, 53, 48, 40, 49, 46, 36, 29, 32,
48, 41,
>> 29,
>> 30, 32, 36, 32, 29, 26, 24, 18, 20, 22, 20, 24, 21, 16, 15, 21,
12,
>> 20,
>> 16, 15, 17, 16, 7, 13, 11, 15, 8, 12, 10, 11, 11, 3, 8, 9, 4,
6, 13,
>> 7,
>> 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4, 5, 2, 5, 7, 6, 5, 2, 2, 5,
4, 5,
>> 3,
>> 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1, 4, 1, 0, 1, 1, 2, 4, 3, 2,
3, 1,
>> 2,
>> 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1, 3, 2, 1, 2, 0, 0, 3, 2,
1, 2,
>> 2,
>> 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0, 4, 0, 0, 1, 1, 0, 2, 0,
1, 0,
>> 1,
>> 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>> 0,
>>
>> and for glosea only
>> 78311, 10419, 6434, 4737, 4187, 3033, 2511, 2393,
>> 2263, 1773, 1759, 1034, 1759, 1052, 1112, 703, 868, 490, 383,
341,
>> 380,
>> 314, 504, 355, 323, 278, 182, 187, 48, 102, 69, 94, 0, 0, 40,
0, 40,
>> 0,
>> 40, 0, 40, 99, 40, 0, 0, 0, 40, 0, 0, 0, 48, 0, 0, 0, 0, 0, 0,
0, 0,
>> 0,
>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>> 0,
>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>> 48,
>> 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>> 0,
>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>> 0,
>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>> 0,
>>
>> Looking at the first bin 93193+78311 = 171504, bin 2 is
7953+10419=18373
>> etc. which are the values in bins 1 and 2 for both the marginal
>> distributions when I input both files together. I am wondering
whether the
>> attribute changes are not pulled through to this somehow? This is
for the
>> sample data you've got and using MET9.1. My workaround for now is
that I
>> will have to run the code 3 times, once for the joint distribution,
which
>> looks ok, and twice more running it with just the single data
stream to get
>> the marginals for the two datasets.
>>
>> Regards
>> Marion
>>
>> -----Original Message-----
>> From: John Halley Gotway via RT <met_help at ucar.edu>
>> Sent: 15 February 2021 17:38
>> To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
>> Subject: Re: FW: [rt.rap.ucar.edu #97766] Need help with GridDiag
>>
>> This email was received from an external source. Always check
sender
>> details, links & attachments.
>>
>> Marion,
>>
>> Thanks for sending sample data. I am a little confused though. The
config
>> file references variables named "precipitation_amount" and
>> "Precipitation_Amount":
>> * name = "precipitation_amount";*
>> * name = "Precipitation_Amount"; *
>>
>> But the actual data you sent does not:
>>
>>
>> * ncdump -h glosea_2019060100_d001_001.nc
>> <http://glosea_2019060100_d001_001.nc> | grep float float
>> precipitation_amount(latitude, longitude) ;*
>>
>> * ncdump -h gpm_2019060100_2019060200.nc
>> <http://gpm_2019060100_2019060200.nc> | grep float float
>> precipitation_amount(latitude, longitude) ;*
>>
>> I do believe the NetCDF is case-sensitive. So I think that's
important.
>> But I decided to just try to get Grid-Diag working on the data you
sent. I
>> did run into an error... different from the one you sent:
>>
>>
>> *terminate called after throwing an instance of
>> 'netCDF::exceptions::NcNameInUse' what(): NetCDF: String match to
name in
>> use*
>>
>> Grid-Diag reads the first variable from the first "-data" source
and the
>> second variable from the second "-data" source. And writes data to
the
>> output NetCDF file. But since those variable names are the same,
there's a
>> name conflict! It's trying to write the same variable names to the
output
>> NetCDF file which causes the NetCDF library to error out.
>>
>> To get around this, I redefined the input variable names and levels
in
>> the config file using:
>> * set_attr_name = "glosea_precip";*
>>
>> * set_attr_level = "sfc"; set_attr_name = "gpm_precip";*
>> * set_attr_level = "sfc";*
>>
>> That works fine and produces the attached output file. I also
attached my
>> run command, config file, and resulting log file.
>>
>> Here's some improvements we could make:
>>
>> (1) Using set_attr_name and set_attr_level works great. But if you
look
>> closely at the log messages, the updated strings are NOT used
there. Seems
>> like they should be. Do you agree?
>>
>> (2) The actual data values don't go much over 250... but you've
defined
>> bins all the way out to 1000. That'll leave you with at least 75%
of the
>> bins empty. Can you think of any enhancements to make the setting
of the
>> number and size of bins easier?
>>
>> (3) The GPM data appears to use a value of 9.96921E36 as missing
data,
>> but no _FillValue or missing_value variable attribute defines that.
So MET
>> processes considers all these pink values to be valid data. Doesn't
matter
>> in this case because you're only using data from India, which has
all valid
>> data.
>> [image: Screen Shot 2021-02-15 at 10.31.31 AM.png] I'd recommend
using
>> _FillValue or missing_value to define bad data values in the GPM
files:
>>
>>
>> https://cfconventions.org/Data/cf-conventions/cf-conventions-
1.8/cf-conventions.html#missing-data
>> And do the same for the glosea files as well.
>>
>> Thanks,
>> John
>>
>> On Mon, Feb 15, 2021 at 4:43 AM marion.mittermaier at metoffice.gov.uk
via
>> RT < met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>> >
>> > Hi all,
>> >
>> > in the mean time I have got regrid_data_plane to work in stand-
alone
>> > model… running the following command (and I’ve tried in both
>> directions):
>> >
>> > regrid_data_plane glosea_2019060100_d001_001.nc
>> > gpm_2019060100_2019060200.nc grid_diag_out.nc -field 019060200.nc
>> > grid_diag_out.nc -field ‘name="precipitation_amount";
level="(*,*)";'
>> > <
>> >
>> > DEBUG 1: Reading data file: glosea_2019060100_d001_001.nc DEBUG
2:
>> > Input grid: Projection: Lat/Lon Nx: 432 Ny: 324 lat_ll: -89.722
>> > lon_ll: -0.417 delta_lat: 0.556 delta_lon: 0.833 DEBUG 2: Output
grid:
>> > Projection: Lat/Lon Nx: 3600 Ny: 1800 lat_ll:
>> > -89.950 lon_ll: 179.950 delta_lat: 0.100 delta_lon: 0.100 DEBUG
2:
>> > Interpolation options: method = NEAREST, width = 1, shape =
SQUARE,
>> > vld_thresh = 0.5 DEBUG 2: Range of input data
>> > (name="precipitation_amount"; level="(*,*)";) is 0 to 262.128.
>> > DEBUG 2: Range of regridded data (name="precipitation_amount";
>> > level="(*,*)";) is 0 to 262.128.
>> > DEBUG 1: Writing output file: grid_diag_out.nc
>> >
>> > Which looks sensible… so it’s NOT regrid_data_plane.
>> >
>> > Marion
>> >
>> > From: Mittermaier, Marion
>> > Sent: 15 February 2021 11:02
>> > To: 'met_help at ucar.edu' <met_help at ucar.edu>
>> > Subject: RE: [rt.rap.ucar.edu #97766] Need help with GridDiag
>> >
>> >
>> > Hi John,
>> >
>> >
>> >
>> > it's taken me a while to get back to this... I do see the
difference
>> > but I still don't get it to work. I can get either on their own
to
>> > work but not together.
>> >
>> > I have split out the forecast days into separate files.
>> >
>> >
>> >
>> > grid_diag -data
>> > /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc
>> > -data /scratch/fryl/GPM/gpm_2019060100_2019060200.nc -config
>> > GridDiag.glosea -out grid_diag_out.nc -v 4
>> >
>> >
>> >
>> > This gives the following error (I think it's in
regrid_data_plane).
>> > Not sure why it is trying to regrid both? Or maybe I’m just not
quite
>> > following the messages….
>> >
>> >
>> >
>> > vld497:/net/home/h02/frmm/WCSSP/India > grid_diag -data
>> > /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc -data
>> > /scratch/fryl/GPM/gpm_2019060100_2019060200.nc
>> >
>> > DEBUG 1: Default Config File:
>> >
/data/users/cfrd/MET_requirements/MET9.0/share/met/config/GridDiagConf
>> > ig_default
>> >
>> > DEBUG 1: User Config File: GridDiag.glosea
>> >
>> > DEBUG 2: Read config files.
>> >
>> > DEBUG 2: Enter get_mtddf.
>> >
>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
>> > Met2dDataFile object of type "FileType_NcCF".
>> >
>> > DEBUG 4: NcCfFile::open() -> parsing units for the time variable
>> > "hours since 1970-01-01"
>> >
>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention
time
>> > unit string "hours since 1970-01-01" as a reference time of
>> > 19700101_000000 and
>> > 3600 second(s) per time step.
>> >
>> > DEBUG 4: NcCfFile::open() -> could not extract init time from the
>> > "forecast_reference_time" variable.
>> >
>> > DEBUG 4: NcCfFile::open() -> could not extract init time from
file name.
>> >
>> > DEBUG 2: Process configuration.
>> >
>> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
>> > of type "FileType_NcCF".
>> >
>> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
>> > of type "FileType_NcCF".
>> >
>> > DEBUG 3: Use the grid defined by file
"/scratch/fryl/WCSSPIndia/GPM/
>> > gpm_2019060100_2019060200.nc".
>> >
>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
>> > Met2dDataFile object of type "FileType_NcCF".
>> >
>> > DEBUG 4: NcCfFile::open() -> parsing units for the time variable
>> > "hours since 1970-01-01"
>> >
>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention
time
>> > unit string "hours since 1970-01-01" as a reference time of
>> > 19700101_000000 and
>> > 3600 second(s) per time step.
>> >
>> > DEBUG 4: NcCfFile::open() -> could not extract init time from the
>> > "forecast_reference_time" variable.
>> >
>> > DEBUG 4: NcCfFile::open() -> could not extract init time from
file name.
>> >
>> > DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 360
lat_ll:
>> > 4.550 lon_ll: -64.550 delta_lat: 0.100 delta_lon: 0.100
>> >
>> > DEBUG 2: Processing masking regions.
>> >
>> > DEBUG 3: Processing poly mask: India.poly
>> >
>> > DEBUG 4: parse_poly_mask() -> parsing poly mask "India.poly"
>> >
>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
>> > Met2dDataFile object of type "FileType_None".
>> >
>> > DEBUG 4: parse_poly_mask() -> parsing poly mask file "India.poly"
>> >
>> > DEBUG 2: Initializing precipitation_amount(*,*) histogram
>> >
>> > DEBUG 2: Initializing Precipitation_Amount(*,*) histogram
>> >
>> > DEBUG 2: Initializing
>> > precipitation_amount(*,*)_Precipitation_Amount(*,*)
>> > joint histogram
>> >
>> > DEBUG 1: grid_diag_out.nc
>> >
>> > DEBUG 1: Length of configuration "data.field" = 2
>> >
>> > DEBUG 1: Length of data file list = 1
>> >
>> > DEBUG 1: Series defined by the data file list of length 1.
>> >
>> > DEBUG 2: 1000
>> >
>> > DEBUG 2: Processing series entry 1: precipitation_amount:
>> > precipitation_amount(*,*)
>> >
>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
>> > Met2dDataFile object of type "FileType_NcCF".
>> >
>> > DEBUG 4: NcCfFile::open() -> parsing units for the time variable
>> > "hours since 1970-01-01"
>> >
>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention
time
>> > unit string "hours since 1970-01-01" as a reference time of
>> > 19700101_000000 and
>> > 3600 second(s) per time step.
>> >
>> > DEBUG 4: NcCfFile::open() -> could not extract init time from the
>> > "forecast_reference_time" variable.
>> >
>> > DEBUG 4: NcCfFile::open() -> could not extract init time from
file name.
>> >
>> > DEBUG 4: NcCfFile::getData() -> setting the unset init time to
the
>> > valid time of 20190601_115959.
>> >
>> > DEBUG 4:
>> >
>> > DEBUG 4: Data plane information:
>> >
>> > DEBUG 4: plane min: 0
>> >
>> > DEBUG 4: plane max: 9.96921e+36
>> >
>> > DEBUG 4: valid time: 20190601_115959
>> >
>> > DEBUG 4: lead time: 000000
>> >
>> > DEBUG 4: init time: 20190601_115959
>> >
>> > DEBUG 4: accum time: 000000
>> >
>> > DEBUG 1: Regridding field precipitation_amount(*,*) to the
>> > verification grid.
>> >
>> > DEBUG 2: Processing series entry 1: Precipitation_Amount:
>> > Precipitation_Amount(*,*)
>> >
>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
>> > Met2dDataFile object of type "FileType_NcCF".
>> >
>> > DEBUG 4: NcCfFile::open() -> parsing units for the time variable
>> > "hours since 1970-01-01"
>> >
>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention
time
>> > unit string "hours since 1970-01-01" as a reference time of
>> > 19700101_000000 and
>> > 3600 second(s) per time step.
>> >
>> > DEBUG 4: NcCfFile::open() -> could not extract init time from the
>> > "forecast_reference_time" variable.
>> >
>> > DEBUG 4: NcCfFile::open() -> could not extract init time from
file name.
>> >
>> > DEBUG 1: Regridding field precipitation_amount(*,*) to the
>> > verification grid.
>> >
>> > ERROR :
>> >
>> > ERROR : DataPlane::two_to_one() -> range check error: (Nx, Ny) =
(0,
>> > 0), (x, y) = (0, 0)
>> >
>> > ERROR :
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: John Halley Gotway via RT <met_help at ucar.edu<mailto:
>> > met_help at ucar.edu>>
>> > Sent: 08 December 2020 20:54
>> > To: Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk<mailto:
>> > marion.mittermaier at metoffice.gov.uk>>
>> > Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
>> >
>> >
>> >
>> > This email was received from an external source. Always check
sender
>> > details, links & attachments.
>> >
>> >
>> >
>> > Marion,
>> >
>> >
>> >
>> > I'm still not convinced. Please take a very careful look at the
>> > command you're executing. I see all the log messages you sent,
but not
>> > the actual command you ran. At that matters a lot!
>> >
>> >
>> >
>> > This command fails on my machine with the SAME ERROR you're
getting:
>> >
>> > * grid_diag -data glosea_2019060100_001.nc
>> >
>> > <http://glosea_2019060100_001.nc/> gpm_2019060100_2019060200.nc <
>> > http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea
-out
>> > grid_diag_out.nc <http://grid_diag_out.nc/>*
>> >
>> >
>> >
>> > ERROR : NcCfFile::getData(NcVar *, const LongArray &,
DataPlane &)
>> > const -> needed 3 arguments for variable precipitation_amount,
got 2
>> >
>> >
>> >
>> > But this command runs WITHOUT ERROR:
>> >
>> > * grid_diag -data glosea_2019060100_001.nc
>> >
>> > <http://glosea_2019060100_001.nc/> -data
gpm_2019060100_2019060200.nc
>> > < http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea
-out
>> > grid_diag_out.nc <http://grid_diag_out.nc/>*
>> >
>> >
>> >
>> > Do you see the difference?
>> >
>> > In the first, I used "-data" once and listed 2 different file
names.
>> >
>> > In the second, I used "-data" twice, once for each of the input
files.
>> >
>> >
>> >
>> > Please try doing exactly the same thing.
>> >
>> >
>> >
>> > Does that enable it to run without error? Note that you'll still
get
>> > warnings about the actual data values falling outside the defined
>> > range of the bins.
>> >
>> >
>> >
>> > As for the bin endpoints, I think this is the relevant line of
code:
>> >
>> >
>> >
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc
>> > 5981bb8/met/src/libcode/vx_series_data/series_pdf.cc#L72
>> >
>> >
>> >
>> > int k = floor((value - min) / delta);
>> >
>> >
>> >
>> > This places the current "value" into the k-th bin where "min" is
the
>> > minimum PDF value and "delta" is the bin width.
>> >
>> > I believe this is consistent with bins being closed on the left
side
>> > and open on the right:
>> >
>> > [0,1) i.e. 0 <= x < 1
>> >
>> >
>> >
>> > John
>> >
>> >
>> >
>> > On Tue, Dec 8, 2020 at 8:30 AM
marion.mittermaier at metoffice.gov.uk
>> <mailto:
>> > marion.mittermaier at metoffice.gov.uk> via RT <
met_help at ucar.edu<mailto:
>> > met_help at ucar.edu>> wrote:
>> >
>> >
>> >
>> > >
>> >
>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>> >
>> > >
>> >
>> > > Hi John,
>> >
>> > >
>> >
>> > > one final question for GridDiag.
>> >
>> > >
>> >
>> > > The bins are set up to be lower bound inclusive? i.e.
>> >
>> > > n_bins = 1000;
>> >
>> > > range = [0, 1000]; // kg/m^2
>> >
>> > >
>> >
>> > > means that the bin is set up as [0,1) i.e. 0 <= x < 1?
>> >
>> > >
>> >
>> > > I'm presuming so.... and the upper bound would be [999,1000)
with no
>> >
>> > > bin for [1000,inf)? i.e. if there were a 1000 in my sample it
would
>> >
>> > > NOT be counted? Or anything bigger.
>> >
>> > >
>> >
>> > > That's what it looks like, i.e. you need to make sure that your
>> > > range
>> >
>> > > encompasses all possible values. Not sure this is in the
>> > documentation...?
>> >
>> > > Would be useful to add.
>> >
>> > >
>> >
>> > > Ta
>> >
>> > > Marion
>> >
>> > >
>> >
>> > > -----Original Message-----
>> >
>> > > From: John Halley Gotway via RT <met_help at ucar.edu<mailto:
>> > met_help at ucar.edu>>
>> >
>> > > Sent: 05 December 2020 06:41
>> >
>> > > To: Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk<mailto:
>> > marion.mittermaier at metoffice.gov.uk>>
>> >
>> > > Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
>> >
>> > >
>> >
>> > > This email was received from an external source. Always check
sender
>> >
>> > > details, links & attachments.
>> >
>> > >
>> >
>> > > Marion,
>> >
>> > >
>> >
>> > > Ugh, I should've just handed this ticket off to David in the
first
>> place.
>> >
>> > > I did more testing and was able to get it to work. I just had
to use
>> >
>> > > separate "-data" command line options for each input data
source.
>> >
>> > >
>> >
>> > > So this command fails:
>> >
>> > > grid_diag -data glosea_2019060100_001.nc
>> > > gpm_2019060100_2019060200.nc
>> >
>> > > -config GridDiag.glosea -out grid_diag_out.nc
>> >
>> > >
>> >
>> > > But this command runs fine:
>> >
>> > > *grid_diag -data glosea_2019060100_001.nc <
>> >
>> > > http://glosea_2019060100_001.nc> -data
gpm_2019060100_2019060200.nc
>> > > <
>> >
>> > > http://gpm_2019060100_2019060200.nc> -config GridDiag.glosea
-out
>> >
>> > > grid_diag_out.nc <http://grid_diag_out.nc>*
>> >
>> > >
>> >
>> > > In the second one, I've specified "-data" separately for each
data
>> > source.
>> >
>> > > Although, you'll see warning about the range of the data
extending
>> >
>> > > beyond the range of the bins defined in the config file.
>> >
>> > >
>> >
>> > > Sorry to have such a rambling and confusing response!
>> >
>> > >
>> >
>> > > So to summarize, I believe the logic is:
>> >
>> > > 1. If "-data" is used once, all specified input fields are read
from
>> >
>> > > all of those data files.
>> >
>> > > 2. If "-data" is used more than once, the number of "-data"
entries
>> >
>> > > must match the number of fields defined in the config file.
>> >
>> > > If not, you get an error message.
>> >
>> > > If so, only the i-th field is read from the files in the i-
th
>> >
>> > > -data source.
>> >
>> > >
>> >
>> > > John
>> >
>> > >
>> >
>> > > On Fri, Dec 4, 2020 at 11:30 PM John Halley Gotway
<johnhg at ucar.edu
>> > <mailto:johnhg at ucar.edu>>
>> >
>> > > wrote:
>> >
>> > >
>> >
>> > > > Marion,
>> >
>> > > >
>> >
>> > > > It occurs to me that perhaps Grid-Diag could be made more
flexible
>> >
>> > > > if we changed that error message to a debug message in this
context.
>> >
>> > > > It could
>> >
>> > > > *ATTEMPT* to read all input fields from all input files...
but
>> > > > when
>> >
>> > > > it's unable to, have it print a log message stating that
rather
>> > > > than
>> >
>> > > > erroring out.
>> >
>> > > >
>> >
>> > > > That would enable you to compute 2D pdf's using different
>> > > > variables
>> >
>> > > > from different input files.
>> >
>> > > >
>> >
>> > > > Do you think that logic is preferable?
>> >
>> > > >
>> >
>> > > > Thanks,
>> >
>> > > > John
>> >
>> > > >
>> >
>> > > > On Fri, Dec 4, 2020 at 11:27 PM The RT System itself via RT <
>> >
>> > > > met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
>> >
>> > > >
>> >
>> > > >>
>> >
>> > > >> Fri Dec 04 23:27:19 2020: Request 97766 was acted upon.
>> >
>> > > >> Transaction: Given to johnhg (John Halley Gotway) by
RT_System
>> >
>> > > >> Queue: met_help
>> >
>> > > >> Subject: Need help with GridDiag
>> >
>> > > >> Owner: johnhg
>> >
>> > > >> Requestors: marion.mittermaier at metoffice.gov.uk<mailto:
>> > marion.mittermaier at metoffice.gov.uk>
>> >
>> > > >> Status: new
>> >
>> > > >> Ticket <URL:
>> >
>> > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
>> >
>> > > >> >
>> >
>> > > >>
>> >
>> > > >>
>> >
>> > > >> This transaction appears to have no content
>> >
>> > > >>
>> >
>> > > >
>> >
>> > >
>> >
>> > >
>> >
>> > >
>> >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>>
>>
------------------------------------------------
Subject: Need help with GridDiag
From: John Halley Gotway
Time: Thu Mar 04 10:37:16 2021
Marion,
Here's the GitHub issue I wrote up to describe this bug:
https://github.com/dtcenter/MET/issues/1694
John
On Thu, Mar 4, 2021 at 9:45 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Marion,
>
> FYI, I'm pretty confident the bug is caused by the fact that we're
reading
> the same variable named "precipitation_amount" from two different
input
> data sources. When I change the variable name of the second one to
> "precipitation_amount_MOD" then the problem goes away. Then 1D
histograms
> are then correct.
>
> So that would provide a short-term, temporary bandaid for now. You
could
> nc_rename precipitaiton_amount to some other (unique) name prior to
running
> grid_diag with met-9.1.
>
> We should just need to update the code to make the string we're
using to
> track the counts unique. And that seems very doable for met-10.0.0.
>
> Thanks,
> John
>
> On Thu, Mar 4, 2021 at 9:30 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
>
>> Marion,
>>
>> You're right. I looked closely at the numbers that grid_diag is
producing
>> and this is pretty obviously a bug. Thanks for finding this and
letting us
>> know!
>>
>> I'll coordinate with David Fillmore, the engineer who originally
worked
>> on this to get this fixed for the met-10.0.0 release.
>>
>> John
>>
>> On Thu, Mar 4, 2021 at 2:29 AM marion.mittermaier at metoffice.gov.uk
via
>> RT <met_help at ucar.edu> wrote:
>>
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>>>
>>> Hi John,
>>>
>>> I thought I had cracked it but something is still not quite
right... the
>>> joint distribution does appear when using the dual -data commands
with the
>>> changed attribute names, and it looks plausible BUT but the
marginal
>>> distributions are wrong.
>>>
>>> If I run the GridDiag code with both data sets then the marginal
>>> distributions are the same, which isn't right.
>>> running together
>>> glosea 171504, 18373, 11192, 7943, 6551, 4922, 4065, 3680,
>>> 3344, 2750, 2583, 1719, 2432, 1602, 1632, 1173, 1358, 899,
774, 685,
>>> 720,
>>> 625, 782, 650, 569, 502, 377, 385, 236, 286, 244, 237, 123,
116,
>>> 156,
>>> 112, 128, 100, 151, 88, 122, 173, 109, 63, 62, 53, 88, 40, 49,
46,
>>> 84,
>>> 29, 32, 48, 41, 29, 30, 32, 36, 32, 29, 26, 24, 18, 20, 22,
20, 24,
>>> 21,
>>> 16, 15, 21, 12, 20, 16, 15, 17, 16, 7, 13, 11, 15, 8, 12, 10,
11,
>>> 11, 3,
>>> 8, 9, 4, 6, 13, 7, 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4, 5, 50,
5, 7,
>>> 6, 5,
>>> 2, 2, 53, 4, 5, 3, 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1, 4, 1,
0, 1,
>>> 1, 2,
>>> 4, 3, 2, 3, 1, 2, 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1, 3, 2,
1, 2,
>>> 0,
>>> 0, 3, 2, 1, 2, 2, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0, 4, 0,
0, 1,
>>> 1,
>>> 0, 2, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0,
0, 0,
>>> 0,
>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>>> 0,
>>> gpm 171504, 18373, 11192, 7943, 6551, 4922, 4065, 3680,
>>> 3344, 2750, 2583, 1719, 2432, 1602, 1632, 1173, 1358, 899,
774, 685,
>>> 720,
>>> 625, 782, 650, 569, 502, 377, 385, 236, 286, 244, 237, 123,
116,
>>> 156,
>>> 112, 128, 100, 151, 88, 122, 173, 109, 63, 62, 53, 88, 40, 49,
46,
>>> 84,
>>> 29, 32, 48, 41, 29, 30, 32, 36, 32, 29, 26, 24, 18, 20, 22,
20, 24,
>>> 21,
>>> 16, 15, 21, 12, 20, 16, 15, 17, 16, 7, 13, 11, 15, 8, 12, 10,
11,
>>> 11, 3,
>>> 8, 9, 4, 6, 13, 7, 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4, 5, 50,
5, 7,
>>> 6, 5,
>>> 2, 2, 53, 4, 5, 3, 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1, 4, 1,
0, 1,
>>> 1, 2,
>>> 4, 3, 2, 3, 1, 2, 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1, 3, 2,
1, 2,
>>> 0,
>>> 0, 3, 2, 1, 2, 2, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0, 4, 0,
0, 1,
>>> 1,
>>> 0, 2, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0,
0, 0,
>>> 0,
>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>>> 0,
>>>
>>> But when I run them separately I get the following for gpm only
(same
>>> input files)
>>> 93193, 7954, 4758, 3206, 2364, 1889, 1554, 1287, 1081,
>>> 977, 824, 685, 673, 550, 520, 470, 490, 409, 391, 344, 340,
311,
>>> 278,
>>> 295, 246, 224, 195, 198, 188, 184, 175, 143, 123, 116, 116,
112, 88,
>>> 100,
>>> 111, 88, 82, 74, 69, 63, 62, 53, 48, 40, 49, 46, 36, 29, 32,
48, 41,
>>> 29,
>>> 30, 32, 36, 32, 29, 26, 24, 18, 20, 22, 20, 24, 21, 16, 15,
21, 12,
>>> 20,
>>> 16, 15, 17, 16, 7, 13, 11, 15, 8, 12, 10, 11, 11, 3, 8, 9, 4,
6, 13,
>>> 7,
>>> 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4, 5, 2, 5, 7, 6, 5, 2, 2, 5,
4, 5,
>>> 3,
>>> 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1, 4, 1, 0, 1, 1, 2, 4, 3, 2,
3, 1,
>>> 2,
>>> 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1, 3, 2, 1, 2, 0, 0, 3, 2,
1, 2,
>>> 2,
>>> 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0, 4, 0, 0, 1, 1, 0, 2, 0,
1, 0,
>>> 1,
>>> 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>>> 0,
>>>
>>> and for glosea only
>>> 78311, 10419, 6434, 4737, 4187, 3033, 2511, 2393,
>>> 2263, 1773, 1759, 1034, 1759, 1052, 1112, 703, 868, 490, 383,
341,
>>> 380,
>>> 314, 504, 355, 323, 278, 182, 187, 48, 102, 69, 94, 0, 0, 40,
0, 40,
>>> 0,
>>> 40, 0, 40, 99, 40, 0, 0, 0, 40, 0, 0, 0, 48, 0, 0, 0, 0, 0, 0,
0, 0,
>>> 0,
>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>>> 0,
>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>>> 48,
>>> 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>>> 0, 0,
>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>>> 0,
>>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>>> 0,
>>>
>>> Looking at the first bin 93193+78311 = 171504, bin 2 is
7953+10419=18373
>>> etc. which are the values in bins 1 and 2 for both the marginal
>>> distributions when I input both files together. I am wondering
whether the
>>> attribute changes are not pulled through to this somehow? This is
for the
>>> sample data you've got and using MET9.1. My workaround for now is
that I
>>> will have to run the code 3 times, once for the joint
distribution, which
>>> looks ok, and twice more running it with just the single data
stream to get
>>> the marginals for the two datasets.
>>>
>>> Regards
>>> Marion
>>>
>>> -----Original Message-----
>>> From: John Halley Gotway via RT <met_help at ucar.edu>
>>> Sent: 15 February 2021 17:38
>>> To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
>>> Subject: Re: FW: [rt.rap.ucar.edu #97766] Need help with GridDiag
>>>
>>> This email was received from an external source. Always check
sender
>>> details, links & attachments.
>>>
>>> Marion,
>>>
>>> Thanks for sending sample data. I am a little confused though. The
>>> config file references variables named "precipitation_amount" and
>>> "Precipitation_Amount":
>>> * name = "precipitation_amount";*
>>> * name = "Precipitation_Amount"; *
>>>
>>> But the actual data you sent does not:
>>>
>>>
>>> * ncdump -h glosea_2019060100_d001_001.nc
>>> <http://glosea_2019060100_d001_001.nc> | grep float float
>>> precipitation_amount(latitude, longitude) ;*
>>>
>>> * ncdump -h gpm_2019060100_2019060200.nc
>>> <http://gpm_2019060100_2019060200.nc> | grep float float
>>> precipitation_amount(latitude, longitude) ;*
>>>
>>> I do believe the NetCDF is case-sensitive. So I think that's
important.
>>> But I decided to just try to get Grid-Diag working on the data you
sent. I
>>> did run into an error... different from the one you sent:
>>>
>>>
>>> *terminate called after throwing an instance of
>>> 'netCDF::exceptions::NcNameInUse' what(): NetCDF: String match
to name in
>>> use*
>>>
>>> Grid-Diag reads the first variable from the first "-data" source
and the
>>> second variable from the second "-data" source. And writes data to
the
>>> output NetCDF file. But since those variable names are the same,
there's a
>>> name conflict! It's trying to write the same variable names to the
output
>>> NetCDF file which causes the NetCDF library to error out.
>>>
>>> To get around this, I redefined the input variable names and
levels in
>>> the config file using:
>>> * set_attr_name = "glosea_precip";*
>>>
>>> * set_attr_level = "sfc"; set_attr_name = "gpm_precip";*
>>> * set_attr_level = "sfc";*
>>>
>>> That works fine and produces the attached output file. I also
attached
>>> my run command, config file, and resulting log file.
>>>
>>> Here's some improvements we could make:
>>>
>>> (1) Using set_attr_name and set_attr_level works great. But if you
look
>>> closely at the log messages, the updated strings are NOT used
there. Seems
>>> like they should be. Do you agree?
>>>
>>> (2) The actual data values don't go much over 250... but you've
defined
>>> bins all the way out to 1000. That'll leave you with at least 75%
of the
>>> bins empty. Can you think of any enhancements to make the setting
of the
>>> number and size of bins easier?
>>>
>>> (3) The GPM data appears to use a value of 9.96921E36 as missing
data,
>>> but no _FillValue or missing_value variable attribute defines
that. So MET
>>> processes considers all these pink values to be valid data.
Doesn't matter
>>> in this case because you're only using data from India, which has
all valid
>>> data.
>>> [image: Screen Shot 2021-02-15 at 10.31.31 AM.png] I'd recommend
using
>>> _FillValue or missing_value to define bad data values in the GPM
files:
>>>
>>>
>>> https://cfconventions.org/Data/cf-conventions/cf-conventions-
1.8/cf-conventions.html#missing-data
>>> And do the same for the glosea files as well.
>>>
>>> Thanks,
>>> John
>>>
>>> On Mon, Feb 15, 2021 at 4:43 AM
marion.mittermaier at metoffice.gov.uk via
>>> RT < met_help at ucar.edu> wrote:
>>>
>>> >
>>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>>> >
>>> > Hi all,
>>> >
>>> > in the mean time I have got regrid_data_plane to work in stand-
alone
>>> > model… running the following command (and I’ve tried in both
>>> directions):
>>> >
>>> > regrid_data_plane glosea_2019060100_d001_001.nc
>>> > gpm_2019060100_2019060200.nc grid_diag_out.nc -field
019060200.nc
>>> > grid_diag_out.nc -field ‘name="precipitation_amount";
level="(*,*)";'
>>> > <
>>> >
>>> > DEBUG 1: Reading data file: glosea_2019060100_d001_001.nc DEBUG
2:
>>> > Input grid: Projection: Lat/Lon Nx: 432 Ny: 324 lat_ll: -89.722
>>> > lon_ll: -0.417 delta_lat: 0.556 delta_lon: 0.833 DEBUG 2: Output
grid:
>>> > Projection: Lat/Lon Nx: 3600 Ny: 1800 lat_ll:
>>> > -89.950 lon_ll: 179.950 delta_lat: 0.100 delta_lon: 0.100 DEBUG
2:
>>> > Interpolation options: method = NEAREST, width = 1, shape =
SQUARE,
>>> > vld_thresh = 0.5 DEBUG 2: Range of input data
>>> > (name="precipitation_amount"; level="(*,*)";) is 0 to 262.128.
>>> > DEBUG 2: Range of regridded data (name="precipitation_amount";
>>> > level="(*,*)";) is 0 to 262.128.
>>> > DEBUG 1: Writing output file: grid_diag_out.nc
>>> >
>>> > Which looks sensible… so it’s NOT regrid_data_plane.
>>> >
>>> > Marion
>>> >
>>> > From: Mittermaier, Marion
>>> > Sent: 15 February 2021 11:02
>>> > To: 'met_help at ucar.edu' <met_help at ucar.edu>
>>> > Subject: RE: [rt.rap.ucar.edu #97766] Need help with GridDiag
>>> >
>>> >
>>> > Hi John,
>>> >
>>> >
>>> >
>>> > it's taken me a while to get back to this... I do see the
difference
>>> > but I still don't get it to work. I can get either on their own
to
>>> > work but not together.
>>> >
>>> > I have split out the forecast days into separate files.
>>> >
>>> >
>>> >
>>> > grid_diag -data
>>> > /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc
>>> > -data /scratch/fryl/GPM/gpm_2019060100_2019060200.nc -config
>>> > GridDiag.glosea -out grid_diag_out.nc -v 4
>>> >
>>> >
>>> >
>>> > This gives the following error (I think it's in
regrid_data_plane).
>>> > Not sure why it is trying to regrid both? Or maybe I’m just not
quite
>>> > following the messages….
>>> >
>>> >
>>> >
>>> > vld497:/net/home/h02/frmm/WCSSP/India > grid_diag -data
>>> > /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc -data
>>> > /scratch/fryl/GPM/gpm_2019060100_2019060200.nc
>>> >
>>> > DEBUG 1: Default Config File:
>>> >
/data/users/cfrd/MET_requirements/MET9.0/share/met/config/GridDiagConf
>>> > ig_default
>>> >
>>> > DEBUG 1: User Config File: GridDiag.glosea
>>> >
>>> > DEBUG 2: Read config files.
>>> >
>>> > DEBUG 2: Enter get_mtddf.
>>> >
>>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
>>> > Met2dDataFile object of type "FileType_NcCF".
>>> >
>>> > DEBUG 4: NcCfFile::open() -> parsing units for the time variable
>>> > "hours since 1970-01-01"
>>> >
>>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention
time
>>> > unit string "hours since 1970-01-01" as a reference time of
>>> > 19700101_000000 and
>>> > 3600 second(s) per time step.
>>> >
>>> > DEBUG 4: NcCfFile::open() -> could not extract init time from
the
>>> > "forecast_reference_time" variable.
>>> >
>>> > DEBUG 4: NcCfFile::open() -> could not extract init time from
file
>>> name.
>>> >
>>> > DEBUG 2: Process configuration.
>>> >
>>> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
>>> > of type "FileType_NcCF".
>>> >
>>> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
>>> > of type "FileType_NcCF".
>>> >
>>> > DEBUG 3: Use the grid defined by file
"/scratch/fryl/WCSSPIndia/GPM/
>>> > gpm_2019060100_2019060200.nc".
>>> >
>>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
>>> > Met2dDataFile object of type "FileType_NcCF".
>>> >
>>> > DEBUG 4: NcCfFile::open() -> parsing units for the time variable
>>> > "hours since 1970-01-01"
>>> >
>>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention
time
>>> > unit string "hours since 1970-01-01" as a reference time of
>>> > 19700101_000000 and
>>> > 3600 second(s) per time step.
>>> >
>>> > DEBUG 4: NcCfFile::open() -> could not extract init time from
the
>>> > "forecast_reference_time" variable.
>>> >
>>> > DEBUG 4: NcCfFile::open() -> could not extract init time from
file
>>> name.
>>> >
>>> > DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 360
lat_ll:
>>> > 4.550 lon_ll: -64.550 delta_lat: 0.100 delta_lon: 0.100
>>> >
>>> > DEBUG 2: Processing masking regions.
>>> >
>>> > DEBUG 3: Processing poly mask: India.poly
>>> >
>>> > DEBUG 4: parse_poly_mask() -> parsing poly mask "India.poly"
>>> >
>>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
>>> > Met2dDataFile object of type "FileType_None".
>>> >
>>> > DEBUG 4: parse_poly_mask() -> parsing poly mask file
"India.poly"
>>> >
>>> > DEBUG 2: Initializing precipitation_amount(*,*) histogram
>>> >
>>> > DEBUG 2: Initializing Precipitation_Amount(*,*) histogram
>>> >
>>> > DEBUG 2: Initializing
>>> > precipitation_amount(*,*)_Precipitation_Amount(*,*)
>>> > joint histogram
>>> >
>>> > DEBUG 1: grid_diag_out.nc
>>> >
>>> > DEBUG 1: Length of configuration "data.field" = 2
>>> >
>>> > DEBUG 1: Length of data file list = 1
>>> >
>>> > DEBUG 1: Series defined by the data file list of length 1.
>>> >
>>> > DEBUG 2: 1000
>>> >
>>> > DEBUG 2: Processing series entry 1: precipitation_amount:
>>> > precipitation_amount(*,*)
>>> >
>>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
>>> > Met2dDataFile object of type "FileType_NcCF".
>>> >
>>> > DEBUG 4: NcCfFile::open() -> parsing units for the time variable
>>> > "hours since 1970-01-01"
>>> >
>>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention
time
>>> > unit string "hours since 1970-01-01" as a reference time of
>>> > 19700101_000000 and
>>> > 3600 second(s) per time step.
>>> >
>>> > DEBUG 4: NcCfFile::open() -> could not extract init time from
the
>>> > "forecast_reference_time" variable.
>>> >
>>> > DEBUG 4: NcCfFile::open() -> could not extract init time from
file
>>> name.
>>> >
>>> > DEBUG 4: NcCfFile::getData() -> setting the unset init time to
the
>>> > valid time of 20190601_115959.
>>> >
>>> > DEBUG 4:
>>> >
>>> > DEBUG 4: Data plane information:
>>> >
>>> > DEBUG 4: plane min: 0
>>> >
>>> > DEBUG 4: plane max: 9.96921e+36
>>> >
>>> > DEBUG 4: valid time: 20190601_115959
>>> >
>>> > DEBUG 4: lead time: 000000
>>> >
>>> > DEBUG 4: init time: 20190601_115959
>>> >
>>> > DEBUG 4: accum time: 000000
>>> >
>>> > DEBUG 1: Regridding field precipitation_amount(*,*) to the
>>> > verification grid.
>>> >
>>> > DEBUG 2: Processing series entry 1: Precipitation_Amount:
>>> > Precipitation_Amount(*,*)
>>> >
>>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created
new
>>> > Met2dDataFile object of type "FileType_NcCF".
>>> >
>>> > DEBUG 4: NcCfFile::open() -> parsing units for the time variable
>>> > "hours since 1970-01-01"
>>> >
>>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention
time
>>> > unit string "hours since 1970-01-01" as a reference time of
>>> > 19700101_000000 and
>>> > 3600 second(s) per time step.
>>> >
>>> > DEBUG 4: NcCfFile::open() -> could not extract init time from
the
>>> > "forecast_reference_time" variable.
>>> >
>>> > DEBUG 4: NcCfFile::open() -> could not extract init time from
file
>>> name.
>>> >
>>> > DEBUG 1: Regridding field precipitation_amount(*,*) to the
>>> > verification grid.
>>> >
>>> > ERROR :
>>> >
>>> > ERROR : DataPlane::two_to_one() -> range check error: (Nx, Ny)
= (0,
>>> > 0), (x, y) = (0, 0)
>>> >
>>> > ERROR :
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > -----Original Message-----
>>> > From: John Halley Gotway via RT <met_help at ucar.edu<mailto:
>>> > met_help at ucar.edu>>
>>> > Sent: 08 December 2020 20:54
>>> > To: Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk<mailto:
>>> > marion.mittermaier at metoffice.gov.uk>>
>>> > Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
>>> >
>>> >
>>> >
>>> > This email was received from an external source. Always check
sender
>>> > details, links & attachments.
>>> >
>>> >
>>> >
>>> > Marion,
>>> >
>>> >
>>> >
>>> > I'm still not convinced. Please take a very careful look at the
>>> > command you're executing. I see all the log messages you sent,
but not
>>> > the actual command you ran. At that matters a lot!
>>> >
>>> >
>>> >
>>> > This command fails on my machine with the SAME ERROR you're
getting:
>>> >
>>> > * grid_diag -data glosea_2019060100_001.nc
>>> >
>>> > <http://glosea_2019060100_001.nc/> gpm_2019060100_2019060200.nc
<
>>> > http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea
-out
>>> > grid_diag_out.nc <http://grid_diag_out.nc/>*
>>> >
>>> >
>>> >
>>> > ERROR : NcCfFile::getData(NcVar *, const LongArray &,
DataPlane &)
>>> > const -> needed 3 arguments for variable precipitation_amount,
got 2
>>> >
>>> >
>>> >
>>> > But this command runs WITHOUT ERROR:
>>> >
>>> > * grid_diag -data glosea_2019060100_001.nc
>>> >
>>> > <http://glosea_2019060100_001.nc/> -data
gpm_2019060100_2019060200.nc
>>> > < http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea
-out
>>> > grid_diag_out.nc <http://grid_diag_out.nc/>*
>>> >
>>> >
>>> >
>>> > Do you see the difference?
>>> >
>>> > In the first, I used "-data" once and listed 2 different file
names.
>>> >
>>> > In the second, I used "-data" twice, once for each of the input
files.
>>> >
>>> >
>>> >
>>> > Please try doing exactly the same thing.
>>> >
>>> >
>>> >
>>> > Does that enable it to run without error? Note that you'll still
get
>>> > warnings about the actual data values falling outside the
defined
>>> > range of the bins.
>>> >
>>> >
>>> >
>>> > As for the bin endpoints, I think this is the relevant line of
code:
>>> >
>>> >
>>> >
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc
>>> > 5981bb8/met/src/libcode/vx_series_data/series_pdf.cc#L72
>>> >
>>> >
>>> >
>>> > int k = floor((value - min) / delta);
>>> >
>>> >
>>> >
>>> > This places the current "value" into the k-th bin where "min" is
the
>>> > minimum PDF value and "delta" is the bin width.
>>> >
>>> > I believe this is consistent with bins being closed on the left
side
>>> > and open on the right:
>>> >
>>> > [0,1) i.e. 0 <= x < 1
>>> >
>>> >
>>> >
>>> > John
>>> >
>>> >
>>> >
>>> > On Tue, Dec 8, 2020 at 8:30 AM
marion.mittermaier at metoffice.gov.uk
>>> <mailto:
>>> > marion.mittermaier at metoffice.gov.uk> via RT < met_help at ucar.edu
>>> <mailto:
>>> > met_help at ucar.edu>> wrote:
>>> >
>>> >
>>> >
>>> > >
>>> >
>>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
>
>>> >
>>> > >
>>> >
>>> > > Hi John,
>>> >
>>> > >
>>> >
>>> > > one final question for GridDiag.
>>> >
>>> > >
>>> >
>>> > > The bins are set up to be lower bound inclusive? i.e.
>>> >
>>> > > n_bins = 1000;
>>> >
>>> > > range = [0, 1000]; // kg/m^2
>>> >
>>> > >
>>> >
>>> > > means that the bin is set up as [0,1) i.e. 0 <= x < 1?
>>> >
>>> > >
>>> >
>>> > > I'm presuming so.... and the upper bound would be [999,1000)
with no
>>> >
>>> > > bin for [1000,inf)? i.e. if there were a 1000 in my sample it
would
>>> >
>>> > > NOT be counted? Or anything bigger.
>>> >
>>> > >
>>> >
>>> > > That's what it looks like, i.e. you need to make sure that
your
>>> > > range
>>> >
>>> > > encompasses all possible values. Not sure this is in the
>>> > documentation...?
>>> >
>>> > > Would be useful to add.
>>> >
>>> > >
>>> >
>>> > > Ta
>>> >
>>> > > Marion
>>> >
>>> > >
>>> >
>>> > > -----Original Message-----
>>> >
>>> > > From: John Halley Gotway via RT <met_help at ucar.edu<mailto:
>>> > met_help at ucar.edu>>
>>> >
>>> > > Sent: 05 December 2020 06:41
>>> >
>>> > > To: Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk<mailto:
>>> > marion.mittermaier at metoffice.gov.uk>>
>>> >
>>> > > Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
>>> >
>>> > >
>>> >
>>> > > This email was received from an external source. Always
check
>>> sender
>>> >
>>> > > details, links & attachments.
>>> >
>>> > >
>>> >
>>> > > Marion,
>>> >
>>> > >
>>> >
>>> > > Ugh, I should've just handed this ticket off to David in the
first
>>> place.
>>> >
>>> > > I did more testing and was able to get it to work. I just had
to use
>>> >
>>> > > separate "-data" command line options for each input data
source.
>>> >
>>> > >
>>> >
>>> > > So this command fails:
>>> >
>>> > > grid_diag -data glosea_2019060100_001.nc
>>> > > gpm_2019060100_2019060200.nc
>>> >
>>> > > -config GridDiag.glosea -out grid_diag_out.nc
>>> >
>>> > >
>>> >
>>> > > But this command runs fine:
>>> >
>>> > > *grid_diag -data glosea_2019060100_001.nc <
>>> >
>>> > > http://glosea_2019060100_001.nc> -data
gpm_2019060100_2019060200.nc
>>> > > <
>>> >
>>> > > http://gpm_2019060100_2019060200.nc> -config GridDiag.glosea
-out
>>> >
>>> > > grid_diag_out.nc <http://grid_diag_out.nc>*
>>> >
>>> > >
>>> >
>>> > > In the second one, I've specified "-data" separately for each
data
>>> > source.
>>> >
>>> > > Although, you'll see warning about the range of the data
extending
>>> >
>>> > > beyond the range of the bins defined in the config file.
>>> >
>>> > >
>>> >
>>> > > Sorry to have such a rambling and confusing response!
>>> >
>>> > >
>>> >
>>> > > So to summarize, I believe the logic is:
>>> >
>>> > > 1. If "-data" is used once, all specified input fields are
read from
>>> >
>>> > > all of those data files.
>>> >
>>> > > 2. If "-data" is used more than once, the number of "-data"
entries
>>> >
>>> > > must match the number of fields defined in the config file.
>>> >
>>> > > If not, you get an error message.
>>> >
>>> > > If so, only the i-th field is read from the files in the
i-th
>>> >
>>> > > -data source.
>>> >
>>> > >
>>> >
>>> > > John
>>> >
>>> > >
>>> >
>>> > > On Fri, Dec 4, 2020 at 11:30 PM John Halley Gotway
<johnhg at ucar.edu
>>> > <mailto:johnhg at ucar.edu>>
>>> >
>>> > > wrote:
>>> >
>>> > >
>>> >
>>> > > > Marion,
>>> >
>>> > > >
>>> >
>>> > > > It occurs to me that perhaps Grid-Diag could be made more
flexible
>>> >
>>> > > > if we changed that error message to a debug message in this
>>> context.
>>> >
>>> > > > It could
>>> >
>>> > > > *ATTEMPT* to read all input fields from all input files...
but
>>> > > > when
>>> >
>>> > > > it's unable to, have it print a log message stating that
rather
>>> > > > than
>>> >
>>> > > > erroring out.
>>> >
>>> > > >
>>> >
>>> > > > That would enable you to compute 2D pdf's using different
>>> > > > variables
>>> >
>>> > > > from different input files.
>>> >
>>> > > >
>>> >
>>> > > > Do you think that logic is preferable?
>>> >
>>> > > >
>>> >
>>> > > > Thanks,
>>> >
>>> > > > John
>>> >
>>> > > >
>>> >
>>> > > > On Fri, Dec 4, 2020 at 11:27 PM The RT System itself via RT
<
>>> >
>>> > > > met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
>>> >
>>> > > >
>>> >
>>> > > >>
>>> >
>>> > > >> Fri Dec 04 23:27:19 2020: Request 97766 was acted upon.
>>> >
>>> > > >> Transaction: Given to johnhg (John Halley Gotway) by
RT_System
>>> >
>>> > > >> Queue: met_help
>>> >
>>> > > >> Subject: Need help with GridDiag
>>> >
>>> > > >> Owner: johnhg
>>> >
>>> > > >> Requestors: marion.mittermaier at metoffice.gov.uk<mailto:
>>> > marion.mittermaier at metoffice.gov.uk>
>>> >
>>> > > >> Status: new
>>> >
>>> > > >> Ticket <URL:
>>> >
>>> > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
>>> >
>>> > > >> >
>>> >
>>> > > >>
>>> >
>>> > > >>
>>> >
>>> > > >> This transaction appears to have no content
>>> >
>>> > > >>
>>> >
>>> > > >
>>> >
>>> > >
>>> >
>>> > >
>>> >
>>> > >
>>> >
>>> > >
>>> >
>>> >
>>> >
>>> >
>>>
>>>
>>>
>>>
------------------------------------------------
Subject: Need help with GridDiag
From: David Fillmore
Time: Thu Mar 04 10:43:38 2021
The description of this issue is correct.
The histograms are keyed by variable name, so one can not have a
duplicate
name from a second dataset.
This was not anticipated and should be remedied.
David
On Thu, Mar 4, 2021 at 10:37 AM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>
> Marion,
>
> Here's the GitHub issue I wrote up to describe this bug:
> https://github.com/dtcenter/MET/issues/1694
>
> John
>
> On Thu, Mar 4, 2021 at 9:45 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
>
> > Marion,
> >
> > FYI, I'm pretty confident the bug is caused by the fact that we're
> reading
> > the same variable named "precipitation_amount" from two different
input
> > data sources. When I change the variable name of the second one to
> > "precipitation_amount_MOD" then the problem goes away. Then 1D
histograms
> > are then correct.
> >
> > So that would provide a short-term, temporary bandaid for now. You
could
> > nc_rename precipitaiton_amount to some other (unique) name prior
to
> running
> > grid_diag with met-9.1.
> >
> > We should just need to update the code to make the string we're
using to
> > track the counts unique. And that seems very doable for met-
10.0.0.
> >
> > Thanks,
> > John
> >
> > On Thu, Mar 4, 2021 at 9:30 AM John Halley Gotway
<johnhg at ucar.edu>
> wrote:
> >
> >> Marion,
> >>
> >> You're right. I looked closely at the numbers that grid_diag is
> producing
> >> and this is pretty obviously a bug. Thanks for finding this and
letting
> us
> >> know!
> >>
> >> I'll coordinate with David Fillmore, the engineer who originally
worked
> >> on this to get this fixed for the met-10.0.0 release.
> >>
> >> John
> >>
> >> On Thu, Mar 4, 2021 at 2:29 AM
marion.mittermaier at metoffice.gov.uk via
> >> RT <met_help at ucar.edu> wrote:
> >>
> >>>
> >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
> >>>
> >>> Hi John,
> >>>
> >>> I thought I had cracked it but something is still not quite
right...
> the
> >>> joint distribution does appear when using the dual -data
commands with
> the
> >>> changed attribute names, and it looks plausible BUT but the
marginal
> >>> distributions are wrong.
> >>>
> >>> If I run the GridDiag code with both data sets then the marginal
> >>> distributions are the same, which isn't right.
> >>> running together
> >>> glosea 171504, 18373, 11192, 7943, 6551, 4922, 4065, 3680,
> >>> 3344, 2750, 2583, 1719, 2432, 1602, 1632, 1173, 1358, 899,
774,
> 685,
> >>> 720,
> >>> 625, 782, 650, 569, 502, 377, 385, 236, 286, 244, 237, 123,
116,
> >>> 156,
> >>> 112, 128, 100, 151, 88, 122, 173, 109, 63, 62, 53, 88, 40,
49, 46,
> >>> 84,
> >>> 29, 32, 48, 41, 29, 30, 32, 36, 32, 29, 26, 24, 18, 20, 22,
20, 24,
> >>> 21,
> >>> 16, 15, 21, 12, 20, 16, 15, 17, 16, 7, 13, 11, 15, 8, 12,
10, 11,
> >>> 11, 3,
> >>> 8, 9, 4, 6, 13, 7, 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4, 5,
50, 5, 7,
> >>> 6, 5,
> >>> 2, 2, 53, 4, 5, 3, 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1, 4, 1,
0, 1,
> >>> 1, 2,
> >>> 4, 3, 2, 3, 1, 2, 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1, 3,
2, 1,
> 2,
> >>> 0,
> >>> 0, 3, 2, 1, 2, 2, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0, 4,
0, 0,
> 1,
> >>> 1,
> >>> 0, 2, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0,
0, 0,
> 0,
> >>> 0,
> >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
> 0,
> >>> 0,
> >>> gpm 171504, 18373, 11192, 7943, 6551, 4922, 4065, 3680,
> >>> 3344, 2750, 2583, 1719, 2432, 1602, 1632, 1173, 1358, 899,
774,
> 685,
> >>> 720,
> >>> 625, 782, 650, 569, 502, 377, 385, 236, 286, 244, 237, 123,
116,
> >>> 156,
> >>> 112, 128, 100, 151, 88, 122, 173, 109, 63, 62, 53, 88, 40,
49, 46,
> >>> 84,
> >>> 29, 32, 48, 41, 29, 30, 32, 36, 32, 29, 26, 24, 18, 20, 22,
20, 24,
> >>> 21,
> >>> 16, 15, 21, 12, 20, 16, 15, 17, 16, 7, 13, 11, 15, 8, 12,
10, 11,
> >>> 11, 3,
> >>> 8, 9, 4, 6, 13, 7, 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4, 5,
50, 5, 7,
> >>> 6, 5,
> >>> 2, 2, 53, 4, 5, 3, 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1, 4, 1,
0, 1,
> >>> 1, 2,
> >>> 4, 3, 2, 3, 1, 2, 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1, 3,
2, 1,
> 2,
> >>> 0,
> >>> 0, 3, 2, 1, 2, 2, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0, 4,
0, 0,
> 1,
> >>> 1,
> >>> 0, 2, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0,
0, 0,
> 0,
> >>> 0,
> >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
> 0,
> >>> 0,
> >>>
> >>> But when I run them separately I get the following for gpm only
(same
> >>> input files)
> >>> 93193, 7954, 4758, 3206, 2364, 1889, 1554, 1287, 1081,
> >>> 977, 824, 685, 673, 550, 520, 470, 490, 409, 391, 344, 340,
311,
> >>> 278,
> >>> 295, 246, 224, 195, 198, 188, 184, 175, 143, 123, 116, 116,
112,
> 88,
> >>> 100,
> >>> 111, 88, 82, 74, 69, 63, 62, 53, 48, 40, 49, 46, 36, 29, 32,
48,
> 41,
> >>> 29,
> >>> 30, 32, 36, 32, 29, 26, 24, 18, 20, 22, 20, 24, 21, 16, 15,
21, 12,
> >>> 20,
> >>> 16, 15, 17, 16, 7, 13, 11, 15, 8, 12, 10, 11, 11, 3, 8, 9,
4, 6,
> 13,
> >>> 7,
> >>> 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4, 5, 2, 5, 7, 6, 5, 2, 2,
5, 4,
> 5,
> >>> 3,
> >>> 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1, 4, 1, 0, 1, 1, 2, 4, 3,
2, 3,
> 1,
> >>> 2,
> >>> 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1, 3, 2, 1, 2, 0, 0, 3,
2, 1,
> 2,
> >>> 2,
> >>> 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0, 4, 0, 0, 1, 1, 0, 2,
0, 1,
> 0,
> >>> 1,
> >>> 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
> 0,
> >>> 0,
> >>>
> >>> and for glosea only
> >>> 78311, 10419, 6434, 4737, 4187, 3033, 2511, 2393,
> >>> 2263, 1773, 1759, 1034, 1759, 1052, 1112, 703, 868, 490,
383, 341,
> >>> 380,
> >>> 314, 504, 355, 323, 278, 182, 187, 48, 102, 69, 94, 0, 0,
40, 0,
> 40,
> >>> 0,
> >>> 40, 0, 40, 99, 40, 0, 0, 0, 40, 0, 0, 0, 48, 0, 0, 0, 0, 0,
0, 0,
> 0,
> >>> 0,
> >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
> 0,
> >>> 0,
> >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
> 0,
> >>> 48,
> >>> 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
> >>> 0, 0,
> >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
> 0,
> >>> 0,
> >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
> 0,
> >>> 0,
> >>>
> >>> Looking at the first bin 93193+78311 = 171504, bin 2 is
> 7953+10419=18373
> >>> etc. which are the values in bins 1 and 2 for both the marginal
> >>> distributions when I input both files together. I am wondering
whether
> the
> >>> attribute changes are not pulled through to this somehow? This
is for
> the
> >>> sample data you've got and using MET9.1. My workaround for now
is that
> I
> >>> will have to run the code 3 times, once for the joint
distribution,
> which
> >>> looks ok, and twice more running it with just the single data
stream
> to get
> >>> the marginals for the two datasets.
> >>>
> >>> Regards
> >>> Marion
> >>>
> >>> -----Original Message-----
> >>> From: John Halley Gotway via RT <met_help at ucar.edu>
> >>> Sent: 15 February 2021 17:38
> >>> To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> >>> Subject: Re: FW: [rt.rap.ucar.edu #97766] Need help with
GridDiag
> >>>
> >>> This email was received from an external source. Always check
sender
> >>> details, links & attachments.
> >>>
> >>> Marion,
> >>>
> >>> Thanks for sending sample data. I am a little confused though.
The
> >>> config file references variables named "precipitation_amount"
and
> >>> "Precipitation_Amount":
> >>> * name = "precipitation_amount";*
> >>> * name = "Precipitation_Amount"; *
> >>>
> >>> But the actual data you sent does not:
> >>>
> >>>
> >>> * ncdump -h glosea_2019060100_d001_001.nc
> >>> <http://glosea_2019060100_d001_001.nc> | grep float float
> >>> precipitation_amount(latitude, longitude) ;*
> >>>
> >>> * ncdump -h gpm_2019060100_2019060200.nc
> >>> <http://gpm_2019060100_2019060200.nc> | grep float float
> >>> precipitation_amount(latitude, longitude) ;*
> >>>
> >>> I do believe the NetCDF is case-sensitive. So I think that's
important.
> >>> But I decided to just try to get Grid-Diag working on the data
you
> sent. I
> >>> did run into an error... different from the one you sent:
> >>>
> >>>
> >>> *terminate called after throwing an instance of
> >>> 'netCDF::exceptions::NcNameInUse' what(): NetCDF: String match
to
> name in
> >>> use*
> >>>
> >>> Grid-Diag reads the first variable from the first "-data" source
and
> the
> >>> second variable from the second "-data" source. And writes data
to the
> >>> output NetCDF file. But since those variable names are the same,
> there's a
> >>> name conflict! It's trying to write the same variable names to
the
> output
> >>> NetCDF file which causes the NetCDF library to error out.
> >>>
> >>> To get around this, I redefined the input variable names and
levels in
> >>> the config file using:
> >>> * set_attr_name = "glosea_precip";*
> >>>
> >>> * set_attr_level = "sfc"; set_attr_name = "gpm_precip";*
> >>> * set_attr_level = "sfc";*
> >>>
> >>> That works fine and produces the attached output file. I also
attached
> >>> my run command, config file, and resulting log file.
> >>>
> >>> Here's some improvements we could make:
> >>>
> >>> (1) Using set_attr_name and set_attr_level works great. But if
you look
> >>> closely at the log messages, the updated strings are NOT used
there.
> Seems
> >>> like they should be. Do you agree?
> >>>
> >>> (2) The actual data values don't go much over 250... but you've
defined
> >>> bins all the way out to 1000. That'll leave you with at least
75% of
> the
> >>> bins empty. Can you think of any enhancements to make the
setting of
> the
> >>> number and size of bins easier?
> >>>
> >>> (3) The GPM data appears to use a value of 9.96921E36 as missing
data,
> >>> but no _FillValue or missing_value variable attribute defines
that. So
> MET
> >>> processes considers all these pink values to be valid data.
Doesn't
> matter
> >>> in this case because you're only using data from India, which
has all
> valid
> >>> data.
> >>> [image: Screen Shot 2021-02-15 at 10.31.31 AM.png] I'd recommend
using
> >>> _FillValue or missing_value to define bad data values in the GPM
files:
> >>>
> >>>
> >>>
> https://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-
conventions.html#missing-data
> >>> And do the same for the glosea files as well.
> >>>
> >>> Thanks,
> >>> John
> >>>
> >>> On Mon, Feb 15, 2021 at 4:43 AM
marion.mittermaier at metoffice.gov.uk
> via
> >>> RT < met_help at ucar.edu> wrote:
> >>>
> >>> >
> >>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
>
> >>> >
> >>> > Hi all,
> >>> >
> >>> > in the mean time I have got regrid_data_plane to work in
stand-alone
> >>> > model… running the following command (and I’ve tried in both
> >>> directions):
> >>> >
> >>> > regrid_data_plane glosea_2019060100_d001_001.nc
> >>> > gpm_2019060100_2019060200.nc grid_diag_out.nc -field
019060200.nc
> >>> > grid_diag_out.nc -field ‘name="precipitation_amount";
> level="(*,*)";'
> >>> > <
> >>> >
> >>> > DEBUG 1: Reading data file: glosea_2019060100_d001_001.nc
DEBUG 2:
> >>> > Input grid: Projection: Lat/Lon Nx: 432 Ny: 324 lat_ll:
-89.722
> >>> > lon_ll: -0.417 delta_lat: 0.556 delta_lon: 0.833 DEBUG 2:
Output
> grid:
> >>> > Projection: Lat/Lon Nx: 3600 Ny: 1800 lat_ll:
> >>> > -89.950 lon_ll: 179.950 delta_lat: 0.100 delta_lon: 0.100
DEBUG 2:
> >>> > Interpolation options: method = NEAREST, width = 1, shape =
SQUARE,
> >>> > vld_thresh = 0.5 DEBUG 2: Range of input data
> >>> > (name="precipitation_amount"; level="(*,*)";) is 0 to 262.128.
> >>> > DEBUG 2: Range of regridded data (name="precipitation_amount";
> >>> > level="(*,*)";) is 0 to 262.128.
> >>> > DEBUG 1: Writing output file: grid_diag_out.nc
> >>> >
> >>> > Which looks sensible… so it’s NOT regrid_data_plane.
> >>> >
> >>> > Marion
> >>> >
> >>> > From: Mittermaier, Marion
> >>> > Sent: 15 February 2021 11:02
> >>> > To: 'met_help at ucar.edu' <met_help at ucar.edu>
> >>> > Subject: RE: [rt.rap.ucar.edu #97766] Need help with GridDiag
> >>> >
> >>> >
> >>> > Hi John,
> >>> >
> >>> >
> >>> >
> >>> > it's taken me a while to get back to this... I do see the
difference
> >>> > but I still don't get it to work. I can get either on their
own to
> >>> > work but not together.
> >>> >
> >>> > I have split out the forecast days into separate files.
> >>> >
> >>> >
> >>> >
> >>> > grid_diag -data
> >>> > /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc
> >>> > -data /scratch/fryl/GPM/gpm_2019060100_2019060200.nc -config
> >>> > GridDiag.glosea -out grid_diag_out.nc -v 4
> >>> >
> >>> >
> >>> >
> >>> > This gives the following error (I think it's in
regrid_data_plane).
> >>> > Not sure why it is trying to regrid both? Or maybe I’m just
not quite
> >>> > following the messages….
> >>> >
> >>> >
> >>> >
> >>> > vld497:/net/home/h02/frmm/WCSSP/India > grid_diag -data
> >>> > /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc
-data
> >>> > /scratch/fryl/GPM/gpm_2019060100_2019060200.nc
> >>> >
> >>> > DEBUG 1: Default Config File:
> >>> >
>
/data/users/cfrd/MET_requirements/MET9.0/share/met/config/GridDiagConf
> >>> > ig_default
> >>> >
> >>> > DEBUG 1: User Config File: GridDiag.glosea
> >>> >
> >>> > DEBUG 2: Read config files.
> >>> >
> >>> > DEBUG 2: Enter get_mtddf.
> >>> >
> >>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created new
> >>> > Met2dDataFile object of type "FileType_NcCF".
> >>> >
> >>> > DEBUG 4: NcCfFile::open() -> parsing units for the time
variable
> >>> > "hours since 1970-01-01"
> >>> >
> >>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention
time
> >>> > unit string "hours since 1970-01-01" as a reference time of
> >>> > 19700101_000000 and
> >>> > 3600 second(s) per time step.
> >>> >
> >>> > DEBUG 4: NcCfFile::open() -> could not extract init time from
the
> >>> > "forecast_reference_time" variable.
> >>> >
> >>> > DEBUG 4: NcCfFile::open() -> could not extract init time from
file
> >>> name.
> >>> >
> >>> > DEBUG 2: Process configuration.
> >>> >
> >>> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> >>> > of type "FileType_NcCF".
> >>> >
> >>> > DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo
object
> >>> > of type "FileType_NcCF".
> >>> >
> >>> > DEBUG 3: Use the grid defined by file
"/scratch/fryl/WCSSPIndia/GPM/
> >>> > gpm_2019060100_2019060200.nc".
> >>> >
> >>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created new
> >>> > Met2dDataFile object of type "FileType_NcCF".
> >>> >
> >>> > DEBUG 4: NcCfFile::open() -> parsing units for the time
variable
> >>> > "hours since 1970-01-01"
> >>> >
> >>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention
time
> >>> > unit string "hours since 1970-01-01" as a reference time of
> >>> > 19700101_000000 and
> >>> > 3600 second(s) per time step.
> >>> >
> >>> > DEBUG 4: NcCfFile::open() -> could not extract init time from
the
> >>> > "forecast_reference_time" variable.
> >>> >
> >>> > DEBUG 4: NcCfFile::open() -> could not extract init time from
file
> >>> name.
> >>> >
> >>> > DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny: 360
lat_ll:
> >>> > 4.550 lon_ll: -64.550 delta_lat: 0.100 delta_lon: 0.100
> >>> >
> >>> > DEBUG 2: Processing masking regions.
> >>> >
> >>> > DEBUG 3: Processing poly mask: India.poly
> >>> >
> >>> > DEBUG 4: parse_poly_mask() -> parsing poly mask "India.poly"
> >>> >
> >>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created new
> >>> > Met2dDataFile object of type "FileType_None".
> >>> >
> >>> > DEBUG 4: parse_poly_mask() -> parsing poly mask file
"India.poly"
> >>> >
> >>> > DEBUG 2: Initializing precipitation_amount(*,*) histogram
> >>> >
> >>> > DEBUG 2: Initializing Precipitation_Amount(*,*) histogram
> >>> >
> >>> > DEBUG 2: Initializing
> >>> > precipitation_amount(*,*)_Precipitation_Amount(*,*)
> >>> > joint histogram
> >>> >
> >>> > DEBUG 1: grid_diag_out.nc
> >>> >
> >>> > DEBUG 1: Length of configuration "data.field" = 2
> >>> >
> >>> > DEBUG 1: Length of data file list = 1
> >>> >
> >>> > DEBUG 1: Series defined by the data file list of length 1.
> >>> >
> >>> > DEBUG 2: 1000
> >>> >
> >>> > DEBUG 2: Processing series entry 1: precipitation_amount:
> >>> > precipitation_amount(*,*)
> >>> >
> >>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created new
> >>> > Met2dDataFile object of type "FileType_NcCF".
> >>> >
> >>> > DEBUG 4: NcCfFile::open() -> parsing units for the time
variable
> >>> > "hours since 1970-01-01"
> >>> >
> >>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention
time
> >>> > unit string "hours since 1970-01-01" as a reference time of
> >>> > 19700101_000000 and
> >>> > 3600 second(s) per time step.
> >>> >
> >>> > DEBUG 4: NcCfFile::open() -> could not extract init time from
the
> >>> > "forecast_reference_time" variable.
> >>> >
> >>> > DEBUG 4: NcCfFile::open() -> could not extract init time from
file
> >>> name.
> >>> >
> >>> > DEBUG 4: NcCfFile::getData() -> setting the unset init time to
the
> >>> > valid time of 20190601_115959.
> >>> >
> >>> > DEBUG 4:
> >>> >
> >>> > DEBUG 4: Data plane information:
> >>> >
> >>> > DEBUG 4: plane min: 0
> >>> >
> >>> > DEBUG 4: plane max: 9.96921e+36
> >>> >
> >>> > DEBUG 4: valid time: 20190601_115959
> >>> >
> >>> > DEBUG 4: lead time: 000000
> >>> >
> >>> > DEBUG 4: init time: 20190601_115959
> >>> >
> >>> > DEBUG 4: accum time: 000000
> >>> >
> >>> > DEBUG 1: Regridding field precipitation_amount(*,*) to the
> >>> > verification grid.
> >>> >
> >>> > DEBUG 2: Processing series entry 1: Precipitation_Amount:
> >>> > Precipitation_Amount(*,*)
> >>> >
> >>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created new
> >>> > Met2dDataFile object of type "FileType_NcCF".
> >>> >
> >>> > DEBUG 4: NcCfFile::open() -> parsing units for the time
variable
> >>> > "hours since 1970-01-01"
> >>> >
> >>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention
time
> >>> > unit string "hours since 1970-01-01" as a reference time of
> >>> > 19700101_000000 and
> >>> > 3600 second(s) per time step.
> >>> >
> >>> > DEBUG 4: NcCfFile::open() -> could not extract init time from
the
> >>> > "forecast_reference_time" variable.
> >>> >
> >>> > DEBUG 4: NcCfFile::open() -> could not extract init time from
file
> >>> name.
> >>> >
> >>> > DEBUG 1: Regridding field precipitation_amount(*,*) to the
> >>> > verification grid.
> >>> >
> >>> > ERROR :
> >>> >
> >>> > ERROR : DataPlane::two_to_one() -> range check error: (Nx,
Ny) = (0,
> >>> > 0), (x, y) = (0, 0)
> >>> >
> >>> > ERROR :
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > -----Original Message-----
> >>> > From: John Halley Gotway via RT <met_help at ucar.edu<mailto:
> >>> > met_help at ucar.edu>>
> >>> > Sent: 08 December 2020 20:54
> >>> > To: Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk<mailto:
> >>> > marion.mittermaier at metoffice.gov.uk>>
> >>> > Subject: Re: [rt.rap.ucar.edu #97766] Need help with GridDiag
> >>> >
> >>> >
> >>> >
> >>> > This email was received from an external source. Always
check
> sender
> >>> > details, links & attachments.
> >>> >
> >>> >
> >>> >
> >>> > Marion,
> >>> >
> >>> >
> >>> >
> >>> > I'm still not convinced. Please take a very careful look at
the
> >>> > command you're executing. I see all the log messages you sent,
but
> not
> >>> > the actual command you ran. At that matters a lot!
> >>> >
> >>> >
> >>> >
> >>> > This command fails on my machine with the SAME ERROR you're
getting:
> >>> >
> >>> > * grid_diag -data glosea_2019060100_001.nc
> >>> >
> >>> > <http://glosea_2019060100_001.nc/>
gpm_2019060100_2019060200.nc <
> >>> > http://gpm_2019060100_2019060200.nc/> -config GridDiag.glosea
-out
> >>> > grid_diag_out.nc <http://grid_diag_out.nc/>*
> >>> >
> >>> >
> >>> >
> >>> > ERROR : NcCfFile::getData(NcVar *, const LongArray &,
DataPlane
> &)
> >>> > const -> needed 3 arguments for variable precipitation_amount,
got 2
> >>> >
> >>> >
> >>> >
> >>> > But this command runs WITHOUT ERROR:
> >>> >
> >>> > * grid_diag -data glosea_2019060100_001.nc
> >>> >
> >>> > <http://glosea_2019060100_001.nc/> -data
> gpm_2019060100_2019060200.nc
> >>> > < http://gpm_2019060100_2019060200.nc/> -config
GridDiag.glosea -out
> >>> > grid_diag_out.nc <http://grid_diag_out.nc/>*
> >>> >
> >>> >
> >>> >
> >>> > Do you see the difference?
> >>> >
> >>> > In the first, I used "-data" once and listed 2 different file
names.
> >>> >
> >>> > In the second, I used "-data" twice, once for each of the
input
> files.
> >>> >
> >>> >
> >>> >
> >>> > Please try doing exactly the same thing.
> >>> >
> >>> >
> >>> >
> >>> > Does that enable it to run without error? Note that you'll
still get
> >>> > warnings about the actual data values falling outside the
defined
> >>> > range of the bins.
> >>> >
> >>> >
> >>> >
> >>> > As for the bin endpoints, I think this is the relevant line of
code:
> >>> >
> >>> >
> >>> >
>
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc
> >>> > 5981bb8/met/src/libcode/vx_series_data/series_pdf.cc#L72
> >>> >
> >>> >
> >>> >
> >>> > int k = floor((value - min) / delta);
> >>> >
> >>> >
> >>> >
> >>> > This places the current "value" into the k-th bin where "min"
is the
> >>> > minimum PDF value and "delta" is the bin width.
> >>> >
> >>> > I believe this is consistent with bins being closed on the
left side
> >>> > and open on the right:
> >>> >
> >>> > [0,1) i.e. 0 <= x < 1
> >>> >
> >>> >
> >>> >
> >>> > John
> >>> >
> >>> >
> >>> >
> >>> > On Tue, Dec 8, 2020 at 8:30 AM
marion.mittermaier at metoffice.gov.uk
> >>> <mailto:
> >>> > marion.mittermaier at metoffice.gov.uk> via RT <
met_help at ucar.edu
> >>> <mailto:
> >>> > met_help at ucar.edu>> wrote:
> >>> >
> >>> >
> >>> >
> >>> > >
> >>> >
> >>> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
> >>> >
> >>> > >
> >>> >
> >>> > > Hi John,
> >>> >
> >>> > >
> >>> >
> >>> > > one final question for GridDiag.
> >>> >
> >>> > >
> >>> >
> >>> > > The bins are set up to be lower bound inclusive? i.e.
> >>> >
> >>> > > n_bins = 1000;
> >>> >
> >>> > > range = [0, 1000]; // kg/m^2
> >>> >
> >>> > >
> >>> >
> >>> > > means that the bin is set up as [0,1) i.e. 0 <= x < 1?
> >>> >
> >>> > >
> >>> >
> >>> > > I'm presuming so.... and the upper bound would be [999,1000)
with
> no
> >>> >
> >>> > > bin for [1000,inf)? i.e. if there were a 1000 in my sample
it would
> >>> >
> >>> > > NOT be counted? Or anything bigger.
> >>> >
> >>> > >
> >>> >
> >>> > > That's what it looks like, i.e. you need to make sure that
your
> >>> > > range
> >>> >
> >>> > > encompasses all possible values. Not sure this is in the
> >>> > documentation...?
> >>> >
> >>> > > Would be useful to add.
> >>> >
> >>> > >
> >>> >
> >>> > > Ta
> >>> >
> >>> > > Marion
> >>> >
> >>> > >
> >>> >
> >>> > > -----Original Message-----
> >>> >
> >>> > > From: John Halley Gotway via RT <met_help at ucar.edu<mailto:
> >>> > met_help at ucar.edu>>
> >>> >
> >>> > > Sent: 05 December 2020 06:41
> >>> >
> >>> > > To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk
> <mailto:
> >>> > marion.mittermaier at metoffice.gov.uk>>
> >>> >
> >>> > > Subject: Re: [rt.rap.ucar.edu #97766] Need help with
GridDiag
> >>> >
> >>> > >
> >>> >
> >>> > > This email was received from an external source. Always
check
> >>> sender
> >>> >
> >>> > > details, links & attachments.
> >>> >
> >>> > >
> >>> >
> >>> > > Marion,
> >>> >
> >>> > >
> >>> >
> >>> > > Ugh, I should've just handed this ticket off to David in the
first
> >>> place.
> >>> >
> >>> > > I did more testing and was able to get it to work. I just
had to
> use
> >>> >
> >>> > > separate "-data" command line options for each input data
source.
> >>> >
> >>> > >
> >>> >
> >>> > > So this command fails:
> >>> >
> >>> > > grid_diag -data glosea_2019060100_001.nc
> >>> > > gpm_2019060100_2019060200.nc
> >>> >
> >>> > > -config GridDiag.glosea -out grid_diag_out.nc
> >>> >
> >>> > >
> >>> >
> >>> > > But this command runs fine:
> >>> >
> >>> > > *grid_diag -data glosea_2019060100_001.nc <
> >>> >
> >>> > > http://glosea_2019060100_001.nc> -data
> gpm_2019060100_2019060200.nc
> >>> > > <
> >>> >
> >>> > > http://gpm_2019060100_2019060200.nc> -config GridDiag.glosea
-out
> >>> >
> >>> > > grid_diag_out.nc <http://grid_diag_out.nc>*
> >>> >
> >>> > >
> >>> >
> >>> > > In the second one, I've specified "-data" separately for
each data
> >>> > source.
> >>> >
> >>> > > Although, you'll see warning about the range of the data
extending
> >>> >
> >>> > > beyond the range of the bins defined in the config file.
> >>> >
> >>> > >
> >>> >
> >>> > > Sorry to have such a rambling and confusing response!
> >>> >
> >>> > >
> >>> >
> >>> > > So to summarize, I believe the logic is:
> >>> >
> >>> > > 1. If "-data" is used once, all specified input fields are
read
> from
> >>> >
> >>> > > all of those data files.
> >>> >
> >>> > > 2. If "-data" is used more than once, the number of "-data"
entries
> >>> >
> >>> > > must match the number of fields defined in the config file.
> >>> >
> >>> > > If not, you get an error message.
> >>> >
> >>> > > If so, only the i-th field is read from the files in the
i-th
> >>> >
> >>> > > -data source.
> >>> >
> >>> > >
> >>> >
> >>> > > John
> >>> >
> >>> > >
> >>> >
> >>> > > On Fri, Dec 4, 2020 at 11:30 PM John Halley Gotway <
> johnhg at ucar.edu
> >>> > <mailto:johnhg at ucar.edu>>
> >>> >
> >>> > > wrote:
> >>> >
> >>> > >
> >>> >
> >>> > > > Marion,
> >>> >
> >>> > > >
> >>> >
> >>> > > > It occurs to me that perhaps Grid-Diag could be made more
> flexible
> >>> >
> >>> > > > if we changed that error message to a debug message in
this
> >>> context.
> >>> >
> >>> > > > It could
> >>> >
> >>> > > > *ATTEMPT* to read all input fields from all input files...
but
> >>> > > > when
> >>> >
> >>> > > > it's unable to, have it print a log message stating that
rather
> >>> > > > than
> >>> >
> >>> > > > erroring out.
> >>> >
> >>> > > >
> >>> >
> >>> > > > That would enable you to compute 2D pdf's using different
> >>> > > > variables
> >>> >
> >>> > > > from different input files.
> >>> >
> >>> > > >
> >>> >
> >>> > > > Do you think that logic is preferable?
> >>> >
> >>> > > >
> >>> >
> >>> > > > Thanks,
> >>> >
> >>> > > > John
> >>> >
> >>> > > >
> >>> >
> >>> > > > On Fri, Dec 4, 2020 at 11:27 PM The RT System itself via
RT <
> >>> >
> >>> > > > met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
> >>> >
> >>> > > >
> >>> >
> >>> > > >>
> >>> >
> >>> > > >> Fri Dec 04 23:27:19 2020: Request 97766 was acted upon.
> >>> >
> >>> > > >> Transaction: Given to johnhg (John Halley Gotway) by
RT_System
> >>> >
> >>> > > >> Queue: met_help
> >>> >
> >>> > > >> Subject: Need help with GridDiag
> >>> >
> >>> > > >> Owner: johnhg
> >>> >
> >>> > > >> Requestors: marion.mittermaier at metoffice.gov.uk<mailto:
> >>> > marion.mittermaier at metoffice.gov.uk>
> >>> >
> >>> > > >> Status: new
> >>> >
> >>> > > >> Ticket <URL:
> >>> >
> >>> > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
> >>> >
> >>> > > >> >
> >>> >
> >>> > > >>
> >>> >
> >>> > > >>
> >>> >
> >>> > > >> This transaction appears to have no content
> >>> >
> >>> > > >>
> >>> >
> >>> > > >
> >>> >
> >>> > >
> >>> >
> >>> > >
> >>> >
> >>> > >
> >>> >
> >>> > >
> >>> >
> >>> >
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>>
>
>
------------------------------------------------
Subject: Need help with GridDiag
From: John Halley Gotway
Time: Thu Mar 04 10:52:37 2021
David,
Thanks for confirming. I know you're working on python enhancements
this
week. So I'll work on a fix but ask you to review the PR when I'm
ready. I
think it'll help get me more familiar with this code anyway.
Thanks,
John
On Thu, Mar 4, 2021 at 10:43 AM David Fillmore via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>
> The description of this issue is correct.
> The histograms are keyed by variable name, so one can not have a
duplicate
> name from a second dataset.
> This was not anticipated and should be remedied.
> David
>
> On Thu, Mar 4, 2021 at 10:37 AM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
> >
> > Marion,
> >
> > Here's the GitHub issue I wrote up to describe this bug:
> > https://github.com/dtcenter/MET/issues/1694
> >
> > John
> >
> > On Thu, Mar 4, 2021 at 9:45 AM John Halley Gotway
<johnhg at ucar.edu>
> wrote:
> >
> > > Marion,
> > >
> > > FYI, I'm pretty confident the bug is caused by the fact that
we're
> > reading
> > > the same variable named "precipitation_amount" from two
different input
> > > data sources. When I change the variable name of the second one
to
> > > "precipitation_amount_MOD" then the problem goes away. Then 1D
> histograms
> > > are then correct.
> > >
> > > So that would provide a short-term, temporary bandaid for now.
You
> could
> > > nc_rename precipitaiton_amount to some other (unique) name prior
to
> > running
> > > grid_diag with met-9.1.
> > >
> > > We should just need to update the code to make the string we're
using
> to
> > > track the counts unique. And that seems very doable for met-
10.0.0.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Mar 4, 2021 at 9:30 AM John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> > >
> > >> Marion,
> > >>
> > >> You're right. I looked closely at the numbers that grid_diag is
> > producing
> > >> and this is pretty obviously a bug. Thanks for finding this and
> letting
> > us
> > >> know!
> > >>
> > >> I'll coordinate with David Fillmore, the engineer who
originally
> worked
> > >> on this to get this fixed for the met-10.0.0 release.
> > >>
> > >> John
> > >>
> > >> On Thu, Mar 4, 2021 at 2:29 AM
marion.mittermaier at metoffice.gov.uk
> via
> > >> RT <met_help at ucar.edu> wrote:
> > >>
> > >>>
> > >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
>
> > >>>
> > >>> Hi John,
> > >>>
> > >>> I thought I had cracked it but something is still not quite
right...
> > the
> > >>> joint distribution does appear when using the dual -data
commands
> with
> > the
> > >>> changed attribute names, and it looks plausible BUT but the
marginal
> > >>> distributions are wrong.
> > >>>
> > >>> If I run the GridDiag code with both data sets then the
marginal
> > >>> distributions are the same, which isn't right.
> > >>> running together
> > >>> glosea 171504, 18373, 11192, 7943, 6551, 4922, 4065, 3680,
> > >>> 3344, 2750, 2583, 1719, 2432, 1602, 1632, 1173, 1358, 899,
774,
> > 685,
> > >>> 720,
> > >>> 625, 782, 650, 569, 502, 377, 385, 236, 286, 244, 237,
123, 116,
> > >>> 156,
> > >>> 112, 128, 100, 151, 88, 122, 173, 109, 63, 62, 53, 88, 40,
49,
> 46,
> > >>> 84,
> > >>> 29, 32, 48, 41, 29, 30, 32, 36, 32, 29, 26, 24, 18, 20,
22, 20,
> 24,
> > >>> 21,
> > >>> 16, 15, 21, 12, 20, 16, 15, 17, 16, 7, 13, 11, 15, 8, 12,
10, 11,
> > >>> 11, 3,
> > >>> 8, 9, 4, 6, 13, 7, 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4, 5,
50, 5,
> 7,
> > >>> 6, 5,
> > >>> 2, 2, 53, 4, 5, 3, 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1, 4,
1, 0,
> 1,
> > >>> 1, 2,
> > >>> 4, 3, 2, 3, 1, 2, 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1,
3, 2, 1,
> > 2,
> > >>> 0,
> > >>> 0, 3, 2, 1, 2, 2, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0,
4, 0, 0,
> > 1,
> > >>> 1,
> > >>> 0, 2, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0,
0, 0, 0,
> > 0,
> > >>> 0,
> > >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0,
> > 0,
> > >>> 0,
> > >>> gpm 171504, 18373, 11192, 7943, 6551, 4922, 4065, 3680,
> > >>> 3344, 2750, 2583, 1719, 2432, 1602, 1632, 1173, 1358, 899,
774,
> > 685,
> > >>> 720,
> > >>> 625, 782, 650, 569, 502, 377, 385, 236, 286, 244, 237,
123, 116,
> > >>> 156,
> > >>> 112, 128, 100, 151, 88, 122, 173, 109, 63, 62, 53, 88, 40,
49,
> 46,
> > >>> 84,
> > >>> 29, 32, 48, 41, 29, 30, 32, 36, 32, 29, 26, 24, 18, 20,
22, 20,
> 24,
> > >>> 21,
> > >>> 16, 15, 21, 12, 20, 16, 15, 17, 16, 7, 13, 11, 15, 8, 12,
10, 11,
> > >>> 11, 3,
> > >>> 8, 9, 4, 6, 13, 7, 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4, 5,
50, 5,
> 7,
> > >>> 6, 5,
> > >>> 2, 2, 53, 4, 5, 3, 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1, 4,
1, 0,
> 1,
> > >>> 1, 2,
> > >>> 4, 3, 2, 3, 1, 2, 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1,
3, 2, 1,
> > 2,
> > >>> 0,
> > >>> 0, 3, 2, 1, 2, 2, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0,
4, 0, 0,
> > 1,
> > >>> 1,
> > >>> 0, 2, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0,
0, 0, 0,
> > 0,
> > >>> 0,
> > >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0,
> > 0,
> > >>> 0,
> > >>>
> > >>> But when I run them separately I get the following for gpm
only (same
> > >>> input files)
> > >>> 93193, 7954, 4758, 3206, 2364, 1889, 1554, 1287, 1081,
> > >>> 977, 824, 685, 673, 550, 520, 470, 490, 409, 391, 344,
340, 311,
> > >>> 278,
> > >>> 295, 246, 224, 195, 198, 188, 184, 175, 143, 123, 116,
116, 112,
> > 88,
> > >>> 100,
> > >>> 111, 88, 82, 74, 69, 63, 62, 53, 48, 40, 49, 46, 36, 29,
32, 48,
> > 41,
> > >>> 29,
> > >>> 30, 32, 36, 32, 29, 26, 24, 18, 20, 22, 20, 24, 21, 16,
15, 21,
> 12,
> > >>> 20,
> > >>> 16, 15, 17, 16, 7, 13, 11, 15, 8, 12, 10, 11, 11, 3, 8, 9,
4, 6,
> > 13,
> > >>> 7,
> > >>> 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4, 5, 2, 5, 7, 6, 5, 2,
2, 5, 4,
> > 5,
> > >>> 3,
> > >>> 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1, 4, 1, 0, 1, 1, 2, 4,
3, 2, 3,
> > 1,
> > >>> 2,
> > >>> 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1, 3, 2, 1, 2, 0, 0,
3, 2, 1,
> > 2,
> > >>> 2,
> > >>> 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0, 4, 0, 0, 1, 1, 0,
2, 0, 1,
> > 0,
> > >>> 1,
> > >>> 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0,
> > 0,
> > >>> 0,
> > >>>
> > >>> and for glosea only
> > >>> 78311, 10419, 6434, 4737, 4187, 3033, 2511, 2393,
> > >>> 2263, 1773, 1759, 1034, 1759, 1052, 1112, 703, 868, 490,
383,
> 341,
> > >>> 380,
> > >>> 314, 504, 355, 323, 278, 182, 187, 48, 102, 69, 94, 0, 0,
40, 0,
> > 40,
> > >>> 0,
> > >>> 40, 0, 40, 99, 40, 0, 0, 0, 40, 0, 0, 0, 48, 0, 0, 0, 0,
0, 0, 0,
> > 0,
> > >>> 0,
> > >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0,
> > 0,
> > >>> 0,
> > >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0,
> > 0,
> > >>> 48,
> > >>> 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
> 0,
> > >>> 0, 0,
> > >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0,
> > 0,
> > >>> 0,
> > >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0,
> > 0,
> > >>> 0,
> > >>>
> > >>> Looking at the first bin 93193+78311 = 171504, bin 2 is
> > 7953+10419=18373
> > >>> etc. which are the values in bins 1 and 2 for both the
marginal
> > >>> distributions when I input both files together. I am wondering
> whether
> > the
> > >>> attribute changes are not pulled through to this somehow? This
is for
> > the
> > >>> sample data you've got and using MET9.1. My workaround for now
is
> that
> > I
> > >>> will have to run the code 3 times, once for the joint
distribution,
> > which
> > >>> looks ok, and twice more running it with just the single data
stream
> > to get
> > >>> the marginals for the two datasets.
> > >>>
> > >>> Regards
> > >>> Marion
> > >>>
> > >>> -----Original Message-----
> > >>> From: John Halley Gotway via RT <met_help at ucar.edu>
> > >>> Sent: 15 February 2021 17:38
> > >>> To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
> > >>> Subject: Re: FW: [rt.rap.ucar.edu #97766] Need help with
GridDiag
> > >>>
> > >>> This email was received from an external source. Always
check
> sender
> > >>> details, links & attachments.
> > >>>
> > >>> Marion,
> > >>>
> > >>> Thanks for sending sample data. I am a little confused though.
The
> > >>> config file references variables named "precipitation_amount"
and
> > >>> "Precipitation_Amount":
> > >>> * name = "precipitation_amount";*
> > >>> * name = "Precipitation_Amount"; *
> > >>>
> > >>> But the actual data you sent does not:
> > >>>
> > >>>
> > >>> * ncdump -h glosea_2019060100_d001_001.nc
> > >>> <http://glosea_2019060100_d001_001.nc> | grep float float
> > >>> precipitation_amount(latitude, longitude) ;*
> > >>>
> > >>> * ncdump -h gpm_2019060100_2019060200.nc
> > >>> <http://gpm_2019060100_2019060200.nc> | grep float float
> > >>> precipitation_amount(latitude, longitude) ;*
> > >>>
> > >>> I do believe the NetCDF is case-sensitive. So I think that's
> important.
> > >>> But I decided to just try to get Grid-Diag working on the data
you
> > sent. I
> > >>> did run into an error... different from the one you sent:
> > >>>
> > >>>
> > >>> *terminate called after throwing an instance of
> > >>> 'netCDF::exceptions::NcNameInUse' what(): NetCDF: String
match to
> > name in
> > >>> use*
> > >>>
> > >>> Grid-Diag reads the first variable from the first "-data"
source and
> > the
> > >>> second variable from the second "-data" source. And writes
data to
> the
> > >>> output NetCDF file. But since those variable names are the
same,
> > there's a
> > >>> name conflict! It's trying to write the same variable names to
the
> > output
> > >>> NetCDF file which causes the NetCDF library to error out.
> > >>>
> > >>> To get around this, I redefined the input variable names and
levels
> in
> > >>> the config file using:
> > >>> * set_attr_name = "glosea_precip";*
> > >>>
> > >>> * set_attr_level = "sfc"; set_attr_name = "gpm_precip";*
> > >>> * set_attr_level = "sfc";*
> > >>>
> > >>> That works fine and produces the attached output file. I also
> attached
> > >>> my run command, config file, and resulting log file.
> > >>>
> > >>> Here's some improvements we could make:
> > >>>
> > >>> (1) Using set_attr_name and set_attr_level works great. But if
you
> look
> > >>> closely at the log messages, the updated strings are NOT used
there.
> > Seems
> > >>> like they should be. Do you agree?
> > >>>
> > >>> (2) The actual data values don't go much over 250... but
you've
> defined
> > >>> bins all the way out to 1000. That'll leave you with at least
75% of
> > the
> > >>> bins empty. Can you think of any enhancements to make the
setting of
> > the
> > >>> number and size of bins easier?
> > >>>
> > >>> (3) The GPM data appears to use a value of 9.96921E36 as
missing
> data,
> > >>> but no _FillValue or missing_value variable attribute defines
that.
> So
> > MET
> > >>> processes considers all these pink values to be valid data.
Doesn't
> > matter
> > >>> in this case because you're only using data from India, which
has all
> > valid
> > >>> data.
> > >>> [image: Screen Shot 2021-02-15 at 10.31.31 AM.png] I'd
recommend
> using
> > >>> _FillValue or missing_value to define bad data values in the
GPM
> files:
> > >>>
> > >>>
> > >>>
> >
> https://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-
conventions.html#missing-data
> > >>> And do the same for the glosea files as well.
> > >>>
> > >>> Thanks,
> > >>> John
> > >>>
> > >>> On Mon, Feb 15, 2021 at 4:43 AM
marion.mittermaier at metoffice.gov.uk
> > via
> > >>> RT < met_help at ucar.edu> wrote:
> > >>>
> > >>> >
> > >>> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
> > >>> >
> > >>> > Hi all,
> > >>> >
> > >>> > in the mean time I have got regrid_data_plane to work in
> stand-alone
> > >>> > model… running the following command (and I’ve tried in both
> > >>> directions):
> > >>> >
> > >>> > regrid_data_plane glosea_2019060100_d001_001.nc
> > >>> > gpm_2019060100_2019060200.nc grid_diag_out.nc -field
019060200.nc
> > >>> > grid_diag_out.nc -field ‘name="precipitation_amount";
> > level="(*,*)";'
> > >>> > <
> > >>> >
> > >>> > DEBUG 1: Reading data file: glosea_2019060100_d001_001.nc
DEBUG 2:
> > >>> > Input grid: Projection: Lat/Lon Nx: 432 Ny: 324 lat_ll:
-89.722
> > >>> > lon_ll: -0.417 delta_lat: 0.556 delta_lon: 0.833 DEBUG 2:
Output
> > grid:
> > >>> > Projection: Lat/Lon Nx: 3600 Ny: 1800 lat_ll:
> > >>> > -89.950 lon_ll: 179.950 delta_lat: 0.100 delta_lon: 0.100
DEBUG 2:
> > >>> > Interpolation options: method = NEAREST, width = 1, shape =
SQUARE,
> > >>> > vld_thresh = 0.5 DEBUG 2: Range of input data
> > >>> > (name="precipitation_amount"; level="(*,*)";) is 0 to
262.128.
> > >>> > DEBUG 2: Range of regridded data
(name="precipitation_amount";
> > >>> > level="(*,*)";) is 0 to 262.128.
> > >>> > DEBUG 1: Writing output file: grid_diag_out.nc
> > >>> >
> > >>> > Which looks sensible… so it’s NOT regrid_data_plane.
> > >>> >
> > >>> > Marion
> > >>> >
> > >>> > From: Mittermaier, Marion
> > >>> > Sent: 15 February 2021 11:02
> > >>> > To: 'met_help at ucar.edu' <met_help at ucar.edu>
> > >>> > Subject: RE: [rt.rap.ucar.edu #97766] Need help with
GridDiag
> > >>> >
> > >>> >
> > >>> > Hi John,
> > >>> >
> > >>> >
> > >>> >
> > >>> > it's taken me a while to get back to this... I do see the
> difference
> > >>> > but I still don't get it to work. I can get either on their
own to
> > >>> > work but not together.
> > >>> >
> > >>> > I have split out the forecast days into separate files.
> > >>> >
> > >>> >
> > >>> >
> > >>> > grid_diag -data
> > >>> > /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc
> > >>> > -data /scratch/fryl/GPM/gpm_2019060100_2019060200.nc -config
> > >>> > GridDiag.glosea -out grid_diag_out.nc -v 4
> > >>> >
> > >>> >
> > >>> >
> > >>> > This gives the following error (I think it's in
regrid_data_plane).
> > >>> > Not sure why it is trying to regrid both? Or maybe I’m just
not
> quite
> > >>> > following the messages….
> > >>> >
> > >>> >
> > >>> >
> > >>> > vld497:/net/home/h02/frmm/WCSSP/India > grid_diag -data
> > >>> > /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc
-data
> > >>> > /scratch/fryl/GPM/gpm_2019060100_2019060200.nc
> > >>> >
> > >>> > DEBUG 1: Default Config File:
> > >>> >
> >
/data/users/cfrd/MET_requirements/MET9.0/share/met/config/GridDiagConf
> > >>> > ig_default
> > >>> >
> > >>> > DEBUG 1: User Config File: GridDiag.glosea
> > >>> >
> > >>> > DEBUG 2: Read config files.
> > >>> >
> > >>> > DEBUG 2: Enter get_mtddf.
> > >>> >
> > >>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> new
> > >>> > Met2dDataFile object of type "FileType_NcCF".
> > >>> >
> > >>> > DEBUG 4: NcCfFile::open() -> parsing units for the time
variable
> > >>> > "hours since 1970-01-01"
> > >>> >
> > >>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF
convention time
> > >>> > unit string "hours since 1970-01-01" as a reference time of
> > >>> > 19700101_000000 and
> > >>> > 3600 second(s) per time step.
> > >>> >
> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from the
> > >>> > "forecast_reference_time" variable.
> > >>> >
> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from file
> > >>> name.
> > >>> >
> > >>> > DEBUG 2: Process configuration.
> > >>> >
> > >>> > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
> object
> > >>> > of type "FileType_NcCF".
> > >>> >
> > >>> > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
> object
> > >>> > of type "FileType_NcCF".
> > >>> >
> > >>> > DEBUG 3: Use the grid defined by file
> "/scratch/fryl/WCSSPIndia/GPM/
> > >>> > gpm_2019060100_2019060200.nc".
> > >>> >
> > >>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> new
> > >>> > Met2dDataFile object of type "FileType_NcCF".
> > >>> >
> > >>> > DEBUG 4: NcCfFile::open() -> parsing units for the time
variable
> > >>> > "hours since 1970-01-01"
> > >>> >
> > >>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF
convention time
> > >>> > unit string "hours since 1970-01-01" as a reference time of
> > >>> > 19700101_000000 and
> > >>> > 3600 second(s) per time step.
> > >>> >
> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from the
> > >>> > "forecast_reference_time" variable.
> > >>> >
> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from file
> > >>> name.
> > >>> >
> > >>> > DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny:
360
> lat_ll:
> > >>> > 4.550 lon_ll: -64.550 delta_lat: 0.100 delta_lon: 0.100
> > >>> >
> > >>> > DEBUG 2: Processing masking regions.
> > >>> >
> > >>> > DEBUG 3: Processing poly mask: India.poly
> > >>> >
> > >>> > DEBUG 4: parse_poly_mask() -> parsing poly mask "India.poly"
> > >>> >
> > >>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> new
> > >>> > Met2dDataFile object of type "FileType_None".
> > >>> >
> > >>> > DEBUG 4: parse_poly_mask() -> parsing poly mask file
"India.poly"
> > >>> >
> > >>> > DEBUG 2: Initializing precipitation_amount(*,*) histogram
> > >>> >
> > >>> > DEBUG 2: Initializing Precipitation_Amount(*,*) histogram
> > >>> >
> > >>> > DEBUG 2: Initializing
> > >>> > precipitation_amount(*,*)_Precipitation_Amount(*,*)
> > >>> > joint histogram
> > >>> >
> > >>> > DEBUG 1: grid_diag_out.nc
> > >>> >
> > >>> > DEBUG 1: Length of configuration "data.field" = 2
> > >>> >
> > >>> > DEBUG 1: Length of data file list = 1
> > >>> >
> > >>> > DEBUG 1: Series defined by the data file list of length 1.
> > >>> >
> > >>> > DEBUG 2: 1000
> > >>> >
> > >>> > DEBUG 2: Processing series entry 1: precipitation_amount:
> > >>> > precipitation_amount(*,*)
> > >>> >
> > >>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> new
> > >>> > Met2dDataFile object of type "FileType_NcCF".
> > >>> >
> > >>> > DEBUG 4: NcCfFile::open() -> parsing units for the time
variable
> > >>> > "hours since 1970-01-01"
> > >>> >
> > >>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF
convention time
> > >>> > unit string "hours since 1970-01-01" as a reference time of
> > >>> > 19700101_000000 and
> > >>> > 3600 second(s) per time step.
> > >>> >
> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from the
> > >>> > "forecast_reference_time" variable.
> > >>> >
> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from file
> > >>> name.
> > >>> >
> > >>> > DEBUG 4: NcCfFile::getData() -> setting the unset init time
to the
> > >>> > valid time of 20190601_115959.
> > >>> >
> > >>> > DEBUG 4:
> > >>> >
> > >>> > DEBUG 4: Data plane information:
> > >>> >
> > >>> > DEBUG 4: plane min: 0
> > >>> >
> > >>> > DEBUG 4: plane max: 9.96921e+36
> > >>> >
> > >>> > DEBUG 4: valid time: 20190601_115959
> > >>> >
> > >>> > DEBUG 4: lead time: 000000
> > >>> >
> > >>> > DEBUG 4: init time: 20190601_115959
> > >>> >
> > >>> > DEBUG 4: accum time: 000000
> > >>> >
> > >>> > DEBUG 1: Regridding field precipitation_amount(*,*) to the
> > >>> > verification grid.
> > >>> >
> > >>> > DEBUG 2: Processing series entry 1: Precipitation_Amount:
> > >>> > Precipitation_Amount(*,*)
> > >>> >
> > >>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
> new
> > >>> > Met2dDataFile object of type "FileType_NcCF".
> > >>> >
> > >>> > DEBUG 4: NcCfFile::open() -> parsing units for the time
variable
> > >>> > "hours since 1970-01-01"
> > >>> >
> > >>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF
convention time
> > >>> > unit string "hours since 1970-01-01" as a reference time of
> > >>> > 19700101_000000 and
> > >>> > 3600 second(s) per time step.
> > >>> >
> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from the
> > >>> > "forecast_reference_time" variable.
> > >>> >
> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from file
> > >>> name.
> > >>> >
> > >>> > DEBUG 1: Regridding field precipitation_amount(*,*) to the
> > >>> > verification grid.
> > >>> >
> > >>> > ERROR :
> > >>> >
> > >>> > ERROR : DataPlane::two_to_one() -> range check error: (Nx,
Ny) =
> (0,
> > >>> > 0), (x, y) = (0, 0)
> > >>> >
> > >>> > ERROR :
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>> > -----Original Message-----
> > >>> > From: John Halley Gotway via RT <met_help at ucar.edu<mailto:
> > >>> > met_help at ucar.edu>>
> > >>> > Sent: 08 December 2020 20:54
> > >>> > To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk
> <mailto:
> > >>> > marion.mittermaier at metoffice.gov.uk>>
> > >>> > Subject: Re: [rt.rap.ucar.edu #97766] Need help with
GridDiag
> > >>> >
> > >>> >
> > >>> >
> > >>> > This email was received from an external source. Always
check
> > sender
> > >>> > details, links & attachments.
> > >>> >
> > >>> >
> > >>> >
> > >>> > Marion,
> > >>> >
> > >>> >
> > >>> >
> > >>> > I'm still not convinced. Please take a very careful look at
the
> > >>> > command you're executing. I see all the log messages you
sent, but
> > not
> > >>> > the actual command you ran. At that matters a lot!
> > >>> >
> > >>> >
> > >>> >
> > >>> > This command fails on my machine with the SAME ERROR you're
> getting:
> > >>> >
> > >>> > * grid_diag -data glosea_2019060100_001.nc
> > >>> >
> > >>> > <http://glosea_2019060100_001.nc/>
gpm_2019060100_2019060200.nc <
> > >>> > http://gpm_2019060100_2019060200.nc/> -config
GridDiag.glosea -out
> > >>> > grid_diag_out.nc <http://grid_diag_out.nc/>*
> > >>> >
> > >>> >
> > >>> >
> > >>> > ERROR : NcCfFile::getData(NcVar *, const LongArray &,
DataPlane
> > &)
> > >>> > const -> needed 3 arguments for variable
precipitation_amount, got
> 2
> > >>> >
> > >>> >
> > >>> >
> > >>> > But this command runs WITHOUT ERROR:
> > >>> >
> > >>> > * grid_diag -data glosea_2019060100_001.nc
> > >>> >
> > >>> > <http://glosea_2019060100_001.nc/> -data
> > gpm_2019060100_2019060200.nc
> > >>> > < http://gpm_2019060100_2019060200.nc/> -config
GridDiag.glosea
> -out
> > >>> > grid_diag_out.nc <http://grid_diag_out.nc/>*
> > >>> >
> > >>> >
> > >>> >
> > >>> > Do you see the difference?
> > >>> >
> > >>> > In the first, I used "-data" once and listed 2 different
file
> names.
> > >>> >
> > >>> > In the second, I used "-data" twice, once for each of the
input
> > files.
> > >>> >
> > >>> >
> > >>> >
> > >>> > Please try doing exactly the same thing.
> > >>> >
> > >>> >
> > >>> >
> > >>> > Does that enable it to run without error? Note that you'll
still
> get
> > >>> > warnings about the actual data values falling outside the
defined
> > >>> > range of the bins.
> > >>> >
> > >>> >
> > >>> >
> > >>> > As for the bin endpoints, I think this is the relevant line
of
> code:
> > >>> >
> > >>> >
> > >>> >
> >
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc
> > >>> > 5981bb8/met/src/libcode/vx_series_data/series_pdf.cc#L72
> > >>> >
> > >>> >
> > >>> >
> > >>> > int k = floor((value - min) / delta);
> > >>> >
> > >>> >
> > >>> >
> > >>> > This places the current "value" into the k-th bin where
"min" is
> the
> > >>> > minimum PDF value and "delta" is the bin width.
> > >>> >
> > >>> > I believe this is consistent with bins being closed on the
left
> side
> > >>> > and open on the right:
> > >>> >
> > >>> > [0,1) i.e. 0 <= x < 1
> > >>> >
> > >>> >
> > >>> >
> > >>> > John
> > >>> >
> > >>> >
> > >>> >
> > >>> > On Tue, Dec 8, 2020 at 8:30 AM
marion.mittermaier at metoffice.gov.uk
> > >>> <mailto:
> > >>> > marion.mittermaier at metoffice.gov.uk> via RT <
met_help at ucar.edu
> > >>> <mailto:
> > >>> > met_help at ucar.edu>> wrote:
> > >>> >
> > >>> >
> > >>> >
> > >>> > >
> > >>> >
> > >>> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
> > >>> >
> > >>> > >
> > >>> >
> > >>> > > Hi John,
> > >>> >
> > >>> > >
> > >>> >
> > >>> > > one final question for GridDiag.
> > >>> >
> > >>> > >
> > >>> >
> > >>> > > The bins are set up to be lower bound inclusive? i.e.
> > >>> >
> > >>> > > n_bins = 1000;
> > >>> >
> > >>> > > range = [0, 1000]; // kg/m^2
> > >>> >
> > >>> > >
> > >>> >
> > >>> > > means that the bin is set up as [0,1) i.e. 0 <= x < 1?
> > >>> >
> > >>> > >
> > >>> >
> > >>> > > I'm presuming so.... and the upper bound would be
[999,1000) with
> > no
> > >>> >
> > >>> > > bin for [1000,inf)? i.e. if there were a 1000 in my sample
it
> would
> > >>> >
> > >>> > > NOT be counted? Or anything bigger.
> > >>> >
> > >>> > >
> > >>> >
> > >>> > > That's what it looks like, i.e. you need to make sure that
your
> > >>> > > range
> > >>> >
> > >>> > > encompasses all possible values. Not sure this is in the
> > >>> > documentation...?
> > >>> >
> > >>> > > Would be useful to add.
> > >>> >
> > >>> > >
> > >>> >
> > >>> > > Ta
> > >>> >
> > >>> > > Marion
> > >>> >
> > >>> > >
> > >>> >
> > >>> > > -----Original Message-----
> > >>> >
> > >>> > > From: John Halley Gotway via RT <met_help at ucar.edu<mailto:
> > >>> > met_help at ucar.edu>>
> > >>> >
> > >>> > > Sent: 05 December 2020 06:41
> > >>> >
> > >>> > > To: Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk
> > <mailto:
> > >>> > marion.mittermaier at metoffice.gov.uk>>
> > >>> >
> > >>> > > Subject: Re: [rt.rap.ucar.edu #97766] Need help with
GridDiag
> > >>> >
> > >>> > >
> > >>> >
> > >>> > > This email was received from an external source. Always
check
> > >>> sender
> > >>> >
> > >>> > > details, links & attachments.
> > >>> >
> > >>> > >
> > >>> >
> > >>> > > Marion,
> > >>> >
> > >>> > >
> > >>> >
> > >>> > > Ugh, I should've just handed this ticket off to David in
the
> first
> > >>> place.
> > >>> >
> > >>> > > I did more testing and was able to get it to work. I just
had to
> > use
> > >>> >
> > >>> > > separate "-data" command line options for each input data
source.
> > >>> >
> > >>> > >
> > >>> >
> > >>> > > So this command fails:
> > >>> >
> > >>> > > grid_diag -data glosea_2019060100_001.nc
> > >>> > > gpm_2019060100_2019060200.nc
> > >>> >
> > >>> > > -config GridDiag.glosea -out grid_diag_out.nc
> > >>> >
> > >>> > >
> > >>> >
> > >>> > > But this command runs fine:
> > >>> >
> > >>> > > *grid_diag -data glosea_2019060100_001.nc <
> > >>> >
> > >>> > > http://glosea_2019060100_001.nc> -data
> > gpm_2019060100_2019060200.nc
> > >>> > > <
> > >>> >
> > >>> > > http://gpm_2019060100_2019060200.nc> -config
GridDiag.glosea
> -out
> > >>> >
> > >>> > > grid_diag_out.nc <http://grid_diag_out.nc>*
> > >>> >
> > >>> > >
> > >>> >
> > >>> > > In the second one, I've specified "-data" separately for
each
> data
> > >>> > source.
> > >>> >
> > >>> > > Although, you'll see warning about the range of the data
> extending
> > >>> >
> > >>> > > beyond the range of the bins defined in the config file.
> > >>> >
> > >>> > >
> > >>> >
> > >>> > > Sorry to have such a rambling and confusing response!
> > >>> >
> > >>> > >
> > >>> >
> > >>> > > So to summarize, I believe the logic is:
> > >>> >
> > >>> > > 1. If "-data" is used once, all specified input fields are
read
> > from
> > >>> >
> > >>> > > all of those data files.
> > >>> >
> > >>> > > 2. If "-data" is used more than once, the number of "-
data"
> entries
> > >>> >
> > >>> > > must match the number of fields defined in the config
file.
> > >>> >
> > >>> > > If not, you get an error message.
> > >>> >
> > >>> > > If so, only the i-th field is read from the files in
the i-th
> > >>> >
> > >>> > > -data source.
> > >>> >
> > >>> > >
> > >>> >
> > >>> > > John
> > >>> >
> > >>> > >
> > >>> >
> > >>> > > On Fri, Dec 4, 2020 at 11:30 PM John Halley Gotway <
> > johnhg at ucar.edu
> > >>> > <mailto:johnhg at ucar.edu>>
> > >>> >
> > >>> > > wrote:
> > >>> >
> > >>> > >
> > >>> >
> > >>> > > > Marion,
> > >>> >
> > >>> > > >
> > >>> >
> > >>> > > > It occurs to me that perhaps Grid-Diag could be made
more
> > flexible
> > >>> >
> > >>> > > > if we changed that error message to a debug message in
this
> > >>> context.
> > >>> >
> > >>> > > > It could
> > >>> >
> > >>> > > > *ATTEMPT* to read all input fields from all input
files... but
> > >>> > > > when
> > >>> >
> > >>> > > > it's unable to, have it print a log message stating that
rather
> > >>> > > > than
> > >>> >
> > >>> > > > erroring out.
> > >>> >
> > >>> > > >
> > >>> >
> > >>> > > > That would enable you to compute 2D pdf's using
different
> > >>> > > > variables
> > >>> >
> > >>> > > > from different input files.
> > >>> >
> > >>> > > >
> > >>> >
> > >>> > > > Do you think that logic is preferable?
> > >>> >
> > >>> > > >
> > >>> >
> > >>> > > > Thanks,
> > >>> >
> > >>> > > > John
> > >>> >
> > >>> > > >
> > >>> >
> > >>> > > > On Fri, Dec 4, 2020 at 11:27 PM The RT System itself via
RT <
> > >>> >
> > >>> > > > met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
> > >>> >
> > >>> > > >
> > >>> >
> > >>> > > >>
> > >>> >
> > >>> > > >> Fri Dec 04 23:27:19 2020: Request 97766 was acted upon.
> > >>> >
> > >>> > > >> Transaction: Given to johnhg (John Halley Gotway) by
RT_System
> > >>> >
> > >>> > > >> Queue: met_help
> > >>> >
> > >>> > > >> Subject: Need help with GridDiag
> > >>> >
> > >>> > > >> Owner: johnhg
> > >>> >
> > >>> > > >> Requestors:
marion.mittermaier at metoffice.gov.uk<mailto:
> > >>> > marion.mittermaier at metoffice.gov.uk>
> > >>> >
> > >>> > > >> Status: new
> > >>> >
> > >>> > > >> Ticket <URL:
> > >>> >
> > >>> > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
> > >>> >
> > >>> > > >> >
> > >>> >
> > >>> > > >>
> > >>> >
> > >>> > > >>
> > >>> >
> > >>> > > >> This transaction appears to have no content
> > >>> >
> > >>> > > >>
> > >>> >
> > >>> > > >
> > >>> >
> > >>> > >
> > >>> >
> > >>> > >
> > >>> >
> > >>> > >
> > >>> >
> > >>> > >
> > >>> >
> > >>> >
> > >>> >
> > >>> >
> > >>>
> > >>>
> > >>>
> > >>>
> >
> >
>
>
------------------------------------------------
Subject: Need help with GridDiag
From: John Halley Gotway
Time: Thu Mar 04 17:31:02 2021
David,
FYI, I just assigned this PR to you to review the fix for this issue:
https://github.com/dtcenter/MET/pull/1696
Thanks,
John
On Thu, Mar 4, 2021 at 10:52 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> David,
>
> Thanks for confirming. I know you're working on python enhancements
this
> week. So I'll work on a fix but ask you to review the PR when I'm
ready. I
> think it'll help get me more familiar with this code anyway.
>
> Thanks,
> John
>
> On Thu, Mar 4, 2021 at 10:43 AM David Fillmore via RT
<met_help at ucar.edu>
> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>>
>> The description of this issue is correct.
>> The histograms are keyed by variable name, so one can not have a
duplicate
>> name from a second dataset.
>> This was not anticipated and should be remedied.
>> David
>>
>> On Thu, Mar 4, 2021 at 10:37 AM John Halley Gotway via RT <
>> met_help at ucar.edu>
>> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>> >
>> > Marion,
>> >
>> > Here's the GitHub issue I wrote up to describe this bug:
>> > https://github.com/dtcenter/MET/issues/1694
>> >
>> > John
>> >
>> > On Thu, Mar 4, 2021 at 9:45 AM John Halley Gotway
<johnhg at ucar.edu>
>> wrote:
>> >
>> > > Marion,
>> > >
>> > > FYI, I'm pretty confident the bug is caused by the fact that
we're
>> > reading
>> > > the same variable named "precipitation_amount" from two
different
>> input
>> > > data sources. When I change the variable name of the second one
to
>> > > "precipitation_amount_MOD" then the problem goes away. Then 1D
>> histograms
>> > > are then correct.
>> > >
>> > > So that would provide a short-term, temporary bandaid for now.
You
>> could
>> > > nc_rename precipitaiton_amount to some other (unique) name
prior to
>> > running
>> > > grid_diag with met-9.1.
>> > >
>> > > We should just need to update the code to make the string we're
using
>> to
>> > > track the counts unique. And that seems very doable for met-
10.0.0.
>> > >
>> > > Thanks,
>> > > John
>> > >
>> > > On Thu, Mar 4, 2021 at 9:30 AM John Halley Gotway
<johnhg at ucar.edu>
>> > wrote:
>> > >
>> > >> Marion,
>> > >>
>> > >> You're right. I looked closely at the numbers that grid_diag
is
>> > producing
>> > >> and this is pretty obviously a bug. Thanks for finding this
and
>> letting
>> > us
>> > >> know!
>> > >>
>> > >> I'll coordinate with David Fillmore, the engineer who
originally
>> worked
>> > >> on this to get this fixed for the met-10.0.0 release.
>> > >>
>> > >> John
>> > >>
>> > >> On Thu, Mar 4, 2021 at 2:29 AM
marion.mittermaier at metoffice.gov.uk
>> via
>> > >> RT <met_help at ucar.edu> wrote:
>> > >>
>> > >>>
>> > >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
>
>> > >>>
>> > >>> Hi John,
>> > >>>
>> > >>> I thought I had cracked it but something is still not quite
right...
>> > the
>> > >>> joint distribution does appear when using the dual -data
commands
>> with
>> > the
>> > >>> changed attribute names, and it looks plausible BUT but the
marginal
>> > >>> distributions are wrong.
>> > >>>
>> > >>> If I run the GridDiag code with both data sets then the
marginal
>> > >>> distributions are the same, which isn't right.
>> > >>> running together
>> > >>> glosea 171504, 18373, 11192, 7943, 6551, 4922, 4065, 3680,
>> > >>> 3344, 2750, 2583, 1719, 2432, 1602, 1632, 1173, 1358,
899, 774,
>> > 685,
>> > >>> 720,
>> > >>> 625, 782, 650, 569, 502, 377, 385, 236, 286, 244, 237,
123, 116,
>> > >>> 156,
>> > >>> 112, 128, 100, 151, 88, 122, 173, 109, 63, 62, 53, 88,
40, 49,
>> 46,
>> > >>> 84,
>> > >>> 29, 32, 48, 41, 29, 30, 32, 36, 32, 29, 26, 24, 18, 20,
22, 20,
>> 24,
>> > >>> 21,
>> > >>> 16, 15, 21, 12, 20, 16, 15, 17, 16, 7, 13, 11, 15, 8, 12,
10,
>> 11,
>> > >>> 11, 3,
>> > >>> 8, 9, 4, 6, 13, 7, 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4, 5,
50,
>> 5, 7,
>> > >>> 6, 5,
>> > >>> 2, 2, 53, 4, 5, 3, 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1, 4,
1, 0,
>> 1,
>> > >>> 1, 2,
>> > >>> 4, 3, 2, 3, 1, 2, 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1,
3, 2,
>> 1,
>> > 2,
>> > >>> 0,
>> > >>> 0, 3, 2, 1, 2, 2, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0,
4, 0,
>> 0,
>> > 1,
>> > >>> 1,
>> > >>> 0, 2, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0,
0, 0,
>> 0,
>> > 0,
>> > >>> 0,
>> > >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>> 0,
>> > 0,
>> > >>> 0,
>> > >>> gpm 171504, 18373, 11192, 7943, 6551, 4922, 4065, 3680,
>> > >>> 3344, 2750, 2583, 1719, 2432, 1602, 1632, 1173, 1358,
899, 774,
>> > 685,
>> > >>> 720,
>> > >>> 625, 782, 650, 569, 502, 377, 385, 236, 286, 244, 237,
123, 116,
>> > >>> 156,
>> > >>> 112, 128, 100, 151, 88, 122, 173, 109, 63, 62, 53, 88,
40, 49,
>> 46,
>> > >>> 84,
>> > >>> 29, 32, 48, 41, 29, 30, 32, 36, 32, 29, 26, 24, 18, 20,
22, 20,
>> 24,
>> > >>> 21,
>> > >>> 16, 15, 21, 12, 20, 16, 15, 17, 16, 7, 13, 11, 15, 8, 12,
10,
>> 11,
>> > >>> 11, 3,
>> > >>> 8, 9, 4, 6, 13, 7, 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4, 5,
50,
>> 5, 7,
>> > >>> 6, 5,
>> > >>> 2, 2, 53, 4, 5, 3, 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1, 4,
1, 0,
>> 1,
>> > >>> 1, 2,
>> > >>> 4, 3, 2, 3, 1, 2, 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1,
3, 2,
>> 1,
>> > 2,
>> > >>> 0,
>> > >>> 0, 3, 2, 1, 2, 2, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0,
4, 0,
>> 0,
>> > 1,
>> > >>> 1,
>> > >>> 0, 2, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0,
0, 0,
>> 0,
>> > 0,
>> > >>> 0,
>> > >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>> 0,
>> > 0,
>> > >>> 0,
>> > >>>
>> > >>> But when I run them separately I get the following for gpm
only
>> (same
>> > >>> input files)
>> > >>> 93193, 7954, 4758, 3206, 2364, 1889, 1554, 1287, 1081,
>> > >>> 977, 824, 685, 673, 550, 520, 470, 490, 409, 391, 344,
340, 311,
>> > >>> 278,
>> > >>> 295, 246, 224, 195, 198, 188, 184, 175, 143, 123, 116,
116, 112,
>> > 88,
>> > >>> 100,
>> > >>> 111, 88, 82, 74, 69, 63, 62, 53, 48, 40, 49, 46, 36, 29,
32, 48,
>> > 41,
>> > >>> 29,
>> > >>> 30, 32, 36, 32, 29, 26, 24, 18, 20, 22, 20, 24, 21, 16,
15, 21,
>> 12,
>> > >>> 20,
>> > >>> 16, 15, 17, 16, 7, 13, 11, 15, 8, 12, 10, 11, 11, 3, 8,
9, 4, 6,
>> > 13,
>> > >>> 7,
>> > >>> 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4, 5, 2, 5, 7, 6, 5, 2,
2, 5,
>> 4,
>> > 5,
>> > >>> 3,
>> > >>> 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1, 4, 1, 0, 1, 1, 2, 4,
3, 2,
>> 3,
>> > 1,
>> > >>> 2,
>> > >>> 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1, 3, 2, 1, 2, 0, 0,
3, 2,
>> 1,
>> > 2,
>> > >>> 2,
>> > >>> 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0, 4, 0, 0, 1, 1, 0,
2, 0,
>> 1,
>> > 0,
>> > >>> 1,
>> > >>> 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>> 0,
>> > 0,
>> > >>> 0,
>> > >>>
>> > >>> and for glosea only
>> > >>> 78311, 10419, 6434, 4737, 4187, 3033, 2511, 2393,
>> > >>> 2263, 1773, 1759, 1034, 1759, 1052, 1112, 703, 868, 490,
383,
>> 341,
>> > >>> 380,
>> > >>> 314, 504, 355, 323, 278, 182, 187, 48, 102, 69, 94, 0, 0,
40, 0,
>> > 40,
>> > >>> 0,
>> > >>> 40, 0, 40, 99, 40, 0, 0, 0, 40, 0, 0, 0, 48, 0, 0, 0, 0,
0, 0,
>> 0,
>> > 0,
>> > >>> 0,
>> > >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>> 0,
>> > 0,
>> > >>> 0,
>> > >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>> 0,
>> > 0,
>> > >>> 48,
>> > >>> 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>> 0,
>> > >>> 0, 0,
>> > >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>> 0,
>> > 0,
>> > >>> 0,
>> > >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>> 0,
>> > 0,
>> > >>> 0,
>> > >>>
>> > >>> Looking at the first bin 93193+78311 = 171504, bin 2 is
>> > 7953+10419=18373
>> > >>> etc. which are the values in bins 1 and 2 for both the
marginal
>> > >>> distributions when I input both files together. I am
wondering
>> whether
>> > the
>> > >>> attribute changes are not pulled through to this somehow?
This is
>> for
>> > the
>> > >>> sample data you've got and using MET9.1. My workaround for
now is
>> that
>> > I
>> > >>> will have to run the code 3 times, once for the joint
distribution,
>> > which
>> > >>> looks ok, and twice more running it with just the single data
stream
>> > to get
>> > >>> the marginals for the two datasets.
>> > >>>
>> > >>> Regards
>> > >>> Marion
>> > >>>
>> > >>> -----Original Message-----
>> > >>> From: John Halley Gotway via RT <met_help at ucar.edu>
>> > >>> Sent: 15 February 2021 17:38
>> > >>> To: Mittermaier, Marion <marion.mittermaier at metoffice.gov.uk>
>> > >>> Subject: Re: FW: [rt.rap.ucar.edu #97766] Need help with
GridDiag
>> > >>>
>> > >>> This email was received from an external source. Always
check
>> sender
>> > >>> details, links & attachments.
>> > >>>
>> > >>> Marion,
>> > >>>
>> > >>> Thanks for sending sample data. I am a little confused
though. The
>> > >>> config file references variables named "precipitation_amount"
and
>> > >>> "Precipitation_Amount":
>> > >>> * name = "precipitation_amount";*
>> > >>> * name = "Precipitation_Amount"; *
>> > >>>
>> > >>> But the actual data you sent does not:
>> > >>>
>> > >>>
>> > >>> * ncdump -h glosea_2019060100_d001_001.nc
>> > >>> <http://glosea_2019060100_d001_001.nc> | grep float float
>> > >>> precipitation_amount(latitude, longitude) ;*
>> > >>>
>> > >>> * ncdump -h gpm_2019060100_2019060200.nc
>> > >>> <http://gpm_2019060100_2019060200.nc> | grep float float
>> > >>> precipitation_amount(latitude, longitude) ;*
>> > >>>
>> > >>> I do believe the NetCDF is case-sensitive. So I think that's
>> important.
>> > >>> But I decided to just try to get Grid-Diag working on the
data you
>> > sent. I
>> > >>> did run into an error... different from the one you sent:
>> > >>>
>> > >>>
>> > >>> *terminate called after throwing an instance of
>> > >>> 'netCDF::exceptions::NcNameInUse' what(): NetCDF: String
match to
>> > name in
>> > >>> use*
>> > >>>
>> > >>> Grid-Diag reads the first variable from the first "-data"
source and
>> > the
>> > >>> second variable from the second "-data" source. And writes
data to
>> the
>> > >>> output NetCDF file. But since those variable names are the
same,
>> > there's a
>> > >>> name conflict! It's trying to write the same variable names
to the
>> > output
>> > >>> NetCDF file which causes the NetCDF library to error out.
>> > >>>
>> > >>> To get around this, I redefined the input variable names and
levels
>> in
>> > >>> the config file using:
>> > >>> * set_attr_name = "glosea_precip";*
>> > >>>
>> > >>> * set_attr_level = "sfc"; set_attr_name =
"gpm_precip";*
>> > >>> * set_attr_level = "sfc";*
>> > >>>
>> > >>> That works fine and produces the attached output file. I also
>> attached
>> > >>> my run command, config file, and resulting log file.
>> > >>>
>> > >>> Here's some improvements we could make:
>> > >>>
>> > >>> (1) Using set_attr_name and set_attr_level works great. But
if you
>> look
>> > >>> closely at the log messages, the updated strings are NOT used
there.
>> > Seems
>> > >>> like they should be. Do you agree?
>> > >>>
>> > >>> (2) The actual data values don't go much over 250... but
you've
>> defined
>> > >>> bins all the way out to 1000. That'll leave you with at least
75% of
>> > the
>> > >>> bins empty. Can you think of any enhancements to make the
setting of
>> > the
>> > >>> number and size of bins easier?
>> > >>>
>> > >>> (3) The GPM data appears to use a value of 9.96921E36 as
missing
>> data,
>> > >>> but no _FillValue or missing_value variable attribute defines
that.
>> So
>> > MET
>> > >>> processes considers all these pink values to be valid data.
Doesn't
>> > matter
>> > >>> in this case because you're only using data from India, which
has
>> all
>> > valid
>> > >>> data.
>> > >>> [image: Screen Shot 2021-02-15 at 10.31.31 AM.png] I'd
recommend
>> using
>> > >>> _FillValue or missing_value to define bad data values in the
GPM
>> files:
>> > >>>
>> > >>>
>> > >>>
>> >
>> https://cfconventions.org/Data/cf-conventions/cf-conventions-
1.8/cf-conventions.html#missing-data
>> > >>> And do the same for the glosea files as well.
>> > >>>
>> > >>> Thanks,
>> > >>> John
>> > >>>
>> > >>> On Mon, Feb 15, 2021 at 4:43 AM
marion.mittermaier at metoffice.gov.uk
>> > via
>> > >>> RT < met_help at ucar.edu> wrote:
>> > >>>
>> > >>> >
>> > >>> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>> > >>> >
>> > >>> > Hi all,
>> > >>> >
>> > >>> > in the mean time I have got regrid_data_plane to work in
>> stand-alone
>> > >>> > model… running the following command (and I’ve tried in
both
>> > >>> directions):
>> > >>> >
>> > >>> > regrid_data_plane glosea_2019060100_d001_001.nc
>> > >>> > gpm_2019060100_2019060200.nc grid_diag_out.nc -field
019060200.nc
>> > >>> > grid_diag_out.nc -field ‘name="precipitation_amount";
>> > level="(*,*)";'
>> > >>> > <
>> > >>> >
>> > >>> > DEBUG 1: Reading data file: glosea_2019060100_d001_001.nc
DEBUG
>> 2:
>> > >>> > Input grid: Projection: Lat/Lon Nx: 432 Ny: 324 lat_ll:
-89.722
>> > >>> > lon_ll: -0.417 delta_lat: 0.556 delta_lon: 0.833 DEBUG 2:
Output
>> > grid:
>> > >>> > Projection: Lat/Lon Nx: 3600 Ny: 1800 lat_ll:
>> > >>> > -89.950 lon_ll: 179.950 delta_lat: 0.100 delta_lon: 0.100
DEBUG 2:
>> > >>> > Interpolation options: method = NEAREST, width = 1, shape =
>> SQUARE,
>> > >>> > vld_thresh = 0.5 DEBUG 2: Range of input data
>> > >>> > (name="precipitation_amount"; level="(*,*)";) is 0 to
262.128.
>> > >>> > DEBUG 2: Range of regridded data
(name="precipitation_amount";
>> > >>> > level="(*,*)";) is 0 to 262.128.
>> > >>> > DEBUG 1: Writing output file: grid_diag_out.nc
>> > >>> >
>> > >>> > Which looks sensible… so it’s NOT regrid_data_plane.
>> > >>> >
>> > >>> > Marion
>> > >>> >
>> > >>> > From: Mittermaier, Marion
>> > >>> > Sent: 15 February 2021 11:02
>> > >>> > To: 'met_help at ucar.edu' <met_help at ucar.edu>
>> > >>> > Subject: RE: [rt.rap.ucar.edu #97766] Need help with
GridDiag
>> > >>> >
>> > >>> >
>> > >>> > Hi John,
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > it's taken me a while to get back to this... I do see the
>> difference
>> > >>> > but I still don't get it to work. I can get either on their
own to
>> > >>> > work but not together.
>> > >>> >
>> > >>> > I have split out the forecast days into separate files.
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > grid_diag -data
>> > >>> > /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc
>> > >>> > -data /scratch/fryl/GPM/gpm_2019060100_2019060200.nc
-config
>> > >>> > GridDiag.glosea -out grid_diag_out.nc -v 4
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > This gives the following error (I think it's in
>> regrid_data_plane).
>> > >>> > Not sure why it is trying to regrid both? Or maybe I’m just
not
>> quite
>> > >>> > following the messages….
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > vld497:/net/home/h02/frmm/WCSSP/India > grid_diag -data
>> > >>> > /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc
-data
>> > >>> > /scratch/fryl/GPM/gpm_2019060100_2019060200.nc
>> > >>> >
>> > >>> > DEBUG 1: Default Config File:
>> > >>> >
>> >
/data/users/cfrd/MET_requirements/MET9.0/share/met/config/GridDiagConf
>> > >>> > ig_default
>> > >>> >
>> > >>> > DEBUG 1: User Config File: GridDiag.glosea
>> > >>> >
>> > >>> > DEBUG 2: Read config files.
>> > >>> >
>> > >>> > DEBUG 2: Enter get_mtddf.
>> > >>> >
>> > >>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
>> new
>> > >>> > Met2dDataFile object of type "FileType_NcCF".
>> > >>> >
>> > >>> > DEBUG 4: NcCfFile::open() -> parsing units for the time
variable
>> > >>> > "hours since 1970-01-01"
>> > >>> >
>> > >>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF
convention
>> time
>> > >>> > unit string "hours since 1970-01-01" as a reference time of
>> > >>> > 19700101_000000 and
>> > >>> > 3600 second(s) per time step.
>> > >>> >
>> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from the
>> > >>> > "forecast_reference_time" variable.
>> > >>> >
>> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from file
>> > >>> name.
>> > >>> >
>> > >>> > DEBUG 2: Process configuration.
>> > >>> >
>> > >>> > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
>> object
>> > >>> > of type "FileType_NcCF".
>> > >>> >
>> > >>> > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
>> object
>> > >>> > of type "FileType_NcCF".
>> > >>> >
>> > >>> > DEBUG 3: Use the grid defined by file
>> "/scratch/fryl/WCSSPIndia/GPM/
>> > >>> > gpm_2019060100_2019060200.nc".
>> > >>> >
>> > >>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
>> new
>> > >>> > Met2dDataFile object of type "FileType_NcCF".
>> > >>> >
>> > >>> > DEBUG 4: NcCfFile::open() -> parsing units for the time
variable
>> > >>> > "hours since 1970-01-01"
>> > >>> >
>> > >>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF
convention
>> time
>> > >>> > unit string "hours since 1970-01-01" as a reference time of
>> > >>> > 19700101_000000 and
>> > >>> > 3600 second(s) per time step.
>> > >>> >
>> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from the
>> > >>> > "forecast_reference_time" variable.
>> > >>> >
>> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from file
>> > >>> name.
>> > >>> >
>> > >>> > DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny:
360
>> lat_ll:
>> > >>> > 4.550 lon_ll: -64.550 delta_lat: 0.100 delta_lon: 0.100
>> > >>> >
>> > >>> > DEBUG 2: Processing masking regions.
>> > >>> >
>> > >>> > DEBUG 3: Processing poly mask: India.poly
>> > >>> >
>> > >>> > DEBUG 4: parse_poly_mask() -> parsing poly mask
"India.poly"
>> > >>> >
>> > >>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
>> new
>> > >>> > Met2dDataFile object of type "FileType_None".
>> > >>> >
>> > >>> > DEBUG 4: parse_poly_mask() -> parsing poly mask file
"India.poly"
>> > >>> >
>> > >>> > DEBUG 2: Initializing precipitation_amount(*,*) histogram
>> > >>> >
>> > >>> > DEBUG 2: Initializing Precipitation_Amount(*,*) histogram
>> > >>> >
>> > >>> > DEBUG 2: Initializing
>> > >>> > precipitation_amount(*,*)_Precipitation_Amount(*,*)
>> > >>> > joint histogram
>> > >>> >
>> > >>> > DEBUG 1: grid_diag_out.nc
>> > >>> >
>> > >>> > DEBUG 1: Length of configuration "data.field" = 2
>> > >>> >
>> > >>> > DEBUG 1: Length of data file list = 1
>> > >>> >
>> > >>> > DEBUG 1: Series defined by the data file list of length 1.
>> > >>> >
>> > >>> > DEBUG 2: 1000
>> > >>> >
>> > >>> > DEBUG 2: Processing series entry 1: precipitation_amount:
>> > >>> > precipitation_amount(*,*)
>> > >>> >
>> > >>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
>> new
>> > >>> > Met2dDataFile object of type "FileType_NcCF".
>> > >>> >
>> > >>> > DEBUG 4: NcCfFile::open() -> parsing units for the time
variable
>> > >>> > "hours since 1970-01-01"
>> > >>> >
>> > >>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF
convention
>> time
>> > >>> > unit string "hours since 1970-01-01" as a reference time of
>> > >>> > 19700101_000000 and
>> > >>> > 3600 second(s) per time step.
>> > >>> >
>> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from the
>> > >>> > "forecast_reference_time" variable.
>> > >>> >
>> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from file
>> > >>> name.
>> > >>> >
>> > >>> > DEBUG 4: NcCfFile::getData() -> setting the unset init time
to the
>> > >>> > valid time of 20190601_115959.
>> > >>> >
>> > >>> > DEBUG 4:
>> > >>> >
>> > >>> > DEBUG 4: Data plane information:
>> > >>> >
>> > >>> > DEBUG 4: plane min: 0
>> > >>> >
>> > >>> > DEBUG 4: plane max: 9.96921e+36
>> > >>> >
>> > >>> > DEBUG 4: valid time: 20190601_115959
>> > >>> >
>> > >>> > DEBUG 4: lead time: 000000
>> > >>> >
>> > >>> > DEBUG 4: init time: 20190601_115959
>> > >>> >
>> > >>> > DEBUG 4: accum time: 000000
>> > >>> >
>> > >>> > DEBUG 1: Regridding field precipitation_amount(*,*) to the
>> > >>> > verification grid.
>> > >>> >
>> > >>> > DEBUG 2: Processing series entry 1: Precipitation_Amount:
>> > >>> > Precipitation_Amount(*,*)
>> > >>> >
>> > >>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
>> new
>> > >>> > Met2dDataFile object of type "FileType_NcCF".
>> > >>> >
>> > >>> > DEBUG 4: NcCfFile::open() -> parsing units for the time
variable
>> > >>> > "hours since 1970-01-01"
>> > >>> >
>> > >>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF
convention
>> time
>> > >>> > unit string "hours since 1970-01-01" as a reference time of
>> > >>> > 19700101_000000 and
>> > >>> > 3600 second(s) per time step.
>> > >>> >
>> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from the
>> > >>> > "forecast_reference_time" variable.
>> > >>> >
>> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from file
>> > >>> name.
>> > >>> >
>> > >>> > DEBUG 1: Regridding field precipitation_amount(*,*) to the
>> > >>> > verification grid.
>> > >>> >
>> > >>> > ERROR :
>> > >>> >
>> > >>> > ERROR : DataPlane::two_to_one() -> range check error: (Nx,
Ny) =
>> (0,
>> > >>> > 0), (x, y) = (0, 0)
>> > >>> >
>> > >>> > ERROR :
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > -----Original Message-----
>> > >>> > From: John Halley Gotway via RT <met_help at ucar.edu<mailto:
>> > >>> > met_help at ucar.edu>>
>> > >>> > Sent: 08 December 2020 20:54
>> > >>> > To: Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk
>> <mailto:
>> > >>> > marion.mittermaier at metoffice.gov.uk>>
>> > >>> > Subject: Re: [rt.rap.ucar.edu #97766] Need help with
GridDiag
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > This email was received from an external source. Always
check
>> > sender
>> > >>> > details, links & attachments.
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > Marion,
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > I'm still not convinced. Please take a very careful look at
the
>> > >>> > command you're executing. I see all the log messages you
sent, but
>> > not
>> > >>> > the actual command you ran. At that matters a lot!
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > This command fails on my machine with the SAME ERROR you're
>> getting:
>> > >>> >
>> > >>> > * grid_diag -data glosea_2019060100_001.nc
>> > >>> >
>> > >>> > <http://glosea_2019060100_001.nc/>
gpm_2019060100_2019060200.nc <
>> > >>> > http://gpm_2019060100_2019060200.nc/> -config
GridDiag.glosea
>> -out
>> > >>> > grid_diag_out.nc <http://grid_diag_out.nc/>*
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > ERROR : NcCfFile::getData(NcVar *, const LongArray &,
>> DataPlane
>> > &)
>> > >>> > const -> needed 3 arguments for variable
precipitation_amount,
>> got 2
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > But this command runs WITHOUT ERROR:
>> > >>> >
>> > >>> > * grid_diag -data glosea_2019060100_001.nc
>> > >>> >
>> > >>> > <http://glosea_2019060100_001.nc/> -data
>> > gpm_2019060100_2019060200.nc
>> > >>> > < http://gpm_2019060100_2019060200.nc/> -config
GridDiag.glosea
>> -out
>> > >>> > grid_diag_out.nc <http://grid_diag_out.nc/>*
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > Do you see the difference?
>> > >>> >
>> > >>> > In the first, I used "-data" once and listed 2 different
file
>> names.
>> > >>> >
>> > >>> > In the second, I used "-data" twice, once for each of the
input
>> > files.
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > Please try doing exactly the same thing.
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > Does that enable it to run without error? Note that you'll
still
>> get
>> > >>> > warnings about the actual data values falling outside the
defined
>> > >>> > range of the bins.
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > As for the bin endpoints, I think this is the relevant line
of
>> code:
>> > >>> >
>> > >>> >
>> > >>> >
>> >
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc
>> > >>> > 5981bb8/met/src/libcode/vx_series_data/series_pdf.cc#L72
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > int k = floor((value - min) / delta);
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > This places the current "value" into the k-th bin where
"min" is
>> the
>> > >>> > minimum PDF value and "delta" is the bin width.
>> > >>> >
>> > >>> > I believe this is consistent with bins being closed on the
left
>> side
>> > >>> > and open on the right:
>> > >>> >
>> > >>> > [0,1) i.e. 0 <= x < 1
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > John
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > On Tue, Dec 8, 2020 at 8:30 AM
>> marion.mittermaier at metoffice.gov.uk
>> > >>> <mailto:
>> > >>> > marion.mittermaier at metoffice.gov.uk> via RT <
met_help at ucar.edu
>> > >>> <mailto:
>> > >>> > met_help at ucar.edu>> wrote:
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > > Hi John,
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > > one final question for GridDiag.
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > > The bins are set up to be lower bound inclusive? i.e.
>> > >>> >
>> > >>> > > n_bins = 1000;
>> > >>> >
>> > >>> > > range = [0, 1000]; // kg/m^2
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > > means that the bin is set up as [0,1) i.e. 0 <= x < 1?
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > > I'm presuming so.... and the upper bound would be
[999,1000)
>> with
>> > no
>> > >>> >
>> > >>> > > bin for [1000,inf)? i.e. if there were a 1000 in my
sample it
>> would
>> > >>> >
>> > >>> > > NOT be counted? Or anything bigger.
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > > That's what it looks like, i.e. you need to make sure
that your
>> > >>> > > range
>> > >>> >
>> > >>> > > encompasses all possible values. Not sure this is in the
>> > >>> > documentation...?
>> > >>> >
>> > >>> > > Would be useful to add.
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > > Ta
>> > >>> >
>> > >>> > > Marion
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > > -----Original Message-----
>> > >>> >
>> > >>> > > From: John Halley Gotway via RT
<met_help at ucar.edu<mailto:
>> > >>> > met_help at ucar.edu>>
>> > >>> >
>> > >>> > > Sent: 05 December 2020 06:41
>> > >>> >
>> > >>> > > To: Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk
>> > <mailto:
>> > >>> > marion.mittermaier at metoffice.gov.uk>>
>> > >>> >
>> > >>> > > Subject: Re: [rt.rap.ucar.edu #97766] Need help with
GridDiag
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > > This email was received from an external source. Always
check
>> > >>> sender
>> > >>> >
>> > >>> > > details, links & attachments.
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > > Marion,
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > > Ugh, I should've just handed this ticket off to David in
the
>> first
>> > >>> place.
>> > >>> >
>> > >>> > > I did more testing and was able to get it to work. I just
had to
>> > use
>> > >>> >
>> > >>> > > separate "-data" command line options for each input data
>> source.
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > > So this command fails:
>> > >>> >
>> > >>> > > grid_diag -data glosea_2019060100_001.nc
>> > >>> > > gpm_2019060100_2019060200.nc
>> > >>> >
>> > >>> > > -config GridDiag.glosea -out grid_diag_out.nc
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > > But this command runs fine:
>> > >>> >
>> > >>> > > *grid_diag -data glosea_2019060100_001.nc <
>> > >>> >
>> > >>> > > http://glosea_2019060100_001.nc> -data
>> > gpm_2019060100_2019060200.nc
>> > >>> > > <
>> > >>> >
>> > >>> > > http://gpm_2019060100_2019060200.nc> -config
GridDiag.glosea
>> -out
>> > >>> >
>> > >>> > > grid_diag_out.nc <http://grid_diag_out.nc>*
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > > In the second one, I've specified "-data" separately for
each
>> data
>> > >>> > source.
>> > >>> >
>> > >>> > > Although, you'll see warning about the range of the data
>> extending
>> > >>> >
>> > >>> > > beyond the range of the bins defined in the config file.
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > > Sorry to have such a rambling and confusing response!
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > > So to summarize, I believe the logic is:
>> > >>> >
>> > >>> > > 1. If "-data" is used once, all specified input fields
are read
>> > from
>> > >>> >
>> > >>> > > all of those data files.
>> > >>> >
>> > >>> > > 2. If "-data" is used more than once, the number of "-
data"
>> entries
>> > >>> >
>> > >>> > > must match the number of fields defined in the config
file.
>> > >>> >
>> > >>> > > If not, you get an error message.
>> > >>> >
>> > >>> > > If so, only the i-th field is read from the files in
the
>> i-th
>> > >>> >
>> > >>> > > -data source.
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > > John
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > > On Fri, Dec 4, 2020 at 11:30 PM John Halley Gotway <
>> > johnhg at ucar.edu
>> > >>> > <mailto:johnhg at ucar.edu>>
>> > >>> >
>> > >>> > > wrote:
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > > > Marion,
>> > >>> >
>> > >>> > > >
>> > >>> >
>> > >>> > > > It occurs to me that perhaps Grid-Diag could be made
more
>> > flexible
>> > >>> >
>> > >>> > > > if we changed that error message to a debug message in
this
>> > >>> context.
>> > >>> >
>> > >>> > > > It could
>> > >>> >
>> > >>> > > > *ATTEMPT* to read all input fields from all input
files... but
>> > >>> > > > when
>> > >>> >
>> > >>> > > > it's unable to, have it print a log message stating
that
>> rather
>> > >>> > > > than
>> > >>> >
>> > >>> > > > erroring out.
>> > >>> >
>> > >>> > > >
>> > >>> >
>> > >>> > > > That would enable you to compute 2D pdf's using
different
>> > >>> > > > variables
>> > >>> >
>> > >>> > > > from different input files.
>> > >>> >
>> > >>> > > >
>> > >>> >
>> > >>> > > > Do you think that logic is preferable?
>> > >>> >
>> > >>> > > >
>> > >>> >
>> > >>> > > > Thanks,
>> > >>> >
>> > >>> > > > John
>> > >>> >
>> > >>> > > >
>> > >>> >
>> > >>> > > > On Fri, Dec 4, 2020 at 11:27 PM The RT System itself
via RT <
>> > >>> >
>> > >>> > > > met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
>> > >>> >
>> > >>> > > >
>> > >>> >
>> > >>> > > >>
>> > >>> >
>> > >>> > > >> Fri Dec 04 23:27:19 2020: Request 97766 was acted
upon.
>> > >>> >
>> > >>> > > >> Transaction: Given to johnhg (John Halley Gotway) by
>> RT_System
>> > >>> >
>> > >>> > > >> Queue: met_help
>> > >>> >
>> > >>> > > >> Subject: Need help with GridDiag
>> > >>> >
>> > >>> > > >> Owner: johnhg
>> > >>> >
>> > >>> > > >> Requestors:
marion.mittermaier at metoffice.gov.uk<mailto:
>> > >>> > marion.mittermaier at metoffice.gov.uk>
>> > >>> >
>> > >>> > > >> Status: new
>> > >>> >
>> > >>> > > >> Ticket <URL:
>> > >>> >
>> > >>> > > >>
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
>> > >>> >
>> > >>> > > >> >
>> > >>> >
>> > >>> > > >>
>> > >>> >
>> > >>> > > >>
>> > >>> >
>> > >>> > > >> This transaction appears to have no content
>> > >>> >
>> > >>> > > >>
>> > >>> >
>> > >>> > > >
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> > >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>> >
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> >
>> >
>>
>>
------------------------------------------------
Subject: Need help with GridDiag
From: John Halley Gotway
Time: Fri Mar 05 08:50:06 2021
Marion,
This issue (https://github.com/dtcenter/MET/issues/1694) is now fixed
and
merged into the develop branch. It will be included in the met-
10.0.0_beta5
development release next week as well as the official met-10.0.0
release at
the end of the month.
I'll go ahead and resolve this ticket.
Thanks again for finding this bug.
John
On Thu, Mar 4, 2021 at 5:30 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> David,
>
> FYI, I just assigned this PR to you to review the fix for this
issue:
> https://github.com/dtcenter/MET/pull/1696
>
> Thanks,
> John
>
> On Thu, Mar 4, 2021 at 10:52 AM John Halley Gotway <johnhg at ucar.edu>
> wrote:
>
>> David,
>>
>> Thanks for confirming. I know you're working on python enhancements
this
>> week. So I'll work on a fix but ask you to review the PR when I'm
ready. I
>> think it'll help get me more familiar with this code anyway.
>>
>> Thanks,
>> John
>>
>> On Thu, Mar 4, 2021 at 10:43 AM David Fillmore via RT
<met_help at ucar.edu>
>> wrote:
>>
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>>>
>>> The description of this issue is correct.
>>> The histograms are keyed by variable name, so one can not have a
>>> duplicate
>>> name from a second dataset.
>>> This was not anticipated and should be remedied.
>>> David
>>>
>>> On Thu, Mar 4, 2021 at 10:37 AM John Halley Gotway via RT <
>>> met_help at ucar.edu>
>>> wrote:
>>>
>>> >
>>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>>> >
>>> > Marion,
>>> >
>>> > Here's the GitHub issue I wrote up to describe this bug:
>>> > https://github.com/dtcenter/MET/issues/1694
>>> >
>>> > John
>>> >
>>> > On Thu, Mar 4, 2021 at 9:45 AM John Halley Gotway
<johnhg at ucar.edu>
>>> wrote:
>>> >
>>> > > Marion,
>>> > >
>>> > > FYI, I'm pretty confident the bug is caused by the fact that
we're
>>> > reading
>>> > > the same variable named "precipitation_amount" from two
different
>>> input
>>> > > data sources. When I change the variable name of the second
one to
>>> > > "precipitation_amount_MOD" then the problem goes away. Then 1D
>>> histograms
>>> > > are then correct.
>>> > >
>>> > > So that would provide a short-term, temporary bandaid for now.
You
>>> could
>>> > > nc_rename precipitaiton_amount to some other (unique) name
prior to
>>> > running
>>> > > grid_diag with met-9.1.
>>> > >
>>> > > We should just need to update the code to make the string
we're
>>> using to
>>> > > track the counts unique. And that seems very doable for met-
10.0.0.
>>> > >
>>> > > Thanks,
>>> > > John
>>> > >
>>> > > On Thu, Mar 4, 2021 at 9:30 AM John Halley Gotway
<johnhg at ucar.edu>
>>> > wrote:
>>> > >
>>> > >> Marion,
>>> > >>
>>> > >> You're right. I looked closely at the numbers that grid_diag
is
>>> > producing
>>> > >> and this is pretty obviously a bug. Thanks for finding this
and
>>> letting
>>> > us
>>> > >> know!
>>> > >>
>>> > >> I'll coordinate with David Fillmore, the engineer who
originally
>>> worked
>>> > >> on this to get this fixed for the met-10.0.0 release.
>>> > >>
>>> > >> John
>>> > >>
>>> > >> On Thu, Mar 4, 2021 at 2:29 AM
marion.mittermaier at metoffice.gov.uk
>>> via
>>> > >> RT <met_help at ucar.edu> wrote:
>>> > >>
>>> > >>>
>>> > >>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>>> > >>>
>>> > >>> Hi John,
>>> > >>>
>>> > >>> I thought I had cracked it but something is still not quite
>>> right...
>>> > the
>>> > >>> joint distribution does appear when using the dual -data
commands
>>> with
>>> > the
>>> > >>> changed attribute names, and it looks plausible BUT but the
>>> marginal
>>> > >>> distributions are wrong.
>>> > >>>
>>> > >>> If I run the GridDiag code with both data sets then the
marginal
>>> > >>> distributions are the same, which isn't right.
>>> > >>> running together
>>> > >>> glosea 171504, 18373, 11192, 7943, 6551, 4922, 4065, 3680,
>>> > >>> 3344, 2750, 2583, 1719, 2432, 1602, 1632, 1173, 1358,
899, 774,
>>> > 685,
>>> > >>> 720,
>>> > >>> 625, 782, 650, 569, 502, 377, 385, 236, 286, 244, 237,
123,
>>> 116,
>>> > >>> 156,
>>> > >>> 112, 128, 100, 151, 88, 122, 173, 109, 63, 62, 53, 88,
40, 49,
>>> 46,
>>> > >>> 84,
>>> > >>> 29, 32, 48, 41, 29, 30, 32, 36, 32, 29, 26, 24, 18, 20,
22,
>>> 20, 24,
>>> > >>> 21,
>>> > >>> 16, 15, 21, 12, 20, 16, 15, 17, 16, 7, 13, 11, 15, 8,
12, 10,
>>> 11,
>>> > >>> 11, 3,
>>> > >>> 8, 9, 4, 6, 13, 7, 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4,
5, 50,
>>> 5, 7,
>>> > >>> 6, 5,
>>> > >>> 2, 2, 53, 4, 5, 3, 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1,
4, 1,
>>> 0, 1,
>>> > >>> 1, 2,
>>> > >>> 4, 3, 2, 3, 1, 2, 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1,
3, 2,
>>> 1,
>>> > 2,
>>> > >>> 0,
>>> > >>> 0, 3, 2, 1, 2, 2, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0,
4, 0,
>>> 0,
>>> > 1,
>>> > >>> 1,
>>> > >>> 0, 2, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0,
0, 0,
>>> 0,
>>> > 0,
>>> > >>> 0,
>>> > >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>>> 0,
>>> > 0,
>>> > >>> 0,
>>> > >>> gpm 171504, 18373, 11192, 7943, 6551, 4922, 4065, 3680,
>>> > >>> 3344, 2750, 2583, 1719, 2432, 1602, 1632, 1173, 1358,
899, 774,
>>> > 685,
>>> > >>> 720,
>>> > >>> 625, 782, 650, 569, 502, 377, 385, 236, 286, 244, 237,
123,
>>> 116,
>>> > >>> 156,
>>> > >>> 112, 128, 100, 151, 88, 122, 173, 109, 63, 62, 53, 88,
40, 49,
>>> 46,
>>> > >>> 84,
>>> > >>> 29, 32, 48, 41, 29, 30, 32, 36, 32, 29, 26, 24, 18, 20,
22,
>>> 20, 24,
>>> > >>> 21,
>>> > >>> 16, 15, 21, 12, 20, 16, 15, 17, 16, 7, 13, 11, 15, 8,
12, 10,
>>> 11,
>>> > >>> 11, 3,
>>> > >>> 8, 9, 4, 6, 13, 7, 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4,
5, 50,
>>> 5, 7,
>>> > >>> 6, 5,
>>> > >>> 2, 2, 53, 4, 5, 3, 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1,
4, 1,
>>> 0, 1,
>>> > >>> 1, 2,
>>> > >>> 4, 3, 2, 3, 1, 2, 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1,
3, 2,
>>> 1,
>>> > 2,
>>> > >>> 0,
>>> > >>> 0, 3, 2, 1, 2, 2, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0,
4, 0,
>>> 0,
>>> > 1,
>>> > >>> 1,
>>> > >>> 0, 2, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0,
0, 0,
>>> 0,
>>> > 0,
>>> > >>> 0,
>>> > >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>>> 0,
>>> > 0,
>>> > >>> 0,
>>> > >>>
>>> > >>> But when I run them separately I get the following for gpm
only
>>> (same
>>> > >>> input files)
>>> > >>> 93193, 7954, 4758, 3206, 2364, 1889, 1554, 1287, 1081,
>>> > >>> 977, 824, 685, 673, 550, 520, 470, 490, 409, 391, 344,
340,
>>> 311,
>>> > >>> 278,
>>> > >>> 295, 246, 224, 195, 198, 188, 184, 175, 143, 123, 116,
116,
>>> 112,
>>> > 88,
>>> > >>> 100,
>>> > >>> 111, 88, 82, 74, 69, 63, 62, 53, 48, 40, 49, 46, 36, 29,
32,
>>> 48,
>>> > 41,
>>> > >>> 29,
>>> > >>> 30, 32, 36, 32, 29, 26, 24, 18, 20, 22, 20, 24, 21, 16,
15,
>>> 21, 12,
>>> > >>> 20,
>>> > >>> 16, 15, 17, 16, 7, 13, 11, 15, 8, 12, 10, 11, 11, 3, 8,
9, 4,
>>> 6,
>>> > 13,
>>> > >>> 7,
>>> > >>> 6, 6, 4, 6, 5, 4, 7, 1, 1, 5, 3, 4, 5, 2, 5, 7, 6, 5, 2,
2, 5,
>>> 4,
>>> > 5,
>>> > >>> 3,
>>> > >>> 2, 3, 3, 2, 0, 2, 2, 1, 1, 4, 3, 1, 4, 1, 0, 1, 1, 2, 4,
3, 2,
>>> 3,
>>> > 1,
>>> > >>> 2,
>>> > >>> 0, 0, 0, 0, 0, 1, 1, 0, 3, 1, 0, 0, 1, 3, 2, 1, 2, 0, 0,
3, 2,
>>> 1,
>>> > 2,
>>> > >>> 2,
>>> > >>> 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2, 1, 0, 4, 0, 0, 1, 1, 0,
2, 0,
>>> 1,
>>> > 0,
>>> > >>> 1,
>>> > >>> 1, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>>> 0,
>>> > 0,
>>> > >>> 0,
>>> > >>>
>>> > >>> and for glosea only
>>> > >>> 78311, 10419, 6434, 4737, 4187, 3033, 2511, 2393,
>>> > >>> 2263, 1773, 1759, 1034, 1759, 1052, 1112, 703, 868, 490,
383,
>>> 341,
>>> > >>> 380,
>>> > >>> 314, 504, 355, 323, 278, 182, 187, 48, 102, 69, 94, 0,
0, 40,
>>> 0,
>>> > 40,
>>> > >>> 0,
>>> > >>> 40, 0, 40, 99, 40, 0, 0, 0, 40, 0, 0, 0, 48, 0, 0, 0, 0,
0, 0,
>>> 0,
>>> > 0,
>>> > >>> 0,
>>> > >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>>> 0,
>>> > 0,
>>> > >>> 0,
>>> > >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>>> 0,
>>> > 0,
>>> > >>> 48,
>>> > >>> 0, 0, 0, 0, 0, 0, 48, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>>> 0, 0,
>>> > >>> 0, 0,
>>> > >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>>> 0,
>>> > 0,
>>> > >>> 0,
>>> > >>> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0,
>>> 0,
>>> > 0,
>>> > >>> 0,
>>> > >>>
>>> > >>> Looking at the first bin 93193+78311 = 171504, bin 2 is
>>> > 7953+10419=18373
>>> > >>> etc. which are the values in bins 1 and 2 for both the
marginal
>>> > >>> distributions when I input both files together. I am
wondering
>>> whether
>>> > the
>>> > >>> attribute changes are not pulled through to this somehow?
This is
>>> for
>>> > the
>>> > >>> sample data you've got and using MET9.1. My workaround for
now is
>>> that
>>> > I
>>> > >>> will have to run the code 3 times, once for the joint
distribution,
>>> > which
>>> > >>> looks ok, and twice more running it with just the single
data
>>> stream
>>> > to get
>>> > >>> the marginals for the two datasets.
>>> > >>>
>>> > >>> Regards
>>> > >>> Marion
>>> > >>>
>>> > >>> -----Original Message-----
>>> > >>> From: John Halley Gotway via RT <met_help at ucar.edu>
>>> > >>> Sent: 15 February 2021 17:38
>>> > >>> To: Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk>
>>> > >>> Subject: Re: FW: [rt.rap.ucar.edu #97766] Need help with
GridDiag
>>> > >>>
>>> > >>> This email was received from an external source. Always
check
>>> sender
>>> > >>> details, links & attachments.
>>> > >>>
>>> > >>> Marion,
>>> > >>>
>>> > >>> Thanks for sending sample data. I am a little confused
though. The
>>> > >>> config file references variables named
"precipitation_amount" and
>>> > >>> "Precipitation_Amount":
>>> > >>> * name = "precipitation_amount";*
>>> > >>> * name = "Precipitation_Amount"; *
>>> > >>>
>>> > >>> But the actual data you sent does not:
>>> > >>>
>>> > >>>
>>> > >>> * ncdump -h glosea_2019060100_d001_001.nc
>>> > >>> <http://glosea_2019060100_d001_001.nc> | grep float float
>>> > >>> precipitation_amount(latitude, longitude) ;*
>>> > >>>
>>> > >>> * ncdump -h gpm_2019060100_2019060200.nc
>>> > >>> <http://gpm_2019060100_2019060200.nc> | grep float float
>>> > >>> precipitation_amount(latitude, longitude) ;*
>>> > >>>
>>> > >>> I do believe the NetCDF is case-sensitive. So I think that's
>>> important.
>>> > >>> But I decided to just try to get Grid-Diag working on the
data you
>>> > sent. I
>>> > >>> did run into an error... different from the one you sent:
>>> > >>>
>>> > >>>
>>> > >>> *terminate called after throwing an instance of
>>> > >>> 'netCDF::exceptions::NcNameInUse' what(): NetCDF: String
match to
>>> > name in
>>> > >>> use*
>>> > >>>
>>> > >>> Grid-Diag reads the first variable from the first "-data"
source
>>> and
>>> > the
>>> > >>> second variable from the second "-data" source. And writes
data to
>>> the
>>> > >>> output NetCDF file. But since those variable names are the
same,
>>> > there's a
>>> > >>> name conflict! It's trying to write the same variable names
to the
>>> > output
>>> > >>> NetCDF file which causes the NetCDF library to error out.
>>> > >>>
>>> > >>> To get around this, I redefined the input variable names and
>>> levels in
>>> > >>> the config file using:
>>> > >>> * set_attr_name = "glosea_precip";*
>>> > >>>
>>> > >>> * set_attr_level = "sfc"; set_attr_name =
"gpm_precip";*
>>> > >>> * set_attr_level = "sfc";*
>>> > >>>
>>> > >>> That works fine and produces the attached output file. I
also
>>> attached
>>> > >>> my run command, config file, and resulting log file.
>>> > >>>
>>> > >>> Here's some improvements we could make:
>>> > >>>
>>> > >>> (1) Using set_attr_name and set_attr_level works great. But
if you
>>> look
>>> > >>> closely at the log messages, the updated strings are NOT
used
>>> there.
>>> > Seems
>>> > >>> like they should be. Do you agree?
>>> > >>>
>>> > >>> (2) The actual data values don't go much over 250... but
you've
>>> defined
>>> > >>> bins all the way out to 1000. That'll leave you with at
least 75%
>>> of
>>> > the
>>> > >>> bins empty. Can you think of any enhancements to make the
setting
>>> of
>>> > the
>>> > >>> number and size of bins easier?
>>> > >>>
>>> > >>> (3) The GPM data appears to use a value of 9.96921E36 as
missing
>>> data,
>>> > >>> but no _FillValue or missing_value variable attribute
defines
>>> that. So
>>> > MET
>>> > >>> processes considers all these pink values to be valid data.
Doesn't
>>> > matter
>>> > >>> in this case because you're only using data from India,
which has
>>> all
>>> > valid
>>> > >>> data.
>>> > >>> [image: Screen Shot 2021-02-15 at 10.31.31 AM.png] I'd
recommend
>>> using
>>> > >>> _FillValue or missing_value to define bad data values in the
GPM
>>> files:
>>> > >>>
>>> > >>>
>>> > >>>
>>> >
>>> https://cfconventions.org/Data/cf-conventions/cf-conventions-
1.8/cf-conventions.html#missing-data
>>> > >>> And do the same for the glosea files as well.
>>> > >>>
>>> > >>> Thanks,
>>> > >>> John
>>> > >>>
>>> > >>> On Mon, Feb 15, 2021 at 4:43 AM
>>> marion.mittermaier at metoffice.gov.uk
>>> > via
>>> > >>> RT < met_help at ucar.edu> wrote:
>>> > >>>
>>> > >>> >
>>> > >>> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766 >
>>> > >>> >
>>> > >>> > Hi all,
>>> > >>> >
>>> > >>> > in the mean time I have got regrid_data_plane to work in
>>> stand-alone
>>> > >>> > model… running the following command (and I’ve tried in
both
>>> > >>> directions):
>>> > >>> >
>>> > >>> > regrid_data_plane glosea_2019060100_d001_001.nc
>>> > >>> > gpm_2019060100_2019060200.nc grid_diag_out.nc -field
>>> 019060200.nc
>>> > >>> > grid_diag_out.nc -field ‘name="precipitation_amount";
>>> > level="(*,*)";'
>>> > >>> > <
>>> > >>> >
>>> > >>> > DEBUG 1: Reading data file: glosea_2019060100_d001_001.nc
DEBUG
>>> 2:
>>> > >>> > Input grid: Projection: Lat/Lon Nx: 432 Ny: 324 lat_ll:
-89.722
>>> > >>> > lon_ll: -0.417 delta_lat: 0.556 delta_lon: 0.833 DEBUG 2:
Output
>>> > grid:
>>> > >>> > Projection: Lat/Lon Nx: 3600 Ny: 1800 lat_ll:
>>> > >>> > -89.950 lon_ll: 179.950 delta_lat: 0.100 delta_lon: 0.100
DEBUG
>>> 2:
>>> > >>> > Interpolation options: method = NEAREST, width = 1, shape
=
>>> SQUARE,
>>> > >>> > vld_thresh = 0.5 DEBUG 2: Range of input data
>>> > >>> > (name="precipitation_amount"; level="(*,*)";) is 0 to
262.128.
>>> > >>> > DEBUG 2: Range of regridded data
(name="precipitation_amount";
>>> > >>> > level="(*,*)";) is 0 to 262.128.
>>> > >>> > DEBUG 1: Writing output file: grid_diag_out.nc
>>> > >>> >
>>> > >>> > Which looks sensible… so it’s NOT regrid_data_plane.
>>> > >>> >
>>> > >>> > Marion
>>> > >>> >
>>> > >>> > From: Mittermaier, Marion
>>> > >>> > Sent: 15 February 2021 11:02
>>> > >>> > To: 'met_help at ucar.edu' <met_help at ucar.edu>
>>> > >>> > Subject: RE: [rt.rap.ucar.edu #97766] Need help with
GridDiag
>>> > >>> >
>>> > >>> >
>>> > >>> > Hi John,
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > it's taken me a while to get back to this... I do see the
>>> difference
>>> > >>> > but I still don't get it to work. I can get either on
their own
>>> to
>>> > >>> > work but not together.
>>> > >>> >
>>> > >>> > I have split out the forecast days into separate files.
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > grid_diag -data
>>> > >>> > /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc
>>> > >>> > -data /scratch/fryl/GPM/gpm_2019060100_2019060200.nc
-config
>>> > >>> > GridDiag.glosea -out grid_diag_out.nc -v 4
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > This gives the following error (I think it's in
>>> regrid_data_plane).
>>> > >>> > Not sure why it is trying to regrid both? Or maybe I’m
just not
>>> quite
>>> > >>> > following the messages….
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > vld497:/net/home/h02/frmm/WCSSP/India > grid_diag -data
>>> > >>> > /scratch/frmm/wcssp/glosea5/glosea_2019060100_d001_001.nc
-data
>>> > >>> > /scratch/fryl/GPM/gpm_2019060100_2019060200.nc
>>> > >>> >
>>> > >>> > DEBUG 1: Default Config File:
>>> > >>> >
>>> >
/data/users/cfrd/MET_requirements/MET9.0/share/met/config/GridDiagConf
>>> > >>> > ig_default
>>> > >>> >
>>> > >>> > DEBUG 1: User Config File: GridDiag.glosea
>>> > >>> >
>>> > >>> > DEBUG 2: Read config files.
>>> > >>> >
>>> > >>> > DEBUG 2: Enter get_mtddf.
>>> > >>> >
>>> > >>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
>>> new
>>> > >>> > Met2dDataFile object of type "FileType_NcCF".
>>> > >>> >
>>> > >>> > DEBUG 4: NcCfFile::open() -> parsing units for the time
variable
>>> > >>> > "hours since 1970-01-01"
>>> > >>> >
>>> > >>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF
convention
>>> time
>>> > >>> > unit string "hours since 1970-01-01" as a reference time
of
>>> > >>> > 19700101_000000 and
>>> > >>> > 3600 second(s) per time step.
>>> > >>> >
>>> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from the
>>> > >>> > "forecast_reference_time" variable.
>>> > >>> >
>>> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from
>>> file
>>> > >>> name.
>>> > >>> >
>>> > >>> > DEBUG 2: Process configuration.
>>> > >>> >
>>> > >>> > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
>>> object
>>> > >>> > of type "FileType_NcCF".
>>> > >>> >
>>> > >>> > DEBUG 4: VarInfoFactory::new_var_info() -> created new
VarInfo
>>> object
>>> > >>> > of type "FileType_NcCF".
>>> > >>> >
>>> > >>> > DEBUG 3: Use the grid defined by file
>>> "/scratch/fryl/WCSSPIndia/GPM/
>>> > >>> > gpm_2019060100_2019060200.nc".
>>> > >>> >
>>> > >>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
>>> new
>>> > >>> > Met2dDataFile object of type "FileType_NcCF".
>>> > >>> >
>>> > >>> > DEBUG 4: NcCfFile::open() -> parsing units for the time
variable
>>> > >>> > "hours since 1970-01-01"
>>> > >>> >
>>> > >>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF
convention
>>> time
>>> > >>> > unit string "hours since 1970-01-01" as a reference time
of
>>> > >>> > 19700101_000000 and
>>> > >>> > 3600 second(s) per time step.
>>> > >>> >
>>> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from the
>>> > >>> > "forecast_reference_time" variable.
>>> > >>> >
>>> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from
>>> file
>>> > >>> name.
>>> > >>> >
>>> > >>> > DEBUG 3: Grid Definition: Projection: Lat/Lon Nx: 360 Ny:
360
>>> lat_ll:
>>> > >>> > 4.550 lon_ll: -64.550 delta_lat: 0.100 delta_lon: 0.100
>>> > >>> >
>>> > >>> > DEBUG 2: Processing masking regions.
>>> > >>> >
>>> > >>> > DEBUG 3: Processing poly mask: India.poly
>>> > >>> >
>>> > >>> > DEBUG 4: parse_poly_mask() -> parsing poly mask
"India.poly"
>>> > >>> >
>>> > >>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
>>> new
>>> > >>> > Met2dDataFile object of type "FileType_None".
>>> > >>> >
>>> > >>> > DEBUG 4: parse_poly_mask() -> parsing poly mask file
"India.poly"
>>> > >>> >
>>> > >>> > DEBUG 2: Initializing precipitation_amount(*,*) histogram
>>> > >>> >
>>> > >>> > DEBUG 2: Initializing Precipitation_Amount(*,*) histogram
>>> > >>> >
>>> > >>> > DEBUG 2: Initializing
>>> > >>> > precipitation_amount(*,*)_Precipitation_Amount(*,*)
>>> > >>> > joint histogram
>>> > >>> >
>>> > >>> > DEBUG 1: grid_diag_out.nc
>>> > >>> >
>>> > >>> > DEBUG 1: Length of configuration "data.field" = 2
>>> > >>> >
>>> > >>> > DEBUG 1: Length of data file list = 1
>>> > >>> >
>>> > >>> > DEBUG 1: Series defined by the data file list of length 1.
>>> > >>> >
>>> > >>> > DEBUG 2: 1000
>>> > >>> >
>>> > >>> > DEBUG 2: Processing series entry 1: precipitation_amount:
>>> > >>> > precipitation_amount(*,*)
>>> > >>> >
>>> > >>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
>>> new
>>> > >>> > Met2dDataFile object of type "FileType_NcCF".
>>> > >>> >
>>> > >>> > DEBUG 4: NcCfFile::open() -> parsing units for the time
variable
>>> > >>> > "hours since 1970-01-01"
>>> > >>> >
>>> > >>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF
convention
>>> time
>>> > >>> > unit string "hours since 1970-01-01" as a reference time
of
>>> > >>> > 19700101_000000 and
>>> > >>> > 3600 second(s) per time step.
>>> > >>> >
>>> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from the
>>> > >>> > "forecast_reference_time" variable.
>>> > >>> >
>>> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from
>>> file
>>> > >>> name.
>>> > >>> >
>>> > >>> > DEBUG 4: NcCfFile::getData() -> setting the unset init
time to
>>> the
>>> > >>> > valid time of 20190601_115959.
>>> > >>> >
>>> > >>> > DEBUG 4:
>>> > >>> >
>>> > >>> > DEBUG 4: Data plane information:
>>> > >>> >
>>> > >>> > DEBUG 4: plane min: 0
>>> > >>> >
>>> > >>> > DEBUG 4: plane max: 9.96921e+36
>>> > >>> >
>>> > >>> > DEBUG 4: valid time: 20190601_115959
>>> > >>> >
>>> > >>> > DEBUG 4: lead time: 000000
>>> > >>> >
>>> > >>> > DEBUG 4: init time: 20190601_115959
>>> > >>> >
>>> > >>> > DEBUG 4: accum time: 000000
>>> > >>> >
>>> > >>> > DEBUG 1: Regridding field precipitation_amount(*,*) to the
>>> > >>> > verification grid.
>>> > >>> >
>>> > >>> > DEBUG 2: Processing series entry 1: Precipitation_Amount:
>>> > >>> > Precipitation_Amount(*,*)
>>> > >>> >
>>> > >>> > DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() ->
created
>>> new
>>> > >>> > Met2dDataFile object of type "FileType_NcCF".
>>> > >>> >
>>> > >>> > DEBUG 4: NcCfFile::open() -> parsing units for the time
variable
>>> > >>> > "hours since 1970-01-01"
>>> > >>> >
>>> > >>> > DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF
convention
>>> time
>>> > >>> > unit string "hours since 1970-01-01" as a reference time
of
>>> > >>> > 19700101_000000 and
>>> > >>> > 3600 second(s) per time step.
>>> > >>> >
>>> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from the
>>> > >>> > "forecast_reference_time" variable.
>>> > >>> >
>>> > >>> > DEBUG 4: NcCfFile::open() -> could not extract init time
from
>>> file
>>> > >>> name.
>>> > >>> >
>>> > >>> > DEBUG 1: Regridding field precipitation_amount(*,*) to the
>>> > >>> > verification grid.
>>> > >>> >
>>> > >>> > ERROR :
>>> > >>> >
>>> > >>> > ERROR : DataPlane::two_to_one() -> range check error:
(Nx, Ny)
>>> = (0,
>>> > >>> > 0), (x, y) = (0, 0)
>>> > >>> >
>>> > >>> > ERROR :
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > -----Original Message-----
>>> > >>> > From: John Halley Gotway via RT <met_help at ucar.edu<mailto:
>>> > >>> > met_help at ucar.edu>>
>>> > >>> > Sent: 08 December 2020 20:54
>>> > >>> > To: Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk
>>> <mailto:
>>> > >>> > marion.mittermaier at metoffice.gov.uk>>
>>> > >>> > Subject: Re: [rt.rap.ucar.edu #97766] Need help with
GridDiag
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > This email was received from an external source. Always
check
>>> > sender
>>> > >>> > details, links & attachments.
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > Marion,
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > I'm still not convinced. Please take a very careful look
at the
>>> > >>> > command you're executing. I see all the log messages you
sent,
>>> but
>>> > not
>>> > >>> > the actual command you ran. At that matters a lot!
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > This command fails on my machine with the SAME ERROR
you're
>>> getting:
>>> > >>> >
>>> > >>> > * grid_diag -data glosea_2019060100_001.nc
>>> > >>> >
>>> > >>> > <http://glosea_2019060100_001.nc/>
gpm_2019060100_2019060200.nc
>>> <
>>> > >>> > http://gpm_2019060100_2019060200.nc/> -config
GridDiag.glosea
>>> -out
>>> > >>> > grid_diag_out.nc <http://grid_diag_out.nc/>*
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > ERROR : NcCfFile::getData(NcVar *, const LongArray &,
>>> DataPlane
>>> > &)
>>> > >>> > const -> needed 3 arguments for variable
precipitation_amount,
>>> got 2
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > But this command runs WITHOUT ERROR:
>>> > >>> >
>>> > >>> > * grid_diag -data glosea_2019060100_001.nc
>>> > >>> >
>>> > >>> > <http://glosea_2019060100_001.nc/> -data
>>> > gpm_2019060100_2019060200.nc
>>> > >>> > < http://gpm_2019060100_2019060200.nc/> -config
GridDiag.glosea
>>> -out
>>> > >>> > grid_diag_out.nc <http://grid_diag_out.nc/>*
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > Do you see the difference?
>>> > >>> >
>>> > >>> > In the first, I used "-data" once and listed 2 different
file
>>> names.
>>> > >>> >
>>> > >>> > In the second, I used "-data" twice, once for each of the
input
>>> > files.
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > Please try doing exactly the same thing.
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > Does that enable it to run without error? Note that you'll
still
>>> get
>>> > >>> > warnings about the actual data values falling outside the
defined
>>> > >>> > range of the bins.
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > As for the bin endpoints, I think this is the relevant
line of
>>> code:
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> >
https://github.com/dtcenter/MET/blob/17e1c9740224bf84a8bd0e84b5d254ccc
>>> > >>> > 5981bb8/met/src/libcode/vx_series_data/series_pdf.cc#L72
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > int k = floor((value - min) / delta);
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > This places the current "value" into the k-th bin where
"min" is
>>> the
>>> > >>> > minimum PDF value and "delta" is the bin width.
>>> > >>> >
>>> > >>> > I believe this is consistent with bins being closed on the
left
>>> side
>>> > >>> > and open on the right:
>>> > >>> >
>>> > >>> > [0,1) i.e. 0 <= x < 1
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > John
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > On Tue, Dec 8, 2020 at 8:30 AM
>>> marion.mittermaier at metoffice.gov.uk
>>> > >>> <mailto:
>>> > >>> > marion.mittermaier at metoffice.gov.uk> via RT <
met_help at ucar.edu
>>> > >>> <mailto:
>>> > >>> > met_help at ucar.edu>> wrote:
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
>>> >
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > > Hi John,
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > > one final question for GridDiag.
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > > The bins are set up to be lower bound inclusive? i.e.
>>> > >>> >
>>> > >>> > > n_bins = 1000;
>>> > >>> >
>>> > >>> > > range = [0, 1000]; // kg/m^2
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > > means that the bin is set up as [0,1) i.e. 0 <= x < 1?
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > > I'm presuming so.... and the upper bound would be
[999,1000)
>>> with
>>> > no
>>> > >>> >
>>> > >>> > > bin for [1000,inf)? i.e. if there were a 1000 in my
sample it
>>> would
>>> > >>> >
>>> > >>> > > NOT be counted? Or anything bigger.
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > > That's what it looks like, i.e. you need to make sure
that your
>>> > >>> > > range
>>> > >>> >
>>> > >>> > > encompasses all possible values. Not sure this is in the
>>> > >>> > documentation...?
>>> > >>> >
>>> > >>> > > Would be useful to add.
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > > Ta
>>> > >>> >
>>> > >>> > > Marion
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > > -----Original Message-----
>>> > >>> >
>>> > >>> > > From: John Halley Gotway via RT
<met_help at ucar.edu<mailto:
>>> > >>> > met_help at ucar.edu>>
>>> > >>> >
>>> > >>> > > Sent: 05 December 2020 06:41
>>> > >>> >
>>> > >>> > > To: Mittermaier, Marion
<marion.mittermaier at metoffice.gov.uk
>>> > <mailto:
>>> > >>> > marion.mittermaier at metoffice.gov.uk>>
>>> > >>> >
>>> > >>> > > Subject: Re: [rt.rap.ucar.edu #97766] Need help with
GridDiag
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > > This email was received from an external source.
Always check
>>> > >>> sender
>>> > >>> >
>>> > >>> > > details, links & attachments.
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > > Marion,
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > > Ugh, I should've just handed this ticket off to David in
the
>>> first
>>> > >>> place.
>>> > >>> >
>>> > >>> > > I did more testing and was able to get it to work. I
just had
>>> to
>>> > use
>>> > >>> >
>>> > >>> > > separate "-data" command line options for each input
data
>>> source.
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > > So this command fails:
>>> > >>> >
>>> > >>> > > grid_diag -data glosea_2019060100_001.nc
>>> > >>> > > gpm_2019060100_2019060200.nc
>>> > >>> >
>>> > >>> > > -config GridDiag.glosea -out grid_diag_out.nc
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > > But this command runs fine:
>>> > >>> >
>>> > >>> > > *grid_diag -data glosea_2019060100_001.nc <
>>> > >>> >
>>> > >>> > > http://glosea_2019060100_001.nc> -data
>>> > gpm_2019060100_2019060200.nc
>>> > >>> > > <
>>> > >>> >
>>> > >>> > > http://gpm_2019060100_2019060200.nc> -config
GridDiag.glosea
>>> -out
>>> > >>> >
>>> > >>> > > grid_diag_out.nc <http://grid_diag_out.nc>*
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > > In the second one, I've specified "-data" separately for
each
>>> data
>>> > >>> > source.
>>> > >>> >
>>> > >>> > > Although, you'll see warning about the range of the data
>>> extending
>>> > >>> >
>>> > >>> > > beyond the range of the bins defined in the config file.
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > > Sorry to have such a rambling and confusing response!
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > > So to summarize, I believe the logic is:
>>> > >>> >
>>> > >>> > > 1. If "-data" is used once, all specified input fields
are read
>>> > from
>>> > >>> >
>>> > >>> > > all of those data files.
>>> > >>> >
>>> > >>> > > 2. If "-data" is used more than once, the number of "-
data"
>>> entries
>>> > >>> >
>>> > >>> > > must match the number of fields defined in the config
file.
>>> > >>> >
>>> > >>> > > If not, you get an error message.
>>> > >>> >
>>> > >>> > > If so, only the i-th field is read from the files in
the
>>> i-th
>>> > >>> >
>>> > >>> > > -data source.
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > > John
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > > On Fri, Dec 4, 2020 at 11:30 PM John Halley Gotway <
>>> > johnhg at ucar.edu
>>> > >>> > <mailto:johnhg at ucar.edu>>
>>> > >>> >
>>> > >>> > > wrote:
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > > > Marion,
>>> > >>> >
>>> > >>> > > >
>>> > >>> >
>>> > >>> > > > It occurs to me that perhaps Grid-Diag could be made
more
>>> > flexible
>>> > >>> >
>>> > >>> > > > if we changed that error message to a debug message in
this
>>> > >>> context.
>>> > >>> >
>>> > >>> > > > It could
>>> > >>> >
>>> > >>> > > > *ATTEMPT* to read all input fields from all input
files...
>>> but
>>> > >>> > > > when
>>> > >>> >
>>> > >>> > > > it's unable to, have it print a log message stating
that
>>> rather
>>> > >>> > > > than
>>> > >>> >
>>> > >>> > > > erroring out.
>>> > >>> >
>>> > >>> > > >
>>> > >>> >
>>> > >>> > > > That would enable you to compute 2D pdf's using
different
>>> > >>> > > > variables
>>> > >>> >
>>> > >>> > > > from different input files.
>>> > >>> >
>>> > >>> > > >
>>> > >>> >
>>> > >>> > > > Do you think that logic is preferable?
>>> > >>> >
>>> > >>> > > >
>>> > >>> >
>>> > >>> > > > Thanks,
>>> > >>> >
>>> > >>> > > > John
>>> > >>> >
>>> > >>> > > >
>>> > >>> >
>>> > >>> > > > On Fri, Dec 4, 2020 at 11:27 PM The RT System itself
via RT <
>>> > >>> >
>>> > >>> > > > met_help at ucar.edu<mailto:met_help at ucar.edu>> wrote:
>>> > >>> >
>>> > >>> > > >
>>> > >>> >
>>> > >>> > > >>
>>> > >>> >
>>> > >>> > > >> Fri Dec 04 23:27:19 2020: Request 97766 was acted
upon.
>>> > >>> >
>>> > >>> > > >> Transaction: Given to johnhg (John Halley Gotway) by
>>> RT_System
>>> > >>> >
>>> > >>> > > >> Queue: met_help
>>> > >>> >
>>> > >>> > > >> Subject: Need help with GridDiag
>>> > >>> >
>>> > >>> > > >> Owner: johnhg
>>> > >>> >
>>> > >>> > > >> Requestors:
marion.mittermaier at metoffice.gov.uk<mailto:
>>> > >>> > marion.mittermaier at metoffice.gov.uk>
>>> > >>> >
>>> > >>> > > >> Status: new
>>> > >>> >
>>> > >>> > > >> Ticket <URL:
>>> > >>> >
>>> > >>> > > >>
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97766
>>> > >>> >
>>> > >>> > > >> >
>>> > >>> >
>>> > >>> > > >>
>>> > >>> >
>>> > >>> > > >>
>>> > >>> >
>>> > >>> > > >> This transaction appears to have no content
>>> > >>> >
>>> > >>> > > >>
>>> > >>> >
>>> > >>> > > >
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> > >
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>> >
>>> > >>>
>>> > >>>
>>> > >>>
>>> > >>>
>>> >
>>> >
>>>
>>>
------------------------------------------------
More information about the Met_help
mailing list