[Met_help] [rt.rap.ucar.edu #97258] History for Configuration file for Grid-stat

Tue Nov 17 15:34:22 MST 2020

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hi George,
I hope you are doing fine. Long back I asked you about the conf file for Gridstat. The link you sent was from the RegridDataPlane. However, the link doesn't work now. Is there any Gridstat conf file example available that utilizes python embedding. I want to use my python script to compare the Radar data with model output. Now that the pressure to height tool is available I need to finish the task within METplus. 
Thanks
Biswas

----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Configuration file for Grid-stat
From: George McCabe
Time: Wed Oct 28 12:49:10 2020

Hi Biswas,

I am moving this email over to MET-Help so we can track our work
easily.

I sent you the RegridDataPlane example because there are currently no
use cases in the METplus repository that call GridStat with Python
embedding, but the variables to set should be the same as
RegridDataPlane. Notably these lines need to be set correctly:

https://github.com/dtcenter/METplus/blob/f54d8305e305a0bfd11964fbf280b390bc7152b5/parm/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.conf#L34

OBS_VAR1_NAME should be set to the path of the python script you want
to call with any arguments to the script. You can use filename
template syntax to specify  time information based on the current run
time, i.e. {init?fmt=%Y%m%d%H} will be replaced with the current
initialization time in YYYYMMDDHH format. If you are using Python
embedding for forecast data, you will set FCST_VAR1_NAME instead.

https://github.com/dtcenter/METplus/blob/f54d8305e305a0bfd11964fbf280b390bc7152b5/parm/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.conf#L61

You will also need to set the appropriate template value to
PYTHON_NUMPY. For obs data, you would set OBS_GRID_STAT_INPUT_TEMPLATE
and for forecast data you would set FCST_GRID_STAT_INPUT_TEMPLATE.

Let me know if you have any issues getting this set up and I can help.

Thanks,
George

On Wed Oct 28 12:42:44 2020, mccabe wrote:
> Hi George,
>  I hope you are doing fine. Long back I asked you about the conf
file
> for Gridstat. The link you sent was from the RegridDataPlane.
However,
> the link doesn't work now. Is there any Gridstat conf file example
> available that utilizes python embedding. I want to use my python
> script to compare the Radar data with model output. Now that the
> pressure to height tool is available I need to finish the task
within
> METplus.
> Thanks
> Biswas

------------------------------------------------
Subject: Configuration file for Grid-stat
From: Mrinal Biswas
Time: Fri Oct 30 11:59:58 2020

Hi George,

Thanks for sending the example again. Let me briefly describe what I
am
trying to do, so it will be easier for you to guide me through the
process.
You probably already know this but just to re-capture.

I have a TDR (tc_radar_v2f_combined_xy_rel_merge_ships.nc). I have a
script
I have tested with this command:

plot_data_plane PYTHON_NUMPY test.ps 'name="read_tdr_lat_lon_final.py
tc_radar_v2f_combined_xy_rel_merge_ships.nc merged_zonal_wind 190901H1
2";'

Where read_tdr_lat_lon_final.py is the name of the python script,
merged_zonal_wind is the name of the observation variable, 190901H1 is
the
mission name and 2 is the height of the observation in km.

I have a NetCDF model output in height (m) which has zonal wind.

The ncdump -h looks like this:.

[Mrinal.Biswas at hfe03 HAFS]$ ncdump -h
dorian05l.2019083112.hafsprs.synoptic.interp.0p03.f024.nc4

netcdf dorian05l.2019083112.hafsprs.synoptic.interp.0p03.f024 {

dimensions:

latitude = 2001 ;

lev = 16 ;

longitude = 3501 ;

variables:

double latitude(latitude) ;

latitude:units = "degrees_north" ;

latitude:standard_name = "latitude" ;

latitude:long_name = "latitude" ;

float lev(lev) ;

lev:units = "meter" ;

double longitude(longitude) ;

longitude:units = "degrees_east" ;

longitude:standard_name = "longitude" ;

longitude:long_name = "longitude" ;

double u(lev, latitude, longitude) ;

u:GRIB_paramId = 131LL ;

u:GRIB_shortName = "u" ;

u:GRIB_units = "m s**-1" ;

u:GRIB_name = "U component of wind" ;

u:GRIB_cfName = "eastward_wind" ;

u:GRIB_cfVarName = "u" ;

u:GRIB_dataType = "fc" ;

u:GRIB_missingValue = 9999LL ;

u:GRIB_numberOfPoints = 7005501LL ;

u:GRIB_typeOfLevel = "isobaricInhPa" ;

u:GRIB_NV = 0LL ;

u:GRIB_stepUnits = 1LL ;

u:GRIB_stepType = "instant" ;

u:GRIB_gridType = "regular_ll" ;

u:GRIB_gridDefinitionDescription = "Latitude/longitude. Also called
equidistant cylindrical, or Plate Carree" ;

u:GRIB_Nx = 3501LL ;

u:GRIB_iDirectionIncrementInDegrees = 0.029999 ;

u:GRIB_iScansNegatively = 0LL ;

u:GRIB_longitudeOfFirstGridPointInDegrees = 245.5 ;

u:GRIB_longitudeOfLastGridPointInDegrees = 350.499998 ;

u:GRIB_Ny = 2001LL ;

u:GRIB_jDirectionIncrementInDegrees = 0.03 ;

u:GRIB_jPointsAreConsecutive = 0LL ;

u:GRIB_jScansPositively = 1LL ;

u:GRIB_latitudeOfFirstGridPointInDegrees = -5. ;

u:GRIB_latitudeOfLastGridPointInDegrees = 55. ;

u:long_name = "U component of wind" ;

u:units = "m s**-1" ;

u:standard_name = "eastward_wind" ;

}

I want to compare the Radar data with the model output to compute
stats
(like RMSE, ME, MAE etc). This will be a use case within METplus. I
hope
GridStat is the correct tool to do this job. I used the
RegridDataPlane
example and created a Configuration file (attached). I have questions
about
some of the variables and what they should be in the conf file. E.g.
Where
do I mention what model output variable to compare against in the conf
file? Should we use "u" or "eastward_wind"? Also I am not sure where
to put
in the arguments that are needed to read the python script. As of now
I
have included it in the OBS_VAR1_NAME.

I will really appreciate it if you can look at the conf file and see
what
corrections need to be made. And then we can diss what next steps
(wrapper
script) need to be written to test the whole case.

Thanks

Biswas

On Wed, Oct 28, 2020 at 12:49 PM George McCabe via RT
<met_help at ucar.edu>
wrote:

> Hi Biswas,
>
> I am moving this email over to MET-Help so we can track our work
easily.
>
> I sent you the RegridDataPlane example because there are currently
no use
> cases in the METplus repository that call GridStat with Python
embedding,
> but the variables to set should be the same as RegridDataPlane.
Notably
> these lines need to be set correctly:
>
>
>
https://github.com/dtcenter/METplus/blob/f54d8305e305a0bfd11964fbf280b390bc7152b5/parm/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.conf#L34
>
> OBS_VAR1_NAME should be set to the path of the python script you
want to
> call with any arguments to the script. You can use filename template
syntax
> to specify  time information based on the current run time, i.e.
> {init?fmt=%Y%m%d%H} will be replaced with the current initialization
time
> in YYYYMMDDHH format. If you are using Python embedding for forecast
data,
> you will set FCST_VAR1_NAME instead.
>
>
>
https://github.com/dtcenter/METplus/blob/f54d8305e305a0bfd11964fbf280b390bc7152b5/parm/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.conf#L61
>
> You will also need to set the appropriate template value to
PYTHON_NUMPY.
> For obs data, you would set OBS_GRID_STAT_INPUT_TEMPLATE and for
forecast
> data you would set FCST_GRID_STAT_INPUT_TEMPLATE.
>
> Let me know if you have any issues getting this set up and I can
help.
>
> Thanks,
> George
>
>
>
> On Wed Oct 28 12:42:44 2020, mccabe wrote:
> > Hi George,
> >  I hope you are doing fine. Long back I asked you about the conf
file
> > for Gridstat. The link you sent was from the RegridDataPlane.
However,
> > the link doesn't work now. Is there any Gridstat conf file example
> > available that utilizes python embedding. I want to use my python
> > script to compare the Radar data with model output. Now that the
> > pressure to height tool is available I need to finish the task
within
> > METplus.
> > Thanks
> > Biswas
>
>
>
>

--

*My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.*

------------------------------------------------
Subject: Configuration file for Grid-stat
From: George McCabe
Time: Mon Nov 02 10:32:58 2020

Hi Biswas,

I'm sorry that I wasn't clear what needed to be changed. It looks like
your
config file took the RegridDataPlane example and changed the names of
variables from including REGRID_DATA_PLANE to GRID_STAT. While the
possible
variables for GridStat and RegridDataPlane have some overlap, they do
differ, as the tools do different things. I can create an example of
using
GridStat using python embedding and send it to you so you can use that
as a
starting point to modify.

Regarding the arguments to your python script, you are correct that
you
will put them after the script name in the [FCST/OBS]_VAR<n>_NAME
variables.

If your data and scripts are available on an NCAR machine, you could
point
me there and I could take a look at them as well.

Thanks,
George

On Fri, Oct 30, 2020 at 12:00 PM Mrinal Biswas via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97258 >
>
> Hi George,
>
> Thanks for sending the example again. Let me briefly describe what I
am
> trying to do, so it will be easier for you to guide me through the
process.
> You probably already know this but just to re-capture.
>
> I have a TDR (tc_radar_v2f_combined_xy_rel_merge_ships.nc). I have a
> script
> I have tested with this command:
>
> plot_data_plane PYTHON_NUMPY test.ps
'name="read_tdr_lat_lon_final.py
> tc_radar_v2f_combined_xy_rel_merge_ships.nc merged_zonal_wind
190901H1
> 2";'
>
>
> Where read_tdr_lat_lon_final.py is the name of the python script,
> merged_zonal_wind is the name of the observation variable, 190901H1
is the
> mission name and 2 is the height of the observation in km.
>
>
> I have a NetCDF model output in height (m) which has zonal wind.
>
>
> The ncdump -h looks like this:.
>
>
> [Mrinal.Biswas at hfe03 HAFS]$ ncdump -h
> dorian05l.2019083112.hafsprs.synoptic.interp.0p03.f024.nc4
>
> netcdf dorian05l.2019083112.hafsprs.synoptic.interp.0p03.f024 {
>
> dimensions:
>
> latitude = 2001 ;
>
> lev = 16 ;
>
> longitude = 3501 ;
>
> variables:
>
> double latitude(latitude) ;
>
> latitude:units = "degrees_north" ;
>
> latitude:standard_name = "latitude" ;
>
> latitude:long_name = "latitude" ;
>
> float lev(lev) ;
>
> lev:units = "meter" ;
>
> double longitude(longitude) ;
>
> longitude:units = "degrees_east" ;
>
> longitude:standard_name = "longitude" ;
>
> longitude:long_name = "longitude" ;
>
> double u(lev, latitude, longitude) ;
>
> u:GRIB_paramId = 131LL ;
>
> u:GRIB_shortName = "u" ;
>
> u:GRIB_units = "m s**-1" ;
>
> u:GRIB_name = "U component of wind" ;
>
> u:GRIB_cfName = "eastward_wind" ;
>
> u:GRIB_cfVarName = "u" ;
>
> u:GRIB_dataType = "fc" ;
>
> u:GRIB_missingValue = 9999LL ;
>
> u:GRIB_numberOfPoints = 7005501LL ;
>
> u:GRIB_typeOfLevel = "isobaricInhPa" ;
>
> u:GRIB_NV = 0LL ;
>
> u:GRIB_stepUnits = 1LL ;
>
> u:GRIB_stepType = "instant" ;
>
> u:GRIB_gridType = "regular_ll" ;
>
> u:GRIB_gridDefinitionDescription = "Latitude/longitude. Also called
> equidistant cylindrical, or Plate Carree" ;
>
> u:GRIB_Nx = 3501LL ;
>
> u:GRIB_iDirectionIncrementInDegrees = 0.029999 ;
>
> u:GRIB_iScansNegatively = 0LL ;
>
> u:GRIB_longitudeOfFirstGridPointInDegrees = 245.5 ;
>
> u:GRIB_longitudeOfLastGridPointInDegrees = 350.499998 ;
>
> u:GRIB_Ny = 2001LL ;
>
> u:GRIB_jDirectionIncrementInDegrees = 0.03 ;
>
> u:GRIB_jPointsAreConsecutive = 0LL ;
>
> u:GRIB_jScansPositively = 1LL ;
>
> u:GRIB_latitudeOfFirstGridPointInDegrees = -5. ;
>
> u:GRIB_latitudeOfLastGridPointInDegrees = 55. ;
>
> u:long_name = "U component of wind" ;
>
> u:units = "m s**-1" ;
>
> u:standard_name = "eastward_wind" ;
>
> }
>
>
> I want to compare the Radar data with the model output to compute
stats
> (like RMSE, ME, MAE etc). This will be a use case within METplus. I
hope
> GridStat is the correct tool to do this job. I used the
RegridDataPlane
> example and created a Configuration file (attached). I have
questions about
> some of the variables and what they should be in the conf file. E.g.
Where
> do I mention what model output variable to compare against in the
conf
> file? Should we use "u" or "eastward_wind"? Also I am not sure where
to put
> in the arguments that are needed to read the python script. As of
now I
> have included it in the OBS_VAR1_NAME.
>
>
> I will really appreciate it if you can look at the conf file and see
what
> corrections need to be made. And then we can diss what next steps
(wrapper
> script) need to be written to test the whole case.
>
>
> Thanks
>
> Biswas
>
> On Wed, Oct 28, 2020 at 12:49 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Biswas,
> >
> > I am moving this email over to MET-Help so we can track our work
easily.
> >
> > I sent you the RegridDataPlane example because there are currently
no use
> > cases in the METplus repository that call GridStat with Python
embedding,
> > but the variables to set should be the same as RegridDataPlane.
Notably
> > these lines need to be set correctly:
> >
> >
> >
>
https://github.com/dtcenter/METplus/blob/f54d8305e305a0bfd11964fbf280b390bc7152b5/parm/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.conf#L34
> >
> > OBS_VAR1_NAME should be set to the path of the python script you
want to
> > call with any arguments to the script. You can use filename
template
> syntax
> > to specify  time information based on the current run time, i.e.
> > {init?fmt=%Y%m%d%H} will be replaced with the current
initialization time
> > in YYYYMMDDHH format. If you are using Python embedding for
forecast
> data,
> > you will set FCST_VAR1_NAME instead.
> >
> >
> >
>
https://github.com/dtcenter/METplus/blob/f54d8305e305a0bfd11964fbf280b390bc7152b5/parm/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.conf#L61
> >
> > You will also need to set the appropriate template value to
PYTHON_NUMPY.
> > For obs data, you would set OBS_GRID_STAT_INPUT_TEMPLATE and for
forecast
> > data you would set FCST_GRID_STAT_INPUT_TEMPLATE.
> >
> > Let me know if you have any issues getting this set up and I can
help.
> >
> > Thanks,
> > George
> >
> >
> >
> > On Wed Oct 28 12:42:44 2020, mccabe wrote:
> > > Hi George,
> > >  I hope you are doing fine. Long back I asked you about the conf
file
> > > for Gridstat. The link you sent was from the RegridDataPlane.
However,
> > > the link doesn't work now. Is there any Gridstat conf file
example
> > > available that utilizes python embedding. I want to use my
python
> > > script to compare the Radar data with model output. Now that the
> > > pressure to height tool is available I need to finish the task
within
> > > METplus.
> > > Thanks
> > > Biswas
> >
> >
> >
> >
>
> --
>
> *My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.*
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: Configuration file for Grid-stat
From: Mrinal Biswas
Time: Mon Nov 02 11:57:32 2020

Hi George,
I think I have some idea. I am using the GridStat.conf file and using
the
variables to make the conf file for METPlus. I will share it soon.
However,
reading the GridStat documentation I reda that both the model and
observation output should be in the same grid. Is my
interpretation correct? My observation and model output is of
different
resolution. Am I using the correct tool?
Thanks
Biswas

On Mon, Nov 2, 2020 at 10:33 AM George McCabe via RT
<met_help at ucar.edu>
wrote:

> Hi Biswas,
>
> I'm sorry that I wasn't clear what needed to be changed. It looks
like your
> config file took the RegridDataPlane example and changed the names
of
> variables from including REGRID_DATA_PLANE to GRID_STAT. While the
possible
> variables for GridStat and RegridDataPlane have some overlap, they
do
> differ, as the tools do different things. I can create an example of
using
> GridStat using python embedding and send it to you so you can use
that as a
> starting point to modify.
>
> Regarding the arguments to your python script, you are correct that
you
> will put them after the script name in the [FCST/OBS]_VAR<n>_NAME
> variables.
>
> If your data and scripts are available on an NCAR machine, you could
point
> me there and I could take a look at them as well.
>
> Thanks,
> George
>
> On Fri, Oct 30, 2020 at 12:00 PM Mrinal Biswas via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97258 >
> >
> > Hi George,
> >
> > Thanks for sending the example again. Let me briefly describe what
I am
> > trying to do, so it will be easier for you to guide me through the
> process.
> > You probably already know this but just to re-capture.
> >
> > I have a TDR (tc_radar_v2f_combined_xy_rel_merge_ships.nc). I have
a
> > script
> > I have tested with this command:
> >
> > plot_data_plane PYTHON_NUMPY test.ps
'name="read_tdr_lat_lon_final.py
> > tc_radar_v2f_combined_xy_rel_merge_ships.nc merged_zonal_wind
190901H1
> > 2";'
> >
> >
> > Where read_tdr_lat_lon_final.py is the name of the python script,
> > merged_zonal_wind is the name of the observation variable,
190901H1 is
> the
> > mission name and 2 is the height of the observation in km.
> >
> >
> > I have a NetCDF model output in height (m) which has zonal wind.
> >
> >
> > The ncdump -h looks like this:.
> >
> >
> > [Mrinal.Biswas at hfe03 HAFS]$ ncdump -h
> > dorian05l.2019083112.hafsprs.synoptic.interp.0p03.f024.nc4
> >
> > netcdf dorian05l.2019083112.hafsprs.synoptic.interp.0p03.f024 {
> >
> > dimensions:
> >
> > latitude = 2001 ;
> >
> > lev = 16 ;
> >
> > longitude = 3501 ;
> >
> > variables:
> >
> > double latitude(latitude) ;
> >
> > latitude:units = "degrees_north" ;
> >
> > latitude:standard_name = "latitude" ;
> >
> > latitude:long_name = "latitude" ;
> >
> > float lev(lev) ;
> >
> > lev:units = "meter" ;
> >
> > double longitude(longitude) ;
> >
> > longitude:units = "degrees_east" ;
> >
> > longitude:standard_name = "longitude" ;
> >
> > longitude:long_name = "longitude" ;
> >
> > double u(lev, latitude, longitude) ;
> >
> > u:GRIB_paramId = 131LL ;
> >
> > u:GRIB_shortName = "u" ;
> >
> > u:GRIB_units = "m s**-1" ;
> >
> > u:GRIB_name = "U component of wind" ;
> >
> > u:GRIB_cfName = "eastward_wind" ;
> >
> > u:GRIB_cfVarName = "u" ;
> >
> > u:GRIB_dataType = "fc" ;
> >
> > u:GRIB_missingValue = 9999LL ;
> >
> > u:GRIB_numberOfPoints = 7005501LL ;
> >
> > u:GRIB_typeOfLevel = "isobaricInhPa" ;
> >
> > u:GRIB_NV = 0LL ;
> >
> > u:GRIB_stepUnits = 1LL ;
> >
> > u:GRIB_stepType = "instant" ;
> >
> > u:GRIB_gridType = "regular_ll" ;
> >
> > u:GRIB_gridDefinitionDescription = "Latitude/longitude. Also
called
> > equidistant cylindrical, or Plate Carree" ;
> >
> > u:GRIB_Nx = 3501LL ;
> >
> > u:GRIB_iDirectionIncrementInDegrees = 0.029999 ;
> >
> > u:GRIB_iScansNegatively = 0LL ;
> >
> > u:GRIB_longitudeOfFirstGridPointInDegrees = 245.5 ;
> >
> > u:GRIB_longitudeOfLastGridPointInDegrees = 350.499998 ;
> >
> > u:GRIB_Ny = 2001LL ;
> >
> > u:GRIB_jDirectionIncrementInDegrees = 0.03 ;
> >
> > u:GRIB_jPointsAreConsecutive = 0LL ;
> >
> > u:GRIB_jScansPositively = 1LL ;
> >
> > u:GRIB_latitudeOfFirstGridPointInDegrees = -5. ;
> >
> > u:GRIB_latitudeOfLastGridPointInDegrees = 55. ;
> >
> > u:long_name = "U component of wind" ;
> >
> > u:units = "m s**-1" ;
> >
> > u:standard_name = "eastward_wind" ;
> >
> > }
> >
> >
> > I want to compare the Radar data with the model output to compute
stats
> > (like RMSE, ME, MAE etc). This will be a use case within METplus.
I hope
> > GridStat is the correct tool to do this job. I used the
RegridDataPlane
> > example and created a Configuration file (attached). I have
questions
> about
> > some of the variables and what they should be in the conf file.
E.g.
> Where
> > do I mention what model output variable to compare against in the
conf
> > file? Should we use "u" or "eastward_wind"? Also I am not sure
where to
> put
> > in the arguments that are needed to read the python script. As of
now I
> > have included it in the OBS_VAR1_NAME.
> >
> >
> > I will really appreciate it if you can look at the conf file and
see what
> > corrections need to be made. And then we can diss what next steps
> (wrapper
> > script) need to be written to test the whole case.
> >
> >
> > Thanks
> >
> > Biswas
> >
> > On Wed, Oct 28, 2020 at 12:49 PM George McCabe via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> > > Hi Biswas,
> > >
> > > I am moving this email over to MET-Help so we can track our work
> easily.
> > >
> > > I sent you the RegridDataPlane example because there are
currently no
> use
> > > cases in the METplus repository that call GridStat with Python
> embedding,
> > > but the variables to set should be the same as RegridDataPlane.
Notably
> > > these lines need to be set correctly:
> > >
> > >
> > >
> >
>
https://github.com/dtcenter/METplus/blob/f54d8305e305a0bfd11964fbf280b390bc7152b5/parm/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.conf#L34
> > >
> > > OBS_VAR1_NAME should be set to the path of the python script you
want
> to
> > > call with any arguments to the script. You can use filename
template
> > syntax
> > > to specify  time information based on the current run time, i.e.
> > > {init?fmt=%Y%m%d%H} will be replaced with the current
initialization
> time
> > > in YYYYMMDDHH format. If you are using Python embedding for
forecast
> > data,
> > > you will set FCST_VAR1_NAME instead.
> > >
> > >
> > >
> >
>
https://github.com/dtcenter/METplus/blob/f54d8305e305a0bfd11964fbf280b390bc7152b5/parm/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.conf#L61
> > >
> > > You will also need to set the appropriate template value to
> PYTHON_NUMPY.
> > > For obs data, you would set OBS_GRID_STAT_INPUT_TEMPLATE and for
> forecast
> > > data you would set FCST_GRID_STAT_INPUT_TEMPLATE.
> > >
> > > Let me know if you have any issues getting this set up and I can
help.
> > >
> > > Thanks,
> > > George
> > >
> > >
> > >
> > > On Wed Oct 28 12:42:44 2020, mccabe wrote:
> > > > Hi George,
> > > >  I hope you are doing fine. Long back I asked you about the
conf file
> > > > for Gridstat. The link you sent was from the RegridDataPlane.
> However,
> > > > the link doesn't work now. Is there any Gridstat conf file
example
> > > > available that utilizes python embedding. I want to use my
python
> > > > script to compare the Radar data with model output. Now that
the
> > > > pressure to height tool is available I need to finish the task
within
> > > > METplus.
> > > > Thanks
> > > > Biswas
> > >
> > >
> > >
> > >
> >
> > --
> >
> > *My working day may not be your working day. Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.*
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

--

*My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.*

------------------------------------------------
Subject: Configuration file for Grid-stat
From: George McCabe
Time: Mon Nov 02 12:03:46 2020

Hi Biswas,

You can configure GridStat to regrid the data to a common grid for
comparisons. If you are on the NCAR VPN, you can see the example I
created
for GridStat with python embedding. It is not yet in the repository,
as it
is being reviewed by other developers, but you can see what it looks
like
here:

http://icculus.rap.ucar.edu/metplus/generated/met_tool_wrapper/GridStat/GridStat_python_embedding.html#metplus-
configuration

The relevant line is:
GRID_STAT_REGRID_TO_GRID = G130

In this example we regrid both datasets to G130. You can also set it
to
regrid the obs to the forecast grid, vice versa, or a grid based on a
file
path. Go to this page and search to "to_grid" for more information on
that
setting:

https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html?highlight=to_grid

On Mon, Nov 2, 2020 at 11:57 AM Mrinal Biswas via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97258 >
>
> Hi George,
> I think I have some idea. I am using the GridStat.conf file and
using the
> variables to make the conf file for METPlus. I will share it soon.
However,
> reading the GridStat documentation I reda that both the model and
> observation output should be in the same grid. Is my
> interpretation correct? My observation and model output is of
different
> resolution. Am I using the correct tool?
> Thanks
> Biswas
>
> On Mon, Nov 2, 2020 at 10:33 AM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Biswas,
> >
> > I'm sorry that I wasn't clear what needed to be changed. It looks
like
> your
> > config file took the RegridDataPlane example and changed the names
of
> > variables from including REGRID_DATA_PLANE to GRID_STAT. While the
> possible
> > variables for GridStat and RegridDataPlane have some overlap, they
do
> > differ, as the tools do different things. I can create an example
of
> using
> > GridStat using python embedding and send it to you so you can use
that
> as a
> > starting point to modify.
> >
> > Regarding the arguments to your python script, you are correct
that you
> > will put them after the script name in the [FCST/OBS]_VAR<n>_NAME
> > variables.
> >
> > If your data and scripts are available on an NCAR machine, you
could
> point
> > me there and I could take a look at them as well.
> >
> > Thanks,
> > George
> >
> > On Fri, Oct 30, 2020 at 12:00 PM Mrinal Biswas via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97258 >
> > >
> > > Hi George,
> > >
> > > Thanks for sending the example again. Let me briefly describe
what I am
> > > trying to do, so it will be easier for you to guide me through
the
> > process.
> > > You probably already know this but just to re-capture.
> > >
> > > I have a TDR (tc_radar_v2f_combined_xy_rel_merge_ships.nc). I
have a
> > > script
> > > I have tested with this command:
> > >
> > > plot_data_plane PYTHON_NUMPY test.ps
'name="read_tdr_lat_lon_final.py
> > > tc_radar_v2f_combined_xy_rel_merge_ships.nc merged_zonal_wind
190901H1
> > > 2";'
> > >
> > >
> > > Where read_tdr_lat_lon_final.py is the name of the python
script,
> > > merged_zonal_wind is the name of the observation variable,
190901H1 is
> > the
> > > mission name and 2 is the height of the observation in km.
> > >
> > >
> > > I have a NetCDF model output in height (m) which has zonal wind.
> > >
> > >
> > > The ncdump -h looks like this:.
> > >
> > >
> > > [Mrinal.Biswas at hfe03 HAFS]$ ncdump -h
> > > dorian05l.2019083112.hafsprs.synoptic.interp.0p03.f024.nc4
> > >
> > > netcdf dorian05l.2019083112.hafsprs.synoptic.interp.0p03.f024 {
> > >
> > > dimensions:
> > >
> > > latitude = 2001 ;
> > >
> > > lev = 16 ;
> > >
> > > longitude = 3501 ;
> > >
> > > variables:
> > >
> > > double latitude(latitude) ;
> > >
> > > latitude:units = "degrees_north" ;
> > >
> > > latitude:standard_name = "latitude" ;
> > >
> > > latitude:long_name = "latitude" ;
> > >
> > > float lev(lev) ;
> > >
> > > lev:units = "meter" ;
> > >
> > > double longitude(longitude) ;
> > >
> > > longitude:units = "degrees_east" ;
> > >
> > > longitude:standard_name = "longitude" ;
> > >
> > > longitude:long_name = "longitude" ;
> > >
> > > double u(lev, latitude, longitude) ;
> > >
> > > u:GRIB_paramId = 131LL ;
> > >
> > > u:GRIB_shortName = "u" ;
> > >
> > > u:GRIB_units = "m s**-1" ;
> > >
> > > u:GRIB_name = "U component of wind" ;
> > >
> > > u:GRIB_cfName = "eastward_wind" ;
> > >
> > > u:GRIB_cfVarName = "u" ;
> > >
> > > u:GRIB_dataType = "fc" ;
> > >
> > > u:GRIB_missingValue = 9999LL ;
> > >
> > > u:GRIB_numberOfPoints = 7005501LL ;
> > >
> > > u:GRIB_typeOfLevel = "isobaricInhPa" ;
> > >
> > > u:GRIB_NV = 0LL ;
> > >
> > > u:GRIB_stepUnits = 1LL ;
> > >
> > > u:GRIB_stepType = "instant" ;
> > >
> > > u:GRIB_gridType = "regular_ll" ;
> > >
> > > u:GRIB_gridDefinitionDescription = "Latitude/longitude. Also
called
> > > equidistant cylindrical, or Plate Carree" ;
> > >
> > > u:GRIB_Nx = 3501LL ;
> > >
> > > u:GRIB_iDirectionIncrementInDegrees = 0.029999 ;
> > >
> > > u:GRIB_iScansNegatively = 0LL ;
> > >
> > > u:GRIB_longitudeOfFirstGridPointInDegrees = 245.5 ;
> > >
> > > u:GRIB_longitudeOfLastGridPointInDegrees = 350.499998 ;
> > >
> > > u:GRIB_Ny = 2001LL ;
> > >
> > > u:GRIB_jDirectionIncrementInDegrees = 0.03 ;
> > >
> > > u:GRIB_jPointsAreConsecutive = 0LL ;
> > >
> > > u:GRIB_jScansPositively = 1LL ;
> > >
> > > u:GRIB_latitudeOfFirstGridPointInDegrees = -5. ;
> > >
> > > u:GRIB_latitudeOfLastGridPointInDegrees = 55. ;
> > >
> > > u:long_name = "U component of wind" ;
> > >
> > > u:units = "m s**-1" ;
> > >
> > > u:standard_name = "eastward_wind" ;
> > >
> > > }
> > >
> > >
> > > I want to compare the Radar data with the model output to
compute stats
> > > (like RMSE, ME, MAE etc). This will be a use case within
METplus. I
> hope
> > > GridStat is the correct tool to do this job. I used the
RegridDataPlane
> > > example and created a Configuration file (attached). I have
questions
> > about
> > > some of the variables and what they should be in the conf file.
E.g.
> > Where
> > > do I mention what model output variable to compare against in
the conf
> > > file? Should we use "u" or "eastward_wind"? Also I am not sure
where to
> > put
> > > in the arguments that are needed to read the python script. As
of now I
> > > have included it in the OBS_VAR1_NAME.
> > >
> > >
> > > I will really appreciate it if you can look at the conf file and
see
> what
> > > corrections need to be made. And then we can diss what next
steps
> > (wrapper
> > > script) need to be written to test the whole case.
> > >
> > >
> > > Thanks
> > >
> > > Biswas
> > >
> > > On Wed, Oct 28, 2020 at 12:49 PM George McCabe via RT <
> met_help at ucar.edu
> > >
> > > wrote:
> > >
> > > > Hi Biswas,
> > > >
> > > > I am moving this email over to MET-Help so we can track our
work
> > easily.
> > > >
> > > > I sent you the RegridDataPlane example because there are
currently no
> > use
> > > > cases in the METplus repository that call GridStat with Python
> > embedding,
> > > > but the variables to set should be the same as
RegridDataPlane.
> Notably
> > > > these lines need to be set correctly:
> > > >
> > > >
> > > >
> > >
> >
>
https://github.com/dtcenter/METplus/blob/f54d8305e305a0bfd11964fbf280b390bc7152b5/parm/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.conf#L34
> > > >
> > > > OBS_VAR1_NAME should be set to the path of the python script
you want
> > to
> > > > call with any arguments to the script. You can use filename
template
> > > syntax
> > > > to specify  time information based on the current run time,
i.e.
> > > > {init?fmt=%Y%m%d%H} will be replaced with the current
initialization
> > time
> > > > in YYYYMMDDHH format. If you are using Python embedding for
forecast
> > > data,
> > > > you will set FCST_VAR1_NAME instead.
> > > >
> > > >
> > > >
> > >
> >
>
https://github.com/dtcenter/METplus/blob/f54d8305e305a0bfd11964fbf280b390bc7152b5/parm/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.conf#L61
> > > >
> > > > You will also need to set the appropriate template value to
> > PYTHON_NUMPY.
> > > > For obs data, you would set OBS_GRID_STAT_INPUT_TEMPLATE and
for
> > forecast
> > > > data you would set FCST_GRID_STAT_INPUT_TEMPLATE.
> > > >
> > > > Let me know if you have any issues getting this set up and I
can
> help.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > >
> > > >
> > > > On Wed Oct 28 12:42:44 2020, mccabe wrote:
> > > > > Hi George,
> > > > >  I hope you are doing fine. Long back I asked you about the
conf
> file
> > > > > for Gridstat. The link you sent was from the
RegridDataPlane.
> > However,
> > > > > the link doesn't work now. Is there any Gridstat conf file
example
> > > > > available that utilizes python embedding. I want to use my
python
> > > > > script to compare the Radar data with model output. Now that
the
> > > > > pressure to height tool is available I need to finish the
task
> within
> > > > > METplus.
> > > > > Thanks
> > > > > Biswas
> > > >
> > > >
> > > >
> > > >
> > >
> > > --
> > >
> > > *My working day may not be your working day. Please do not feel
obliged
> > to
> > > reply to this email outside of your normal working hours.*
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
>
> *My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.*
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: Configuration file for Grid-stat
From: Mrinal Biswas
Time: Mon Nov 02 13:02:26 2020

Hi George,
Many thanks for the info. After my corrections to the conf file it
looks
similar to the example you have created. I have put the input data,
the
python script, model output and the GridStat_python_enbedding.conf
file on
eyewall (/d1/biswas/HRDdata-MET/mccabe). Please let me know if it
looks ok
to you.
Thanks for all your help
Biswas

On Mon, Nov 2, 2020 at 12:04 PM George McCabe via RT
<met_help at ucar.edu>
wrote:

> Hi Biswas,
>
> You can configure GridStat to regrid the data to a common grid for
> comparisons. If you are on the NCAR VPN, you can see the example I
created
> for GridStat with python embedding. It is not yet in the repository,
as it
> is being reviewed by other developers, but you can see what it looks
like
> here:
>
>
>
http://icculus.rap.ucar.edu/metplus/generated/met_tool_wrapper/GridStat/GridStat_python_embedding.html#metplus-
configuration
>
> The relevant line is:
> GRID_STAT_REGRID_TO_GRID = G130
>
> In this example we regrid both datasets to G130. You can also set it
to
> regrid the obs to the forecast grid, vice versa, or a grid based on
a file
> path. Go to this page and search to "to_grid" for more information
on that
> setting:
>
>
>
https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html?highlight=to_grid
>
>
>
> On Mon, Nov 2, 2020 at 11:57 AM Mrinal Biswas via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97258 >
> >
> > Hi George,
> > I think I have some idea. I am using the GridStat.conf file and
using the
> > variables to make the conf file for METPlus. I will share it soon.
> However,
> > reading the GridStat documentation I reda that both the model and
> > observation output should be in the same grid. Is my
> > interpretation correct? My observation and model output is of
different
> > resolution. Am I using the correct tool?
> > Thanks
> > Biswas
> >
> > On Mon, Nov 2, 2020 at 10:33 AM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Biswas,
> > >
> > > I'm sorry that I wasn't clear what needed to be changed. It
looks like
> > your
> > > config file took the RegridDataPlane example and changed the
names of
> > > variables from including REGRID_DATA_PLANE to GRID_STAT. While
the
> > possible
> > > variables for GridStat and RegridDataPlane have some overlap,
they do
> > > differ, as the tools do different things. I can create an
example of
> > using
> > > GridStat using python embedding and send it to you so you can
use that
> > as a
> > > starting point to modify.
> > >
> > > Regarding the arguments to your python script, you are correct
that you
> > > will put them after the script name in the
[FCST/OBS]_VAR<n>_NAME
> > > variables.
> > >
> > > If your data and scripts are available on an NCAR machine, you
could
> > point
> > > me there and I could take a look at them as well.
> > >
> > > Thanks,
> > > George
> > >
> > > On Fri, Oct 30, 2020 at 12:00 PM Mrinal Biswas via RT <
> met_help at ucar.edu
> > >
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97258
>
> > > >
> > > > Hi George,
> > > >
> > > > Thanks for sending the example again. Let me briefly describe
what I
> am
> > > > trying to do, so it will be easier for you to guide me through
the
> > > process.
> > > > You probably already know this but just to re-capture.
> > > >
> > > > I have a TDR (tc_radar_v2f_combined_xy_rel_merge_ships.nc). I
have a
> > > > script
> > > > I have tested with this command:
> > > >
> > > > plot_data_plane PYTHON_NUMPY test.ps
> 'name="read_tdr_lat_lon_final.py
> > > > tc_radar_v2f_combined_xy_rel_merge_ships.nc merged_zonal_wind
> 190901H1
> > > > 2";'
> > > >
> > > >
> > > > Where read_tdr_lat_lon_final.py is the name of the python
script,
> > > > merged_zonal_wind is the name of the observation variable,
190901H1
> is
> > > the
> > > > mission name and 2 is the height of the observation in km.
> > > >
> > > >
> > > > I have a NetCDF model output in height (m) which has zonal
wind.
> > > >
> > > >
> > > > The ncdump -h looks like this:.
> > > >
> > > >
> > > > [Mrinal.Biswas at hfe03 HAFS]$ ncdump -h
> > > > dorian05l.2019083112.hafsprs.synoptic.interp.0p03.f024.nc4
> > > >
> > > > netcdf dorian05l.2019083112.hafsprs.synoptic.interp.0p03.f024
{
> > > >
> > > > dimensions:
> > > >
> > > > latitude = 2001 ;
> > > >
> > > > lev = 16 ;
> > > >
> > > > longitude = 3501 ;
> > > >
> > > > variables:
> > > >
> > > > double latitude(latitude) ;
> > > >
> > > > latitude:units = "degrees_north" ;
> > > >
> > > > latitude:standard_name = "latitude" ;
> > > >
> > > > latitude:long_name = "latitude" ;
> > > >
> > > > float lev(lev) ;
> > > >
> > > > lev:units = "meter" ;
> > > >
> > > > double longitude(longitude) ;
> > > >
> > > > longitude:units = "degrees_east" ;
> > > >
> > > > longitude:standard_name = "longitude" ;
> > > >
> > > > longitude:long_name = "longitude" ;
> > > >
> > > > double u(lev, latitude, longitude) ;
> > > >
> > > > u:GRIB_paramId = 131LL ;
> > > >
> > > > u:GRIB_shortName = "u" ;
> > > >
> > > > u:GRIB_units = "m s**-1" ;
> > > >
> > > > u:GRIB_name = "U component of wind" ;
> > > >
> > > > u:GRIB_cfName = "eastward_wind" ;
> > > >
> > > > u:GRIB_cfVarName = "u" ;
> > > >
> > > > u:GRIB_dataType = "fc" ;
> > > >
> > > > u:GRIB_missingValue = 9999LL ;
> > > >
> > > > u:GRIB_numberOfPoints = 7005501LL ;
> > > >
> > > > u:GRIB_typeOfLevel = "isobaricInhPa" ;
> > > >
> > > > u:GRIB_NV = 0LL ;
> > > >
> > > > u:GRIB_stepUnits = 1LL ;
> > > >
> > > > u:GRIB_stepType = "instant" ;
> > > >
> > > > u:GRIB_gridType = "regular_ll" ;
> > > >
> > > > u:GRIB_gridDefinitionDescription = "Latitude/longitude. Also
called
> > > > equidistant cylindrical, or Plate Carree" ;
> > > >
> > > > u:GRIB_Nx = 3501LL ;
> > > >
> > > > u:GRIB_iDirectionIncrementInDegrees = 0.029999 ;
> > > >
> > > > u:GRIB_iScansNegatively = 0LL ;
> > > >
> > > > u:GRIB_longitudeOfFirstGridPointInDegrees = 245.5 ;
> > > >
> > > > u:GRIB_longitudeOfLastGridPointInDegrees = 350.499998 ;
> > > >
> > > > u:GRIB_Ny = 2001LL ;
> > > >
> > > > u:GRIB_jDirectionIncrementInDegrees = 0.03 ;
> > > >
> > > > u:GRIB_jPointsAreConsecutive = 0LL ;
> > > >
> > > > u:GRIB_jScansPositively = 1LL ;
> > > >
> > > > u:GRIB_latitudeOfFirstGridPointInDegrees = -5. ;
> > > >
> > > > u:GRIB_latitudeOfLastGridPointInDegrees = 55. ;
> > > >
> > > > u:long_name = "U component of wind" ;
> > > >
> > > > u:units = "m s**-1" ;
> > > >
> > > > u:standard_name = "eastward_wind" ;
> > > >
> > > > }
> > > >
> > > >
> > > > I want to compare the Radar data with the model output to
compute
> stats
> > > > (like RMSE, ME, MAE etc). This will be a use case within
METplus. I
> > hope
> > > > GridStat is the correct tool to do this job. I used the
> RegridDataPlane
> > > > example and created a Configuration file (attached). I have
questions
> > > about
> > > > some of the variables and what they should be in the conf
file. E.g.
> > > Where
> > > > do I mention what model output variable to compare against in
the
> conf
> > > > file? Should we use "u" or "eastward_wind"? Also I am not sure
where
> to
> > > put
> > > > in the arguments that are needed to read the python script. As
of
> now I
> > > > have included it in the OBS_VAR1_NAME.
> > > >
> > > >
> > > > I will really appreciate it if you can look at the conf file
and see
> > what
> > > > corrections need to be made. And then we can diss what next
steps
> > > (wrapper
> > > > script) need to be written to test the whole case.
> > > >
> > > >
> > > > Thanks
> > > >
> > > > Biswas
> > > >
> > > > On Wed, Oct 28, 2020 at 12:49 PM George McCabe via RT <
> > met_help at ucar.edu
> > > >
> > > > wrote:
> > > >
> > > > > Hi Biswas,
> > > > >
> > > > > I am moving this email over to MET-Help so we can track our
work
> > > easily.
> > > > >
> > > > > I sent you the RegridDataPlane example because there are
currently
> no
> > > use
> > > > > cases in the METplus repository that call GridStat with
Python
> > > embedding,
> > > > > but the variables to set should be the same as
RegridDataPlane.
> > Notably
> > > > > these lines need to be set correctly:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
https://github.com/dtcenter/METplus/blob/f54d8305e305a0bfd11964fbf280b390bc7152b5/parm/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.conf#L34
> > > > >
> > > > > OBS_VAR1_NAME should be set to the path of the python script
you
> want
> > > to
> > > > > call with any arguments to the script. You can use filename
> template
> > > > syntax
> > > > > to specify  time information based on the current run time,
i.e.
> > > > > {init?fmt=%Y%m%d%H} will be replaced with the current
> initialization
> > > time
> > > > > in YYYYMMDDHH format. If you are using Python embedding for
> forecast
> > > > data,
> > > > > you will set FCST_VAR1_NAME instead.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
https://github.com/dtcenter/METplus/blob/f54d8305e305a0bfd11964fbf280b390bc7152b5/parm/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.conf#L61
> > > > >
> > > > > You will also need to set the appropriate template value to
> > > PYTHON_NUMPY.
> > > > > For obs data, you would set OBS_GRID_STAT_INPUT_TEMPLATE and
for
> > > forecast
> > > > > data you would set FCST_GRID_STAT_INPUT_TEMPLATE.
> > > > >
> > > > > Let me know if you have any issues getting this set up and I
can
> > help.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > >
> > > > >
> > > > > On Wed Oct 28 12:42:44 2020, mccabe wrote:
> > > > > > Hi George,
> > > > > >  I hope you are doing fine. Long back I asked you about
the conf
> > file
> > > > > > for Gridstat. The link you sent was from the
RegridDataPlane.
> > > However,
> > > > > > the link doesn't work now. Is there any Gridstat conf file
> example
> > > > > > available that utilizes python embedding. I want to use my
python
> > > > > > script to compare the Radar data with model output. Now
that the
> > > > > > pressure to height tool is available I need to finish the
task
> > within
> > > > > > METplus.
> > > > > > Thanks
> > > > > > Biswas
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > > *My working day may not be your working day. Please do not
feel
> obliged
> > > to
> > > > reply to this email outside of your normal working hours.*
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> > --
> >
> > *My working day may not be your working day. Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.*
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

--

*My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.*

------------------------------------------------
Subject: Configuration file for Grid-stat
From: George McCabe
Time: Mon Nov 02 13:12:22 2020

Hi Biswas,

Am I understanding correctly that you are using Python embedding to
read
the observation data and passing in the model data directly? If so,
your
use case conf should refer to the GridStat.conf example to read the
forecast data properly. I am working on another task now, but I can
help
you get this configured for your case when I wrap that work up.

Thanks,
George

On Mon, Nov 2, 2020 at 1:02 PM Mrinal Biswas via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97258 >
>
> Hi George,
> Many thanks for the info. After my corrections to the conf file it
looks
> similar to the example you have created. I have put the input data,
the
> python script, model output and the GridStat_python_enbedding.conf
file on
> eyewall (/d1/biswas/HRDdata-MET/mccabe). Please let me know if it
looks ok
> to you.
> Thanks for all your help
> Biswas
>
> On Mon, Nov 2, 2020 at 12:04 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Biswas,
> >
> > You can configure GridStat to regrid the data to a common grid for
> > comparisons. If you are on the NCAR VPN, you can see the example I
> created
> > for GridStat with python embedding. It is not yet in the
repository, as
> it
> > is being reviewed by other developers, but you can see what it
looks like
> > here:
> >
> >
> >
>
http://icculus.rap.ucar.edu/metplus/generated/met_tool_wrapper/GridStat/GridStat_python_embedding.html#metplus-
configuration
> >
> > The relevant line is:
> > GRID_STAT_REGRID_TO_GRID = G130
> >
> > In this example we regrid both datasets to G130. You can also set
it to
> > regrid the obs to the forecast grid, vice versa, or a grid based
on a
> file
> > path. Go to this page and search to "to_grid" for more information
on
> that
> > setting:
> >
> >
> >
>
https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html?highlight=to_grid
> >
> >
> >
> > On Mon, Nov 2, 2020 at 11:57 AM Mrinal Biswas via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97258 >
> > >
> > > Hi George,
> > > I think I have some idea. I am using the GridStat.conf file and
using
> the
> > > variables to make the conf file for METPlus. I will share it
soon.
> > However,
> > > reading the GridStat documentation I reda that both the model
and
> > > observation output should be in the same grid. Is my
> > > interpretation correct? My observation and model output is of
different
> > > resolution. Am I using the correct tool?
> > > Thanks
> > > Biswas
> > >
> > > On Mon, Nov 2, 2020 at 10:33 AM George McCabe via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Biswas,
> > > >
> > > > I'm sorry that I wasn't clear what needed to be changed. It
looks
> like
> > > your
> > > > config file took the RegridDataPlane example and changed the
names of
> > > > variables from including REGRID_DATA_PLANE to GRID_STAT. While
the
> > > possible
> > > > variables for GridStat and RegridDataPlane have some overlap,
they do
> > > > differ, as the tools do different things. I can create an
example of
> > > using
> > > > GridStat using python embedding and send it to you so you can
use
> that
> > > as a
> > > > starting point to modify.
> > > >
> > > > Regarding the arguments to your python script, you are correct
that
> you
> > > > will put them after the script name in the
[FCST/OBS]_VAR<n>_NAME
> > > > variables.
> > > >
> > > > If your data and scripts are available on an NCAR machine, you
could
> > > point
> > > > me there and I could take a look at them as well.
> > > >
> > > > Thanks,
> > > > George
> > > >
> > > > On Fri, Oct 30, 2020 at 12:00 PM Mrinal Biswas via RT <
> > met_help at ucar.edu
> > > >
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97258 >
> > > > >
> > > > > Hi George,
> > > > >
> > > > > Thanks for sending the example again. Let me briefly
describe what
> I
> > am
> > > > > trying to do, so it will be easier for you to guide me
through the
> > > > process.
> > > > > You probably already know this but just to re-capture.
> > > > >
> > > > > I have a TDR (tc_radar_v2f_combined_xy_rel_merge_ships.nc).
I
> have a
> > > > > script
> > > > > I have tested with this command:
> > > > >
> > > > > plot_data_plane PYTHON_NUMPY test.ps
> > 'name="read_tdr_lat_lon_final.py
> > > > > tc_radar_v2f_combined_xy_rel_merge_ships.nc
merged_zonal_wind
> > 190901H1
> > > > > 2";'
> > > > >
> > > > >
> > > > > Where read_tdr_lat_lon_final.py is the name of the python
script,
> > > > > merged_zonal_wind is the name of the observation variable,
190901H1
> > is
> > > > the
> > > > > mission name and 2 is the height of the observation in km.
> > > > >
> > > > >
> > > > > I have a NetCDF model output in height (m) which has zonal
wind.
> > > > >
> > > > >
> > > > > The ncdump -h looks like this:.
> > > > >
> > > > >
> > > > > [Mrinal.Biswas at hfe03 HAFS]$ ncdump -h
> > > > > dorian05l.2019083112.hafsprs.synoptic.interp.0p03.f024.nc4
> > > > >
> > > > > netcdf
dorian05l.2019083112.hafsprs.synoptic.interp.0p03.f024 {
> > > > >
> > > > > dimensions:
> > > > >
> > > > > latitude = 2001 ;
> > > > >
> > > > > lev = 16 ;
> > > > >
> > > > > longitude = 3501 ;
> > > > >
> > > > > variables:
> > > > >
> > > > > double latitude(latitude) ;
> > > > >
> > > > > latitude:units = "degrees_north" ;
> > > > >
> > > > > latitude:standard_name = "latitude" ;
> > > > >
> > > > > latitude:long_name = "latitude" ;
> > > > >
> > > > > float lev(lev) ;
> > > > >
> > > > > lev:units = "meter" ;
> > > > >
> > > > > double longitude(longitude) ;
> > > > >
> > > > > longitude:units = "degrees_east" ;
> > > > >
> > > > > longitude:standard_name = "longitude" ;
> > > > >
> > > > > longitude:long_name = "longitude" ;
> > > > >
> > > > > double u(lev, latitude, longitude) ;
> > > > >
> > > > > u:GRIB_paramId = 131LL ;
> > > > >
> > > > > u:GRIB_shortName = "u" ;
> > > > >
> > > > > u:GRIB_units = "m s**-1" ;
> > > > >
> > > > > u:GRIB_name = "U component of wind" ;
> > > > >
> > > > > u:GRIB_cfName = "eastward_wind" ;
> > > > >
> > > > > u:GRIB_cfVarName = "u" ;
> > > > >
> > > > > u:GRIB_dataType = "fc" ;
> > > > >
> > > > > u:GRIB_missingValue = 9999LL ;
> > > > >
> > > > > u:GRIB_numberOfPoints = 7005501LL ;
> > > > >
> > > > > u:GRIB_typeOfLevel = "isobaricInhPa" ;
> > > > >
> > > > > u:GRIB_NV = 0LL ;
> > > > >
> > > > > u:GRIB_stepUnits = 1LL ;
> > > > >
> > > > > u:GRIB_stepType = "instant" ;
> > > > >
> > > > > u:GRIB_gridType = "regular_ll" ;
> > > > >
> > > > > u:GRIB_gridDefinitionDescription = "Latitude/longitude. Also
called
> > > > > equidistant cylindrical, or Plate Carree" ;
> > > > >
> > > > > u:GRIB_Nx = 3501LL ;
> > > > >
> > > > > u:GRIB_iDirectionIncrementInDegrees = 0.029999 ;
> > > > >
> > > > > u:GRIB_iScansNegatively = 0LL ;
> > > > >
> > > > > u:GRIB_longitudeOfFirstGridPointInDegrees = 245.5 ;
> > > > >
> > > > > u:GRIB_longitudeOfLastGridPointInDegrees = 350.499998 ;
> > > > >
> > > > > u:GRIB_Ny = 2001LL ;
> > > > >
> > > > > u:GRIB_jDirectionIncrementInDegrees = 0.03 ;
> > > > >
> > > > > u:GRIB_jPointsAreConsecutive = 0LL ;
> > > > >
> > > > > u:GRIB_jScansPositively = 1LL ;
> > > > >
> > > > > u:GRIB_latitudeOfFirstGridPointInDegrees = -5. ;
> > > > >
> > > > > u:GRIB_latitudeOfLastGridPointInDegrees = 55. ;
> > > > >
> > > > > u:long_name = "U component of wind" ;
> > > > >
> > > > > u:units = "m s**-1" ;
> > > > >
> > > > > u:standard_name = "eastward_wind" ;
> > > > >
> > > > > }
> > > > >
> > > > >
> > > > > I want to compare the Radar data with the model output to
compute
> > stats
> > > > > (like RMSE, ME, MAE etc). This will be a use case within
METplus. I
> > > hope
> > > > > GridStat is the correct tool to do this job. I used the
> > RegridDataPlane
> > > > > example and created a Configuration file (attached). I have
> questions
> > > > about
> > > > > some of the variables and what they should be in the conf
file.
> E.g.
> > > > Where
> > > > > do I mention what model output variable to compare against
in the
> > conf
> > > > > file? Should we use "u" or "eastward_wind"? Also I am not
sure
> where
> > to
> > > > put
> > > > > in the arguments that are needed to read the python script.
As of
> > now I
> > > > > have included it in the OBS_VAR1_NAME.
> > > > >
> > > > >
> > > > > I will really appreciate it if you can look at the conf file
and
> see
> > > what
> > > > > corrections need to be made. And then we can diss what next
steps
> > > > (wrapper
> > > > > script) need to be written to test the whole case.
> > > > >
> > > > >
> > > > > Thanks
> > > > >
> > > > > Biswas
> > > > >
> > > > > On Wed, Oct 28, 2020 at 12:49 PM George McCabe via RT <
> > > met_help at ucar.edu
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Biswas,
> > > > > >
> > > > > > I am moving this email over to MET-Help so we can track
our work
> > > > easily.
> > > > > >
> > > > > > I sent you the RegridDataPlane example because there are
> currently
> > no
> > > > use
> > > > > > cases in the METplus repository that call GridStat with
Python
> > > > embedding,
> > > > > > but the variables to set should be the same as
RegridDataPlane.
> > > Notably
> > > > > > these lines need to be set correctly:
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://github.com/dtcenter/METplus/blob/f54d8305e305a0bfd11964fbf280b390bc7152b5/parm/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.conf#L34
> > > > > >
> > > > > > OBS_VAR1_NAME should be set to the path of the python
script you
> > want
> > > > to
> > > > > > call with any arguments to the script. You can use
filename
> > template
> > > > > syntax
> > > > > > to specify  time information based on the current run
time, i.e.
> > > > > > {init?fmt=%Y%m%d%H} will be replaced with the current
> > initialization
> > > > time
> > > > > > in YYYYMMDDHH format. If you are using Python embedding
for
> > forecast
> > > > > data,
> > > > > > you will set FCST_VAR1_NAME instead.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://github.com/dtcenter/METplus/blob/f54d8305e305a0bfd11964fbf280b390bc7152b5/parm/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.conf#L61
> > > > > >
> > > > > > You will also need to set the appropriate template value
to
> > > > PYTHON_NUMPY.
> > > > > > For obs data, you would set OBS_GRID_STAT_INPUT_TEMPLATE
and for
> > > > forecast
> > > > > > data you would set FCST_GRID_STAT_INPUT_TEMPLATE.
> > > > > >
> > > > > > Let me know if you have any issues getting this set up and
I can
> > > help.
> > > > > >
> > > > > > Thanks,
> > > > > > George
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed Oct 28 12:42:44 2020, mccabe wrote:
> > > > > > > Hi George,
> > > > > > >  I hope you are doing fine. Long back I asked you about
the
> conf
> > > file
> > > > > > > for Gridstat. The link you sent was from the
RegridDataPlane.
> > > > However,
> > > > > > > the link doesn't work now. Is there any Gridstat conf
file
> > example
> > > > > > > available that utilizes python embedding. I want to use
my
> python
> > > > > > > script to compare the Radar data with model output. Now
that
> the
> > > > > > > pressure to height tool is available I need to finish
the task
> > > within
> > > > > > > METplus.
> > > > > > > Thanks
> > > > > > > Biswas
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > > *My working day may not be your working day. Please do not
feel
> > obliged
> > > > to
> > > > > reply to this email outside of your normal working hours.*
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > > --
> > >
> > > *My working day may not be your working day. Please do not feel
obliged
> > to
> > > reply to this email outside of your normal working hours.*
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
>
> *My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.*
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: Configuration file for Grid-stat
From: Mrinal Biswas
Time: Mon Nov 02 13:29:51 2020

Hi George,
Yes, that is correct. We want this example as a MetPlus use case for
other
users. Please let me know if it is better to schedule a time to meet
rather
than explaining on the email, whatever works for you. I will check on
the
GridStat conf file.
Thanks
Biswas

On Mon, Nov 2, 2020 at 1:12 PM George McCabe via RT
<met_help at ucar.edu>
wrote:

> Hi Biswas,
>
> Am I understanding correctly that you are using Python embedding to
read
> the observation data and passing in the model data directly? If so,
your
> use case conf should refer to the GridStat.conf example to read the
> forecast data properly. I am working on another task now, but I can
help
> you get this configured for your case when I wrap that work up.
>
> Thanks,
> George
>
> On Mon, Nov 2, 2020 at 1:02 PM Mrinal Biswas via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97258 >
> >
> > Hi George,
> > Many thanks for the info. After my corrections to the conf file it
looks
> > similar to the example you have created. I have put the input
data, the
> > python script, model output and the GridStat_python_enbedding.conf
file
> on
> > eyewall (/d1/biswas/HRDdata-MET/mccabe). Please let me know if it
looks
> ok
> > to you.
> > Thanks for all your help
> > Biswas
> >
> > On Mon, Nov 2, 2020 at 12:04 PM George McCabe via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Biswas,
> > >
> > > You can configure GridStat to regrid the data to a common grid
for
> > > comparisons. If you are on the NCAR VPN, you can see the example
I
> > created
> > > for GridStat with python embedding. It is not yet in the
repository, as
> > it
> > > is being reviewed by other developers, but you can see what it
looks
> like
> > > here:
> > >
> > >
> > >
> >
>
http://icculus.rap.ucar.edu/metplus/generated/met_tool_wrapper/GridStat/GridStat_python_embedding.html#metplus-
configuration
> > >
> > > The relevant line is:
> > > GRID_STAT_REGRID_TO_GRID = G130
> > >
> > > In this example we regrid both datasets to G130. You can also
set it to
> > > regrid the obs to the forecast grid, vice versa, or a grid based
on a
> > file
> > > path. Go to this page and search to "to_grid" for more
information on
> > that
> > > setting:
> > >
> > >
> > >
> >
>
https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html?highlight=to_grid
> > >
> > >
> > >
> > > On Mon, Nov 2, 2020 at 11:57 AM Mrinal Biswas via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97258
>
> > > >
> > > > Hi George,
> > > > I think I have some idea. I am using the GridStat.conf file
and using
> > the
> > > > variables to make the conf file for METPlus. I will share it
soon.
> > > However,
> > > > reading the GridStat documentation I reda that both the model
and
> > > > observation output should be in the same grid. Is my
> > > > interpretation correct? My observation and model output is of
> different
> > > > resolution. Am I using the correct tool?
> > > > Thanks
> > > > Biswas
> > > >
> > > > On Mon, Nov 2, 2020 at 10:33 AM George McCabe via RT <
> > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Biswas,
> > > > >
> > > > > I'm sorry that I wasn't clear what needed to be changed. It
looks
> > like
> > > > your
> > > > > config file took the RegridDataPlane example and changed the
names
> of
> > > > > variables from including REGRID_DATA_PLANE to GRID_STAT.
While the
> > > > possible
> > > > > variables for GridStat and RegridDataPlane have some
overlap, they
> do
> > > > > differ, as the tools do different things. I can create an
example
> of
> > > > using
> > > > > GridStat using python embedding and send it to you so you
can use
> > that
> > > > as a
> > > > > starting point to modify.
> > > > >
> > > > > Regarding the arguments to your python script, you are
correct that
> > you
> > > > > will put them after the script name in the
[FCST/OBS]_VAR<n>_NAME
> > > > > variables.
> > > > >
> > > > > If your data and scripts are available on an NCAR machine,
you
> could
> > > > point
> > > > > me there and I could take a look at them as well.
> > > > >
> > > > > Thanks,
> > > > > George
> > > > >
> > > > > On Fri, Oct 30, 2020 at 12:00 PM Mrinal Biswas via RT <
> > > met_help at ucar.edu
> > > > >
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97258 >
> > > > > >
> > > > > > Hi George,
> > > > > >
> > > > > > Thanks for sending the example again. Let me briefly
describe
> what
> > I
> > > am
> > > > > > trying to do, so it will be easier for you to guide me
through
> the
> > > > > process.
> > > > > > You probably already know this but just to re-capture.
> > > > > >
> > > > > > I have a TDR
(tc_radar_v2f_combined_xy_rel_merge_ships.nc). I
> > have a
> > > > > > script
> > > > > > I have tested with this command:
> > > > > >
> > > > > > plot_data_plane PYTHON_NUMPY test.ps
> > > 'name="read_tdr_lat_lon_final.py
> > > > > > tc_radar_v2f_combined_xy_rel_merge_ships.nc
merged_zonal_wind
> > > 190901H1
> > > > > > 2";'
> > > > > >
> > > > > >
> > > > > > Where read_tdr_lat_lon_final.py is the name of the python
script,
> > > > > > merged_zonal_wind is the name of the observation variable,
> 190901H1
> > > is
> > > > > the
> > > > > > mission name and 2 is the height of the observation in km.
> > > > > >
> > > > > >
> > > > > > I have a NetCDF model output in height (m) which has zonal
wind.
> > > > > >
> > > > > >
> > > > > > The ncdump -h looks like this:.
> > > > > >
> > > > > >
> > > > > > [Mrinal.Biswas at hfe03 HAFS]$ ncdump -h
> > > > > > dorian05l.2019083112.hafsprs.synoptic.interp.0p03.f024.nc4
> > > > > >
> > > > > > netcdf
dorian05l.2019083112.hafsprs.synoptic.interp.0p03.f024 {
> > > > > >
> > > > > > dimensions:
> > > > > >
> > > > > > latitude = 2001 ;
> > > > > >
> > > > > > lev = 16 ;
> > > > > >
> > > > > > longitude = 3501 ;
> > > > > >
> > > > > > variables:
> > > > > >
> > > > > > double latitude(latitude) ;
> > > > > >
> > > > > > latitude:units = "degrees_north" ;
> > > > > >
> > > > > > latitude:standard_name = "latitude" ;
> > > > > >
> > > > > > latitude:long_name = "latitude" ;
> > > > > >
> > > > > > float lev(lev) ;
> > > > > >
> > > > > > lev:units = "meter" ;
> > > > > >
> > > > > > double longitude(longitude) ;
> > > > > >
> > > > > > longitude:units = "degrees_east" ;
> > > > > >
> > > > > > longitude:standard_name = "longitude" ;
> > > > > >
> > > > > > longitude:long_name = "longitude" ;
> > > > > >
> > > > > > double u(lev, latitude, longitude) ;
> > > > > >
> > > > > > u:GRIB_paramId = 131LL ;
> > > > > >
> > > > > > u:GRIB_shortName = "u" ;
> > > > > >
> > > > > > u:GRIB_units = "m s**-1" ;
> > > > > >
> > > > > > u:GRIB_name = "U component of wind" ;
> > > > > >
> > > > > > u:GRIB_cfName = "eastward_wind" ;
> > > > > >
> > > > > > u:GRIB_cfVarName = "u" ;
> > > > > >
> > > > > > u:GRIB_dataType = "fc" ;
> > > > > >
> > > > > > u:GRIB_missingValue = 9999LL ;
> > > > > >
> > > > > > u:GRIB_numberOfPoints = 7005501LL ;
> > > > > >
> > > > > > u:GRIB_typeOfLevel = "isobaricInhPa" ;
> > > > > >
> > > > > > u:GRIB_NV = 0LL ;
> > > > > >
> > > > > > u:GRIB_stepUnits = 1LL ;
> > > > > >
> > > > > > u:GRIB_stepType = "instant" ;
> > > > > >
> > > > > > u:GRIB_gridType = "regular_ll" ;
> > > > > >
> > > > > > u:GRIB_gridDefinitionDescription = "Latitude/longitude.
Also
> called
> > > > > > equidistant cylindrical, or Plate Carree" ;
> > > > > >
> > > > > > u:GRIB_Nx = 3501LL ;
> > > > > >
> > > > > > u:GRIB_iDirectionIncrementInDegrees = 0.029999 ;
> > > > > >
> > > > > > u:GRIB_iScansNegatively = 0LL ;
> > > > > >
> > > > > > u:GRIB_longitudeOfFirstGridPointInDegrees = 245.5 ;
> > > > > >
> > > > > > u:GRIB_longitudeOfLastGridPointInDegrees = 350.499998 ;
> > > > > >
> > > > > > u:GRIB_Ny = 2001LL ;
> > > > > >
> > > > > > u:GRIB_jDirectionIncrementInDegrees = 0.03 ;
> > > > > >
> > > > > > u:GRIB_jPointsAreConsecutive = 0LL ;
> > > > > >
> > > > > > u:GRIB_jScansPositively = 1LL ;
> > > > > >
> > > > > > u:GRIB_latitudeOfFirstGridPointInDegrees = -5. ;
> > > > > >
> > > > > > u:GRIB_latitudeOfLastGridPointInDegrees = 55. ;
> > > > > >
> > > > > > u:long_name = "U component of wind" ;
> > > > > >
> > > > > > u:units = "m s**-1" ;
> > > > > >
> > > > > > u:standard_name = "eastward_wind" ;
> > > > > >
> > > > > > }
> > > > > >
> > > > > >
> > > > > > I want to compare the Radar data with the model output to
compute
> > > stats
> > > > > > (like RMSE, ME, MAE etc). This will be a use case within
> METplus. I
> > > > hope
> > > > > > GridStat is the correct tool to do this job. I used the
> > > RegridDataPlane
> > > > > > example and created a Configuration file (attached). I
have
> > questions
> > > > > about
> > > > > > some of the variables and what they should be in the conf
file.
> > E.g.
> > > > > Where
> > > > > > do I mention what model output variable to compare against
in the
> > > conf
> > > > > > file? Should we use "u" or "eastward_wind"? Also I am not
sure
> > where
> > > to
> > > > > put
> > > > > > in the arguments that are needed to read the python
script. As of
> > > now I
> > > > > > have included it in the OBS_VAR1_NAME.
> > > > > >
> > > > > >
> > > > > > I will really appreciate it if you can look at the conf
file and
> > see
> > > > what
> > > > > > corrections need to be made. And then we can diss what
next steps
> > > > > (wrapper
> > > > > > script) need to be written to test the whole case.
> > > > > >
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > Biswas
> > > > > >
> > > > > > On Wed, Oct 28, 2020 at 12:49 PM George McCabe via RT <
> > > > met_help at ucar.edu
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Biswas,
> > > > > > >
> > > > > > > I am moving this email over to MET-Help so we can track
our
> work
> > > > > easily.
> > > > > > >
> > > > > > > I sent you the RegridDataPlane example because there are
> > currently
> > > no
> > > > > use
> > > > > > > cases in the METplus repository that call GridStat with
Python
> > > > > embedding,
> > > > > > > but the variables to set should be the same as
RegridDataPlane.
> > > > Notably
> > > > > > > these lines need to be set correctly:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://github.com/dtcenter/METplus/blob/f54d8305e305a0bfd11964fbf280b390bc7152b5/parm/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.conf#L34
> > > > > > >
> > > > > > > OBS_VAR1_NAME should be set to the path of the python
script
> you
> > > want
> > > > > to
> > > > > > > call with any arguments to the script. You can use
filename
> > > template
> > > > > > syntax
> > > > > > > to specify  time information based on the current run
time,
> i.e.
> > > > > > > {init?fmt=%Y%m%d%H} will be replaced with the current
> > > initialization
> > > > > time
> > > > > > > in YYYYMMDDHH format. If you are using Python embedding
for
> > > forecast
> > > > > > data,
> > > > > > > you will set FCST_VAR1_NAME instead.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://github.com/dtcenter/METplus/blob/f54d8305e305a0bfd11964fbf280b390bc7152b5/parm/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.conf#L61
> > > > > > >
> > > > > > > You will also need to set the appropriate template value
to
> > > > > PYTHON_NUMPY.
> > > > > > > For obs data, you would set OBS_GRID_STAT_INPUT_TEMPLATE
and
> for
> > > > > forecast
> > > > > > > data you would set FCST_GRID_STAT_INPUT_TEMPLATE.
> > > > > > >
> > > > > > > Let me know if you have any issues getting this set up
and I
> can
> > > > help.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > George
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed Oct 28 12:42:44 2020, mccabe wrote:
> > > > > > > > Hi George,
> > > > > > > >  I hope you are doing fine. Long back I asked you
about the
> > conf
> > > > file
> > > > > > > > for Gridstat. The link you sent was from the
RegridDataPlane.
> > > > > However,
> > > > > > > > the link doesn't work now. Is there any Gridstat conf
file
> > > example
> > > > > > > > available that utilizes python embedding. I want to
use my
> > python
> > > > > > > > script to compare the Radar data with model output.
Now that
> > the
> > > > > > > > pressure to height tool is available I need to finish
the
> task
> > > > within
> > > > > > > > METplus.
> > > > > > > > Thanks
> > > > > > > > Biswas
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > *My working day may not be your working day. Please do not
feel
> > > obliged
> > > > > to
> > > > > > reply to this email outside of your normal working hours.*
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > George McCabe - Software Engineer III
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > 303-497-2768
> > > > > ---
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > > *My working day may not be your working day. Please do not
feel
> obliged
> > > to
> > > > reply to this email outside of your normal working hours.*
> > > >
> > > >
> > >
> > > --
> > > George McCabe - Software Engineer III
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > 303-497-2768
> > > ---
> > > My working day may not be your working day. Please do not feel
obliged
> to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> > --
> >
> > *My working day may not be your working day. Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.*
> >
> >
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>

--

*My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.*

------------------------------------------------
Subject: Configuration file for Grid-stat
From: George McCabe
Time: Tue Nov 03 08:46:48 2020

Hi Biswas,

This page has instructions and guidelines for adding a new METplus use
case. Please refer to this page and let me know if anything is
unclear. I
have filled in a lot of information here recently but we are relying
on
users to provide feedback to improve the process.

https://dtcenter.github.io/METplus/develop/Contributors_Guide/add_use_case.html

I will take a look at your use case and help you get it configured to
read
your data, but you will need to make the necessary adjustments to
produce
useful output from the tools.

Thanks,
George

On Mon, Nov 2, 2020 at 1:30 PM Mrinal Biswas via RT
<met_help at ucar.edu>
wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97258 >
>
> Hi George,
> Yes, that is correct. We want this example as a MetPlus use case for
other
> users. Please let me know if it is better to schedule a time to meet
rather
> than explaining on the email, whatever works for you. I will check
on the
> GridStat conf file.
> Thanks
> Biswas
>
> On Mon, Nov 2, 2020 at 1:12 PM George McCabe via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Biswas,
> >
> > Am I understanding correctly that you are using Python embedding
to read
> > the observation data and passing in the model data directly? If
so, your
> > use case conf should refer to the GridStat.conf example to read
the
> > forecast data properly. I am working on another task now, but I
can help
> > you get this configured for your case when I wrap that work up.
> >
> > Thanks,
> > George
> >
> > On Mon, Nov 2, 2020 at 1:02 PM Mrinal Biswas via RT
<met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97258 >
> > >
> > > Hi George,
> > > Many thanks for the info. After my corrections to the conf file
it
> looks
> > > similar to the example you have created. I have put the input
data, the
> > > python script, model output and the
GridStat_python_enbedding.conf file
> > on
> > > eyewall (/d1/biswas/HRDdata-MET/mccabe). Please let me know if
it looks
> > ok
> > > to you.
> > > Thanks for all your help
> > > Biswas
> > >
> > > On Mon, Nov 2, 2020 at 12:04 PM George McCabe via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Biswas,
> > > >
> > > > You can configure GridStat to regrid the data to a common grid
for
> > > > comparisons. If you are on the NCAR VPN, you can see the
example I
> > > created
> > > > for GridStat with python embedding. It is not yet in the
repository,
> as
> > > it
> > > > is being reviewed by other developers, but you can see what it
looks
> > like
> > > > here:
> > > >
> > > >
> > > >
> > >
> >
>
http://icculus.rap.ucar.edu/metplus/generated/met_tool_wrapper/GridStat/GridStat_python_embedding.html#metplus-
configuration
> > > >
> > > > The relevant line is:
> > > > GRID_STAT_REGRID_TO_GRID = G130
> > > >
> > > > In this example we regrid both datasets to G130. You can also
set it
> to
> > > > regrid the obs to the forecast grid, vice versa, or a grid
based on a
> > > file
> > > > path. Go to this page and search to "to_grid" for more
information on
> > > that
> > > > setting:
> > > >
> > > >
> > > >
> > >
> >
>
https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html?highlight=to_grid
> > > >
> > > >
> > > >
> > > > On Mon, Nov 2, 2020 at 11:57 AM Mrinal Biswas via RT <
> > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97258 >
> > > > >
> > > > > Hi George,
> > > > > I think I have some idea. I am using the GridStat.conf file
and
> using
> > > the
> > > > > variables to make the conf file for METPlus. I will share it
soon.
> > > > However,
> > > > > reading the GridStat documentation I reda that both the
model and
> > > > > observation output should be in the same grid. Is my
> > > > > interpretation correct? My observation and model output is
of
> > different
> > > > > resolution. Am I using the correct tool?
> > > > > Thanks
> > > > > Biswas
> > > > >
> > > > > On Mon, Nov 2, 2020 at 10:33 AM George McCabe via RT <
> > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Hi Biswas,
> > > > > >
> > > > > > I'm sorry that I wasn't clear what needed to be changed.
It looks
> > > like
> > > > > your
> > > > > > config file took the RegridDataPlane example and changed
the
> names
> > of
> > > > > > variables from including REGRID_DATA_PLANE to GRID_STAT.
While
> the
> > > > > possible
> > > > > > variables for GridStat and RegridDataPlane have some
overlap,
> they
> > do
> > > > > > differ, as the tools do different things. I can create an
example
> > of
> > > > > using
> > > > > > GridStat using python embedding and send it to you so you
can use
> > > that
> > > > > as a
> > > > > > starting point to modify.
> > > > > >
> > > > > > Regarding the arguments to your python script, you are
correct
> that
> > > you
> > > > > > will put them after the script name in the
[FCST/OBS]_VAR<n>_NAME
> > > > > > variables.
> > > > > >
> > > > > > If your data and scripts are available on an NCAR machine,
you
> > could
> > > > > point
> > > > > > me there and I could take a look at them as well.
> > > > > >
> > > > > > Thanks,
> > > > > > George
> > > > > >
> > > > > > On Fri, Oct 30, 2020 at 12:00 PM Mrinal Biswas via RT <
> > > > met_help at ucar.edu
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97258
> >
> > > > > > >
> > > > > > > Hi George,
> > > > > > >
> > > > > > > Thanks for sending the example again. Let me briefly
describe
> > what
> > > I
> > > > am
> > > > > > > trying to do, so it will be easier for you to guide me
through
> > the
> > > > > > process.
> > > > > > > You probably already know this but just to re-capture.
> > > > > > >
> > > > > > > I have a TDR
(tc_radar_v2f_combined_xy_rel_merge_ships.nc). I
> > > have a
> > > > > > > script
> > > > > > > I have tested with this command:
> > > > > > >
> > > > > > > plot_data_plane PYTHON_NUMPY test.ps
> > > > 'name="read_tdr_lat_lon_final.py
> > > > > > > tc_radar_v2f_combined_xy_rel_merge_ships.nc
merged_zonal_wind
> > > > 190901H1
> > > > > > > 2";'
> > > > > > >
> > > > > > >
> > > > > > > Where read_tdr_lat_lon_final.py is the name of the
python
> script,
> > > > > > > merged_zonal_wind is the name of the observation
variable,
> > 190901H1
> > > > is
> > > > > > the
> > > > > > > mission name and 2 is the height of the observation in
km.
> > > > > > >
> > > > > > >
> > > > > > > I have a NetCDF model output in height (m) which has
zonal
> wind.
> > > > > > >
> > > > > > >
> > > > > > > The ncdump -h looks like this:.
> > > > > > >
> > > > > > >
> > > > > > > [Mrinal.Biswas at hfe03 HAFS]$ ncdump -h
> > > > > > >
dorian05l.2019083112.hafsprs.synoptic.interp.0p03.f024.nc4
> > > > > > >
> > > > > > > netcdf
dorian05l.2019083112.hafsprs.synoptic.interp.0p03.f024 {
> > > > > > >
> > > > > > > dimensions:
> > > > > > >
> > > > > > > latitude = 2001 ;
> > > > > > >
> > > > > > > lev = 16 ;
> > > > > > >
> > > > > > > longitude = 3501 ;
> > > > > > >
> > > > > > > variables:
> > > > > > >
> > > > > > > double latitude(latitude) ;
> > > > > > >
> > > > > > > latitude:units = "degrees_north" ;
> > > > > > >
> > > > > > > latitude:standard_name = "latitude" ;
> > > > > > >
> > > > > > > latitude:long_name = "latitude" ;
> > > > > > >
> > > > > > > float lev(lev) ;
> > > > > > >
> > > > > > > lev:units = "meter" ;
> > > > > > >
> > > > > > > double longitude(longitude) ;
> > > > > > >
> > > > > > > longitude:units = "degrees_east" ;
> > > > > > >
> > > > > > > longitude:standard_name = "longitude" ;
> > > > > > >
> > > > > > > longitude:long_name = "longitude" ;
> > > > > > >
> > > > > > > double u(lev, latitude, longitude) ;
> > > > > > >
> > > > > > > u:GRIB_paramId = 131LL ;
> > > > > > >
> > > > > > > u:GRIB_shortName = "u" ;
> > > > > > >
> > > > > > > u:GRIB_units = "m s**-1" ;
> > > > > > >
> > > > > > > u:GRIB_name = "U component of wind" ;
> > > > > > >
> > > > > > > u:GRIB_cfName = "eastward_wind" ;
> > > > > > >
> > > > > > > u:GRIB_cfVarName = "u" ;
> > > > > > >
> > > > > > > u:GRIB_dataType = "fc" ;
> > > > > > >
> > > > > > > u:GRIB_missingValue = 9999LL ;
> > > > > > >
> > > > > > > u:GRIB_numberOfPoints = 7005501LL ;
> > > > > > >
> > > > > > > u:GRIB_typeOfLevel = "isobaricInhPa" ;
> > > > > > >
> > > > > > > u:GRIB_NV = 0LL ;
> > > > > > >
> > > > > > > u:GRIB_stepUnits = 1LL ;
> > > > > > >
> > > > > > > u:GRIB_stepType = "instant" ;
> > > > > > >
> > > > > > > u:GRIB_gridType = "regular_ll" ;
> > > > > > >
> > > > > > > u:GRIB_gridDefinitionDescription = "Latitude/longitude.
Also
> > called
> > > > > > > equidistant cylindrical, or Plate Carree" ;
> > > > > > >
> > > > > > > u:GRIB_Nx = 3501LL ;
> > > > > > >
> > > > > > > u:GRIB_iDirectionIncrementInDegrees = 0.029999 ;
> > > > > > >
> > > > > > > u:GRIB_iScansNegatively = 0LL ;
> > > > > > >
> > > > > > > u:GRIB_longitudeOfFirstGridPointInDegrees = 245.5 ;
> > > > > > >
> > > > > > > u:GRIB_longitudeOfLastGridPointInDegrees = 350.499998 ;
> > > > > > >
> > > > > > > u:GRIB_Ny = 2001LL ;
> > > > > > >
> > > > > > > u:GRIB_jDirectionIncrementInDegrees = 0.03 ;
> > > > > > >
> > > > > > > u:GRIB_jPointsAreConsecutive = 0LL ;
> > > > > > >
> > > > > > > u:GRIB_jScansPositively = 1LL ;
> > > > > > >
> > > > > > > u:GRIB_latitudeOfFirstGridPointInDegrees = -5. ;
> > > > > > >
> > > > > > > u:GRIB_latitudeOfLastGridPointInDegrees = 55. ;
> > > > > > >
> > > > > > > u:long_name = "U component of wind" ;
> > > > > > >
> > > > > > > u:units = "m s**-1" ;
> > > > > > >
> > > > > > > u:standard_name = "eastward_wind" ;
> > > > > > >
> > > > > > > }
> > > > > > >
> > > > > > >
> > > > > > > I want to compare the Radar data with the model output
to
> compute
> > > > stats
> > > > > > > (like RMSE, ME, MAE etc). This will be a use case within
> > METplus. I
> > > > > hope
> > > > > > > GridStat is the correct tool to do this job. I used the
> > > > RegridDataPlane
> > > > > > > example and created a Configuration file (attached). I
have
> > > questions
> > > > > > about
> > > > > > > some of the variables and what they should be in the
conf file.
> > > E.g.
> > > > > > Where
> > > > > > > do I mention what model output variable to compare
against in
> the
> > > > conf
> > > > > > > file? Should we use "u" or "eastward_wind"? Also I am
not sure
> > > where
> > > > to
> > > > > > put
> > > > > > > in the arguments that are needed to read the python
script. As
> of
> > > > now I
> > > > > > > have included it in the OBS_VAR1_NAME.
> > > > > > >
> > > > > > >
> > > > > > > I will really appreciate it if you can look at the conf
file
> and
> > > see
> > > > > what
> > > > > > > corrections need to be made. And then we can diss what
next
> steps
> > > > > > (wrapper
> > > > > > > script) need to be written to test the whole case.
> > > > > > >
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > Biswas
> > > > > > >
> > > > > > > On Wed, Oct 28, 2020 at 12:49 PM George McCabe via RT <
> > > > > met_help at ucar.edu
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Biswas,
> > > > > > > >
> > > > > > > > I am moving this email over to MET-Help so we can
track our
> > work
> > > > > > easily.
> > > > > > > >
> > > > > > > > I sent you the RegridDataPlane example because there
are
> > > currently
> > > > no
> > > > > > use
> > > > > > > > cases in the METplus repository that call GridStat
with
> Python
> > > > > > embedding,
> > > > > > > > but the variables to set should be the same as
> RegridDataPlane.
> > > > > Notably
> > > > > > > > these lines need to be set correctly:
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://github.com/dtcenter/METplus/blob/f54d8305e305a0bfd11964fbf280b390bc7152b5/parm/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.conf#L34
> > > > > > > >
> > > > > > > > OBS_VAR1_NAME should be set to the path of the python
script
> > you
> > > > want
> > > > > > to
> > > > > > > > call with any arguments to the script. You can use
filename
> > > > template
> > > > > > > syntax
> > > > > > > > to specify  time information based on the current run
time,
> > i.e.
> > > > > > > > {init?fmt=%Y%m%d%H} will be replaced with the current
> > > > initialization
> > > > > > time
> > > > > > > > in YYYYMMDDHH format. If you are using Python
embedding for
> > > > forecast
> > > > > > > data,
> > > > > > > > you will set FCST_VAR1_NAME instead.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
https://github.com/dtcenter/METplus/blob/f54d8305e305a0bfd11964fbf280b390bc7152b5/parm/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.conf#L61
> > > > > > > >
> > > > > > > > You will also need to set the appropriate template
value to
> > > > > > PYTHON_NUMPY.
> > > > > > > > For obs data, you would set
OBS_GRID_STAT_INPUT_TEMPLATE and
> > for
> > > > > > forecast
> > > > > > > > data you would set FCST_GRID_STAT_INPUT_TEMPLATE.
> > > > > > > >
> > > > > > > > Let me know if you have any issues getting this set up
and I
> > can
> > > > > help.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > George
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed Oct 28 12:42:44 2020, mccabe wrote:
> > > > > > > > > Hi George,
> > > > > > > > >  I hope you are doing fine. Long back I asked you
about the
> > > conf
> > > > > file
> > > > > > > > > for Gridstat. The link you sent was from the
> RegridDataPlane.
> > > > > > However,
> > > > > > > > > the link doesn't work now. Is there any Gridstat
conf file
> > > > example
> > > > > > > > > available that utilizes python embedding. I want to
use my
> > > python
> > > > > > > > > script to compare the Radar data with model output.
Now
> that
> > > the
> > > > > > > > > pressure to height tool is available I need to
finish the
> > task
> > > > > within
> > > > > > > > > METplus.
> > > > > > > > > Thanks
> > > > > > > > > Biswas
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > *My working day may not be your working day. Please do
not feel
> > > > obliged
> > > > > > to
> > > > > > > reply to this email outside of your normal working
hours.*
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > George McCabe - Software Engineer III
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > 303-497-2768
> > > > > > ---
> > > > > > My working day may not be your working day. Please do not
feel
> > > obliged
> > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > >
> > > > > *My working day may not be your working day. Please do not
feel
> > obliged
> > > > to
> > > > > reply to this email outside of your normal working hours.*
> > > > >
> > > > >
> > > >
> > > > --
> > > > George McCabe - Software Engineer III
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > 303-497-2768
> > > > ---
> > > > My working day may not be your working day. Please do not feel
> obliged
> > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > > --
> > >
> > > *My working day may not be your working day. Please do not feel
obliged
> > to
> > > reply to this email outside of your normal working hours.*
> > >
> > >
> >
> > --
> > George McCabe - Software Engineer III
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > 303-497-2768
> > ---
> > My working day may not be your working day. Please do not feel
obliged to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
>
> *My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.*
>
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------
Subject: Configuration file for Grid-stat
From: George McCabe
Time: Tue Nov 03 09:56:26 2020

Hi Biswas,

It would be useful to have a meeting to discuss this use case. It
would
probably be useful to have John HG there as well to help you determine
the
best tools to run for your analysis. Let me know your availability or
refer
to our Google calendar schedules to find a time that works.

Thanks,
George

On Tue, Nov 3, 2020 at 8:46 AM George McCabe <mccabe at ucar.edu> wrote:

> Hi Biswas,
>
> This page has instructions and guidelines for adding a new METplus
use
> case. Please refer to this page and let me know if anything is
unclear. I
> have filled in a lot of information here recently but we are relying
on
> users to provide feedback to improve the process.
>
>
>
https://dtcenter.github.io/METplus/develop/Contributors_Guide/add_use_case.html
>
> I will take a look at your use case and help you get it configured
to read
> your data, but you will need to make the necessary adjustments to
produce
> useful output from the tools.
>
> Thanks,
> George
>
> On Mon, Nov 2, 2020 at 1:30 PM Mrinal Biswas via RT
<met_help at ucar.edu>
> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97258 >
>>
>> Hi George,
>> Yes, that is correct. We want this example as a MetPlus use case
for other
>> users. Please let me know if it is better to schedule a time to
meet
>> rather
>> than explaining on the email, whatever works for you. I will check
on the
>> GridStat conf file.
>> Thanks
>> Biswas
>>
>> On Mon, Nov 2, 2020 at 1:12 PM George McCabe via RT
<met_help at ucar.edu>
>> wrote:
>>
>> > Hi Biswas,
>> >
>> > Am I understanding correctly that you are using Python embedding
to read
>> > the observation data and passing in the model data directly? If
so, your
>> > use case conf should refer to the GridStat.conf example to read
the
>> > forecast data properly. I am working on another task now, but I
can help
>> > you get this configured for your case when I wrap that work up.
>> >
>> > Thanks,
>> > George
>> >
>> > On Mon, Nov 2, 2020 at 1:02 PM Mrinal Biswas via RT
<met_help at ucar.edu>
>> > wrote:
>> >
>> > >
>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97258 >
>> > >
>> > > Hi George,
>> > > Many thanks for the info. After my corrections to the conf file
it
>> looks
>> > > similar to the example you have created. I have put the input
data,
>> the
>> > > python script, model output and the
GridStat_python_enbedding.conf
>> file
>> > on
>> > > eyewall (/d1/biswas/HRDdata-MET/mccabe). Please let me know if
it
>> looks
>> > ok
>> > > to you.
>> > > Thanks for all your help
>> > > Biswas
>> > >
>> > > On Mon, Nov 2, 2020 at 12:04 PM George McCabe via RT <
>> met_help at ucar.edu>
>> > > wrote:
>> > >
>> > > > Hi Biswas,
>> > > >
>> > > > You can configure GridStat to regrid the data to a common
grid for
>> > > > comparisons. If you are on the NCAR VPN, you can see the
example I
>> > > created
>> > > > for GridStat with python embedding. It is not yet in the
>> repository, as
>> > > it
>> > > > is being reviewed by other developers, but you can see what
it looks
>> > like
>> > > > here:
>> > > >
>> > > >
>> > > >
>> > >
>> >
>>
http://icculus.rap.ucar.edu/metplus/generated/met_tool_wrapper/GridStat/GridStat_python_embedding.html#metplus-
configuration
>> > > >
>> > > > The relevant line is:
>> > > > GRID_STAT_REGRID_TO_GRID = G130
>> > > >
>> > > > In this example we regrid both datasets to G130. You can also
set
>> it to
>> > > > regrid the obs to the forecast grid, vice versa, or a grid
based on
>> a
>> > > file
>> > > > path. Go to this page and search to "to_grid" for more
information
>> on
>> > > that
>> > > > setting:
>> > > >
>> > > >
>> > > >
>> > >
>> >
>>
https://dtcenter.github.io/MET/latest/Users_Guide/data_io.html?highlight=to_grid
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Nov 2, 2020 at 11:57 AM Mrinal Biswas via RT <
>> > met_help at ucar.edu>
>> > > > wrote:
>> > > >
>> > > > >
>> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97258 >
>> > > > >
>> > > > > Hi George,
>> > > > > I think I have some idea. I am using the GridStat.conf file
and
>> using
>> > > the
>> > > > > variables to make the conf file for METPlus. I will share
it soon.
>> > > > However,
>> > > > > reading the GridStat documentation I reda that both the
model and
>> > > > > observation output should be in the same grid. Is my
>> > > > > interpretation correct? My observation and model output is
of
>> > different
>> > > > > resolution. Am I using the correct tool?
>> > > > > Thanks
>> > > > > Biswas
>> > > > >
>> > > > > On Mon, Nov 2, 2020 at 10:33 AM George McCabe via RT <
>> > > met_help at ucar.edu>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi Biswas,
>> > > > > >
>> > > > > > I'm sorry that I wasn't clear what needed to be changed.
It
>> looks
>> > > like
>> > > > > your
>> > > > > > config file took the RegridDataPlane example and changed
the
>> names
>> > of
>> > > > > > variables from including REGRID_DATA_PLANE to GRID_STAT.
While
>> the
>> > > > > possible
>> > > > > > variables for GridStat and RegridDataPlane have some
overlap,
>> they
>> > do
>> > > > > > differ, as the tools do different things. I can create an
>> example
>> > of
>> > > > > using
>> > > > > > GridStat using python embedding and send it to you so you
can
>> use
>> > > that
>> > > > > as a
>> > > > > > starting point to modify.
>> > > > > >
>> > > > > > Regarding the arguments to your python script, you are
correct
>> that
>> > > you
>> > > > > > will put them after the script name in the
>> [FCST/OBS]_VAR<n>_NAME
>> > > > > > variables.
>> > > > > >
>> > > > > > If your data and scripts are available on an NCAR
machine, you
>> > could
>> > > > > point
>> > > > > > me there and I could take a look at them as well.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > George
>> > > > > >
>> > > > > > On Fri, Oct 30, 2020 at 12:00 PM Mrinal Biswas via RT <
>> > > > met_help at ucar.edu
>> > > > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > >
>> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97258
>> >
>> > > > > > >
>> > > > > > > Hi George,
>> > > > > > >
>> > > > > > > Thanks for sending the example again. Let me briefly
describe
>> > what
>> > > I
>> > > > am
>> > > > > > > trying to do, so it will be easier for you to guide me
through
>> > the
>> > > > > > process.
>> > > > > > > You probably already know this but just to re-capture.
>> > > > > > >
>> > > > > > > I have a TDR
(tc_radar_v2f_combined_xy_rel_merge_ships.nc). I
>> > > have a
>> > > > > > > script
>> > > > > > > I have tested with this command:
>> > > > > > >
>> > > > > > > plot_data_plane PYTHON_NUMPY test.ps
>> > > > 'name="read_tdr_lat_lon_final.py
>> > > > > > > tc_radar_v2f_combined_xy_rel_merge_ships.nc
merged_zonal_wind
>> > > > 190901H1
>> > > > > > > 2";'
>> > > > > > >
>> > > > > > >
>> > > > > > > Where read_tdr_lat_lon_final.py is the name of the
python
>> script,
>> > > > > > > merged_zonal_wind is the name of the observation
variable,
>> > 190901H1
>> > > > is
>> > > > > > the
>> > > > > > > mission name and 2 is the height of the observation in
km.
>> > > > > > >
>> > > > > > >
>> > > > > > > I have a NetCDF model output in height (m) which has
zonal
>> wind.
>> > > > > > >
>> > > > > > >
>> > > > > > > The ncdump -h looks like this:.
>> > > > > > >
>> > > > > > >
>> > > > > > > [Mrinal.Biswas at hfe03 HAFS]$ ncdump -h
>> > > > > > >
dorian05l.2019083112.hafsprs.synoptic.interp.0p03.f024.nc4
>> > > > > > >
>> > > > > > > netcdf
dorian05l.2019083112.hafsprs.synoptic.interp.0p03.f024
>> {
>> > > > > > >
>> > > > > > > dimensions:
>> > > > > > >
>> > > > > > > latitude = 2001 ;
>> > > > > > >
>> > > > > > > lev = 16 ;
>> > > > > > >
>> > > > > > > longitude = 3501 ;
>> > > > > > >
>> > > > > > > variables:
>> > > > > > >
>> > > > > > > double latitude(latitude) ;
>> > > > > > >
>> > > > > > > latitude:units = "degrees_north" ;
>> > > > > > >
>> > > > > > > latitude:standard_name = "latitude" ;
>> > > > > > >
>> > > > > > > latitude:long_name = "latitude" ;
>> > > > > > >
>> > > > > > > float lev(lev) ;
>> > > > > > >
>> > > > > > > lev:units = "meter" ;
>> > > > > > >
>> > > > > > > double longitude(longitude) ;
>> > > > > > >
>> > > > > > > longitude:units = "degrees_east" ;
>> > > > > > >
>> > > > > > > longitude:standard_name = "longitude" ;
>> > > > > > >
>> > > > > > > longitude:long_name = "longitude" ;
>> > > > > > >
>> > > > > > > double u(lev, latitude, longitude) ;
>> > > > > > >
>> > > > > > > u:GRIB_paramId = 131LL ;
>> > > > > > >
>> > > > > > > u:GRIB_shortName = "u" ;
>> > > > > > >
>> > > > > > > u:GRIB_units = "m s**-1" ;
>> > > > > > >
>> > > > > > > u:GRIB_name = "U component of wind" ;
>> > > > > > >
>> > > > > > > u:GRIB_cfName = "eastward_wind" ;
>> > > > > > >
>> > > > > > > u:GRIB_cfVarName = "u" ;
>> > > > > > >
>> > > > > > > u:GRIB_dataType = "fc" ;
>> > > > > > >
>> > > > > > > u:GRIB_missingValue = 9999LL ;
>> > > > > > >
>> > > > > > > u:GRIB_numberOfPoints = 7005501LL ;
>> > > > > > >
>> > > > > > > u:GRIB_typeOfLevel = "isobaricInhPa" ;
>> > > > > > >
>> > > > > > > u:GRIB_NV = 0LL ;
>> > > > > > >
>> > > > > > > u:GRIB_stepUnits = 1LL ;
>> > > > > > >
>> > > > > > > u:GRIB_stepType = "instant" ;
>> > > > > > >
>> > > > > > > u:GRIB_gridType = "regular_ll" ;
>> > > > > > >
>> > > > > > > u:GRIB_gridDefinitionDescription = "Latitude/longitude.
Also
>> > called
>> > > > > > > equidistant cylindrical, or Plate Carree" ;
>> > > > > > >
>> > > > > > > u:GRIB_Nx = 3501LL ;
>> > > > > > >
>> > > > > > > u:GRIB_iDirectionIncrementInDegrees = 0.029999 ;
>> > > > > > >
>> > > > > > > u:GRIB_iScansNegatively = 0LL ;
>> > > > > > >
>> > > > > > > u:GRIB_longitudeOfFirstGridPointInDegrees = 245.5 ;
>> > > > > > >
>> > > > > > > u:GRIB_longitudeOfLastGridPointInDegrees = 350.499998 ;
>> > > > > > >
>> > > > > > > u:GRIB_Ny = 2001LL ;
>> > > > > > >
>> > > > > > > u:GRIB_jDirectionIncrementInDegrees = 0.03 ;
>> > > > > > >
>> > > > > > > u:GRIB_jPointsAreConsecutive = 0LL ;
>> > > > > > >
>> > > > > > > u:GRIB_jScansPositively = 1LL ;
>> > > > > > >
>> > > > > > > u:GRIB_latitudeOfFirstGridPointInDegrees = -5. ;
>> > > > > > >
>> > > > > > > u:GRIB_latitudeOfLastGridPointInDegrees = 55. ;
>> > > > > > >
>> > > > > > > u:long_name = "U component of wind" ;
>> > > > > > >
>> > > > > > > u:units = "m s**-1" ;
>> > > > > > >
>> > > > > > > u:standard_name = "eastward_wind" ;
>> > > > > > >
>> > > > > > > }
>> > > > > > >
>> > > > > > >
>> > > > > > > I want to compare the Radar data with the model output
to
>> compute
>> > > > stats
>> > > > > > > (like RMSE, ME, MAE etc). This will be a use case
within
>> > METplus. I
>> > > > > hope
>> > > > > > > GridStat is the correct tool to do this job. I used the
>> > > > RegridDataPlane
>> > > > > > > example and created a Configuration file (attached). I
have
>> > > questions
>> > > > > > about
>> > > > > > > some of the variables and what they should be in the
conf
>> file.
>> > > E.g.
>> > > > > > Where
>> > > > > > > do I mention what model output variable to compare
against in
>> the
>> > > > conf
>> > > > > > > file? Should we use "u" or "eastward_wind"? Also I am
not sure
>> > > where
>> > > > to
>> > > > > > put
>> > > > > > > in the arguments that are needed to read the python
script.
>> As of
>> > > > now I
>> > > > > > > have included it in the OBS_VAR1_NAME.
>> > > > > > >
>> > > > > > >
>> > > > > > > I will really appreciate it if you can look at the conf
file
>> and
>> > > see
>> > > > > what
>> > > > > > > corrections need to be made. And then we can diss what
next
>> steps
>> > > > > > (wrapper
>> > > > > > > script) need to be written to test the whole case.
>> > > > > > >
>> > > > > > >
>> > > > > > > Thanks
>> > > > > > >
>> > > > > > > Biswas
>> > > > > > >
>> > > > > > > On Wed, Oct 28, 2020 at 12:49 PM George McCabe via RT <
>> > > > > met_help at ucar.edu
>> > > > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Hi Biswas,
>> > > > > > > >
>> > > > > > > > I am moving this email over to MET-Help so we can
track our
>> > work
>> > > > > > easily.
>> > > > > > > >
>> > > > > > > > I sent you the RegridDataPlane example because there
are
>> > > currently
>> > > > no
>> > > > > > use
>> > > > > > > > cases in the METplus repository that call GridStat
with
>> Python
>> > > > > > embedding,
>> > > > > > > > but the variables to set should be the same as
>> RegridDataPlane.
>> > > > > Notably
>> > > > > > > > these lines need to be set correctly:
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
https://github.com/dtcenter/METplus/blob/f54d8305e305a0bfd11964fbf280b390bc7152b5/parm/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.conf#L34
>> > > > > > > >
>> > > > > > > > OBS_VAR1_NAME should be set to the path of the python
script
>> > you
>> > > > want
>> > > > > > to
>> > > > > > > > call with any arguments to the script. You can use
filename
>> > > > template
>> > > > > > > syntax
>> > > > > > > > to specify  time information based on the current run
time,
>> > i.e.
>> > > > > > > > {init?fmt=%Y%m%d%H} will be replaced with the current
>> > > > initialization
>> > > > > > time
>> > > > > > > > in YYYYMMDDHH format. If you are using Python
embedding for
>> > > > forecast
>> > > > > > > data,
>> > > > > > > > you will set FCST_VAR1_NAME instead.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
https://github.com/dtcenter/METplus/blob/f54d8305e305a0bfd11964fbf280b390bc7152b5/parm/use_cases/met_tool_wrapper/RegridDataPlane/RegridDataPlane_python_embedding.conf#L61
>> > > > > > > >
>> > > > > > > > You will also need to set the appropriate template
value to
>> > > > > > PYTHON_NUMPY.
>> > > > > > > > For obs data, you would set
OBS_GRID_STAT_INPUT_TEMPLATE and
>> > for
>> > > > > > forecast
>> > > > > > > > data you would set FCST_GRID_STAT_INPUT_TEMPLATE.
>> > > > > > > >
>> > > > > > > > Let me know if you have any issues getting this set
up and I
>> > can
>> > > > > help.
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > > George
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Wed Oct 28 12:42:44 2020, mccabe wrote:
>> > > > > > > > > Hi George,
>> > > > > > > > >  I hope you are doing fine. Long back I asked you
about
>> the
>> > > conf
>> > > > > file
>> > > > > > > > > for Gridstat. The link you sent was from the
>> RegridDataPlane.
>> > > > > > However,
>> > > > > > > > > the link doesn't work now. Is there any Gridstat
conf file
>> > > > example
>> > > > > > > > > available that utilizes python embedding. I want to
use my
>> > > python
>> > > > > > > > > script to compare the Radar data with model output.
Now
>> that
>> > > the
>> > > > > > > > > pressure to height tool is available I need to
finish the
>> > task
>> > > > > within
>> > > > > > > > > METplus.
>> > > > > > > > > Thanks
>> > > > > > > > > Biswas
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > >
>> > > > > > > *My working day may not be your working day. Please do
not
>> feel
>> > > > obliged
>> > > > > > to
>> > > > > > > reply to this email outside of your normal working
hours.*
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > George McCabe - Software Engineer III
>> > > > > > National Center for Atmospheric Research
>> > > > > > Research Applications Laboratory
>> > > > > > 303-497-2768
>> > > > > > ---
>> > > > > > My working day may not be your working day. Please do not
feel
>> > > obliged
>> > > > to
>> > > > > > reply to this email outside of your normal working hours.
>> > > > > >
>> > > > > >
>> > > > >
>> > > > > --
>> > > > >
>> > > > > *My working day may not be your working day. Please do not
feel
>> > obliged
>> > > > to
>> > > > > reply to this email outside of your normal working hours.*
>> > > > >
>> > > > >
>> > > >
>> > > > --
>> > > > George McCabe - Software Engineer III
>> > > > National Center for Atmospheric Research
>> > > > Research Applications Laboratory
>> > > > 303-497-2768
>> > > > ---
>> > > > My working day may not be your working day. Please do not
feel
>> obliged
>> > to
>> > > > reply to this email outside of your normal working hours.
>> > > >
>> > > >
>> > >
>> > > --
>> > >
>> > > *My working day may not be your working day. Please do not feel
>> obliged
>> > to
>> > > reply to this email outside of your normal working hours.*
>> > >
>> > >
>> >
>> > --
>> > George McCabe - Software Engineer III
>> > National Center for Atmospheric Research
>> > Research Applications Laboratory
>> > 303-497-2768
>> > ---
>> > My working day may not be your working day. Please do not feel
obliged
>> to
>> > reply to this email outside of your normal working hours.
>> >
>> >
>>
>> --
>>
>> *My working day may not be your working day. Please do not feel
obliged to
>> reply to this email outside of your normal working hours.*
>>
>>
>
> --
> George McCabe - Software Engineer III
> National Center for Atmospheric Research
> Research Applications Laboratory
> 303-497-2768
> ---
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>

--
George McCabe - Software Engineer III
National Center for Atmospheric Research
Research Applications Laboratory
303-497-2768
---
My working day may not be your working day. Please do not feel obliged
to
reply to this email outside of your normal working hours.

------------------------------------------------