[Met_help] [rt.rap.ucar.edu #94783] History for Help on MODE Time Domain Tool
John Halley Gotway via RT
met_help at ucar.edu
Wed May 6 16:36:04 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Dear all,
I am a PostDoc at ICTP in Trieste, working with convection-permitting
climate simulations in the projects CORDEX-FPS and EUCP, and I am trying
to replicate the analysis done by Prein et al., 2017, but for Europe,
meaning to identify and sample mesoscale convective systems, using MODE mtd.
I got MET successfully installed and I here only want to use mtd on the
simulations, no observations for the moment.
So what I do is:
> ../MET/met-9.0/bin/mtd -single
> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
> Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config MODEConfig_SKM
where Month_oooo__pr_....nc is my model output, to be found here (1GB):
https://owncloud.gwdg.de/index.php/s/aVGr0Gql5tKVUcR
and the config file is attached and basically unchanged.
The error message is:
> DEBUG 2: mtd_read_data() -> processing file
> "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
> WARNING:
> WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) ->
> returns the first available time for "pr" variable).
> WARNING:
> ERROR :
> ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &)
> const -> needed 3 arguments for variable pr, got 2
> ERROR :
The remapping would be specified in this part of the config file!?:
regrid = {
to_grid = NONE;
method = NEAREST;
width = 1;
vld_thresh = 0.5;
shape = SQUARE;
}
Should that move into fcst?
Also attached you find info on the output i'm working on.
I would be very thankful if you could help me.
With best regards
Sebastian
--
*********************************************************************
Sebastian K. Müller
Earth System Physics Section
The Abdus Salam International Centre for Theoretical Physics
Strada Costiera 11
34100 Trieste, ITALY
phone: + 39 040 2240 438
email: smueller at ictp.it
*********************************************************************
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Help on MODE Time Domain Tool
From: John Halley Gotway
Time: Mon Mar 30 21:33:50 2020
Hello Sebastian,
I see you have questions about running the MODE time domain tool. I
was
able to download your sample NetCDF file. Looking at the history
attribute
in the NetCDF header, I see that you constructed this file by running
the
"cdo mergetime" utility. So you've merged many output files into a
single
one that contains many timesteps:
:history = "Thu Dec 12 08:06:55 2019: cdo mergetime
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060100-2002060200.nc
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060200-2002060300.nc
...
MTD is setup to process the raw, unmerged files. It expects to read
the
same field of data from multiple files. So if you still have the
original
files available, each containing a single time step, please try
running it
that way.
Here is the usage statement for mtd:
Usage: mtd
-fcst file_1 ... file_n | file_list
-obs file_1 ... file_n | file_list
-single file_1 ... file_n | file_list
-config config_file
[-outdir path]
[-log file]
[-v level]
Since you're running with a single data source, you should use the
"-single" command line option. You can either list all the input
files on
the command line or put that list into an ascii file and put that
ascii
file list on the command line.
You're running MTD but you sent me a MODE config file. Make sure
you're
using the default config file for MTD:
fcst = {
field = {
name = "pr";
level = "(0,*,*)";
}
...
Hope that helps.
Thanks,
John Halley Gotway
On Mon, Mar 30, 2020 at 5:21 PM Sebastian K. Müller via RT <
met_help at ucar.edu> wrote:
>
> Mon Mar 30 17:20:51 2020: Request 94783 was acted upon.
> Transaction: Ticket created by smueller at ictp.it
> Queue: met_help
> Subject: Help on MODE Time Domain Tool
> Owner: Nobody
> Requestors: smueller at ictp.it
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
>
>
> Dear all,
>
> I am a PostDoc at ICTP in Trieste, working with convection-
permitting
> climate simulations in the projects CORDEX-FPS and EUCP, and I am
trying
> to replicate the analysis done by Prein et al., 2017, but for
Europe,
> meaning to identify and sample mesoscale convective systems, using
MODE
> mtd.
>
> I got MET successfully installed and I here only want to use mtd on
the
> simulations, no observations for the moment.
>
> So what I do is:
>
> > ../MET/met-9.0/bin/mtd -single
> > Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
> > Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config MODEConfig_SKM
> where Month_oooo__pr_....nc is my model output, to be found here
(1GB):
> https://owncloud.gwdg.de/index.php/s/aVGr0Gql5tKVUcR
> and the config file is attached and basically unchanged.
>
> The error message is:
>
> > DEBUG 2: mtd_read_data() -> processing file
> > "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
> > WARNING:
> > WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) ->
> > returns the first available time for "pr" variable).
> > WARNING:
> > ERROR :
> > ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane
&)
> > const -> needed 3 arguments for variable pr, got 2
> > ERROR :
>
> The remapping would be specified in this part of the config file!?:
>
> regrid = {
> to_grid = NONE;
> method = NEAREST;
> width = 1;
> vld_thresh = 0.5;
> shape = SQUARE;
> }
>
> Should that move into fcst?
>
> Also attached you find info on the output i'm working on.
>
> I would be very thankful if you could help me.
>
> With best regards
> Sebastian
>
> --
>
*********************************************************************
> Sebastian K. Müller
> Earth System Physics Section
> The Abdus Salam International Centre for Theoretical Physics
> Strada Costiera 11
> 34100 Trieste, ITALY
> phone: + 39 040 2240 438
> email: smueller at ictp.it
>
*********************************************************************
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #94783] Help on MODE Time Domain Tool
From: Sebastian K. Müller
Time: Tue Mar 31 02:19:11 2020
Hello Hailey,
thank you. That already heldped a lot. Now I at least work with the
right configure file. And it is not giving me errors, but not writing
content to the output files either...
> smueller at login02 E_OBS $ ../MET/met-9.0/bin/mtd -single
> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
> Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config MTDConfig_SKM
> -outdir /local_scratch/smueller/MTD_output/
> DEBUG 2: mtd_read_data() -> processing file
> "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
> DEBUG 2: mtd_read_data() -> processing file
> "Month_oooo__pr_HadGEM_HIST_072002_pyco.nc"
> DEBUG 2: regridding, if needed ...
> DEBUG 2: Splitting object field
> DEBUG 2: Done splitting
> DEBUG 2: Calculating 3D fcst single attributes
> DEBUG 2: Calculating 2D fcst attributes
> DEBUG 2: Creating 2D constant-time slice attributes file:
> "/local_scratch/smueller/MTD_output//mtd_20010830_003000V_2d.txt"
> DEBUG 2: Creating 3D single simple attributes file:
>
"/local_scratch/smueller/MTD_output//mtd_20010830_003000V_3d_single_simple.txt"
> DEBUG 2: Creating NetCDF file:
> "/local_scratch/smueller/MTD_output//mtd_20010830_003000V_obj.nc"
So, it would be a big effort to produce files of every output time
step.
We save the original output "daily", and I merge them to months. Is
there a way around this?
Thanks again,
Cheers
Sebastian
Am 31.03.20 um 05:33 schrieb John Halley Gotway via RT:
> Hello Sebastian,
>
> I see you have questions about running the MODE time domain tool. I
was
> able to download your sample NetCDF file. Looking at the history
attribute
> in the NetCDF header, I see that you constructed this file by
running the
> "cdo mergetime" utility. So you've merged many output files into a
single
> one that contains many timesteps:
>
> :history = "Thu Dec 12 08:06:55 2019: cdo mergetime
>
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060100-2002060200.nc
>
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060200-2002060300.nc
> ...
>
> MTD is setup to process the raw, unmerged files. It expects to read
the
> same field of data from multiple files. So if you still have the
original
> files available, each containing a single time step, please try
running it
> that way.
>
> Here is the usage statement for mtd:
>
> Usage: mtd
> -fcst file_1 ... file_n | file_list
> -obs file_1 ... file_n | file_list
> -single file_1 ... file_n | file_list
> -config config_file
> [-outdir path]
> [-log file]
> [-v level]
>
> Since you're running with a single data source, you should use the
> "-single" command line option. You can either list all the input
files on
> the command line or put that list into an ascii file and put that
ascii
> file list on the command line.
>
> You're running MTD but you sent me a MODE config file. Make sure
you're
> using the default config file for MTD:
>
> fcst = {
> field = {
> name = "pr";
> level = "(0,*,*)";
> }
> ...
>
>
> Hope that helps.
>
> Thanks,
> John Halley Gotway
>
>
> On Mon, Mar 30, 2020 at 5:21 PM Sebastian K. Müller via RT <
> met_help at ucar.edu> wrote:
>
>> Mon Mar 30 17:20:51 2020: Request 94783 was acted upon.
>> Transaction: Ticket created by smueller at ictp.it
>> Queue: met_help
>> Subject: Help on MODE Time Domain Tool
>> Owner: Nobody
>> Requestors: smueller at ictp.it
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
>>
>>
>> Dear all,
>>
>> I am a PostDoc at ICTP in Trieste, working with convection-
permitting
>> climate simulations in the projects CORDEX-FPS and EUCP, and I am
trying
>> to replicate the analysis done by Prein et al., 2017, but for
Europe,
>> meaning to identify and sample mesoscale convective systems, using
MODE
>> mtd.
>>
>> I got MET successfully installed and I here only want to use mtd on
the
>> simulations, no observations for the moment.
>>
>> So what I do is:
>>
>>> ../MET/met-9.0/bin/mtd -single
>>> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
>>> Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config MODEConfig_SKM
>> where Month_oooo__pr_....nc is my model output, to be found here
(1GB):
>> https://owncloud.gwdg.de/index.php/s/aVGr0Gql5tKVUcR
>> and the config file is attached and basically unchanged.
>>
>> The error message is:
>>
>>> DEBUG 2: mtd_read_data() -> processing file
>>> "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
>>> WARNING:
>>> WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) ->
>>> returns the first available time for "pr" variable).
>>> WARNING:
>>> ERROR :
>>> ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane
&)
>>> const -> needed 3 arguments for variable pr, got 2
>>> ERROR :
>> The remapping would be specified in this part of the config file!?:
>>
>> regrid = {
>> to_grid = NONE;
>> method = NEAREST;
>> width = 1;
>> vld_thresh = 0.5;
>> shape = SQUARE;
>> }
>>
>> Should that move into fcst?
>>
>> Also attached you find info on the output i'm working on.
>>
>> I would be very thankful if you could help me.
>>
>> With best regards
>> Sebastian
>>
>> --
>>
*********************************************************************
>> Sebastian K. Müller
>> Earth System Physics Section
>> The Abdus Salam International Centre for Theoretical Physics
>> Strada Costiera 11
>> 34100 Trieste, ITALY
>> phone: + 39 040 2240 438
>> email: smueller at ictp.it
>>
*********************************************************************
>>
>>
>>
--
*********************************************************************
Sebastian K. Müller
Earth System Physics Section
The Abdus Salam International Centre for Theoretical Physics
Strada Costiera 11
34100 Trieste, ITALY
phone: + 39 040 2240 438
email: smueller at ictp.it
*********************************************************************
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #94783] Help on MODE Time Domain Tool
From: Sebastian K. Müller
Time: Tue Mar 31 15:22:33 2020
Hello again,
now, splitting up the file into individual time steps works and is not
too much of an effort. Still I would be interested in whether it could
be possible to do mtg with monthly files...
mtg works now and I will be tuning the parameters now. Maybe just a
few
questions:
the "smoothing radius" of Prein et al. 2017 is "conv_radius" in the
config_file? And by default "60/grid_res", meaning that the smoothing
is
done over a square of 20*20 grid cells, with "grid_res=3.". Is that
correct? Does this interfere in any with the regrid command?
I think that's it for now.
Thanks for your help,
Cheers
Sebastian
Am 31.03.20 um 10:19 schrieb Sebastian K. Müller:
> Hello Hailey,
>
> thank you. That already heldped a lot. Now I at least work with the
> right configure file. And it is not giving me errors, but not
writing
> content to the output files either...
>
>> smueller at login02 E_OBS $ ../MET/met-9.0/bin/mtd -single
>> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
>> Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config MTDConfig_SKM
>> -outdir /local_scratch/smueller/MTD_output/
>> DEBUG 2: mtd_read_data() -> processing file
>> "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
>> DEBUG 2: mtd_read_data() -> processing file
>> "Month_oooo__pr_HadGEM_HIST_072002_pyco.nc"
>> DEBUG 2: regridding, if needed ...
>> DEBUG 2: Splitting object field
>> DEBUG 2: Done splitting
>> DEBUG 2: Calculating 3D fcst single attributes
>> DEBUG 2: Calculating 2D fcst attributes
>> DEBUG 2: Creating 2D constant-time slice attributes file:
>> "/local_scratch/smueller/MTD_output//mtd_20010830_003000V_2d.txt"
>> DEBUG 2: Creating 3D single simple attributes file:
>>
"/local_scratch/smueller/MTD_output//mtd_20010830_003000V_3d_single_simple.txt"
>> DEBUG 2: Creating NetCDF file:
>> "/local_scratch/smueller/MTD_output//mtd_20010830_003000V_obj.nc"
> So, it would be a big effort to produce files of every output time
> step. We save the original output "daily", and I merge them to
months.
> Is there a way around this?
>
> Thanks again,
> Cheers
> Sebastian
>
> Am 31.03.20 um 05:33 schrieb John Halley Gotway via RT:
>> Hello Sebastian,
>>
>> I see you have questions about running the MODE time domain tool.
I was
>> able to download your sample NetCDF file. Looking at the history
>> attribute
>> in the NetCDF header, I see that you constructed this file by
running
>> the
>> "cdo mergetime" utility. So you've merged many output files into a
>> single
>> one that contains many timesteps:
>>
>> :history = "Thu Dec 12 08:06:55 2019: cdo mergetime
>>
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060100-2002060200.nc
>>
>>
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060200-2002060300.nc
>>
>> ...
>>
>> MTD is setup to process the raw, unmerged files. It expects to
read the
>> same field of data from multiple files. So if you still have the
>> original
>> files available, each containing a single time step, please try
>> running it
>> that way.
>>
>> Here is the usage statement for mtd:
>>
>> Usage: mtd
>> -fcst file_1 ... file_n | file_list
>> -obs file_1 ... file_n | file_list
>> -single file_1 ... file_n | file_list
>> -config config_file
>> [-outdir path]
>> [-log file]
>> [-v level]
>>
>> Since you're running with a single data source, you should use the
>> "-single" command line option. You can either list all the input
>> files on
>> the command line or put that list into an ascii file and put that
ascii
>> file list on the command line.
>>
>> You're running MTD but you sent me a MODE config file. Make sure
you're
>> using the default config file for MTD:
>>
>> fcst = {
>> field = {
>> name = "pr";
>> level = "(0,*,*)";
>> }
>> ...
>>
>>
>> Hope that helps.
>>
>> Thanks,
>> John Halley Gotway
>>
>>
>> On Mon, Mar 30, 2020 at 5:21 PM Sebastian K. Müller via RT <
>> met_help at ucar.edu> wrote:
>>
>>> Mon Mar 30 17:20:51 2020: Request 94783 was acted upon.
>>> Transaction: Ticket created by smueller at ictp.it
>>> Queue: met_help
>>> Subject: Help on MODE Time Domain Tool
>>> Owner: Nobody
>>> Requestors: smueller at ictp.it
>>> Status: new
>>> Ticket <URL:
>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
>>>
>>>
>>> Dear all,
>>>
>>> I am a PostDoc at ICTP in Trieste, working with convection-
permitting
>>> climate simulations in the projects CORDEX-FPS and EUCP, and I am
>>> trying
>>> to replicate the analysis done by Prein et al., 2017, but for
Europe,
>>> meaning to identify and sample mesoscale convective systems, using
MODE
>>> mtd.
>>>
>>> I got MET successfully installed and I here only want to use mtd
on the
>>> simulations, no observations for the moment.
>>>
>>> So what I do is:
>>>
>>>> ../MET/met-9.0/bin/mtd -single
>>>> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
>>>> Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config MODEConfig_SKM
>>> where Month_oooo__pr_....nc is my model output, to be found here
(1GB):
>>> https://owncloud.gwdg.de/index.php/s/aVGr0Gql5tKVUcR
>>> and the config file is attached and basically unchanged.
>>>
>>> The error message is:
>>>
>>>> DEBUG 2: mtd_read_data() -> processing file
>>>> "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
>>>> WARNING:
>>>> WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) ->
>>>> returns the first available time for "pr" variable).
>>>> WARNING:
>>>> ERROR :
>>>> ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane
&)
>>>> const -> needed 3 arguments for variable pr, got 2
>>>> ERROR :
>>> The remapping would be specified in this part of the config
file!?:
>>>
>>> regrid = {
>>> to_grid = NONE;
>>> method = NEAREST;
>>> width = 1;
>>> vld_thresh = 0.5;
>>> shape = SQUARE;
>>> }
>>>
>>> Should that move into fcst?
>>>
>>> Also attached you find info on the output i'm working on.
>>>
>>> I would be very thankful if you could help me.
>>>
>>> With best regards
>>> Sebastian
>>>
>>> --
>>>
*********************************************************************
>>> Sebastian K. Müller
>>> Earth System Physics Section
>>> The Abdus Salam International Centre for Theoretical Physics
>>> Strada Costiera 11
>>> 34100 Trieste, ITALY
>>> phone: + 39 040 2240 438
>>> email: smueller at ictp.it
>>>
*********************************************************************
>>>
>>>
>>>
--
*********************************************************************
Sebastian K. Müller
Earth System Physics Section
The Abdus Salam International Centre for Theoretical Physics
Strada Costiera 11
34100 Trieste, ITALY
phone: + 39 040 2240 438
email: smueller at ictp.it
*********************************************************************
------------------------------------------------
Subject: Help on MODE Time Domain Tool
From: John Halley Gotway
Time: Tue Mar 31 17:31:41 2020
Sebastian,
Yes, the smoothing radius is the "conv_radius" in the config file.
That
stands for convolution radius. It's the radius of the smoother
defined in
grid squares. If your data is already very smooth, you could set it
to 0
to disable the spatial smoothing. But your precip data probably is
NOT
very smooth. We intend the "grid_res" to be the nominal grid spacing
in
km. It's used a lot in the MODE config file so that you only need to
change one setting to get reasonable defaults for a few other
settings.
Looks like it's only used once in MTD... to set the conv_radius. You
can
manually define the conv_radius however you'd like (e.g. conv_radius =
5;).
Also note that met-9.0 includes the ability to define the amount of
smoothing in time. The default conv_time_window smooths data across
one
prior time step (beg = -1) and the next time step (end = 1). Prior to
met-9.0, those settings were hard-coded, but they're configurable now.
In MTD, as in all the other MET tools, regridding is done right after
the
2D data is read from the file. It operates totally independently of
the
smoothing that's done in MTD.
As for reading multiple time steps from the same input file, there is
a
workaround we can do using python embedding. Instead of having MET
read
from the NetCDF input file directly, we run a python script to do the
reading and then pass that gridded data and metadata to the MET tools
in
memory. I adapted one of the examples from this page
https://dtcenter.org/community-code/model-evaluation-tools-met/sample-
analysis-scripts
to
create the attached read_pr_timestep.py script.
Run that script like this to make sure it works for you:
*python3 read_pr_timestep.py Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
10*
This should read the 11-th (0-based) timestep of the "pr" variable.
If
that works, then try running it with MET's plot_data_plane tool:
*plot_data_plane PYTHON_NUMPY pr_python.ps <http://pr_python.ps>
'name="read_pr_timestep.py Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
10";'*
Next, I compared the resulting image with MET plotting that data
directly
to make sure it's oriented correctly.
*plot_data_plane Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
pr_direct.ps
<http://pr_direct.ps> 'name="pr"; level="(10,*,*)";'*
I've attached the resulting png's and they match.
So why go to all this trouble? Here's the punchline... you can now
run MTD
like this:
*mtd -single 0 1 2 3 4 5 6 7 8 9 10 11 12 -config MTDConfig_SKM_python
-outdir out*
And inside the MTD config file, use this:
*fcst = {*
* file_type = PYTHON_NUMPY;*
* field = { name = "read_pr_timestep.py
Month_oooo__pr_HadGEM_HIST_062002_pyco.nc MET_PYTHON_INPUT_ARG";*
* }*
See the attached MTD config file.
This runs MTD for the first 13 timesteps. For each input argument in
the
"-single" list, it executes the python script defined in the config
file,
substituting in the current value into the "MET_PYTHON_INPUT_ARG"
slot.
This is a pretty nice usage of features we added in met-9.0!
John
On Tue, Mar 31, 2020 at 3:22 PM Sebastian K. Müller via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
>
> Hello again,
>
> now, splitting up the file into individual time steps works and is
not
> too much of an effort. Still I would be interested in whether it
could
> be possible to do mtg with monthly files...
>
> mtg works now and I will be tuning the parameters now. Maybe just a
few
> questions:
>
> the "smoothing radius" of Prein et al. 2017 is "conv_radius" in the
> config_file? And by default "60/grid_res", meaning that the
smoothing is
> done over a square of 20*20 grid cells, with "grid_res=3.". Is that
> correct? Does this interfere in any with the regrid command?
>
> I think that's it for now.
>
> Thanks for your help,
> Cheers
> Sebastian
>
> Am 31.03.20 um 10:19 schrieb Sebastian K. Müller:
> > Hello Hailey,
> >
> > thank you. That already heldped a lot. Now I at least work with
the
> > right configure file. And it is not giving me errors, but not
writing
> > content to the output files either...
> >
> >> smueller at login02 E_OBS $ ../MET/met-9.0/bin/mtd -single
> >> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
> >> Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config MTDConfig_SKM
> >> -outdir /local_scratch/smueller/MTD_output/
> >> DEBUG 2: mtd_read_data() -> processing file
> >> "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
> >> DEBUG 2: mtd_read_data() -> processing file
> >> "Month_oooo__pr_HadGEM_HIST_072002_pyco.nc"
> >> DEBUG 2: regridding, if needed ...
> >> DEBUG 2: Splitting object field
> >> DEBUG 2: Done splitting
> >> DEBUG 2: Calculating 3D fcst single attributes
> >> DEBUG 2: Calculating 2D fcst attributes
> >> DEBUG 2: Creating 2D constant-time slice attributes file:
> >> "/local_scratch/smueller/MTD_output//mtd_20010830_003000V_2d.txt"
> >> DEBUG 2: Creating 3D single simple attributes file:
> >>
>
"/local_scratch/smueller/MTD_output//mtd_20010830_003000V_3d_single_simple.txt"
> >> DEBUG 2: Creating NetCDF file:
> >> "/local_scratch/smueller/MTD_output//mtd_20010830_003000V_obj.nc"
> > So, it would be a big effort to produce files of every output time
> > step. We save the original output "daily", and I merge them to
months.
> > Is there a way around this?
> >
> > Thanks again,
> > Cheers
> > Sebastian
> >
> > Am 31.03.20 um 05:33 schrieb John Halley Gotway via RT:
> >> Hello Sebastian,
> >>
> >> I see you have questions about running the MODE time domain tool.
I was
> >> able to download your sample NetCDF file. Looking at the history
> >> attribute
> >> in the NetCDF header, I see that you constructed this file by
running
> >> the
> >> "cdo mergetime" utility. So you've merged many output files into
a
> >> single
> >> one that contains many timesteps:
> >>
> >> :history = "Thu Dec 12 08:06:55 2019: cdo mergetime
> >>
>
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060100-
> 2002060200.nc
> >>
> >>
>
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060200-
> 2002060300.nc
> >>
> >> ...
> >>
> >> MTD is setup to process the raw, unmerged files. It expects to
read the
> >> same field of data from multiple files. So if you still have the
> >> original
> >> files available, each containing a single time step, please try
> >> running it
> >> that way.
> >>
> >> Here is the usage statement for mtd:
> >>
> >> Usage: mtd
> >> -fcst file_1 ... file_n | file_list
> >> -obs file_1 ... file_n | file_list
> >> -single file_1 ... file_n | file_list
> >> -config config_file
> >> [-outdir path]
> >> [-log file]
> >> [-v level]
> >>
> >> Since you're running with a single data source, you should use
the
> >> "-single" command line option. You can either list all the input
> >> files on
> >> the command line or put that list into an ascii file and put that
ascii
> >> file list on the command line.
> >>
> >> You're running MTD but you sent me a MODE config file. Make sure
you're
> >> using the default config file for MTD:
> >>
> >> fcst = {
> >> field = {
> >> name = "pr";
> >> level = "(0,*,*)";
> >> }
> >> ...
> >>
> >>
> >> Hope that helps.
> >>
> >> Thanks,
> >> John Halley Gotway
> >>
> >>
> >> On Mon, Mar 30, 2020 at 5:21 PM Sebastian K. Müller via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >>> Mon Mar 30 17:20:51 2020: Request 94783 was acted upon.
> >>> Transaction: Ticket created by smueller at ictp.it
> >>> Queue: met_help
> >>> Subject: Help on MODE Time Domain Tool
> >>> Owner: Nobody
> >>> Requestors: smueller at ictp.it
> >>> Status: new
> >>> Ticket <URL:
> >>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
> >>>
> >>>
> >>> Dear all,
> >>>
> >>> I am a PostDoc at ICTP in Trieste, working with convection-
permitting
> >>> climate simulations in the projects CORDEX-FPS and EUCP, and I
am
> >>> trying
> >>> to replicate the analysis done by Prein et al., 2017, but for
Europe,
> >>> meaning to identify and sample mesoscale convective systems,
using MODE
> >>> mtd.
> >>>
> >>> I got MET successfully installed and I here only want to use mtd
on the
> >>> simulations, no observations for the moment.
> >>>
> >>> So what I do is:
> >>>
> >>>> ../MET/met-9.0/bin/mtd -single
> >>>> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
> >>>> Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config
MODEConfig_SKM
> >>> where Month_oooo__pr_....nc is my model output, to be found here
(1GB):
> >>> https://owncloud.gwdg.de/index.php/s/aVGr0Gql5tKVUcR
> >>> and the config file is attached and basically unchanged.
> >>>
> >>> The error message is:
> >>>
> >>>> DEBUG 2: mtd_read_data() -> processing file
> >>>> "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
> >>>> WARNING:
> >>>> WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) ->
> >>>> returns the first available time for "pr" variable).
> >>>> WARNING:
> >>>> ERROR :
> >>>> ERROR : NcCfFile::getData(NcVar *, const LongArray &,
DataPlane &)
> >>>> const -> needed 3 arguments for variable pr, got 2
> >>>> ERROR :
> >>> The remapping would be specified in this part of the config
file!?:
> >>>
> >>> regrid = {
> >>> to_grid = NONE;
> >>> method = NEAREST;
> >>> width = 1;
> >>> vld_thresh = 0.5;
> >>> shape = SQUARE;
> >>> }
> >>>
> >>> Should that move into fcst?
> >>>
> >>> Also attached you find info on the output i'm working on.
> >>>
> >>> I would be very thankful if you could help me.
> >>>
> >>> With best regards
> >>> Sebastian
> >>>
> >>> --
> >>>
*********************************************************************
> >>> Sebastian K. Müller
> >>> Earth System Physics Section
> >>> The Abdus Salam International Centre for Theoretical Physics
> >>> Strada Costiera 11
> >>> 34100 Trieste, ITALY
> >>> phone: + 39 040 2240 438
> >>> email: smueller at ictp.it
> >>>
*********************************************************************
> >>>
> >>>
> >>>
> --
>
*********************************************************************
> Sebastian K. Müller
> Earth System Physics Section
> The Abdus Salam International Centre for Theoretical Physics
> Strada Costiera 11
> 34100 Trieste, ITALY
> phone: + 39 040 2240 438
> email: smueller at ictp.it
>
*********************************************************************
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #94783] Help on MODE Time Domain Tool
From: Sebastian K. Müller
Time: Mon Apr 06 19:31:52 2020
John,
thank you very much! The python-interface you suggested works
perfectly
and I get around splitting up the output into time steps.
Just two more little bits:
First, to be sure: conv_radius is eventually set as an integer of grid
cells!? I assume so.
Second: the smoothing in time is done by setting again a "radius of
timesteps". So beg=-1 and end=-1, smoothes over three timesteps, that
are [-1,0,1]. So if I set beg=0, and end=0, there is no smoothing in
time!? From Prein et al., 2017, Clim. Dyn, I do not see whether they
applied smoothing in time. Do you know if they did?
Generally, my many thanks: this toolbox is very well documented, easy
to
apply and the help of the team is great!
Best regards
Sebastian
Am 01.04.20 um 01:31 schrieb John Halley Gotway via RT:
> Sebastian,
>
> Yes, the smoothing radius is the "conv_radius" in the config file.
That
> stands for convolution radius. It's the radius of the smoother
defined in
> grid squares. If your data is already very smooth, you could set it
to 0
> to disable the spatial smoothing. But your precip data probably is
NOT
> very smooth. We intend the "grid_res" to be the nominal grid
spacing in
> km. It's used a lot in the MODE config file so that you only need
to
> change one setting to get reasonable defaults for a few other
settings.
> Looks like it's only used once in MTD... to set the conv_radius.
You can
> manually define the conv_radius however you'd like (e.g. conv_radius
= 5;).
>
> Also note that met-9.0 includes the ability to define the amount of
> smoothing in time. The default conv_time_window smooths data across
one
> prior time step (beg = -1) and the next time step (end = 1). Prior
to
> met-9.0, those settings were hard-coded, but they're configurable
now.
>
> In MTD, as in all the other MET tools, regridding is done right
after the
> 2D data is read from the file. It operates totally independently of
the
> smoothing that's done in MTD.
>
> As for reading multiple time steps from the same input file, there
is a
> workaround we can do using python embedding. Instead of having MET
read
> from the NetCDF input file directly, we run a python script to do
the
> reading and then pass that gridded data and metadata to the MET
tools in
> memory. I adapted one of the examples from this page
> https://dtcenter.org/community-code/model-evaluation-tools-
met/sample-analysis-scripts
> to
> create the attached read_pr_timestep.py script.
>
> Run that script like this to make sure it works for you:
> *python3 read_pr_timestep.py
Month_oooo__pr_HadGEM_HIST_062002_pyco.nc 10*
>
> This should read the 11-th (0-based) timestep of the "pr" variable.
If
> that works, then try running it with MET's plot_data_plane tool:
> *plot_data_plane PYTHON_NUMPY pr_python.ps<http://pr_python.ps>
> 'name="read_pr_timestep.py Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
10";'*
>
> Next, I compared the resulting image with MET plotting that data
directly
> to make sure it's oriented correctly.
> *plot_data_plane Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
pr_direct.ps
> <http://pr_direct.ps> 'name="pr"; level="(10,*,*)";'*
>
> I've attached the resulting png's and they match.
>
> So why go to all this trouble? Here's the punchline... you can now
run MTD
> like this:
>
> *mtd -single 0 1 2 3 4 5 6 7 8 9 10 11 12 -config
MTDConfig_SKM_python
> -outdir out*
>
> And inside the MTD config file, use this:
> *fcst = {*
> * file_type = PYTHON_NUMPY;*
>
> * field = { name = "read_pr_timestep.py
> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc MET_PYTHON_INPUT_ARG";*
> * }*
>
> See the attached MTD config file.
>
> This runs MTD for the first 13 timesteps. For each input argument
in the
> "-single" list, it executes the python script defined in the config
file,
> substituting in the current value into the "MET_PYTHON_INPUT_ARG"
slot.
>
> This is a pretty nice usage of features we added in met-9.0!
>
> John
>
> On Tue, Mar 31, 2020 at 3:22 PM Sebastian K. Müller via RT <
> met_help at ucar.edu> wrote:
>
>> <URL:https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
>>
>> Hello again,
>>
>> now, splitting up the file into individual time steps works and is
not
>> too much of an effort. Still I would be interested in whether it
could
>> be possible to do mtg with monthly files...
>>
>> mtg works now and I will be tuning the parameters now. Maybe just a
few
>> questions:
>>
>> the "smoothing radius" of Prein et al. 2017 is "conv_radius" in the
>> config_file? And by default "60/grid_res", meaning that the
smoothing is
>> done over a square of 20*20 grid cells, with "grid_res=3.". Is that
>> correct? Does this interfere in any with the regrid command?
>>
>> I think that's it for now.
>>
>> Thanks for your help,
>> Cheers
>> Sebastian
>>
>> Am 31.03.20 um 10:19 schrieb Sebastian K. Müller:
>>> Hello Hailey,
>>>
>>> thank you. That already heldped a lot. Now I at least work with
the
>>> right configure file. And it is not giving me errors, but not
writing
>>> content to the output files either...
>>>
>>>> smueller at login02 E_OBS $ ../MET/met-9.0/bin/mtd -single
>>>> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
>>>> Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config MTDConfig_SKM
>>>> -outdir /local_scratch/smueller/MTD_output/
>>>> DEBUG 2: mtd_read_data() -> processing file
>>>> "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
>>>> DEBUG 2: mtd_read_data() -> processing file
>>>> "Month_oooo__pr_HadGEM_HIST_072002_pyco.nc"
>>>> DEBUG 2: regridding, if needed ...
>>>> DEBUG 2: Splitting object field
>>>> DEBUG 2: Done splitting
>>>> DEBUG 2: Calculating 3D fcst single attributes
>>>> DEBUG 2: Calculating 2D fcst attributes
>>>> DEBUG 2: Creating 2D constant-time slice attributes file:
>>>> "/local_scratch/smueller/MTD_output//mtd_20010830_003000V_2d.txt"
>>>> DEBUG 2: Creating 3D single simple attributes file:
>>>>
>>
"/local_scratch/smueller/MTD_output//mtd_20010830_003000V_3d_single_simple.txt"
>>>> DEBUG 2: Creating NetCDF file:
>>>> "/local_scratch/smueller/MTD_output//mtd_20010830_003000V_obj.nc"
>>> So, it would be a big effort to produce files of every output time
>>> step. We save the original output "daily", and I merge them to
months.
>>> Is there a way around this?
>>>
>>> Thanks again,
>>> Cheers
>>> Sebastian
>>>
>>> Am 31.03.20 um 05:33 schrieb John Halley Gotway via RT:
>>>> Hello Sebastian,
>>>>
>>>> I see you have questions about running the MODE time domain tool.
I was
>>>> able to download your sample NetCDF file. Looking at the history
>>>> attribute
>>>> in the NetCDF header, I see that you constructed this file by
running
>>>> the
>>>> "cdo mergetime" utility. So you've merged many output files into
a
>>>> single
>>>> one that contains many timesteps:
>>>>
>>>> :history = "Thu Dec 12 08:06:55 2019: cdo mergetime
>>>>
>>
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060100-
>> 2002060200.nc
>>
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060200-
>> 2002060300.nc
>>>> ...
>>>>
>>>> MTD is setup to process the raw, unmerged files. It expects to
read the
>>>> same field of data from multiple files. So if you still have the
>>>> original
>>>> files available, each containing a single time step, please try
>>>> running it
>>>> that way.
>>>>
>>>> Here is the usage statement for mtd:
>>>>
>>>> Usage: mtd
>>>> -fcst file_1 ... file_n | file_list
>>>> -obs file_1 ... file_n | file_list
>>>> -single file_1 ... file_n | file_list
>>>> -config config_file
>>>> [-outdir path]
>>>> [-log file]
>>>> [-v level]
>>>>
>>>> Since you're running with a single data source, you should use
the
>>>> "-single" command line option. You can either list all the input
>>>> files on
>>>> the command line or put that list into an ascii file and put that
ascii
>>>> file list on the command line.
>>>>
>>>> You're running MTD but you sent me a MODE config file. Make sure
you're
>>>> using the default config file for MTD:
>>>>
>>>> fcst = {
>>>> field = {
>>>> name = "pr";
>>>> level = "(0,*,*)";
>>>> }
>>>> ...
>>>>
>>>>
>>>> Hope that helps.
>>>>
>>>> Thanks,
>>>> John Halley Gotway
>>>>
>>>>
>>>> On Mon, Mar 30, 2020 at 5:21 PM Sebastian K. Müller via RT <
>>>> met_help at ucar.edu> wrote:
>>>>
>>>>> Mon Mar 30 17:20:51 2020: Request 94783 was acted upon.
>>>>> Transaction: Ticket created bysmueller at ictp.it
>>>>> Queue: met_help
>>>>> Subject: Help on MODE Time Domain Tool
>>>>> Owner: Nobody
>>>>> Requestors:smueller at ictp.it
>>>>> Status: new
>>>>> Ticket<URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
>>>>>
>>>>>
>>>>> Dear all,
>>>>>
>>>>> I am a PostDoc at ICTP in Trieste, working with convection-
permitting
>>>>> climate simulations in the projects CORDEX-FPS and EUCP, and I
am
>>>>> trying
>>>>> to replicate the analysis done by Prein et al., 2017, but for
Europe,
>>>>> meaning to identify and sample mesoscale convective systems,
using MODE
>>>>> mtd.
>>>>>
>>>>> I got MET successfully installed and I here only want to use mtd
on the
>>>>> simulations, no observations for the moment.
>>>>>
>>>>> So what I do is:
>>>>>
>>>>>> ../MET/met-9.0/bin/mtd -single
>>>>>> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
>>>>>> Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config
MODEConfig_SKM
>>>>> where Month_oooo__pr_....nc is my model output, to be found here
(1GB):
>>>>> https://owncloud.gwdg.de/index.php/s/aVGr0Gql5tKVUcR
>>>>> and the config file is attached and basically unchanged.
>>>>>
>>>>> The error message is:
>>>>>
>>>>>> DEBUG 2: mtd_read_data() -> processing file
>>>>>> "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
>>>>>> WARNING:
>>>>>> WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) ->
>>>>>> returns the first available time for "pr" variable).
>>>>>> WARNING:
>>>>>> ERROR :
>>>>>> ERROR : NcCfFile::getData(NcVar *, const LongArray &,
DataPlane &)
>>>>>> const -> needed 3 arguments for variable pr, got 2
>>>>>> ERROR :
>>>>> The remapping would be specified in this part of the config
file!?:
>>>>>
>>>>> regrid = {
>>>>> to_grid = NONE;
>>>>> method = NEAREST;
>>>>> width = 1;
>>>>> vld_thresh = 0.5;
>>>>> shape = SQUARE;
>>>>> }
>>>>>
>>>>> Should that move into fcst?
>>>>>
>>>>> Also attached you find info on the output i'm working on.
>>>>>
>>>>> I would be very thankful if you could help me.
>>>>>
>>>>> With best regards
>>>>> Sebastian
>>>>>
>>>>> --
>>>>>
*********************************************************************
>>>>> Sebastian K. Müller
>>>>> Earth System Physics Section
>>>>> The Abdus Salam International Centre for Theoretical Physics
>>>>> Strada Costiera 11
>>>>> 34100 Trieste, ITALY
>>>>> phone: + 39 040 2240 438
>>>>> email:smueller at ictp.it
>>>>>
*********************************************************************
>>>>>
>>>>>
>>>>>
>> --
>>
*********************************************************************
>> Sebastian K. Müller
>> Earth System Physics Section
>> The Abdus Salam International Centre for Theoretical Physics
>> Strada Costiera 11
>> 34100 Trieste, ITALY
>> phone: + 39 040 2240 438
>> email:smueller at ictp.it
>>
*********************************************************************
>>
>>
>>
--
*********************************************************************
Sebastian K. Müller
Earth System Physics Section
The Abdus Salam International Centre for Theoretical Physics
Strada Costiera 11
34100 Trieste, ITALY
phone: + 39 040 2240 438
email:smueller at ictp.it
*********************************************************************
------------------------------------------------
Subject: Help on MODE Time Domain Tool
From: John Halley Gotway
Time: Tue Apr 07 07:01:17 2020
Sebastian,
Great, I’m glad to hear that did the trick. In answer to your
questions,
yes, the conv_radius is ultimately set as an integer number of grid
cells.
And prior to met-9.0, the temporal smoothing was hard-coded as 3 time-
steps
(t-1, t, and t+1). It wasn’t mentioned in Prein because it wasn’t a
configurable option at that point. If you set beg=end=0 to skip the
temporal smoothing, be aware that objects from one time step to the
next
must have some overlap in order to be included in the same 3D space-
time
object.
Thanks,
John
On Mon, Apr 6, 2020 at 7:31 PM Sebastian K. Müller via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
>
> John,
>
> thank you very much! The python-interface you suggested works
perfectly
> and I get around splitting up the output into time steps.
>
> Just two more little bits:
> First, to be sure: conv_radius is eventually set as an integer of
grid
> cells!? I assume so.
> Second: the smoothing in time is done by setting again a "radius of
> timesteps". So beg=-1 and end=-1, smoothes over three timesteps,
that
> are [-1,0,1]. So if I set beg=0, and end=0, there is no smoothing in
> time!? From Prein et al., 2017, Clim. Dyn, I do not see whether they
> applied smoothing in time. Do you know if they did?
>
> Generally, my many thanks: this toolbox is very well documented,
easy to
> apply and the help of the team is great!
>
> Best regards
> Sebastian
>
>
> Am 01.04.20 um 01:31 schrieb John Halley Gotway via RT:
> > Sebastian,
> >
> > Yes, the smoothing radius is the "conv_radius" in the config file.
That
> > stands for convolution radius. It's the radius of the smoother
defined
> in
> > grid squares. If your data is already very smooth, you could set
it to 0
> > to disable the spatial smoothing. But your precip data probably
is NOT
> > very smooth. We intend the "grid_res" to be the nominal grid
spacing in
> > km. It's used a lot in the MODE config file so that you only need
to
> > change one setting to get reasonable defaults for a few other
settings.
> > Looks like it's only used once in MTD... to set the conv_radius.
You can
> > manually define the conv_radius however you'd like (e.g.
conv_radius =
> 5;).
> >
> > Also note that met-9.0 includes the ability to define the amount
of
> > smoothing in time. The default conv_time_window smooths data
across one
> > prior time step (beg = -1) and the next time step (end = 1).
Prior to
> > met-9.0, those settings were hard-coded, but they're configurable
now.
> >
> > In MTD, as in all the other MET tools, regridding is done right
after the
> > 2D data is read from the file. It operates totally independently
of the
> > smoothing that's done in MTD.
> >
> > As for reading multiple time steps from the same input file, there
is a
> > workaround we can do using python embedding. Instead of having
MET read
> > from the NetCDF input file directly, we run a python script to do
the
> > reading and then pass that gridded data and metadata to the MET
tools in
> > memory. I adapted one of the examples from this page
> >
> https://dtcenter.org/community-code/model-evaluation-tools-
met/sample-analysis-scripts
> > to
> > create the attached read_pr_timestep.py script.
> >
> > Run that script like this to make sure it works for you:
> > *python3 read_pr_timestep.py
Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
> 10*
> >
> > This should read the 11-th (0-based) timestep of the "pr"
variable. If
> > that works, then try running it with MET's plot_data_plane tool:
> > *plot_data_plane PYTHON_NUMPY pr_python.ps<http://pr_python.ps>
> > 'name="read_pr_timestep.py
Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
> 10";'*
> >
> > Next, I compared the resulting image with MET plotting that data
directly
> > to make sure it's oriented correctly.
> > *plot_data_plane Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
pr_direct.ps
> > <http://pr_direct.ps> 'name="pr"; level="(10,*,*)";'*
> >
> > I've attached the resulting png's and they match.
> >
> > So why go to all this trouble? Here's the punchline... you can
now run
> MTD
> > like this:
> >
> > *mtd -single 0 1 2 3 4 5 6 7 8 9 10 11 12 -config
MTDConfig_SKM_python
> > -outdir out*
> >
> > And inside the MTD config file, use this:
> > *fcst = {*
> > * file_type = PYTHON_NUMPY;*
> >
> > * field = { name = "read_pr_timestep.py
> > Month_oooo__pr_HadGEM_HIST_062002_pyco.nc MET_PYTHON_INPUT_ARG";*
> > * }*
> >
> > See the attached MTD config file.
> >
> > This runs MTD for the first 13 timesteps. For each input argument
in the
> > "-single" list, it executes the python script defined in the
config file,
> > substituting in the current value into the "MET_PYTHON_INPUT_ARG"
slot.
> >
> > This is a pretty nice usage of features we added in met-9.0!
> >
> > John
> >
> > On Tue, Mar 31, 2020 at 3:22 PM Sebastian K. Müller via RT <
> > met_help at ucar.edu> wrote:
> >
> >> <URL:https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
> >>
> >> Hello again,
> >>
> >> now, splitting up the file into individual time steps works and
is not
> >> too much of an effort. Still I would be interested in whether it
could
> >> be possible to do mtg with monthly files...
> >>
> >> mtg works now and I will be tuning the parameters now. Maybe just
a few
> >> questions:
> >>
> >> the "smoothing radius" of Prein et al. 2017 is "conv_radius" in
the
> >> config_file? And by default "60/grid_res", meaning that the
smoothing is
> >> done over a square of 20*20 grid cells, with "grid_res=3.". Is
that
> >> correct? Does this interfere in any with the regrid command?
> >>
> >> I think that's it for now.
> >>
> >> Thanks for your help,
> >> Cheers
> >> Sebastian
> >>
> >> Am 31.03.20 um 10:19 schrieb Sebastian K. Müller:
> >>> Hello Hailey,
> >>>
> >>> thank you. That already heldped a lot. Now I at least work with
the
> >>> right configure file. And it is not giving me errors, but not
writing
> >>> content to the output files either...
> >>>
> >>>> smueller at login02 E_OBS $ ../MET/met-9.0/bin/mtd -single
> >>>> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
> >>>> Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config MTDConfig_SKM
> >>>> -outdir /local_scratch/smueller/MTD_output/
> >>>> DEBUG 2: mtd_read_data() -> processing file
> >>>> "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
> >>>> DEBUG 2: mtd_read_data() -> processing file
> >>>> "Month_oooo__pr_HadGEM_HIST_072002_pyco.nc"
> >>>> DEBUG 2: regridding, if needed ...
> >>>> DEBUG 2: Splitting object field
> >>>> DEBUG 2: Done splitting
> >>>> DEBUG 2: Calculating 3D fcst single attributes
> >>>> DEBUG 2: Calculating 2D fcst attributes
> >>>> DEBUG 2: Creating 2D constant-time slice attributes file:
> >>>>
"/local_scratch/smueller/MTD_output//mtd_20010830_003000V_2d.txt"
> >>>> DEBUG 2: Creating 3D single simple attributes file:
> >>>>
> >>
>
"/local_scratch/smueller/MTD_output//mtd_20010830_003000V_3d_single_simple.txt"
> >>>> DEBUG 2: Creating NetCDF file:
> >>>>
"/local_scratch/smueller/MTD_output//mtd_20010830_003000V_obj.nc"
> >>> So, it would be a big effort to produce files of every output
time
> >>> step. We save the original output "daily", and I merge them to
months.
> >>> Is there a way around this?
> >>>
> >>> Thanks again,
> >>> Cheers
> >>> Sebastian
> >>>
> >>> Am 31.03.20 um 05:33 schrieb John Halley Gotway via RT:
> >>>> Hello Sebastian,
> >>>>
> >>>> I see you have questions about running the MODE time domain
tool. I
> was
> >>>> able to download your sample NetCDF file. Looking at the
history
> >>>> attribute
> >>>> in the NetCDF header, I see that you constructed this file by
running
> >>>> the
> >>>> "cdo mergetime" utility. So you've merged many output files
into a
> >>>> single
> >>>> one that contains many timesteps:
> >>>>
> >>>> :history = "Thu Dec 12 08:06:55 2019: cdo mergetime
> >>>>
> >>
>
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060100-
> >> 2002060200.nc
> >>
>
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060200-
> >> 2002060300.nc
> >>>> ...
> >>>>
> >>>> MTD is setup to process the raw, unmerged files. It expects to
read
> the
> >>>> same field of data from multiple files. So if you still have
the
> >>>> original
> >>>> files available, each containing a single time step, please try
> >>>> running it
> >>>> that way.
> >>>>
> >>>> Here is the usage statement for mtd:
> >>>>
> >>>> Usage: mtd
> >>>> -fcst file_1 ... file_n | file_list
> >>>> -obs file_1 ... file_n | file_list
> >>>> -single file_1 ... file_n | file_list
> >>>> -config config_file
> >>>> [-outdir path]
> >>>> [-log file]
> >>>> [-v level]
> >>>>
> >>>> Since you're running with a single data source, you should use
the
> >>>> "-single" command line option. You can either list all the
input
> >>>> files on
> >>>> the command line or put that list into an ascii file and put
that
> ascii
> >>>> file list on the command line.
> >>>>
> >>>> You're running MTD but you sent me a MODE config file. Make
sure
> you're
> >>>> using the default config file for MTD:
> >>>>
> >>>> fcst = {
> >>>> field = {
> >>>> name = "pr";
> >>>> level = "(0,*,*)";
> >>>> }
> >>>> ...
> >>>>
> >>>>
> >>>> Hope that helps.
> >>>>
> >>>> Thanks,
> >>>> John Halley Gotway
> >>>>
> >>>>
> >>>> On Mon, Mar 30, 2020 at 5:21 PM Sebastian K. Müller via RT <
> >>>> met_help at ucar.edu> wrote:
> >>>>
> >>>>> Mon Mar 30 17:20:51 2020: Request 94783 was acted upon.
> >>>>> Transaction: Ticket created bysmueller at ictp.it
> >>>>> Queue: met_help
> >>>>> Subject: Help on MODE Time Domain Tool
> >>>>> Owner: Nobody
> >>>>> Requestors:smueller at ictp.it
> >>>>> Status: new
> >>>>> Ticket<URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
> >>>>>
> >>>>>
> >>>>> Dear all,
> >>>>>
> >>>>> I am a PostDoc at ICTP in Trieste, working with convection-
permitting
> >>>>> climate simulations in the projects CORDEX-FPS and EUCP, and I
am
> >>>>> trying
> >>>>> to replicate the analysis done by Prein et al., 2017, but for
Europe,
> >>>>> meaning to identify and sample mesoscale convective systems,
using
> MODE
> >>>>> mtd.
> >>>>>
> >>>>> I got MET successfully installed and I here only want to use
mtd on
> the
> >>>>> simulations, no observations for the moment.
> >>>>>
> >>>>> So what I do is:
> >>>>>
> >>>>>> ../MET/met-9.0/bin/mtd -single
> >>>>>> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
> >>>>>> Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config
MODEConfig_SKM
> >>>>> where Month_oooo__pr_....nc is my model output, to be found
here
> (1GB):
> >>>>> https://owncloud.gwdg.de/index.php/s/aVGr0Gql5tKVUcR
> >>>>> and the config file is attached and basically unchanged.
> >>>>>
> >>>>> The error message is:
> >>>>>
> >>>>>> DEBUG 2: mtd_read_data() -> processing file
> >>>>>> "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
> >>>>>> WARNING:
> >>>>>> WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &)
->
> >>>>>> returns the first available time for "pr" variable).
> >>>>>> WARNING:
> >>>>>> ERROR :
> >>>>>> ERROR : NcCfFile::getData(NcVar *, const LongArray &,
DataPlane &)
> >>>>>> const -> needed 3 arguments for variable pr, got 2
> >>>>>> ERROR :
> >>>>> The remapping would be specified in this part of the config
file!?:
> >>>>>
> >>>>> regrid = {
> >>>>> to_grid = NONE;
> >>>>> method = NEAREST;
> >>>>> width = 1;
> >>>>> vld_thresh = 0.5;
> >>>>> shape = SQUARE;
> >>>>> }
> >>>>>
> >>>>> Should that move into fcst?
> >>>>>
> >>>>> Also attached you find info on the output i'm working on.
> >>>>>
> >>>>> I would be very thankful if you could help me.
> >>>>>
> >>>>> With best regards
> >>>>> Sebastian
> >>>>>
> >>>>> --
> >>>>>
*********************************************************************
> >>>>> Sebastian K. Müller
> >>>>> Earth System Physics Section
> >>>>> The Abdus Salam International Centre for Theoretical Physics
> >>>>> Strada Costiera 11
> >>>>> 34100 Trieste, ITALY
> >>>>> phone: + 39 040 2240 438
> >>>>> email:smueller at ictp.it
> >>>>>
*********************************************************************
> >>>>>
> >>>>>
> >>>>>
> >> --
> >>
*********************************************************************
> >> Sebastian K. Müller
> >> Earth System Physics Section
> >> The Abdus Salam International Centre for Theoretical Physics
> >> Strada Costiera 11
> >> 34100 Trieste, ITALY
> >> phone: + 39 040 2240 438
> >> email:smueller at ictp.it
> >>
*********************************************************************
> >>
> >>
> >>
> --
>
*********************************************************************
> Sebastian K. Müller
> Earth System Physics Section
> The Abdus Salam International Centre for Theoretical Physics
> Strada Costiera 11
> 34100 Trieste, ITALY
> phone: + 39 040 2240 438
> email:smueller at ictp.it
>
*********************************************************************
>
>
>
------------------------------------------------
Subject: Help on MODE Time Domain Tool
From: Sebastian K. Müller
Time: Wed Apr 29 12:10:54 2020
Dear John,
true, my issues have been solved, thanks again!
But now a new issue has come up and I seek advice once more. I apply
mtd
onto a model forecast and observations on Italy. I mask the forecast
such that it only exists where observations exist, that is exclusively
over land. Now I realize that close to the coasts, as well as at all
other boundaries, no clusters are identified. I expect that mtd works
only on the area "1 smoothing radius inwards from the boundary". The
two
plots illustrate the issue, one is the total precipitation of the
"raw_pr" and the other the total precipitation of all clusters. Our
missing value is as well -9999. So my questions are:
A0) If I would set observations to zero, for the area "1 smoothing
radius outside of the boundary", I would identify clusters allover the
original domain. The points along the boundary would get blurred out
into the sea. Likewise I could do with my model-forecast.
A1) Or I could set all points "1smooR outside" to nearest neighbor
along
the coast?
A2) Or I smoothen, just like mtd, towards outside of the domain?
I expect that you never really had that problem as boundaries were
less
important to your application!? And I rather do not want to modify
mtd,
but prepare my input such that I have a sensible solution.
Eventually I believe that A2) is the solution I should go for. Thanks
for your attention! Often just to phrase the problem let's you solve
it
yourself. Still I would be curious for your opinion! Otherwise, mtd
works perfectly and is just what I was looking for!
Best regards
Sebastian
Am 07.04.20 um 17:43 schrieb John Halley Gotway via RT:
> According to our records, your request has been resolved. If you
have any
> further questions or concerns, please respond to this message.
--
*********************************************************************
Sebastian K. Müller
Earth System Physics Section
The Abdus Salam International Centre for Theoretical Physics
Strada Costiera 11
34100 Trieste, ITALY
phone: + 39 040 2240 438
email: smueller at ictp.it
*********************************************************************
------------------------------------------------
Subject: Help on MODE Time Domain Tool
From: John Halley Gotway
Time: Wed Apr 29 17:19:31 2020
Sebastian,
Thanks for sending the images to illustrate the situation.
Unfortunately, MTD does not include logic to handle missing data
values
well. In fact, we have a very brief issue defined in GitHub about
this:
https://github.com/NCAR/MET/issues/1132
The developer of MTD has been reluctant to add logic for keeping track
of
missing data values since it'd slow down the 3D space/time convolution
processing step. As it stands in the latest versions of MET, a bad
data
value occurring anywhere within the convolution area results in a bad
data
value in the output. In other MET tools, like MODE and Grid-Stat,
this is
handled using a "vld_thresh" configuration option. The user sets it
between 0 and 1 to indicate the required ratio of data that must be
valid
in order to get a value in the output. So the current behavior is
equivalent to setting "vld_thresh = 1.0", requiring that 100% of the
grid
points contain valid data values.
MTD just doesn't support that as a configurable option yet, but if it
did,
I suspect that setting "vld_thresh = 0.1" or so might produce the
desired
result.
So what can you do with the current tools?
One option is to reset all those bad data values to some constant
value.
And you could do that by "censoring" the data. I'll use some sample
data
that's included in the MET tarball to illustrate:
# Plot 2-meter temperature to see gray areas of bad data values
*bin/plot_data_plane
data/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212
TMP_Z2.ps 'name="TMP"; level="Z2";' *
# Plot 2-meter temperature using censoring to replace bad data with a
constant value of 275
*bin/plot_data_plane
data/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212
TMP_Z2_no_bad_data.ps 'name="TMP"; level="Z2"; censor_thresh=[eq-
9999];
censor_val=[275];'*
I was thinking of some other ways how you might use the
regrid_data_plane,
gen_vx_mask, and/or pcp_combine tools to smooth the data out over the
ocean. But it's a whole of extra processing you'd need to do for each
time
step.
MET does also support python embedding. If you're comfortable with
python,
you could write a script to pre-process the data however you'd like
and
then hand that to MTD via memory.
Thanks,
John
On Wed, Apr 29, 2020 at 12:11 PM Sebastian K. Müller via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
>
> Dear John,
>
> true, my issues have been solved, thanks again!
>
> But now a new issue has come up and I seek advice once more. I apply
mtd
> onto a model forecast and observations on Italy. I mask the forecast
> such that it only exists where observations exist, that is
exclusively
> over land. Now I realize that close to the coasts, as well as at all
> other boundaries, no clusters are identified. I expect that mtd
works
> only on the area "1 smoothing radius inwards from the boundary". The
two
> plots illustrate the issue, one is the total precipitation of the
> "raw_pr" and the other the total precipitation of all clusters. Our
> missing value is as well -9999. So my questions are:
>
> A0) If I would set observations to zero, for the area "1 smoothing
> radius outside of the boundary", I would identify clusters allover
the
> original domain. The points along the boundary would get blurred out
> into the sea. Likewise I could do with my model-forecast.
> A1) Or I could set all points "1smooR outside" to nearest neighbor
along
> the coast?
> A2) Or I smoothen, just like mtd, towards outside of the domain?
>
> I expect that you never really had that problem as boundaries were
less
> important to your application!? And I rather do not want to modify
mtd,
> but prepare my input such that I have a sensible solution.
>
> Eventually I believe that A2) is the solution I should go for.
Thanks
> for your attention! Often just to phrase the problem let's you solve
it
> yourself. Still I would be curious for your opinion! Otherwise, mtd
> works perfectly and is just what I was looking for!
>
> Best regards
> Sebastian
>
>
> Am 07.04.20 um 17:43 schrieb John Halley Gotway via RT:
> > According to our records, your request has been resolved. If you
have any
> > further questions or concerns, please respond to this message.
>
> --
>
*********************************************************************
> Sebastian K. Müller
> Earth System Physics Section
> The Abdus Salam International Centre for Theoretical Physics
> Strada Costiera 11
> 34100 Trieste, ITALY
> phone: + 39 040 2240 438
> email: smueller at ictp.it
>
*********************************************************************
>
>
>
------------------------------------------------
Subject: Help on MODE Time Domain Tool
From: Sebastian K. Müller
Time: Mon May 04 17:35:41 2020
John,
thanks a lot again!
Of course I now work around the problem with setting the mask to
zeroes.
That is the best I can do, as everything else would get sketchy.
Then, for reducing the computational effort, I reduced the grid to
around Italy only. Still on the same grid, but extracting an area
through 'cdo selindexbox'. I now have to change the metadata
accordingly
in 'read_pr_timestep_....py'? It seems so, because all clusters I now
detect are located off in Spain, still at the shape of Italy.
How do I retrieve this metadata from the input lat-lon fields? In
particular 'scale_lat1/2', 'lat/lon_pin', 'x/y_pin', 'lon_orient'? Are
these specific to the projection? I see their values (not x/y_pin) in
the metadata of the original input.
So, it seems MTD relies on these values, as well as on the given lat-
lon
fields? That advice would be really precious, and I couldn't find the
solution online. Or maybe you give me a good reference from which to
learn about projections?
Attached you find the current situation.
Best regards
Sebastian
> ##
> ## create the metadata dictionary
> ##
>
> attrs = {
> 'valid': valid_time.strftime("%Y%m%d_%H%M%S"),
> 'init': valid_time.strftime("%Y%m%d_%H%M%S"),
> 'lead': '00',
> 'accum': accum_time,
>
> 'name': 'pr',
> 'long_name': var.long_name,
> 'level': 'Surface',
> 'units': var.units,
>
> 'grid': {
> 'name' : 'Lambert Conformal',
> 'type' : 'Lambert Conformal',
> 'hemisphere' : 'N',
> 'scale_lat_1' : float(41),
> 'scale_lat_2' : float(48),
> 'lat_pin' : float(45.441),
> 'lon_pin' : float(8.062),
> 'x_pin' : float(300.5),
> 'y_pin' : float(285.5),
> 'lon_orient' : float(8.062),
> 'd_km' : float(3),
> 'r_km' : float(6371.2),
> 'nx' : 317,
> 'ny' : 333
> }
> }
Am 30.04.20 um 01:19 schrieb John Halley Gotway via RT:
> Sebastian,
>
> Thanks for sending the images to illustrate the situation.
>
> Unfortunately, MTD does not include logic to handle missing data
values
> well. In fact, we have a very brief issue defined in GitHub about
this:
> https://github.com/NCAR/MET/issues/1132
>
> The developer of MTD has been reluctant to add logic for keeping
track of
> missing data values since it'd slow down the 3D space/time
convolution
> processing step. As it stands in the latest versions of MET, a bad
data
> value occurring anywhere within the convolution area results in a
bad data
> value in the output. In other MET tools, like MODE and Grid-Stat,
this is
> handled using a "vld_thresh" configuration option. The user sets it
> between 0 and 1 to indicate the required ratio of data that must be
valid
> in order to get a value in the output. So the current behavior is
> equivalent to setting "vld_thresh = 1.0", requiring that 100% of the
grid
> points contain valid data values.
>
> MTD just doesn't support that as a configurable option yet, but if
it did,
> I suspect that setting "vld_thresh = 0.1" or so might produce the
desired
> result.
>
> So what can you do with the current tools?
>
> One option is to reset all those bad data values to some constant
value.
> And you could do that by "censoring" the data. I'll use some sample
data
> that's included in the MET tarball to illustrate:
>
> # Plot 2-meter temperature to see gray areas of bad data values
> *bin/plot_data_plane
data/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212
> TMP_Z2.ps 'name="TMP"; level="Z2";' *
>
> # Plot 2-meter temperature using censoring to replace bad data with
a
> constant value of 275
> *bin/plot_data_plane
data/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212
> TMP_Z2_no_bad_data.ps 'name="TMP"; level="Z2"; censor_thresh=[eq-
9999];
> censor_val=[275];'*
>
> I was thinking of some other ways how you might use the
regrid_data_plane,
> gen_vx_mask, and/or pcp_combine tools to smooth the data out over
the
> ocean. But it's a whole of extra processing you'd need to do for
each time
> step.
>
> MET does also support python embedding. If you're comfortable with
python,
> you could write a script to pre-process the data however you'd like
and
> then hand that to MTD via memory.
>
> Thanks,
> John
>
>
> On Wed, Apr 29, 2020 at 12:11 PM Sebastian K. Müller via RT <
> met_help at ucar.edu> wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
>>
>> Dear John,
>>
>> true, my issues have been solved, thanks again!
>>
>> But now a new issue has come up and I seek advice once more. I
apply mtd
>> onto a model forecast and observations on Italy. I mask the
forecast
>> such that it only exists where observations exist, that is
exclusively
>> over land. Now I realize that close to the coasts, as well as at
all
>> other boundaries, no clusters are identified. I expect that mtd
works
>> only on the area "1 smoothing radius inwards from the boundary".
The two
>> plots illustrate the issue, one is the total precipitation of the
>> "raw_pr" and the other the total precipitation of all clusters. Our
>> missing value is as well -9999. So my questions are:
>>
>> A0) If I would set observations to zero, for the area "1 smoothing
>> radius outside of the boundary", I would identify clusters allover
the
>> original domain. The points along the boundary would get blurred
out
>> into the sea. Likewise I could do with my model-forecast.
>> A1) Or I could set all points "1smooR outside" to nearest neighbor
along
>> the coast?
>> A2) Or I smoothen, just like mtd, towards outside of the domain?
>>
>> I expect that you never really had that problem as boundaries were
less
>> important to your application!? And I rather do not want to modify
mtd,
>> but prepare my input such that I have a sensible solution.
>>
>> Eventually I believe that A2) is the solution I should go for.
Thanks
>> for your attention! Often just to phrase the problem let's you
solve it
>> yourself. Still I would be curious for your opinion! Otherwise, mtd
>> works perfectly and is just what I was looking for!
>>
>> Best regards
>> Sebastian
>>
>>
>> Am 07.04.20 um 17:43 schrieb John Halley Gotway via RT:
>>> According to our records, your request has been resolved. If you
have any
>>> further questions or concerns, please respond to this message.
>> --
>>
*********************************************************************
>> Sebastian K. Müller
>> Earth System Physics Section
>> The Abdus Salam International Centre for Theoretical Physics
>> Strada Costiera 11
>> 34100 Trieste, ITALY
>> phone: + 39 040 2240 438
>> email: smueller at ictp.it
>>
*********************************************************************
>>
>>
>>
--
*********************************************************************
Sebastian K. Müller
Earth System Physics Section
The Abdus Salam International Centre for Theoretical Physics
Strada Costiera 11
34100 Trieste, ITALY
phone: + 39 040 2240 438
email: smueller at ictp.it
*********************************************************************
------------------------------------------------
Subject: Help on MODE Time Domain Tool
From: John Halley Gotway
Time: Mon May 04 21:20:42 2020
Sebastian,
I’ll need to ask my co-worker, Randy, about these grid questions.
Yes, MET
does use those grid/projection parameters to instantiate the grid. It
does
not actually use any input Lat/Lon fields at all.
I see that you’re subsetting the lower-right corner of the grid. Can
you
tell me exactly the starting/ending grid points you’re including in
the
subset? And I’ll ask Randy if the grid can easily be subsetted. If
not,
we could always use the MET tools to do the regridding to a smaller
domain.
Thanks,
John
On Mon, May 4, 2020 at 5:35 PM Sebastian K. Müller via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
>
> John,
>
> thanks a lot again!
>
> Of course I now work around the problem with setting the mask to
zeroes.
> That is the best I can do, as everything else would get sketchy.
>
> Then, for reducing the computational effort, I reduced the grid to
> around Italy only. Still on the same grid, but extracting an area
> through 'cdo selindexbox'. I now have to change the metadata
accordingly
> in 'read_pr_timestep_....py'? It seems so, because all clusters I
now
> detect are located off in Spain, still at the shape of Italy.
>
> How do I retrieve this metadata from the input lat-lon fields? In
> particular 'scale_lat1/2', 'lat/lon_pin', 'x/y_pin', 'lon_orient'?
Are
> these specific to the projection? I see their values (not x/y_pin)
in
> the metadata of the original input.
> So, it seems MTD relies on these values, as well as on the given
lat-lon
> fields? That advice would be really precious, and I couldn't find
the
> solution online. Or maybe you give me a good reference from which to
> learn about projections?
>
> Attached you find the current situation.
>
> Best regards
> Sebastian
>
> > ##
> > ## create the metadata dictionary
> > ##
> >
> > attrs = {
> > 'valid': valid_time.strftime("%Y%m%d_%H%M%S"),
> > 'init': valid_time.strftime("%Y%m%d_%H%M%S"),
> > 'lead': '00',
> > 'accum': accum_time,
> >
> > 'name': 'pr',
> > 'long_name': var.long_name,
> > 'level': 'Surface',
> > 'units': var.units,
> >
> > 'grid': {
> > 'name' : 'Lambert Conformal',
> > 'type' : 'Lambert Conformal',
> > 'hemisphere' : 'N',
> > 'scale_lat_1' : float(41),
> > 'scale_lat_2' : float(48),
> > 'lat_pin' : float(45.441),
> > 'lon_pin' : float(8.062),
> > 'x_pin' : float(300.5),
> > 'y_pin' : float(285.5),
> > 'lon_orient' : float(8.062),
> > 'd_km' : float(3),
> > 'r_km' : float(6371.2),
> > 'nx' : 317,
> > 'ny' : 333
> > }
> > }
>
> Am 30.04.20 um 01:19 schrieb John Halley Gotway via RT:
> > Sebastian,
> >
> > Thanks for sending the images to illustrate the situation.
> >
> > Unfortunately, MTD does not include logic to handle missing data
values
> > well. In fact, we have a very brief issue defined in GitHub about
this:
> > https://github.com/NCAR/MET/issues/1132
> >
> > The developer of MTD has been reluctant to add logic for keeping
track of
> > missing data values since it'd slow down the 3D space/time
convolution
> > processing step. As it stands in the latest versions of MET, a
bad data
> > value occurring anywhere within the convolution area results in a
bad
> data
> > value in the output. In other MET tools, like MODE and Grid-Stat,
this
> is
> > handled using a "vld_thresh" configuration option. The user sets
it
> > between 0 and 1 to indicate the required ratio of data that must
be valid
> > in order to get a value in the output. So the current behavior is
> > equivalent to setting "vld_thresh = 1.0", requiring that 100% of
the grid
> > points contain valid data values.
> >
> > MTD just doesn't support that as a configurable option yet, but if
it
> did,
> > I suspect that setting "vld_thresh = 0.1" or so might produce the
desired
> > result.
> >
> > So what can you do with the current tools?
> >
> > One option is to reset all those bad data values to some constant
value.
> > And you could do that by "censoring" the data. I'll use some
sample data
> > that's included in the MET tarball to illustrate:
> >
> > # Plot 2-meter temperature to see gray areas of bad data values
> > *bin/plot_data_plane
> data/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212
> > TMP_Z2.ps 'name="TMP"; level="Z2";' *
> >
> > # Plot 2-meter temperature using censoring to replace bad data
with a
> > constant value of 275
> > *bin/plot_data_plane
> data/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212
> > TMP_Z2_no_bad_data.ps 'name="TMP"; level="Z2"; censor_thresh=[eq-
9999];
> > censor_val=[275];'*
> >
> > I was thinking of some other ways how you might use the
> regrid_data_plane,
> > gen_vx_mask, and/or pcp_combine tools to smooth the data out over
the
> > ocean. But it's a whole of extra processing you'd need to do for
each
> time
> > step.
> >
> > MET does also support python embedding. If you're comfortable
with
> python,
> > you could write a script to pre-process the data however you'd
like and
> > then hand that to MTD via memory.
> >
> > Thanks,
> > John
> >
> >
> > On Wed, Apr 29, 2020 at 12:11 PM Sebastian K. Müller via RT <
> > met_help at ucar.edu> wrote:
> >
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
> >>
> >> Dear John,
> >>
> >> true, my issues have been solved, thanks again!
> >>
> >> But now a new issue has come up and I seek advice once more. I
apply mtd
> >> onto a model forecast and observations on Italy. I mask the
forecast
> >> such that it only exists where observations exist, that is
exclusively
> >> over land. Now I realize that close to the coasts, as well as at
all
> >> other boundaries, no clusters are identified. I expect that mtd
works
> >> only on the area "1 smoothing radius inwards from the boundary".
The two
> >> plots illustrate the issue, one is the total precipitation of the
> >> "raw_pr" and the other the total precipitation of all clusters.
Our
> >> missing value is as well -9999. So my questions are:
> >>
> >> A0) If I would set observations to zero, for the area "1
smoothing
> >> radius outside of the boundary", I would identify clusters
allover the
> >> original domain. The points along the boundary would get blurred
out
> >> into the sea. Likewise I could do with my model-forecast.
> >> A1) Or I could set all points "1smooR outside" to nearest
neighbor along
> >> the coast?
> >> A2) Or I smoothen, just like mtd, towards outside of the domain?
> >>
> >> I expect that you never really had that problem as boundaries
were less
> >> important to your application!? And I rather do not want to
modify mtd,
> >> but prepare my input such that I have a sensible solution.
> >>
> >> Eventually I believe that A2) is the solution I should go for.
Thanks
> >> for your attention! Often just to phrase the problem let's you
solve it
> >> yourself. Still I would be curious for your opinion! Otherwise,
mtd
> >> works perfectly and is just what I was looking for!
> >>
> >> Best regards
> >> Sebastian
> >>
> >>
> >> Am 07.04.20 um 17:43 schrieb John Halley Gotway via RT:
> >>> According to our records, your request has been resolved. If you
have
> any
> >>> further questions or concerns, please respond to this message.
> >> --
> >>
*********************************************************************
> >> Sebastian K. Müller
> >> Earth System Physics Section
> >> The Abdus Salam International Centre for Theoretical Physics
> >> Strada Costiera 11
> >> 34100 Trieste, ITALY
> >> phone: + 39 040 2240 438
> >> email: smueller at ictp.it
> >>
*********************************************************************
> >>
> >>
> >>
> --
>
*********************************************************************
> Sebastian K. Müller
> Earth System Physics Section
> The Abdus Salam International Centre for Theoretical Physics
> Strada Costiera 11
> 34100 Trieste, ITALY
> phone: + 39 040 2240 438
> email: smueller at ictp.it
>
*********************************************************************
>
>
>
------------------------------------------------
Subject: Help on MODE Time Domain Tool
From: Sebastian K. Müller
Time: Tue May 05 03:37:51 2020
Right, the plot is cut out.
The original grid spans from [lon=-2.089°,lat=37.238°] at lower left
corner to [21,320°,52,471°] upper right corner. (602*572) From the
netCDF-metadata I find:
:latitude_of_projection_origin = 45.441 ;
:longitude_of_projection_origin = 8.062 ;
:standard_parallel = 41., 48. ;
From that I cut out Italy ranging from [6.512, 37.970] to
[21.320,52.471°] with (317*333). It would be great to be able to
determine the grid attributes from that field. Attached you find
npy-files of both lat-lon fields.
Thanks
Sebastian
Am 05.05.20 um 05:20 schrieb John Halley Gotway via RT:
> Sebastian,
>
> I’ll need to ask my co-worker, Randy, about these grid questions.
Yes, MET
> does use those grid/projection parameters to instantiate the grid.
It does
> not actually use any input Lat/Lon fields at all.
>
> I see that you’re subsetting the lower-right corner of the grid.
Can you
> tell me exactly the starting/ending grid points you’re including in
the
> subset? And I’ll ask Randy if the grid can easily be subsetted. If
not,
> we could always use the MET tools to do the regridding to a smaller
domain.
>
> Thanks,
> John
>
> On Mon, May 4, 2020 at 5:35 PM Sebastian K. Müller via RT
<met_help at ucar.edu>
> wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
>>
>> John,
>>
>> thanks a lot again!
>>
>> Of course I now work around the problem with setting the mask to
zeroes.
>> That is the best I can do, as everything else would get sketchy.
>>
>> Then, for reducing the computational effort, I reduced the grid to
>> around Italy only. Still on the same grid, but extracting an area
>> through 'cdo selindexbox'. I now have to change the metadata
accordingly
>> in 'read_pr_timestep_....py'? It seems so, because all clusters I
now
>> detect are located off in Spain, still at the shape of Italy.
>>
>> How do I retrieve this metadata from the input lat-lon fields? In
>> particular 'scale_lat1/2', 'lat/lon_pin', 'x/y_pin', 'lon_orient'?
Are
>> these specific to the projection? I see their values (not x/y_pin)
in
>> the metadata of the original input.
>> So, it seems MTD relies on these values, as well as on the given
lat-lon
>> fields? That advice would be really precious, and I couldn't find
the
>> solution online. Or maybe you give me a good reference from which
to
>> learn about projections?
>>
>> Attached you find the current situation.
>>
>> Best regards
>> Sebastian
>>
>>> ##
>>> ## create the metadata dictionary
>>> ##
>>>
>>> attrs = {
>>> 'valid': valid_time.strftime("%Y%m%d_%H%M%S"),
>>> 'init': valid_time.strftime("%Y%m%d_%H%M%S"),
>>> 'lead': '00',
>>> 'accum': accum_time,
>>>
>>> 'name': 'pr',
>>> 'long_name': var.long_name,
>>> 'level': 'Surface',
>>> 'units': var.units,
>>>
>>> 'grid': {
>>> 'name' : 'Lambert Conformal',
>>> 'type' : 'Lambert Conformal',
>>> 'hemisphere' : 'N',
>>> 'scale_lat_1' : float(41),
>>> 'scale_lat_2' : float(48),
>>> 'lat_pin' : float(45.441),
>>> 'lon_pin' : float(8.062),
>>> 'x_pin' : float(300.5),
>>> 'y_pin' : float(285.5),
>>> 'lon_orient' : float(8.062),
>>> 'd_km' : float(3),
>>> 'r_km' : float(6371.2),
>>> 'nx' : 317,
>>> 'ny' : 333
>>> }
>>> }
>> Am 30.04.20 um 01:19 schrieb John Halley Gotway via RT:
>>> Sebastian,
>>>
>>> Thanks for sending the images to illustrate the situation.
>>>
>>> Unfortunately, MTD does not include logic to handle missing data
values
>>> well. In fact, we have a very brief issue defined in GitHub about
this:
>>> https://github.com/NCAR/MET/issues/1132
>>>
>>> The developer of MTD has been reluctant to add logic for keeping
track of
>>> missing data values since it'd slow down the 3D space/time
convolution
>>> processing step. As it stands in the latest versions of MET, a
bad data
>>> value occurring anywhere within the convolution area results in a
bad
>> data
>>> value in the output. In other MET tools, like MODE and Grid-Stat,
this
>> is
>>> handled using a "vld_thresh" configuration option. The user sets
it
>>> between 0 and 1 to indicate the required ratio of data that must
be valid
>>> in order to get a value in the output. So the current behavior is
>>> equivalent to setting "vld_thresh = 1.0", requiring that 100% of
the grid
>>> points contain valid data values.
>>>
>>> MTD just doesn't support that as a configurable option yet, but if
it
>> did,
>>> I suspect that setting "vld_thresh = 0.1" or so might produce the
desired
>>> result.
>>>
>>> So what can you do with the current tools?
>>>
>>> One option is to reset all those bad data values to some constant
value.
>>> And you could do that by "censoring" the data. I'll use some
sample data
>>> that's included in the MET tarball to illustrate:
>>>
>>> # Plot 2-meter temperature to see gray areas of bad data values
>>> *bin/plot_data_plane
>> data/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212
>>> TMP_Z2.ps 'name="TMP"; level="Z2";' *
>>>
>>> # Plot 2-meter temperature using censoring to replace bad data
with a
>>> constant value of 275
>>> *bin/plot_data_plane
>> data/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212
>>> TMP_Z2_no_bad_data.ps 'name="TMP"; level="Z2"; censor_thresh=[eq-
9999];
>>> censor_val=[275];'*
>>>
>>> I was thinking of some other ways how you might use the
>> regrid_data_plane,
>>> gen_vx_mask, and/or pcp_combine tools to smooth the data out over
the
>>> ocean. But it's a whole of extra processing you'd need to do for
each
>> time
>>> step.
>>>
>>> MET does also support python embedding. If you're comfortable
with
>> python,
>>> you could write a script to pre-process the data however you'd
like and
>>> then hand that to MTD via memory.
>>>
>>> Thanks,
>>> John
>>>
>>>
>>> On Wed, Apr 29, 2020 at 12:11 PM Sebastian K. Müller via RT <
>>> met_help at ucar.edu> wrote:
>>>
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
>>>>
>>>> Dear John,
>>>>
>>>> true, my issues have been solved, thanks again!
>>>>
>>>> But now a new issue has come up and I seek advice once more. I
apply mtd
>>>> onto a model forecast and observations on Italy. I mask the
forecast
>>>> such that it only exists where observations exist, that is
exclusively
>>>> over land. Now I realize that close to the coasts, as well as at
all
>>>> other boundaries, no clusters are identified. I expect that mtd
works
>>>> only on the area "1 smoothing radius inwards from the boundary".
The two
>>>> plots illustrate the issue, one is the total precipitation of the
>>>> "raw_pr" and the other the total precipitation of all clusters.
Our
>>>> missing value is as well -9999. So my questions are:
>>>>
>>>> A0) If I would set observations to zero, for the area "1
smoothing
>>>> radius outside of the boundary", I would identify clusters
allover the
>>>> original domain. The points along the boundary would get blurred
out
>>>> into the sea. Likewise I could do with my model-forecast.
>>>> A1) Or I could set all points "1smooR outside" to nearest
neighbor along
>>>> the coast?
>>>> A2) Or I smoothen, just like mtd, towards outside of the domain?
>>>>
>>>> I expect that you never really had that problem as boundaries
were less
>>>> important to your application!? And I rather do not want to
modify mtd,
>>>> but prepare my input such that I have a sensible solution.
>>>>
>>>> Eventually I believe that A2) is the solution I should go for.
Thanks
>>>> for your attention! Often just to phrase the problem let's you
solve it
>>>> yourself. Still I would be curious for your opinion! Otherwise,
mtd
>>>> works perfectly and is just what I was looking for!
>>>>
>>>> Best regards
>>>> Sebastian
>>>>
>>>>
>>>> Am 07.04.20 um 17:43 schrieb John Halley Gotway via RT:
>>>>> According to our records, your request has been resolved. If you
have
>> any
>>>>> further questions or concerns, please respond to this message.
>>>> --
>>>>
*********************************************************************
>>>> Sebastian K. Müller
>>>> Earth System Physics Section
>>>> The Abdus Salam International Centre for Theoretical Physics
>>>> Strada Costiera 11
>>>> 34100 Trieste, ITALY
>>>> phone: + 39 040 2240 438
>>>> email: smueller at ictp.it
>>>>
*********************************************************************
>>>>
>>>>
>>>>
>> --
>>
*********************************************************************
>> Sebastian K. Müller
>> Earth System Physics Section
>> The Abdus Salam International Centre for Theoretical Physics
>> Strada Costiera 11
>> 34100 Trieste, ITALY
>> phone: + 39 040 2240 438
>> email: smueller at ictp.it
>>
*********************************************************************
>>
>>
>>
--
*********************************************************************
Sebastian K. Müller
Earth System Physics Section
The Abdus Salam International Centre for Theoretical Physics
Strada Costiera 11
34100 Trieste, ITALY
phone: + 39 040 2240 438
email: smueller at ictp.it
*********************************************************************
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #94783] Resolved: Help on MODE Time Domain Tool
From: Sebastian K. Müller
Time: Tue May 05 15:02:00 2020
Ciao John,
the problem is solved. I finally understood that x_pin and y_pin are
the
indices of lon_pin and lat_pin, respectively.
All is working.
Many thanks and best regards
Sebastian
Am 05.05.20 um 05:20 schrieb John Halley Gotway via RT:
> Sebastian,
>
> I’ll need to ask my co-worker, Randy, about these grid questions.
Yes, MET
> does use those grid/projection parameters to instantiate the grid.
It does
> not actually use any input Lat/Lon fields at all.
>
> I see that you’re subsetting the lower-right corner of the grid.
Can you
> tell me exactly the starting/ending grid points you’re including in
the
> subset? And I’ll ask Randy if the grid can easily be subsetted. If
not,
> we could always use the MET tools to do the regridding to a smaller
domain.
>
> Thanks,
> John
>
> On Mon, May 4, 2020 at 5:35 PM Sebastian K. Müller via RT
<met_help at ucar.edu>
> wrote:
>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
>>
>> John,
>>
>> thanks a lot again!
>>
>> Of course I now work around the problem with setting the mask to
zeroes.
>> That is the best I can do, as everything else would get sketchy.
>>
>> Then, for reducing the computational effort, I reduced the grid to
>> around Italy only. Still on the same grid, but extracting an area
>> through 'cdo selindexbox'. I now have to change the metadata
accordingly
>> in 'read_pr_timestep_....py'? It seems so, because all clusters I
now
>> detect are located off in Spain, still at the shape of Italy.
>>
>> How do I retrieve this metadata from the input lat-lon fields? In
>> particular 'scale_lat1/2', 'lat/lon_pin', 'x/y_pin', 'lon_orient'?
Are
>> these specific to the projection? I see their values (not x/y_pin)
in
>> the metadata of the original input.
>> So, it seems MTD relies on these values, as well as on the given
lat-lon
>> fields? That advice would be really precious, and I couldn't find
the
>> solution online. Or maybe you give me a good reference from which
to
>> learn about projections?
>>
>> Attached you find the current situation.
>>
>> Best regards
>> Sebastian
>>
>>> ##
>>> ## create the metadata dictionary
>>> ##
>>>
>>> attrs = {
>>> 'valid': valid_time.strftime("%Y%m%d_%H%M%S"),
>>> 'init': valid_time.strftime("%Y%m%d_%H%M%S"),
>>> 'lead': '00',
>>> 'accum': accum_time,
>>>
>>> 'name': 'pr',
>>> 'long_name': var.long_name,
>>> 'level': 'Surface',
>>> 'units': var.units,
>>>
>>> 'grid': {
>>> 'name' : 'Lambert Conformal',
>>> 'type' : 'Lambert Conformal',
>>> 'hemisphere' : 'N',
>>> 'scale_lat_1' : float(41),
>>> 'scale_lat_2' : float(48),
>>> 'lat_pin' : float(45.441),
>>> 'lon_pin' : float(8.062),
>>> 'x_pin' : float(300.5),
>>> 'y_pin' : float(285.5),
>>> 'lon_orient' : float(8.062),
>>> 'd_km' : float(3),
>>> 'r_km' : float(6371.2),
>>> 'nx' : 317,
>>> 'ny' : 333
>>> }
>>> }
>> Am 30.04.20 um 01:19 schrieb John Halley Gotway via RT:
>>> Sebastian,
>>>
>>> Thanks for sending the images to illustrate the situation.
>>>
>>> Unfortunately, MTD does not include logic to handle missing data
values
>>> well. In fact, we have a very brief issue defined in GitHub about
this:
>>> https://github.com/NCAR/MET/issues/1132
>>>
>>> The developer of MTD has been reluctant to add logic for keeping
track of
>>> missing data values since it'd slow down the 3D space/time
convolution
>>> processing step. As it stands in the latest versions of MET, a
bad data
>>> value occurring anywhere within the convolution area results in a
bad
>> data
>>> value in the output. In other MET tools, like MODE and Grid-Stat,
this
>> is
>>> handled using a "vld_thresh" configuration option. The user sets
it
>>> between 0 and 1 to indicate the required ratio of data that must
be valid
>>> in order to get a value in the output. So the current behavior is
>>> equivalent to setting "vld_thresh = 1.0", requiring that 100% of
the grid
>>> points contain valid data values.
>>>
>>> MTD just doesn't support that as a configurable option yet, but if
it
>> did,
>>> I suspect that setting "vld_thresh = 0.1" or so might produce the
desired
>>> result.
>>>
>>> So what can you do with the current tools?
>>>
>>> One option is to reset all those bad data values to some constant
value.
>>> And you could do that by "censoring" the data. I'll use some
sample data
>>> that's included in the MET tarball to illustrate:
>>>
>>> # Plot 2-meter temperature to see gray areas of bad data values
>>> *bin/plot_data_plane
>> data/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212
>>> TMP_Z2.ps 'name="TMP"; level="Z2";' *
>>>
>>> # Plot 2-meter temperature using censoring to replace bad data
with a
>>> constant value of 275
>>> *bin/plot_data_plane
>> data/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212
>>> TMP_Z2_no_bad_data.ps 'name="TMP"; level="Z2"; censor_thresh=[eq-
9999];
>>> censor_val=[275];'*
>>>
>>> I was thinking of some other ways how you might use the
>> regrid_data_plane,
>>> gen_vx_mask, and/or pcp_combine tools to smooth the data out over
the
>>> ocean. But it's a whole of extra processing you'd need to do for
each
>> time
>>> step.
>>>
>>> MET does also support python embedding. If you're comfortable
with
>> python,
>>> you could write a script to pre-process the data however you'd
like and
>>> then hand that to MTD via memory.
>>>
>>> Thanks,
>>> John
>>>
>>>
>>> On Wed, Apr 29, 2020 at 12:11 PM Sebastian K. Müller via RT <
>>> met_help at ucar.edu> wrote:
>>>
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
>>>>
>>>> Dear John,
>>>>
>>>> true, my issues have been solved, thanks again!
>>>>
>>>> But now a new issue has come up and I seek advice once more. I
apply mtd
>>>> onto a model forecast and observations on Italy. I mask the
forecast
>>>> such that it only exists where observations exist, that is
exclusively
>>>> over land. Now I realize that close to the coasts, as well as at
all
>>>> other boundaries, no clusters are identified. I expect that mtd
works
>>>> only on the area "1 smoothing radius inwards from the boundary".
The two
>>>> plots illustrate the issue, one is the total precipitation of the
>>>> "raw_pr" and the other the total precipitation of all clusters.
Our
>>>> missing value is as well -9999. So my questions are:
>>>>
>>>> A0) If I would set observations to zero, for the area "1
smoothing
>>>> radius outside of the boundary", I would identify clusters
allover the
>>>> original domain. The points along the boundary would get blurred
out
>>>> into the sea. Likewise I could do with my model-forecast.
>>>> A1) Or I could set all points "1smooR outside" to nearest
neighbor along
>>>> the coast?
>>>> A2) Or I smoothen, just like mtd, towards outside of the domain?
>>>>
>>>> I expect that you never really had that problem as boundaries
were less
>>>> important to your application!? And I rather do not want to
modify mtd,
>>>> but prepare my input such that I have a sensible solution.
>>>>
>>>> Eventually I believe that A2) is the solution I should go for.
Thanks
>>>> for your attention! Often just to phrase the problem let's you
solve it
>>>> yourself. Still I would be curious for your opinion! Otherwise,
mtd
>>>> works perfectly and is just what I was looking for!
>>>>
>>>> Best regards
>>>> Sebastian
>>>>
>>>>
>>>> Am 07.04.20 um 17:43 schrieb John Halley Gotway via RT:
>>>>> According to our records, your request has been resolved. If you
have
>> any
>>>>> further questions or concerns, please respond to this message.
>>>> --
>>>>
*********************************************************************
>>>> Sebastian K. Müller
>>>> Earth System Physics Section
>>>> The Abdus Salam International Centre for Theoretical Physics
>>>> Strada Costiera 11
>>>> 34100 Trieste, ITALY
>>>> phone: + 39 040 2240 438
>>>> email: smueller at ictp.it
>>>>
*********************************************************************
>>>>
>>>>
>>>>
>> --
>>
*********************************************************************
>> Sebastian K. Müller
>> Earth System Physics Section
>> The Abdus Salam International Centre for Theoretical Physics
>> Strada Costiera 11
>> 34100 Trieste, ITALY
>> phone: + 39 040 2240 438
>> email: smueller at ictp.it
>>
*********************************************************************
>>
>>
>>
--
*********************************************************************
Sebastian K. Müller
Earth System Physics Section
The Abdus Salam International Centre for Theoretical Physics
Strada Costiera 11
34100 Trieste, ITALY
phone: + 39 040 2240 438
email: smueller at ictp.it
*********************************************************************
------------------------------------------------
Subject: Help on MODE Time Domain Tool
From: John Halley Gotway
Time: Wed May 06 16:35:58 2020
Fantastic, thanks for confirming.
John
On Tue, May 5, 2020 at 3:02 PM Sebastian K. Müller via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
>
> Ciao John,
>
> the problem is solved. I finally understood that x_pin and y_pin are
the
> indices of lon_pin and lat_pin, respectively.
>
> All is working.
>
> Many thanks and best regards
> Sebastian
>
> Am 05.05.20 um 05:20 schrieb John Halley Gotway via RT:
> > Sebastian,
> >
> > I’ll need to ask my co-worker, Randy, about these grid questions.
Yes,
> MET
> > does use those grid/projection parameters to instantiate the grid.
It
> does
> > not actually use any input Lat/Lon fields at all.
> >
> > I see that you’re subsetting the lower-right corner of the grid.
Can you
> > tell me exactly the starting/ending grid points you’re including
in the
> > subset? And I’ll ask Randy if the grid can easily be subsetted.
If not,
> > we could always use the MET tools to do the regridding to a
smaller
> domain.
> >
> > Thanks,
> > John
> >
> > On Mon, May 4, 2020 at 5:35 PM Sebastian K. Müller via RT <
> met_help at ucar.edu>
> > wrote:
> >
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
> >>
> >> John,
> >>
> >> thanks a lot again!
> >>
> >> Of course I now work around the problem with setting the mask to
zeroes.
> >> That is the best I can do, as everything else would get sketchy.
> >>
> >> Then, for reducing the computational effort, I reduced the grid
to
> >> around Italy only. Still on the same grid, but extracting an area
> >> through 'cdo selindexbox'. I now have to change the metadata
accordingly
> >> in 'read_pr_timestep_....py'? It seems so, because all clusters I
now
> >> detect are located off in Spain, still at the shape of Italy.
> >>
> >> How do I retrieve this metadata from the input lat-lon fields? In
> >> particular 'scale_lat1/2', 'lat/lon_pin', 'x/y_pin',
'lon_orient'? Are
> >> these specific to the projection? I see their values (not
x/y_pin) in
> >> the metadata of the original input.
> >> So, it seems MTD relies on these values, as well as on the given
lat-lon
> >> fields? That advice would be really precious, and I couldn't find
the
> >> solution online. Or maybe you give me a good reference from which
to
> >> learn about projections?
> >>
> >> Attached you find the current situation.
> >>
> >> Best regards
> >> Sebastian
> >>
> >>> ##
> >>> ## create the metadata dictionary
> >>> ##
> >>>
> >>> attrs = {
> >>> 'valid': valid_time.strftime("%Y%m%d_%H%M%S"),
> >>> 'init': valid_time.strftime("%Y%m%d_%H%M%S"),
> >>> 'lead': '00',
> >>> 'accum': accum_time,
> >>>
> >>> 'name': 'pr',
> >>> 'long_name': var.long_name,
> >>> 'level': 'Surface',
> >>> 'units': var.units,
> >>>
> >>> 'grid': {
> >>> 'name' : 'Lambert Conformal',
> >>> 'type' : 'Lambert Conformal',
> >>> 'hemisphere' : 'N',
> >>> 'scale_lat_1' : float(41),
> >>> 'scale_lat_2' : float(48),
> >>> 'lat_pin' : float(45.441),
> >>> 'lon_pin' : float(8.062),
> >>> 'x_pin' : float(300.5),
> >>> 'y_pin' : float(285.5),
> >>> 'lon_orient' : float(8.062),
> >>> 'd_km' : float(3),
> >>> 'r_km' : float(6371.2),
> >>> 'nx' : 317,
> >>> 'ny' : 333
> >>> }
> >>> }
> >> Am 30.04.20 um 01:19 schrieb John Halley Gotway via RT:
> >>> Sebastian,
> >>>
> >>> Thanks for sending the images to illustrate the situation.
> >>>
> >>> Unfortunately, MTD does not include logic to handle missing data
values
> >>> well. In fact, we have a very brief issue defined in GitHub
about
> this:
> >>> https://github.com/NCAR/MET/issues/1132
> >>>
> >>> The developer of MTD has been reluctant to add logic for keeping
track
> of
> >>> missing data values since it'd slow down the 3D space/time
convolution
> >>> processing step. As it stands in the latest versions of MET, a
bad
> data
> >>> value occurring anywhere within the convolution area results in
a bad
> >> data
> >>> value in the output. In other MET tools, like MODE and Grid-
Stat, this
> >> is
> >>> handled using a "vld_thresh" configuration option. The user
sets it
> >>> between 0 and 1 to indicate the required ratio of data that must
be
> valid
> >>> in order to get a value in the output. So the current behavior
is
> >>> equivalent to setting "vld_thresh = 1.0", requiring that 100% of
the
> grid
> >>> points contain valid data values.
> >>>
> >>> MTD just doesn't support that as a configurable option yet, but
if it
> >> did,
> >>> I suspect that setting "vld_thresh = 0.1" or so might produce
the
> desired
> >>> result.
> >>>
> >>> So what can you do with the current tools?
> >>>
> >>> One option is to reset all those bad data values to some
constant
> value.
> >>> And you could do that by "censoring" the data. I'll use some
sample
> data
> >>> that's included in the MET tarball to illustrate:
> >>>
> >>> # Plot 2-meter temperature to see gray areas of bad data values
> >>> *bin/plot_data_plane
> >> data/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212
> >>> TMP_Z2.ps 'name="TMP"; level="Z2";' *
> >>>
> >>> # Plot 2-meter temperature using censoring to replace bad data
with a
> >>> constant value of 275
> >>> *bin/plot_data_plane
> >> data/sample_fcst/2005080700/wrfprs_ruc13_12.tm00_G212
> >>> TMP_Z2_no_bad_data.ps 'name="TMP"; level="Z2";
censor_thresh=[eq-9999];
> >>> censor_val=[275];'*
> >>>
> >>> I was thinking of some other ways how you might use the
> >> regrid_data_plane,
> >>> gen_vx_mask, and/or pcp_combine tools to smooth the data out
over the
> >>> ocean. But it's a whole of extra processing you'd need to do
for each
> >> time
> >>> step.
> >>>
> >>> MET does also support python embedding. If you're comfortable
with
> >> python,
> >>> you could write a script to pre-process the data however you'd
like and
> >>> then hand that to MTD via memory.
> >>>
> >>> Thanks,
> >>> John
> >>>
> >>>
> >>> On Wed, Apr 29, 2020 at 12:11 PM Sebastian K. Müller via RT <
> >>> met_help at ucar.edu> wrote:
> >>>
> >>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
> >>>>
> >>>> Dear John,
> >>>>
> >>>> true, my issues have been solved, thanks again!
> >>>>
> >>>> But now a new issue has come up and I seek advice once more. I
apply
> mtd
> >>>> onto a model forecast and observations on Italy. I mask the
forecast
> >>>> such that it only exists where observations exist, that is
exclusively
> >>>> over land. Now I realize that close to the coasts, as well as
at all
> >>>> other boundaries, no clusters are identified. I expect that mtd
works
> >>>> only on the area "1 smoothing radius inwards from the
boundary". The
> two
> >>>> plots illustrate the issue, one is the total precipitation of
the
> >>>> "raw_pr" and the other the total precipitation of all clusters.
Our
> >>>> missing value is as well -9999. So my questions are:
> >>>>
> >>>> A0) If I would set observations to zero, for the area "1
smoothing
> >>>> radius outside of the boundary", I would identify clusters
allover the
> >>>> original domain. The points along the boundary would get
blurred out
> >>>> into the sea. Likewise I could do with my model-forecast.
> >>>> A1) Or I could set all points "1smooR outside" to nearest
neighbor
> along
> >>>> the coast?
> >>>> A2) Or I smoothen, just like mtd, towards outside of the
domain?
> >>>>
> >>>> I expect that you never really had that problem as boundaries
were
> less
> >>>> important to your application!? And I rather do not want to
modify
> mtd,
> >>>> but prepare my input such that I have a sensible solution.
> >>>>
> >>>> Eventually I believe that A2) is the solution I should go for.
Thanks
> >>>> for your attention! Often just to phrase the problem let's you
solve
> it
> >>>> yourself. Still I would be curious for your opinion! Otherwise,
mtd
> >>>> works perfectly and is just what I was looking for!
> >>>>
> >>>> Best regards
> >>>> Sebastian
> >>>>
> >>>>
> >>>> Am 07.04.20 um 17:43 schrieb John Halley Gotway via RT:
> >>>>> According to our records, your request has been resolved. If
you have
> >> any
> >>>>> further questions or concerns, please respond to this message.
> >>>> --
> >>>>
*********************************************************************
> >>>> Sebastian K. Müller
> >>>> Earth System Physics Section
> >>>> The Abdus Salam International Centre for Theoretical Physics
> >>>> Strada Costiera 11
> >>>> 34100 Trieste, ITALY
> >>>> phone: + 39 040 2240 438
> >>>> email: smueller at ictp.it
> >>>>
*********************************************************************
> >>>>
> >>>>
> >>>>
> >> --
> >>
*********************************************************************
> >> Sebastian K. Müller
> >> Earth System Physics Section
> >> The Abdus Salam International Centre for Theoretical Physics
> >> Strada Costiera 11
> >> 34100 Trieste, ITALY
> >> phone: + 39 040 2240 438
> >> email: smueller at ictp.it
> >>
*********************************************************************
> >>
> >>
> >>
> --
>
*********************************************************************
> Sebastian K. Müller
> Earth System Physics Section
> The Abdus Salam International Centre for Theoretical Physics
> Strada Costiera 11
> 34100 Trieste, ITALY
> phone: + 39 040 2240 438
> email: smueller at ictp.it
>
*********************************************************************
>
>
>
------------------------------------------------
More information about the Met_help
mailing list