[Met_help] [rt.rap.ucar.edu #94783] History for Help on MODE Time Domain Tool
John Halley Gotway via RT
met_help at ucar.edu
Tue Apr 7 09:43:36 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Dear all,
I am a PostDoc at ICTP in Trieste, working with convection-permitting
climate simulations in the projects CORDEX-FPS and EUCP, and I am trying
to replicate the analysis done by Prein et al., 2017, but for Europe,
meaning to identify and sample mesoscale convective systems, using MODE mtd.
I got MET successfully installed and I here only want to use mtd on the
simulations, no observations for the moment.
So what I do is:
> ../MET/met-9.0/bin/mtd -single
> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
> Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config MODEConfig_SKM
where Month_oooo__pr_....nc is my model output, to be found here (1GB):
https://owncloud.gwdg.de/index.php/s/aVGr0Gql5tKVUcR
and the config file is attached and basically unchanged.
The error message is:
> DEBUG 2: mtd_read_data() -> processing file
> "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
> WARNING:
> WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) ->
> returns the first available time for "pr" variable).
> WARNING:
> ERROR :
> ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane &)
> const -> needed 3 arguments for variable pr, got 2
> ERROR :
The remapping would be specified in this part of the config file!?:
regrid = {
to_grid = NONE;
method = NEAREST;
width = 1;
vld_thresh = 0.5;
shape = SQUARE;
}
Should that move into fcst?
Also attached you find info on the output i'm working on.
I would be very thankful if you could help me.
With best regards
Sebastian
--
*********************************************************************
Sebastian K. Müller
Earth System Physics Section
The Abdus Salam International Centre for Theoretical Physics
Strada Costiera 11
34100 Trieste, ITALY
phone: + 39 040 2240 438
email: smueller at ictp.it
*********************************************************************
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Help on MODE Time Domain Tool
From: John Halley Gotway
Time: Mon Mar 30 21:33:50 2020
Hello Sebastian,
I see you have questions about running the MODE time domain tool. I
was
able to download your sample NetCDF file. Looking at the history
attribute
in the NetCDF header, I see that you constructed this file by running
the
"cdo mergetime" utility. So you've merged many output files into a
single
one that contains many timesteps:
:history = "Thu Dec 12 08:06:55 2019: cdo mergetime
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060100-2002060200.nc
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060200-2002060300.nc
...
MTD is setup to process the raw, unmerged files. It expects to read
the
same field of data from multiple files. So if you still have the
original
files available, each containing a single time step, please try
running it
that way.
Here is the usage statement for mtd:
Usage: mtd
-fcst file_1 ... file_n | file_list
-obs file_1 ... file_n | file_list
-single file_1 ... file_n | file_list
-config config_file
[-outdir path]
[-log file]
[-v level]
Since you're running with a single data source, you should use the
"-single" command line option. You can either list all the input
files on
the command line or put that list into an ascii file and put that
ascii
file list on the command line.
You're running MTD but you sent me a MODE config file. Make sure
you're
using the default config file for MTD:
fcst = {
field = {
name = "pr";
level = "(0,*,*)";
}
...
Hope that helps.
Thanks,
John Halley Gotway
On Mon, Mar 30, 2020 at 5:21 PM Sebastian K. Müller via RT <
met_help at ucar.edu> wrote:
>
> Mon Mar 30 17:20:51 2020: Request 94783 was acted upon.
> Transaction: Ticket created by smueller at ictp.it
> Queue: met_help
> Subject: Help on MODE Time Domain Tool
> Owner: Nobody
> Requestors: smueller at ictp.it
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
>
>
> Dear all,
>
> I am a PostDoc at ICTP in Trieste, working with convection-
permitting
> climate simulations in the projects CORDEX-FPS and EUCP, and I am
trying
> to replicate the analysis done by Prein et al., 2017, but for
Europe,
> meaning to identify and sample mesoscale convective systems, using
MODE
> mtd.
>
> I got MET successfully installed and I here only want to use mtd on
the
> simulations, no observations for the moment.
>
> So what I do is:
>
> > ../MET/met-9.0/bin/mtd -single
> > Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
> > Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config MODEConfig_SKM
> where Month_oooo__pr_....nc is my model output, to be found here
(1GB):
> https://owncloud.gwdg.de/index.php/s/aVGr0Gql5tKVUcR
> and the config file is attached and basically unchanged.
>
> The error message is:
>
> > DEBUG 2: mtd_read_data() -> processing file
> > "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
> > WARNING:
> > WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) ->
> > returns the first available time for "pr" variable).
> > WARNING:
> > ERROR :
> > ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane
&)
> > const -> needed 3 arguments for variable pr, got 2
> > ERROR :
>
> The remapping would be specified in this part of the config file!?:
>
> regrid = {
> to_grid = NONE;
> method = NEAREST;
> width = 1;
> vld_thresh = 0.5;
> shape = SQUARE;
> }
>
> Should that move into fcst?
>
> Also attached you find info on the output i'm working on.
>
> I would be very thankful if you could help me.
>
> With best regards
> Sebastian
>
> --
>
*********************************************************************
> Sebastian K. Müller
> Earth System Physics Section
> The Abdus Salam International Centre for Theoretical Physics
> Strada Costiera 11
> 34100 Trieste, ITALY
> phone: + 39 040 2240 438
> email: smueller at ictp.it
>
*********************************************************************
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #94783] Help on MODE Time Domain Tool
From: Sebastian K. Müller
Time: Tue Mar 31 02:19:11 2020
Hello Hailey,
thank you. That already heldped a lot. Now I at least work with the
right configure file. And it is not giving me errors, but not writing
content to the output files either...
> smueller at login02 E_OBS $ ../MET/met-9.0/bin/mtd -single
> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
> Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config MTDConfig_SKM
> -outdir /local_scratch/smueller/MTD_output/
> DEBUG 2: mtd_read_data() -> processing file
> "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
> DEBUG 2: mtd_read_data() -> processing file
> "Month_oooo__pr_HadGEM_HIST_072002_pyco.nc"
> DEBUG 2: regridding, if needed ...
> DEBUG 2: Splitting object field
> DEBUG 2: Done splitting
> DEBUG 2: Calculating 3D fcst single attributes
> DEBUG 2: Calculating 2D fcst attributes
> DEBUG 2: Creating 2D constant-time slice attributes file:
> "/local_scratch/smueller/MTD_output//mtd_20010830_003000V_2d.txt"
> DEBUG 2: Creating 3D single simple attributes file:
>
"/local_scratch/smueller/MTD_output//mtd_20010830_003000V_3d_single_simple.txt"
> DEBUG 2: Creating NetCDF file:
> "/local_scratch/smueller/MTD_output//mtd_20010830_003000V_obj.nc"
So, it would be a big effort to produce files of every output time
step.
We save the original output "daily", and I merge them to months. Is
there a way around this?
Thanks again,
Cheers
Sebastian
Am 31.03.20 um 05:33 schrieb John Halley Gotway via RT:
> Hello Sebastian,
>
> I see you have questions about running the MODE time domain tool. I
was
> able to download your sample NetCDF file. Looking at the history
attribute
> in the NetCDF header, I see that you constructed this file by
running the
> "cdo mergetime" utility. So you've merged many output files into a
single
> one that contains many timesteps:
>
> :history = "Thu Dec 12 08:06:55 2019: cdo mergetime
>
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060100-2002060200.nc
>
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060200-2002060300.nc
> ...
>
> MTD is setup to process the raw, unmerged files. It expects to read
the
> same field of data from multiple files. So if you still have the
original
> files available, each containing a single time step, please try
running it
> that way.
>
> Here is the usage statement for mtd:
>
> Usage: mtd
> -fcst file_1 ... file_n | file_list
> -obs file_1 ... file_n | file_list
> -single file_1 ... file_n | file_list
> -config config_file
> [-outdir path]
> [-log file]
> [-v level]
>
> Since you're running with a single data source, you should use the
> "-single" command line option. You can either list all the input
files on
> the command line or put that list into an ascii file and put that
ascii
> file list on the command line.
>
> You're running MTD but you sent me a MODE config file. Make sure
you're
> using the default config file for MTD:
>
> fcst = {
> field = {
> name = "pr";
> level = "(0,*,*)";
> }
> ...
>
>
> Hope that helps.
>
> Thanks,
> John Halley Gotway
>
>
> On Mon, Mar 30, 2020 at 5:21 PM Sebastian K. Müller via RT <
> met_help at ucar.edu> wrote:
>
>> Mon Mar 30 17:20:51 2020: Request 94783 was acted upon.
>> Transaction: Ticket created by smueller at ictp.it
>> Queue: met_help
>> Subject: Help on MODE Time Domain Tool
>> Owner: Nobody
>> Requestors: smueller at ictp.it
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
>>
>>
>> Dear all,
>>
>> I am a PostDoc at ICTP in Trieste, working with convection-
permitting
>> climate simulations in the projects CORDEX-FPS and EUCP, and I am
trying
>> to replicate the analysis done by Prein et al., 2017, but for
Europe,
>> meaning to identify and sample mesoscale convective systems, using
MODE
>> mtd.
>>
>> I got MET successfully installed and I here only want to use mtd on
the
>> simulations, no observations for the moment.
>>
>> So what I do is:
>>
>>> ../MET/met-9.0/bin/mtd -single
>>> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
>>> Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config MODEConfig_SKM
>> where Month_oooo__pr_....nc is my model output, to be found here
(1GB):
>> https://owncloud.gwdg.de/index.php/s/aVGr0Gql5tKVUcR
>> and the config file is attached and basically unchanged.
>>
>> The error message is:
>>
>>> DEBUG 2: mtd_read_data() -> processing file
>>> "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
>>> WARNING:
>>> WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) ->
>>> returns the first available time for "pr" variable).
>>> WARNING:
>>> ERROR :
>>> ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane
&)
>>> const -> needed 3 arguments for variable pr, got 2
>>> ERROR :
>> The remapping would be specified in this part of the config file!?:
>>
>> regrid = {
>> to_grid = NONE;
>> method = NEAREST;
>> width = 1;
>> vld_thresh = 0.5;
>> shape = SQUARE;
>> }
>>
>> Should that move into fcst?
>>
>> Also attached you find info on the output i'm working on.
>>
>> I would be very thankful if you could help me.
>>
>> With best regards
>> Sebastian
>>
>> --
>>
*********************************************************************
>> Sebastian K. Müller
>> Earth System Physics Section
>> The Abdus Salam International Centre for Theoretical Physics
>> Strada Costiera 11
>> 34100 Trieste, ITALY
>> phone: + 39 040 2240 438
>> email: smueller at ictp.it
>>
*********************************************************************
>>
>>
>>
--
*********************************************************************
Sebastian K. Müller
Earth System Physics Section
The Abdus Salam International Centre for Theoretical Physics
Strada Costiera 11
34100 Trieste, ITALY
phone: + 39 040 2240 438
email: smueller at ictp.it
*********************************************************************
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #94783] Help on MODE Time Domain Tool
From: Sebastian K. Müller
Time: Tue Mar 31 15:22:33 2020
Hello again,
now, splitting up the file into individual time steps works and is not
too much of an effort. Still I would be interested in whether it could
be possible to do mtg with monthly files...
mtg works now and I will be tuning the parameters now. Maybe just a
few
questions:
the "smoothing radius" of Prein et al. 2017 is "conv_radius" in the
config_file? And by default "60/grid_res", meaning that the smoothing
is
done over a square of 20*20 grid cells, with "grid_res=3.". Is that
correct? Does this interfere in any with the regrid command?
I think that's it for now.
Thanks for your help,
Cheers
Sebastian
Am 31.03.20 um 10:19 schrieb Sebastian K. Müller:
> Hello Hailey,
>
> thank you. That already heldped a lot. Now I at least work with the
> right configure file. And it is not giving me errors, but not
writing
> content to the output files either...
>
>> smueller at login02 E_OBS $ ../MET/met-9.0/bin/mtd -single
>> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
>> Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config MTDConfig_SKM
>> -outdir /local_scratch/smueller/MTD_output/
>> DEBUG 2: mtd_read_data() -> processing file
>> "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
>> DEBUG 2: mtd_read_data() -> processing file
>> "Month_oooo__pr_HadGEM_HIST_072002_pyco.nc"
>> DEBUG 2: regridding, if needed ...
>> DEBUG 2: Splitting object field
>> DEBUG 2: Done splitting
>> DEBUG 2: Calculating 3D fcst single attributes
>> DEBUG 2: Calculating 2D fcst attributes
>> DEBUG 2: Creating 2D constant-time slice attributes file:
>> "/local_scratch/smueller/MTD_output//mtd_20010830_003000V_2d.txt"
>> DEBUG 2: Creating 3D single simple attributes file:
>>
"/local_scratch/smueller/MTD_output//mtd_20010830_003000V_3d_single_simple.txt"
>> DEBUG 2: Creating NetCDF file:
>> "/local_scratch/smueller/MTD_output//mtd_20010830_003000V_obj.nc"
> So, it would be a big effort to produce files of every output time
> step. We save the original output "daily", and I merge them to
months.
> Is there a way around this?
>
> Thanks again,
> Cheers
> Sebastian
>
> Am 31.03.20 um 05:33 schrieb John Halley Gotway via RT:
>> Hello Sebastian,
>>
>> I see you have questions about running the MODE time domain tool.
I was
>> able to download your sample NetCDF file. Looking at the history
>> attribute
>> in the NetCDF header, I see that you constructed this file by
running
>> the
>> "cdo mergetime" utility. So you've merged many output files into a
>> single
>> one that contains many timesteps:
>>
>> :history = "Thu Dec 12 08:06:55 2019: cdo mergetime
>>
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060100-2002060200.nc
>>
>>
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060200-2002060300.nc
>>
>> ...
>>
>> MTD is setup to process the raw, unmerged files. It expects to
read the
>> same field of data from multiple files. So if you still have the
>> original
>> files available, each containing a single time step, please try
>> running it
>> that way.
>>
>> Here is the usage statement for mtd:
>>
>> Usage: mtd
>> -fcst file_1 ... file_n | file_list
>> -obs file_1 ... file_n | file_list
>> -single file_1 ... file_n | file_list
>> -config config_file
>> [-outdir path]
>> [-log file]
>> [-v level]
>>
>> Since you're running with a single data source, you should use the
>> "-single" command line option. You can either list all the input
>> files on
>> the command line or put that list into an ascii file and put that
ascii
>> file list on the command line.
>>
>> You're running MTD but you sent me a MODE config file. Make sure
you're
>> using the default config file for MTD:
>>
>> fcst = {
>> field = {
>> name = "pr";
>> level = "(0,*,*)";
>> }
>> ...
>>
>>
>> Hope that helps.
>>
>> Thanks,
>> John Halley Gotway
>>
>>
>> On Mon, Mar 30, 2020 at 5:21 PM Sebastian K. Müller via RT <
>> met_help at ucar.edu> wrote:
>>
>>> Mon Mar 30 17:20:51 2020: Request 94783 was acted upon.
>>> Transaction: Ticket created by smueller at ictp.it
>>> Queue: met_help
>>> Subject: Help on MODE Time Domain Tool
>>> Owner: Nobody
>>> Requestors: smueller at ictp.it
>>> Status: new
>>> Ticket <URL:
>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
>>>
>>>
>>> Dear all,
>>>
>>> I am a PostDoc at ICTP in Trieste, working with convection-
permitting
>>> climate simulations in the projects CORDEX-FPS and EUCP, and I am
>>> trying
>>> to replicate the analysis done by Prein et al., 2017, but for
Europe,
>>> meaning to identify and sample mesoscale convective systems, using
MODE
>>> mtd.
>>>
>>> I got MET successfully installed and I here only want to use mtd
on the
>>> simulations, no observations for the moment.
>>>
>>> So what I do is:
>>>
>>>> ../MET/met-9.0/bin/mtd -single
>>>> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
>>>> Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config MODEConfig_SKM
>>> where Month_oooo__pr_....nc is my model output, to be found here
(1GB):
>>> https://owncloud.gwdg.de/index.php/s/aVGr0Gql5tKVUcR
>>> and the config file is attached and basically unchanged.
>>>
>>> The error message is:
>>>
>>>> DEBUG 2: mtd_read_data() -> processing file
>>>> "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
>>>> WARNING:
>>>> WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) ->
>>>> returns the first available time for "pr" variable).
>>>> WARNING:
>>>> ERROR :
>>>> ERROR : NcCfFile::getData(NcVar *, const LongArray &, DataPlane
&)
>>>> const -> needed 3 arguments for variable pr, got 2
>>>> ERROR :
>>> The remapping would be specified in this part of the config
file!?:
>>>
>>> regrid = {
>>> to_grid = NONE;
>>> method = NEAREST;
>>> width = 1;
>>> vld_thresh = 0.5;
>>> shape = SQUARE;
>>> }
>>>
>>> Should that move into fcst?
>>>
>>> Also attached you find info on the output i'm working on.
>>>
>>> I would be very thankful if you could help me.
>>>
>>> With best regards
>>> Sebastian
>>>
>>> --
>>>
*********************************************************************
>>> Sebastian K. Müller
>>> Earth System Physics Section
>>> The Abdus Salam International Centre for Theoretical Physics
>>> Strada Costiera 11
>>> 34100 Trieste, ITALY
>>> phone: + 39 040 2240 438
>>> email: smueller at ictp.it
>>>
*********************************************************************
>>>
>>>
>>>
--
*********************************************************************
Sebastian K. Müller
Earth System Physics Section
The Abdus Salam International Centre for Theoretical Physics
Strada Costiera 11
34100 Trieste, ITALY
phone: + 39 040 2240 438
email: smueller at ictp.it
*********************************************************************
------------------------------------------------
Subject: Help on MODE Time Domain Tool
From: John Halley Gotway
Time: Tue Mar 31 17:31:41 2020
Sebastian,
Yes, the smoothing radius is the "conv_radius" in the config file.
That
stands for convolution radius. It's the radius of the smoother
defined in
grid squares. If your data is already very smooth, you could set it
to 0
to disable the spatial smoothing. But your precip data probably is
NOT
very smooth. We intend the "grid_res" to be the nominal grid spacing
in
km. It's used a lot in the MODE config file so that you only need to
change one setting to get reasonable defaults for a few other
settings.
Looks like it's only used once in MTD... to set the conv_radius. You
can
manually define the conv_radius however you'd like (e.g. conv_radius =
5;).
Also note that met-9.0 includes the ability to define the amount of
smoothing in time. The default conv_time_window smooths data across
one
prior time step (beg = -1) and the next time step (end = 1). Prior to
met-9.0, those settings were hard-coded, but they're configurable now.
In MTD, as in all the other MET tools, regridding is done right after
the
2D data is read from the file. It operates totally independently of
the
smoothing that's done in MTD.
As for reading multiple time steps from the same input file, there is
a
workaround we can do using python embedding. Instead of having MET
read
from the NetCDF input file directly, we run a python script to do the
reading and then pass that gridded data and metadata to the MET tools
in
memory. I adapted one of the examples from this page
https://dtcenter.org/community-code/model-evaluation-tools-met/sample-
analysis-scripts
to
create the attached read_pr_timestep.py script.
Run that script like this to make sure it works for you:
*python3 read_pr_timestep.py Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
10*
This should read the 11-th (0-based) timestep of the "pr" variable.
If
that works, then try running it with MET's plot_data_plane tool:
*plot_data_plane PYTHON_NUMPY pr_python.ps <http://pr_python.ps>
'name="read_pr_timestep.py Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
10";'*
Next, I compared the resulting image with MET plotting that data
directly
to make sure it's oriented correctly.
*plot_data_plane Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
pr_direct.ps
<http://pr_direct.ps> 'name="pr"; level="(10,*,*)";'*
I've attached the resulting png's and they match.
So why go to all this trouble? Here's the punchline... you can now
run MTD
like this:
*mtd -single 0 1 2 3 4 5 6 7 8 9 10 11 12 -config MTDConfig_SKM_python
-outdir out*
And inside the MTD config file, use this:
*fcst = {*
* file_type = PYTHON_NUMPY;*
* field = { name = "read_pr_timestep.py
Month_oooo__pr_HadGEM_HIST_062002_pyco.nc MET_PYTHON_INPUT_ARG";*
* }*
See the attached MTD config file.
This runs MTD for the first 13 timesteps. For each input argument in
the
"-single" list, it executes the python script defined in the config
file,
substituting in the current value into the "MET_PYTHON_INPUT_ARG"
slot.
This is a pretty nice usage of features we added in met-9.0!
John
On Tue, Mar 31, 2020 at 3:22 PM Sebastian K. Müller via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
>
> Hello again,
>
> now, splitting up the file into individual time steps works and is
not
> too much of an effort. Still I would be interested in whether it
could
> be possible to do mtg with monthly files...
>
> mtg works now and I will be tuning the parameters now. Maybe just a
few
> questions:
>
> the "smoothing radius" of Prein et al. 2017 is "conv_radius" in the
> config_file? And by default "60/grid_res", meaning that the
smoothing is
> done over a square of 20*20 grid cells, with "grid_res=3.". Is that
> correct? Does this interfere in any with the regrid command?
>
> I think that's it for now.
>
> Thanks for your help,
> Cheers
> Sebastian
>
> Am 31.03.20 um 10:19 schrieb Sebastian K. Müller:
> > Hello Hailey,
> >
> > thank you. That already heldped a lot. Now I at least work with
the
> > right configure file. And it is not giving me errors, but not
writing
> > content to the output files either...
> >
> >> smueller at login02 E_OBS $ ../MET/met-9.0/bin/mtd -single
> >> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
> >> Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config MTDConfig_SKM
> >> -outdir /local_scratch/smueller/MTD_output/
> >> DEBUG 2: mtd_read_data() -> processing file
> >> "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
> >> DEBUG 2: mtd_read_data() -> processing file
> >> "Month_oooo__pr_HadGEM_HIST_072002_pyco.nc"
> >> DEBUG 2: regridding, if needed ...
> >> DEBUG 2: Splitting object field
> >> DEBUG 2: Done splitting
> >> DEBUG 2: Calculating 3D fcst single attributes
> >> DEBUG 2: Calculating 2D fcst attributes
> >> DEBUG 2: Creating 2D constant-time slice attributes file:
> >> "/local_scratch/smueller/MTD_output//mtd_20010830_003000V_2d.txt"
> >> DEBUG 2: Creating 3D single simple attributes file:
> >>
>
"/local_scratch/smueller/MTD_output//mtd_20010830_003000V_3d_single_simple.txt"
> >> DEBUG 2: Creating NetCDF file:
> >> "/local_scratch/smueller/MTD_output//mtd_20010830_003000V_obj.nc"
> > So, it would be a big effort to produce files of every output time
> > step. We save the original output "daily", and I merge them to
months.
> > Is there a way around this?
> >
> > Thanks again,
> > Cheers
> > Sebastian
> >
> > Am 31.03.20 um 05:33 schrieb John Halley Gotway via RT:
> >> Hello Sebastian,
> >>
> >> I see you have questions about running the MODE time domain tool.
I was
> >> able to download your sample NetCDF file. Looking at the history
> >> attribute
> >> in the NetCDF header, I see that you constructed this file by
running
> >> the
> >> "cdo mergetime" utility. So you've merged many output files into
a
> >> single
> >> one that contains many timesteps:
> >>
> >> :history = "Thu Dec 12 08:06:55 2019: cdo mergetime
> >>
>
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060100-
> 2002060200.nc
> >>
> >>
>
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060200-
> 2002060300.nc
> >>
> >> ...
> >>
> >> MTD is setup to process the raw, unmerged files. It expects to
read the
> >> same field of data from multiple files. So if you still have the
> >> original
> >> files available, each containing a single time step, please try
> >> running it
> >> that way.
> >>
> >> Here is the usage statement for mtd:
> >>
> >> Usage: mtd
> >> -fcst file_1 ... file_n | file_list
> >> -obs file_1 ... file_n | file_list
> >> -single file_1 ... file_n | file_list
> >> -config config_file
> >> [-outdir path]
> >> [-log file]
> >> [-v level]
> >>
> >> Since you're running with a single data source, you should use
the
> >> "-single" command line option. You can either list all the input
> >> files on
> >> the command line or put that list into an ascii file and put that
ascii
> >> file list on the command line.
> >>
> >> You're running MTD but you sent me a MODE config file. Make sure
you're
> >> using the default config file for MTD:
> >>
> >> fcst = {
> >> field = {
> >> name = "pr";
> >> level = "(0,*,*)";
> >> }
> >> ...
> >>
> >>
> >> Hope that helps.
> >>
> >> Thanks,
> >> John Halley Gotway
> >>
> >>
> >> On Mon, Mar 30, 2020 at 5:21 PM Sebastian K. Müller via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >>> Mon Mar 30 17:20:51 2020: Request 94783 was acted upon.
> >>> Transaction: Ticket created by smueller at ictp.it
> >>> Queue: met_help
> >>> Subject: Help on MODE Time Domain Tool
> >>> Owner: Nobody
> >>> Requestors: smueller at ictp.it
> >>> Status: new
> >>> Ticket <URL:
> >>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
> >>>
> >>>
> >>> Dear all,
> >>>
> >>> I am a PostDoc at ICTP in Trieste, working with convection-
permitting
> >>> climate simulations in the projects CORDEX-FPS and EUCP, and I
am
> >>> trying
> >>> to replicate the analysis done by Prein et al., 2017, but for
Europe,
> >>> meaning to identify and sample mesoscale convective systems,
using MODE
> >>> mtd.
> >>>
> >>> I got MET successfully installed and I here only want to use mtd
on the
> >>> simulations, no observations for the moment.
> >>>
> >>> So what I do is:
> >>>
> >>>> ../MET/met-9.0/bin/mtd -single
> >>>> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
> >>>> Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config
MODEConfig_SKM
> >>> where Month_oooo__pr_....nc is my model output, to be found here
(1GB):
> >>> https://owncloud.gwdg.de/index.php/s/aVGr0Gql5tKVUcR
> >>> and the config file is attached and basically unchanged.
> >>>
> >>> The error message is:
> >>>
> >>>> DEBUG 2: mtd_read_data() -> processing file
> >>>> "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
> >>>> WARNING:
> >>>> WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) ->
> >>>> returns the first available time for "pr" variable).
> >>>> WARNING:
> >>>> ERROR :
> >>>> ERROR : NcCfFile::getData(NcVar *, const LongArray &,
DataPlane &)
> >>>> const -> needed 3 arguments for variable pr, got 2
> >>>> ERROR :
> >>> The remapping would be specified in this part of the config
file!?:
> >>>
> >>> regrid = {
> >>> to_grid = NONE;
> >>> method = NEAREST;
> >>> width = 1;
> >>> vld_thresh = 0.5;
> >>> shape = SQUARE;
> >>> }
> >>>
> >>> Should that move into fcst?
> >>>
> >>> Also attached you find info on the output i'm working on.
> >>>
> >>> I would be very thankful if you could help me.
> >>>
> >>> With best regards
> >>> Sebastian
> >>>
> >>> --
> >>>
*********************************************************************
> >>> Sebastian K. Müller
> >>> Earth System Physics Section
> >>> The Abdus Salam International Centre for Theoretical Physics
> >>> Strada Costiera 11
> >>> 34100 Trieste, ITALY
> >>> phone: + 39 040 2240 438
> >>> email: smueller at ictp.it
> >>>
*********************************************************************
> >>>
> >>>
> >>>
> --
>
*********************************************************************
> Sebastian K. Müller
> Earth System Physics Section
> The Abdus Salam International Centre for Theoretical Physics
> Strada Costiera 11
> 34100 Trieste, ITALY
> phone: + 39 040 2240 438
> email: smueller at ictp.it
>
*********************************************************************
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #94783] Help on MODE Time Domain Tool
From: Sebastian K. Müller
Time: Mon Apr 06 19:31:52 2020
John,
thank you very much! The python-interface you suggested works
perfectly
and I get around splitting up the output into time steps.
Just two more little bits:
First, to be sure: conv_radius is eventually set as an integer of grid
cells!? I assume so.
Second: the smoothing in time is done by setting again a "radius of
timesteps". So beg=-1 and end=-1, smoothes over three timesteps, that
are [-1,0,1]. So if I set beg=0, and end=0, there is no smoothing in
time!? From Prein et al., 2017, Clim. Dyn, I do not see whether they
applied smoothing in time. Do you know if they did?
Generally, my many thanks: this toolbox is very well documented, easy
to
apply and the help of the team is great!
Best regards
Sebastian
Am 01.04.20 um 01:31 schrieb John Halley Gotway via RT:
> Sebastian,
>
> Yes, the smoothing radius is the "conv_radius" in the config file.
That
> stands for convolution radius. It's the radius of the smoother
defined in
> grid squares. If your data is already very smooth, you could set it
to 0
> to disable the spatial smoothing. But your precip data probably is
NOT
> very smooth. We intend the "grid_res" to be the nominal grid
spacing in
> km. It's used a lot in the MODE config file so that you only need
to
> change one setting to get reasonable defaults for a few other
settings.
> Looks like it's only used once in MTD... to set the conv_radius.
You can
> manually define the conv_radius however you'd like (e.g. conv_radius
= 5;).
>
> Also note that met-9.0 includes the ability to define the amount of
> smoothing in time. The default conv_time_window smooths data across
one
> prior time step (beg = -1) and the next time step (end = 1). Prior
to
> met-9.0, those settings were hard-coded, but they're configurable
now.
>
> In MTD, as in all the other MET tools, regridding is done right
after the
> 2D data is read from the file. It operates totally independently of
the
> smoothing that's done in MTD.
>
> As for reading multiple time steps from the same input file, there
is a
> workaround we can do using python embedding. Instead of having MET
read
> from the NetCDF input file directly, we run a python script to do
the
> reading and then pass that gridded data and metadata to the MET
tools in
> memory. I adapted one of the examples from this page
> https://dtcenter.org/community-code/model-evaluation-tools-
met/sample-analysis-scripts
> to
> create the attached read_pr_timestep.py script.
>
> Run that script like this to make sure it works for you:
> *python3 read_pr_timestep.py
Month_oooo__pr_HadGEM_HIST_062002_pyco.nc 10*
>
> This should read the 11-th (0-based) timestep of the "pr" variable.
If
> that works, then try running it with MET's plot_data_plane tool:
> *plot_data_plane PYTHON_NUMPY pr_python.ps<http://pr_python.ps>
> 'name="read_pr_timestep.py Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
10";'*
>
> Next, I compared the resulting image with MET plotting that data
directly
> to make sure it's oriented correctly.
> *plot_data_plane Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
pr_direct.ps
> <http://pr_direct.ps> 'name="pr"; level="(10,*,*)";'*
>
> I've attached the resulting png's and they match.
>
> So why go to all this trouble? Here's the punchline... you can now
run MTD
> like this:
>
> *mtd -single 0 1 2 3 4 5 6 7 8 9 10 11 12 -config
MTDConfig_SKM_python
> -outdir out*
>
> And inside the MTD config file, use this:
> *fcst = {*
> * file_type = PYTHON_NUMPY;*
>
> * field = { name = "read_pr_timestep.py
> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc MET_PYTHON_INPUT_ARG";*
> * }*
>
> See the attached MTD config file.
>
> This runs MTD for the first 13 timesteps. For each input argument
in the
> "-single" list, it executes the python script defined in the config
file,
> substituting in the current value into the "MET_PYTHON_INPUT_ARG"
slot.
>
> This is a pretty nice usage of features we added in met-9.0!
>
> John
>
> On Tue, Mar 31, 2020 at 3:22 PM Sebastian K. Müller via RT <
> met_help at ucar.edu> wrote:
>
>> <URL:https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
>>
>> Hello again,
>>
>> now, splitting up the file into individual time steps works and is
not
>> too much of an effort. Still I would be interested in whether it
could
>> be possible to do mtg with monthly files...
>>
>> mtg works now and I will be tuning the parameters now. Maybe just a
few
>> questions:
>>
>> the "smoothing radius" of Prein et al. 2017 is "conv_radius" in the
>> config_file? And by default "60/grid_res", meaning that the
smoothing is
>> done over a square of 20*20 grid cells, with "grid_res=3.". Is that
>> correct? Does this interfere in any with the regrid command?
>>
>> I think that's it for now.
>>
>> Thanks for your help,
>> Cheers
>> Sebastian
>>
>> Am 31.03.20 um 10:19 schrieb Sebastian K. Müller:
>>> Hello Hailey,
>>>
>>> thank you. That already heldped a lot. Now I at least work with
the
>>> right configure file. And it is not giving me errors, but not
writing
>>> content to the output files either...
>>>
>>>> smueller at login02 E_OBS $ ../MET/met-9.0/bin/mtd -single
>>>> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
>>>> Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config MTDConfig_SKM
>>>> -outdir /local_scratch/smueller/MTD_output/
>>>> DEBUG 2: mtd_read_data() -> processing file
>>>> "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
>>>> DEBUG 2: mtd_read_data() -> processing file
>>>> "Month_oooo__pr_HadGEM_HIST_072002_pyco.nc"
>>>> DEBUG 2: regridding, if needed ...
>>>> DEBUG 2: Splitting object field
>>>> DEBUG 2: Done splitting
>>>> DEBUG 2: Calculating 3D fcst single attributes
>>>> DEBUG 2: Calculating 2D fcst attributes
>>>> DEBUG 2: Creating 2D constant-time slice attributes file:
>>>> "/local_scratch/smueller/MTD_output//mtd_20010830_003000V_2d.txt"
>>>> DEBUG 2: Creating 3D single simple attributes file:
>>>>
>>
"/local_scratch/smueller/MTD_output//mtd_20010830_003000V_3d_single_simple.txt"
>>>> DEBUG 2: Creating NetCDF file:
>>>> "/local_scratch/smueller/MTD_output//mtd_20010830_003000V_obj.nc"
>>> So, it would be a big effort to produce files of every output time
>>> step. We save the original output "daily", and I merge them to
months.
>>> Is there a way around this?
>>>
>>> Thanks again,
>>> Cheers
>>> Sebastian
>>>
>>> Am 31.03.20 um 05:33 schrieb John Halley Gotway via RT:
>>>> Hello Sebastian,
>>>>
>>>> I see you have questions about running the MODE time domain tool.
I was
>>>> able to download your sample NetCDF file. Looking at the history
>>>> attribute
>>>> in the NetCDF header, I see that you constructed this file by
running
>>>> the
>>>> "cdo mergetime" utility. So you've merged many output files into
a
>>>> single
>>>> one that contains many timesteps:
>>>>
>>>> :history = "Thu Dec 12 08:06:55 2019: cdo mergetime
>>>>
>>
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060100-
>> 2002060200.nc
>>
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060200-
>> 2002060300.nc
>>>> ...
>>>>
>>>> MTD is setup to process the raw, unmerged files. It expects to
read the
>>>> same field of data from multiple files. So if you still have the
>>>> original
>>>> files available, each containing a single time step, please try
>>>> running it
>>>> that way.
>>>>
>>>> Here is the usage statement for mtd:
>>>>
>>>> Usage: mtd
>>>> -fcst file_1 ... file_n | file_list
>>>> -obs file_1 ... file_n | file_list
>>>> -single file_1 ... file_n | file_list
>>>> -config config_file
>>>> [-outdir path]
>>>> [-log file]
>>>> [-v level]
>>>>
>>>> Since you're running with a single data source, you should use
the
>>>> "-single" command line option. You can either list all the input
>>>> files on
>>>> the command line or put that list into an ascii file and put that
ascii
>>>> file list on the command line.
>>>>
>>>> You're running MTD but you sent me a MODE config file. Make sure
you're
>>>> using the default config file for MTD:
>>>>
>>>> fcst = {
>>>> field = {
>>>> name = "pr";
>>>> level = "(0,*,*)";
>>>> }
>>>> ...
>>>>
>>>>
>>>> Hope that helps.
>>>>
>>>> Thanks,
>>>> John Halley Gotway
>>>>
>>>>
>>>> On Mon, Mar 30, 2020 at 5:21 PM Sebastian K. Müller via RT <
>>>> met_help at ucar.edu> wrote:
>>>>
>>>>> Mon Mar 30 17:20:51 2020: Request 94783 was acted upon.
>>>>> Transaction: Ticket created bysmueller at ictp.it
>>>>> Queue: met_help
>>>>> Subject: Help on MODE Time Domain Tool
>>>>> Owner: Nobody
>>>>> Requestors:smueller at ictp.it
>>>>> Status: new
>>>>> Ticket<URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
>>>>>
>>>>>
>>>>> Dear all,
>>>>>
>>>>> I am a PostDoc at ICTP in Trieste, working with convection-
permitting
>>>>> climate simulations in the projects CORDEX-FPS and EUCP, and I
am
>>>>> trying
>>>>> to replicate the analysis done by Prein et al., 2017, but for
Europe,
>>>>> meaning to identify and sample mesoscale convective systems,
using MODE
>>>>> mtd.
>>>>>
>>>>> I got MET successfully installed and I here only want to use mtd
on the
>>>>> simulations, no observations for the moment.
>>>>>
>>>>> So what I do is:
>>>>>
>>>>>> ../MET/met-9.0/bin/mtd -single
>>>>>> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
>>>>>> Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config
MODEConfig_SKM
>>>>> where Month_oooo__pr_....nc is my model output, to be found here
(1GB):
>>>>> https://owncloud.gwdg.de/index.php/s/aVGr0Gql5tKVUcR
>>>>> and the config file is attached and basically unchanged.
>>>>>
>>>>> The error message is:
>>>>>
>>>>>> DEBUG 2: mtd_read_data() -> processing file
>>>>>> "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
>>>>>> WARNING:
>>>>>> WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &) ->
>>>>>> returns the first available time for "pr" variable).
>>>>>> WARNING:
>>>>>> ERROR :
>>>>>> ERROR : NcCfFile::getData(NcVar *, const LongArray &,
DataPlane &)
>>>>>> const -> needed 3 arguments for variable pr, got 2
>>>>>> ERROR :
>>>>> The remapping would be specified in this part of the config
file!?:
>>>>>
>>>>> regrid = {
>>>>> to_grid = NONE;
>>>>> method = NEAREST;
>>>>> width = 1;
>>>>> vld_thresh = 0.5;
>>>>> shape = SQUARE;
>>>>> }
>>>>>
>>>>> Should that move into fcst?
>>>>>
>>>>> Also attached you find info on the output i'm working on.
>>>>>
>>>>> I would be very thankful if you could help me.
>>>>>
>>>>> With best regards
>>>>> Sebastian
>>>>>
>>>>> --
>>>>>
*********************************************************************
>>>>> Sebastian K. Müller
>>>>> Earth System Physics Section
>>>>> The Abdus Salam International Centre for Theoretical Physics
>>>>> Strada Costiera 11
>>>>> 34100 Trieste, ITALY
>>>>> phone: + 39 040 2240 438
>>>>> email:smueller at ictp.it
>>>>>
*********************************************************************
>>>>>
>>>>>
>>>>>
>> --
>>
*********************************************************************
>> Sebastian K. Müller
>> Earth System Physics Section
>> The Abdus Salam International Centre for Theoretical Physics
>> Strada Costiera 11
>> 34100 Trieste, ITALY
>> phone: + 39 040 2240 438
>> email:smueller at ictp.it
>>
*********************************************************************
>>
>>
>>
--
*********************************************************************
Sebastian K. Müller
Earth System Physics Section
The Abdus Salam International Centre for Theoretical Physics
Strada Costiera 11
34100 Trieste, ITALY
phone: + 39 040 2240 438
email:smueller at ictp.it
*********************************************************************
------------------------------------------------
Subject: Help on MODE Time Domain Tool
From: John Halley Gotway
Time: Tue Apr 07 07:01:17 2020
Sebastian,
Great, I’m glad to hear that did the trick. In answer to your
questions,
yes, the conv_radius is ultimately set as an integer number of grid
cells.
And prior to met-9.0, the temporal smoothing was hard-coded as 3 time-
steps
(t-1, t, and t+1). It wasn’t mentioned in Prein because it wasn’t a
configurable option at that point. If you set beg=end=0 to skip the
temporal smoothing, be aware that objects from one time step to the
next
must have some overlap in order to be included in the same 3D space-
time
object.
Thanks,
John
On Mon, Apr 6, 2020 at 7:31 PM Sebastian K. Müller via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
>
> John,
>
> thank you very much! The python-interface you suggested works
perfectly
> and I get around splitting up the output into time steps.
>
> Just two more little bits:
> First, to be sure: conv_radius is eventually set as an integer of
grid
> cells!? I assume so.
> Second: the smoothing in time is done by setting again a "radius of
> timesteps". So beg=-1 and end=-1, smoothes over three timesteps,
that
> are [-1,0,1]. So if I set beg=0, and end=0, there is no smoothing in
> time!? From Prein et al., 2017, Clim. Dyn, I do not see whether they
> applied smoothing in time. Do you know if they did?
>
> Generally, my many thanks: this toolbox is very well documented,
easy to
> apply and the help of the team is great!
>
> Best regards
> Sebastian
>
>
> Am 01.04.20 um 01:31 schrieb John Halley Gotway via RT:
> > Sebastian,
> >
> > Yes, the smoothing radius is the "conv_radius" in the config file.
That
> > stands for convolution radius. It's the radius of the smoother
defined
> in
> > grid squares. If your data is already very smooth, you could set
it to 0
> > to disable the spatial smoothing. But your precip data probably
is NOT
> > very smooth. We intend the "grid_res" to be the nominal grid
spacing in
> > km. It's used a lot in the MODE config file so that you only need
to
> > change one setting to get reasonable defaults for a few other
settings.
> > Looks like it's only used once in MTD... to set the conv_radius.
You can
> > manually define the conv_radius however you'd like (e.g.
conv_radius =
> 5;).
> >
> > Also note that met-9.0 includes the ability to define the amount
of
> > smoothing in time. The default conv_time_window smooths data
across one
> > prior time step (beg = -1) and the next time step (end = 1).
Prior to
> > met-9.0, those settings were hard-coded, but they're configurable
now.
> >
> > In MTD, as in all the other MET tools, regridding is done right
after the
> > 2D data is read from the file. It operates totally independently
of the
> > smoothing that's done in MTD.
> >
> > As for reading multiple time steps from the same input file, there
is a
> > workaround we can do using python embedding. Instead of having
MET read
> > from the NetCDF input file directly, we run a python script to do
the
> > reading and then pass that gridded data and metadata to the MET
tools in
> > memory. I adapted one of the examples from this page
> >
> https://dtcenter.org/community-code/model-evaluation-tools-
met/sample-analysis-scripts
> > to
> > create the attached read_pr_timestep.py script.
> >
> > Run that script like this to make sure it works for you:
> > *python3 read_pr_timestep.py
Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
> 10*
> >
> > This should read the 11-th (0-based) timestep of the "pr"
variable. If
> > that works, then try running it with MET's plot_data_plane tool:
> > *plot_data_plane PYTHON_NUMPY pr_python.ps<http://pr_python.ps>
> > 'name="read_pr_timestep.py
Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
> 10";'*
> >
> > Next, I compared the resulting image with MET plotting that data
directly
> > to make sure it's oriented correctly.
> > *plot_data_plane Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
pr_direct.ps
> > <http://pr_direct.ps> 'name="pr"; level="(10,*,*)";'*
> >
> > I've attached the resulting png's and they match.
> >
> > So why go to all this trouble? Here's the punchline... you can
now run
> MTD
> > like this:
> >
> > *mtd -single 0 1 2 3 4 5 6 7 8 9 10 11 12 -config
MTDConfig_SKM_python
> > -outdir out*
> >
> > And inside the MTD config file, use this:
> > *fcst = {*
> > * file_type = PYTHON_NUMPY;*
> >
> > * field = { name = "read_pr_timestep.py
> > Month_oooo__pr_HadGEM_HIST_062002_pyco.nc MET_PYTHON_INPUT_ARG";*
> > * }*
> >
> > See the attached MTD config file.
> >
> > This runs MTD for the first 13 timesteps. For each input argument
in the
> > "-single" list, it executes the python script defined in the
config file,
> > substituting in the current value into the "MET_PYTHON_INPUT_ARG"
slot.
> >
> > This is a pretty nice usage of features we added in met-9.0!
> >
> > John
> >
> > On Tue, Mar 31, 2020 at 3:22 PM Sebastian K. Müller via RT <
> > met_help at ucar.edu> wrote:
> >
> >> <URL:https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
> >>
> >> Hello again,
> >>
> >> now, splitting up the file into individual time steps works and
is not
> >> too much of an effort. Still I would be interested in whether it
could
> >> be possible to do mtg with monthly files...
> >>
> >> mtg works now and I will be tuning the parameters now. Maybe just
a few
> >> questions:
> >>
> >> the "smoothing radius" of Prein et al. 2017 is "conv_radius" in
the
> >> config_file? And by default "60/grid_res", meaning that the
smoothing is
> >> done over a square of 20*20 grid cells, with "grid_res=3.". Is
that
> >> correct? Does this interfere in any with the regrid command?
> >>
> >> I think that's it for now.
> >>
> >> Thanks for your help,
> >> Cheers
> >> Sebastian
> >>
> >> Am 31.03.20 um 10:19 schrieb Sebastian K. Müller:
> >>> Hello Hailey,
> >>>
> >>> thank you. That already heldped a lot. Now I at least work with
the
> >>> right configure file. And it is not giving me errors, but not
writing
> >>> content to the output files either...
> >>>
> >>>> smueller at login02 E_OBS $ ../MET/met-9.0/bin/mtd -single
> >>>> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
> >>>> Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config MTDConfig_SKM
> >>>> -outdir /local_scratch/smueller/MTD_output/
> >>>> DEBUG 2: mtd_read_data() -> processing file
> >>>> "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
> >>>> DEBUG 2: mtd_read_data() -> processing file
> >>>> "Month_oooo__pr_HadGEM_HIST_072002_pyco.nc"
> >>>> DEBUG 2: regridding, if needed ...
> >>>> DEBUG 2: Splitting object field
> >>>> DEBUG 2: Done splitting
> >>>> DEBUG 2: Calculating 3D fcst single attributes
> >>>> DEBUG 2: Calculating 2D fcst attributes
> >>>> DEBUG 2: Creating 2D constant-time slice attributes file:
> >>>>
"/local_scratch/smueller/MTD_output//mtd_20010830_003000V_2d.txt"
> >>>> DEBUG 2: Creating 3D single simple attributes file:
> >>>>
> >>
>
"/local_scratch/smueller/MTD_output//mtd_20010830_003000V_3d_single_simple.txt"
> >>>> DEBUG 2: Creating NetCDF file:
> >>>>
"/local_scratch/smueller/MTD_output//mtd_20010830_003000V_obj.nc"
> >>> So, it would be a big effort to produce files of every output
time
> >>> step. We save the original output "daily", and I merge them to
months.
> >>> Is there a way around this?
> >>>
> >>> Thanks again,
> >>> Cheers
> >>> Sebastian
> >>>
> >>> Am 31.03.20 um 05:33 schrieb John Halley Gotway via RT:
> >>>> Hello Sebastian,
> >>>>
> >>>> I see you have questions about running the MODE time domain
tool. I
> was
> >>>> able to download your sample NetCDF file. Looking at the
history
> >>>> attribute
> >>>> in the NetCDF header, I see that you constructed this file by
running
> >>>> the
> >>>> "cdo mergetime" utility. So you've merged many output files
into a
> >>>> single
> >>>> one that contains many timesteps:
> >>>>
> >>>> :history = "Thu Dec 12 08:06:55 2019: cdo mergetime
> >>>>
> >>
>
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060100-
> >> 2002060200.nc
> >>
>
/marconi_scratch/userexternal/smueller/output/alps_3km/ICTP/HadGEM/HIST/NN/ICTP-
RegCM4-7-0/v0/1hr/pr/pr_alps_3km_HadGEM_HIST_NN_ICTP-RegCM4-7-
0_v0_1hr_2002060200-
> >> 2002060300.nc
> >>>> ...
> >>>>
> >>>> MTD is setup to process the raw, unmerged files. It expects to
read
> the
> >>>> same field of data from multiple files. So if you still have
the
> >>>> original
> >>>> files available, each containing a single time step, please try
> >>>> running it
> >>>> that way.
> >>>>
> >>>> Here is the usage statement for mtd:
> >>>>
> >>>> Usage: mtd
> >>>> -fcst file_1 ... file_n | file_list
> >>>> -obs file_1 ... file_n | file_list
> >>>> -single file_1 ... file_n | file_list
> >>>> -config config_file
> >>>> [-outdir path]
> >>>> [-log file]
> >>>> [-v level]
> >>>>
> >>>> Since you're running with a single data source, you should use
the
> >>>> "-single" command line option. You can either list all the
input
> >>>> files on
> >>>> the command line or put that list into an ascii file and put
that
> ascii
> >>>> file list on the command line.
> >>>>
> >>>> You're running MTD but you sent me a MODE config file. Make
sure
> you're
> >>>> using the default config file for MTD:
> >>>>
> >>>> fcst = {
> >>>> field = {
> >>>> name = "pr";
> >>>> level = "(0,*,*)";
> >>>> }
> >>>> ...
> >>>>
> >>>>
> >>>> Hope that helps.
> >>>>
> >>>> Thanks,
> >>>> John Halley Gotway
> >>>>
> >>>>
> >>>> On Mon, Mar 30, 2020 at 5:21 PM Sebastian K. Müller via RT <
> >>>> met_help at ucar.edu> wrote:
> >>>>
> >>>>> Mon Mar 30 17:20:51 2020: Request 94783 was acted upon.
> >>>>> Transaction: Ticket created bysmueller at ictp.it
> >>>>> Queue: met_help
> >>>>> Subject: Help on MODE Time Domain Tool
> >>>>> Owner: Nobody
> >>>>> Requestors:smueller at ictp.it
> >>>>> Status: new
> >>>>> Ticket<URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=94783 >
> >>>>>
> >>>>>
> >>>>> Dear all,
> >>>>>
> >>>>> I am a PostDoc at ICTP in Trieste, working with convection-
permitting
> >>>>> climate simulations in the projects CORDEX-FPS and EUCP, and I
am
> >>>>> trying
> >>>>> to replicate the analysis done by Prein et al., 2017, but for
Europe,
> >>>>> meaning to identify and sample mesoscale convective systems,
using
> MODE
> >>>>> mtd.
> >>>>>
> >>>>> I got MET successfully installed and I here only want to use
mtd on
> the
> >>>>> simulations, no observations for the moment.
> >>>>>
> >>>>> So what I do is:
> >>>>>
> >>>>>> ../MET/met-9.0/bin/mtd -single
> >>>>>> Month_oooo__pr_HadGEM_HIST_062002_pyco.nc
> >>>>>> Month_oooo__pr_HadGEM_HIST_072002_pyco.nc -config
MODEConfig_SKM
> >>>>> where Month_oooo__pr_....nc is my model output, to be found
here
> (1GB):
> >>>>> https://owncloud.gwdg.de/index.php/s/aVGr0Gql5tKVUcR
> >>>>> and the config file is attached and basically unchanged.
> >>>>>
> >>>>> The error message is:
> >>>>>
> >>>>>> DEBUG 2: mtd_read_data() -> processing file
> >>>>>> "Month_oooo__pr_HadGEM_HIST_062002_pyco.nc"
> >>>>>> WARNING:
> >>>>>> WARNING: MetNcCFDataFile::data_plane(VarInfo &, DataPlane &)
->
> >>>>>> returns the first available time for "pr" variable).
> >>>>>> WARNING:
> >>>>>> ERROR :
> >>>>>> ERROR : NcCfFile::getData(NcVar *, const LongArray &,
DataPlane &)
> >>>>>> const -> needed 3 arguments for variable pr, got 2
> >>>>>> ERROR :
> >>>>> The remapping would be specified in this part of the config
file!?:
> >>>>>
> >>>>> regrid = {
> >>>>> to_grid = NONE;
> >>>>> method = NEAREST;
> >>>>> width = 1;
> >>>>> vld_thresh = 0.5;
> >>>>> shape = SQUARE;
> >>>>> }
> >>>>>
> >>>>> Should that move into fcst?
> >>>>>
> >>>>> Also attached you find info on the output i'm working on.
> >>>>>
> >>>>> I would be very thankful if you could help me.
> >>>>>
> >>>>> With best regards
> >>>>> Sebastian
> >>>>>
> >>>>> --
> >>>>>
*********************************************************************
> >>>>> Sebastian K. Müller
> >>>>> Earth System Physics Section
> >>>>> The Abdus Salam International Centre for Theoretical Physics
> >>>>> Strada Costiera 11
> >>>>> 34100 Trieste, ITALY
> >>>>> phone: + 39 040 2240 438
> >>>>> email:smueller at ictp.it
> >>>>>
*********************************************************************
> >>>>>
> >>>>>
> >>>>>
> >> --
> >>
*********************************************************************
> >> Sebastian K. Müller
> >> Earth System Physics Section
> >> The Abdus Salam International Centre for Theoretical Physics
> >> Strada Costiera 11
> >> 34100 Trieste, ITALY
> >> phone: + 39 040 2240 438
> >> email:smueller at ictp.it
> >>
*********************************************************************
> >>
> >>
> >>
> --
>
*********************************************************************
> Sebastian K. Müller
> Earth System Physics Section
> The Abdus Salam International Centre for Theoretical Physics
> Strada Costiera 11
> 34100 Trieste, ITALY
> phone: + 39 040 2240 438
> email:smueller at ictp.it
>
*********************************************************************
>
>
>
------------------------------------------------
More information about the Met_help
mailing list