[Met_help] [rt.rap.ucar.edu #94917] History for Series-Analysis

[John Halley Gotway via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

I am running series analysis and looks like it producing reasonable outputs and processing all the files I am providing through the script. However, I am noticing one error line after every file it processes. I want to make sure that the computations are done correctly. I am attaching the log and the config file. I am processing V10 variable. 

I am running the script run_serana_imd.sh on Cheyenne (/glade/work/biswas/India_aq/eval/MET/series_analysis) and the output are in /glade/work/biswas/India_aq/eval/MET/series_analysis_out. 


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Series-Analysis
From: John Halley Gotway
Time: Tue Apr 14 15:37:44 2020

Biswas,

Thanks for sending your log file.  Judging by those warnings, no, I do
not think that Series-Analysis is doing what you want here.

Your forecast data are wrfout NetCDF files, each containing a single
output time.  Your observation data are GRIB2 files, each containing
many records for each month.  And the warning message to tool is
printing says it all:

DEBUG 2: Found data for V10(0,*,*) in file:
/glade/scratch/rkumar/india_fcst/20161010/wrf_run/wrfout_d01_2016-10-
10_12:00:00
WARNING:
WARNING: MetGrib2DataFile::data_plane() -> Using the first of 124
matching records for "VGRD/Z10":

It finds data for "V10(0,*,*)" from the forecast file but then finds
124 records for "VGRD/Z10" from the observation file
"ngfs_reanl_201610.grb2".

I realize that you want it to pick one record out of the 124 whose
valid time matches the forecast valid time, but that's not what it's
doing.  It's simpler picking the first one and printing a
warning message about it.  So how can we fix this?

I see 2 options.
(1) Run a script to break out the GRIB2 gfs reanalysis files into a
separate file for each valid time.  That way we'd no longer have
multiple matching records.
(2) Switch to using met-9.0 and use python embedding to serve up the
data you want.  I helped a user do this this past week, and it
actually worked.

I can think of how we might change the logic of the compiled code in
Series-Analysis to do what you want in this case.  But I'm a little
worried about these sort of changes to the logic.  For example, we
could have the code use the fcst valid time for each timestep to only
search for obs data with that exact same valid time.  But that'd
require the times to match exactly... down to the second.  And that
would make it less useful when using radar data where the times may
differ by a couple of minutes.

But maybe there's something we could add to the config file to
facilitate this.  Any thoughts?

John

------------------------------------------------
Subject: Series-Analysis
From: Mrinal Biswas
Time: Tue Apr 14 15:55:16 2020

Hi John,
Many thanks for looking into this for me. Both options are okay for
me. I
do want to try the second option and see if it works for me too. Do
you
have any instructions on how to use the python embedding option.
Thanks
Biswas


On Tue, Apr 14, 2020 at 3:38 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:

> Biswas,
>
> Thanks for sending your log file.  Judging by those warnings, no, I
do not
> think that Series-Analysis is doing what you want here.
>
> Your forecast data are wrfout NetCDF files, each containing a single
> output time.  Your observation data are GRIB2 files, each containing
many
> records for each month.  And the warning message to tool is printing
says
> it all:
>
> DEBUG 2: Found data for V10(0,*,*) in file:
> /glade/scratch/rkumar/india_fcst/20161010/wrf_run/wrfout_d01_2016-
10-10_12:00:00
> WARNING:
> WARNING: MetGrib2DataFile::data_plane() -> Using the first of 124
matching
> records for "VGRD/Z10":
>
> It finds data for "V10(0,*,*)" from the forecast file but then finds
124
> records for "VGRD/Z10" from the observation file
"ngfs_reanl_201610.grb2".
>
> I realize that you want it to pick one record out of the 124 whose
valid
> time matches the forecast valid time, but that's not what it's
doing.  It's
> simpler picking the first one and printing a warning message about
it.  So
> how can we fix this?
>
> I see 2 options.
> (1) Run a script to break out the GRIB2 gfs reanalysis files into a
> separate file for each valid time.  That way we'd no longer have
multiple
> matching records.
> (2) Switch to using met-9.0 and use python embedding to serve up the
data
> you want.  I helped a user do this this past week, and it actually
worked.
>
> I can think of how we might change the logic of the compiled code in
> Series-Analysis to do what you want in this case.  But I'm a little
worried
> about these sort of changes to the logic.  For example, we could
have the
> code use the fcst valid time for each timestep to only search for
obs data
> with that exact same valid time.  But that'd require the times to
match
> exactly... down to the second.  And that would make it less useful
when
> using radar data where the times may differ by a couple of minutes.
>
> But maybe there's something we could add to the config file to
facilitate
> this.  Any thoughts?
>
> John
>


--

*My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.*

------------------------------------------------
Subject: Series-Analysis
From: John Halley Gotway
Time: Tue Apr 14 16:15:10 2020

Biswas,

Here's a link to the RT ticket itself:
https://rt.rap.ucar.edu/rt/Ticket/History.html?id=94783

And here's a link to what was posted to the MET-Help email archives:
http://mailman.ucar.edu/pipermail/met_help/2020-April/003931.html

Here's the relevant part of that:
-----------------------

So why go to all this trouble?  Here's the punchline... you can now
run MTD like this:

mtd -single 0 1 2 3 4 5 6 7 8 9 10 11 12 -config MTDConfig_SKM_python
-outdir out

And inside the MTD config file, use this:
*fcst = {*
*   file_type = PYTHON_NUMPY;*

*   field = {      name  = "read_pr_timestep.py
Month_oooo__pr_HadGEM_HIST_062002_pyco.nc MET_PYTHON_INPUT_ARG";*
*   }*

See the attached MTD config file.

This runs MTD for the first 13 timesteps.  For each input argument in
the "-single" list, it executes the python script defined in the
config file, substituting in the current value into the
"MET_PYTHON_INPUT_ARG" slot.

-----------------------

For you, we could use a python script such as:

*read_gfs_reanal.py ngfs_reanl_201610.grb2:VGRD:Z10:20161010_12*

And have that script parse the argument and read VGRD/Z10 from the
input file for that timestamp.  And you'd put those args on the
Series-Analysis command line:

-obs ngfs_reanl_201610.grb2:VGRD:Z10:20161010_12
ngfs_reanl_201610.grb2:VGRD:Z10:20161010_18 ... and so on


Of course there's any number of ways you could structure the logic for
this.

John

------------------------------------------------