[Met_help] [rt.rap.ucar.edu #75766] History for problem running series-analysis
John Halley Gotway via RT
met_help at ucar.edu
Tue Apr 26 13:58:47 MDT 2016
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Using MET V4.1, I tried running series-analysis on input data which has successfully been used in grid-stat. My script to run series analysis, my config file, my input forecast file list, observations list, and a sample observation file are all attached. The error dialog seems to indicate that the variable TMP/Z2 cannot be found in the observation files. When I run grid-stat it successfully finds this variable and completed the run with all desired results. I would like some assistance to find out why I'm having this issue if possible. See the following excerpt from the error dialog:
WARNING:
WARNING: MetGrib1DataFile::data_plane() -> No exact match found for VarInfo "TMP/Z2" in GRIB file "/home/jraby/MET_obs/GSD_LAPS/20120207/120382300_laps_288.grb1".
WARNING:
ERROR :
ERROR : get_series_entry() -> Could not find data for TMP/Z2 in file list:
ERROR : /home/jraby/MET_obs/GSD_LAPS/20120207/120381200_laps_288.grb1
ERROR : /home/jraby/MET_obs/GSD_LAPS/20120207/120381300_laps_288.grb1
ERROR : /home/jraby/MET_obs/GSD_LAPS/20120207/120381400_laps_288.grb1
ERROR : /home/jraby/MET_obs/GSD_LAPS/20120207/120381500_laps_288.grb1
ERROR : /home/jraby/MET_obs/GSD_LAPS/20120207/120381600_laps_288.grb1
ERROR : /home/jraby/MET_obs/GSD_LAPS/20120207/120381700_laps_288.grb1
ERROR : /home/jraby/MET_obs/GSD_LAPS/20120207/120381800_laps_288.grb1
ERROR : /home/jraby/MET_obs/GSD_LAPS/20120207/120381900_laps_288.grb1
ERROR : /home/jraby/MET_obs/GSD_LAPS/20120207/120382000_laps_288.grb1
ERROR : /home/jraby/MET_obs/GSD_LAPS/20120207/120382100_laps_288.grb1
ERROR : /home/jraby/MET_obs/GSD_LAPS/20120207/120382200_laps_288.grb1
ERROR : /home/jraby/MET_obs/GSD_LAPS/20120207/120382300_laps_288.grb1
ERROR :
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: problem running series-analysis
From: Randy Bullock
Time: Wed Apr 06 09:09:34 2016
Hi John -
John Halley Gotway is out on vacation this week, so I'll hand your
questions off to him when he gets back. I don't know much about the
series_analysis tool myself, or I'd help you myself.
Randy
On Tue, Apr 5, 2016 at 12:02 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Tue Apr 05 12:02:46 2016: Request 75766 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: problem running series-analysis
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
>
> Using MET V4.1, I tried running series-analysis on input data which
has
> successfully been used in grid-stat. My script to run series
analysis, my
> config file, my input forecast file list, observations list, and a
sample
> observation file are all attached. The error dialog seems to
indicate that
> the variable TMP/Z2 cannot be found in the observation files. When I
run
> grid-stat it successfully finds this variable and completed the run
with
> all desired results. I would like some assistance to find out why
I'm
> having this issue if possible. See the following excerpt from the
error
> dialog:
>
> WARNING:
> WARNING: MetGrib1DataFile::data_plane() -> No exact match found for
> VarInfo "TMP/Z2" in GRIB file
> "/home/jraby/MET_obs/GSD_LAPS/20120207/120382300_laps_288.grb1".
> WARNING:
> ERROR :
> ERROR : get_series_entry() -> Could not find data for TMP/Z2 in
file list:
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120381200_laps_288.grb1
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120381300_laps_288.grb1
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120381400_laps_288.grb1
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120381500_laps_288.grb1
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120381600_laps_288.grb1
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120381700_laps_288.grb1
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120381800_laps_288.grb1
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120381900_laps_288.grb1
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120382000_laps_288.grb1
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120382100_laps_288.grb1
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120382200_laps_288.grb1
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120382300_laps_288.grb1
> ERROR :
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem running series-analysis
From: Raby, John W USA CIV
Time: Thu Apr 07 07:24:47 2016
Randy -
Thanks for the update on my question. Next week if fine for me.
R/
John
-----Original Message-----
From: Randy Bullock via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, April 06, 2016 9:10 AM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem running
series-analysis
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
Hi John -
John Halley Gotway is out on vacation this week, so I'll hand your
questions off to him when he gets back. I don't know much about the
series_analysis tool myself, or I'd help you myself.
Randy
On Tue, Apr 5, 2016 at 12:02 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Tue Apr 05 12:02:46 2016: Request 75766 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: problem running series-analysis
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
>
> Using MET V4.1, I tried running series-analysis on input data which
> has successfully been used in grid-stat. My script to run series
> analysis, my config file, my input forecast file list, observations
> list, and a sample observation file are all attached. The error
dialog
> seems to indicate that the variable TMP/Z2 cannot be found in the
> observation files. When I run grid-stat it successfully finds this
> variable and completed the run with all desired results. I would
like
> some assistance to find out why I'm having this issue if possible.
See
> the following excerpt from the error
> dialog:
>
> WARNING:
> WARNING: MetGrib1DataFile::data_plane() -> No exact match found for
> VarInfo "TMP/Z2" in GRIB file
> "/home/jraby/MET_obs/GSD_LAPS/20120207/120382300_laps_288.grb1".
> WARNING:
> ERROR :
> ERROR : get_series_entry() -> Could not find data for TMP/Z2 in
file list:
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120381200_laps_288.grb1
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120381300_laps_288.grb1
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120381400_laps_288.grb1
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120381500_laps_288.grb1
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120381600_laps_288.grb1
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120381700_laps_288.grb1
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120381800_laps_288.grb1
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120381900_laps_288.grb1
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120382000_laps_288.grb1
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120382100_laps_288.grb1
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120382200_laps_288.grb1
> ERROR :
/home/jraby/MET_obs/GSD_LAPS/20120207/120382300_laps_288.grb1
> ERROR :
>
>
>
>
------------------------------------------------
Subject: problem running series-analysis
From: John Halley Gotway
Time: Wed Apr 13 12:34:05 2016
Hello John,
I see that you were having trouble running series-analysis on 2-m
temperature data last week.
Is this still an issue? If so, I'd be happy to take a look.
Thanks,
John Halley Gotway
On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT
<met_help at ucar.edu>
wrote:
>
> Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by bullock
> Queue: met_help
> Subject: problem running series-analysis
> Owner: johnhg
> Requestors: john.w.raby2.civ at mail.mil
> Status: open
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
>
> This transaction appears to have no content
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem running series-analysis
From: Raby, John W USA CIV
Time: Wed Apr 13 13:27:24 2016
Hi John -
Yes. I got stuck on trying to figure out the cause of that error.
Seems like if Grid-Stat can read the 2-m temperature from the gridded
obs, then why would Series-Analysis not be able to read it?
Thanks.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, April 13, 2016 12:34 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem running
series-analysis
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
Hello John,
I see that you were having trouble running series-analysis on 2-m
temperature data last week.
Is this still an issue? If so, I'd be happy to take a look.
Thanks,
John Halley Gotway
On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT
<met_help at ucar.edu>
wrote:
>
> Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by bullock
> Queue: met_help
> Subject: problem running series-analysis
> Owner: johnhg
> Requestors: john.w.raby2.civ at mail.mil
> Status: open
> Ticket <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
>
> This transaction appears to have no content
>
------------------------------------------------
Subject: problem running series-analysis
From: John Halley Gotway
Time: Wed Apr 13 15:48:01 2016
John,
I don't see any immediately obvious problems in the data you sent to
us. I
see that you're writing a log file named:
~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
Are you able to send me that complete log file? That might help me
diagnose what's going on.
Here's my initial guess...
You are running series-analysis by defining a single field in the
configuration file. And you're passing to it lists of 12 forecast
files
and 12 observation files.
It reads TMP/Z2 from the first forecast file and check the valid time
for
that data. Next, it looks in the observation file list (starting with
the
first file) for TMP/Z2 with the *SAME* valid time.
Next, it reads TMP/Z2 from the second forecast file, gets the valid
time,
and looks in the observation file list (starting with the second file)
for
TMP/Z2 with the *SAME* valid time.
I looked at the observation file (120382300_laps_288.grb1) you sent me
and
see that it's valid time is 2012020723, which happens to match the
timestamp in the forecast file name. The first thing I'd check is the
valid time of the corresponding forecast file:
131127A_WRFPRS_d03_2012-02-07_23:00:00
Is it actually valid at the same time, 2012020723? If not, that's the
problem. If so, that's not the problem.
Thanks,
John
On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> Hi John -
>
> Yes. I got stuck on trying to figure out the cause of that error.
Seems
> like if Grid-Stat can read the 2-m temperature from the gridded obs,
then
> why would Series-Analysis not be able to read it?
>
> Thanks.
>
> R/
> John
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, April 13, 2016 12:34 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running
> series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> Hello John,
>
> I see that you were having trouble running series-analysis on 2-m
> temperature data last week.
>
> Is this still an issue? If so, I'd be happy to take a look.
>
> Thanks,
> John Halley Gotway
>
> On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
> > Transaction: Given to johnhg (John Halley Gotway) by bullock
> > Queue: met_help
> > Subject: problem running series-analysis
> > Owner: johnhg
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: open
> > Ticket <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> >
> > This transaction appears to have no content
> >
>
>
>
>
------------------------------------------------
Subject: problem running series-analysis
From: Raby, John W USA CIV
Time: Wed Apr 13 15:56:35 2016
John -
Thanks for taking a look and offering your initial guess at some
possible issues.Sorry I had not included the log file in the first
email. It's attached. I will consider your initial guess ideas and
offer my thoughts on those by sep email.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Wednesday, April 13, 2016 3:48 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running series-analysis
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
I don't see any immediately obvious problems in the data you sent to
us. I
see that you're writing a log file named:
~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
Are you able to send me that complete log file? That might help me
diagnose what's going on.
Here's my initial guess...
You are running series-analysis by defining a single field in the
configuration file. And you're passing to it lists of 12 forecast
files
and 12 observation files.
It reads TMP/Z2 from the first forecast file and check the valid time
for
that data. Next, it looks in the observation file list (starting with
the
first file) for TMP/Z2 with the *SAME* valid time.
Next, it reads TMP/Z2 from the second forecast file, gets the valid
time,
and looks in the observation file list (starting with the second file)
for
TMP/Z2 with the *SAME* valid time.
I looked at the observation file (120382300_laps_288.grb1) you sent me
and
see that it's valid time is 2012020723, which happens to match the
timestamp in the forecast file name. The first thing I'd check is the
valid time of the corresponding forecast file:
131127A_WRFPRS_d03_2012-02-07_23:00:00
Is it actually valid at the same time, 2012020723? If not, that's the
problem. If so, that's not the problem.
Thanks,
John
On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> Hi John -
>
> Yes. I got stuck on trying to figure out the cause of that error.
Seems
> like if Grid-Stat can read the 2-m temperature from the gridded obs,
then
> why would Series-Analysis not be able to read it?
>
> Thanks.
>
> R/
> John
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Wednesday, April 13, 2016 12:34 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running
> series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> Hello John,
>
> I see that you were having trouble running series-analysis on 2-m
> temperature data last week.
>
> Is this still an issue? If so, I'd be happy to take a look.
>
> Thanks,
> John Halley Gotway
>
> On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
> > Transaction: Given to johnhg (John Halley Gotway) by bullock
> > Queue: met_help
> > Subject: problem running series-analysis
> > Owner: johnhg
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: open
> > Ticket <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> >
> > This transaction appears to have no content
> >
>
>
>
>
------------------------------------------------
Subject: problem running series-analysis
From: John Halley Gotway
Time: Wed Apr 13 16:10:27 2016
John,
Thanks for sending the log file.
I see the following line in there indicating that it read 2m
temperature
from the first forecast file:
DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match for
VarInfo "TMP/Z2" in GRIB record 270 of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_D_FDDA/131127A_WRFPRS_d03_2012-
02-07_12:00:00".
The question is what's the timing info for that data? Try running the
following:
wgrib -d 270 -V
/home/jraby/MET_WRFpostprd/20120207_D_FDDA/131127A_WRFPRS_d03_2012-02-
07_12:00:00
And please send me the output. We want to see if it the valid time is
actually 2012020712. <2012020723>
John
On Wed, Apr 13, 2016 at 3:56 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> Thanks for taking a look and offering your initial guess at some
possible
> issues.Sorry I had not included the log file in the first email.
It's
> attached. I will consider your initial guess ideas and offer my
thoughts on
> those by sep email.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Wednesday, April 13, 2016 3:48 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> I don't see any immediately obvious problems in the data you sent to
us. I
> see that you're writing a log file named:
> ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
>
> Are you able to send me that complete log file? That might help me
> diagnose what's going on.
>
> Here's my initial guess...
>
> You are running series-analysis by defining a single field in the
> configuration file. And you're passing to it lists of 12 forecast
files
> and 12 observation files.
>
> It reads TMP/Z2 from the first forecast file and check the valid
time for
> that data. Next, it looks in the observation file list (starting
with the
> first file) for TMP/Z2 with the *SAME* valid time.
>
> Next, it reads TMP/Z2 from the second forecast file, gets the valid
time,
> and looks in the observation file list (starting with the second
file) for
> TMP/Z2 with the *SAME* valid time.
>
> I looked at the observation file (120382300_laps_288.grb1) you sent
me and
> see that it's valid time is 2012020723, which happens to match the
> timestamp in the forecast file name. The first thing I'd check is
the
> valid time of the corresponding forecast file:
> 131127A_WRFPRS_d03_2012-02-07_23:00:00
>
> Is it actually valid at the same time, 2012020723? If not, that's
the
> problem. If so, that's not the problem.
>
> Thanks,
> John
>
>
> On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url: Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> > Hi John -
> >
> > Yes. I got stuck on trying to figure out the cause of that error.
Seems
> > like if Grid-Stat can read the 2-m temperature from the gridded
obs, then
> > why would Series-Analysis not be able to read it?
> >
> > Thanks.
> >
> > R/
> > John
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> > Sent: Wednesday, April 13, 2016 12:34 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running
> > series-analysis
> >
> > All active links contained in this email were disabled. Please
verify
> the
> > identity of the sender, and confirm the authenticity of all links
> contained
> > within the message prior to copying and pasting the address to a
Web
> > browser.
> >
> >
> >
> >
> > ----
> >
> > Hello John,
> >
> > I see that you were having trouble running series-analysis on 2-m
> > temperature data last week.
> >
> > Is this still an issue? If so, I'd be happy to take a look.
> >
> > Thanks,
> > John Halley Gotway
> >
> > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> > >
> > > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
> > > Transaction: Given to johnhg (John Halley Gotway) by bullock
> > > Queue: met_help
> > > Subject: problem running series-analysis
> > > Owner: johnhg
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: open
> > > Ticket <Caution-Caution-url:
> > > Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >
> > >
> > > This transaction appears to have no content
> > >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem running series-analysis
From: Raby, John W USA CIV
Time: Wed Apr 13 16:20:22 2016
John -
Based on your analysis of the reading of the input files, I can offer
a possibility that I'm aware of regarding the valid times.The hours
for the example input data (observations) I sent you is indeed 23
which matches that of the forecast data valid at the same hour (23),
but the minutes of the forecast file is 00 while for the observations
it is 15. I was aware of this mismatch in the minutes from when I ran
Grid-Stat and it flagged it as a "Warning", but it went ahead and
generated output with no Error. The presence of the min value of 15
is an artifact of the fact that this data was produced by the LAPS
system. The valid time of the data is actually 2300 instead of 2315.
That said, I will run wgrib on the forecast file per your suggestion
to confirm the valid time.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Wednesday, April 13, 2016 3:48 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running series-analysis
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
I don't see any immediately obvious problems in the data you sent to
us. I
see that you're writing a log file named:
~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
Are you able to send me that complete log file? That might help me
diagnose what's going on.
Here's my initial guess...
You are running series-analysis by defining a single field in the
configuration file. And you're passing to it lists of 12 forecast
files
and 12 observation files.
It reads TMP/Z2 from the first forecast file and check the valid time
for
that data. Next, it looks in the observation file list (starting with
the
first file) for TMP/Z2 with the *SAME* valid time.
Next, it reads TMP/Z2 from the second forecast file, gets the valid
time,
and looks in the observation file list (starting with the second file)
for
TMP/Z2 with the *SAME* valid time.
I looked at the observation file (120382300_laps_288.grb1) you sent me
and
see that it's valid time is 2012020723, which happens to match the
timestamp in the forecast file name. The first thing I'd check is the
valid time of the corresponding forecast file:
131127A_WRFPRS_d03_2012-02-07_23:00:00
Is it actually valid at the same time, 2012020723? If not, that's the
problem. If so, that's not the problem.
Thanks,
John
On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> Hi John -
>
> Yes. I got stuck on trying to figure out the cause of that error.
Seems
> like if Grid-Stat can read the 2-m temperature from the gridded obs,
then
> why would Series-Analysis not be able to read it?
>
> Thanks.
>
> R/
> John
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Wednesday, April 13, 2016 12:34 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running
> series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> Hello John,
>
> I see that you were having trouble running series-analysis on 2-m
> temperature data last week.
>
> Is this still an issue? If so, I'd be happy to take a look.
>
> Thanks,
> John Halley Gotway
>
> On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT
<met_help at ucar.edu>
> wrote:
>
> >
> > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
> > Transaction: Given to johnhg (John Halley Gotway) by bullock
> > Queue: met_help
> > Subject: problem running series-analysis
> > Owner: johnhg
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: open
> > Ticket <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> >
> > This transaction appears to have no content
> >
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem running series-analysis
From: Raby, John W USA CIV
Time: Wed Apr 13 16:31:43 2016
John -
I ran the wgrib with the following results:
carson:jraby ~/MET_WRFpostprd/20120207_D_FDDA
> wgrib -d 270 -V 131127A_WRFPRS_d03_2012-02-07_12:00:00
rec 270:33865104:date 2012020712 TMP kpds5=11 kpds6=105 kpds7=2
levels=(0,2) grid=255 2 m above gnd anl:
TMP=Temp. [K]
timerange 0 P1 0 P2 0 TimeU 1 nx 288 ny 288 GDS grid 3 num_in_ave 0
missing 0
center 7 subcenter 0 process 125 Table 2 scan: WE:SN winds(grid)
Lambert Conf: Lat1 30.614000 Lon1 -119.723000 Lov -117.100000
Latin1 32.900000 Latin2 32.900000 LatSP 0.000000 LonSP 0.000000
North Pole (288 x 288) Dx 1.750000 Dy 1.750000 scan 64 mode 136
min/max data 279.42 290.752 num bits 14 BDS_Ref 279420 DecScale 3
BinScale 0
carson:jraby ~/MET_WRFpostprd/20120207_D_FDDA
For the observations I ran a modified wgrib command which removed the
"270" in order to make it work. Hope is has the right info for you.
carson:jraby ~/MET_obs/GSD_LAPS/20120207
> wgrib -d -V 120381200_laps_288.grb1
1:0:d=12020712:PRMSL:kpds5=2:kpds6=1:kpds7=0:TR=0:P1=15:P2=0:TimeU=0:sfc:15min
fcst:NAve=0
carson:jraby ~/MET_obs/GSD_LAPS/20120207
The following is a more complete listing, if that helps:
carson:jraby ~/MET_obs/GSD_LAPS/20120207
> wgrib -s 120381200_laps_288.grb1
1:0:d=12020712:PRMSL:sfc:15min fcst:NAve=0
2:124568:d=12020712:PRES:sfc:15min fcst:NAve=0
3:280240:d=12020712:TMP:2 m above gnd:15min fcst:NAve=0
4:363336:d=12020712:DPT:2 m above gnd:15min fcst:NAve=0
5:456800:d=12020712:RH:2 m above gnd:15min fcst:NAve=0
6:529528:d=12020712:UGRD:10 m above gnd:15min fcst:NAve=0
7:602256:d=12020712:VGRD:10 m above gnd:15min fcst:NAve=0
carson:jraby ~/MET_obs/GSD_LAPS/20120207
This shows that LAPS introduces the "15min" as the minutes, but in
reality it's valid at 1200 rather than 1215.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Wednesday, April 13, 2016 4:10 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running series-analysis
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
Thanks for sending the log file.
I see the following line in there indicating that it read 2m
temperature
from the first forecast file:
DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match for
VarInfo "TMP/Z2" in GRIB record 270 of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_D_FDDA/131127A_WRFPRS_d03_2012-
02-07_12:00:00".
The question is what's the timing info for that data? Try running the
following:
wgrib -d 270 -V
/home/jraby/MET_WRFpostprd/20120207_D_FDDA/131127A_WRFPRS_d03_2012-02-
07_12:00:00
And please send me the output. We want to see if it the valid time is
actually 2012020712. <2012020723>
John
On Wed, Apr 13, 2016 at 3:56 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> Thanks for taking a look and offering your initial guess at some
possible
> issues.Sorry I had not included the log file in the first email.
It's
> attached. I will consider your initial guess ideas and offer my
thoughts on
> those by sep email.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Wednesday, April 13, 2016 3:48 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> I don't see any immediately obvious problems in the data you sent to
us. I
> see that you're writing a log file named:
> ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
>
> Are you able to send me that complete log file? That might help me
> diagnose what's going on.
>
> Here's my initial guess...
>
> You are running series-analysis by defining a single field in the
> configuration file. And you're passing to it lists of 12 forecast
files
> and 12 observation files.
>
> It reads TMP/Z2 from the first forecast file and check the valid
time for
> that data. Next, it looks in the observation file list (starting
with the
> first file) for TMP/Z2 with the *SAME* valid time.
>
> Next, it reads TMP/Z2 from the second forecast file, gets the valid
time,
> and looks in the observation file list (starting with the second
file) for
> TMP/Z2 with the *SAME* valid time.
>
> I looked at the observation file (120382300_laps_288.grb1) you sent
me and
> see that it's valid time is 2012020723, which happens to match the
> timestamp in the forecast file name. The first thing I'd check is
the
> valid time of the corresponding forecast file:
> 131127A_WRFPRS_d03_2012-02-07_23:00:00
>
> Is it actually valid at the same time, 2012020723? If not, that's
the
> problem. If so, that's not the problem.
>
> Thanks,
> John
>
>
> On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url: Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> > Hi John -
> >
> > Yes. I got stuck on trying to figure out the cause of that error.
Seems
> > like if Grid-Stat can read the 2-m temperature from the gridded
obs, then
> > why would Series-Analysis not be able to read it?
> >
> > Thanks.
> >
> > R/
> > John
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-Caution-
mailto:met_help at ucar.edu]
> > Sent: Wednesday, April 13, 2016 12:34 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running
> > series-analysis
> >
> > All active links contained in this email were disabled. Please
verify
> the
> > identity of the sender, and confirm the authenticity of all links
> contained
> > within the message prior to copying and pasting the address to a
Web
> > browser.
> >
> >
> >
> >
> > ----
> >
> > Hello John,
> >
> > I see that you were having trouble running series-analysis on 2-m
> > temperature data last week.
> >
> > Is this still an issue? If so, I'd be happy to take a look.
> >
> > Thanks,
> > John Halley Gotway
> >
> > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> > >
> > > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
> > > Transaction: Given to johnhg (John Halley Gotway) by bullock
> > > Queue: met_help
> > > Subject: problem running series-analysis
> > > Owner: johnhg
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: open
> > > Ticket <Caution-Caution-Caution-url:
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >
> > >
> > > This transaction appears to have no content
> > >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem running series-analysis
From: Raby, John W USA CIV
Time: Wed Apr 13 16:37:06 2016
John -
If you want, I could send you the forecast file valid at 2012020723 to
match the valid time of the observations which I provided in the first
email. It's over 40MB, so I would need to send it to your email
address (which I have in my contact list) using a file transfer system
we have here at ARL. Let me know.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Wednesday, April 13, 2016 4:10 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running series-analysis
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
Thanks for sending the log file.
I see the following line in there indicating that it read 2m
temperature
from the first forecast file:
DEBUG 3: MetGrib1DataFile::data_plane_array() -> Found range match for
VarInfo "TMP/Z2" in GRIB record 270 of GRIB file
"/home/jraby/MET_WRFpostprd/20120207_D_FDDA/131127A_WRFPRS_d03_2012-
02-07_12:00:00".
The question is what's the timing info for that data? Try running the
following:
wgrib -d 270 -V
/home/jraby/MET_WRFpostprd/20120207_D_FDDA/131127A_WRFPRS_d03_2012-02-
07_12:00:00
And please send me the output. We want to see if it the valid time is
actually 2012020712. <2012020723>
John
On Wed, Apr 13, 2016 at 3:56 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> Thanks for taking a look and offering your initial guess at some
possible
> issues.Sorry I had not included the log file in the first email.
It's
> attached. I will consider your initial guess ideas and offer my
thoughts on
> those by sep email.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Wednesday, April 13, 2016 3:48 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> I don't see any immediately obvious problems in the data you sent to
us. I
> see that you're writing a log file named:
> ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
>
> Are you able to send me that complete log file? That might help me
> diagnose what's going on.
>
> Here's my initial guess...
>
> You are running series-analysis by defining a single field in the
> configuration file. And you're passing to it lists of 12 forecast
files
> and 12 observation files.
>
> It reads TMP/Z2 from the first forecast file and check the valid
time for
> that data. Next, it looks in the observation file list (starting
with the
> first file) for TMP/Z2 with the *SAME* valid time.
>
> Next, it reads TMP/Z2 from the second forecast file, gets the valid
time,
> and looks in the observation file list (starting with the second
file) for
> TMP/Z2 with the *SAME* valid time.
>
> I looked at the observation file (120382300_laps_288.grb1) you sent
me and
> see that it's valid time is 2012020723, which happens to match the
> timestamp in the forecast file name. The first thing I'd check is
the
> valid time of the corresponding forecast file:
> 131127A_WRFPRS_d03_2012-02-07_23:00:00
>
> Is it actually valid at the same time, 2012020723? If not, that's
the
> problem. If so, that's not the problem.
>
> Thanks,
> John
>
>
> On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url: Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> > Hi John -
> >
> > Yes. I got stuck on trying to figure out the cause of that error.
Seems
> > like if Grid-Stat can read the 2-m temperature from the gridded
obs, then
> > why would Series-Analysis not be able to read it?
> >
> > Thanks.
> >
> > R/
> > John
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-Caution-
mailto:met_help at ucar.edu]
> > Sent: Wednesday, April 13, 2016 12:34 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running
> > series-analysis
> >
> > All active links contained in this email were disabled. Please
verify
> the
> > identity of the sender, and confirm the authenticity of all links
> contained
> > within the message prior to copying and pasting the address to a
Web
> > browser.
> >
> >
> >
> >
> > ----
> >
> > Hello John,
> >
> > I see that you were having trouble running series-analysis on 2-m
> > temperature data last week.
> >
> > Is this still an issue? If so, I'd be happy to take a look.
> >
> > Thanks,
> > John Halley Gotway
> >
> > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> > >
> > > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
> > > Transaction: Given to johnhg (John Halley Gotway) by bullock
> > > Queue: met_help
> > > Subject: problem running series-analysis
> > > Owner: johnhg
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: open
> > > Ticket <Caution-Caution-Caution-url:
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >
> > >
> > > This transaction appears to have no content
> > >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: problem running series-analysis
From: John Halley Gotway
Time: Wed Apr 13 16:37:13 2016
John,
Ah yes, that time offset is the problem here.
And it would have been nice for Series-Analysis to be more verbose
about
the timing information it's processing.
Please try using the updated config file that I've attached, and see
if
Series-Analysis actually runs.
We pass grid-stat 1 forecast file and 1 observation file. So even if
the
times don't match up exactly, it writes a warning and compares them.
Series-Analysis tries to be a bit fancier. You pass it N forecast and
N
observation files. For each forecast files, it searches all the
observation files for a match.
But that more complex logic breaks down when there's a 15 minute time
offset. So we should think about how better to handle this use case.
Thanks,
John
On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> Based on your analysis of the reading of the input files, I can
offer a
> possibility that I'm aware of regarding the valid times.The hours
for the
> example input data (observations) I sent you is indeed 23 which
matches
> that of the forecast data valid at the same hour (23), but the
minutes of
> the forecast file is 00 while for the observations it is 15. I was
aware of
> this mismatch in the minutes from when I ran Grid-Stat and it
flagged it as
> a "Warning", but it went ahead and generated output with no Error.
The
> presence of the min value of 15 is an artifact of the fact that
this data
> was produced by the LAPS system. The valid time of the data is
actually
> 2300 instead of 2315. That said, I will run wgrib on the forecast
file per
> your suggestion to confirm the valid time.
> R/
> John
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Wednesday, April 13, 2016 3:48 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> I don't see any immediately obvious problems in the data you sent to
us. I
> see that you're writing a log file named:
> ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
>
> Are you able to send me that complete log file? That might help me
> diagnose what's going on.
>
> Here's my initial guess...
>
> You are running series-analysis by defining a single field in the
> configuration file. And you're passing to it lists of 12 forecast
files
> and 12 observation files.
>
> It reads TMP/Z2 from the first forecast file and check the valid
time for
> that data. Next, it looks in the observation file list (starting
with the
> first file) for TMP/Z2 with the *SAME* valid time.
>
> Next, it reads TMP/Z2 from the second forecast file, gets the valid
time,
> and looks in the observation file list (starting with the second
file) for
> TMP/Z2 with the *SAME* valid time.
>
> I looked at the observation file (120382300_laps_288.grb1) you sent
me and
> see that it's valid time is 2012020723, which happens to match the
> timestamp in the forecast file name. The first thing I'd check is
the
> valid time of the corresponding forecast file:
> 131127A_WRFPRS_d03_2012-02-07_23:00:00
>
> Is it actually valid at the same time, 2012020723? If not, that's
the
> problem. If so, that's not the problem.
>
> Thanks,
> John
>
>
> On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url: Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> > Hi John -
> >
> > Yes. I got stuck on trying to figure out the cause of that error.
Seems
> > like if Grid-Stat can read the 2-m temperature from the gridded
obs, then
> > why would Series-Analysis not be able to read it?
> >
> > Thanks.
> >
> > R/
> > John
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> > Sent: Wednesday, April 13, 2016 12:34 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running
> > series-analysis
> >
> > All active links contained in this email were disabled. Please
verify
> the
> > identity of the sender, and confirm the authenticity of all links
> contained
> > within the message prior to copying and pasting the address to a
Web
> > browser.
> >
> >
> >
> >
> > ----
> >
> > Hello John,
> >
> > I see that you were having trouble running series-analysis on 2-m
> > temperature data last week.
> >
> > Is this still an issue? If so, I'd be happy to take a look.
> >
> > Thanks,
> > John Halley Gotway
> >
> > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> > >
> > > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
> > > Transaction: Given to johnhg (John Halley Gotway) by bullock
> > > Queue: met_help
> > > Subject: problem running series-analysis
> > > Owner: johnhg
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: open
> > > Ticket <Caution-Caution-url:
> > > Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >
> > >
> > > This transaction appears to have no content
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: problem running series-analysis
From: Raby, John W USA CIV
Time: Wed Apr 13 16:50:37 2016
John -
I looked at your config file and it looks like a nice workaround. I
reran using your config file. Ran considerably longer this time. I
also have a 14MB output NetCDF file which is definitely encouraging! I
have to leave for the day in about 10 mins and will be back in touch
tomorrow. Thanks for your help.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Wednesday, April 13, 2016 4:37 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running series-analysis
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
Ah yes, that time offset is the problem here.
And it would have been nice for Series-Analysis to be more verbose
about
the timing information it's processing.
Please try using the updated config file that I've attached, and see
if
Series-Analysis actually runs.
We pass grid-stat 1 forecast file and 1 observation file. So even if
the
times don't match up exactly, it writes a warning and compares them.
Series-Analysis tries to be a bit fancier. You pass it N forecast and
N
observation files. For each forecast files, it searches all the
observation files for a match.
But that more complex logic breaks down when there's a 15 minute time
offset. So we should think about how better to handle this use case.
Thanks,
John
On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> Based on your analysis of the reading of the input files, I can
offer a
> possibility that I'm aware of regarding the valid times.The hours
for the
> example input data (observations) I sent you is indeed 23 which
matches
> that of the forecast data valid at the same hour (23), but the
minutes of
> the forecast file is 00 while for the observations it is 15. I was
aware of
> this mismatch in the minutes from when I ran Grid-Stat and it
flagged it as
> a "Warning", but it went ahead and generated output with no Error.
The
> presence of the min value of 15 is an artifact of the fact that
this data
> was produced by the LAPS system. The valid time of the data is
actually
> 2300 instead of 2315. That said, I will run wgrib on the forecast
file per
> your suggestion to confirm the valid time.
> R/
> John
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Wednesday, April 13, 2016 3:48 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> I don't see any immediately obvious problems in the data you sent to
us. I
> see that you're writing a log file named:
> ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
>
> Are you able to send me that complete log file? That might help me
> diagnose what's going on.
>
> Here's my initial guess...
>
> You are running series-analysis by defining a single field in the
> configuration file. And you're passing to it lists of 12 forecast
files
> and 12 observation files.
>
> It reads TMP/Z2 from the first forecast file and check the valid
time for
> that data. Next, it looks in the observation file list (starting
with the
> first file) for TMP/Z2 with the *SAME* valid time.
>
> Next, it reads TMP/Z2 from the second forecast file, gets the valid
time,
> and looks in the observation file list (starting with the second
file) for
> TMP/Z2 with the *SAME* valid time.
>
> I looked at the observation file (120382300_laps_288.grb1) you sent
me and
> see that it's valid time is 2012020723, which happens to match the
> timestamp in the forecast file name. The first thing I'd check is
the
> valid time of the corresponding forecast file:
> 131127A_WRFPRS_d03_2012-02-07_23:00:00
>
> Is it actually valid at the same time, 2012020723? If not, that's
the
> problem. If so, that's not the problem.
>
> Thanks,
> John
>
>
> On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url: Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> > Hi John -
> >
> > Yes. I got stuck on trying to figure out the cause of that error.
Seems
> > like if Grid-Stat can read the 2-m temperature from the gridded
obs, then
> > why would Series-Analysis not be able to read it?
> >
> > Thanks.
> >
> > R/
> > John
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-Caution-
mailto:met_help at ucar.edu]
> > Sent: Wednesday, April 13, 2016 12:34 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running
> > series-analysis
> >
> > All active links contained in this email were disabled. Please
verify
> the
> > identity of the sender, and confirm the authenticity of all links
> contained
> > within the message prior to copying and pasting the address to a
Web
> > browser.
> >
> >
> >
> >
> > ----
> >
> > Hello John,
> >
> > I see that you were having trouble running series-analysis on 2-m
> > temperature data last week.
> >
> > Is this still an issue? If so, I'd be happy to take a look.
> >
> > Thanks,
> > John Halley Gotway
> >
> > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> > >
> > > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
> > > Transaction: Given to johnhg (John Halley Gotway) by bullock
> > > Queue: met_help
> > > Subject: problem running series-analysis
> > > Owner: johnhg
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: open
> > > Ticket <Caution-Caution-Caution-url:
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >
> > >
> > > This transaction appears to have no content
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: problem running series-analysis
From: John Halley Gotway
Time: Wed Apr 13 16:56:04 2016
John,
Ah yes, I meant to mention this. Please try increasing the block size
from
1024 to something much larger.
Your grid is 288x288 = 82944 grid points. Try setting:
block_size = 100000;
That will process it all in one pass through the data and should make
it
run much faster.
John
On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> I looked at your config file and it looks like a nice workaround. I
reran
> using your config file. Ran considerably longer this time. I also
have a
> 14MB output NetCDF file which is definitely encouraging! I have to
leave
> for the day in about 10 mins and will be back in touch tomorrow.
Thanks for
> your help.
>
> R/
> John
>
>
>
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Wednesday, April 13, 2016 4:37 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Ah yes, that time offset is the problem here.
>
> And it would have been nice for Series-Analysis to be more verbose
about
> the timing information it's processing.
>
> Please try using the updated config file that I've attached, and see
if
> Series-Analysis actually runs.
>
> We pass grid-stat 1 forecast file and 1 observation file. So even
if the
> times don't match up exactly, it writes a warning and compares them.
> Series-Analysis tries to be a bit fancier. You pass it N forecast
and N
> observation files. For each forecast files, it searches all the
> observation files for a match.
>
> But that more complex logic breaks down when there's a 15 minute
time
> offset. So we should think about how better to handle this use
case.
>
> Thanks,
> John
>
> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url: Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> > John -
> >
> > Based on your analysis of the reading of the input files, I can
offer a
> > possibility that I'm aware of regarding the valid times.The hours
for the
> > example input data (observations) I sent you is indeed 23 which
matches
> > that of the forecast data valid at the same hour (23), but the
minutes of
> > the forecast file is 00 while for the observations it is 15. I was
aware
> of
> > this mismatch in the minutes from when I ran Grid-Stat and it
flagged it
> as
> > a "Warning", but it went ahead and generated output with no Error.
The
> > presence of the min value of 15 is an artifact of the fact that
this
> data
> > was produced by the LAPS system. The valid time of the data is
actually
> > 2300 instead of 2315. That said, I will run wgrib on the forecast
file
> per
> > your suggestion to confirm the valid time.
> > R/
> > John
> >
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Wednesday, April 13, 2016 3:48 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> > running series-analysis
> >
> > All active links contained in this email were disabled. Please
verify
> the
> > identity of the sender, and confirm the authenticity of all links
> contained
> > within the message prior to copying and pasting the address to a
Web
> > browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > I don't see any immediately obvious problems in the data you sent
to
> us. I
> > see that you're writing a log file named:
> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
> >
> > Are you able to send me that complete log file? That might help
me
> > diagnose what's going on.
> >
> > Here's my initial guess...
> >
> > You are running series-analysis by defining a single field in the
> > configuration file. And you're passing to it lists of 12 forecast
files
> > and 12 observation files.
> >
> > It reads TMP/Z2 from the first forecast file and check the valid
time for
> > that data. Next, it looks in the observation file list (starting
with
> the
> > first file) for TMP/Z2 with the *SAME* valid time.
> >
> > Next, it reads TMP/Z2 from the second forecast file, gets the
valid time,
> > and looks in the observation file list (starting with the second
file)
> for
> > TMP/Z2 with the *SAME* valid time.
> >
> > I looked at the observation file (120382300_laps_288.grb1) you
sent me
> and
> > see that it's valid time is 2012020723, which happens to match the
> > timestamp in the forecast file name. The first thing I'd check is
the
> > valid time of the corresponding forecast file:
> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
> >
> > Is it actually valid at the same time, 2012020723? If not, that's
the
> > problem. If so, that's not the problem.
> >
> > Thanks,
> > John
> >
> >
> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-url: Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >
> > > Hi John -
> > >
> > > Yes. I got stuck on trying to figure out the cause of that
error. Seems
> > > like if Grid-Stat can read the 2-m temperature from the gridded
obs,
> then
> > > why would Series-Analysis not be able to read it?
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [Caution-Caution-mailto:
> met_help at ucar.edu]
> > > Sent: Wednesday, April 13, 2016 12:34 PM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil
> >
> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running
> > > series-analysis
> > >
> > > All active links contained in this email were disabled. Please
verify
> > the
> > > identity of the sender, and confirm the authenticity of all
links
> > contained
> > > within the message prior to copying and pasting the address to a
Web
> > > browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > Hello John,
> > >
> > > I see that you were having trouble running series-analysis on 2-
m
> > > temperature data last week.
> > >
> > > Is this still an issue? If so, I'd be happy to take a look.
> > >
> > > Thanks,
> > > John Halley Gotway
> > >
> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT <
> met_help at ucar.edu
> > >
> > > wrote:
> > >
> > > >
> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
> > > > Transaction: Given to johnhg (John Halley Gotway) by bullock
> > > > Queue: met_help
> > > > Subject: problem running series-analysis
> > > > Owner: johnhg
> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > Status: open
> > > > Ticket <Caution-Caution-Caution-url:
> > > > Caution-Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >
> > > >
> > > > This transaction appears to have no content
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: problem running series-analysis
From: John Halley Gotway
Time: Wed Apr 13 16:57:40 2016
Also, I see the following error message in the log file you sent:
ERROR : store_stat_cnt() -> unsupported column name requested "IQR"
IQR was added in version 5.1, but it not defined in version 4.1.
John
On Wed, Apr 13, 2016 at 4:55 PM, John Halley Gotway <johnhg at ucar.edu>
wrote:
> John,
>
> Ah yes, I meant to mention this. Please try increasing the block
size
> from 1024 to something much larger.
>
> Your grid is 288x288 = 82944 grid points. Try setting:
> block_size = 100000;
>
> That will process it all in one pass through the data and should
make it
> run much faster.
>
> John
>
> On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>>
>> John -
>>
>> I looked at your config file and it looks like a nice workaround. I
reran
>> using your config file. Ran considerably longer this time. I also
have a
>> 14MB output NetCDF file which is definitely encouraging! I have to
leave
>> for the day in about 10 mins and will be back in touch tomorrow.
Thanks for
>> your help.
>>
>> R/
>> John
>>
>>
>>
>>
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Wednesday, April 13, 2016 4:37 PM
>> To: Raby, John W CIV USARMY RDECOM ARL (US)
>> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
>> running series-analysis
>>
>> All active links contained in this email were disabled. Please
verify
>> the identity of the sender, and confirm the authenticity of all
links
>> contained within the message prior to copying and pasting the
address to a
>> Web browser.
>>
>>
>>
>>
>> ----
>>
>> John,
>>
>> Ah yes, that time offset is the problem here.
>>
>> And it would have been nice for Series-Analysis to be more verbose
about
>> the timing information it's processing.
>>
>> Please try using the updated config file that I've attached, and
see if
>> Series-Analysis actually runs.
>>
>> We pass grid-stat 1 forecast file and 1 observation file. So even
if the
>> times don't match up exactly, it writes a warning and compares
them.
>> Series-Analysis tries to be a bit fancier. You pass it N forecast
and N
>> observation files. For each forecast files, it searches all the
>> observation files for a match.
>>
>> But that more complex logic breaks down when there's a 15 minute
time
>> offset. So we should think about how better to handle this use
case.
>>
>> Thanks,
>> John
>>
>> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <Caution-url: Caution-
>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>> >
>> > John -
>> >
>> > Based on your analysis of the reading of the input files, I can
offer a
>> > possibility that I'm aware of regarding the valid times.The hours
for
>> the
>> > example input data (observations) I sent you is indeed 23 which
matches
>> > that of the forecast data valid at the same hour (23), but the
minutes
>> of
>> > the forecast file is 00 while for the observations it is 15. I
was
>> aware of
>> > this mismatch in the minutes from when I ran Grid-Stat and it
flagged
>> it as
>> > a "Warning", but it went ahead and generated output with no
Error. The
>> > presence of the min value of 15 is an artifact of the fact that
this
>> data
>> > was produced by the LAPS system. The valid time of the data is
actually
>> > 2300 instead of 2315. That said, I will run wgrib on the forecast
file
>> per
>> > your suggestion to confirm the valid time.
>> > R/
>> > John
>> >
>> > ________________________________________
>> > From: John Halley Gotway via RT [met_help at ucar.edu]
>> > Sent: Wednesday, April 13, 2016 3:48 PM
>> > To: Raby, John W CIV USARMY RDECOM ARL (US)
>> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
>> > running series-analysis
>> >
>> > All active links contained in this email were disabled. Please
verify
>> the
>> > identity of the sender, and confirm the authenticity of all links
>> contained
>> > within the message prior to copying and pasting the address to a
Web
>> > browser.
>> >
>> >
>> >
>> >
>> > ----
>> >
>> > John,
>> >
>> > I don't see any immediately obvious problems in the data you sent
to
>> us. I
>> > see that you're writing a log file named:
>> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
>> >
>> > Are you able to send me that complete log file? That might help
me
>> > diagnose what's going on.
>> >
>> > Here's my initial guess...
>> >
>> > You are running series-analysis by defining a single field in the
>> > configuration file. And you're passing to it lists of 12
forecast files
>> > and 12 observation files.
>> >
>> > It reads TMP/Z2 from the first forecast file and check the valid
time
>> for
>> > that data. Next, it looks in the observation file list (starting
with
>> the
>> > first file) for TMP/Z2 with the *SAME* valid time.
>> >
>> > Next, it reads TMP/Z2 from the second forecast file, gets the
valid
>> time,
>> > and looks in the observation file list (starting with the second
file)
>> for
>> > TMP/Z2 with the *SAME* valid time.
>> >
>> > I looked at the observation file (120382300_laps_288.grb1) you
sent me
>> and
>> > see that it's valid time is 2012020723, which happens to match
the
>> > timestamp in the forecast file name. The first thing I'd check
is the
>> > valid time of the corresponding forecast file:
>> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
>> >
>> > Is it actually valid at the same time, 2012020723? If not,
that's the
>> > problem. If so, that's not the problem.
>> >
>> > Thanks,
>> > John
>> >
>> >
>> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > <Caution-Caution-url: Caution-
>> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>> > >
>> > > Hi John -
>> > >
>> > > Yes. I got stuck on trying to figure out the cause of that
error.
>> Seems
>> > > like if Grid-Stat can read the 2-m temperature from the gridded
obs,
>> then
>> > > why would Series-Analysis not be able to read it?
>> > >
>> > > Thanks.
>> > >
>> > > R/
>> > > John
>> > >
>> > >
>> > > -----Original Message-----
>> > > From: John Halley Gotway via RT [Caution-Caution-mailto:
>> met_help at ucar.edu]
>> > > Sent: Wednesday, April 13, 2016 12:34 PM
>> > > To: Raby, John W CIV USARMY RDECOM ARL (US) <
>> john.w.raby2.civ at mail.mil>
>> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
>> running
>> > > series-analysis
>> > >
>> > > All active links contained in this email were disabled. Please
verify
>> > the
>> > > identity of the sender, and confirm the authenticity of all
links
>> > contained
>> > > within the message prior to copying and pasting the address to
a Web
>> > > browser.
>> > >
>> > >
>> > >
>> > >
>> > > ----
>> > >
>> > > Hello John,
>> > >
>> > > I see that you were having trouble running series-analysis on
2-m
>> > > temperature data last week.
>> > >
>> > > Is this still an issue? If so, I'd be happy to take a look.
>> > >
>> > > Thanks,
>> > > John Halley Gotway
>> > >
>> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT <
>> met_help at ucar.edu
>> > >
>> > > wrote:
>> > >
>> > > >
>> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
>> > > > Transaction: Given to johnhg (John Halley Gotway) by bullock
>> > > > Queue: met_help
>> > > > Subject: problem running series-analysis
>> > > > Owner: johnhg
>> > > > Requestors: john.w.raby2.civ at mail.mil
>> > > > Status: open
>> > > > Ticket <Caution-Caution-Caution-url:
>> > > > Caution-Caution-
>> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>> > > >
>> > > >
>> > > > This transaction appears to have no content
>> > > >
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem running series-analysis
From: Raby, John W USA CIV
Time: Thu Apr 14 08:13:39 2016
John -
Thanks for the tips on the block size and on why I had the error
described below. I learned something new by seeing how you used the
"valid_time" entry to define the valid times when you are working with
data which may have different valid times. Will definitely document
this example for my bag of tricks. I had never used this before.
I assume that using the valid_time workaround that this is what I will
need to do as long as I'm using V4.1 unless you come up with something
else. This workaround works for me.
I used IDV to visualize the NetCDF output file and noted the following
issues:
RMSE - missing info, can't plot.
ME - single value=.745, no plot
GSS - missing, no plot for all threshold values
FBIAS - same as GSS
FMEAN - Same as GSS
CSI - data is either 1 or missing, no plot.
Base rate - data range is 0-80, no plot.
Accuracy - data is 1 for all thresholds, no plot.
I am using plot_data_plane to try plotting the NetCDF with log file
output to see what that does. I'll send you the results.
Thanks.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, April 13, 2016 4:58 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running series-analysis
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
Also, I see the following error message in the log file you sent:
ERROR : store_stat_cnt() -> unsupported column name requested "IQR"
IQR was added in version 5.1, but it not defined in version 4.1.
John
On Wed, Apr 13, 2016 at 4:55 PM, John Halley Gotway <johnhg at ucar.edu>
wrote:
> John,
>
> Ah yes, I meant to mention this. Please try increasing the block
size
> from 1024 to something much larger.
>
> Your grid is 288x288 = 82944 grid points. Try setting:
> block_size = 100000;
>
> That will process it all in one pass through the data and should
make
> it run much faster.
>
> John
>
> On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <Caution-url:
>> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>>
>> John -
>>
>> I looked at your config file and it looks like a nice workaround. I
>> reran using your config file. Ran considerably longer this time. I
>> also have a 14MB output NetCDF file which is definitely
encouraging!
>> I have to leave for the day in about 10 mins and will be back in
>> touch tomorrow. Thanks for your help.
>>
>> R/
>> John
>>
>>
>>
>>
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Wednesday, April 13, 2016 4:37 PM
>> To: Raby, John W CIV USARMY RDECOM ARL (US)
>> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
>> running series-analysis
>>
>> All active links contained in this email were disabled. Please
>> verify the identity of the sender, and confirm the authenticity of
>> all links contained within the message prior to copying and pasting
>> the address to a Web browser.
>>
>>
>>
>>
>> ----
>>
>> John,
>>
>> Ah yes, that time offset is the problem here.
>>
>> And it would have been nice for Series-Analysis to be more verbose
>> about the timing information it's processing.
>>
>> Please try using the updated config file that I've attached, and
see
>> if Series-Analysis actually runs.
>>
>> We pass grid-stat 1 forecast file and 1 observation file. So even
if
>> the times don't match up exactly, it writes a warning and compares
them.
>> Series-Analysis tries to be a bit fancier. You pass it N forecast
>> and N observation files. For each forecast files, it searches all
>> the observation files for a match.
>>
>> But that more complex logic breaks down when there's a 15 minute
time
>> offset. So we should think about how better to handle this use
case.
>>
>> Thanks,
>> John
>>
>> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <Caution-Caution-url: Caution-
>> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>> >
>> > John -
>> >
>> > Based on your analysis of the reading of the input files, I can
>> > offer a possibility that I'm aware of regarding the valid
times.The
>> > hours for
>> the
>> > example input data (observations) I sent you is indeed 23 which
>> > matches that of the forecast data valid at the same hour (23),
but
>> > the minutes
>> of
>> > the forecast file is 00 while for the observations it is 15. I
was
>> aware of
>> > this mismatch in the minutes from when I ran Grid-Stat and it
>> > flagged
>> it as
>> > a "Warning", but it went ahead and generated output with no
Error.
>> > The presence of the min value of 15 is an artifact of the fact
>> > that this
>> data
>> > was produced by the LAPS system. The valid time of the data is
>> > actually
>> > 2300 instead of 2315. That said, I will run wgrib on the forecast
>> > file
>> per
>> > your suggestion to confirm the valid time.
>> > R/
>> > John
>> >
>> > ________________________________________
>> > From: John Halley Gotway via RT [met_help at ucar.edu]
>> > Sent: Wednesday, April 13, 2016 3:48 PM
>> > To: Raby, John W CIV USARMY RDECOM ARL (US)
>> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
>> > running series-analysis
>> >
>> > All active links contained in this email were disabled. Please
>> > verify
>> the
>> > identity of the sender, and confirm the authenticity of all links
>> contained
>> > within the message prior to copying and pasting the address to a
>> > Web browser.
>> >
>> >
>> >
>> >
>> > ----
>> >
>> > John,
>> >
>> > I don't see any immediately obvious problems in the data you sent
>> > to
>> us. I
>> > see that you're writing a log file named:
>> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
>> >
>> > Are you able to send me that complete log file? That might help
me
>> > diagnose what's going on.
>> >
>> > Here's my initial guess...
>> >
>> > You are running series-analysis by defining a single field in the
>> > configuration file. And you're passing to it lists of 12
forecast
>> > files and 12 observation files.
>> >
>> > It reads TMP/Z2 from the first forecast file and check the valid
>> > time
>> for
>> > that data. Next, it looks in the observation file list (starting
>> > with
>> the
>> > first file) for TMP/Z2 with the *SAME* valid time.
>> >
>> > Next, it reads TMP/Z2 from the second forecast file, gets the
valid
>> time,
>> > and looks in the observation file list (starting with the second
>> > file)
>> for
>> > TMP/Z2 with the *SAME* valid time.
>> >
>> > I looked at the observation file (120382300_laps_288.grb1) you
sent
>> > me
>> and
>> > see that it's valid time is 2012020723, which happens to match
the
>> > timestamp in the forecast file name. The first thing I'd check
is
>> > the valid time of the corresponding forecast file:
>> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
>> >
>> > Is it actually valid at the same time, 2012020723? If not,
that's
>> > the problem. If so, that's not the problem.
>> >
>> > Thanks,
>> > John
>> >
>> >
>> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > <Caution-Caution-Caution-url: Caution-
>> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=7
>> > 5766 >
>> > >
>> > > Hi John -
>> > >
>> > > Yes. I got stuck on trying to figure out the cause of that
error.
>> Seems
>> > > like if Grid-Stat can read the 2-m temperature from the gridded
>> > > obs,
>> then
>> > > why would Series-Analysis not be able to read it?
>> > >
>> > > Thanks.
>> > >
>> > > R/
>> > > John
>> > >
>> > >
>> > > -----Original Message-----
>> > > From: John Halley Gotway via RT [Caution-Caution-Caution-
mailto:
>> met_help at ucar.edu]
>> > > Sent: Wednesday, April 13, 2016 12:34 PM
>> > > To: Raby, John W CIV USARMY RDECOM ARL (US) <
>> john.w.raby2.civ at mail.mil>
>> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
>> running
>> > > series-analysis
>> > >
>> > > All active links contained in this email were disabled. Please
>> > > verify
>> > the
>> > > identity of the sender, and confirm the authenticity of all
links
>> > contained
>> > > within the message prior to copying and pasting the address to
a
>> > > Web browser.
>> > >
>> > >
>> > >
>> > >
>> > > ----
>> > >
>> > > Hello John,
>> > >
>> > > I see that you were having trouble running series-analysis on
2-m
>> > > temperature data last week.
>> > >
>> > > Is this still an issue? If so, I'd be happy to take a look.
>> > >
>> > > Thanks,
>> > > John Halley Gotway
>> > >
>> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT <
>> met_help at ucar.edu
>> > >
>> > > wrote:
>> > >
>> > > >
>> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
>> > > > Transaction: Given to johnhg (John Halley Gotway) by bullock
>> > > > Queue: met_help
>> > > > Subject: problem running series-analysis
>> > > > Owner: johnhg
>> > > > Requestors: john.w.raby2.civ at mail.mil
>> > > > Status: open
>> > > > Ticket <Caution-Caution-Caution-Caution-url:
>> > > > Caution-Caution-
>> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=7
>> > 5766 >
>> > > >
>> > > >
>> > > > This transaction appears to have no content
>> > > >
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>>
>
------------------------------------------------
Subject: problem running series-analysis
From: Raby, John W USA CIV
Time: Thu Apr 14 09:04:34 2016
John - I am having an issue with using plot_data_plane to plot the
series-analysis output file.
I've attached the log file which shows a whitespace problem which I am
trying to debug. I also attached the output of "ncdump" generated from
the output file "results_m3o3_TMP".
Any suggestions on the whitespace error?
Thanks.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Wednesday, April 13, 2016 4:57 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running series-analysis
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
Also, I see the following error message in the log file you sent:
ERROR : store_stat_cnt() -> unsupported column name requested "IQR"
IQR was added in version 5.1, but it not defined in version 4.1.
John
On Wed, Apr 13, 2016 at 4:55 PM, John Halley Gotway <johnhg at ucar.edu>
wrote:
> John,
>
> Ah yes, I meant to mention this. Please try increasing the block
size
> from 1024 to something much larger.
>
> Your grid is 288x288 = 82944 grid points. Try setting:
> block_size = 100000;
>
> That will process it all in one pass through the data and should
make it
> run much faster.
>
> John
>
> On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>>
>> John -
>>
>> I looked at your config file and it looks like a nice workaround. I
reran
>> using your config file. Ran considerably longer this time. I also
have a
>> 14MB output NetCDF file which is definitely encouraging! I have to
leave
>> for the day in about 10 mins and will be back in touch tomorrow.
Thanks for
>> your help.
>>
>> R/
>> John
>>
>>
>>
>>
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Wednesday, April 13, 2016 4:37 PM
>> To: Raby, John W CIV USARMY RDECOM ARL (US)
>> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
>> running series-analysis
>>
>> All active links contained in this email were disabled. Please
verify
>> the identity of the sender, and confirm the authenticity of all
links
>> contained within the message prior to copying and pasting the
address to a
>> Web browser.
>>
>>
>>
>>
>> ----
>>
>> John,
>>
>> Ah yes, that time offset is the problem here.
>>
>> And it would have been nice for Series-Analysis to be more verbose
about
>> the timing information it's processing.
>>
>> Please try using the updated config file that I've attached, and
see if
>> Series-Analysis actually runs.
>>
>> We pass grid-stat 1 forecast file and 1 observation file. So even
if the
>> times don't match up exactly, it writes a warning and compares
them.
>> Series-Analysis tries to be a bit fancier. You pass it N forecast
and N
>> observation files. For each forecast files, it searches all the
>> observation files for a match.
>>
>> But that more complex logic breaks down when there's a 15 minute
time
>> offset. So we should think about how better to handle this use
case.
>>
>> Thanks,
>> John
>>
>> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <Caution-Caution-url: Caution-
>> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>> >
>> > John -
>> >
>> > Based on your analysis of the reading of the input files, I can
offer a
>> > possibility that I'm aware of regarding the valid times.The hours
for
>> the
>> > example input data (observations) I sent you is indeed 23 which
matches
>> > that of the forecast data valid at the same hour (23), but the
minutes
>> of
>> > the forecast file is 00 while for the observations it is 15. I
was
>> aware of
>> > this mismatch in the minutes from when I ran Grid-Stat and it
flagged
>> it as
>> > a "Warning", but it went ahead and generated output with no
Error. The
>> > presence of the min value of 15 is an artifact of the fact that
this
>> data
>> > was produced by the LAPS system. The valid time of the data is
actually
>> > 2300 instead of 2315. That said, I will run wgrib on the forecast
file
>> per
>> > your suggestion to confirm the valid time.
>> > R/
>> > John
>> >
>> > ________________________________________
>> > From: John Halley Gotway via RT [met_help at ucar.edu]
>> > Sent: Wednesday, April 13, 2016 3:48 PM
>> > To: Raby, John W CIV USARMY RDECOM ARL (US)
>> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
>> > running series-analysis
>> >
>> > All active links contained in this email were disabled. Please
verify
>> the
>> > identity of the sender, and confirm the authenticity of all links
>> contained
>> > within the message prior to copying and pasting the address to a
Web
>> > browser.
>> >
>> >
>> >
>> >
>> > ----
>> >
>> > John,
>> >
>> > I don't see any immediately obvious problems in the data you sent
to
>> us. I
>> > see that you're writing a log file named:
>> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
>> >
>> > Are you able to send me that complete log file? That might help
me
>> > diagnose what's going on.
>> >
>> > Here's my initial guess...
>> >
>> > You are running series-analysis by defining a single field in the
>> > configuration file. And you're passing to it lists of 12
forecast files
>> > and 12 observation files.
>> >
>> > It reads TMP/Z2 from the first forecast file and check the valid
time
>> for
>> > that data. Next, it looks in the observation file list (starting
with
>> the
>> > first file) for TMP/Z2 with the *SAME* valid time.
>> >
>> > Next, it reads TMP/Z2 from the second forecast file, gets the
valid
>> time,
>> > and looks in the observation file list (starting with the second
file)
>> for
>> > TMP/Z2 with the *SAME* valid time.
>> >
>> > I looked at the observation file (120382300_laps_288.grb1) you
sent me
>> and
>> > see that it's valid time is 2012020723, which happens to match
the
>> > timestamp in the forecast file name. The first thing I'd check
is the
>> > valid time of the corresponding forecast file:
>> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
>> >
>> > Is it actually valid at the same time, 2012020723? If not,
that's the
>> > problem. If so, that's not the problem.
>> >
>> > Thanks,
>> > John
>> >
>> >
>> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > <Caution-Caution-Caution-url: Caution-
>> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>> > >
>> > > Hi John -
>> > >
>> > > Yes. I got stuck on trying to figure out the cause of that
error.
>> Seems
>> > > like if Grid-Stat can read the 2-m temperature from the gridded
obs,
>> then
>> > > why would Series-Analysis not be able to read it?
>> > >
>> > > Thanks.
>> > >
>> > > R/
>> > > John
>> > >
>> > >
>> > > -----Original Message-----
>> > > From: John Halley Gotway via RT [Caution-Caution-Caution-
mailto:
>> met_help at ucar.edu]
>> > > Sent: Wednesday, April 13, 2016 12:34 PM
>> > > To: Raby, John W CIV USARMY RDECOM ARL (US) <
>> john.w.raby2.civ at mail.mil>
>> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
>> running
>> > > series-analysis
>> > >
>> > > All active links contained in this email were disabled. Please
verify
>> > the
>> > > identity of the sender, and confirm the authenticity of all
links
>> > contained
>> > > within the message prior to copying and pasting the address to
a Web
>> > > browser.
>> > >
>> > >
>> > >
>> > >
>> > > ----
>> > >
>> > > Hello John,
>> > >
>> > > I see that you were having trouble running series-analysis on
2-m
>> > > temperature data last week.
>> > >
>> > > Is this still an issue? If so, I'd be happy to take a look.
>> > >
>> > > Thanks,
>> > > John Halley Gotway
>> > >
>> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT <
>> met_help at ucar.edu
>> > >
>> > > wrote:
>> > >
>> > > >
>> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
>> > > > Transaction: Given to johnhg (John Halley Gotway) by bullock
>> > > > Queue: met_help
>> > > > Subject: problem running series-analysis
>> > > > Owner: johnhg
>> > > > Requestors: john.w.raby2.civ at mail.mil
>> > > > Status: open
>> > > > Ticket <Caution-Caution-Caution-Caution-url:
>> > > > Caution-Caution-
>> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>> > > >
>> > > >
>> > > > This transaction appears to have no content
>> > > >
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>>
>
------------------------------------------------
Subject: problem running series-analysis
From: Raby, John W USA CIV
Time: Thu Apr 14 09:14:37 2016
John -
I resolved the issue with whitespace by using the name "RMSE" vice
"Root mean squared error".
Now it has an issue with the field "RMSE" from the output file. See
attached log.
Thanks.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Wednesday, April 13, 2016 4:57 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running series-analysis
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
Also, I see the following error message in the log file you sent:
ERROR : store_stat_cnt() -> unsupported column name requested "IQR"
IQR was added in version 5.1, but it not defined in version 4.1.
John
On Wed, Apr 13, 2016 at 4:55 PM, John Halley Gotway <johnhg at ucar.edu>
wrote:
> John,
>
> Ah yes, I meant to mention this. Please try increasing the block
size
> from 1024 to something much larger.
>
> Your grid is 288x288 = 82944 grid points. Try setting:
> block_size = 100000;
>
> That will process it all in one pass through the data and should
make it
> run much faster.
>
> John
>
> On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>>
>> John -
>>
>> I looked at your config file and it looks like a nice workaround. I
reran
>> using your config file. Ran considerably longer this time. I also
have a
>> 14MB output NetCDF file which is definitely encouraging! I have to
leave
>> for the day in about 10 mins and will be back in touch tomorrow.
Thanks for
>> your help.
>>
>> R/
>> John
>>
>>
>>
>>
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Wednesday, April 13, 2016 4:37 PM
>> To: Raby, John W CIV USARMY RDECOM ARL (US)
>> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
>> running series-analysis
>>
>> All active links contained in this email were disabled. Please
verify
>> the identity of the sender, and confirm the authenticity of all
links
>> contained within the message prior to copying and pasting the
address to a
>> Web browser.
>>
>>
>>
>>
>> ----
>>
>> John,
>>
>> Ah yes, that time offset is the problem here.
>>
>> And it would have been nice for Series-Analysis to be more verbose
about
>> the timing information it's processing.
>>
>> Please try using the updated config file that I've attached, and
see if
>> Series-Analysis actually runs.
>>
>> We pass grid-stat 1 forecast file and 1 observation file. So even
if the
>> times don't match up exactly, it writes a warning and compares
them.
>> Series-Analysis tries to be a bit fancier. You pass it N forecast
and N
>> observation files. For each forecast files, it searches all the
>> observation files for a match.
>>
>> But that more complex logic breaks down when there's a 15 minute
time
>> offset. So we should think about how better to handle this use
case.
>>
>> Thanks,
>> John
>>
>> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <Caution-Caution-url: Caution-
>> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>> >
>> > John -
>> >
>> > Based on your analysis of the reading of the input files, I can
offer a
>> > possibility that I'm aware of regarding the valid times.The hours
for
>> the
>> > example input data (observations) I sent you is indeed 23 which
matches
>> > that of the forecast data valid at the same hour (23), but the
minutes
>> of
>> > the forecast file is 00 while for the observations it is 15. I
was
>> aware of
>> > this mismatch in the minutes from when I ran Grid-Stat and it
flagged
>> it as
>> > a "Warning", but it went ahead and generated output with no
Error. The
>> > presence of the min value of 15 is an artifact of the fact that
this
>> data
>> > was produced by the LAPS system. The valid time of the data is
actually
>> > 2300 instead of 2315. That said, I will run wgrib on the forecast
file
>> per
>> > your suggestion to confirm the valid time.
>> > R/
>> > John
>> >
>> > ________________________________________
>> > From: John Halley Gotway via RT [met_help at ucar.edu]
>> > Sent: Wednesday, April 13, 2016 3:48 PM
>> > To: Raby, John W CIV USARMY RDECOM ARL (US)
>> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
>> > running series-analysis
>> >
>> > All active links contained in this email were disabled. Please
verify
>> the
>> > identity of the sender, and confirm the authenticity of all links
>> contained
>> > within the message prior to copying and pasting the address to a
Web
>> > browser.
>> >
>> >
>> >
>> >
>> > ----
>> >
>> > John,
>> >
>> > I don't see any immediately obvious problems in the data you sent
to
>> us. I
>> > see that you're writing a log file named:
>> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
>> >
>> > Are you able to send me that complete log file? That might help
me
>> > diagnose what's going on.
>> >
>> > Here's my initial guess...
>> >
>> > You are running series-analysis by defining a single field in the
>> > configuration file. And you're passing to it lists of 12
forecast files
>> > and 12 observation files.
>> >
>> > It reads TMP/Z2 from the first forecast file and check the valid
time
>> for
>> > that data. Next, it looks in the observation file list (starting
with
>> the
>> > first file) for TMP/Z2 with the *SAME* valid time.
>> >
>> > Next, it reads TMP/Z2 from the second forecast file, gets the
valid
>> time,
>> > and looks in the observation file list (starting with the second
file)
>> for
>> > TMP/Z2 with the *SAME* valid time.
>> >
>> > I looked at the observation file (120382300_laps_288.grb1) you
sent me
>> and
>> > see that it's valid time is 2012020723, which happens to match
the
>> > timestamp in the forecast file name. The first thing I'd check
is the
>> > valid time of the corresponding forecast file:
>> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
>> >
>> > Is it actually valid at the same time, 2012020723? If not,
that's the
>> > problem. If so, that's not the problem.
>> >
>> > Thanks,
>> > John
>> >
>> >
>> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > <Caution-Caution-Caution-url: Caution-
>> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>> > >
>> > > Hi John -
>> > >
>> > > Yes. I got stuck on trying to figure out the cause of that
error.
>> Seems
>> > > like if Grid-Stat can read the 2-m temperature from the gridded
obs,
>> then
>> > > why would Series-Analysis not be able to read it?
>> > >
>> > > Thanks.
>> > >
>> > > R/
>> > > John
>> > >
>> > >
>> > > -----Original Message-----
>> > > From: John Halley Gotway via RT [Caution-Caution-Caution-
mailto:
>> met_help at ucar.edu]
>> > > Sent: Wednesday, April 13, 2016 12:34 PM
>> > > To: Raby, John W CIV USARMY RDECOM ARL (US) <
>> john.w.raby2.civ at mail.mil>
>> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
>> running
>> > > series-analysis
>> > >
>> > > All active links contained in this email were disabled. Please
verify
>> > the
>> > > identity of the sender, and confirm the authenticity of all
links
>> > contained
>> > > within the message prior to copying and pasting the address to
a Web
>> > > browser.
>> > >
>> > >
>> > >
>> > >
>> > > ----
>> > >
>> > > Hello John,
>> > >
>> > > I see that you were having trouble running series-analysis on
2-m
>> > > temperature data last week.
>> > >
>> > > Is this still an issue? If so, I'd be happy to take a look.
>> > >
>> > > Thanks,
>> > > John Halley Gotway
>> > >
>> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT <
>> met_help at ucar.edu
>> > >
>> > > wrote:
>> > >
>> > > >
>> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
>> > > > Transaction: Given to johnhg (John Halley Gotway) by bullock
>> > > > Queue: met_help
>> > > > Subject: problem running series-analysis
>> > > > Owner: johnhg
>> > > > Requestors: john.w.raby2.civ at mail.mil
>> > > > Status: open
>> > > > Ticket <Caution-Caution-Caution-Caution-url:
>> > > > Caution-Caution-
>> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>> > > >
>> > > >
>> > > > This transaction appears to have no content
>> > > >
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem running series-analysis
From: Raby, John W USA CIV
Time: Thu Apr 14 09:23:52 2016
John -
I tried plotting all of the other stats/scores which I had specified
for output using the plot_data_plane command I used for RMSE using the
"name" from the ncdump file as the field name, but I received the same
error for all:
"trouble getting field "name="RMSE"; level="Z2";" from file
"results_m3o3_TMP"
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Wednesday, April 13, 2016 4:57 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running series-analysis
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
Also, I see the following error message in the log file you sent:
ERROR : store_stat_cnt() -> unsupported column name requested "IQR"
IQR was added in version 5.1, but it not defined in version 4.1.
John
On Wed, Apr 13, 2016 at 4:55 PM, John Halley Gotway <johnhg at ucar.edu>
wrote:
> John,
>
> Ah yes, I meant to mention this. Please try increasing the block
size
> from 1024 to something much larger.
>
> Your grid is 288x288 = 82944 grid points. Try setting:
> block_size = 100000;
>
> That will process it all in one pass through the data and should
make it
> run much faster.
>
> John
>
> On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
>>
>> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>>
>> John -
>>
>> I looked at your config file and it looks like a nice workaround. I
reran
>> using your config file. Ran considerably longer this time. I also
have a
>> 14MB output NetCDF file which is definitely encouraging! I have to
leave
>> for the day in about 10 mins and will be back in touch tomorrow.
Thanks for
>> your help.
>>
>> R/
>> John
>>
>>
>>
>>
>> ________________________________________
>> From: John Halley Gotway via RT [met_help at ucar.edu]
>> Sent: Wednesday, April 13, 2016 4:37 PM
>> To: Raby, John W CIV USARMY RDECOM ARL (US)
>> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
>> running series-analysis
>>
>> All active links contained in this email were disabled. Please
verify
>> the identity of the sender, and confirm the authenticity of all
links
>> contained within the message prior to copying and pasting the
address to a
>> Web browser.
>>
>>
>>
>>
>> ----
>>
>> John,
>>
>> Ah yes, that time offset is the problem here.
>>
>> And it would have been nice for Series-Analysis to be more verbose
about
>> the timing information it's processing.
>>
>> Please try using the updated config file that I've attached, and
see if
>> Series-Analysis actually runs.
>>
>> We pass grid-stat 1 forecast file and 1 observation file. So even
if the
>> times don't match up exactly, it writes a warning and compares
them.
>> Series-Analysis tries to be a bit fancier. You pass it N forecast
and N
>> observation files. For each forecast files, it searches all the
>> observation files for a match.
>>
>> But that more complex logic breaks down when there's a 15 minute
time
>> offset. So we should think about how better to handle this use
case.
>>
>> Thanks,
>> John
>>
>> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <Caution-Caution-url: Caution-
>> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>> >
>> > John -
>> >
>> > Based on your analysis of the reading of the input files, I can
offer a
>> > possibility that I'm aware of regarding the valid times.The hours
for
>> the
>> > example input data (observations) I sent you is indeed 23 which
matches
>> > that of the forecast data valid at the same hour (23), but the
minutes
>> of
>> > the forecast file is 00 while for the observations it is 15. I
was
>> aware of
>> > this mismatch in the minutes from when I ran Grid-Stat and it
flagged
>> it as
>> > a "Warning", but it went ahead and generated output with no
Error. The
>> > presence of the min value of 15 is an artifact of the fact that
this
>> data
>> > was produced by the LAPS system. The valid time of the data is
actually
>> > 2300 instead of 2315. That said, I will run wgrib on the forecast
file
>> per
>> > your suggestion to confirm the valid time.
>> > R/
>> > John
>> >
>> > ________________________________________
>> > From: John Halley Gotway via RT [met_help at ucar.edu]
>> > Sent: Wednesday, April 13, 2016 3:48 PM
>> > To: Raby, John W CIV USARMY RDECOM ARL (US)
>> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
>> > running series-analysis
>> >
>> > All active links contained in this email were disabled. Please
verify
>> the
>> > identity of the sender, and confirm the authenticity of all links
>> contained
>> > within the message prior to copying and pasting the address to a
Web
>> > browser.
>> >
>> >
>> >
>> >
>> > ----
>> >
>> > John,
>> >
>> > I don't see any immediately obvious problems in the data you sent
to
>> us. I
>> > see that you're writing a log file named:
>> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
>> >
>> > Are you able to send me that complete log file? That might help
me
>> > diagnose what's going on.
>> >
>> > Here's my initial guess...
>> >
>> > You are running series-analysis by defining a single field in the
>> > configuration file. And you're passing to it lists of 12
forecast files
>> > and 12 observation files.
>> >
>> > It reads TMP/Z2 from the first forecast file and check the valid
time
>> for
>> > that data. Next, it looks in the observation file list (starting
with
>> the
>> > first file) for TMP/Z2 with the *SAME* valid time.
>> >
>> > Next, it reads TMP/Z2 from the second forecast file, gets the
valid
>> time,
>> > and looks in the observation file list (starting with the second
file)
>> for
>> > TMP/Z2 with the *SAME* valid time.
>> >
>> > I looked at the observation file (120382300_laps_288.grb1) you
sent me
>> and
>> > see that it's valid time is 2012020723, which happens to match
the
>> > timestamp in the forecast file name. The first thing I'd check
is the
>> > valid time of the corresponding forecast file:
>> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
>> >
>> > Is it actually valid at the same time, 2012020723? If not,
that's the
>> > problem. If so, that's not the problem.
>> >
>> > Thanks,
>> > John
>> >
>> >
>> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > <Caution-Caution-Caution-url: Caution-
>> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>> > >
>> > > Hi John -
>> > >
>> > > Yes. I got stuck on trying to figure out the cause of that
error.
>> Seems
>> > > like if Grid-Stat can read the 2-m temperature from the gridded
obs,
>> then
>> > > why would Series-Analysis not be able to read it?
>> > >
>> > > Thanks.
>> > >
>> > > R/
>> > > John
>> > >
>> > >
>> > > -----Original Message-----
>> > > From: John Halley Gotway via RT [Caution-Caution-Caution-
mailto:
>> met_help at ucar.edu]
>> > > Sent: Wednesday, April 13, 2016 12:34 PM
>> > > To: Raby, John W CIV USARMY RDECOM ARL (US) <
>> john.w.raby2.civ at mail.mil>
>> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
>> running
>> > > series-analysis
>> > >
>> > > All active links contained in this email were disabled. Please
verify
>> > the
>> > > identity of the sender, and confirm the authenticity of all
links
>> > contained
>> > > within the message prior to copying and pasting the address to
a Web
>> > > browser.
>> > >
>> > >
>> > >
>> > >
>> > > ----
>> > >
>> > > Hello John,
>> > >
>> > > I see that you were having trouble running series-analysis on
2-m
>> > > temperature data last week.
>> > >
>> > > Is this still an issue? If so, I'd be happy to take a look.
>> > >
>> > > Thanks,
>> > > John Halley Gotway
>> > >
>> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT <
>> met_help at ucar.edu
>> > >
>> > > wrote:
>> > >
>> > > >
>> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
>> > > > Transaction: Given to johnhg (John Halley Gotway) by bullock
>> > > > Queue: met_help
>> > > > Subject: problem running series-analysis
>> > > > Owner: johnhg
>> > > > Requestors: john.w.raby2.civ at mail.mil
>> > > > Status: open
>> > > > Ticket <Caution-Caution-Caution-Caution-url:
>> > > > Caution-Caution-
>> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>> > > >
>> > > >
>> > > > This transaction appears to have no content
>> > > >
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>>
>
------------------------------------------------
Subject: problem running series-analysis
From: John Halley Gotway
Time: Mon Apr 18 13:41:41 2016
John,
I see the following error in the log file you sent me:
ERROR :
ERROR : plot_data_plane -> trouble getting field "name="RMSE";
level="Z2";" from file "results_m3o3_TMP"
ERROR :
You are calling plot_data_plane by passing it a NetCDF file to plot.
The
way you set "name" and "level" is different between GRIB files and
gridded
NetCDF files. For NetCDF files, you set the name equal to the name of
the
NetCDF variable you want to plot and you set level to "(*,*)". I know
that
looks weird but you're telling MET which NetCDF dimensions are the
gridded
ones that it should plot.
In your case, the output of Series-Analysis will only have 2
dimensions...
thus "(*,*)".
Just use "ncdump -h" to look at the name of the variables and run
plot_data_plane something like this:
plot_data_plane results_m3o3_TMP results_m3o3_TMP_RMSE.ps
'name="series_cnt_RMSE"; level="(*,*)";'
Hope that helps clarify.
Thanks,
John
On Thu, Apr 14, 2016 at 9:14 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> I resolved the issue with whitespace by using the name "RMSE" vice
"Root
> mean squared error".
>
> Now it has an issue with the field "RMSE" from the output file. See
> attached log.
>
> Thanks.
>
> R/
> John
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Wednesday, April 13, 2016 4:57 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> Also, I see the following error message in the log file you sent:
>
> ERROR : store_stat_cnt() -> unsupported column name requested "IQR"
>
> IQR was added in version 5.1, but it not defined in version 4.1.
>
> John
>
> On Wed, Apr 13, 2016 at 4:55 PM, John Halley Gotway
<johnhg at ucar.edu>
> wrote:
>
> > John,
> >
> > Ah yes, I meant to mention this. Please try increasing the block
size
> > from 1024 to something much larger.
> >
> > Your grid is 288x288 = 82944 grid points. Try setting:
> > block_size = 100000;
> >
> > That will process it all in one pass through the data and should
make it
> > run much faster.
> >
> > John
> >
> > On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <Caution-url: Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >>
> >> John -
> >>
> >> I looked at your config file and it looks like a nice workaround.
I
> reran
> >> using your config file. Ran considerably longer this time. I also
have a
> >> 14MB output NetCDF file which is definitely encouraging! I have
to leave
> >> for the day in about 10 mins and will be back in touch tomorrow.
Thanks
> for
> >> your help.
> >>
> >> R/
> >> John
> >>
> >>
> >>
> >>
> >> ________________________________________
> >> From: John Halley Gotway via RT [met_help at ucar.edu]
> >> Sent: Wednesday, April 13, 2016 4:37 PM
> >> To: Raby, John W CIV USARMY RDECOM ARL (US)
> >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> >> running series-analysis
> >>
> >> All active links contained in this email were disabled. Please
verify
> >> the identity of the sender, and confirm the authenticity of all
links
> >> contained within the message prior to copying and pasting the
address
> to a
> >> Web browser.
> >>
> >>
> >>
> >>
> >> ----
> >>
> >> John,
> >>
> >> Ah yes, that time offset is the problem here.
> >>
> >> And it would have been nice for Series-Analysis to be more
verbose about
> >> the timing information it's processing.
> >>
> >> Please try using the updated config file that I've attached, and
see if
> >> Series-Analysis actually runs.
> >>
> >> We pass grid-stat 1 forecast file and 1 observation file. So
even if
> the
> >> times don't match up exactly, it writes a warning and compares
them.
> >> Series-Analysis tries to be a bit fancier. You pass it N
forecast and N
> >> observation files. For each forecast files, it searches all the
> >> observation files for a match.
> >>
> >> But that more complex logic breaks down when there's a 15 minute
time
> >> offset. So we should think about how better to handle this use
case.
> >>
> >> Thanks,
> >> John
> >>
> >> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <Caution-Caution-url: Caution-
> >> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >> >
> >> > John -
> >> >
> >> > Based on your analysis of the reading of the input files, I can
offer
> a
> >> > possibility that I'm aware of regarding the valid times.The
hours for
> >> the
> >> > example input data (observations) I sent you is indeed 23 which
> matches
> >> > that of the forecast data valid at the same hour (23), but the
minutes
> >> of
> >> > the forecast file is 00 while for the observations it is 15. I
was
> >> aware of
> >> > this mismatch in the minutes from when I ran Grid-Stat and it
flagged
> >> it as
> >> > a "Warning", but it went ahead and generated output with no
Error. The
> >> > presence of the min value of 15 is an artifact of the fact
that this
> >> data
> >> > was produced by the LAPS system. The valid time of the data is
> actually
> >> > 2300 instead of 2315. That said, I will run wgrib on the
forecast file
> >> per
> >> > your suggestion to confirm the valid time.
> >> > R/
> >> > John
> >> >
> >> > ________________________________________
> >> > From: John Halley Gotway via RT [met_help at ucar.edu]
> >> > Sent: Wednesday, April 13, 2016 3:48 PM
> >> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> >> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> >> > running series-analysis
> >> >
> >> > All active links contained in this email were disabled. Please
verify
> >> the
> >> > identity of the sender, and confirm the authenticity of all
links
> >> contained
> >> > within the message prior to copying and pasting the address to
a Web
> >> > browser.
> >> >
> >> >
> >> >
> >> >
> >> > ----
> >> >
> >> > John,
> >> >
> >> > I don't see any immediately obvious problems in the data you
sent to
> >> us. I
> >> > see that you're writing a log file named:
> >> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
> >> >
> >> > Are you able to send me that complete log file? That might
help me
> >> > diagnose what's going on.
> >> >
> >> > Here's my initial guess...
> >> >
> >> > You are running series-analysis by defining a single field in
the
> >> > configuration file. And you're passing to it lists of 12
forecast
> files
> >> > and 12 observation files.
> >> >
> >> > It reads TMP/Z2 from the first forecast file and check the
valid time
> >> for
> >> > that data. Next, it looks in the observation file list
(starting with
> >> the
> >> > first file) for TMP/Z2 with the *SAME* valid time.
> >> >
> >> > Next, it reads TMP/Z2 from the second forecast file, gets the
valid
> >> time,
> >> > and looks in the observation file list (starting with the
second file)
> >> for
> >> > TMP/Z2 with the *SAME* valid time.
> >> >
> >> > I looked at the observation file (120382300_laps_288.grb1) you
sent me
> >> and
> >> > see that it's valid time is 2012020723, which happens to match
the
> >> > timestamp in the forecast file name. The first thing I'd check
is the
> >> > valid time of the corresponding forecast file:
> >> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
> >> >
> >> > Is it actually valid at the same time, 2012020723? If not,
that's the
> >> > problem. If so, that's not the problem.
> >> >
> >> > Thanks,
> >> > John
> >> >
> >> >
> >> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > <Caution-Caution-Caution-url: Caution-
> >> > Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >> > >
> >> > > Hi John -
> >> > >
> >> > > Yes. I got stuck on trying to figure out the cause of that
error.
> >> Seems
> >> > > like if Grid-Stat can read the 2-m temperature from the
gridded obs,
> >> then
> >> > > why would Series-Analysis not be able to read it?
> >> > >
> >> > > Thanks.
> >> > >
> >> > > R/
> >> > > John
> >> > >
> >> > >
> >> > > -----Original Message-----
> >> > > From: John Halley Gotway via RT [Caution-Caution-Caution-
mailto:
> >> met_help at ucar.edu]
> >> > > Sent: Wednesday, April 13, 2016 12:34 PM
> >> > > To: Raby, John W CIV USARMY RDECOM ARL (US) <
> >> john.w.raby2.civ at mail.mil>
> >> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> >> running
> >> > > series-analysis
> >> > >
> >> > > All active links contained in this email were disabled.
Please
> verify
> >> > the
> >> > > identity of the sender, and confirm the authenticity of all
links
> >> > contained
> >> > > within the message prior to copying and pasting the address
to a Web
> >> > > browser.
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > ----
> >> > >
> >> > > Hello John,
> >> > >
> >> > > I see that you were having trouble running series-analysis on
2-m
> >> > > temperature data last week.
> >> > >
> >> > > Is this still an issue? If so, I'd be happy to take a look.
> >> > >
> >> > > Thanks,
> >> > > John Halley Gotway
> >> > >
> >> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT <
> >> met_help at ucar.edu
> >> > >
> >> > > wrote:
> >> > >
> >> > > >
> >> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
> >> > > > Transaction: Given to johnhg (John Halley Gotway) by
bullock
> >> > > > Queue: met_help
> >> > > > Subject: problem running series-analysis
> >> > > > Owner: johnhg
> >> > > > Requestors: john.w.raby2.civ at mail.mil
> >> > > > Status: open
> >> > > > Ticket <Caution-Caution-Caution-Caution-url:
> >> > > > Caution-Caution-
> >> > Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >> > > >
> >> > > >
> >> > > > This transaction appears to have no content
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> >
> >> >
> >> >
> >> >
> >>
> >>
> >>
> >
>
>
>
------------------------------------------------
Subject: problem running series-analysis
From: Raby, John W USA CIV
Time: Mon Apr 18 15:23:34 2016
John -
The attached plot was the result of using your command. No errors!
The plot doesn't look right.
Any thoughts?
Thanks.
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Monday, April 18, 2016 1:41 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running series-analysis
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
I see the following error in the log file you sent me:
ERROR :
ERROR : plot_data_plane -> trouble getting field "name="RMSE";
level="Z2";" from file "results_m3o3_TMP"
ERROR :
You are calling plot_data_plane by passing it a NetCDF file to plot.
The
way you set "name" and "level" is different between GRIB files and
gridded
NetCDF files. For NetCDF files, you set the name equal to the name of
the
NetCDF variable you want to plot and you set level to "(*,*)". I know
that
looks weird but you're telling MET which NetCDF dimensions are the
gridded
ones that it should plot.
In your case, the output of Series-Analysis will only have 2
dimensions...
thus "(*,*)".
Just use "ncdump -h" to look at the name of the variables and run
plot_data_plane something like this:
plot_data_plane results_m3o3_TMP results_m3o3_TMP_RMSE.ps
'name="series_cnt_RMSE"; level="(*,*)";'
Hope that helps clarify.
Thanks,
John
On Thu, Apr 14, 2016 at 9:14 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> I resolved the issue with whitespace by using the name "RMSE" vice
"Root
> mean squared error".
>
> Now it has an issue with the field "RMSE" from the output file. See
> attached log.
>
> Thanks.
>
> R/
> John
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Wednesday, April 13, 2016 4:57 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> Also, I see the following error message in the log file you sent:
>
> ERROR : store_stat_cnt() -> unsupported column name requested "IQR"
>
> IQR was added in version 5.1, but it not defined in version 4.1.
>
> John
>
> On Wed, Apr 13, 2016 at 4:55 PM, John Halley Gotway
<johnhg at ucar.edu>
> wrote:
>
> > John,
> >
> > Ah yes, I meant to mention this. Please try increasing the block
size
> > from 1024 to something much larger.
> >
> > Your grid is 288x288 = 82944 grid points. Try setting:
> > block_size = 100000;
> >
> > That will process it all in one pass through the data and should
make it
> > run much faster.
> >
> > John
> >
> > On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <Caution-Caution-url: Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >>
> >> John -
> >>
> >> I looked at your config file and it looks like a nice workaround.
I
> reran
> >> using your config file. Ran considerably longer this time. I also
have a
> >> 14MB output NetCDF file which is definitely encouraging! I have
to leave
> >> for the day in about 10 mins and will be back in touch tomorrow.
Thanks
> for
> >> your help.
> >>
> >> R/
> >> John
> >>
> >>
> >>
> >>
> >> ________________________________________
> >> From: John Halley Gotway via RT [met_help at ucar.edu]
> >> Sent: Wednesday, April 13, 2016 4:37 PM
> >> To: Raby, John W CIV USARMY RDECOM ARL (US)
> >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> >> running series-analysis
> >>
> >> All active links contained in this email were disabled. Please
verify
> >> the identity of the sender, and confirm the authenticity of all
links
> >> contained within the message prior to copying and pasting the
address
> to a
> >> Web browser.
> >>
> >>
> >>
> >>
> >> ----
> >>
> >> John,
> >>
> >> Ah yes, that time offset is the problem here.
> >>
> >> And it would have been nice for Series-Analysis to be more
verbose about
> >> the timing information it's processing.
> >>
> >> Please try using the updated config file that I've attached, and
see if
> >> Series-Analysis actually runs.
> >>
> >> We pass grid-stat 1 forecast file and 1 observation file. So
even if
> the
> >> times don't match up exactly, it writes a warning and compares
them.
> >> Series-Analysis tries to be a bit fancier. You pass it N
forecast and N
> >> observation files. For each forecast files, it searches all the
> >> observation files for a match.
> >>
> >> But that more complex logic breaks down when there's a 15 minute
time
> >> offset. So we should think about how better to handle this use
case.
> >>
> >> Thanks,
> >> John
> >>
> >> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <Caution-Caution-Caution-url: Caution-
> >> Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >> >
> >> > John -
> >> >
> >> > Based on your analysis of the reading of the input files, I can
offer
> a
> >> > possibility that I'm aware of regarding the valid times.The
hours for
> >> the
> >> > example input data (observations) I sent you is indeed 23 which
> matches
> >> > that of the forecast data valid at the same hour (23), but the
minutes
> >> of
> >> > the forecast file is 00 while for the observations it is 15. I
was
> >> aware of
> >> > this mismatch in the minutes from when I ran Grid-Stat and it
flagged
> >> it as
> >> > a "Warning", but it went ahead and generated output with no
Error. The
> >> > presence of the min value of 15 is an artifact of the fact
that this
> >> data
> >> > was produced by the LAPS system. The valid time of the data is
> actually
> >> > 2300 instead of 2315. That said, I will run wgrib on the
forecast file
> >> per
> >> > your suggestion to confirm the valid time.
> >> > R/
> >> > John
> >> >
> >> > ________________________________________
> >> > From: John Halley Gotway via RT [met_help at ucar.edu]
> >> > Sent: Wednesday, April 13, 2016 3:48 PM
> >> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> >> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> >> > running series-analysis
> >> >
> >> > All active links contained in this email were disabled. Please
verify
> >> the
> >> > identity of the sender, and confirm the authenticity of all
links
> >> contained
> >> > within the message prior to copying and pasting the address to
a Web
> >> > browser.
> >> >
> >> >
> >> >
> >> >
> >> > ----
> >> >
> >> > John,
> >> >
> >> > I don't see any immediately obvious problems in the data you
sent to
> >> us. I
> >> > see that you're writing a log file named:
> >> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
> >> >
> >> > Are you able to send me that complete log file? That might
help me
> >> > diagnose what's going on.
> >> >
> >> > Here's my initial guess...
> >> >
> >> > You are running series-analysis by defining a single field in
the
> >> > configuration file. And you're passing to it lists of 12
forecast
> files
> >> > and 12 observation files.
> >> >
> >> > It reads TMP/Z2 from the first forecast file and check the
valid time
> >> for
> >> > that data. Next, it looks in the observation file list
(starting with
> >> the
> >> > first file) for TMP/Z2 with the *SAME* valid time.
> >> >
> >> > Next, it reads TMP/Z2 from the second forecast file, gets the
valid
> >> time,
> >> > and looks in the observation file list (starting with the
second file)
> >> for
> >> > TMP/Z2 with the *SAME* valid time.
> >> >
> >> > I looked at the observation file (120382300_laps_288.grb1) you
sent me
> >> and
> >> > see that it's valid time is 2012020723, which happens to match
the
> >> > timestamp in the forecast file name. The first thing I'd check
is the
> >> > valid time of the corresponding forecast file:
> >> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
> >> >
> >> > Is it actually valid at the same time, 2012020723? If not,
that's the
> >> > problem. If so, that's not the problem.
> >> >
> >> > Thanks,
> >> > John
> >> >
> >> >
> >> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > <Caution-Caution-Caution-Caution-url: Caution-
> >> > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >> > >
> >> > > Hi John -
> >> > >
> >> > > Yes. I got stuck on trying to figure out the cause of that
error.
> >> Seems
> >> > > like if Grid-Stat can read the 2-m temperature from the
gridded obs,
> >> then
> >> > > why would Series-Analysis not be able to read it?
> >> > >
> >> > > Thanks.
> >> > >
> >> > > R/
> >> > > John
> >> > >
> >> > >
> >> > > -----Original Message-----
> >> > > From: John Halley Gotway via RT [Caution-Caution-Caution-
Caution-mailto:
> >> met_help at ucar.edu]
> >> > > Sent: Wednesday, April 13, 2016 12:34 PM
> >> > > To: Raby, John W CIV USARMY RDECOM ARL (US) <
> >> john.w.raby2.civ at mail.mil>
> >> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> >> running
> >> > > series-analysis
> >> > >
> >> > > All active links contained in this email were disabled.
Please
> verify
> >> > the
> >> > > identity of the sender, and confirm the authenticity of all
links
> >> > contained
> >> > > within the message prior to copying and pasting the address
to a Web
> >> > > browser.
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > ----
> >> > >
> >> > > Hello John,
> >> > >
> >> > > I see that you were having trouble running series-analysis on
2-m
> >> > > temperature data last week.
> >> > >
> >> > > Is this still an issue? If so, I'd be happy to take a look.
> >> > >
> >> > > Thanks,
> >> > > John Halley Gotway
> >> > >
> >> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT <
> >> met_help at ucar.edu
> >> > >
> >> > > wrote:
> >> > >
> >> > > >
> >> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
> >> > > > Transaction: Given to johnhg (John Halley Gotway) by
bullock
> >> > > > Queue: met_help
> >> > > > Subject: problem running series-analysis
> >> > > > Owner: johnhg
> >> > > > Requestors: john.w.raby2.civ at mail.mil
> >> > > > Status: open
> >> > > > Ticket <Caution-Caution-Caution-Caution-Caution-url:
> >> > > > Caution-Caution-
> >> > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >> > > >
> >> > > >
> >> > > > This transaction appears to have no content
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> >
> >> >
> >> >
> >> >
> >>
> >>
> >>
> >
>
>
>
------------------------------------------------
Subject: problem running series-analysis
From: Raby, John W USA CIV
Time: Mon Apr 18 16:13:32 2016
John -
I used the following command to plot the variable:
series_cts_CSI_ge285.000
carson:jraby ~/MET_SeriesAnalysis
> plot_data_plane results_m3o3_TMP results_m3o3_TMP_CSI.ps
'name="series_cts_CSI_ge285.000"; level="(*,*)";'
DEBUG 1: Opening data file: results_m3o3_TMP
DEBUG 1: Creating postscript file: results_m3o3_TMP_CSI.ps
carson:jraby ~/MET_SeriesAnalysis
Seem to run fine, but the plot doesn't look like I expected.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Monday, April 18, 2016 1:41 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running series-analysis
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
I see the following error in the log file you sent me:
ERROR :
ERROR : plot_data_plane -> trouble getting field "name="RMSE";
level="Z2";" from file "results_m3o3_TMP"
ERROR :
You are calling plot_data_plane by passing it a NetCDF file to plot.
The
way you set "name" and "level" is different between GRIB files and
gridded
NetCDF files. For NetCDF files, you set the name equal to the name of
the
NetCDF variable you want to plot and you set level to "(*,*)". I know
that
looks weird but you're telling MET which NetCDF dimensions are the
gridded
ones that it should plot.
In your case, the output of Series-Analysis will only have 2
dimensions...
thus "(*,*)".
Just use "ncdump -h" to look at the name of the variables and run
plot_data_plane something like this:
plot_data_plane results_m3o3_TMP results_m3o3_TMP_RMSE.ps
'name="series_cnt_RMSE"; level="(*,*)";'
Hope that helps clarify.
Thanks,
John
On Thu, Apr 14, 2016 at 9:14 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> I resolved the issue with whitespace by using the name "RMSE" vice
"Root
> mean squared error".
>
> Now it has an issue with the field "RMSE" from the output file. See
> attached log.
>
> Thanks.
>
> R/
> John
>
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Wednesday, April 13, 2016 4:57 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> Also, I see the following error message in the log file you sent:
>
> ERROR : store_stat_cnt() -> unsupported column name requested "IQR"
>
> IQR was added in version 5.1, but it not defined in version 4.1.
>
> John
>
> On Wed, Apr 13, 2016 at 4:55 PM, John Halley Gotway
<johnhg at ucar.edu>
> wrote:
>
> > John,
> >
> > Ah yes, I meant to mention this. Please try increasing the block
size
> > from 1024 to something much larger.
> >
> > Your grid is 288x288 = 82944 grid points. Try setting:
> > block_size = 100000;
> >
> > That will process it all in one pass through the data and should
make it
> > run much faster.
> >
> > John
> >
> > On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <Caution-Caution-url: Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >>
> >> John -
> >>
> >> I looked at your config file and it looks like a nice workaround.
I
> reran
> >> using your config file. Ran considerably longer this time. I also
have a
> >> 14MB output NetCDF file which is definitely encouraging! I have
to leave
> >> for the day in about 10 mins and will be back in touch tomorrow.
Thanks
> for
> >> your help.
> >>
> >> R/
> >> John
> >>
> >>
> >>
> >>
> >> ________________________________________
> >> From: John Halley Gotway via RT [met_help at ucar.edu]
> >> Sent: Wednesday, April 13, 2016 4:37 PM
> >> To: Raby, John W CIV USARMY RDECOM ARL (US)
> >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> >> running series-analysis
> >>
> >> All active links contained in this email were disabled. Please
verify
> >> the identity of the sender, and confirm the authenticity of all
links
> >> contained within the message prior to copying and pasting the
address
> to a
> >> Web browser.
> >>
> >>
> >>
> >>
> >> ----
> >>
> >> John,
> >>
> >> Ah yes, that time offset is the problem here.
> >>
> >> And it would have been nice for Series-Analysis to be more
verbose about
> >> the timing information it's processing.
> >>
> >> Please try using the updated config file that I've attached, and
see if
> >> Series-Analysis actually runs.
> >>
> >> We pass grid-stat 1 forecast file and 1 observation file. So
even if
> the
> >> times don't match up exactly, it writes a warning and compares
them.
> >> Series-Analysis tries to be a bit fancier. You pass it N
forecast and N
> >> observation files. For each forecast files, it searches all the
> >> observation files for a match.
> >>
> >> But that more complex logic breaks down when there's a 15 minute
time
> >> offset. So we should think about how better to handle this use
case.
> >>
> >> Thanks,
> >> John
> >>
> >> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <Caution-Caution-Caution-url: Caution-
> >> Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >> >
> >> > John -
> >> >
> >> > Based on your analysis of the reading of the input files, I can
offer
> a
> >> > possibility that I'm aware of regarding the valid times.The
hours for
> >> the
> >> > example input data (observations) I sent you is indeed 23 which
> matches
> >> > that of the forecast data valid at the same hour (23), but the
minutes
> >> of
> >> > the forecast file is 00 while for the observations it is 15. I
was
> >> aware of
> >> > this mismatch in the minutes from when I ran Grid-Stat and it
flagged
> >> it as
> >> > a "Warning", but it went ahead and generated output with no
Error. The
> >> > presence of the min value of 15 is an artifact of the fact
that this
> >> data
> >> > was produced by the LAPS system. The valid time of the data is
> actually
> >> > 2300 instead of 2315. That said, I will run wgrib on the
forecast file
> >> per
> >> > your suggestion to confirm the valid time.
> >> > R/
> >> > John
> >> >
> >> > ________________________________________
> >> > From: John Halley Gotway via RT [met_help at ucar.edu]
> >> > Sent: Wednesday, April 13, 2016 3:48 PM
> >> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> >> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> >> > running series-analysis
> >> >
> >> > All active links contained in this email were disabled. Please
verify
> >> the
> >> > identity of the sender, and confirm the authenticity of all
links
> >> contained
> >> > within the message prior to copying and pasting the address to
a Web
> >> > browser.
> >> >
> >> >
> >> >
> >> >
> >> > ----
> >> >
> >> > John,
> >> >
> >> > I don't see any immediately obvious problems in the data you
sent to
> >> us. I
> >> > see that you're writing a log file named:
> >> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
> >> >
> >> > Are you able to send me that complete log file? That might
help me
> >> > diagnose what's going on.
> >> >
> >> > Here's my initial guess...
> >> >
> >> > You are running series-analysis by defining a single field in
the
> >> > configuration file. And you're passing to it lists of 12
forecast
> files
> >> > and 12 observation files.
> >> >
> >> > It reads TMP/Z2 from the first forecast file and check the
valid time
> >> for
> >> > that data. Next, it looks in the observation file list
(starting with
> >> the
> >> > first file) for TMP/Z2 with the *SAME* valid time.
> >> >
> >> > Next, it reads TMP/Z2 from the second forecast file, gets the
valid
> >> time,
> >> > and looks in the observation file list (starting with the
second file)
> >> for
> >> > TMP/Z2 with the *SAME* valid time.
> >> >
> >> > I looked at the observation file (120382300_laps_288.grb1) you
sent me
> >> and
> >> > see that it's valid time is 2012020723, which happens to match
the
> >> > timestamp in the forecast file name. The first thing I'd check
is the
> >> > valid time of the corresponding forecast file:
> >> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
> >> >
> >> > Is it actually valid at the same time, 2012020723? If not,
that's the
> >> > problem. If so, that's not the problem.
> >> >
> >> > Thanks,
> >> > John
> >> >
> >> >
> >> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > <Caution-Caution-Caution-Caution-url: Caution-
> >> > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >> > >
> >> > > Hi John -
> >> > >
> >> > > Yes. I got stuck on trying to figure out the cause of that
error.
> >> Seems
> >> > > like if Grid-Stat can read the 2-m temperature from the
gridded obs,
> >> then
> >> > > why would Series-Analysis not be able to read it?
> >> > >
> >> > > Thanks.
> >> > >
> >> > > R/
> >> > > John
> >> > >
> >> > >
> >> > > -----Original Message-----
> >> > > From: John Halley Gotway via RT [Caution-Caution-Caution-
Caution-mailto:
> >> met_help at ucar.edu]
> >> > > Sent: Wednesday, April 13, 2016 12:34 PM
> >> > > To: Raby, John W CIV USARMY RDECOM ARL (US) <
> >> john.w.raby2.civ at mail.mil>
> >> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> >> running
> >> > > series-analysis
> >> > >
> >> > > All active links contained in this email were disabled.
Please
> verify
> >> > the
> >> > > identity of the sender, and confirm the authenticity of all
links
> >> > contained
> >> > > within the message prior to copying and pasting the address
to a Web
> >> > > browser.
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > ----
> >> > >
> >> > > Hello John,
> >> > >
> >> > > I see that you were having trouble running series-analysis on
2-m
> >> > > temperature data last week.
> >> > >
> >> > > Is this still an issue? If so, I'd be happy to take a look.
> >> > >
> >> > > Thanks,
> >> > > John Halley Gotway
> >> > >
> >> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT <
> >> met_help at ucar.edu
> >> > >
> >> > > wrote:
> >> > >
> >> > > >
> >> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
> >> > > > Transaction: Given to johnhg (John Halley Gotway) by
bullock
> >> > > > Queue: met_help
> >> > > > Subject: problem running series-analysis
> >> > > > Owner: johnhg
> >> > > > Requestors: john.w.raby2.civ at mail.mil
> >> > > > Status: open
> >> > > > Ticket <Caution-Caution-Caution-Caution-Caution-url:
> >> > > > Caution-Caution-
> >> > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >> > > >
> >> > > >
> >> > > > This transaction appears to have no content
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> >
> >> >
> >> >
> >> >
> >>
> >>
> >>
> >
>
>
>
------------------------------------------------
Subject: problem running series-analysis
From: John Halley Gotway
Time: Tue Apr 19 09:45:48 2016
John,
I agree with you. That output doesn't look good at all!
Please just open up the NetCDF output file from Series-Analysis using
the
"ncview" tool. Click through the variables and check whether any of
them
look reasonable. If they look OK, then the problem is in plotting
them
with plot_data_plane. If so, please just post that Series-Analysis
output
file to our anonymous ftp site? And I'll figure out what's going on
with
the plotting.
But my guess is that none of the fields look good, which means the
issue is
coming from Series-Analysis. Please try rerunning your Series-
Analysis
command at verbosity level 5 (-v 5) and then save the output to a log
file
(-log run_series_analysis.log). Then send me that log file.
Thanks,
John
On Mon, Apr 18, 2016 at 4:13 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> I used the following command to plot the variable:
series_cts_CSI_ge285.000
> carson:jraby ~/MET_SeriesAnalysis
> > plot_data_plane results_m3o3_TMP results_m3o3_TMP_CSI.ps
> 'name="series_cts_CSI_ge285.000"; level="(*,*)";'
> DEBUG 1: Opening data file: results_m3o3_TMP
> DEBUG 1: Creating postscript file: results_m3o3_TMP_CSI.ps
> carson:jraby ~/MET_SeriesAnalysis
>
> Seem to run fine, but the plot doesn't look like I expected.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Monday, April 18, 2016 1:41 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> I see the following error in the log file you sent me:
> ERROR :
> ERROR : plot_data_plane -> trouble getting field "name="RMSE";
> level="Z2";" from file "results_m3o3_TMP"
> ERROR :
>
> You are calling plot_data_plane by passing it a NetCDF file to plot.
The
> way you set "name" and "level" is different between GRIB files and
gridded
> NetCDF files. For NetCDF files, you set the name equal to the name
of the
> NetCDF variable you want to plot and you set level to "(*,*)". I
know that
> looks weird but you're telling MET which NetCDF dimensions are the
gridded
> ones that it should plot.
>
> In your case, the output of Series-Analysis will only have 2
dimensions...
> thus "(*,*)".
>
> Just use "ncdump -h" to look at the name of the variables and run
> plot_data_plane something like this:
>
> plot_data_plane results_m3o3_TMP results_m3o3_TMP_RMSE.ps
> 'name="series_cnt_RMSE"; level="(*,*)";'
>
> Hope that helps clarify.
>
> Thanks,
> John
>
> On Thu, Apr 14, 2016 at 9:14 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url: Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> > John -
> >
> > I resolved the issue with whitespace by using the name "RMSE" vice
"Root
> > mean squared error".
> >
> > Now it has an issue with the field "RMSE" from the output file.
See
> > attached log.
> >
> > Thanks.
> >
> > R/
> > John
> >
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Wednesday, April 13, 2016 4:57 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> > running series-analysis
> >
> > All active links contained in this email were disabled. Please
verify
> the
> > identity of the sender, and confirm the authenticity of all links
> contained
> > within the message prior to copying and pasting the address to a
Web
> > browser.
> >
> >
> >
> >
> > ----
> >
> > Also, I see the following error message in the log file you sent:
> >
> > ERROR : store_stat_cnt() -> unsupported column name requested
"IQR"
> >
> > IQR was added in version 5.1, but it not defined in version 4.1.
> >
> > John
> >
> > On Wed, Apr 13, 2016 at 4:55 PM, John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> >
> > > John,
> > >
> > > Ah yes, I meant to mention this. Please try increasing the
block size
> > > from 1024 to something much larger.
> > >
> > > Your grid is 288x288 = 82944 grid points. Try setting:
> > > block_size = 100000;
> > >
> > > That will process it all in one pass through the data and should
make
> it
> > > run much faster.
> > >
> > > John
> > >
> > > On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> <Caution-Caution-url: Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >>
> > >> John -
> > >>
> > >> I looked at your config file and it looks like a nice
workaround. I
> > reran
> > >> using your config file. Ran considerably longer this time. I
also
> have a
> > >> 14MB output NetCDF file which is definitely encouraging! I have
to
> leave
> > >> for the day in about 10 mins and will be back in touch
tomorrow.
> Thanks
> > for
> > >> your help.
> > >>
> > >> R/
> > >> John
> > >>
> > >>
> > >>
> > >>
> > >> ________________________________________
> > >> From: John Halley Gotway via RT [met_help at ucar.edu]
> > >> Sent: Wednesday, April 13, 2016 4:37 PM
> > >> To: Raby, John W CIV USARMY RDECOM ARL (US)
> > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > >> running series-analysis
> > >>
> > >> All active links contained in this email were disabled. Please
verify
> > >> the identity of the sender, and confirm the authenticity of all
links
> > >> contained within the message prior to copying and pasting the
address
> > to a
> > >> Web browser.
> > >>
> > >>
> > >>
> > >>
> > >> ----
> > >>
> > >> John,
> > >>
> > >> Ah yes, that time offset is the problem here.
> > >>
> > >> And it would have been nice for Series-Analysis to be more
verbose
> about
> > >> the timing information it's processing.
> > >>
> > >> Please try using the updated config file that I've attached,
and see
> if
> > >> Series-Analysis actually runs.
> > >>
> > >> We pass grid-stat 1 forecast file and 1 observation file. So
even if
> > the
> > >> times don't match up exactly, it writes a warning and compares
them.
> > >> Series-Analysis tries to be a bit fancier. You pass it N
forecast
> and N
> > >> observation files. For each forecast files, it searches all
the
> > >> observation files for a match.
> > >>
> > >> But that more complex logic breaks down when there's a 15
minute time
> > >> offset. So we should think about how better to handle this use
case.
> > >>
> > >> Thanks,
> > >> John
> > >>
> > >> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via RT <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >> >
> > >> > <Caution-Caution-Caution-url: Caution-
> > >> Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >> >
> > >> > John -
> > >> >
> > >> > Based on your analysis of the reading of the input files, I
can
> offer
> > a
> > >> > possibility that I'm aware of regarding the valid times.The
hours
> for
> > >> the
> > >> > example input data (observations) I sent you is indeed 23
which
> > matches
> > >> > that of the forecast data valid at the same hour (23), but
the
> minutes
> > >> of
> > >> > the forecast file is 00 while for the observations it is 15.
I was
> > >> aware of
> > >> > this mismatch in the minutes from when I ran Grid-Stat and it
> flagged
> > >> it as
> > >> > a "Warning", but it went ahead and generated output with no
Error.
> The
> > >> > presence of the min value of 15 is an artifact of the fact
that
> this
> > >> data
> > >> > was produced by the LAPS system. The valid time of the data
is
> > actually
> > >> > 2300 instead of 2315. That said, I will run wgrib on the
forecast
> file
> > >> per
> > >> > your suggestion to confirm the valid time.
> > >> > R/
> > >> > John
> > >> >
> > >> > ________________________________________
> > >> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > >> > Sent: Wednesday, April 13, 2016 3:48 PM
> > >> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > >> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > >> > running series-analysis
> > >> >
> > >> > All active links contained in this email were disabled.
Please
> verify
> > >> the
> > >> > identity of the sender, and confirm the authenticity of all
links
> > >> contained
> > >> > within the message prior to copying and pasting the address
to a Web
> > >> > browser.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > ----
> > >> >
> > >> > John,
> > >> >
> > >> > I don't see any immediately obvious problems in the data you
sent to
> > >> us. I
> > >> > see that you're writing a log file named:
> > >> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
> > >> >
> > >> > Are you able to send me that complete log file? That might
help me
> > >> > diagnose what's going on.
> > >> >
> > >> > Here's my initial guess...
> > >> >
> > >> > You are running series-analysis by defining a single field in
the
> > >> > configuration file. And you're passing to it lists of 12
forecast
> > files
> > >> > and 12 observation files.
> > >> >
> > >> > It reads TMP/Z2 from the first forecast file and check the
valid
> time
> > >> for
> > >> > that data. Next, it looks in the observation file list
(starting
> with
> > >> the
> > >> > first file) for TMP/Z2 with the *SAME* valid time.
> > >> >
> > >> > Next, it reads TMP/Z2 from the second forecast file, gets the
valid
> > >> time,
> > >> > and looks in the observation file list (starting with the
second
> file)
> > >> for
> > >> > TMP/Z2 with the *SAME* valid time.
> > >> >
> > >> > I looked at the observation file (120382300_laps_288.grb1)
you sent
> me
> > >> and
> > >> > see that it's valid time is 2012020723, which happens to
match the
> > >> > timestamp in the forecast file name. The first thing I'd
check is
> the
> > >> > valid time of the corresponding forecast file:
> > >> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
> > >> >
> > >> > Is it actually valid at the same time, 2012020723? If not,
that's
> the
> > >> > problem. If so, that's not the problem.
> > >> >
> > >> > Thanks,
> > >> > John
> > >> >
> > >> >
> > >> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via RT
<
> > >> > met_help at ucar.edu> wrote:
> > >> >
> > >> > >
> > >> > > <Caution-Caution-Caution-Caution-url: Caution-
> > >> > Caution-Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >> > >
> > >> > > Hi John -
> > >> > >
> > >> > > Yes. I got stuck on trying to figure out the cause of that
error.
> > >> Seems
> > >> > > like if Grid-Stat can read the 2-m temperature from the
gridded
> obs,
> > >> then
> > >> > > why would Series-Analysis not be able to read it?
> > >> > >
> > >> > > Thanks.
> > >> > >
> > >> > > R/
> > >> > > John
> > >> > >
> > >> > >
> > >> > > -----Original Message-----
> > >> > > From: John Halley Gotway via RT
> [Caution-Caution-Caution-Caution-mailto:
> > >> met_help at ucar.edu]
> > >> > > Sent: Wednesday, April 13, 2016 12:34 PM
> > >> > > To: Raby, John W CIV USARMY RDECOM ARL (US) <
> > >> john.w.raby2.civ at mail.mil>
> > >> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > >> running
> > >> > > series-analysis
> > >> > >
> > >> > > All active links contained in this email were disabled.
Please
> > verify
> > >> > the
> > >> > > identity of the sender, and confirm the authenticity of all
links
> > >> > contained
> > >> > > within the message prior to copying and pasting the address
to a
> Web
> > >> > > browser.
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > ----
> > >> > >
> > >> > > Hello John,
> > >> > >
> > >> > > I see that you were having trouble running series-analysis
on 2-m
> > >> > > temperature data last week.
> > >> > >
> > >> > > Is this still an issue? If so, I'd be happy to take a
look.
> > >> > >
> > >> > > Thanks,
> > >> > > John Halley Gotway
> > >> > >
> > >> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT <
> > >> met_help at ucar.edu
> > >> > >
> > >> > > wrote:
> > >> > >
> > >> > > >
> > >> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
> > >> > > > Transaction: Given to johnhg (John Halley Gotway) by
bullock
> > >> > > > Queue: met_help
> > >> > > > Subject: problem running series-analysis
> > >> > > > Owner: johnhg
> > >> > > > Requestors: john.w.raby2.civ at mail.mil
> > >> > > > Status: open
> > >> > > > Ticket <Caution-Caution-Caution-Caution-Caution-url:
> > >> > > > Caution-Caution-
> > >> > Caution-Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >> > > >
> > >> > > >
> > >> > > > This transaction appears to have no content
> > >> > > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> >
> > >>
> > >>
> > >>
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem running series-analysis
From: Raby, John W USA CIV
Time: Tue Apr 19 11:51:05 2016
John -
Thanks for taking a look at that. For large sized files, my only
method of transfer to you is by the ARL Secure Access File Exchange
(SAFE) which requires an email address to which they send you a link
to download the file. FTP is blocked here at WSMR. In the past, I have
used your email address @ucar.edu. Will this still work? If so, do you
also want the forecast file which has the same valid time (almost-
2300Z) as the observations file I sent you which was valid at 2315Z on
07 FEB 2012?
I don't have ncview running here. I did try visualizing the output
file using the Unidata IDV software and none of the fields looked
right to me.
I'll start working on a re-run of Series-Analysis to generate the log
per your instructions.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Tuesday, April 19, 2016 9:45 AM
To: Raby, John W CIV USARMY RDECOM ARL (US)
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running series-analysis
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
I agree with you. That output doesn't look good at all!
Please just open up the NetCDF output file from Series-Analysis using
the
"ncview" tool. Click through the variables and check whether any of
them
look reasonable. If they look OK, then the problem is in plotting
them
with plot_data_plane. If so, please just post that Series-Analysis
output
file to our anonymous ftp site? And I'll figure out what's going on
with
the plotting.
But my guess is that none of the fields look good, which means the
issue is
coming from Series-Analysis. Please try rerunning your Series-
Analysis
command at verbosity level 5 (-v 5) and then save the output to a log
file
(-log run_series_analysis.log). Then send me that log file.
Thanks,
John
On Mon, Apr 18, 2016 at 4:13 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> I used the following command to plot the variable:
series_cts_CSI_ge285.000
> carson:jraby ~/MET_SeriesAnalysis
> > plot_data_plane results_m3o3_TMP results_m3o3_TMP_CSI.ps
> 'name="series_cts_CSI_ge285.000"; level="(*,*)";'
> DEBUG 1: Opening data file: results_m3o3_TMP
> DEBUG 1: Creating postscript file: results_m3o3_TMP_CSI.ps
> carson:jraby ~/MET_SeriesAnalysis
>
> Seem to run fine, but the plot doesn't look like I expected.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Monday, April 18, 2016 1:41 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> I see the following error in the log file you sent me:
> ERROR :
> ERROR : plot_data_plane -> trouble getting field "name="RMSE";
> level="Z2";" from file "results_m3o3_TMP"
> ERROR :
>
> You are calling plot_data_plane by passing it a NetCDF file to plot.
The
> way you set "name" and "level" is different between GRIB files and
gridded
> NetCDF files. For NetCDF files, you set the name equal to the name
of the
> NetCDF variable you want to plot and you set level to "(*,*)". I
know that
> looks weird but you're telling MET which NetCDF dimensions are the
gridded
> ones that it should plot.
>
> In your case, the output of Series-Analysis will only have 2
dimensions...
> thus "(*,*)".
>
> Just use "ncdump -h" to look at the name of the variables and run
> plot_data_plane something like this:
>
> plot_data_plane results_m3o3_TMP results_m3o3_TMP_RMSE.ps
> 'name="series_cnt_RMSE"; level="(*,*)";'
>
> Hope that helps clarify.
>
> Thanks,
> John
>
> On Thu, Apr 14, 2016 at 9:14 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url: Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> > John -
> >
> > I resolved the issue with whitespace by using the name "RMSE" vice
"Root
> > mean squared error".
> >
> > Now it has an issue with the field "RMSE" from the output file.
See
> > attached log.
> >
> > Thanks.
> >
> > R/
> > John
> >
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Wednesday, April 13, 2016 4:57 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> > running series-analysis
> >
> > All active links contained in this email were disabled. Please
verify
> the
> > identity of the sender, and confirm the authenticity of all links
> contained
> > within the message prior to copying and pasting the address to a
Web
> > browser.
> >
> >
> >
> >
> > ----
> >
> > Also, I see the following error message in the log file you sent:
> >
> > ERROR : store_stat_cnt() -> unsupported column name requested
"IQR"
> >
> > IQR was added in version 5.1, but it not defined in version 4.1.
> >
> > John
> >
> > On Wed, Apr 13, 2016 at 4:55 PM, John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> >
> > > John,
> > >
> > > Ah yes, I meant to mention this. Please try increasing the
block size
> > > from 1024 to something much larger.
> > >
> > > Your grid is 288x288 = 82944 grid points. Try setting:
> > > block_size = 100000;
> > >
> > > That will process it all in one pass through the data and should
make
> it
> > > run much faster.
> > >
> > > John
> > >
> > > On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> <Caution-Caution-Caution-url: Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >>
> > >> John -
> > >>
> > >> I looked at your config file and it looks like a nice
workaround. I
> > reran
> > >> using your config file. Ran considerably longer this time. I
also
> have a
> > >> 14MB output NetCDF file which is definitely encouraging! I have
to
> leave
> > >> for the day in about 10 mins and will be back in touch
tomorrow.
> Thanks
> > for
> > >> your help.
> > >>
> > >> R/
> > >> John
> > >>
> > >>
> > >>
> > >>
> > >> ________________________________________
> > >> From: John Halley Gotway via RT [met_help at ucar.edu]
> > >> Sent: Wednesday, April 13, 2016 4:37 PM
> > >> To: Raby, John W CIV USARMY RDECOM ARL (US)
> > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > >> running series-analysis
> > >>
> > >> All active links contained in this email were disabled. Please
verify
> > >> the identity of the sender, and confirm the authenticity of all
links
> > >> contained within the message prior to copying and pasting the
address
> > to a
> > >> Web browser.
> > >>
> > >>
> > >>
> > >>
> > >> ----
> > >>
> > >> John,
> > >>
> > >> Ah yes, that time offset is the problem here.
> > >>
> > >> And it would have been nice for Series-Analysis to be more
verbose
> about
> > >> the timing information it's processing.
> > >>
> > >> Please try using the updated config file that I've attached,
and see
> if
> > >> Series-Analysis actually runs.
> > >>
> > >> We pass grid-stat 1 forecast file and 1 observation file. So
even if
> > the
> > >> times don't match up exactly, it writes a warning and compares
them.
> > >> Series-Analysis tries to be a bit fancier. You pass it N
forecast
> and N
> > >> observation files. For each forecast files, it searches all
the
> > >> observation files for a match.
> > >>
> > >> But that more complex logic breaks down when there's a 15
minute time
> > >> offset. So we should think about how better to handle this use
case.
> > >>
> > >> Thanks,
> > >> John
> > >>
> > >> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via RT <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >> >
> > >> > <Caution-Caution-Caution-Caution-url: Caution-
> > >> Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >> >
> > >> > John -
> > >> >
> > >> > Based on your analysis of the reading of the input files, I
can
> offer
> > a
> > >> > possibility that I'm aware of regarding the valid times.The
hours
> for
> > >> the
> > >> > example input data (observations) I sent you is indeed 23
which
> > matches
> > >> > that of the forecast data valid at the same hour (23), but
the
> minutes
> > >> of
> > >> > the forecast file is 00 while for the observations it is 15.
I was
> > >> aware of
> > >> > this mismatch in the minutes from when I ran Grid-Stat and it
> flagged
> > >> it as
> > >> > a "Warning", but it went ahead and generated output with no
Error.
> The
> > >> > presence of the min value of 15 is an artifact of the fact
that
> this
> > >> data
> > >> > was produced by the LAPS system. The valid time of the data
is
> > actually
> > >> > 2300 instead of 2315. That said, I will run wgrib on the
forecast
> file
> > >> per
> > >> > your suggestion to confirm the valid time.
> > >> > R/
> > >> > John
> > >> >
> > >> > ________________________________________
> > >> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > >> > Sent: Wednesday, April 13, 2016 3:48 PM
> > >> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > >> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > >> > running series-analysis
> > >> >
> > >> > All active links contained in this email were disabled.
Please
> verify
> > >> the
> > >> > identity of the sender, and confirm the authenticity of all
links
> > >> contained
> > >> > within the message prior to copying and pasting the address
to a Web
> > >> > browser.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > ----
> > >> >
> > >> > John,
> > >> >
> > >> > I don't see any immediately obvious problems in the data you
sent to
> > >> us. I
> > >> > see that you're writing a log file named:
> > >> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
> > >> >
> > >> > Are you able to send me that complete log file? That might
help me
> > >> > diagnose what's going on.
> > >> >
> > >> > Here's my initial guess...
> > >> >
> > >> > You are running series-analysis by defining a single field in
the
> > >> > configuration file. And you're passing to it lists of 12
forecast
> > files
> > >> > and 12 observation files.
> > >> >
> > >> > It reads TMP/Z2 from the first forecast file and check the
valid
> time
> > >> for
> > >> > that data. Next, it looks in the observation file list
(starting
> with
> > >> the
> > >> > first file) for TMP/Z2 with the *SAME* valid time.
> > >> >
> > >> > Next, it reads TMP/Z2 from the second forecast file, gets the
valid
> > >> time,
> > >> > and looks in the observation file list (starting with the
second
> file)
> > >> for
> > >> > TMP/Z2 with the *SAME* valid time.
> > >> >
> > >> > I looked at the observation file (120382300_laps_288.grb1)
you sent
> me
> > >> and
> > >> > see that it's valid time is 2012020723, which happens to
match the
> > >> > timestamp in the forecast file name. The first thing I'd
check is
> the
> > >> > valid time of the corresponding forecast file:
> > >> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
> > >> >
> > >> > Is it actually valid at the same time, 2012020723? If not,
that's
> the
> > >> > problem. If so, that's not the problem.
> > >> >
> > >> > Thanks,
> > >> > John
> > >> >
> > >> >
> > >> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via RT
<
> > >> > met_help at ucar.edu> wrote:
> > >> >
> > >> > >
> > >> > > <Caution-Caution-Caution-Caution-Caution-url: Caution-
> > >> > Caution-Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >> > >
> > >> > > Hi John -
> > >> > >
> > >> > > Yes. I got stuck on trying to figure out the cause of that
error.
> > >> Seems
> > >> > > like if Grid-Stat can read the 2-m temperature from the
gridded
> obs,
> > >> then
> > >> > > why would Series-Analysis not be able to read it?
> > >> > >
> > >> > > Thanks.
> > >> > >
> > >> > > R/
> > >> > > John
> > >> > >
> > >> > >
> > >> > > -----Original Message-----
> > >> > > From: John Halley Gotway via RT
> [Caution-Caution-Caution-Caution-Caution-mailto:
> > >> met_help at ucar.edu]
> > >> > > Sent: Wednesday, April 13, 2016 12:34 PM
> > >> > > To: Raby, John W CIV USARMY RDECOM ARL (US) <
> > >> john.w.raby2.civ at mail.mil>
> > >> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > >> running
> > >> > > series-analysis
> > >> > >
> > >> > > All active links contained in this email were disabled.
Please
> > verify
> > >> > the
> > >> > > identity of the sender, and confirm the authenticity of all
links
> > >> > contained
> > >> > > within the message prior to copying and pasting the address
to a
> Web
> > >> > > browser.
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > ----
> > >> > >
> > >> > > Hello John,
> > >> > >
> > >> > > I see that you were having trouble running series-analysis
on 2-m
> > >> > > temperature data last week.
> > >> > >
> > >> > > Is this still an issue? If so, I'd be happy to take a
look.
> > >> > >
> > >> > > Thanks,
> > >> > > John Halley Gotway
> > >> > >
> > >> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT <
> > >> met_help at ucar.edu
> > >> > >
> > >> > > wrote:
> > >> > >
> > >> > > >
> > >> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
> > >> > > > Transaction: Given to johnhg (John Halley Gotway) by
bullock
> > >> > > > Queue: met_help
> > >> > > > Subject: problem running series-analysis
> > >> > > > Owner: johnhg
> > >> > > > Requestors: john.w.raby2.civ at mail.mil
> > >> > > > Status: open
> > >> > > > Ticket <Caution-Caution-Caution-Caution-Caution-Caution-
url:
> > >> > > > Caution-Caution-
> > >> > Caution-Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >> > > >
> > >> > > >
> > >> > > > This transaction appears to have no content
> > >> > > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> >
> > >>
> > >>
> > >>
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: problem running series-analysis
From: John Halley Gotway
Time: Tue Apr 19 11:56:12 2016
John,
Sure, you can use my email address, johnhg at ucar.edu. Although looking
in
the "fcst_file_list" I see you're running Series-Analysis on 12 files.
With just a single fcst/obs file, running it on a series of length 1
won't
be all that informative.
For now please just send me a verbose log file for your run and I'll
look
to see if issues jump out.
Thanks,
John
On Tue, Apr 19, 2016 at 11:51 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> Thanks for taking a look at that. For large sized files, my only
method of
> transfer to you is by the ARL Secure Access File Exchange (SAFE)
which
> requires an email address to which they send you a link to download
the
> file. FTP is blocked here at WSMR. In the past, I have used your
email
> address @ucar.edu. Will this still work? If so, do you also want the
> forecast file which has the same valid time (almost-2300Z) as the
> observations file I sent you which was valid at 2315Z on 07 FEB
2012?
>
> I don't have ncview running here. I did try visualizing the output
file
> using the Unidata IDV software and none of the fields looked right
to me.
>
> I'll start working on a re-run of Series-Analysis to generate the
log per
> your instructions.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Tuesday, April 19, 2016 9:45 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> I agree with you. That output doesn't look good at all!
>
> Please just open up the NetCDF output file from Series-Analysis
using the
> "ncview" tool. Click through the variables and check whether any of
them
> look reasonable. If they look OK, then the problem is in plotting
them
> with plot_data_plane. If so, please just post that Series-Analysis
output
> file to our anonymous ftp site? And I'll figure out what's going on
with
> the plotting.
>
> But my guess is that none of the fields look good, which means the
issue is
> coming from Series-Analysis. Please try rerunning your Series-
Analysis
> command at verbosity level 5 (-v 5) and then save the output to a
log file
> (-log run_series_analysis.log). Then send me that log file.
>
> Thanks,
> John
>
> On Mon, Apr 18, 2016 at 4:13 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url: Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> > John -
> >
> > I used the following command to plot the variable:
> series_cts_CSI_ge285.000
> > carson:jraby ~/MET_SeriesAnalysis
> > > plot_data_plane results_m3o3_TMP results_m3o3_TMP_CSI.ps
> > 'name="series_cts_CSI_ge285.000"; level="(*,*)";'
> > DEBUG 1: Opening data file: results_m3o3_TMP
> > DEBUG 1: Creating postscript file: results_m3o3_TMP_CSI.ps
> > carson:jraby ~/MET_SeriesAnalysis
> >
> > Seem to run fine, but the plot doesn't look like I expected.
> >
> > R/
> > John
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Monday, April 18, 2016 1:41 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> > running series-analysis
> >
> > All active links contained in this email were disabled. Please
verify
> the
> > identity of the sender, and confirm the authenticity of all links
> contained
> > within the message prior to copying and pasting the address to a
Web
> > browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > I see the following error in the log file you sent me:
> > ERROR :
> > ERROR : plot_data_plane -> trouble getting field "name="RMSE";
> > level="Z2";" from file "results_m3o3_TMP"
> > ERROR :
> >
> > You are calling plot_data_plane by passing it a NetCDF file to
plot. The
> > way you set "name" and "level" is different between GRIB files and
> gridded
> > NetCDF files. For NetCDF files, you set the name equal to the
name of
> the
> > NetCDF variable you want to plot and you set level to "(*,*)". I
know
> that
> > looks weird but you're telling MET which NetCDF dimensions are the
> gridded
> > ones that it should plot.
> >
> > In your case, the output of Series-Analysis will only have 2
> dimensions...
> > thus "(*,*)".
> >
> > Just use "ncdump -h" to look at the name of the variables and run
> > plot_data_plane something like this:
> >
> > plot_data_plane results_m3o3_TMP results_m3o3_TMP_RMSE.ps
> > 'name="series_cnt_RMSE"; level="(*,*)";'
> >
> > Hope that helps clarify.
> >
> > Thanks,
> > John
> >
> > On Thu, Apr 14, 2016 at 9:14 AM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-url: Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >
> > > John -
> > >
> > > I resolved the issue with whitespace by using the name "RMSE"
vice
> "Root
> > > mean squared error".
> > >
> > > Now it has an issue with the field "RMSE" from the output file.
See
> > > attached log.
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Wednesday, April 13, 2016 4:57 PM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > running series-analysis
> > >
> > > All active links contained in this email were disabled. Please
verify
> > the
> > > identity of the sender, and confirm the authenticity of all
links
> > contained
> > > within the message prior to copying and pasting the address to a
Web
> > > browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > Also, I see the following error message in the log file you
sent:
> > >
> > > ERROR : store_stat_cnt() -> unsupported column name requested
"IQR"
> > >
> > > IQR was added in version 5.1, but it not defined in version 4.1.
> > >
> > > John
> > >
> > > On Wed, Apr 13, 2016 at 4:55 PM, John Halley Gotway
<johnhg at ucar.edu>
> > > wrote:
> > >
> > > > John,
> > > >
> > > > Ah yes, I meant to mention this. Please try increasing the
block
> size
> > > > from 1024 to something much larger.
> > > >
> > > > Your grid is 288x288 = 82944 grid points. Try setting:
> > > > block_size = 100000;
> > > >
> > > > That will process it all in one pass through the data and
should make
> > it
> > > > run much faster.
> > > >
> > > > John
> > > >
> > > > On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >>
> > > >> <Caution-Caution-Caution-url: Caution-
> > > Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >>
> > > >> John -
> > > >>
> > > >> I looked at your config file and it looks like a nice
workaround. I
> > > reran
> > > >> using your config file. Ran considerably longer this time. I
also
> > have a
> > > >> 14MB output NetCDF file which is definitely encouraging! I
have to
> > leave
> > > >> for the day in about 10 mins and will be back in touch
tomorrow.
> > Thanks
> > > for
> > > >> your help.
> > > >>
> > > >> R/
> > > >> John
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> ________________________________________
> > > >> From: John Halley Gotway via RT [met_help at ucar.edu]
> > > >> Sent: Wednesday, April 13, 2016 4:37 PM
> > > >> To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > >> running series-analysis
> > > >>
> > > >> All active links contained in this email were disabled.
Please
> verify
> > > >> the identity of the sender, and confirm the authenticity of
all
> links
> > > >> contained within the message prior to copying and pasting the
> address
> > > to a
> > > >> Web browser.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> ----
> > > >>
> > > >> John,
> > > >>
> > > >> Ah yes, that time offset is the problem here.
> > > >>
> > > >> And it would have been nice for Series-Analysis to be more
verbose
> > about
> > > >> the timing information it's processing.
> > > >>
> > > >> Please try using the updated config file that I've attached,
and see
> > if
> > > >> Series-Analysis actually runs.
> > > >>
> > > >> We pass grid-stat 1 forecast file and 1 observation file. So
even
> if
> > > the
> > > >> times don't match up exactly, it writes a warning and
compares them.
> > > >> Series-Analysis tries to be a bit fancier. You pass it N
forecast
> > and N
> > > >> observation files. For each forecast files, it searches all
the
> > > >> observation files for a match.
> > > >>
> > > >> But that more complex logic breaks down when there's a 15
minute
> time
> > > >> offset. So we should think about how better to handle this
use
> case.
> > > >>
> > > >> Thanks,
> > > >> John
> > > >>
> > > >> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via RT
<
> > > >> met_help at ucar.edu> wrote:
> > > >>
> > > >> >
> > > >> > <Caution-Caution-Caution-Caution-url: Caution-
> > > >> Caution-Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >> >
> > > >> > John -
> > > >> >
> > > >> > Based on your analysis of the reading of the input files, I
can
> > offer
> > > a
> > > >> > possibility that I'm aware of regarding the valid times.The
hours
> > for
> > > >> the
> > > >> > example input data (observations) I sent you is indeed 23
which
> > > matches
> > > >> > that of the forecast data valid at the same hour (23), but
the
> > minutes
> > > >> of
> > > >> > the forecast file is 00 while for the observations it is
15. I was
> > > >> aware of
> > > >> > this mismatch in the minutes from when I ran Grid-Stat and
it
> > flagged
> > > >> it as
> > > >> > a "Warning", but it went ahead and generated output with no
Error.
> > The
> > > >> > presence of the min value of 15 is an artifact of the fact
that
> > this
> > > >> data
> > > >> > was produced by the LAPS system. The valid time of the data
is
> > > actually
> > > >> > 2300 instead of 2315. That said, I will run wgrib on the
forecast
> > file
> > > >> per
> > > >> > your suggestion to confirm the valid time.
> > > >> > R/
> > > >> > John
> > > >> >
> > > >> > ________________________________________
> > > >> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > >> > Sent: Wednesday, April 13, 2016 3:48 PM
> > > >> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > >> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
> problem
> > > >> > running series-analysis
> > > >> >
> > > >> > All active links contained in this email were disabled.
Please
> > verify
> > > >> the
> > > >> > identity of the sender, and confirm the authenticity of all
links
> > > >> contained
> > > >> > within the message prior to copying and pasting the address
to a
> Web
> > > >> > browser.
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > ----
> > > >> >
> > > >> > John,
> > > >> >
> > > >> > I don't see any immediately obvious problems in the data
you sent
> to
> > > >> us. I
> > > >> > see that you're writing a log file named:
> > > >> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
> > > >> >
> > > >> > Are you able to send me that complete log file? That might
help
> me
> > > >> > diagnose what's going on.
> > > >> >
> > > >> > Here's my initial guess...
> > > >> >
> > > >> > You are running series-analysis by defining a single field
in the
> > > >> > configuration file. And you're passing to it lists of 12
forecast
> > > files
> > > >> > and 12 observation files.
> > > >> >
> > > >> > It reads TMP/Z2 from the first forecast file and check the
valid
> > time
> > > >> for
> > > >> > that data. Next, it looks in the observation file list
(starting
> > with
> > > >> the
> > > >> > first file) for TMP/Z2 with the *SAME* valid time.
> > > >> >
> > > >> > Next, it reads TMP/Z2 from the second forecast file, gets
the
> valid
> > > >> time,
> > > >> > and looks in the observation file list (starting with the
second
> > file)
> > > >> for
> > > >> > TMP/Z2 with the *SAME* valid time.
> > > >> >
> > > >> > I looked at the observation file (120382300_laps_288.grb1)
you
> sent
> > me
> > > >> and
> > > >> > see that it's valid time is 2012020723, which happens to
match the
> > > >> > timestamp in the forecast file name. The first thing I'd
check is
> > the
> > > >> > valid time of the corresponding forecast file:
> > > >> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
> > > >> >
> > > >> > Is it actually valid at the same time, 2012020723? If not,
that's
> > the
> > > >> > problem. If so, that's not the problem.
> > > >> >
> > > >> > Thanks,
> > > >> > John
> > > >> >
> > > >> >
> > > >> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via
RT <
> > > >> > met_help at ucar.edu> wrote:
> > > >> >
> > > >> > >
> > > >> > > <Caution-Caution-Caution-Caution-Caution-url: Caution-
> > > >> > Caution-Caution-
> > > Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >> > >
> > > >> > > Hi John -
> > > >> > >
> > > >> > > Yes. I got stuck on trying to figure out the cause of
that
> error.
> > > >> Seems
> > > >> > > like if Grid-Stat can read the 2-m temperature from the
gridded
> > obs,
> > > >> then
> > > >> > > why would Series-Analysis not be able to read it?
> > > >> > >
> > > >> > > Thanks.
> > > >> > >
> > > >> > > R/
> > > >> > > John
> > > >> > >
> > > >> > >
> > > >> > > -----Original Message-----
> > > >> > > From: John Halley Gotway via RT
> > [Caution-Caution-Caution-Caution-Caution-mailto:
> > > >> met_help at ucar.edu]
> > > >> > > Sent: Wednesday, April 13, 2016 12:34 PM
> > > >> > > To: Raby, John W CIV USARMY RDECOM ARL (US) <
> > > >> john.w.raby2.civ at mail.mil>
> > > >> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > >> running
> > > >> > > series-analysis
> > > >> > >
> > > >> > > All active links contained in this email were disabled.
Please
> > > verify
> > > >> > the
> > > >> > > identity of the sender, and confirm the authenticity of
all
> links
> > > >> > contained
> > > >> > > within the message prior to copying and pasting the
address to a
> > Web
> > > >> > > browser.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > ----
> > > >> > >
> > > >> > > Hello John,
> > > >> > >
> > > >> > > I see that you were having trouble running series-
analysis on
> 2-m
> > > >> > > temperature data last week.
> > > >> > >
> > > >> > > Is this still an issue? If so, I'd be happy to take a
look.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > John Halley Gotway
> > > >> > >
> > > >> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT <
> > > >> met_help at ucar.edu
> > > >> > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > >
> > > >> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
> > > >> > > > Transaction: Given to johnhg (John Halley Gotway) by
bullock
> > > >> > > > Queue: met_help
> > > >> > > > Subject: problem running series-analysis
> > > >> > > > Owner: johnhg
> > > >> > > > Requestors: john.w.raby2.civ at mail.mil
> > > >> > > > Status: open
> > > >> > > > Ticket <Caution-Caution-Caution-Caution-Caution-
Caution-url:
> > > >> > > > Caution-Caution-
> > > >> > Caution-Caution-
> > > Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >> > > >
> > > >> > > >
> > > >> > > > This transaction appears to have no content
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: problem running series-analysis
From: Raby, John W USA CIV
Time: Tue Apr 19 15:12:16 2016
John -
I looked at my Series-Analysis run script and it has verbose level 5
specifed and has been that way since the 13th of APR when I used a
config file with the string "JHG" appended to the name which you had
provided to run series-analysis to generate the 14 mb output NetCDF
file which was plotted recently using plot_data_plane. I have attached
the log file which was generated for that run which used verbosity 5.
Let me know if this gives you the info you need.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Tuesday, April 19, 2016 11:56 AM
To: Raby, John W CIV USARMY RDECOM ARL (US)
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running series-analysis
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
Sure, you can use my email address, johnhg at ucar.edu. Although looking
in
the "fcst_file_list" I see you're running Series-Analysis on 12 files.
With just a single fcst/obs file, running it on a series of length 1
won't
be all that informative.
For now please just send me a verbose log file for your run and I'll
look
to see if issues jump out.
Thanks,
John
On Tue, Apr 19, 2016 at 11:51 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> Thanks for taking a look at that. For large sized files, my only
method of
> transfer to you is by the ARL Secure Access File Exchange (SAFE)
which
> requires an email address to which they send you a link to download
the
> file. FTP is blocked here at WSMR. In the past, I have used your
email
> address @ucar.edu. Will this still work? If so, do you also want the
> forecast file which has the same valid time (almost-2300Z) as the
> observations file I sent you which was valid at 2315Z on 07 FEB
2012?
>
> I don't have ncview running here. I did try visualizing the output
file
> using the Unidata IDV software and none of the fields looked right
to me.
>
> I'll start working on a re-run of Series-Analysis to generate the
log per
> your instructions.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Tuesday, April 19, 2016 9:45 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> I agree with you. That output doesn't look good at all!
>
> Please just open up the NetCDF output file from Series-Analysis
using the
> "ncview" tool. Click through the variables and check whether any of
them
> look reasonable. If they look OK, then the problem is in plotting
them
> with plot_data_plane. If so, please just post that Series-Analysis
output
> file to our anonymous ftp site? And I'll figure out what's going on
with
> the plotting.
>
> But my guess is that none of the fields look good, which means the
issue is
> coming from Series-Analysis. Please try rerunning your Series-
Analysis
> command at verbosity level 5 (-v 5) and then save the output to a
log file
> (-log run_series_analysis.log). Then send me that log file.
>
> Thanks,
> John
>
> On Mon, Apr 18, 2016 at 4:13 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url: Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> > John -
> >
> > I used the following command to plot the variable:
> series_cts_CSI_ge285.000
> > carson:jraby ~/MET_SeriesAnalysis
> > > plot_data_plane results_m3o3_TMP results_m3o3_TMP_CSI.ps
> > 'name="series_cts_CSI_ge285.000"; level="(*,*)";'
> > DEBUG 1: Opening data file: results_m3o3_TMP
> > DEBUG 1: Creating postscript file: results_m3o3_TMP_CSI.ps
> > carson:jraby ~/MET_SeriesAnalysis
> >
> > Seem to run fine, but the plot doesn't look like I expected.
> >
> > R/
> > John
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Monday, April 18, 2016 1:41 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> > running series-analysis
> >
> > All active links contained in this email were disabled. Please
verify
> the
> > identity of the sender, and confirm the authenticity of all links
> contained
> > within the message prior to copying and pasting the address to a
Web
> > browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > I see the following error in the log file you sent me:
> > ERROR :
> > ERROR : plot_data_plane -> trouble getting field "name="RMSE";
> > level="Z2";" from file "results_m3o3_TMP"
> > ERROR :
> >
> > You are calling plot_data_plane by passing it a NetCDF file to
plot. The
> > way you set "name" and "level" is different between GRIB files and
> gridded
> > NetCDF files. For NetCDF files, you set the name equal to the
name of
> the
> > NetCDF variable you want to plot and you set level to "(*,*)". I
know
> that
> > looks weird but you're telling MET which NetCDF dimensions are the
> gridded
> > ones that it should plot.
> >
> > In your case, the output of Series-Analysis will only have 2
> dimensions...
> > thus "(*,*)".
> >
> > Just use "ncdump -h" to look at the name of the variables and run
> > plot_data_plane something like this:
> >
> > plot_data_plane results_m3o3_TMP results_m3o3_TMP_RMSE.ps
> > 'name="series_cnt_RMSE"; level="(*,*)";'
> >
> > Hope that helps clarify.
> >
> > Thanks,
> > John
> >
> > On Thu, Apr 14, 2016 at 9:14 AM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url: Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >
> > > John -
> > >
> > > I resolved the issue with whitespace by using the name "RMSE"
vice
> "Root
> > > mean squared error".
> > >
> > > Now it has an issue with the field "RMSE" from the output file.
See
> > > attached log.
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Wednesday, April 13, 2016 4:57 PM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > running series-analysis
> > >
> > > All active links contained in this email were disabled. Please
verify
> > the
> > > identity of the sender, and confirm the authenticity of all
links
> > contained
> > > within the message prior to copying and pasting the address to a
Web
> > > browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > Also, I see the following error message in the log file you
sent:
> > >
> > > ERROR : store_stat_cnt() -> unsupported column name requested
"IQR"
> > >
> > > IQR was added in version 5.1, but it not defined in version 4.1.
> > >
> > > John
> > >
> > > On Wed, Apr 13, 2016 at 4:55 PM, John Halley Gotway
<johnhg at ucar.edu>
> > > wrote:
> > >
> > > > John,
> > > >
> > > > Ah yes, I meant to mention this. Please try increasing the
block
> size
> > > > from 1024 to something much larger.
> > > >
> > > > Your grid is 288x288 = 82944 grid points. Try setting:
> > > > block_size = 100000;
> > > >
> > > > That will process it all in one pass through the data and
should make
> > it
> > > > run much faster.
> > > >
> > > > John
> > > >
> > > > On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >>
> > > >> <Caution-Caution-Caution-Caution-url: Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >>
> > > >> John -
> > > >>
> > > >> I looked at your config file and it looks like a nice
workaround. I
> > > reran
> > > >> using your config file. Ran considerably longer this time. I
also
> > have a
> > > >> 14MB output NetCDF file which is definitely encouraging! I
have to
> > leave
> > > >> for the day in about 10 mins and will be back in touch
tomorrow.
> > Thanks
> > > for
> > > >> your help.
> > > >>
> > > >> R/
> > > >> John
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> ________________________________________
> > > >> From: John Halley Gotway via RT [met_help at ucar.edu]
> > > >> Sent: Wednesday, April 13, 2016 4:37 PM
> > > >> To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > >> running series-analysis
> > > >>
> > > >> All active links contained in this email were disabled.
Please
> verify
> > > >> the identity of the sender, and confirm the authenticity of
all
> links
> > > >> contained within the message prior to copying and pasting the
> address
> > > to a
> > > >> Web browser.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> ----
> > > >>
> > > >> John,
> > > >>
> > > >> Ah yes, that time offset is the problem here.
> > > >>
> > > >> And it would have been nice for Series-Analysis to be more
verbose
> > about
> > > >> the timing information it's processing.
> > > >>
> > > >> Please try using the updated config file that I've attached,
and see
> > if
> > > >> Series-Analysis actually runs.
> > > >>
> > > >> We pass grid-stat 1 forecast file and 1 observation file. So
even
> if
> > > the
> > > >> times don't match up exactly, it writes a warning and
compares them.
> > > >> Series-Analysis tries to be a bit fancier. You pass it N
forecast
> > and N
> > > >> observation files. For each forecast files, it searches all
the
> > > >> observation files for a match.
> > > >>
> > > >> But that more complex logic breaks down when there's a 15
minute
> time
> > > >> offset. So we should think about how better to handle this
use
> case.
> > > >>
> > > >> Thanks,
> > > >> John
> > > >>
> > > >> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via RT
<
> > > >> met_help at ucar.edu> wrote:
> > > >>
> > > >> >
> > > >> > <Caution-Caution-Caution-Caution-Caution-url: Caution-
> > > >> Caution-Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >> >
> > > >> > John -
> > > >> >
> > > >> > Based on your analysis of the reading of the input files, I
can
> > offer
> > > a
> > > >> > possibility that I'm aware of regarding the valid times.The
hours
> > for
> > > >> the
> > > >> > example input data (observations) I sent you is indeed 23
which
> > > matches
> > > >> > that of the forecast data valid at the same hour (23), but
the
> > minutes
> > > >> of
> > > >> > the forecast file is 00 while for the observations it is
15. I was
> > > >> aware of
> > > >> > this mismatch in the minutes from when I ran Grid-Stat and
it
> > flagged
> > > >> it as
> > > >> > a "Warning", but it went ahead and generated output with no
Error.
> > The
> > > >> > presence of the min value of 15 is an artifact of the fact
that
> > this
> > > >> data
> > > >> > was produced by the LAPS system. The valid time of the data
is
> > > actually
> > > >> > 2300 instead of 2315. That said, I will run wgrib on the
forecast
> > file
> > > >> per
> > > >> > your suggestion to confirm the valid time.
> > > >> > R/
> > > >> > John
> > > >> >
> > > >> > ________________________________________
> > > >> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > >> > Sent: Wednesday, April 13, 2016 3:48 PM
> > > >> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > >> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
> problem
> > > >> > running series-analysis
> > > >> >
> > > >> > All active links contained in this email were disabled.
Please
> > verify
> > > >> the
> > > >> > identity of the sender, and confirm the authenticity of all
links
> > > >> contained
> > > >> > within the message prior to copying and pasting the address
to a
> Web
> > > >> > browser.
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > ----
> > > >> >
> > > >> > John,
> > > >> >
> > > >> > I don't see any immediately obvious problems in the data
you sent
> to
> > > >> us. I
> > > >> > see that you're writing a log file named:
> > > >> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
> > > >> >
> > > >> > Are you able to send me that complete log file? That might
help
> me
> > > >> > diagnose what's going on.
> > > >> >
> > > >> > Here's my initial guess...
> > > >> >
> > > >> > You are running series-analysis by defining a single field
in the
> > > >> > configuration file. And you're passing to it lists of 12
forecast
> > > files
> > > >> > and 12 observation files.
> > > >> >
> > > >> > It reads TMP/Z2 from the first forecast file and check the
valid
> > time
> > > >> for
> > > >> > that data. Next, it looks in the observation file list
(starting
> > with
> > > >> the
> > > >> > first file) for TMP/Z2 with the *SAME* valid time.
> > > >> >
> > > >> > Next, it reads TMP/Z2 from the second forecast file, gets
the
> valid
> > > >> time,
> > > >> > and looks in the observation file list (starting with the
second
> > file)
> > > >> for
> > > >> > TMP/Z2 with the *SAME* valid time.
> > > >> >
> > > >> > I looked at the observation file (120382300_laps_288.grb1)
you
> sent
> > me
> > > >> and
> > > >> > see that it's valid time is 2012020723, which happens to
match the
> > > >> > timestamp in the forecast file name. The first thing I'd
check is
> > the
> > > >> > valid time of the corresponding forecast file:
> > > >> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
> > > >> >
> > > >> > Is it actually valid at the same time, 2012020723? If not,
that's
> > the
> > > >> > problem. If so, that's not the problem.
> > > >> >
> > > >> > Thanks,
> > > >> > John
> > > >> >
> > > >> >
> > > >> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via
RT <
> > > >> > met_help at ucar.edu> wrote:
> > > >> >
> > > >> > >
> > > >> > > <Caution-Caution-Caution-Caution-Caution-Caution-url:
Caution-
> > > >> > Caution-Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >> > >
> > > >> > > Hi John -
> > > >> > >
> > > >> > > Yes. I got stuck on trying to figure out the cause of
that
> error.
> > > >> Seems
> > > >> > > like if Grid-Stat can read the 2-m temperature from the
gridded
> > obs,
> > > >> then
> > > >> > > why would Series-Analysis not be able to read it?
> > > >> > >
> > > >> > > Thanks.
> > > >> > >
> > > >> > > R/
> > > >> > > John
> > > >> > >
> > > >> > >
> > > >> > > -----Original Message-----
> > > >> > > From: John Halley Gotway via RT
> > [Caution-Caution-Caution-Caution-Caution-Caution-mailto:
> > > >> met_help at ucar.edu]
> > > >> > > Sent: Wednesday, April 13, 2016 12:34 PM
> > > >> > > To: Raby, John W CIV USARMY RDECOM ARL (US) <
> > > >> john.w.raby2.civ at mail.mil>
> > > >> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > >> running
> > > >> > > series-analysis
> > > >> > >
> > > >> > > All active links contained in this email were disabled.
Please
> > > verify
> > > >> > the
> > > >> > > identity of the sender, and confirm the authenticity of
all
> links
> > > >> > contained
> > > >> > > within the message prior to copying and pasting the
address to a
> > Web
> > > >> > > browser.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > ----
> > > >> > >
> > > >> > > Hello John,
> > > >> > >
> > > >> > > I see that you were having trouble running series-
analysis on
> 2-m
> > > >> > > temperature data last week.
> > > >> > >
> > > >> > > Is this still an issue? If so, I'd be happy to take a
look.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > John Halley Gotway
> > > >> > >
> > > >> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT <
> > > >> met_help at ucar.edu
> > > >> > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > >
> > > >> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
> > > >> > > > Transaction: Given to johnhg (John Halley Gotway) by
bullock
> > > >> > > > Queue: met_help
> > > >> > > > Subject: problem running series-analysis
> > > >> > > > Owner: johnhg
> > > >> > > > Requestors: john.w.raby2.civ at mail.mil
> > > >> > > > Status: open
> > > >> > > > Ticket <Caution-Caution-Caution-Caution-Caution-
Caution-Caution-url:
> > > >> > > > Caution-Caution-
> > > >> > Caution-Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >> > > >
> > > >> > > >
> > > >> > > > This transaction appears to have no content
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem running series-analysis
From: Raby, John W USA CIV
Time: Tue Apr 19 15:39:55 2016
John -
FYI, I just now reran series-analysis (-v 5) as before, but this time
I edited the config file to remove the source of the error from
specifying "IQR" which you mentioned was not supported by V4.1. This
time the run took MUCH longer and, in fact, is still running!
Apparently, that IQR error shut down the process, even through a 14mb
output file was generated. I will send you the log file from this run
ASAP after completion of the run.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Tuesday, April 19, 2016 11:56 AM
To: Raby, John W CIV USARMY RDECOM ARL (US)
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running series-analysis
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
Sure, you can use my email address, johnhg at ucar.edu. Although looking
in
the "fcst_file_list" I see you're running Series-Analysis on 12 files.
With just a single fcst/obs file, running it on a series of length 1
won't
be all that informative.
For now please just send me a verbose log file for your run and I'll
look
to see if issues jump out.
Thanks,
John
On Tue, Apr 19, 2016 at 11:51 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> Thanks for taking a look at that. For large sized files, my only
method of
> transfer to you is by the ARL Secure Access File Exchange (SAFE)
which
> requires an email address to which they send you a link to download
the
> file. FTP is blocked here at WSMR. In the past, I have used your
email
> address @ucar.edu. Will this still work? If so, do you also want the
> forecast file which has the same valid time (almost-2300Z) as the
> observations file I sent you which was valid at 2315Z on 07 FEB
2012?
>
> I don't have ncview running here. I did try visualizing the output
file
> using the Unidata IDV software and none of the fields looked right
to me.
>
> I'll start working on a re-run of Series-Analysis to generate the
log per
> your instructions.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Tuesday, April 19, 2016 9:45 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> I agree with you. That output doesn't look good at all!
>
> Please just open up the NetCDF output file from Series-Analysis
using the
> "ncview" tool. Click through the variables and check whether any of
them
> look reasonable. If they look OK, then the problem is in plotting
them
> with plot_data_plane. If so, please just post that Series-Analysis
output
> file to our anonymous ftp site? And I'll figure out what's going on
with
> the plotting.
>
> But my guess is that none of the fields look good, which means the
issue is
> coming from Series-Analysis. Please try rerunning your Series-
Analysis
> command at verbosity level 5 (-v 5) and then save the output to a
log file
> (-log run_series_analysis.log). Then send me that log file.
>
> Thanks,
> John
>
> On Mon, Apr 18, 2016 at 4:13 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url: Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> > John -
> >
> > I used the following command to plot the variable:
> series_cts_CSI_ge285.000
> > carson:jraby ~/MET_SeriesAnalysis
> > > plot_data_plane results_m3o3_TMP results_m3o3_TMP_CSI.ps
> > 'name="series_cts_CSI_ge285.000"; level="(*,*)";'
> > DEBUG 1: Opening data file: results_m3o3_TMP
> > DEBUG 1: Creating postscript file: results_m3o3_TMP_CSI.ps
> > carson:jraby ~/MET_SeriesAnalysis
> >
> > Seem to run fine, but the plot doesn't look like I expected.
> >
> > R/
> > John
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Monday, April 18, 2016 1:41 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> > running series-analysis
> >
> > All active links contained in this email were disabled. Please
verify
> the
> > identity of the sender, and confirm the authenticity of all links
> contained
> > within the message prior to copying and pasting the address to a
Web
> > browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > I see the following error in the log file you sent me:
> > ERROR :
> > ERROR : plot_data_plane -> trouble getting field "name="RMSE";
> > level="Z2";" from file "results_m3o3_TMP"
> > ERROR :
> >
> > You are calling plot_data_plane by passing it a NetCDF file to
plot. The
> > way you set "name" and "level" is different between GRIB files and
> gridded
> > NetCDF files. For NetCDF files, you set the name equal to the
name of
> the
> > NetCDF variable you want to plot and you set level to "(*,*)". I
know
> that
> > looks weird but you're telling MET which NetCDF dimensions are the
> gridded
> > ones that it should plot.
> >
> > In your case, the output of Series-Analysis will only have 2
> dimensions...
> > thus "(*,*)".
> >
> > Just use "ncdump -h" to look at the name of the variables and run
> > plot_data_plane something like this:
> >
> > plot_data_plane results_m3o3_TMP results_m3o3_TMP_RMSE.ps
> > 'name="series_cnt_RMSE"; level="(*,*)";'
> >
> > Hope that helps clarify.
> >
> > Thanks,
> > John
> >
> > On Thu, Apr 14, 2016 at 9:14 AM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url: Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >
> > > John -
> > >
> > > I resolved the issue with whitespace by using the name "RMSE"
vice
> "Root
> > > mean squared error".
> > >
> > > Now it has an issue with the field "RMSE" from the output file.
See
> > > attached log.
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Wednesday, April 13, 2016 4:57 PM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > running series-analysis
> > >
> > > All active links contained in this email were disabled. Please
verify
> > the
> > > identity of the sender, and confirm the authenticity of all
links
> > contained
> > > within the message prior to copying and pasting the address to a
Web
> > > browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > Also, I see the following error message in the log file you
sent:
> > >
> > > ERROR : store_stat_cnt() -> unsupported column name requested
"IQR"
> > >
> > > IQR was added in version 5.1, but it not defined in version 4.1.
> > >
> > > John
> > >
> > > On Wed, Apr 13, 2016 at 4:55 PM, John Halley Gotway
<johnhg at ucar.edu>
> > > wrote:
> > >
> > > > John,
> > > >
> > > > Ah yes, I meant to mention this. Please try increasing the
block
> size
> > > > from 1024 to something much larger.
> > > >
> > > > Your grid is 288x288 = 82944 grid points. Try setting:
> > > > block_size = 100000;
> > > >
> > > > That will process it all in one pass through the data and
should make
> > it
> > > > run much faster.
> > > >
> > > > John
> > > >
> > > > On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >>
> > > >> <Caution-Caution-Caution-Caution-url: Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >>
> > > >> John -
> > > >>
> > > >> I looked at your config file and it looks like a nice
workaround. I
> > > reran
> > > >> using your config file. Ran considerably longer this time. I
also
> > have a
> > > >> 14MB output NetCDF file which is definitely encouraging! I
have to
> > leave
> > > >> for the day in about 10 mins and will be back in touch
tomorrow.
> > Thanks
> > > for
> > > >> your help.
> > > >>
> > > >> R/
> > > >> John
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> ________________________________________
> > > >> From: John Halley Gotway via RT [met_help at ucar.edu]
> > > >> Sent: Wednesday, April 13, 2016 4:37 PM
> > > >> To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > >> running series-analysis
> > > >>
> > > >> All active links contained in this email were disabled.
Please
> verify
> > > >> the identity of the sender, and confirm the authenticity of
all
> links
> > > >> contained within the message prior to copying and pasting the
> address
> > > to a
> > > >> Web browser.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> ----
> > > >>
> > > >> John,
> > > >>
> > > >> Ah yes, that time offset is the problem here.
> > > >>
> > > >> And it would have been nice for Series-Analysis to be more
verbose
> > about
> > > >> the timing information it's processing.
> > > >>
> > > >> Please try using the updated config file that I've attached,
and see
> > if
> > > >> Series-Analysis actually runs.
> > > >>
> > > >> We pass grid-stat 1 forecast file and 1 observation file. So
even
> if
> > > the
> > > >> times don't match up exactly, it writes a warning and
compares them.
> > > >> Series-Analysis tries to be a bit fancier. You pass it N
forecast
> > and N
> > > >> observation files. For each forecast files, it searches all
the
> > > >> observation files for a match.
> > > >>
> > > >> But that more complex logic breaks down when there's a 15
minute
> time
> > > >> offset. So we should think about how better to handle this
use
> case.
> > > >>
> > > >> Thanks,
> > > >> John
> > > >>
> > > >> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via RT
<
> > > >> met_help at ucar.edu> wrote:
> > > >>
> > > >> >
> > > >> > <Caution-Caution-Caution-Caution-Caution-url: Caution-
> > > >> Caution-Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >> >
> > > >> > John -
> > > >> >
> > > >> > Based on your analysis of the reading of the input files, I
can
> > offer
> > > a
> > > >> > possibility that I'm aware of regarding the valid times.The
hours
> > for
> > > >> the
> > > >> > example input data (observations) I sent you is indeed 23
which
> > > matches
> > > >> > that of the forecast data valid at the same hour (23), but
the
> > minutes
> > > >> of
> > > >> > the forecast file is 00 while for the observations it is
15. I was
> > > >> aware of
> > > >> > this mismatch in the minutes from when I ran Grid-Stat and
it
> > flagged
> > > >> it as
> > > >> > a "Warning", but it went ahead and generated output with no
Error.
> > The
> > > >> > presence of the min value of 15 is an artifact of the fact
that
> > this
> > > >> data
> > > >> > was produced by the LAPS system. The valid time of the data
is
> > > actually
> > > >> > 2300 instead of 2315. That said, I will run wgrib on the
forecast
> > file
> > > >> per
> > > >> > your suggestion to confirm the valid time.
> > > >> > R/
> > > >> > John
> > > >> >
> > > >> > ________________________________________
> > > >> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > >> > Sent: Wednesday, April 13, 2016 3:48 PM
> > > >> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > >> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
> problem
> > > >> > running series-analysis
> > > >> >
> > > >> > All active links contained in this email were disabled.
Please
> > verify
> > > >> the
> > > >> > identity of the sender, and confirm the authenticity of all
links
> > > >> contained
> > > >> > within the message prior to copying and pasting the address
to a
> Web
> > > >> > browser.
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > ----
> > > >> >
> > > >> > John,
> > > >> >
> > > >> > I don't see any immediately obvious problems in the data
you sent
> to
> > > >> us. I
> > > >> > see that you're writing a log file named:
> > > >> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
> > > >> >
> > > >> > Are you able to send me that complete log file? That might
help
> me
> > > >> > diagnose what's going on.
> > > >> >
> > > >> > Here's my initial guess...
> > > >> >
> > > >> > You are running series-analysis by defining a single field
in the
> > > >> > configuration file. And you're passing to it lists of 12
forecast
> > > files
> > > >> > and 12 observation files.
> > > >> >
> > > >> > It reads TMP/Z2 from the first forecast file and check the
valid
> > time
> > > >> for
> > > >> > that data. Next, it looks in the observation file list
(starting
> > with
> > > >> the
> > > >> > first file) for TMP/Z2 with the *SAME* valid time.
> > > >> >
> > > >> > Next, it reads TMP/Z2 from the second forecast file, gets
the
> valid
> > > >> time,
> > > >> > and looks in the observation file list (starting with the
second
> > file)
> > > >> for
> > > >> > TMP/Z2 with the *SAME* valid time.
> > > >> >
> > > >> > I looked at the observation file (120382300_laps_288.grb1)
you
> sent
> > me
> > > >> and
> > > >> > see that it's valid time is 2012020723, which happens to
match the
> > > >> > timestamp in the forecast file name. The first thing I'd
check is
> > the
> > > >> > valid time of the corresponding forecast file:
> > > >> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
> > > >> >
> > > >> > Is it actually valid at the same time, 2012020723? If not,
that's
> > the
> > > >> > problem. If so, that's not the problem.
> > > >> >
> > > >> > Thanks,
> > > >> > John
> > > >> >
> > > >> >
> > > >> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via
RT <
> > > >> > met_help at ucar.edu> wrote:
> > > >> >
> > > >> > >
> > > >> > > <Caution-Caution-Caution-Caution-Caution-Caution-url:
Caution-
> > > >> > Caution-Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >> > >
> > > >> > > Hi John -
> > > >> > >
> > > >> > > Yes. I got stuck on trying to figure out the cause of
that
> error.
> > > >> Seems
> > > >> > > like if Grid-Stat can read the 2-m temperature from the
gridded
> > obs,
> > > >> then
> > > >> > > why would Series-Analysis not be able to read it?
> > > >> > >
> > > >> > > Thanks.
> > > >> > >
> > > >> > > R/
> > > >> > > John
> > > >> > >
> > > >> > >
> > > >> > > -----Original Message-----
> > > >> > > From: John Halley Gotway via RT
> > [Caution-Caution-Caution-Caution-Caution-Caution-mailto:
> > > >> met_help at ucar.edu]
> > > >> > > Sent: Wednesday, April 13, 2016 12:34 PM
> > > >> > > To: Raby, John W CIV USARMY RDECOM ARL (US) <
> > > >> john.w.raby2.civ at mail.mil>
> > > >> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > >> running
> > > >> > > series-analysis
> > > >> > >
> > > >> > > All active links contained in this email were disabled.
Please
> > > verify
> > > >> > the
> > > >> > > identity of the sender, and confirm the authenticity of
all
> links
> > > >> > contained
> > > >> > > within the message prior to copying and pasting the
address to a
> > Web
> > > >> > > browser.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > ----
> > > >> > >
> > > >> > > Hello John,
> > > >> > >
> > > >> > > I see that you were having trouble running series-
analysis on
> 2-m
> > > >> > > temperature data last week.
> > > >> > >
> > > >> > > Is this still an issue? If so, I'd be happy to take a
look.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > John Halley Gotway
> > > >> > >
> > > >> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT <
> > > >> met_help at ucar.edu
> > > >> > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > >
> > > >> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
> > > >> > > > Transaction: Given to johnhg (John Halley Gotway) by
bullock
> > > >> > > > Queue: met_help
> > > >> > > > Subject: problem running series-analysis
> > > >> > > > Owner: johnhg
> > > >> > > > Requestors: john.w.raby2.civ at mail.mil
> > > >> > > > Status: open
> > > >> > > > Ticket <Caution-Caution-Caution-Caution-Caution-
Caution-Caution-url:
> > > >> > > > Caution-Caution-
> > > >> > Caution-Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >> > > >
> > > >> > > >
> > > >> > > > This transaction appears to have no content
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem running series-analysis
From: Raby, John W USA CIV
Time: Tue Apr 19 15:50:53 2016
John -
This run continues! Now I see why you specified block size for pass
reduction. This baby is gonna take some time, judging form the way
it's iterating through the grid. Removal of that source of the error
definitely changed the character of the run. With a 288X288 grid, we
may see some results tomorrow at the rate it's going.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Tuesday, April 19, 2016 11:56 AM
To: Raby, John W CIV USARMY RDECOM ARL (US)
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running series-analysis
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
Sure, you can use my email address, johnhg at ucar.edu. Although looking
in
the "fcst_file_list" I see you're running Series-Analysis on 12 files.
With just a single fcst/obs file, running it on a series of length 1
won't
be all that informative.
For now please just send me a verbose log file for your run and I'll
look
to see if issues jump out.
Thanks,
John
On Tue, Apr 19, 2016 at 11:51 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> Thanks for taking a look at that. For large sized files, my only
method of
> transfer to you is by the ARL Secure Access File Exchange (SAFE)
which
> requires an email address to which they send you a link to download
the
> file. FTP is blocked here at WSMR. In the past, I have used your
email
> address @ucar.edu. Will this still work? If so, do you also want the
> forecast file which has the same valid time (almost-2300Z) as the
> observations file I sent you which was valid at 2315Z on 07 FEB
2012?
>
> I don't have ncview running here. I did try visualizing the output
file
> using the Unidata IDV software and none of the fields looked right
to me.
>
> I'll start working on a re-run of Series-Analysis to generate the
log per
> your instructions.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Tuesday, April 19, 2016 9:45 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> I agree with you. That output doesn't look good at all!
>
> Please just open up the NetCDF output file from Series-Analysis
using the
> "ncview" tool. Click through the variables and check whether any of
them
> look reasonable. If they look OK, then the problem is in plotting
them
> with plot_data_plane. If so, please just post that Series-Analysis
output
> file to our anonymous ftp site? And I'll figure out what's going on
with
> the plotting.
>
> But my guess is that none of the fields look good, which means the
issue is
> coming from Series-Analysis. Please try rerunning your Series-
Analysis
> command at verbosity level 5 (-v 5) and then save the output to a
log file
> (-log run_series_analysis.log). Then send me that log file.
>
> Thanks,
> John
>
> On Mon, Apr 18, 2016 at 4:13 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url: Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> > John -
> >
> > I used the following command to plot the variable:
> series_cts_CSI_ge285.000
> > carson:jraby ~/MET_SeriesAnalysis
> > > plot_data_plane results_m3o3_TMP results_m3o3_TMP_CSI.ps
> > 'name="series_cts_CSI_ge285.000"; level="(*,*)";'
> > DEBUG 1: Opening data file: results_m3o3_TMP
> > DEBUG 1: Creating postscript file: results_m3o3_TMP_CSI.ps
> > carson:jraby ~/MET_SeriesAnalysis
> >
> > Seem to run fine, but the plot doesn't look like I expected.
> >
> > R/
> > John
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Monday, April 18, 2016 1:41 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> > running series-analysis
> >
> > All active links contained in this email were disabled. Please
verify
> the
> > identity of the sender, and confirm the authenticity of all links
> contained
> > within the message prior to copying and pasting the address to a
Web
> > browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > I see the following error in the log file you sent me:
> > ERROR :
> > ERROR : plot_data_plane -> trouble getting field "name="RMSE";
> > level="Z2";" from file "results_m3o3_TMP"
> > ERROR :
> >
> > You are calling plot_data_plane by passing it a NetCDF file to
plot. The
> > way you set "name" and "level" is different between GRIB files and
> gridded
> > NetCDF files. For NetCDF files, you set the name equal to the
name of
> the
> > NetCDF variable you want to plot and you set level to "(*,*)". I
know
> that
> > looks weird but you're telling MET which NetCDF dimensions are the
> gridded
> > ones that it should plot.
> >
> > In your case, the output of Series-Analysis will only have 2
> dimensions...
> > thus "(*,*)".
> >
> > Just use "ncdump -h" to look at the name of the variables and run
> > plot_data_plane something like this:
> >
> > plot_data_plane results_m3o3_TMP results_m3o3_TMP_RMSE.ps
> > 'name="series_cnt_RMSE"; level="(*,*)";'
> >
> > Hope that helps clarify.
> >
> > Thanks,
> > John
> >
> > On Thu, Apr 14, 2016 at 9:14 AM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url: Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >
> > > John -
> > >
> > > I resolved the issue with whitespace by using the name "RMSE"
vice
> "Root
> > > mean squared error".
> > >
> > > Now it has an issue with the field "RMSE" from the output file.
See
> > > attached log.
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Wednesday, April 13, 2016 4:57 PM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > running series-analysis
> > >
> > > All active links contained in this email were disabled. Please
verify
> > the
> > > identity of the sender, and confirm the authenticity of all
links
> > contained
> > > within the message prior to copying and pasting the address to a
Web
> > > browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > Also, I see the following error message in the log file you
sent:
> > >
> > > ERROR : store_stat_cnt() -> unsupported column name requested
"IQR"
> > >
> > > IQR was added in version 5.1, but it not defined in version 4.1.
> > >
> > > John
> > >
> > > On Wed, Apr 13, 2016 at 4:55 PM, John Halley Gotway
<johnhg at ucar.edu>
> > > wrote:
> > >
> > > > John,
> > > >
> > > > Ah yes, I meant to mention this. Please try increasing the
block
> size
> > > > from 1024 to something much larger.
> > > >
> > > > Your grid is 288x288 = 82944 grid points. Try setting:
> > > > block_size = 100000;
> > > >
> > > > That will process it all in one pass through the data and
should make
> > it
> > > > run much faster.
> > > >
> > > > John
> > > >
> > > > On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >>
> > > >> <Caution-Caution-Caution-Caution-url: Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >>
> > > >> John -
> > > >>
> > > >> I looked at your config file and it looks like a nice
workaround. I
> > > reran
> > > >> using your config file. Ran considerably longer this time. I
also
> > have a
> > > >> 14MB output NetCDF file which is definitely encouraging! I
have to
> > leave
> > > >> for the day in about 10 mins and will be back in touch
tomorrow.
> > Thanks
> > > for
> > > >> your help.
> > > >>
> > > >> R/
> > > >> John
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> ________________________________________
> > > >> From: John Halley Gotway via RT [met_help at ucar.edu]
> > > >> Sent: Wednesday, April 13, 2016 4:37 PM
> > > >> To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > >> running series-analysis
> > > >>
> > > >> All active links contained in this email were disabled.
Please
> verify
> > > >> the identity of the sender, and confirm the authenticity of
all
> links
> > > >> contained within the message prior to copying and pasting the
> address
> > > to a
> > > >> Web browser.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> ----
> > > >>
> > > >> John,
> > > >>
> > > >> Ah yes, that time offset is the problem here.
> > > >>
> > > >> And it would have been nice for Series-Analysis to be more
verbose
> > about
> > > >> the timing information it's processing.
> > > >>
> > > >> Please try using the updated config file that I've attached,
and see
> > if
> > > >> Series-Analysis actually runs.
> > > >>
> > > >> We pass grid-stat 1 forecast file and 1 observation file. So
even
> if
> > > the
> > > >> times don't match up exactly, it writes a warning and
compares them.
> > > >> Series-Analysis tries to be a bit fancier. You pass it N
forecast
> > and N
> > > >> observation files. For each forecast files, it searches all
the
> > > >> observation files for a match.
> > > >>
> > > >> But that more complex logic breaks down when there's a 15
minute
> time
> > > >> offset. So we should think about how better to handle this
use
> case.
> > > >>
> > > >> Thanks,
> > > >> John
> > > >>
> > > >> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via RT
<
> > > >> met_help at ucar.edu> wrote:
> > > >>
> > > >> >
> > > >> > <Caution-Caution-Caution-Caution-Caution-url: Caution-
> > > >> Caution-Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >> >
> > > >> > John -
> > > >> >
> > > >> > Based on your analysis of the reading of the input files, I
can
> > offer
> > > a
> > > >> > possibility that I'm aware of regarding the valid times.The
hours
> > for
> > > >> the
> > > >> > example input data (observations) I sent you is indeed 23
which
> > > matches
> > > >> > that of the forecast data valid at the same hour (23), but
the
> > minutes
> > > >> of
> > > >> > the forecast file is 00 while for the observations it is
15. I was
> > > >> aware of
> > > >> > this mismatch in the minutes from when I ran Grid-Stat and
it
> > flagged
> > > >> it as
> > > >> > a "Warning", but it went ahead and generated output with no
Error.
> > The
> > > >> > presence of the min value of 15 is an artifact of the fact
that
> > this
> > > >> data
> > > >> > was produced by the LAPS system. The valid time of the data
is
> > > actually
> > > >> > 2300 instead of 2315. That said, I will run wgrib on the
forecast
> > file
> > > >> per
> > > >> > your suggestion to confirm the valid time.
> > > >> > R/
> > > >> > John
> > > >> >
> > > >> > ________________________________________
> > > >> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > >> > Sent: Wednesday, April 13, 2016 3:48 PM
> > > >> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > >> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
> problem
> > > >> > running series-analysis
> > > >> >
> > > >> > All active links contained in this email were disabled.
Please
> > verify
> > > >> the
> > > >> > identity of the sender, and confirm the authenticity of all
links
> > > >> contained
> > > >> > within the message prior to copying and pasting the address
to a
> Web
> > > >> > browser.
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > ----
> > > >> >
> > > >> > John,
> > > >> >
> > > >> > I don't see any immediately obvious problems in the data
you sent
> to
> > > >> us. I
> > > >> > see that you're writing a log file named:
> > > >> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
> > > >> >
> > > >> > Are you able to send me that complete log file? That might
help
> me
> > > >> > diagnose what's going on.
> > > >> >
> > > >> > Here's my initial guess...
> > > >> >
> > > >> > You are running series-analysis by defining a single field
in the
> > > >> > configuration file. And you're passing to it lists of 12
forecast
> > > files
> > > >> > and 12 observation files.
> > > >> >
> > > >> > It reads TMP/Z2 from the first forecast file and check the
valid
> > time
> > > >> for
> > > >> > that data. Next, it looks in the observation file list
(starting
> > with
> > > >> the
> > > >> > first file) for TMP/Z2 with the *SAME* valid time.
> > > >> >
> > > >> > Next, it reads TMP/Z2 from the second forecast file, gets
the
> valid
> > > >> time,
> > > >> > and looks in the observation file list (starting with the
second
> > file)
> > > >> for
> > > >> > TMP/Z2 with the *SAME* valid time.
> > > >> >
> > > >> > I looked at the observation file (120382300_laps_288.grb1)
you
> sent
> > me
> > > >> and
> > > >> > see that it's valid time is 2012020723, which happens to
match the
> > > >> > timestamp in the forecast file name. The first thing I'd
check is
> > the
> > > >> > valid time of the corresponding forecast file:
> > > >> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
> > > >> >
> > > >> > Is it actually valid at the same time, 2012020723? If not,
that's
> > the
> > > >> > problem. If so, that's not the problem.
> > > >> >
> > > >> > Thanks,
> > > >> > John
> > > >> >
> > > >> >
> > > >> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via
RT <
> > > >> > met_help at ucar.edu> wrote:
> > > >> >
> > > >> > >
> > > >> > > <Caution-Caution-Caution-Caution-Caution-Caution-url:
Caution-
> > > >> > Caution-Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >> > >
> > > >> > > Hi John -
> > > >> > >
> > > >> > > Yes. I got stuck on trying to figure out the cause of
that
> error.
> > > >> Seems
> > > >> > > like if Grid-Stat can read the 2-m temperature from the
gridded
> > obs,
> > > >> then
> > > >> > > why would Series-Analysis not be able to read it?
> > > >> > >
> > > >> > > Thanks.
> > > >> > >
> > > >> > > R/
> > > >> > > John
> > > >> > >
> > > >> > >
> > > >> > > -----Original Message-----
> > > >> > > From: John Halley Gotway via RT
> > [Caution-Caution-Caution-Caution-Caution-Caution-mailto:
> > > >> met_help at ucar.edu]
> > > >> > > Sent: Wednesday, April 13, 2016 12:34 PM
> > > >> > > To: Raby, John W CIV USARMY RDECOM ARL (US) <
> > > >> john.w.raby2.civ at mail.mil>
> > > >> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > >> running
> > > >> > > series-analysis
> > > >> > >
> > > >> > > All active links contained in this email were disabled.
Please
> > > verify
> > > >> > the
> > > >> > > identity of the sender, and confirm the authenticity of
all
> links
> > > >> > contained
> > > >> > > within the message prior to copying and pasting the
address to a
> > Web
> > > >> > > browser.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > ----
> > > >> > >
> > > >> > > Hello John,
> > > >> > >
> > > >> > > I see that you were having trouble running series-
analysis on
> 2-m
> > > >> > > temperature data last week.
> > > >> > >
> > > >> > > Is this still an issue? If so, I'd be happy to take a
look.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > John Halley Gotway
> > > >> > >
> > > >> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT <
> > > >> met_help at ucar.edu
> > > >> > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > >
> > > >> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
> > > >> > > > Transaction: Given to johnhg (John Halley Gotway) by
bullock
> > > >> > > > Queue: met_help
> > > >> > > > Subject: problem running series-analysis
> > > >> > > > Owner: johnhg
> > > >> > > > Requestors: john.w.raby2.civ at mail.mil
> > > >> > > > Status: open
> > > >> > > > Ticket <Caution-Caution-Caution-Caution-Caution-
Caution-Caution-url:
> > > >> > > > Caution-Caution-
> > > >> > Caution-Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >> > > >
> > > >> > > >
> > > >> > > > This transaction appears to have no content
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem running series-analysis
From: Raby, John W USA CIV
Time: Wed Apr 20 07:12:59 2016
John -
The series-analysis run I started yesterday is still running. Judging
from the screen output, the grid position info which changes at ~ 1sec
intervals to show the point on which the processing is occurring
(categorical and continuous statistics), it looks like it is about 40%
complete.I estimate that it will complete early tomorrow morning.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Tuesday, April 19, 2016 11:56 AM
To: Raby, John W CIV USARMY RDECOM ARL (US)
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running series-analysis
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
Sure, you can use my email address, johnhg at ucar.edu. Although looking
in
the "fcst_file_list" I see you're running Series-Analysis on 12 files.
With just a single fcst/obs file, running it on a series of length 1
won't
be all that informative.
For now please just send me a verbose log file for your run and I'll
look
to see if issues jump out.
Thanks,
John
On Tue, Apr 19, 2016 at 11:51 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> Thanks for taking a look at that. For large sized files, my only
method of
> transfer to you is by the ARL Secure Access File Exchange (SAFE)
which
> requires an email address to which they send you a link to download
the
> file. FTP is blocked here at WSMR. In the past, I have used your
email
> address @ucar.edu. Will this still work? If so, do you also want the
> forecast file which has the same valid time (almost-2300Z) as the
> observations file I sent you which was valid at 2315Z on 07 FEB
2012?
>
> I don't have ncview running here. I did try visualizing the output
file
> using the Unidata IDV software and none of the fields looked right
to me.
>
> I'll start working on a re-run of Series-Analysis to generate the
log per
> your instructions.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Tuesday, April 19, 2016 9:45 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> I agree with you. That output doesn't look good at all!
>
> Please just open up the NetCDF output file from Series-Analysis
using the
> "ncview" tool. Click through the variables and check whether any of
them
> look reasonable. If they look OK, then the problem is in plotting
them
> with plot_data_plane. If so, please just post that Series-Analysis
output
> file to our anonymous ftp site? And I'll figure out what's going on
with
> the plotting.
>
> But my guess is that none of the fields look good, which means the
issue is
> coming from Series-Analysis. Please try rerunning your Series-
Analysis
> command at verbosity level 5 (-v 5) and then save the output to a
log file
> (-log run_series_analysis.log). Then send me that log file.
>
> Thanks,
> John
>
> On Mon, Apr 18, 2016 at 4:13 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url: Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> > John -
> >
> > I used the following command to plot the variable:
> series_cts_CSI_ge285.000
> > carson:jraby ~/MET_SeriesAnalysis
> > > plot_data_plane results_m3o3_TMP results_m3o3_TMP_CSI.ps
> > 'name="series_cts_CSI_ge285.000"; level="(*,*)";'
> > DEBUG 1: Opening data file: results_m3o3_TMP
> > DEBUG 1: Creating postscript file: results_m3o3_TMP_CSI.ps
> > carson:jraby ~/MET_SeriesAnalysis
> >
> > Seem to run fine, but the plot doesn't look like I expected.
> >
> > R/
> > John
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Monday, April 18, 2016 1:41 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> > running series-analysis
> >
> > All active links contained in this email were disabled. Please
verify
> the
> > identity of the sender, and confirm the authenticity of all links
> contained
> > within the message prior to copying and pasting the address to a
Web
> > browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > I see the following error in the log file you sent me:
> > ERROR :
> > ERROR : plot_data_plane -> trouble getting field "name="RMSE";
> > level="Z2";" from file "results_m3o3_TMP"
> > ERROR :
> >
> > You are calling plot_data_plane by passing it a NetCDF file to
plot. The
> > way you set "name" and "level" is different between GRIB files and
> gridded
> > NetCDF files. For NetCDF files, you set the name equal to the
name of
> the
> > NetCDF variable you want to plot and you set level to "(*,*)". I
know
> that
> > looks weird but you're telling MET which NetCDF dimensions are the
> gridded
> > ones that it should plot.
> >
> > In your case, the output of Series-Analysis will only have 2
> dimensions...
> > thus "(*,*)".
> >
> > Just use "ncdump -h" to look at the name of the variables and run
> > plot_data_plane something like this:
> >
> > plot_data_plane results_m3o3_TMP results_m3o3_TMP_RMSE.ps
> > 'name="series_cnt_RMSE"; level="(*,*)";'
> >
> > Hope that helps clarify.
> >
> > Thanks,
> > John
> >
> > On Thu, Apr 14, 2016 at 9:14 AM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url: Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >
> > > John -
> > >
> > > I resolved the issue with whitespace by using the name "RMSE"
vice
> "Root
> > > mean squared error".
> > >
> > > Now it has an issue with the field "RMSE" from the output file.
See
> > > attached log.
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Wednesday, April 13, 2016 4:57 PM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > running series-analysis
> > >
> > > All active links contained in this email were disabled. Please
verify
> > the
> > > identity of the sender, and confirm the authenticity of all
links
> > contained
> > > within the message prior to copying and pasting the address to a
Web
> > > browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > Also, I see the following error message in the log file you
sent:
> > >
> > > ERROR : store_stat_cnt() -> unsupported column name requested
"IQR"
> > >
> > > IQR was added in version 5.1, but it not defined in version 4.1.
> > >
> > > John
> > >
> > > On Wed, Apr 13, 2016 at 4:55 PM, John Halley Gotway
<johnhg at ucar.edu>
> > > wrote:
> > >
> > > > John,
> > > >
> > > > Ah yes, I meant to mention this. Please try increasing the
block
> size
> > > > from 1024 to something much larger.
> > > >
> > > > Your grid is 288x288 = 82944 grid points. Try setting:
> > > > block_size = 100000;
> > > >
> > > > That will process it all in one pass through the data and
should make
> > it
> > > > run much faster.
> > > >
> > > > John
> > > >
> > > > On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >>
> > > >> <Caution-Caution-Caution-Caution-url: Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >>
> > > >> John -
> > > >>
> > > >> I looked at your config file and it looks like a nice
workaround. I
> > > reran
> > > >> using your config file. Ran considerably longer this time. I
also
> > have a
> > > >> 14MB output NetCDF file which is definitely encouraging! I
have to
> > leave
> > > >> for the day in about 10 mins and will be back in touch
tomorrow.
> > Thanks
> > > for
> > > >> your help.
> > > >>
> > > >> R/
> > > >> John
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> ________________________________________
> > > >> From: John Halley Gotway via RT [met_help at ucar.edu]
> > > >> Sent: Wednesday, April 13, 2016 4:37 PM
> > > >> To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > >> running series-analysis
> > > >>
> > > >> All active links contained in this email were disabled.
Please
> verify
> > > >> the identity of the sender, and confirm the authenticity of
all
> links
> > > >> contained within the message prior to copying and pasting the
> address
> > > to a
> > > >> Web browser.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> ----
> > > >>
> > > >> John,
> > > >>
> > > >> Ah yes, that time offset is the problem here.
> > > >>
> > > >> And it would have been nice for Series-Analysis to be more
verbose
> > about
> > > >> the timing information it's processing.
> > > >>
> > > >> Please try using the updated config file that I've attached,
and see
> > if
> > > >> Series-Analysis actually runs.
> > > >>
> > > >> We pass grid-stat 1 forecast file and 1 observation file. So
even
> if
> > > the
> > > >> times don't match up exactly, it writes a warning and
compares them.
> > > >> Series-Analysis tries to be a bit fancier. You pass it N
forecast
> > and N
> > > >> observation files. For each forecast files, it searches all
the
> > > >> observation files for a match.
> > > >>
> > > >> But that more complex logic breaks down when there's a 15
minute
> time
> > > >> offset. So we should think about how better to handle this
use
> case.
> > > >>
> > > >> Thanks,
> > > >> John
> > > >>
> > > >> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via RT
<
> > > >> met_help at ucar.edu> wrote:
> > > >>
> > > >> >
> > > >> > <Caution-Caution-Caution-Caution-Caution-url: Caution-
> > > >> Caution-Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >> >
> > > >> > John -
> > > >> >
> > > >> > Based on your analysis of the reading of the input files, I
can
> > offer
> > > a
> > > >> > possibility that I'm aware of regarding the valid times.The
hours
> > for
> > > >> the
> > > >> > example input data (observations) I sent you is indeed 23
which
> > > matches
> > > >> > that of the forecast data valid at the same hour (23), but
the
> > minutes
> > > >> of
> > > >> > the forecast file is 00 while for the observations it is
15. I was
> > > >> aware of
> > > >> > this mismatch in the minutes from when I ran Grid-Stat and
it
> > flagged
> > > >> it as
> > > >> > a "Warning", but it went ahead and generated output with no
Error.
> > The
> > > >> > presence of the min value of 15 is an artifact of the fact
that
> > this
> > > >> data
> > > >> > was produced by the LAPS system. The valid time of the data
is
> > > actually
> > > >> > 2300 instead of 2315. That said, I will run wgrib on the
forecast
> > file
> > > >> per
> > > >> > your suggestion to confirm the valid time.
> > > >> > R/
> > > >> > John
> > > >> >
> > > >> > ________________________________________
> > > >> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > >> > Sent: Wednesday, April 13, 2016 3:48 PM
> > > >> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > >> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
> problem
> > > >> > running series-analysis
> > > >> >
> > > >> > All active links contained in this email were disabled.
Please
> > verify
> > > >> the
> > > >> > identity of the sender, and confirm the authenticity of all
links
> > > >> contained
> > > >> > within the message prior to copying and pasting the address
to a
> Web
> > > >> > browser.
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > ----
> > > >> >
> > > >> > John,
> > > >> >
> > > >> > I don't see any immediately obvious problems in the data
you sent
> to
> > > >> us. I
> > > >> > see that you're writing a log file named:
> > > >> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
> > > >> >
> > > >> > Are you able to send me that complete log file? That might
help
> me
> > > >> > diagnose what's going on.
> > > >> >
> > > >> > Here's my initial guess...
> > > >> >
> > > >> > You are running series-analysis by defining a single field
in the
> > > >> > configuration file. And you're passing to it lists of 12
forecast
> > > files
> > > >> > and 12 observation files.
> > > >> >
> > > >> > It reads TMP/Z2 from the first forecast file and check the
valid
> > time
> > > >> for
> > > >> > that data. Next, it looks in the observation file list
(starting
> > with
> > > >> the
> > > >> > first file) for TMP/Z2 with the *SAME* valid time.
> > > >> >
> > > >> > Next, it reads TMP/Z2 from the second forecast file, gets
the
> valid
> > > >> time,
> > > >> > and looks in the observation file list (starting with the
second
> > file)
> > > >> for
> > > >> > TMP/Z2 with the *SAME* valid time.
> > > >> >
> > > >> > I looked at the observation file (120382300_laps_288.grb1)
you
> sent
> > me
> > > >> and
> > > >> > see that it's valid time is 2012020723, which happens to
match the
> > > >> > timestamp in the forecast file name. The first thing I'd
check is
> > the
> > > >> > valid time of the corresponding forecast file:
> > > >> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
> > > >> >
> > > >> > Is it actually valid at the same time, 2012020723? If not,
that's
> > the
> > > >> > problem. If so, that's not the problem.
> > > >> >
> > > >> > Thanks,
> > > >> > John
> > > >> >
> > > >> >
> > > >> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via
RT <
> > > >> > met_help at ucar.edu> wrote:
> > > >> >
> > > >> > >
> > > >> > > <Caution-Caution-Caution-Caution-Caution-Caution-url:
Caution-
> > > >> > Caution-Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >> > >
> > > >> > > Hi John -
> > > >> > >
> > > >> > > Yes. I got stuck on trying to figure out the cause of
that
> error.
> > > >> Seems
> > > >> > > like if Grid-Stat can read the 2-m temperature from the
gridded
> > obs,
> > > >> then
> > > >> > > why would Series-Analysis not be able to read it?
> > > >> > >
> > > >> > > Thanks.
> > > >> > >
> > > >> > > R/
> > > >> > > John
> > > >> > >
> > > >> > >
> > > >> > > -----Original Message-----
> > > >> > > From: John Halley Gotway via RT
> > [Caution-Caution-Caution-Caution-Caution-Caution-mailto:
> > > >> met_help at ucar.edu]
> > > >> > > Sent: Wednesday, April 13, 2016 12:34 PM
> > > >> > > To: Raby, John W CIV USARMY RDECOM ARL (US) <
> > > >> john.w.raby2.civ at mail.mil>
> > > >> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > >> running
> > > >> > > series-analysis
> > > >> > >
> > > >> > > All active links contained in this email were disabled.
Please
> > > verify
> > > >> > the
> > > >> > > identity of the sender, and confirm the authenticity of
all
> links
> > > >> > contained
> > > >> > > within the message prior to copying and pasting the
address to a
> > Web
> > > >> > > browser.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > ----
> > > >> > >
> > > >> > > Hello John,
> > > >> > >
> > > >> > > I see that you were having trouble running series-
analysis on
> 2-m
> > > >> > > temperature data last week.
> > > >> > >
> > > >> > > Is this still an issue? If so, I'd be happy to take a
look.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > John Halley Gotway
> > > >> > >
> > > >> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT <
> > > >> met_help at ucar.edu
> > > >> > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > >
> > > >> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted upon.
> > > >> > > > Transaction: Given to johnhg (John Halley Gotway) by
bullock
> > > >> > > > Queue: met_help
> > > >> > > > Subject: problem running series-analysis
> > > >> > > > Owner: johnhg
> > > >> > > > Requestors: john.w.raby2.civ at mail.mil
> > > >> > > > Status: open
> > > >> > > > Ticket <Caution-Caution-Caution-Caution-Caution-
Caution-Caution-url:
> > > >> > > > Caution-Caution-
> > > >> > Caution-Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >> > > >
> > > >> > > >
> > > >> > > > This transaction appears to have no content
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: problem running series-analysis
From: John Halley Gotway
Time: Wed Apr 20 10:47:59 2016
John,
Running this long is obviously not the desired behavior! Is this
really
just being run on series of length 12 and a grid with dimension
288x288?
If so, this should run in minutes, not hours.
Please take a look at the following settings in the Series-Analysis
config
file...
(1) In the "boot" section, set:
n_rep = 0;
That will turn off the computation of bootstrap CI's which is slow.
(2) Check to see how "block_size" is set. The default is 1024. This
is
the number of grid points to process concurrently. Using the default
value
of 1024 will require 81 (288*288/104) passes through those files. And
that'll slow it down a lot. Setting it to 82944 will enable it to
process
your data in a single pass (much faster). BUT if your computer
doesn't
have enough memory, it'll start using swap space and make it run
REALLY
slowly. If you have it set high, try reducing it. If it's set low,
try
increasing it.
(3) Set "rank_corr_flag = FALSE;" to disable rank correlation
statistics
which can be slow.
John
On Wed, Apr 20, 2016 at 7:12 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> The series-analysis run I started yesterday is still running.
Judging from
> the screen output, the grid position info which changes at ~ 1sec
intervals
> to show the point on which the processing is occurring (categorical
and
> continuous statistics), it looks like it is about 40% complete.I
estimate
> that it will complete early tomorrow morning.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Tuesday, April 19, 2016 11:56 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Sure, you can use my email address, johnhg at ucar.edu. Although
looking in
> the "fcst_file_list" I see you're running Series-Analysis on 12
files.
> With just a single fcst/obs file, running it on a series of length 1
won't
> be all that informative.
>
> For now please just send me a verbose log file for your run and I'll
look
> to see if issues jump out.
>
> Thanks,
> John
>
> On Tue, Apr 19, 2016 at 11:51 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url: Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> > John -
> >
> > Thanks for taking a look at that. For large sized files, my only
method
> of
> > transfer to you is by the ARL Secure Access File Exchange (SAFE)
which
> > requires an email address to which they send you a link to
download the
> > file. FTP is blocked here at WSMR. In the past, I have used your
email
> > address @ucar.edu. Will this still work? If so, do you also want
the
> > forecast file which has the same valid time (almost-2300Z) as the
> > observations file I sent you which was valid at 2315Z on 07 FEB
2012?
> >
> > I don't have ncview running here. I did try visualizing the output
file
> > using the Unidata IDV software and none of the fields looked right
to me.
> >
> > I'll start working on a re-run of Series-Analysis to generate the
log per
> > your instructions.
> >
> > R/
> > John
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Tuesday, April 19, 2016 9:45 AM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> > running series-analysis
> >
> > All active links contained in this email were disabled. Please
verify
> the
> > identity of the sender, and confirm the authenticity of all links
> contained
> > within the message prior to copying and pasting the address to a
Web
> > browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > I agree with you. That output doesn't look good at all!
> >
> > Please just open up the NetCDF output file from Series-Analysis
using the
> > "ncview" tool. Click through the variables and check whether any
of them
> > look reasonable. If they look OK, then the problem is in plotting
them
> > with plot_data_plane. If so, please just post that Series-
Analysis
> output
> > file to our anonymous ftp site? And I'll figure out what's going
on with
> > the plotting.
> >
> > But my guess is that none of the fields look good, which means the
issue
> is
> > coming from Series-Analysis. Please try rerunning your Series-
Analysis
> > command at verbosity level 5 (-v 5) and then save the output to a
log
> file
> > (-log run_series_analysis.log). Then send me that log file.
> >
> > Thanks,
> > John
> >
> > On Mon, Apr 18, 2016 at 4:13 PM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-url: Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >
> > > John -
> > >
> > > I used the following command to plot the variable:
> > series_cts_CSI_ge285.000
> > > carson:jraby ~/MET_SeriesAnalysis
> > > > plot_data_plane results_m3o3_TMP results_m3o3_TMP_CSI.ps
> > > 'name="series_cts_CSI_ge285.000"; level="(*,*)";'
> > > DEBUG 1: Opening data file: results_m3o3_TMP
> > > DEBUG 1: Creating postscript file: results_m3o3_TMP_CSI.ps
> > > carson:jraby ~/MET_SeriesAnalysis
> > >
> > > Seem to run fine, but the plot doesn't look like I expected.
> > >
> > > R/
> > > John
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Monday, April 18, 2016 1:41 PM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > running series-analysis
> > >
> > > All active links contained in this email were disabled. Please
verify
> > the
> > > identity of the sender, and confirm the authenticity of all
links
> > contained
> > > within the message prior to copying and pasting the address to a
Web
> > > browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > I see the following error in the log file you sent me:
> > > ERROR :
> > > ERROR : plot_data_plane -> trouble getting field "name="RMSE";
> > > level="Z2";" from file "results_m3o3_TMP"
> > > ERROR :
> > >
> > > You are calling plot_data_plane by passing it a NetCDF file to
plot.
> The
> > > way you set "name" and "level" is different between GRIB files
and
> > gridded
> > > NetCDF files. For NetCDF files, you set the name equal to the
name of
> > the
> > > NetCDF variable you want to plot and you set level to "(*,*)".
I know
> > that
> > > looks weird but you're telling MET which NetCDF dimensions are
the
> > gridded
> > > ones that it should plot.
> > >
> > > In your case, the output of Series-Analysis will only have 2
> > dimensions...
> > > thus "(*,*)".
> > >
> > > Just use "ncdump -h" to look at the name of the variables and
run
> > > plot_data_plane something like this:
> > >
> > > plot_data_plane results_m3o3_TMP results_m3o3_TMP_RMSE.ps
> > > 'name="series_cnt_RMSE"; level="(*,*)";'
> > >
> > > Hope that helps clarify.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Apr 14, 2016 at 9:14 AM, Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <Caution-Caution-Caution-url: Caution-
> > > Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >
> > > > John -
> > > >
> > > > I resolved the issue with whitespace by using the name "RMSE"
vice
> > "Root
> > > > mean squared error".
> > > >
> > > > Now it has an issue with the field "RMSE" from the output
file. See
> > > > attached log.
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > >
> > > > ________________________________________
> > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > Sent: Wednesday, April 13, 2016 4:57 PM
> > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > > running series-analysis
> > > >
> > > > All active links contained in this email were disabled.
Please
> verify
> > > the
> > > > identity of the sender, and confirm the authenticity of all
links
> > > contained
> > > > within the message prior to copying and pasting the address to
a Web
> > > > browser.
> > > >
> > > >
> > > >
> > > >
> > > > ----
> > > >
> > > > Also, I see the following error message in the log file you
sent:
> > > >
> > > > ERROR : store_stat_cnt() -> unsupported column name requested
"IQR"
> > > >
> > > > IQR was added in version 5.1, but it not defined in version
4.1.
> > > >
> > > > John
> > > >
> > > > On Wed, Apr 13, 2016 at 4:55 PM, John Halley Gotway
<johnhg at ucar.edu
> >
> > > > wrote:
> > > >
> > > > > John,
> > > > >
> > > > > Ah yes, I meant to mention this. Please try increasing the
block
> > size
> > > > > from 1024 to something much larger.
> > > > >
> > > > > Your grid is 288x288 = 82944 grid points. Try setting:
> > > > > block_size = 100000;
> > > > >
> > > > > That will process it all in one pass through the data and
should
> make
> > > it
> > > > > run much faster.
> > > > >
> > > > > John
> > > > >
> > > > > On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV via RT
<
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > >>
> > > > >> <Caution-Caution-Caution-Caution-url: Caution-
> > > > Caution-Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > >>
> > > > >> John -
> > > > >>
> > > > >> I looked at your config file and it looks like a nice
workaround.
> I
> > > > reran
> > > > >> using your config file. Ran considerably longer this time.
I also
> > > have a
> > > > >> 14MB output NetCDF file which is definitely encouraging! I
have to
> > > leave
> > > > >> for the day in about 10 mins and will be back in touch
tomorrow.
> > > Thanks
> > > > for
> > > > >> your help.
> > > > >>
> > > > >> R/
> > > > >> John
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> ________________________________________
> > > > >> From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > >> Sent: Wednesday, April 13, 2016 4:37 PM
> > > > >> To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
> problem
> > > > >> running series-analysis
> > > > >>
> > > > >> All active links contained in this email were disabled.
Please
> > verify
> > > > >> the identity of the sender, and confirm the authenticity of
all
> > links
> > > > >> contained within the message prior to copying and pasting
the
> > address
> > > > to a
> > > > >> Web browser.
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> ----
> > > > >>
> > > > >> John,
> > > > >>
> > > > >> Ah yes, that time offset is the problem here.
> > > > >>
> > > > >> And it would have been nice for Series-Analysis to be more
verbose
> > > about
> > > > >> the timing information it's processing.
> > > > >>
> > > > >> Please try using the updated config file that I've
attached, and
> see
> > > if
> > > > >> Series-Analysis actually runs.
> > > > >>
> > > > >> We pass grid-stat 1 forecast file and 1 observation file.
So even
> > if
> > > > the
> > > > >> times don't match up exactly, it writes a warning and
compares
> them.
> > > > >> Series-Analysis tries to be a bit fancier. You pass it N
forecast
> > > and N
> > > > >> observation files. For each forecast files, it searches
all the
> > > > >> observation files for a match.
> > > > >>
> > > > >> But that more complex logic breaks down when there's a 15
minute
> > time
> > > > >> offset. So we should think about how better to handle this
use
> > case.
> > > > >>
> > > > >> Thanks,
> > > > >> John
> > > > >>
> > > > >> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via
RT <
> > > > >> met_help at ucar.edu> wrote:
> > > > >>
> > > > >> >
> > > > >> > <Caution-Caution-Caution-Caution-Caution-url: Caution-
> > > > >> Caution-Caution-
> > > Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > >> >
> > > > >> > John -
> > > > >> >
> > > > >> > Based on your analysis of the reading of the input files,
I can
> > > offer
> > > > a
> > > > >> > possibility that I'm aware of regarding the valid
times.The
> hours
> > > for
> > > > >> the
> > > > >> > example input data (observations) I sent you is indeed 23
which
> > > > matches
> > > > >> > that of the forecast data valid at the same hour (23),
but the
> > > minutes
> > > > >> of
> > > > >> > the forecast file is 00 while for the observations it is
15. I
> was
> > > > >> aware of
> > > > >> > this mismatch in the minutes from when I ran Grid-Stat
and it
> > > flagged
> > > > >> it as
> > > > >> > a "Warning", but it went ahead and generated output with
no
> Error.
> > > The
> > > > >> > presence of the min value of 15 is an artifact of the
fact that
> > > this
> > > > >> data
> > > > >> > was produced by the LAPS system. The valid time of the
data is
> > > > actually
> > > > >> > 2300 instead of 2315. That said, I will run wgrib on the
> forecast
> > > file
> > > > >> per
> > > > >> > your suggestion to confirm the valid time.
> > > > >> > R/
> > > > >> > John
> > > > >> >
> > > > >> > ________________________________________
> > > > >> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > >> > Sent: Wednesday, April 13, 2016 3:48 PM
> > > > >> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > >> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#75766]
> > problem
> > > > >> > running series-analysis
> > > > >> >
> > > > >> > All active links contained in this email were disabled.
Please
> > > verify
> > > > >> the
> > > > >> > identity of the sender, and confirm the authenticity of
all
> links
> > > > >> contained
> > > > >> > within the message prior to copying and pasting the
address to a
> > Web
> > > > >> > browser.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > ----
> > > > >> >
> > > > >> > John,
> > > > >> >
> > > > >> > I don't see any immediately obvious problems in the data
you
> sent
> > to
> > > > >> us. I
> > > > >> > see that you're writing a log file named:
> > > > >> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
> > > > >> >
> > > > >> > Are you able to send me that complete log file? That
might help
> > me
> > > > >> > diagnose what's going on.
> > > > >> >
> > > > >> > Here's my initial guess...
> > > > >> >
> > > > >> > You are running series-analysis by defining a single
field in
> the
> > > > >> > configuration file. And you're passing to it lists of 12
> forecast
> > > > files
> > > > >> > and 12 observation files.
> > > > >> >
> > > > >> > It reads TMP/Z2 from the first forecast file and check
the valid
> > > time
> > > > >> for
> > > > >> > that data. Next, it looks in the observation file list
> (starting
> > > with
> > > > >> the
> > > > >> > first file) for TMP/Z2 with the *SAME* valid time.
> > > > >> >
> > > > >> > Next, it reads TMP/Z2 from the second forecast file, gets
the
> > valid
> > > > >> time,
> > > > >> > and looks in the observation file list (starting with the
second
> > > file)
> > > > >> for
> > > > >> > TMP/Z2 with the *SAME* valid time.
> > > > >> >
> > > > >> > I looked at the observation file
(120382300_laps_288.grb1) you
> > sent
> > > me
> > > > >> and
> > > > >> > see that it's valid time is 2012020723, which happens to
match
> the
> > > > >> > timestamp in the forecast file name. The first thing I'd
check
> is
> > > the
> > > > >> > valid time of the corresponding forecast file:
> > > > >> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
> > > > >> >
> > > > >> > Is it actually valid at the same time, 2012020723? If
not,
> that's
> > > the
> > > > >> > problem. If so, that's not the problem.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > John
> > > > >> >
> > > > >> >
> > > > >> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via
RT <
> > > > >> > met_help at ucar.edu> wrote:
> > > > >> >
> > > > >> > >
> > > > >> > > <Caution-Caution-Caution-Caution-Caution-Caution-url:
Caution-
> > > > >> > Caution-Caution-
> > > > Caution-Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > >> > >
> > > > >> > > Hi John -
> > > > >> > >
> > > > >> > > Yes. I got stuck on trying to figure out the cause of
that
> > error.
> > > > >> Seems
> > > > >> > > like if Grid-Stat can read the 2-m temperature from the
> gridded
> > > obs,
> > > > >> then
> > > > >> > > why would Series-Analysis not be able to read it?
> > > > >> > >
> > > > >> > > Thanks.
> > > > >> > >
> > > > >> > > R/
> > > > >> > > John
> > > > >> > >
> > > > >> > >
> > > > >> > > -----Original Message-----
> > > > >> > > From: John Halley Gotway via RT
> > > [Caution-Caution-Caution-Caution-Caution-Caution-mailto:
> > > > >> met_help at ucar.edu]
> > > > >> > > Sent: Wednesday, April 13, 2016 12:34 PM
> > > > >> > > To: Raby, John W CIV USARMY RDECOM ARL (US) <
> > > > >> john.w.raby2.civ at mail.mil>
> > > > >> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
> problem
> > > > >> running
> > > > >> > > series-analysis
> > > > >> > >
> > > > >> > > All active links contained in this email were disabled.
> Please
> > > > verify
> > > > >> > the
> > > > >> > > identity of the sender, and confirm the authenticity of
all
> > links
> > > > >> > contained
> > > > >> > > within the message prior to copying and pasting the
address
> to a
> > > Web
> > > > >> > > browser.
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > ----
> > > > >> > >
> > > > >> > > Hello John,
> > > > >> > >
> > > > >> > > I see that you were having trouble running series-
analysis on
> > 2-m
> > > > >> > > temperature data last week.
> > > > >> > >
> > > > >> > > Is this still an issue? If so, I'd be happy to take a
look.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > John Halley Gotway
> > > > >> > >
> > > > >> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT <
> > > > >> met_help at ucar.edu
> > > > >> > >
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > >
> > > > >> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted
upon.
> > > > >> > > > Transaction: Given to johnhg (John Halley Gotway) by
bullock
> > > > >> > > > Queue: met_help
> > > > >> > > > Subject: problem running series-analysis
> > > > >> > > > Owner: johnhg
> > > > >> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > >> > > > Status: open
> > > > >> > > > Ticket
> <Caution-Caution-Caution-Caution-Caution-Caution-Caution-url:
> > > > >> > > > Caution-Caution-
> > > > >> > Caution-Caution-
> > > > Caution-Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > This transaction appears to have no content
> > > > >> > > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem running series-analysis
From: Raby, John W USA CIV
Time: Wed Apr 20 12:26:53 2016
John -
I checked my config file for the particular settings you mentioned:
n_rep = 1000
block_size = 100000
rank_corr_flag = TRUE
The screen dialog says it's processing "with 12 Matched pairs" and the
(i, j) values increment j, then cycle through all 287 values of i,
then increment j by one and start over incrementing i for 1-287 and
this is repeating so that now I have j=160.
My script is as follows:
series_analysis -fcst ~/MET_WRFpostprd/20120207_D_FDDA/fcst_file_list
-obs ~/MET_obs/GSD_LAPS/20120207/obs_file_list -out
~/MET_SeriesAnalysis/results_m3o3_TMP_041916 -config
~/MET_SeriesAnalysis/SeriesAnalysisConfig_m3o3_gsdlaps_LAX_JHG -log
~/MET_SeriesAnalysis/logs/run_series_analysis.log -v 5
You saw the fcst and obs file lists which I sent you.
I could send you the log file which is still being written if you want
via SAFE.
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Wednesday, April 20, 2016 10:47 AM
To: Raby, John W CIV USARMY RDECOM ARL (US)
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running series-analysis
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
Running this long is obviously not the desired behavior! Is this
really
just being run on series of length 12 and a grid with dimension
288x288?
If so, this should run in minutes, not hours.
Please take a look at the following settings in the Series-Analysis
config
file...
(1) In the "boot" section, set:
n_rep = 0;
That will turn off the computation of bootstrap CI's which is slow.
(2) Check to see how "block_size" is set. The default is 1024. This
is
the number of grid points to process concurrently. Using the default
value
of 1024 will require 81 (288*288/104) passes through those files. And
that'll slow it down a lot. Setting it to 82944 will enable it to
process
your data in a single pass (much faster). BUT if your computer
doesn't
have enough memory, it'll start using swap space and make it run
REALLY
slowly. If you have it set high, try reducing it. If it's set low,
try
increasing it.
(3) Set "rank_corr_flag = FALSE;" to disable rank correlation
statistics
which can be slow.
John
On Wed, Apr 20, 2016 at 7:12 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> The series-analysis run I started yesterday is still running.
Judging from
> the screen output, the grid position info which changes at ~ 1sec
intervals
> to show the point on which the processing is occurring (categorical
and
> continuous statistics), it looks like it is about 40% complete.I
estimate
> that it will complete early tomorrow morning.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Tuesday, April 19, 2016 11:56 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Sure, you can use my email address, johnhg at ucar.edu. Although
looking in
> the "fcst_file_list" I see you're running Series-Analysis on 12
files.
> With just a single fcst/obs file, running it on a series of length 1
won't
> be all that informative.
>
> For now please just send me a verbose log file for your run and I'll
look
> to see if issues jump out.
>
> Thanks,
> John
>
> On Tue, Apr 19, 2016 at 11:51 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url: Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> > John -
> >
> > Thanks for taking a look at that. For large sized files, my only
method
> of
> > transfer to you is by the ARL Secure Access File Exchange (SAFE)
which
> > requires an email address to which they send you a link to
download the
> > file. FTP is blocked here at WSMR. In the past, I have used your
email
> > address @ucar.edu. Will this still work? If so, do you also want
the
> > forecast file which has the same valid time (almost-2300Z) as the
> > observations file I sent you which was valid at 2315Z on 07 FEB
2012?
> >
> > I don't have ncview running here. I did try visualizing the output
file
> > using the Unidata IDV software and none of the fields looked right
to me.
> >
> > I'll start working on a re-run of Series-Analysis to generate the
log per
> > your instructions.
> >
> > R/
> > John
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Tuesday, April 19, 2016 9:45 AM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> > running series-analysis
> >
> > All active links contained in this email were disabled. Please
verify
> the
> > identity of the sender, and confirm the authenticity of all links
> contained
> > within the message prior to copying and pasting the address to a
Web
> > browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > I agree with you. That output doesn't look good at all!
> >
> > Please just open up the NetCDF output file from Series-Analysis
using the
> > "ncview" tool. Click through the variables and check whether any
of them
> > look reasonable. If they look OK, then the problem is in plotting
them
> > with plot_data_plane. If so, please just post that Series-
Analysis
> output
> > file to our anonymous ftp site? And I'll figure out what's going
on with
> > the plotting.
> >
> > But my guess is that none of the fields look good, which means the
issue
> is
> > coming from Series-Analysis. Please try rerunning your Series-
Analysis
> > command at verbosity level 5 (-v 5) and then save the output to a
log
> file
> > (-log run_series_analysis.log). Then send me that log file.
> >
> > Thanks,
> > John
> >
> > On Mon, Apr 18, 2016 at 4:13 PM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url: Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >
> > > John -
> > >
> > > I used the following command to plot the variable:
> > series_cts_CSI_ge285.000
> > > carson:jraby ~/MET_SeriesAnalysis
> > > > plot_data_plane results_m3o3_TMP results_m3o3_TMP_CSI.ps
> > > 'name="series_cts_CSI_ge285.000"; level="(*,*)";'
> > > DEBUG 1: Opening data file: results_m3o3_TMP
> > > DEBUG 1: Creating postscript file: results_m3o3_TMP_CSI.ps
> > > carson:jraby ~/MET_SeriesAnalysis
> > >
> > > Seem to run fine, but the plot doesn't look like I expected.
> > >
> > > R/
> > > John
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Monday, April 18, 2016 1:41 PM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > running series-analysis
> > >
> > > All active links contained in this email were disabled. Please
verify
> > the
> > > identity of the sender, and confirm the authenticity of all
links
> > contained
> > > within the message prior to copying and pasting the address to a
Web
> > > browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > I see the following error in the log file you sent me:
> > > ERROR :
> > > ERROR : plot_data_plane -> trouble getting field "name="RMSE";
> > > level="Z2";" from file "results_m3o3_TMP"
> > > ERROR :
> > >
> > > You are calling plot_data_plane by passing it a NetCDF file to
plot.
> The
> > > way you set "name" and "level" is different between GRIB files
and
> > gridded
> > > NetCDF files. For NetCDF files, you set the name equal to the
name of
> > the
> > > NetCDF variable you want to plot and you set level to "(*,*)".
I know
> > that
> > > looks weird but you're telling MET which NetCDF dimensions are
the
> > gridded
> > > ones that it should plot.
> > >
> > > In your case, the output of Series-Analysis will only have 2
> > dimensions...
> > > thus "(*,*)".
> > >
> > > Just use "ncdump -h" to look at the name of the variables and
run
> > > plot_data_plane something like this:
> > >
> > > plot_data_plane results_m3o3_TMP results_m3o3_TMP_RMSE.ps
> > > 'name="series_cnt_RMSE"; level="(*,*)";'
> > >
> > > Hope that helps clarify.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Apr 14, 2016 at 9:14 AM, Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <Caution-Caution-Caution-Caution-url: Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >
> > > > John -
> > > >
> > > > I resolved the issue with whitespace by using the name "RMSE"
vice
> > "Root
> > > > mean squared error".
> > > >
> > > > Now it has an issue with the field "RMSE" from the output
file. See
> > > > attached log.
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > >
> > > > ________________________________________
> > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > Sent: Wednesday, April 13, 2016 4:57 PM
> > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > > running series-analysis
> > > >
> > > > All active links contained in this email were disabled.
Please
> verify
> > > the
> > > > identity of the sender, and confirm the authenticity of all
links
> > > contained
> > > > within the message prior to copying and pasting the address to
a Web
> > > > browser.
> > > >
> > > >
> > > >
> > > >
> > > > ----
> > > >
> > > > Also, I see the following error message in the log file you
sent:
> > > >
> > > > ERROR : store_stat_cnt() -> unsupported column name requested
"IQR"
> > > >
> > > > IQR was added in version 5.1, but it not defined in version
4.1.
> > > >
> > > > John
> > > >
> > > > On Wed, Apr 13, 2016 at 4:55 PM, John Halley Gotway
<johnhg at ucar.edu
> >
> > > > wrote:
> > > >
> > > > > John,
> > > > >
> > > > > Ah yes, I meant to mention this. Please try increasing the
block
> > size
> > > > > from 1024 to something much larger.
> > > > >
> > > > > Your grid is 288x288 = 82944 grid points. Try setting:
> > > > > block_size = 100000;
> > > > >
> > > > > That will process it all in one pass through the data and
should
> make
> > > it
> > > > > run much faster.
> > > > >
> > > > > John
> > > > >
> > > > > On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV via RT
<
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > >>
> > > > >> <Caution-Caution-Caution-Caution-Caution-url: Caution-
> > > > Caution-Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > >>
> > > > >> John -
> > > > >>
> > > > >> I looked at your config file and it looks like a nice
workaround.
> I
> > > > reran
> > > > >> using your config file. Ran considerably longer this time.
I also
> > > have a
> > > > >> 14MB output NetCDF file which is definitely encouraging! I
have to
> > > leave
> > > > >> for the day in about 10 mins and will be back in touch
tomorrow.
> > > Thanks
> > > > for
> > > > >> your help.
> > > > >>
> > > > >> R/
> > > > >> John
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> ________________________________________
> > > > >> From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > >> Sent: Wednesday, April 13, 2016 4:37 PM
> > > > >> To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
> problem
> > > > >> running series-analysis
> > > > >>
> > > > >> All active links contained in this email were disabled.
Please
> > verify
> > > > >> the identity of the sender, and confirm the authenticity of
all
> > links
> > > > >> contained within the message prior to copying and pasting
the
> > address
> > > > to a
> > > > >> Web browser.
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> ----
> > > > >>
> > > > >> John,
> > > > >>
> > > > >> Ah yes, that time offset is the problem here.
> > > > >>
> > > > >> And it would have been nice for Series-Analysis to be more
verbose
> > > about
> > > > >> the timing information it's processing.
> > > > >>
> > > > >> Please try using the updated config file that I've
attached, and
> see
> > > if
> > > > >> Series-Analysis actually runs.
> > > > >>
> > > > >> We pass grid-stat 1 forecast file and 1 observation file.
So even
> > if
> > > > the
> > > > >> times don't match up exactly, it writes a warning and
compares
> them.
> > > > >> Series-Analysis tries to be a bit fancier. You pass it N
forecast
> > > and N
> > > > >> observation files. For each forecast files, it searches
all the
> > > > >> observation files for a match.
> > > > >>
> > > > >> But that more complex logic breaks down when there's a 15
minute
> > time
> > > > >> offset. So we should think about how better to handle this
use
> > case.
> > > > >>
> > > > >> Thanks,
> > > > >> John
> > > > >>
> > > > >> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via
RT <
> > > > >> met_help at ucar.edu> wrote:
> > > > >>
> > > > >> >
> > > > >> > <Caution-Caution-Caution-Caution-Caution-Caution-url:
Caution-
> > > > >> Caution-Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > >> >
> > > > >> > John -
> > > > >> >
> > > > >> > Based on your analysis of the reading of the input files,
I can
> > > offer
> > > > a
> > > > >> > possibility that I'm aware of regarding the valid
times.The
> hours
> > > for
> > > > >> the
> > > > >> > example input data (observations) I sent you is indeed 23
which
> > > > matches
> > > > >> > that of the forecast data valid at the same hour (23),
but the
> > > minutes
> > > > >> of
> > > > >> > the forecast file is 00 while for the observations it is
15. I
> was
> > > > >> aware of
> > > > >> > this mismatch in the minutes from when I ran Grid-Stat
and it
> > > flagged
> > > > >> it as
> > > > >> > a "Warning", but it went ahead and generated output with
no
> Error.
> > > The
> > > > >> > presence of the min value of 15 is an artifact of the
fact that
> > > this
> > > > >> data
> > > > >> > was produced by the LAPS system. The valid time of the
data is
> > > > actually
> > > > >> > 2300 instead of 2315. That said, I will run wgrib on the
> forecast
> > > file
> > > > >> per
> > > > >> > your suggestion to confirm the valid time.
> > > > >> > R/
> > > > >> > John
> > > > >> >
> > > > >> > ________________________________________
> > > > >> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > >> > Sent: Wednesday, April 13, 2016 3:48 PM
> > > > >> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > >> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#75766]
> > problem
> > > > >> > running series-analysis
> > > > >> >
> > > > >> > All active links contained in this email were disabled.
Please
> > > verify
> > > > >> the
> > > > >> > identity of the sender, and confirm the authenticity of
all
> links
> > > > >> contained
> > > > >> > within the message prior to copying and pasting the
address to a
> > Web
> > > > >> > browser.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > ----
> > > > >> >
> > > > >> > John,
> > > > >> >
> > > > >> > I don't see any immediately obvious problems in the data
you
> sent
> > to
> > > > >> us. I
> > > > >> > see that you're writing a log file named:
> > > > >> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
> > > > >> >
> > > > >> > Are you able to send me that complete log file? That
might help
> > me
> > > > >> > diagnose what's going on.
> > > > >> >
> > > > >> > Here's my initial guess...
> > > > >> >
> > > > >> > You are running series-analysis by defining a single
field in
> the
> > > > >> > configuration file. And you're passing to it lists of 12
> forecast
> > > > files
> > > > >> > and 12 observation files.
> > > > >> >
> > > > >> > It reads TMP/Z2 from the first forecast file and check
the valid
> > > time
> > > > >> for
> > > > >> > that data. Next, it looks in the observation file list
> (starting
> > > with
> > > > >> the
> > > > >> > first file) for TMP/Z2 with the *SAME* valid time.
> > > > >> >
> > > > >> > Next, it reads TMP/Z2 from the second forecast file, gets
the
> > valid
> > > > >> time,
> > > > >> > and looks in the observation file list (starting with the
second
> > > file)
> > > > >> for
> > > > >> > TMP/Z2 with the *SAME* valid time.
> > > > >> >
> > > > >> > I looked at the observation file
(120382300_laps_288.grb1) you
> > sent
> > > me
> > > > >> and
> > > > >> > see that it's valid time is 2012020723, which happens to
match
> the
> > > > >> > timestamp in the forecast file name. The first thing I'd
check
> is
> > > the
> > > > >> > valid time of the corresponding forecast file:
> > > > >> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
> > > > >> >
> > > > >> > Is it actually valid at the same time, 2012020723? If
not,
> that's
> > > the
> > > > >> > problem. If so, that's not the problem.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > John
> > > > >> >
> > > > >> >
> > > > >> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV via
RT <
> > > > >> > met_help at ucar.edu> wrote:
> > > > >> >
> > > > >> > >
> > > > >> > > <Caution-Caution-Caution-Caution-Caution-Caution-
Caution-url: Caution-
> > > > >> > Caution-Caution-
> > > > Caution-Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > >> > >
> > > > >> > > Hi John -
> > > > >> > >
> > > > >> > > Yes. I got stuck on trying to figure out the cause of
that
> > error.
> > > > >> Seems
> > > > >> > > like if Grid-Stat can read the 2-m temperature from the
> gridded
> > > obs,
> > > > >> then
> > > > >> > > why would Series-Analysis not be able to read it?
> > > > >> > >
> > > > >> > > Thanks.
> > > > >> > >
> > > > >> > > R/
> > > > >> > > John
> > > > >> > >
> > > > >> > >
> > > > >> > > -----Original Message-----
> > > > >> > > From: John Halley Gotway via RT
> > > [Caution-Caution-Caution-Caution-Caution-Caution-Caution-mailto:
> > > > >> met_help at ucar.edu]
> > > > >> > > Sent: Wednesday, April 13, 2016 12:34 PM
> > > > >> > > To: Raby, John W CIV USARMY RDECOM ARL (US) <
> > > > >> john.w.raby2.civ at mail.mil>
> > > > >> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
> problem
> > > > >> running
> > > > >> > > series-analysis
> > > > >> > >
> > > > >> > > All active links contained in this email were disabled.
> Please
> > > > verify
> > > > >> > the
> > > > >> > > identity of the sender, and confirm the authenticity of
all
> > links
> > > > >> > contained
> > > > >> > > within the message prior to copying and pasting the
address
> to a
> > > Web
> > > > >> > > browser.
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > ----
> > > > >> > >
> > > > >> > > Hello John,
> > > > >> > >
> > > > >> > > I see that you were having trouble running series-
analysis on
> > 2-m
> > > > >> > > temperature data last week.
> > > > >> > >
> > > > >> > > Is this still an issue? If so, I'd be happy to take a
look.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > > John Halley Gotway
> > > > >> > >
> > > > >> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT <
> > > > >> met_help at ucar.edu
> > > > >> > >
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > >
> > > > >> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted
upon.
> > > > >> > > > Transaction: Given to johnhg (John Halley Gotway) by
bullock
> > > > >> > > > Queue: met_help
> > > > >> > > > Subject: problem running series-analysis
> > > > >> > > > Owner: johnhg
> > > > >> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > >> > > > Status: open
> > > > >> > > > Ticket
> <Caution-Caution-Caution-Caution-Caution-Caution-Caution-Caution-
url:
> > > > >> > > > Caution-Caution-
> > > > >> > Caution-Caution-
> > > > Caution-Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > This transaction appears to have no content
> > > > >> > > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: problem running series-analysis
From: John Halley Gotway
Time: Wed Apr 20 13:33:24 2016
John,
OK, I'd suggest running again with:
n_rep = 0
block_size = 100000
rank_corr_flag = FALSE
And set "-v 3"... using 4 or higher will cause that very detailed
output
listing info for each pair. Let's skip over that.
John
And then set reduce the verbosity level
On Wed, Apr 20, 2016 at 12:26 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> I checked my config file for the particular settings you mentioned:
> n_rep = 1000
> block_size = 100000
> rank_corr_flag = TRUE
>
> The screen dialog says it's processing "with 12 Matched pairs" and
the (i,
> j) values increment j, then cycle through all 287 values of i, then
> increment j by one and start over incrementing i for 1-287 and this
is
> repeating so that now I have j=160.
>
> My script is as follows:
> series_analysis -fcst
~/MET_WRFpostprd/20120207_D_FDDA/fcst_file_list -obs
> ~/MET_obs/GSD_LAPS/20120207/obs_file_list -out
> ~/MET_SeriesAnalysis/results_m3o3_TMP_041916 -config
> ~/MET_SeriesAnalysis/SeriesAnalysisConfig_m3o3_gsdlaps_LAX_JHG -log
> ~/MET_SeriesAnalysis/logs/run_series_analysis.log -v 5
>
> You saw the fcst and obs file lists which I sent you.
>
> I could send you the log file which is still being written if you
want via
> SAFE.
>
>
>
> ________________________________________
>
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Wednesday, April 20, 2016 10:47 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Running this long is obviously not the desired behavior! Is this
really
> just being run on series of length 12 and a grid with dimension
288x288?
> If so, this should run in minutes, not hours.
>
> Please take a look at the following settings in the Series-Analysis
config
> file...
>
> (1) In the "boot" section, set:
> n_rep = 0;
> That will turn off the computation of bootstrap CI's which is slow.
>
> (2) Check to see how "block_size" is set. The default is 1024.
This is
> the number of grid points to process concurrently. Using the
default value
> of 1024 will require 81 (288*288/104) passes through those files.
And
> that'll slow it down a lot. Setting it to 82944 will enable it to
process
> your data in a single pass (much faster). BUT if your computer
doesn't
> have enough memory, it'll start using swap space and make it run
REALLY
> slowly. If you have it set high, try reducing it. If it's set low,
try
> increasing it.
>
> (3) Set "rank_corr_flag = FALSE;" to disable rank correlation
statistics
> which can be slow.
>
> John
>
>
>
> On Wed, Apr 20, 2016 at 7:12 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url: Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> > John -
> >
> > The series-analysis run I started yesterday is still running.
Judging
> from
> > the screen output, the grid position info which changes at ~ 1sec
> intervals
> > to show the point on which the processing is occurring
(categorical and
> > continuous statistics), it looks like it is about 40% complete.I
estimate
> > that it will complete early tomorrow morning.
> >
> > R/
> > John
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Tuesday, April 19, 2016 11:56 AM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> > running series-analysis
> >
> > All active links contained in this email were disabled. Please
verify
> the
> > identity of the sender, and confirm the authenticity of all links
> contained
> > within the message prior to copying and pasting the address to a
Web
> > browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Sure, you can use my email address, johnhg at ucar.edu. Although
looking
> in
> > the "fcst_file_list" I see you're running Series-Analysis on 12
files.
> > With just a single fcst/obs file, running it on a series of length
1
> won't
> > be all that informative.
> >
> > For now please just send me a verbose log file for your run and
I'll look
> > to see if issues jump out.
> >
> > Thanks,
> > John
> >
> > On Tue, Apr 19, 2016 at 11:51 AM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-url: Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >
> > > John -
> > >
> > > Thanks for taking a look at that. For large sized files, my only
method
> > of
> > > transfer to you is by the ARL Secure Access File Exchange (SAFE)
which
> > > requires an email address to which they send you a link to
download the
> > > file. FTP is blocked here at WSMR. In the past, I have used your
email
> > > address @ucar.edu. Will this still work? If so, do you also want
the
> > > forecast file which has the same valid time (almost-2300Z) as
the
> > > observations file I sent you which was valid at 2315Z on 07 FEB
2012?
> > >
> > > I don't have ncview running here. I did try visualizing the
output file
> > > using the Unidata IDV software and none of the fields looked
right to
> me.
> > >
> > > I'll start working on a re-run of Series-Analysis to generate
the log
> per
> > > your instructions.
> > >
> > > R/
> > > John
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Tuesday, April 19, 2016 9:45 AM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > running series-analysis
> > >
> > > All active links contained in this email were disabled. Please
verify
> > the
> > > identity of the sender, and confirm the authenticity of all
links
> > contained
> > > within the message prior to copying and pasting the address to a
Web
> > > browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > I agree with you. That output doesn't look good at all!
> > >
> > > Please just open up the NetCDF output file from Series-Analysis
using
> the
> > > "ncview" tool. Click through the variables and check whether
any of
> them
> > > look reasonable. If they look OK, then the problem is in
plotting them
> > > with plot_data_plane. If so, please just post that Series-
Analysis
> > output
> > > file to our anonymous ftp site? And I'll figure out what's
going on
> with
> > > the plotting.
> > >
> > > But my guess is that none of the fields look good, which means
the
> issue
> > is
> > > coming from Series-Analysis. Please try rerunning your Series-
Analysis
> > > command at verbosity level 5 (-v 5) and then save the output to
a log
> > file
> > > (-log run_series_analysis.log). Then send me that log file.
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Apr 18, 2016 at 4:13 PM, Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <Caution-Caution-Caution-url: Caution-
> > > Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >
> > > > John -
> > > >
> > > > I used the following command to plot the variable:
> > > series_cts_CSI_ge285.000
> > > > carson:jraby ~/MET_SeriesAnalysis
> > > > > plot_data_plane results_m3o3_TMP results_m3o3_TMP_CSI.ps
> > > > 'name="series_cts_CSI_ge285.000"; level="(*,*)";'
> > > > DEBUG 1: Opening data file: results_m3o3_TMP
> > > > DEBUG 1: Creating postscript file: results_m3o3_TMP_CSI.ps
> > > > carson:jraby ~/MET_SeriesAnalysis
> > > >
> > > > Seem to run fine, but the plot doesn't look like I expected.
> > > >
> > > > R/
> > > > John
> > > > ________________________________________
> > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > Sent: Monday, April 18, 2016 1:41 PM
> > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > > running series-analysis
> > > >
> > > > All active links contained in this email were disabled.
Please
> verify
> > > the
> > > > identity of the sender, and confirm the authenticity of all
links
> > > contained
> > > > within the message prior to copying and pasting the address to
a Web
> > > > browser.
> > > >
> > > >
> > > >
> > > >
> > > > ----
> > > >
> > > > John,
> > > >
> > > > I see the following error in the log file you sent me:
> > > > ERROR :
> > > > ERROR : plot_data_plane -> trouble getting field
"name="RMSE";
> > > > level="Z2";" from file "results_m3o3_TMP"
> > > > ERROR :
> > > >
> > > > You are calling plot_data_plane by passing it a NetCDF file to
plot.
> > The
> > > > way you set "name" and "level" is different between GRIB files
and
> > > gridded
> > > > NetCDF files. For NetCDF files, you set the name equal to the
name
> of
> > > the
> > > > NetCDF variable you want to plot and you set level to "(*,*)".
I
> know
> > > that
> > > > looks weird but you're telling MET which NetCDF dimensions are
the
> > > gridded
> > > > ones that it should plot.
> > > >
> > > > In your case, the output of Series-Analysis will only have 2
> > > dimensions...
> > > > thus "(*,*)".
> > > >
> > > > Just use "ncdump -h" to look at the name of the variables and
run
> > > > plot_data_plane something like this:
> > > >
> > > > plot_data_plane results_m3o3_TMP results_m3o3_TMP_RMSE.ps
> > > > 'name="series_cnt_RMSE"; level="(*,*)";'
> > > >
> > > > Hope that helps clarify.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Thu, Apr 14, 2016 at 9:14 AM, Raby, John W USA CIV via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <Caution-Caution-Caution-Caution-url: Caution-
> > > > Caution-Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > >
> > > > > John -
> > > > >
> > > > > I resolved the issue with whitespace by using the name
"RMSE" vice
> > > "Root
> > > > > mean squared error".
> > > > >
> > > > > Now it has an issue with the field "RMSE" from the output
file. See
> > > > > attached log.
> > > > >
> > > > > Thanks.
> > > > >
> > > > > R/
> > > > > John
> > > > >
> > > > > ________________________________________
> > > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > > Sent: Wednesday, April 13, 2016 4:57 PM
> > > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > > > running series-analysis
> > > > >
> > > > > All active links contained in this email were disabled.
Please
> > verify
> > > > the
> > > > > identity of the sender, and confirm the authenticity of all
links
> > > > contained
> > > > > within the message prior to copying and pasting the address
to a
> Web
> > > > > browser.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ----
> > > > >
> > > > > Also, I see the following error message in the log file you
sent:
> > > > >
> > > > > ERROR : store_stat_cnt() -> unsupported column name
requested
> "IQR"
> > > > >
> > > > > IQR was added in version 5.1, but it not defined in version
4.1.
> > > > >
> > > > > John
> > > > >
> > > > > On Wed, Apr 13, 2016 at 4:55 PM, John Halley Gotway <
> johnhg at ucar.edu
> > >
> > > > > wrote:
> > > > >
> > > > > > John,
> > > > > >
> > > > > > Ah yes, I meant to mention this. Please try increasing
the block
> > > size
> > > > > > from 1024 to something much larger.
> > > > > >
> > > > > > Your grid is 288x288 = 82944 grid points. Try setting:
> > > > > > block_size = 100000;
> > > > > >
> > > > > > That will process it all in one pass through the data and
should
> > make
> > > > it
> > > > > > run much faster.
> > > > > >
> > > > > > John
> > > > > >
> > > > > > On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV via
RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > >>
> > > > > >> <Caution-Caution-Caution-Caution-Caution-url: Caution-
> > > > > Caution-Caution-
> > > Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > > >>
> > > > > >> John -
> > > > > >>
> > > > > >> I looked at your config file and it looks like a nice
> workaround.
> > I
> > > > > reran
> > > > > >> using your config file. Ran considerably longer this
time. I
> also
> > > > have a
> > > > > >> 14MB output NetCDF file which is definitely encouraging!
I have
> to
> > > > leave
> > > > > >> for the day in about 10 mins and will be back in touch
tomorrow.
> > > > Thanks
> > > > > for
> > > > > >> your help.
> > > > > >>
> > > > > >> R/
> > > > > >> John
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> ________________________________________
> > > > > >> From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > > >> Sent: Wednesday, April 13, 2016 4:37 PM
> > > > > >> To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#75766]
> > problem
> > > > > >> running series-analysis
> > > > > >>
> > > > > >> All active links contained in this email were disabled.
Please
> > > verify
> > > > > >> the identity of the sender, and confirm the authenticity
of all
> > > links
> > > > > >> contained within the message prior to copying and pasting
the
> > > address
> > > > > to a
> > > > > >> Web browser.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> ----
> > > > > >>
> > > > > >> John,
> > > > > >>
> > > > > >> Ah yes, that time offset is the problem here.
> > > > > >>
> > > > > >> And it would have been nice for Series-Analysis to be
more
> verbose
> > > > about
> > > > > >> the timing information it's processing.
> > > > > >>
> > > > > >> Please try using the updated config file that I've
attached, and
> > see
> > > > if
> > > > > >> Series-Analysis actually runs.
> > > > > >>
> > > > > >> We pass grid-stat 1 forecast file and 1 observation file.
So
> even
> > > if
> > > > > the
> > > > > >> times don't match up exactly, it writes a warning and
compares
> > them.
> > > > > >> Series-Analysis tries to be a bit fancier. You pass it N
> forecast
> > > > and N
> > > > > >> observation files. For each forecast files, it searches
all the
> > > > > >> observation files for a match.
> > > > > >>
> > > > > >> But that more complex logic breaks down when there's a 15
minute
> > > time
> > > > > >> offset. So we should think about how better to handle
this use
> > > case.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> John
> > > > > >>
> > > > > >> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via
RT <
> > > > > >> met_help at ucar.edu> wrote:
> > > > > >>
> > > > > >> >
> > > > > >> > <Caution-Caution-Caution-Caution-Caution-Caution-url:
Caution-
> > > > > >> Caution-Caution-
> > > > Caution-Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > > >> >
> > > > > >> > John -
> > > > > >> >
> > > > > >> > Based on your analysis of the reading of the input
files, I
> can
> > > > offer
> > > > > a
> > > > > >> > possibility that I'm aware of regarding the valid
times.The
> > hours
> > > > for
> > > > > >> the
> > > > > >> > example input data (observations) I sent you is indeed
23
> which
> > > > > matches
> > > > > >> > that of the forecast data valid at the same hour (23),
but the
> > > > minutes
> > > > > >> of
> > > > > >> > the forecast file is 00 while for the observations it
is 15. I
> > was
> > > > > >> aware of
> > > > > >> > this mismatch in the minutes from when I ran Grid-Stat
and it
> > > > flagged
> > > > > >> it as
> > > > > >> > a "Warning", but it went ahead and generated output
with no
> > Error.
> > > > The
> > > > > >> > presence of the min value of 15 is an artifact of the
fact
> that
> > > > this
> > > > > >> data
> > > > > >> > was produced by the LAPS system. The valid time of the
data is
> > > > > actually
> > > > > >> > 2300 instead of 2315. That said, I will run wgrib on
the
> > forecast
> > > > file
> > > > > >> per
> > > > > >> > your suggestion to confirm the valid time.
> > > > > >> > R/
> > > > > >> > John
> > > > > >> >
> > > > > >> > ________________________________________
> > > > > >> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > > >> > Sent: Wednesday, April 13, 2016 3:48 PM
> > > > > >> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > >> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#75766]
> > > problem
> > > > > >> > running series-analysis
> > > > > >> >
> > > > > >> > All active links contained in this email were disabled.
> Please
> > > > verify
> > > > > >> the
> > > > > >> > identity of the sender, and confirm the authenticity of
all
> > links
> > > > > >> contained
> > > > > >> > within the message prior to copying and pasting the
address
> to a
> > > Web
> > > > > >> > browser.
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > ----
> > > > > >> >
> > > > > >> > John,
> > > > > >> >
> > > > > >> > I don't see any immediately obvious problems in the
data you
> > sent
> > > to
> > > > > >> us. I
> > > > > >> > see that you're writing a log file named:
> > > > > >> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
> > > > > >> >
> > > > > >> > Are you able to send me that complete log file? That
might
> help
> > > me
> > > > > >> > diagnose what's going on.
> > > > > >> >
> > > > > >> > Here's my initial guess...
> > > > > >> >
> > > > > >> > You are running series-analysis by defining a single
field in
> > the
> > > > > >> > configuration file. And you're passing to it lists of
12
> > forecast
> > > > > files
> > > > > >> > and 12 observation files.
> > > > > >> >
> > > > > >> > It reads TMP/Z2 from the first forecast file and check
the
> valid
> > > > time
> > > > > >> for
> > > > > >> > that data. Next, it looks in the observation file list
> > (starting
> > > > with
> > > > > >> the
> > > > > >> > first file) for TMP/Z2 with the *SAME* valid time.
> > > > > >> >
> > > > > >> > Next, it reads TMP/Z2 from the second forecast file,
gets the
> > > valid
> > > > > >> time,
> > > > > >> > and looks in the observation file list (starting with
the
> second
> > > > file)
> > > > > >> for
> > > > > >> > TMP/Z2 with the *SAME* valid time.
> > > > > >> >
> > > > > >> > I looked at the observation file
(120382300_laps_288.grb1) you
> > > sent
> > > > me
> > > > > >> and
> > > > > >> > see that it's valid time is 2012020723, which happens
to
> match
> > the
> > > > > >> > timestamp in the forecast file name. The first thing
I'd
> check
> > is
> > > > the
> > > > > >> > valid time of the corresponding forecast file:
> > > > > >> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
> > > > > >> >
> > > > > >> > Is it actually valid at the same time, 2012020723? If
not,
> > that's
> > > > the
> > > > > >> > problem. If so, that's not the problem.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > John
> > > > > >> >
> > > > > >> >
> > > > > >> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV
via RT <
> > > > > >> > met_help at ucar.edu> wrote:
> > > > > >> >
> > > > > >> > >
> > > > > >> > >
> <Caution-Caution-Caution-Caution-Caution-Caution-Caution-url:
Caution-
> > > > > >> > Caution-Caution-
> > > > > Caution-Caution-
> > > Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > > >> > >
> > > > > >> > > Hi John -
> > > > > >> > >
> > > > > >> > > Yes. I got stuck on trying to figure out the cause of
that
> > > error.
> > > > > >> Seems
> > > > > >> > > like if Grid-Stat can read the 2-m temperature from
the
> > gridded
> > > > obs,
> > > > > >> then
> > > > > >> > > why would Series-Analysis not be able to read it?
> > > > > >> > >
> > > > > >> > > Thanks.
> > > > > >> > >
> > > > > >> > > R/
> > > > > >> > > John
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > -----Original Message-----
> > > > > >> > > From: John Halley Gotway via RT
> > > > [Caution-Caution-Caution-Caution-Caution-Caution-Caution-
mailto:
> > > > > >> met_help at ucar.edu]
> > > > > >> > > Sent: Wednesday, April 13, 2016 12:34 PM
> > > > > >> > > To: Raby, John W CIV USARMY RDECOM ARL (US) <
> > > > > >> john.w.raby2.civ at mail.mil>
> > > > > >> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu
#75766]
> > problem
> > > > > >> running
> > > > > >> > > series-analysis
> > > > > >> > >
> > > > > >> > > All active links contained in this email were
disabled.
> > Please
> > > > > verify
> > > > > >> > the
> > > > > >> > > identity of the sender, and confirm the authenticity
of all
> > > links
> > > > > >> > contained
> > > > > >> > > within the message prior to copying and pasting the
address
> > to a
> > > > Web
> > > > > >> > > browser.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > ----
> > > > > >> > >
> > > > > >> > > Hello John,
> > > > > >> > >
> > > > > >> > > I see that you were having trouble running series-
analysis
> on
> > > 2-m
> > > > > >> > > temperature data last week.
> > > > > >> > >
> > > > > >> > > Is this still an issue? If so, I'd be happy to take
a look.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > John Halley Gotway
> > > > > >> > >
> > > > > >> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT
<
> > > > > >> met_help at ucar.edu
> > > > > >> > >
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > >
> > > > > >> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted
upon.
> > > > > >> > > > Transaction: Given to johnhg (John Halley Gotway)
by
> bullock
> > > > > >> > > > Queue: met_help
> > > > > >> > > > Subject: problem running series-analysis
> > > > > >> > > > Owner: johnhg
> > > > > >> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > > >> > > > Status: open
> > > > > >> > > > Ticket
> > <Caution-Caution-Caution-Caution-Caution-Caution-Caution-Caution-
url:
> > > > > >> > > > Caution-Caution-
> > > > > >> > Caution-Caution-
> > > > > Caution-Caution-
> > > Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > This transaction appears to have no content
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem running series-analysis
From: Raby, John W USA CIV
Time: Wed Apr 20 13:48:21 2016
John -
Thanks for your quick response.
OK. I'll stop the run, delete the output file and rename the log file
to save and then make the changes you indicate and then rerun.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Wednesday, April 20, 2016 1:33 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running series-analysis
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
OK, I'd suggest running again with:
n_rep = 0
block_size = 100000
rank_corr_flag = FALSE
And set "-v 3"... using 4 or higher will cause that very detailed
output
listing info for each pair. Let's skip over that.
John
And then set reduce the verbosity level
On Wed, Apr 20, 2016 at 12:26 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> I checked my config file for the particular settings you mentioned:
> n_rep = 1000
> block_size = 100000
> rank_corr_flag = TRUE
>
> The screen dialog says it's processing "with 12 Matched pairs" and
the (i,
> j) values increment j, then cycle through all 287 values of i, then
> increment j by one and start over incrementing i for 1-287 and this
is
> repeating so that now I have j=160.
>
> My script is as follows:
> series_analysis -fcst
~/MET_WRFpostprd/20120207_D_FDDA/fcst_file_list -obs
> ~/MET_obs/GSD_LAPS/20120207/obs_file_list -out
> ~/MET_SeriesAnalysis/results_m3o3_TMP_041916 -config
> ~/MET_SeriesAnalysis/SeriesAnalysisConfig_m3o3_gsdlaps_LAX_JHG -log
> ~/MET_SeriesAnalysis/logs/run_series_analysis.log -v 5
>
> You saw the fcst and obs file lists which I sent you.
>
> I could send you the log file which is still being written if you
want via
> SAFE.
>
>
>
> ________________________________________
>
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Wednesday, April 20, 2016 10:47 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Running this long is obviously not the desired behavior! Is this
really
> just being run on series of length 12 and a grid with dimension
288x288?
> If so, this should run in minutes, not hours.
>
> Please take a look at the following settings in the Series-Analysis
config
> file...
>
> (1) In the "boot" section, set:
> n_rep = 0;
> That will turn off the computation of bootstrap CI's which is slow.
>
> (2) Check to see how "block_size" is set. The default is 1024.
This is
> the number of grid points to process concurrently. Using the
default value
> of 1024 will require 81 (288*288/104) passes through those files.
And
> that'll slow it down a lot. Setting it to 82944 will enable it to
process
> your data in a single pass (much faster). BUT if your computer
doesn't
> have enough memory, it'll start using swap space and make it run
REALLY
> slowly. If you have it set high, try reducing it. If it's set low,
try
> increasing it.
>
> (3) Set "rank_corr_flag = FALSE;" to disable rank correlation
statistics
> which can be slow.
>
> John
>
>
>
> On Wed, Apr 20, 2016 at 7:12 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url: Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> > John -
> >
> > The series-analysis run I started yesterday is still running.
Judging
> from
> > the screen output, the grid position info which changes at ~ 1sec
> intervals
> > to show the point on which the processing is occurring
(categorical and
> > continuous statistics), it looks like it is about 40% complete.I
estimate
> > that it will complete early tomorrow morning.
> >
> > R/
> > John
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Tuesday, April 19, 2016 11:56 AM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> > running series-analysis
> >
> > All active links contained in this email were disabled. Please
verify
> the
> > identity of the sender, and confirm the authenticity of all links
> contained
> > within the message prior to copying and pasting the address to a
Web
> > browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Sure, you can use my email address, johnhg at ucar.edu. Although
looking
> in
> > the "fcst_file_list" I see you're running Series-Analysis on 12
files.
> > With just a single fcst/obs file, running it on a series of length
1
> won't
> > be all that informative.
> >
> > For now please just send me a verbose log file for your run and
I'll look
> > to see if issues jump out.
> >
> > Thanks,
> > John
> >
> > On Tue, Apr 19, 2016 at 11:51 AM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url: Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >
> > > John -
> > >
> > > Thanks for taking a look at that. For large sized files, my only
method
> > of
> > > transfer to you is by the ARL Secure Access File Exchange (SAFE)
which
> > > requires an email address to which they send you a link to
download the
> > > file. FTP is blocked here at WSMR. In the past, I have used your
email
> > > address @ucar.edu. Will this still work? If so, do you also want
the
> > > forecast file which has the same valid time (almost-2300Z) as
the
> > > observations file I sent you which was valid at 2315Z on 07 FEB
2012?
> > >
> > > I don't have ncview running here. I did try visualizing the
output file
> > > using the Unidata IDV software and none of the fields looked
right to
> me.
> > >
> > > I'll start working on a re-run of Series-Analysis to generate
the log
> per
> > > your instructions.
> > >
> > > R/
> > > John
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Tuesday, April 19, 2016 9:45 AM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > running series-analysis
> > >
> > > All active links contained in this email were disabled. Please
verify
> > the
> > > identity of the sender, and confirm the authenticity of all
links
> > contained
> > > within the message prior to copying and pasting the address to a
Web
> > > browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > I agree with you. That output doesn't look good at all!
> > >
> > > Please just open up the NetCDF output file from Series-Analysis
using
> the
> > > "ncview" tool. Click through the variables and check whether
any of
> them
> > > look reasonable. If they look OK, then the problem is in
plotting them
> > > with plot_data_plane. If so, please just post that Series-
Analysis
> > output
> > > file to our anonymous ftp site? And I'll figure out what's
going on
> with
> > > the plotting.
> > >
> > > But my guess is that none of the fields look good, which means
the
> issue
> > is
> > > coming from Series-Analysis. Please try rerunning your Series-
Analysis
> > > command at verbosity level 5 (-v 5) and then save the output to
a log
> > file
> > > (-log run_series_analysis.log). Then send me that log file.
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Apr 18, 2016 at 4:13 PM, Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <Caution-Caution-Caution-Caution-url: Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >
> > > > John -
> > > >
> > > > I used the following command to plot the variable:
> > > series_cts_CSI_ge285.000
> > > > carson:jraby ~/MET_SeriesAnalysis
> > > > > plot_data_plane results_m3o3_TMP results_m3o3_TMP_CSI.ps
> > > > 'name="series_cts_CSI_ge285.000"; level="(*,*)";'
> > > > DEBUG 1: Opening data file: results_m3o3_TMP
> > > > DEBUG 1: Creating postscript file: results_m3o3_TMP_CSI.ps
> > > > carson:jraby ~/MET_SeriesAnalysis
> > > >
> > > > Seem to run fine, but the plot doesn't look like I expected.
> > > >
> > > > R/
> > > > John
> > > > ________________________________________
> > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > Sent: Monday, April 18, 2016 1:41 PM
> > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > > running series-analysis
> > > >
> > > > All active links contained in this email were disabled.
Please
> verify
> > > the
> > > > identity of the sender, and confirm the authenticity of all
links
> > > contained
> > > > within the message prior to copying and pasting the address to
a Web
> > > > browser.
> > > >
> > > >
> > > >
> > > >
> > > > ----
> > > >
> > > > John,
> > > >
> > > > I see the following error in the log file you sent me:
> > > > ERROR :
> > > > ERROR : plot_data_plane -> trouble getting field
"name="RMSE";
> > > > level="Z2";" from file "results_m3o3_TMP"
> > > > ERROR :
> > > >
> > > > You are calling plot_data_plane by passing it a NetCDF file to
plot.
> > The
> > > > way you set "name" and "level" is different between GRIB files
and
> > > gridded
> > > > NetCDF files. For NetCDF files, you set the name equal to the
name
> of
> > > the
> > > > NetCDF variable you want to plot and you set level to "(*,*)".
I
> know
> > > that
> > > > looks weird but you're telling MET which NetCDF dimensions are
the
> > > gridded
> > > > ones that it should plot.
> > > >
> > > > In your case, the output of Series-Analysis will only have 2
> > > dimensions...
> > > > thus "(*,*)".
> > > >
> > > > Just use "ncdump -h" to look at the name of the variables and
run
> > > > plot_data_plane something like this:
> > > >
> > > > plot_data_plane results_m3o3_TMP results_m3o3_TMP_RMSE.ps
> > > > 'name="series_cnt_RMSE"; level="(*,*)";'
> > > >
> > > > Hope that helps clarify.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Thu, Apr 14, 2016 at 9:14 AM, Raby, John W USA CIV via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <Caution-Caution-Caution-Caution-Caution-url: Caution-
> > > > Caution-Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > >
> > > > > John -
> > > > >
> > > > > I resolved the issue with whitespace by using the name
"RMSE" vice
> > > "Root
> > > > > mean squared error".
> > > > >
> > > > > Now it has an issue with the field "RMSE" from the output
file. See
> > > > > attached log.
> > > > >
> > > > > Thanks.
> > > > >
> > > > > R/
> > > > > John
> > > > >
> > > > > ________________________________________
> > > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > > Sent: Wednesday, April 13, 2016 4:57 PM
> > > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > > > running series-analysis
> > > > >
> > > > > All active links contained in this email were disabled.
Please
> > verify
> > > > the
> > > > > identity of the sender, and confirm the authenticity of all
links
> > > > contained
> > > > > within the message prior to copying and pasting the address
to a
> Web
> > > > > browser.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ----
> > > > >
> > > > > Also, I see the following error message in the log file you
sent:
> > > > >
> > > > > ERROR : store_stat_cnt() -> unsupported column name
requested
> "IQR"
> > > > >
> > > > > IQR was added in version 5.1, but it not defined in version
4.1.
> > > > >
> > > > > John
> > > > >
> > > > > On Wed, Apr 13, 2016 at 4:55 PM, John Halley Gotway <
> johnhg at ucar.edu
> > >
> > > > > wrote:
> > > > >
> > > > > > John,
> > > > > >
> > > > > > Ah yes, I meant to mention this. Please try increasing
the block
> > > size
> > > > > > from 1024 to something much larger.
> > > > > >
> > > > > > Your grid is 288x288 = 82944 grid points. Try setting:
> > > > > > block_size = 100000;
> > > > > >
> > > > > > That will process it all in one pass through the data and
should
> > make
> > > > it
> > > > > > run much faster.
> > > > > >
> > > > > > John
> > > > > >
> > > > > > On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV via
RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > >>
> > > > > >> <Caution-Caution-Caution-Caution-Caution-Caution-url:
Caution-
> > > > > Caution-Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > > >>
> > > > > >> John -
> > > > > >>
> > > > > >> I looked at your config file and it looks like a nice
> workaround.
> > I
> > > > > reran
> > > > > >> using your config file. Ran considerably longer this
time. I
> also
> > > > have a
> > > > > >> 14MB output NetCDF file which is definitely encouraging!
I have
> to
> > > > leave
> > > > > >> for the day in about 10 mins and will be back in touch
tomorrow.
> > > > Thanks
> > > > > for
> > > > > >> your help.
> > > > > >>
> > > > > >> R/
> > > > > >> John
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> ________________________________________
> > > > > >> From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > > >> Sent: Wednesday, April 13, 2016 4:37 PM
> > > > > >> To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#75766]
> > problem
> > > > > >> running series-analysis
> > > > > >>
> > > > > >> All active links contained in this email were disabled.
Please
> > > verify
> > > > > >> the identity of the sender, and confirm the authenticity
of all
> > > links
> > > > > >> contained within the message prior to copying and pasting
the
> > > address
> > > > > to a
> > > > > >> Web browser.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> ----
> > > > > >>
> > > > > >> John,
> > > > > >>
> > > > > >> Ah yes, that time offset is the problem here.
> > > > > >>
> > > > > >> And it would have been nice for Series-Analysis to be
more
> verbose
> > > > about
> > > > > >> the timing information it's processing.
> > > > > >>
> > > > > >> Please try using the updated config file that I've
attached, and
> > see
> > > > if
> > > > > >> Series-Analysis actually runs.
> > > > > >>
> > > > > >> We pass grid-stat 1 forecast file and 1 observation file.
So
> even
> > > if
> > > > > the
> > > > > >> times don't match up exactly, it writes a warning and
compares
> > them.
> > > > > >> Series-Analysis tries to be a bit fancier. You pass it N
> forecast
> > > > and N
> > > > > >> observation files. For each forecast files, it searches
all the
> > > > > >> observation files for a match.
> > > > > >>
> > > > > >> But that more complex logic breaks down when there's a 15
minute
> > > time
> > > > > >> offset. So we should think about how better to handle
this use
> > > case.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> John
> > > > > >>
> > > > > >> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via
RT <
> > > > > >> met_help at ucar.edu> wrote:
> > > > > >>
> > > > > >> >
> > > > > >> > <Caution-Caution-Caution-Caution-Caution-Caution-
Caution-url: Caution-
> > > > > >> Caution-Caution-
> > > > Caution-Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > > >> >
> > > > > >> > John -
> > > > > >> >
> > > > > >> > Based on your analysis of the reading of the input
files, I
> can
> > > > offer
> > > > > a
> > > > > >> > possibility that I'm aware of regarding the valid
times.The
> > hours
> > > > for
> > > > > >> the
> > > > > >> > example input data (observations) I sent you is indeed
23
> which
> > > > > matches
> > > > > >> > that of the forecast data valid at the same hour (23),
but the
> > > > minutes
> > > > > >> of
> > > > > >> > the forecast file is 00 while for the observations it
is 15. I
> > was
> > > > > >> aware of
> > > > > >> > this mismatch in the minutes from when I ran Grid-Stat
and it
> > > > flagged
> > > > > >> it as
> > > > > >> > a "Warning", but it went ahead and generated output
with no
> > Error.
> > > > The
> > > > > >> > presence of the min value of 15 is an artifact of the
fact
> that
> > > > this
> > > > > >> data
> > > > > >> > was produced by the LAPS system. The valid time of the
data is
> > > > > actually
> > > > > >> > 2300 instead of 2315. That said, I will run wgrib on
the
> > forecast
> > > > file
> > > > > >> per
> > > > > >> > your suggestion to confirm the valid time.
> > > > > >> > R/
> > > > > >> > John
> > > > > >> >
> > > > > >> > ________________________________________
> > > > > >> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > > >> > Sent: Wednesday, April 13, 2016 3:48 PM
> > > > > >> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > >> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#75766]
> > > problem
> > > > > >> > running series-analysis
> > > > > >> >
> > > > > >> > All active links contained in this email were disabled.
> Please
> > > > verify
> > > > > >> the
> > > > > >> > identity of the sender, and confirm the authenticity of
all
> > links
> > > > > >> contained
> > > > > >> > within the message prior to copying and pasting the
address
> to a
> > > Web
> > > > > >> > browser.
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > ----
> > > > > >> >
> > > > > >> > John,
> > > > > >> >
> > > > > >> > I don't see any immediately obvious problems in the
data you
> > sent
> > > to
> > > > > >> us. I
> > > > > >> > see that you're writing a log file named:
> > > > > >> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
> > > > > >> >
> > > > > >> > Are you able to send me that complete log file? That
might
> help
> > > me
> > > > > >> > diagnose what's going on.
> > > > > >> >
> > > > > >> > Here's my initial guess...
> > > > > >> >
> > > > > >> > You are running series-analysis by defining a single
field in
> > the
> > > > > >> > configuration file. And you're passing to it lists of
12
> > forecast
> > > > > files
> > > > > >> > and 12 observation files.
> > > > > >> >
> > > > > >> > It reads TMP/Z2 from the first forecast file and check
the
> valid
> > > > time
> > > > > >> for
> > > > > >> > that data. Next, it looks in the observation file list
> > (starting
> > > > with
> > > > > >> the
> > > > > >> > first file) for TMP/Z2 with the *SAME* valid time.
> > > > > >> >
> > > > > >> > Next, it reads TMP/Z2 from the second forecast file,
gets the
> > > valid
> > > > > >> time,
> > > > > >> > and looks in the observation file list (starting with
the
> second
> > > > file)
> > > > > >> for
> > > > > >> > TMP/Z2 with the *SAME* valid time.
> > > > > >> >
> > > > > >> > I looked at the observation file
(120382300_laps_288.grb1) you
> > > sent
> > > > me
> > > > > >> and
> > > > > >> > see that it's valid time is 2012020723, which happens
to
> match
> > the
> > > > > >> > timestamp in the forecast file name. The first thing
I'd
> check
> > is
> > > > the
> > > > > >> > valid time of the corresponding forecast file:
> > > > > >> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
> > > > > >> >
> > > > > >> > Is it actually valid at the same time, 2012020723? If
not,
> > that's
> > > > the
> > > > > >> > problem. If so, that's not the problem.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > John
> > > > > >> >
> > > > > >> >
> > > > > >> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV
via RT <
> > > > > >> > met_help at ucar.edu> wrote:
> > > > > >> >
> > > > > >> > >
> > > > > >> > >
> <Caution-Caution-Caution-Caution-Caution-Caution-Caution-Caution-
url: Caution-
> > > > > >> > Caution-Caution-
> > > > > Caution-Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > > >> > >
> > > > > >> > > Hi John -
> > > > > >> > >
> > > > > >> > > Yes. I got stuck on trying to figure out the cause of
that
> > > error.
> > > > > >> Seems
> > > > > >> > > like if Grid-Stat can read the 2-m temperature from
the
> > gridded
> > > > obs,
> > > > > >> then
> > > > > >> > > why would Series-Analysis not be able to read it?
> > > > > >> > >
> > > > > >> > > Thanks.
> > > > > >> > >
> > > > > >> > > R/
> > > > > >> > > John
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > -----Original Message-----
> > > > > >> > > From: John Halley Gotway via RT
> > > > [Caution-Caution-Caution-Caution-Caution-Caution-Caution-
Caution-mailto:
> > > > > >> met_help at ucar.edu]
> > > > > >> > > Sent: Wednesday, April 13, 2016 12:34 PM
> > > > > >> > > To: Raby, John W CIV USARMY RDECOM ARL (US) <
> > > > > >> john.w.raby2.civ at mail.mil>
> > > > > >> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu
#75766]
> > problem
> > > > > >> running
> > > > > >> > > series-analysis
> > > > > >> > >
> > > > > >> > > All active links contained in this email were
disabled.
> > Please
> > > > > verify
> > > > > >> > the
> > > > > >> > > identity of the sender, and confirm the authenticity
of all
> > > links
> > > > > >> > contained
> > > > > >> > > within the message prior to copying and pasting the
address
> > to a
> > > > Web
> > > > > >> > > browser.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > ----
> > > > > >> > >
> > > > > >> > > Hello John,
> > > > > >> > >
> > > > > >> > > I see that you were having trouble running series-
analysis
> on
> > > 2-m
> > > > > >> > > temperature data last week.
> > > > > >> > >
> > > > > >> > > Is this still an issue? If so, I'd be happy to take
a look.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > John Halley Gotway
> > > > > >> > >
> > > > > >> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT
<
> > > > > >> met_help at ucar.edu
> > > > > >> > >
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > >
> > > > > >> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted
upon.
> > > > > >> > > > Transaction: Given to johnhg (John Halley Gotway)
by
> bullock
> > > > > >> > > > Queue: met_help
> > > > > >> > > > Subject: problem running series-analysis
> > > > > >> > > > Owner: johnhg
> > > > > >> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > > >> > > > Status: open
> > > > > >> > > > Ticket
> > <Caution-Caution-Caution-Caution-Caution-Caution-Caution-Caution-
Caution-url:
> > > > > >> > > > Caution-Caution-
> > > > > >> > Caution-Caution-
> > > > > Caution-Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > This transaction appears to have no content
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: problem running series-analysis
From: Raby, John W USA CIV
Time: Wed Apr 20 14:18:39 2016
John -
Series-Analysis is restarted with the recommended changes. Appears to
have read in all12 files and successfully found the 2m TMP fields, but
I don't see screen updates beyond that due to the reduced verbosity.
The run completed after 6 mins duration. The log file indicates no
ERROR or WARNING and otherwise appears normal. I plotted the CSI error
using plot_data_plane and it looks very promising as good output from
a successful run compared to the plots from earlier in the week.
What do you think? I'm ready to do more testing or provide more
information as needed to confirm.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Wednesday, April 20, 2016 1:33 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running series-analysis
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
OK, I'd suggest running again with:
n_rep = 0
block_size = 100000
rank_corr_flag = FALSE
And set "-v 3"... using 4 or higher will cause that very detailed
output
listing info for each pair. Let's skip over that.
John
And then set reduce the verbosity level
On Wed, Apr 20, 2016 at 12:26 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> I checked my config file for the particular settings you mentioned:
> n_rep = 1000
> block_size = 100000
> rank_corr_flag = TRUE
>
> The screen dialog says it's processing "with 12 Matched pairs" and
the (i,
> j) values increment j, then cycle through all 287 values of i, then
> increment j by one and start over incrementing i for 1-287 and this
is
> repeating so that now I have j=160.
>
> My script is as follows:
> series_analysis -fcst
~/MET_WRFpostprd/20120207_D_FDDA/fcst_file_list -obs
> ~/MET_obs/GSD_LAPS/20120207/obs_file_list -out
> ~/MET_SeriesAnalysis/results_m3o3_TMP_041916 -config
> ~/MET_SeriesAnalysis/SeriesAnalysisConfig_m3o3_gsdlaps_LAX_JHG -log
> ~/MET_SeriesAnalysis/logs/run_series_analysis.log -v 5
>
> You saw the fcst and obs file lists which I sent you.
>
> I could send you the log file which is still being written if you
want via
> SAFE.
>
>
>
> ________________________________________
>
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Wednesday, April 20, 2016 10:47 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Running this long is obviously not the desired behavior! Is this
really
> just being run on series of length 12 and a grid with dimension
288x288?
> If so, this should run in minutes, not hours.
>
> Please take a look at the following settings in the Series-Analysis
config
> file...
>
> (1) In the "boot" section, set:
> n_rep = 0;
> That will turn off the computation of bootstrap CI's which is slow.
>
> (2) Check to see how "block_size" is set. The default is 1024.
This is
> the number of grid points to process concurrently. Using the
default value
> of 1024 will require 81 (288*288/104) passes through those files.
And
> that'll slow it down a lot. Setting it to 82944 will enable it to
process
> your data in a single pass (much faster). BUT if your computer
doesn't
> have enough memory, it'll start using swap space and make it run
REALLY
> slowly. If you have it set high, try reducing it. If it's set low,
try
> increasing it.
>
> (3) Set "rank_corr_flag = FALSE;" to disable rank correlation
statistics
> which can be slow.
>
> John
>
>
>
> On Wed, Apr 20, 2016 at 7:12 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url: Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> > John -
> >
> > The series-analysis run I started yesterday is still running.
Judging
> from
> > the screen output, the grid position info which changes at ~ 1sec
> intervals
> > to show the point on which the processing is occurring
(categorical and
> > continuous statistics), it looks like it is about 40% complete.I
estimate
> > that it will complete early tomorrow morning.
> >
> > R/
> > John
> > ________________________________________
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Tuesday, April 19, 2016 11:56 AM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> > running series-analysis
> >
> > All active links contained in this email were disabled. Please
verify
> the
> > identity of the sender, and confirm the authenticity of all links
> contained
> > within the message prior to copying and pasting the address to a
Web
> > browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Sure, you can use my email address, johnhg at ucar.edu. Although
looking
> in
> > the "fcst_file_list" I see you're running Series-Analysis on 12
files.
> > With just a single fcst/obs file, running it on a series of length
1
> won't
> > be all that informative.
> >
> > For now please just send me a verbose log file for your run and
I'll look
> > to see if issues jump out.
> >
> > Thanks,
> > John
> >
> > On Tue, Apr 19, 2016 at 11:51 AM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url: Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >
> > > John -
> > >
> > > Thanks for taking a look at that. For large sized files, my only
method
> > of
> > > transfer to you is by the ARL Secure Access File Exchange (SAFE)
which
> > > requires an email address to which they send you a link to
download the
> > > file. FTP is blocked here at WSMR. In the past, I have used your
email
> > > address @ucar.edu. Will this still work? If so, do you also want
the
> > > forecast file which has the same valid time (almost-2300Z) as
the
> > > observations file I sent you which was valid at 2315Z on 07 FEB
2012?
> > >
> > > I don't have ncview running here. I did try visualizing the
output file
> > > using the Unidata IDV software and none of the fields looked
right to
> me.
> > >
> > > I'll start working on a re-run of Series-Analysis to generate
the log
> per
> > > your instructions.
> > >
> > > R/
> > > John
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Tuesday, April 19, 2016 9:45 AM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > running series-analysis
> > >
> > > All active links contained in this email were disabled. Please
verify
> > the
> > > identity of the sender, and confirm the authenticity of all
links
> > contained
> > > within the message prior to copying and pasting the address to a
Web
> > > browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > I agree with you. That output doesn't look good at all!
> > >
> > > Please just open up the NetCDF output file from Series-Analysis
using
> the
> > > "ncview" tool. Click through the variables and check whether
any of
> them
> > > look reasonable. If they look OK, then the problem is in
plotting them
> > > with plot_data_plane. If so, please just post that Series-
Analysis
> > output
> > > file to our anonymous ftp site? And I'll figure out what's
going on
> with
> > > the plotting.
> > >
> > > But my guess is that none of the fields look good, which means
the
> issue
> > is
> > > coming from Series-Analysis. Please try rerunning your Series-
Analysis
> > > command at verbosity level 5 (-v 5) and then save the output to
a log
> > file
> > > (-log run_series_analysis.log). Then send me that log file.
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Apr 18, 2016 at 4:13 PM, Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <Caution-Caution-Caution-Caution-url: Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >
> > > > John -
> > > >
> > > > I used the following command to plot the variable:
> > > series_cts_CSI_ge285.000
> > > > carson:jraby ~/MET_SeriesAnalysis
> > > > > plot_data_plane results_m3o3_TMP results_m3o3_TMP_CSI.ps
> > > > 'name="series_cts_CSI_ge285.000"; level="(*,*)";'
> > > > DEBUG 1: Opening data file: results_m3o3_TMP
> > > > DEBUG 1: Creating postscript file: results_m3o3_TMP_CSI.ps
> > > > carson:jraby ~/MET_SeriesAnalysis
> > > >
> > > > Seem to run fine, but the plot doesn't look like I expected.
> > > >
> > > > R/
> > > > John
> > > > ________________________________________
> > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > Sent: Monday, April 18, 2016 1:41 PM
> > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > > running series-analysis
> > > >
> > > > All active links contained in this email were disabled.
Please
> verify
> > > the
> > > > identity of the sender, and confirm the authenticity of all
links
> > > contained
> > > > within the message prior to copying and pasting the address to
a Web
> > > > browser.
> > > >
> > > >
> > > >
> > > >
> > > > ----
> > > >
> > > > John,
> > > >
> > > > I see the following error in the log file you sent me:
> > > > ERROR :
> > > > ERROR : plot_data_plane -> trouble getting field
"name="RMSE";
> > > > level="Z2";" from file "results_m3o3_TMP"
> > > > ERROR :
> > > >
> > > > You are calling plot_data_plane by passing it a NetCDF file to
plot.
> > The
> > > > way you set "name" and "level" is different between GRIB files
and
> > > gridded
> > > > NetCDF files. For NetCDF files, you set the name equal to the
name
> of
> > > the
> > > > NetCDF variable you want to plot and you set level to "(*,*)".
I
> know
> > > that
> > > > looks weird but you're telling MET which NetCDF dimensions are
the
> > > gridded
> > > > ones that it should plot.
> > > >
> > > > In your case, the output of Series-Analysis will only have 2
> > > dimensions...
> > > > thus "(*,*)".
> > > >
> > > > Just use "ncdump -h" to look at the name of the variables and
run
> > > > plot_data_plane something like this:
> > > >
> > > > plot_data_plane results_m3o3_TMP results_m3o3_TMP_RMSE.ps
> > > > 'name="series_cnt_RMSE"; level="(*,*)";'
> > > >
> > > > Hope that helps clarify.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Thu, Apr 14, 2016 at 9:14 AM, Raby, John W USA CIV via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <Caution-Caution-Caution-Caution-Caution-url: Caution-
> > > > Caution-Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > >
> > > > > John -
> > > > >
> > > > > I resolved the issue with whitespace by using the name
"RMSE" vice
> > > "Root
> > > > > mean squared error".
> > > > >
> > > > > Now it has an issue with the field "RMSE" from the output
file. See
> > > > > attached log.
> > > > >
> > > > > Thanks.
> > > > >
> > > > > R/
> > > > > John
> > > > >
> > > > > ________________________________________
> > > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > > Sent: Wednesday, April 13, 2016 4:57 PM
> > > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > > > running series-analysis
> > > > >
> > > > > All active links contained in this email were disabled.
Please
> > verify
> > > > the
> > > > > identity of the sender, and confirm the authenticity of all
links
> > > > contained
> > > > > within the message prior to copying and pasting the address
to a
> Web
> > > > > browser.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ----
> > > > >
> > > > > Also, I see the following error message in the log file you
sent:
> > > > >
> > > > > ERROR : store_stat_cnt() -> unsupported column name
requested
> "IQR"
> > > > >
> > > > > IQR was added in version 5.1, but it not defined in version
4.1.
> > > > >
> > > > > John
> > > > >
> > > > > On Wed, Apr 13, 2016 at 4:55 PM, John Halley Gotway <
> johnhg at ucar.edu
> > >
> > > > > wrote:
> > > > >
> > > > > > John,
> > > > > >
> > > > > > Ah yes, I meant to mention this. Please try increasing
the block
> > > size
> > > > > > from 1024 to something much larger.
> > > > > >
> > > > > > Your grid is 288x288 = 82944 grid points. Try setting:
> > > > > > block_size = 100000;
> > > > > >
> > > > > > That will process it all in one pass through the data and
should
> > make
> > > > it
> > > > > > run much faster.
> > > > > >
> > > > > > John
> > > > > >
> > > > > > On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV via
RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > >>
> > > > > >> <Caution-Caution-Caution-Caution-Caution-Caution-url:
Caution-
> > > > > Caution-Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > > >>
> > > > > >> John -
> > > > > >>
> > > > > >> I looked at your config file and it looks like a nice
> workaround.
> > I
> > > > > reran
> > > > > >> using your config file. Ran considerably longer this
time. I
> also
> > > > have a
> > > > > >> 14MB output NetCDF file which is definitely encouraging!
I have
> to
> > > > leave
> > > > > >> for the day in about 10 mins and will be back in touch
tomorrow.
> > > > Thanks
> > > > > for
> > > > > >> your help.
> > > > > >>
> > > > > >> R/
> > > > > >> John
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> ________________________________________
> > > > > >> From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > > >> Sent: Wednesday, April 13, 2016 4:37 PM
> > > > > >> To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#75766]
> > problem
> > > > > >> running series-analysis
> > > > > >>
> > > > > >> All active links contained in this email were disabled.
Please
> > > verify
> > > > > >> the identity of the sender, and confirm the authenticity
of all
> > > links
> > > > > >> contained within the message prior to copying and pasting
the
> > > address
> > > > > to a
> > > > > >> Web browser.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> ----
> > > > > >>
> > > > > >> John,
> > > > > >>
> > > > > >> Ah yes, that time offset is the problem here.
> > > > > >>
> > > > > >> And it would have been nice for Series-Analysis to be
more
> verbose
> > > > about
> > > > > >> the timing information it's processing.
> > > > > >>
> > > > > >> Please try using the updated config file that I've
attached, and
> > see
> > > > if
> > > > > >> Series-Analysis actually runs.
> > > > > >>
> > > > > >> We pass grid-stat 1 forecast file and 1 observation file.
So
> even
> > > if
> > > > > the
> > > > > >> times don't match up exactly, it writes a warning and
compares
> > them.
> > > > > >> Series-Analysis tries to be a bit fancier. You pass it N
> forecast
> > > > and N
> > > > > >> observation files. For each forecast files, it searches
all the
> > > > > >> observation files for a match.
> > > > > >>
> > > > > >> But that more complex logic breaks down when there's a 15
minute
> > > time
> > > > > >> offset. So we should think about how better to handle
this use
> > > case.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> John
> > > > > >>
> > > > > >> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV via
RT <
> > > > > >> met_help at ucar.edu> wrote:
> > > > > >>
> > > > > >> >
> > > > > >> > <Caution-Caution-Caution-Caution-Caution-Caution-
Caution-url: Caution-
> > > > > >> Caution-Caution-
> > > > Caution-Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > > >> >
> > > > > >> > John -
> > > > > >> >
> > > > > >> > Based on your analysis of the reading of the input
files, I
> can
> > > > offer
> > > > > a
> > > > > >> > possibility that I'm aware of regarding the valid
times.The
> > hours
> > > > for
> > > > > >> the
> > > > > >> > example input data (observations) I sent you is indeed
23
> which
> > > > > matches
> > > > > >> > that of the forecast data valid at the same hour (23),
but the
> > > > minutes
> > > > > >> of
> > > > > >> > the forecast file is 00 while for the observations it
is 15. I
> > was
> > > > > >> aware of
> > > > > >> > this mismatch in the minutes from when I ran Grid-Stat
and it
> > > > flagged
> > > > > >> it as
> > > > > >> > a "Warning", but it went ahead and generated output
with no
> > Error.
> > > > The
> > > > > >> > presence of the min value of 15 is an artifact of the
fact
> that
> > > > this
> > > > > >> data
> > > > > >> > was produced by the LAPS system. The valid time of the
data is
> > > > > actually
> > > > > >> > 2300 instead of 2315. That said, I will run wgrib on
the
> > forecast
> > > > file
> > > > > >> per
> > > > > >> > your suggestion to confirm the valid time.
> > > > > >> > R/
> > > > > >> > John
> > > > > >> >
> > > > > >> > ________________________________________
> > > > > >> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > > >> > Sent: Wednesday, April 13, 2016 3:48 PM
> > > > > >> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > >> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#75766]
> > > problem
> > > > > >> > running series-analysis
> > > > > >> >
> > > > > >> > All active links contained in this email were disabled.
> Please
> > > > verify
> > > > > >> the
> > > > > >> > identity of the sender, and confirm the authenticity of
all
> > links
> > > > > >> contained
> > > > > >> > within the message prior to copying and pasting the
address
> to a
> > > Web
> > > > > >> > browser.
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > ----
> > > > > >> >
> > > > > >> > John,
> > > > > >> >
> > > > > >> > I don't see any immediately obvious problems in the
data you
> > sent
> > > to
> > > > > >> us. I
> > > > > >> > see that you're writing a log file named:
> > > > > >> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
> > > > > >> >
> > > > > >> > Are you able to send me that complete log file? That
might
> help
> > > me
> > > > > >> > diagnose what's going on.
> > > > > >> >
> > > > > >> > Here's my initial guess...
> > > > > >> >
> > > > > >> > You are running series-analysis by defining a single
field in
> > the
> > > > > >> > configuration file. And you're passing to it lists of
12
> > forecast
> > > > > files
> > > > > >> > and 12 observation files.
> > > > > >> >
> > > > > >> > It reads TMP/Z2 from the first forecast file and check
the
> valid
> > > > time
> > > > > >> for
> > > > > >> > that data. Next, it looks in the observation file list
> > (starting
> > > > with
> > > > > >> the
> > > > > >> > first file) for TMP/Z2 with the *SAME* valid time.
> > > > > >> >
> > > > > >> > Next, it reads TMP/Z2 from the second forecast file,
gets the
> > > valid
> > > > > >> time,
> > > > > >> > and looks in the observation file list (starting with
the
> second
> > > > file)
> > > > > >> for
> > > > > >> > TMP/Z2 with the *SAME* valid time.
> > > > > >> >
> > > > > >> > I looked at the observation file
(120382300_laps_288.grb1) you
> > > sent
> > > > me
> > > > > >> and
> > > > > >> > see that it's valid time is 2012020723, which happens
to
> match
> > the
> > > > > >> > timestamp in the forecast file name. The first thing
I'd
> check
> > is
> > > > the
> > > > > >> > valid time of the corresponding forecast file:
> > > > > >> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
> > > > > >> >
> > > > > >> > Is it actually valid at the same time, 2012020723? If
not,
> > that's
> > > > the
> > > > > >> > problem. If so, that's not the problem.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > John
> > > > > >> >
> > > > > >> >
> > > > > >> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV
via RT <
> > > > > >> > met_help at ucar.edu> wrote:
> > > > > >> >
> > > > > >> > >
> > > > > >> > >
> <Caution-Caution-Caution-Caution-Caution-Caution-Caution-Caution-
url: Caution-
> > > > > >> > Caution-Caution-
> > > > > Caution-Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > > >> > >
> > > > > >> > > Hi John -
> > > > > >> > >
> > > > > >> > > Yes. I got stuck on trying to figure out the cause of
that
> > > error.
> > > > > >> Seems
> > > > > >> > > like if Grid-Stat can read the 2-m temperature from
the
> > gridded
> > > > obs,
> > > > > >> then
> > > > > >> > > why would Series-Analysis not be able to read it?
> > > > > >> > >
> > > > > >> > > Thanks.
> > > > > >> > >
> > > > > >> > > R/
> > > > > >> > > John
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > -----Original Message-----
> > > > > >> > > From: John Halley Gotway via RT
> > > > [Caution-Caution-Caution-Caution-Caution-Caution-Caution-
Caution-mailto:
> > > > > >> met_help at ucar.edu]
> > > > > >> > > Sent: Wednesday, April 13, 2016 12:34 PM
> > > > > >> > > To: Raby, John W CIV USARMY RDECOM ARL (US) <
> > > > > >> john.w.raby2.civ at mail.mil>
> > > > > >> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu
#75766]
> > problem
> > > > > >> running
> > > > > >> > > series-analysis
> > > > > >> > >
> > > > > >> > > All active links contained in this email were
disabled.
> > Please
> > > > > verify
> > > > > >> > the
> > > > > >> > > identity of the sender, and confirm the authenticity
of all
> > > links
> > > > > >> > contained
> > > > > >> > > within the message prior to copying and pasting the
address
> > to a
> > > > Web
> > > > > >> > > browser.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > ----
> > > > > >> > >
> > > > > >> > > Hello John,
> > > > > >> > >
> > > > > >> > > I see that you were having trouble running series-
analysis
> on
> > > 2-m
> > > > > >> > > temperature data last week.
> > > > > >> > >
> > > > > >> > > Is this still an issue? If so, I'd be happy to take
a look.
> > > > > >> > >
> > > > > >> > > Thanks,
> > > > > >> > > John Halley Gotway
> > > > > >> > >
> > > > > >> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via RT
<
> > > > > >> met_help at ucar.edu
> > > > > >> > >
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > >
> > > > > >> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted
upon.
> > > > > >> > > > Transaction: Given to johnhg (John Halley Gotway)
by
> bullock
> > > > > >> > > > Queue: met_help
> > > > > >> > > > Subject: problem running series-analysis
> > > > > >> > > > Owner: johnhg
> > > > > >> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > > >> > > > Status: open
> > > > > >> > > > Ticket
> > <Caution-Caution-Caution-Caution-Caution-Caution-Caution-Caution-
Caution-url:
> > > > > >> > > > Caution-Caution-
> > > > > >> > Caution-Caution-
> > > > > Caution-Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > This transaction appears to have no content
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
------------------------------------------------
Subject: problem running series-analysis
From: John Halley Gotway
Time: Wed Apr 20 15:32:09 2016
John,
That looks good to me! Perhaps it was all the bootstrapping and rank
correlation stats that was slowing it down to a crawl.
>From my perspective I think you're all set. Just let us know if more
issues or questions arise.
FYI, I'm working on adding logic to Series-Analysis for the next
release
for the user to say the list of forecast files and observation files
are
already "paired" up. When that logic is enabled, your 15 minute
timing
offset will resulting in warnings instead of errors. So you'll get
the
output you want without having to manually define the times in the
configuration file.
Thanks,
John
On Wed, Apr 20, 2016 at 2:18 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> Series-Analysis is restarted with the recommended changes. Appears
to have
> read in all12 files and successfully found the 2m TMP fields, but I
don't
> see screen updates beyond that due to the reduced verbosity.
>
> The run completed after 6 mins duration. The log file indicates no
ERROR
> or WARNING and otherwise appears normal. I plotted the CSI error
using
> plot_data_plane and it looks very promising as good output from a
> successful run compared to the plots from earlier in the week.
>
> What do you think? I'm ready to do more testing or provide more
> information as needed to confirm.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Wednesday, April 20, 2016 1:33 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> OK, I'd suggest running again with:
>
> n_rep = 0
> block_size = 100000
> rank_corr_flag = FALSE
>
> And set "-v 3"... using 4 or higher will cause that very detailed
output
> listing info for each pair. Let's skip over that.
>
> John
>
> And then set reduce the verbosity level
>
> On Wed, Apr 20, 2016 at 12:26 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url: Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> > John -
> >
> > I checked my config file for the particular settings you
mentioned:
> > n_rep = 1000
> > block_size = 100000
> > rank_corr_flag = TRUE
> >
> > The screen dialog says it's processing "with 12 Matched pairs" and
the
> (i,
> > j) values increment j, then cycle through all 287 values of i,
then
> > increment j by one and start over incrementing i for 1-287 and
this is
> > repeating so that now I have j=160.
> >
> > My script is as follows:
> > series_analysis -fcst
~/MET_WRFpostprd/20120207_D_FDDA/fcst_file_list
> -obs
> > ~/MET_obs/GSD_LAPS/20120207/obs_file_list -out
> > ~/MET_SeriesAnalysis/results_m3o3_TMP_041916 -config
> > ~/MET_SeriesAnalysis/SeriesAnalysisConfig_m3o3_gsdlaps_LAX_JHG
-log
> > ~/MET_SeriesAnalysis/logs/run_series_analysis.log -v 5
> >
> > You saw the fcst and obs file lists which I sent you.
> >
> > I could send you the log file which is still being written if you
want
> via
> > SAFE.
> >
> >
> >
> > ________________________________________
> >
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Wednesday, April 20, 2016 10:47 AM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> > running series-analysis
> >
> > All active links contained in this email were disabled. Please
verify
> the
> > identity of the sender, and confirm the authenticity of all links
> contained
> > within the message prior to copying and pasting the address to a
Web
> > browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Running this long is obviously not the desired behavior! Is this
really
> > just being run on series of length 12 and a grid with dimension
288x288?
> > If so, this should run in minutes, not hours.
> >
> > Please take a look at the following settings in the Series-
Analysis
> config
> > file...
> >
> > (1) In the "boot" section, set:
> > n_rep = 0;
> > That will turn off the computation of bootstrap CI's which is
slow.
> >
> > (2) Check to see how "block_size" is set. The default is 1024.
This is
> > the number of grid points to process concurrently. Using the
default
> value
> > of 1024 will require 81 (288*288/104) passes through those files.
And
> > that'll slow it down a lot. Setting it to 82944 will enable it to
> process
> > your data in a single pass (much faster). BUT if your computer
doesn't
> > have enough memory, it'll start using swap space and make it run
REALLY
> > slowly. If you have it set high, try reducing it. If it's set
low, try
> > increasing it.
> >
> > (3) Set "rank_corr_flag = FALSE;" to disable rank correlation
statistics
> > which can be slow.
> >
> > John
> >
> >
> >
> > On Wed, Apr 20, 2016 at 7:12 AM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-url: Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >
> > > John -
> > >
> > > The series-analysis run I started yesterday is still running.
Judging
> > from
> > > the screen output, the grid position info which changes at ~
1sec
> > intervals
> > > to show the point on which the processing is occurring
(categorical and
> > > continuous statistics), it looks like it is about 40% complete.I
> estimate
> > > that it will complete early tomorrow morning.
> > >
> > > R/
> > > John
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Tuesday, April 19, 2016 11:56 AM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > running series-analysis
> > >
> > > All active links contained in this email were disabled. Please
verify
> > the
> > > identity of the sender, and confirm the authenticity of all
links
> > contained
> > > within the message prior to copying and pasting the address to a
Web
> > > browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > Sure, you can use my email address, johnhg at ucar.edu. Although
looking
> > in
> > > the "fcst_file_list" I see you're running Series-Analysis on 12
files.
> > > With just a single fcst/obs file, running it on a series of
length 1
> > won't
> > > be all that informative.
> > >
> > > For now please just send me a verbose log file for your run and
I'll
> look
> > > to see if issues jump out.
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Apr 19, 2016 at 11:51 AM, Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <Caution-Caution-Caution-url: Caution-
> > > Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >
> > > > John -
> > > >
> > > > Thanks for taking a look at that. For large sized files, my
only
> method
> > > of
> > > > transfer to you is by the ARL Secure Access File Exchange
(SAFE)
> which
> > > > requires an email address to which they send you a link to
download
> the
> > > > file. FTP is blocked here at WSMR. In the past, I have used
your
> email
> > > > address @ucar.edu. Will this still work? If so, do you also
want the
> > > > forecast file which has the same valid time (almost-2300Z) as
the
> > > > observations file I sent you which was valid at 2315Z on 07
FEB 2012?
> > > >
> > > > I don't have ncview running here. I did try visualizing the
output
> file
> > > > using the Unidata IDV software and none of the fields looked
right to
> > me.
> > > >
> > > > I'll start working on a re-run of Series-Analysis to generate
the log
> > per
> > > > your instructions.
> > > >
> > > > R/
> > > > John
> > > > ________________________________________
> > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > Sent: Tuesday, April 19, 2016 9:45 AM
> > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > > running series-analysis
> > > >
> > > > All active links contained in this email were disabled.
Please
> verify
> > > the
> > > > identity of the sender, and confirm the authenticity of all
links
> > > contained
> > > > within the message prior to copying and pasting the address to
a Web
> > > > browser.
> > > >
> > > >
> > > >
> > > >
> > > > ----
> > > >
> > > > John,
> > > >
> > > > I agree with you. That output doesn't look good at all!
> > > >
> > > > Please just open up the NetCDF output file from Series-
Analysis using
> > the
> > > > "ncview" tool. Click through the variables and check whether
any of
> > them
> > > > look reasonable. If they look OK, then the problem is in
plotting
> them
> > > > with plot_data_plane. If so, please just post that Series-
Analysis
> > > output
> > > > file to our anonymous ftp site? And I'll figure out what's
going on
> > with
> > > > the plotting.
> > > >
> > > > But my guess is that none of the fields look good, which means
the
> > issue
> > > is
> > > > coming from Series-Analysis. Please try rerunning your
> Series-Analysis
> > > > command at verbosity level 5 (-v 5) and then save the output
to a log
> > > file
> > > > (-log run_series_analysis.log). Then send me that log file.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Mon, Apr 18, 2016 at 4:13 PM, Raby, John W USA CIV via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <Caution-Caution-Caution-Caution-url: Caution-
> > > > Caution-Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > >
> > > > > John -
> > > > >
> > > > > I used the following command to plot the variable:
> > > > series_cts_CSI_ge285.000
> > > > > carson:jraby ~/MET_SeriesAnalysis
> > > > > > plot_data_plane results_m3o3_TMP results_m3o3_TMP_CSI.ps
> > > > > 'name="series_cts_CSI_ge285.000"; level="(*,*)";'
> > > > > DEBUG 1: Opening data file: results_m3o3_TMP
> > > > > DEBUG 1: Creating postscript file: results_m3o3_TMP_CSI.ps
> > > > > carson:jraby ~/MET_SeriesAnalysis
> > > > >
> > > > > Seem to run fine, but the plot doesn't look like I expected.
> > > > >
> > > > > R/
> > > > > John
> > > > > ________________________________________
> > > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > > Sent: Monday, April 18, 2016 1:41 PM
> > > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > > > running series-analysis
> > > > >
> > > > > All active links contained in this email were disabled.
Please
> > verify
> > > > the
> > > > > identity of the sender, and confirm the authenticity of all
links
> > > > contained
> > > > > within the message prior to copying and pasting the address
to a
> Web
> > > > > browser.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ----
> > > > >
> > > > > John,
> > > > >
> > > > > I see the following error in the log file you sent me:
> > > > > ERROR :
> > > > > ERROR : plot_data_plane -> trouble getting field
"name="RMSE";
> > > > > level="Z2";" from file "results_m3o3_TMP"
> > > > > ERROR :
> > > > >
> > > > > You are calling plot_data_plane by passing it a NetCDF file
to
> plot.
> > > The
> > > > > way you set "name" and "level" is different between GRIB
files and
> > > > gridded
> > > > > NetCDF files. For NetCDF files, you set the name equal to
the name
> > of
> > > > the
> > > > > NetCDF variable you want to plot and you set level to
"(*,*)". I
> > know
> > > > that
> > > > > looks weird but you're telling MET which NetCDF dimensions
are the
> > > > gridded
> > > > > ones that it should plot.
> > > > >
> > > > > In your case, the output of Series-Analysis will only have 2
> > > > dimensions...
> > > > > thus "(*,*)".
> > > > >
> > > > > Just use "ncdump -h" to look at the name of the variables
and run
> > > > > plot_data_plane something like this:
> > > > >
> > > > > plot_data_plane results_m3o3_TMP results_m3o3_TMP_RMSE.ps
> > > > > 'name="series_cnt_RMSE"; level="(*,*)";'
> > > > >
> > > > > Hope that helps clarify.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Thu, Apr 14, 2016 at 9:14 AM, Raby, John W USA CIV via RT
<
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <Caution-Caution-Caution-Caution-Caution-url: Caution-
> > > > > Caution-Caution-
> > > Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > > >
> > > > > > John -
> > > > > >
> > > > > > I resolved the issue with whitespace by using the name
"RMSE"
> vice
> > > > "Root
> > > > > > mean squared error".
> > > > > >
> > > > > > Now it has an issue with the field "RMSE" from the output
file.
> See
> > > > > > attached log.
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > R/
> > > > > > John
> > > > > >
> > > > > > ________________________________________
> > > > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > > > Sent: Wednesday, April 13, 2016 4:57 PM
> > > > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
> problem
> > > > > > running series-analysis
> > > > > >
> > > > > > All active links contained in this email were disabled.
Please
> > > verify
> > > > > the
> > > > > > identity of the sender, and confirm the authenticity of
all links
> > > > > contained
> > > > > > within the message prior to copying and pasting the
address to a
> > Web
> > > > > > browser.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ----
> > > > > >
> > > > > > Also, I see the following error message in the log file
you sent:
> > > > > >
> > > > > > ERROR : store_stat_cnt() -> unsupported column name
requested
> > "IQR"
> > > > > >
> > > > > > IQR was added in version 5.1, but it not defined in
version 4.1.
> > > > > >
> > > > > > John
> > > > > >
> > > > > > On Wed, Apr 13, 2016 at 4:55 PM, John Halley Gotway <
> > johnhg at ucar.edu
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > John,
> > > > > > >
> > > > > > > Ah yes, I meant to mention this. Please try increasing
the
> block
> > > > size
> > > > > > > from 1024 to something much larger.
> > > > > > >
> > > > > > > Your grid is 288x288 = 82944 grid points. Try setting:
> > > > > > > block_size = 100000;
> > > > > > >
> > > > > > > That will process it all in one pass through the data
and
> should
> > > make
> > > > > it
> > > > > > > run much faster.
> > > > > > >
> > > > > > > John
> > > > > > >
> > > > > > > On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV
via RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > >>
> > > > > > >> <Caution-Caution-Caution-Caution-Caution-Caution-url:
Caution-
> > > > > > Caution-Caution-
> > > > Caution-Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > > > >>
> > > > > > >> John -
> > > > > > >>
> > > > > > >> I looked at your config file and it looks like a nice
> > workaround.
> > > I
> > > > > > reran
> > > > > > >> using your config file. Ran considerably longer this
time. I
> > also
> > > > > have a
> > > > > > >> 14MB output NetCDF file which is definitely
encouraging! I
> have
> > to
> > > > > leave
> > > > > > >> for the day in about 10 mins and will be back in touch
> tomorrow.
> > > > > Thanks
> > > > > > for
> > > > > > >> your help.
> > > > > > >>
> > > > > > >> R/
> > > > > > >> John
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> ________________________________________
> > > > > > >> From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > > > >> Sent: Wednesday, April 13, 2016 4:37 PM
> > > > > > >> To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#75766]
> > > problem
> > > > > > >> running series-analysis
> > > > > > >>
> > > > > > >> All active links contained in this email were disabled.
> Please
> > > > verify
> > > > > > >> the identity of the sender, and confirm the
authenticity of
> all
> > > > links
> > > > > > >> contained within the message prior to copying and
pasting the
> > > > address
> > > > > > to a
> > > > > > >> Web browser.
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> ----
> > > > > > >>
> > > > > > >> John,
> > > > > > >>
> > > > > > >> Ah yes, that time offset is the problem here.
> > > > > > >>
> > > > > > >> And it would have been nice for Series-Analysis to be
more
> > verbose
> > > > > about
> > > > > > >> the timing information it's processing.
> > > > > > >>
> > > > > > >> Please try using the updated config file that I've
attached,
> and
> > > see
> > > > > if
> > > > > > >> Series-Analysis actually runs.
> > > > > > >>
> > > > > > >> We pass grid-stat 1 forecast file and 1 observation
file. So
> > even
> > > > if
> > > > > > the
> > > > > > >> times don't match up exactly, it writes a warning and
compares
> > > them.
> > > > > > >> Series-Analysis tries to be a bit fancier. You pass it
N
> > forecast
> > > > > and N
> > > > > > >> observation files. For each forecast files, it
searches all
> the
> > > > > > >> observation files for a match.
> > > > > > >>
> > > > > > >> But that more complex logic breaks down when there's a
15
> minute
> > > > time
> > > > > > >> offset. So we should think about how better to handle
this
> use
> > > > case.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> John
> > > > > > >>
> > > > > > >> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV
via RT <
> > > > > > >> met_help at ucar.edu> wrote:
> > > > > > >>
> > > > > > >> >
> > > > > > >> >
> <Caution-Caution-Caution-Caution-Caution-Caution-Caution-url:
Caution-
> > > > > > >> Caution-Caution-
> > > > > Caution-Caution-
> > > Caution-Caution-
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > > > >> >
> > > > > > >> > John -
> > > > > > >> >
> > > > > > >> > Based on your analysis of the reading of the input
files, I
> > can
> > > > > offer
> > > > > > a
> > > > > > >> > possibility that I'm aware of regarding the valid
times.The
> > > hours
> > > > > for
> > > > > > >> the
> > > > > > >> > example input data (observations) I sent you is
indeed 23
> > which
> > > > > > matches
> > > > > > >> > that of the forecast data valid at the same hour
(23), but
> the
> > > > > minutes
> > > > > > >> of
> > > > > > >> > the forecast file is 00 while for the observations it
is
> 15. I
> > > was
> > > > > > >> aware of
> > > > > > >> > this mismatch in the minutes from when I ran Grid-
Stat and
> it
> > > > > flagged
> > > > > > >> it as
> > > > > > >> > a "Warning", but it went ahead and generated output
with no
> > > Error.
> > > > > The
> > > > > > >> > presence of the min value of 15 is an artifact of
the fact
> > that
> > > > > this
> > > > > > >> data
> > > > > > >> > was produced by the LAPS system. The valid time of
the data
> is
> > > > > > actually
> > > > > > >> > 2300 instead of 2315. That said, I will run wgrib on
the
> > > forecast
> > > > > file
> > > > > > >> per
> > > > > > >> > your suggestion to confirm the valid time.
> > > > > > >> > R/
> > > > > > >> > John
> > > > > > >> >
> > > > > > >> > ________________________________________
> > > > > > >> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > > > >> > Sent: Wednesday, April 13, 2016 3:48 PM
> > > > > > >> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > > >> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#75766]
> > > > problem
> > > > > > >> > running series-analysis
> > > > > > >> >
> > > > > > >> > All active links contained in this email were
disabled.
> > Please
> > > > > verify
> > > > > > >> the
> > > > > > >> > identity of the sender, and confirm the authenticity
of all
> > > links
> > > > > > >> contained
> > > > > > >> > within the message prior to copying and pasting the
address
> > to a
> > > > Web
> > > > > > >> > browser.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > ----
> > > > > > >> >
> > > > > > >> > John,
> > > > > > >> >
> > > > > > >> > I don't see any immediately obvious problems in the
data you
> > > sent
> > > > to
> > > > > > >> us. I
> > > > > > >> > see that you're writing a log file named:
> > > > > > >> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
> > > > > > >> >
> > > > > > >> > Are you able to send me that complete log file? That
might
> > help
> > > > me
> > > > > > >> > diagnose what's going on.
> > > > > > >> >
> > > > > > >> > Here's my initial guess...
> > > > > > >> >
> > > > > > >> > You are running series-analysis by defining a single
field
> in
> > > the
> > > > > > >> > configuration file. And you're passing to it lists
of 12
> > > forecast
> > > > > > files
> > > > > > >> > and 12 observation files.
> > > > > > >> >
> > > > > > >> > It reads TMP/Z2 from the first forecast file and
check the
> > valid
> > > > > time
> > > > > > >> for
> > > > > > >> > that data. Next, it looks in the observation file
list
> > > (starting
> > > > > with
> > > > > > >> the
> > > > > > >> > first file) for TMP/Z2 with the *SAME* valid time.
> > > > > > >> >
> > > > > > >> > Next, it reads TMP/Z2 from the second forecast file,
gets
> the
> > > > valid
> > > > > > >> time,
> > > > > > >> > and looks in the observation file list (starting with
the
> > second
> > > > > file)
> > > > > > >> for
> > > > > > >> > TMP/Z2 with the *SAME* valid time.
> > > > > > >> >
> > > > > > >> > I looked at the observation file
(120382300_laps_288.grb1)
> you
> > > > sent
> > > > > me
> > > > > > >> and
> > > > > > >> > see that it's valid time is 2012020723, which happens
to
> > match
> > > the
> > > > > > >> > timestamp in the forecast file name. The first thing
I'd
> > check
> > > is
> > > > > the
> > > > > > >> > valid time of the corresponding forecast file:
> > > > > > >> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
> > > > > > >> >
> > > > > > >> > Is it actually valid at the same time, 2012020723?
If not,
> > > that's
> > > > > the
> > > > > > >> > problem. If so, that's not the problem.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > John
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV
via
> RT <
> > > > > > >> > met_help at ucar.edu> wrote:
> > > > > > >> >
> > > > > > >> > >
> > > > > > >> > >
> > <Caution-Caution-Caution-Caution-Caution-Caution-Caution-Caution-
url:
> Caution-
> > > > > > >> > Caution-Caution-
> > > > > > Caution-Caution-
> > > > Caution-Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > > > >> > >
> > > > > > >> > > Hi John -
> > > > > > >> > >
> > > > > > >> > > Yes. I got stuck on trying to figure out the cause
of that
> > > > error.
> > > > > > >> Seems
> > > > > > >> > > like if Grid-Stat can read the 2-m temperature from
the
> > > gridded
> > > > > obs,
> > > > > > >> then
> > > > > > >> > > why would Series-Analysis not be able to read it?
> > > > > > >> > >
> > > > > > >> > > Thanks.
> > > > > > >> > >
> > > > > > >> > > R/
> > > > > > >> > > John
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > -----Original Message-----
> > > > > > >> > > From: John Halley Gotway via RT
> > > > >
> [Caution-Caution-Caution-Caution-Caution-Caution-Caution-Caution-
mailto:
> > > > > > >> met_help at ucar.edu]
> > > > > > >> > > Sent: Wednesday, April 13, 2016 12:34 PM
> > > > > > >> > > To: Raby, John W CIV USARMY RDECOM ARL (US) <
> > > > > > >> john.w.raby2.civ at mail.mil>
> > > > > > >> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu
#75766]
> > > problem
> > > > > > >> running
> > > > > > >> > > series-analysis
> > > > > > >> > >
> > > > > > >> > > All active links contained in this email were
disabled.
> > > Please
> > > > > > verify
> > > > > > >> > the
> > > > > > >> > > identity of the sender, and confirm the
authenticity of
> all
> > > > links
> > > > > > >> > contained
> > > > > > >> > > within the message prior to copying and pasting the
> address
> > > to a
> > > > > Web
> > > > > > >> > > browser.
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > ----
> > > > > > >> > >
> > > > > > >> > > Hello John,
> > > > > > >> > >
> > > > > > >> > > I see that you were having trouble running series-
analysis
> > on
> > > > 2-m
> > > > > > >> > > temperature data last week.
> > > > > > >> > >
> > > > > > >> > > Is this still an issue? If so, I'd be happy to
take a
> look.
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > John Halley Gotway
> > > > > > >> > >
> > > > > > >> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via
RT <
> > > > > > >> met_help at ucar.edu
> > > > > > >> > >
> > > > > > >> > > wrote:
> > > > > > >> > >
> > > > > > >> > > >
> > > > > > >> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted
upon.
> > > > > > >> > > > Transaction: Given to johnhg (John Halley Gotway)
by
> > bullock
> > > > > > >> > > > Queue: met_help
> > > > > > >> > > > Subject: problem running series-analysis
> > > > > > >> > > > Owner: johnhg
> > > > > > >> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > > > >> > > > Status: open
> > > > > > >> > > > Ticket
> > >
> <Caution-Caution-Caution-Caution-Caution-Caution-Caution-Caution-
Caution-url:
> > > > > > >> > > > Caution-Caution-
> > > > > > >> > Caution-Caution-
> > > > > > Caution-Caution-
> > > > Caution-Caution-
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > This transaction appears to have no content
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem running series-analysis
From: Raby, John W USA CIV
Time: Wed Apr 20 15:58:21 2016
John -
Thanks for all your help is getting series-analysis running. I also
plotted the CNT statistic "RMSE" and it looked good. We are now
scratching our heads over the significance of this new way of seeing
the stats spatially. Great to have new information to consider! think
the proposed change to the logic would be worth it. The user is
probably already aware of what files he's pairing up. I like having
the warning anyway to remind the user.
R/
John
________________________________________
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Wednesday, April 20, 2016 3:32 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
running series-analysis
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
That looks good to me! Perhaps it was all the bootstrapping and rank
correlation stats that was slowing it down to a crawl.
>From my perspective I think you're all set. Just let us know if more
issues or questions arise.
FYI, I'm working on adding logic to Series-Analysis for the next
release
for the user to say the list of forecast files and observation files
are
already "paired" up. When that logic is enabled, your 15 minute
timing
offset will resulting in warnings instead of errors. So you'll get
the
output you want without having to manually define the times in the
configuration file.
Thanks,
John
On Wed, Apr 20, 2016 at 2:18 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
>
> John -
>
> Series-Analysis is restarted with the recommended changes. Appears
to have
> read in all12 files and successfully found the 2m TMP fields, but I
don't
> see screen updates beyond that due to the reduced verbosity.
>
> The run completed after 6 mins duration. The log file indicates no
ERROR
> or WARNING and otherwise appears normal. I plotted the CSI error
using
> plot_data_plane and it looks very promising as good output from a
> successful run compared to the plots from earlier in the week.
>
> What do you think? I'm ready to do more testing or provide more
> information as needed to confirm.
>
> R/
> John
> ________________________________________
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Wednesday, April 20, 2016 1:33 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> running series-analysis
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> OK, I'd suggest running again with:
>
> n_rep = 0
> block_size = 100000
> rank_corr_flag = FALSE
>
> And set "-v 3"... using 4 or higher will cause that very detailed
output
> listing info for each pair. Let's skip over that.
>
> John
>
> And then set reduce the verbosity level
>
> On Wed, Apr 20, 2016 at 12:26 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url: Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> >
> > John -
> >
> > I checked my config file for the particular settings you
mentioned:
> > n_rep = 1000
> > block_size = 100000
> > rank_corr_flag = TRUE
> >
> > The screen dialog says it's processing "with 12 Matched pairs" and
the
> (i,
> > j) values increment j, then cycle through all 287 values of i,
then
> > increment j by one and start over incrementing i for 1-287 and
this is
> > repeating so that now I have j=160.
> >
> > My script is as follows:
> > series_analysis -fcst
~/MET_WRFpostprd/20120207_D_FDDA/fcst_file_list
> -obs
> > ~/MET_obs/GSD_LAPS/20120207/obs_file_list -out
> > ~/MET_SeriesAnalysis/results_m3o3_TMP_041916 -config
> > ~/MET_SeriesAnalysis/SeriesAnalysisConfig_m3o3_gsdlaps_LAX_JHG
-log
> > ~/MET_SeriesAnalysis/logs/run_series_analysis.log -v 5
> >
> > You saw the fcst and obs file lists which I sent you.
> >
> > I could send you the log file which is still being written if you
want
> via
> > SAFE.
> >
> >
> >
> > ________________________________________
> >
> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > Sent: Wednesday, April 20, 2016 10:47 AM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766] problem
> > running series-analysis
> >
> > All active links contained in this email were disabled. Please
verify
> the
> > identity of the sender, and confirm the authenticity of all links
> contained
> > within the message prior to copying and pasting the address to a
Web
> > browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Running this long is obviously not the desired behavior! Is this
really
> > just being run on series of length 12 and a grid with dimension
288x288?
> > If so, this should run in minutes, not hours.
> >
> > Please take a look at the following settings in the Series-
Analysis
> config
> > file...
> >
> > (1) In the "boot" section, set:
> > n_rep = 0;
> > That will turn off the computation of bootstrap CI's which is
slow.
> >
> > (2) Check to see how "block_size" is set. The default is 1024.
This is
> > the number of grid points to process concurrently. Using the
default
> value
> > of 1024 will require 81 (288*288/104) passes through those files.
And
> > that'll slow it down a lot. Setting it to 82944 will enable it to
> process
> > your data in a single pass (much faster). BUT if your computer
doesn't
> > have enough memory, it'll start using swap space and make it run
REALLY
> > slowly. If you have it set high, try reducing it. If it's set
low, try
> > increasing it.
> >
> > (3) Set "rank_corr_flag = FALSE;" to disable rank correlation
statistics
> > which can be slow.
> >
> > John
> >
> >
> >
> > On Wed, Apr 20, 2016 at 7:12 AM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url: Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > >
> > > John -
> > >
> > > The series-analysis run I started yesterday is still running.
Judging
> > from
> > > the screen output, the grid position info which changes at ~
1sec
> > intervals
> > > to show the point on which the processing is occurring
(categorical and
> > > continuous statistics), it looks like it is about 40% complete.I
> estimate
> > > that it will complete early tomorrow morning.
> > >
> > > R/
> > > John
> > > ________________________________________
> > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > Sent: Tuesday, April 19, 2016 11:56 AM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > running series-analysis
> > >
> > > All active links contained in this email were disabled. Please
verify
> > the
> > > identity of the sender, and confirm the authenticity of all
links
> > contained
> > > within the message prior to copying and pasting the address to a
Web
> > > browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > Sure, you can use my email address, johnhg at ucar.edu. Although
looking
> > in
> > > the "fcst_file_list" I see you're running Series-Analysis on 12
files.
> > > With just a single fcst/obs file, running it on a series of
length 1
> > won't
> > > be all that informative.
> > >
> > > For now please just send me a verbose log file for your run and
I'll
> look
> > > to see if issues jump out.
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Apr 19, 2016 at 11:51 AM, Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <Caution-Caution-Caution-Caution-url: Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > >
> > > > John -
> > > >
> > > > Thanks for taking a look at that. For large sized files, my
only
> method
> > > of
> > > > transfer to you is by the ARL Secure Access File Exchange
(SAFE)
> which
> > > > requires an email address to which they send you a link to
download
> the
> > > > file. FTP is blocked here at WSMR. In the past, I have used
your
> email
> > > > address @ucar.edu. Will this still work? If so, do you also
want the
> > > > forecast file which has the same valid time (almost-2300Z) as
the
> > > > observations file I sent you which was valid at 2315Z on 07
FEB 2012?
> > > >
> > > > I don't have ncview running here. I did try visualizing the
output
> file
> > > > using the Unidata IDV software and none of the fields looked
right to
> > me.
> > > >
> > > > I'll start working on a re-run of Series-Analysis to generate
the log
> > per
> > > > your instructions.
> > > >
> > > > R/
> > > > John
> > > > ________________________________________
> > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > Sent: Tuesday, April 19, 2016 9:45 AM
> > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > > running series-analysis
> > > >
> > > > All active links contained in this email were disabled.
Please
> verify
> > > the
> > > > identity of the sender, and confirm the authenticity of all
links
> > > contained
> > > > within the message prior to copying and pasting the address to
a Web
> > > > browser.
> > > >
> > > >
> > > >
> > > >
> > > > ----
> > > >
> > > > John,
> > > >
> > > > I agree with you. That output doesn't look good at all!
> > > >
> > > > Please just open up the NetCDF output file from Series-
Analysis using
> > the
> > > > "ncview" tool. Click through the variables and check whether
any of
> > them
> > > > look reasonable. If they look OK, then the problem is in
plotting
> them
> > > > with plot_data_plane. If so, please just post that Series-
Analysis
> > > output
> > > > file to our anonymous ftp site? And I'll figure out what's
going on
> > with
> > > > the plotting.
> > > >
> > > > But my guess is that none of the fields look good, which means
the
> > issue
> > > is
> > > > coming from Series-Analysis. Please try rerunning your
> Series-Analysis
> > > > command at verbosity level 5 (-v 5) and then save the output
to a log
> > > file
> > > > (-log run_series_analysis.log). Then send me that log file.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Mon, Apr 18, 2016 at 4:13 PM, Raby, John W USA CIV via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <Caution-Caution-Caution-Caution-Caution-url: Caution-
> > > > Caution-Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > >
> > > > > John -
> > > > >
> > > > > I used the following command to plot the variable:
> > > > series_cts_CSI_ge285.000
> > > > > carson:jraby ~/MET_SeriesAnalysis
> > > > > > plot_data_plane results_m3o3_TMP results_m3o3_TMP_CSI.ps
> > > > > 'name="series_cts_CSI_ge285.000"; level="(*,*)";'
> > > > > DEBUG 1: Opening data file: results_m3o3_TMP
> > > > > DEBUG 1: Creating postscript file: results_m3o3_TMP_CSI.ps
> > > > > carson:jraby ~/MET_SeriesAnalysis
> > > > >
> > > > > Seem to run fine, but the plot doesn't look like I expected.
> > > > >
> > > > > R/
> > > > > John
> > > > > ________________________________________
> > > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > > Sent: Monday, April 18, 2016 1:41 PM
> > > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
problem
> > > > > running series-analysis
> > > > >
> > > > > All active links contained in this email were disabled.
Please
> > verify
> > > > the
> > > > > identity of the sender, and confirm the authenticity of all
links
> > > > contained
> > > > > within the message prior to copying and pasting the address
to a
> Web
> > > > > browser.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ----
> > > > >
> > > > > John,
> > > > >
> > > > > I see the following error in the log file you sent me:
> > > > > ERROR :
> > > > > ERROR : plot_data_plane -> trouble getting field
"name="RMSE";
> > > > > level="Z2";" from file "results_m3o3_TMP"
> > > > > ERROR :
> > > > >
> > > > > You are calling plot_data_plane by passing it a NetCDF file
to
> plot.
> > > The
> > > > > way you set "name" and "level" is different between GRIB
files and
> > > > gridded
> > > > > NetCDF files. For NetCDF files, you set the name equal to
the name
> > of
> > > > the
> > > > > NetCDF variable you want to plot and you set level to
"(*,*)". I
> > know
> > > > that
> > > > > looks weird but you're telling MET which NetCDF dimensions
are the
> > > > gridded
> > > > > ones that it should plot.
> > > > >
> > > > > In your case, the output of Series-Analysis will only have 2
> > > > dimensions...
> > > > > thus "(*,*)".
> > > > >
> > > > > Just use "ncdump -h" to look at the name of the variables
and run
> > > > > plot_data_plane something like this:
> > > > >
> > > > > plot_data_plane results_m3o3_TMP results_m3o3_TMP_RMSE.ps
> > > > > 'name="series_cnt_RMSE"; level="(*,*)";'
> > > > >
> > > > > Hope that helps clarify.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Thu, Apr 14, 2016 at 9:14 AM, Raby, John W USA CIV via RT
<
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <Caution-Caution-Caution-Caution-Caution-Caution-url:
Caution-
> > > > > Caution-Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > > >
> > > > > > John -
> > > > > >
> > > > > > I resolved the issue with whitespace by using the name
"RMSE"
> vice
> > > > "Root
> > > > > > mean squared error".
> > > > > >
> > > > > > Now it has an issue with the field "RMSE" from the output
file.
> See
> > > > > > attached log.
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > R/
> > > > > > John
> > > > > >
> > > > > > ________________________________________
> > > > > > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > > > Sent: Wednesday, April 13, 2016 4:57 PM
> > > > > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #75766]
> problem
> > > > > > running series-analysis
> > > > > >
> > > > > > All active links contained in this email were disabled.
Please
> > > verify
> > > > > the
> > > > > > identity of the sender, and confirm the authenticity of
all links
> > > > > contained
> > > > > > within the message prior to copying and pasting the
address to a
> > Web
> > > > > > browser.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > ----
> > > > > >
> > > > > > Also, I see the following error message in the log file
you sent:
> > > > > >
> > > > > > ERROR : store_stat_cnt() -> unsupported column name
requested
> > "IQR"
> > > > > >
> > > > > > IQR was added in version 5.1, but it not defined in
version 4.1.
> > > > > >
> > > > > > John
> > > > > >
> > > > > > On Wed, Apr 13, 2016 at 4:55 PM, John Halley Gotway <
> > johnhg at ucar.edu
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > John,
> > > > > > >
> > > > > > > Ah yes, I meant to mention this. Please try increasing
the
> block
> > > > size
> > > > > > > from 1024 to something much larger.
> > > > > > >
> > > > > > > Your grid is 288x288 = 82944 grid points. Try setting:
> > > > > > > block_size = 100000;
> > > > > > >
> > > > > > > That will process it all in one pass through the data
and
> should
> > > make
> > > > > it
> > > > > > > run much faster.
> > > > > > >
> > > > > > > John
> > > > > > >
> > > > > > > On Wed, Apr 13, 2016 at 4:50 PM, Raby, John W USA CIV
via RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > >>
> > > > > > >> <Caution-Caution-Caution-Caution-Caution-Caution-
Caution-url: Caution-
> > > > > > Caution-Caution-
> > > > Caution-Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > > > >>
> > > > > > >> John -
> > > > > > >>
> > > > > > >> I looked at your config file and it looks like a nice
> > workaround.
> > > I
> > > > > > reran
> > > > > > >> using your config file. Ran considerably longer this
time. I
> > also
> > > > > have a
> > > > > > >> 14MB output NetCDF file which is definitely
encouraging! I
> have
> > to
> > > > > leave
> > > > > > >> for the day in about 10 mins and will be back in touch
> tomorrow.
> > > > > Thanks
> > > > > > for
> > > > > > >> your help.
> > > > > > >>
> > > > > > >> R/
> > > > > > >> John
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> ________________________________________
> > > > > > >> From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > > > >> Sent: Wednesday, April 13, 2016 4:37 PM
> > > > > > >> To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > > >> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#75766]
> > > problem
> > > > > > >> running series-analysis
> > > > > > >>
> > > > > > >> All active links contained in this email were disabled.
> Please
> > > > verify
> > > > > > >> the identity of the sender, and confirm the
authenticity of
> all
> > > > links
> > > > > > >> contained within the message prior to copying and
pasting the
> > > > address
> > > > > > to a
> > > > > > >> Web browser.
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> ----
> > > > > > >>
> > > > > > >> John,
> > > > > > >>
> > > > > > >> Ah yes, that time offset is the problem here.
> > > > > > >>
> > > > > > >> And it would have been nice for Series-Analysis to be
more
> > verbose
> > > > > about
> > > > > > >> the timing information it's processing.
> > > > > > >>
> > > > > > >> Please try using the updated config file that I've
attached,
> and
> > > see
> > > > > if
> > > > > > >> Series-Analysis actually runs.
> > > > > > >>
> > > > > > >> We pass grid-stat 1 forecast file and 1 observation
file. So
> > even
> > > > if
> > > > > > the
> > > > > > >> times don't match up exactly, it writes a warning and
compares
> > > them.
> > > > > > >> Series-Analysis tries to be a bit fancier. You pass it
N
> > forecast
> > > > > and N
> > > > > > >> observation files. For each forecast files, it
searches all
> the
> > > > > > >> observation files for a match.
> > > > > > >>
> > > > > > >> But that more complex logic breaks down when there's a
15
> minute
> > > > time
> > > > > > >> offset. So we should think about how better to handle
this
> use
> > > > case.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> John
> > > > > > >>
> > > > > > >> On Wed, Apr 13, 2016 at 4:20 PM, Raby, John W USA CIV
via RT <
> > > > > > >> met_help at ucar.edu> wrote:
> > > > > > >>
> > > > > > >> >
> > > > > > >> >
> <Caution-Caution-Caution-Caution-Caution-Caution-Caution-Caution-
url: Caution-
> > > > > > >> Caution-Caution-
> > > > > Caution-Caution-
> > > Caution-Caution-
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > > > >> >
> > > > > > >> > John -
> > > > > > >> >
> > > > > > >> > Based on your analysis of the reading of the input
files, I
> > can
> > > > > offer
> > > > > > a
> > > > > > >> > possibility that I'm aware of regarding the valid
times.The
> > > hours
> > > > > for
> > > > > > >> the
> > > > > > >> > example input data (observations) I sent you is
indeed 23
> > which
> > > > > > matches
> > > > > > >> > that of the forecast data valid at the same hour
(23), but
> the
> > > > > minutes
> > > > > > >> of
> > > > > > >> > the forecast file is 00 while for the observations it
is
> 15. I
> > > was
> > > > > > >> aware of
> > > > > > >> > this mismatch in the minutes from when I ran Grid-
Stat and
> it
> > > > > flagged
> > > > > > >> it as
> > > > > > >> > a "Warning", but it went ahead and generated output
with no
> > > Error.
> > > > > The
> > > > > > >> > presence of the min value of 15 is an artifact of
the fact
> > that
> > > > > this
> > > > > > >> data
> > > > > > >> > was produced by the LAPS system. The valid time of
the data
> is
> > > > > > actually
> > > > > > >> > 2300 instead of 2315. That said, I will run wgrib on
the
> > > forecast
> > > > > file
> > > > > > >> per
> > > > > > >> > your suggestion to confirm the valid time.
> > > > > > >> > R/
> > > > > > >> > John
> > > > > > >> >
> > > > > > >> > ________________________________________
> > > > > > >> > From: John Halley Gotway via RT [met_help at ucar.edu]
> > > > > > >> > Sent: Wednesday, April 13, 2016 3:48 PM
> > > > > > >> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > > > > >> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu
#75766]
> > > > problem
> > > > > > >> > running series-analysis
> > > > > > >> >
> > > > > > >> > All active links contained in this email were
disabled.
> > Please
> > > > > verify
> > > > > > >> the
> > > > > > >> > identity of the sender, and confirm the authenticity
of all
> > > links
> > > > > > >> contained
> > > > > > >> > within the message prior to copying and pasting the
address
> > to a
> > > > Web
> > > > > > >> > browser.
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > ----
> > > > > > >> >
> > > > > > >> > John,
> > > > > > >> >
> > > > > > >> > I don't see any immediately obvious problems in the
data you
> > > sent
> > > > to
> > > > > > >> us. I
> > > > > > >> > see that you're writing a log file named:
> > > > > > >> > ~/MET_SeriesAnalysis/logs/m3o3_1.75kmWRE-N_TMP_log
> > > > > > >> >
> > > > > > >> > Are you able to send me that complete log file? That
might
> > help
> > > > me
> > > > > > >> > diagnose what's going on.
> > > > > > >> >
> > > > > > >> > Here's my initial guess...
> > > > > > >> >
> > > > > > >> > You are running series-analysis by defining a single
field
> in
> > > the
> > > > > > >> > configuration file. And you're passing to it lists
of 12
> > > forecast
> > > > > > files
> > > > > > >> > and 12 observation files.
> > > > > > >> >
> > > > > > >> > It reads TMP/Z2 from the first forecast file and
check the
> > valid
> > > > > time
> > > > > > >> for
> > > > > > >> > that data. Next, it looks in the observation file
list
> > > (starting
> > > > > with
> > > > > > >> the
> > > > > > >> > first file) for TMP/Z2 with the *SAME* valid time.
> > > > > > >> >
> > > > > > >> > Next, it reads TMP/Z2 from the second forecast file,
gets
> the
> > > > valid
> > > > > > >> time,
> > > > > > >> > and looks in the observation file list (starting with
the
> > second
> > > > > file)
> > > > > > >> for
> > > > > > >> > TMP/Z2 with the *SAME* valid time.
> > > > > > >> >
> > > > > > >> > I looked at the observation file
(120382300_laps_288.grb1)
> you
> > > > sent
> > > > > me
> > > > > > >> and
> > > > > > >> > see that it's valid time is 2012020723, which happens
to
> > match
> > > the
> > > > > > >> > timestamp in the forecast file name. The first thing
I'd
> > check
> > > is
> > > > > the
> > > > > > >> > valid time of the corresponding forecast file:
> > > > > > >> > 131127A_WRFPRS_d03_2012-02-07_23:00:00
> > > > > > >> >
> > > > > > >> > Is it actually valid at the same time, 2012020723?
If not,
> > > that's
> > > > > the
> > > > > > >> > problem. If so, that's not the problem.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > John
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Wed, Apr 13, 2016 at 1:27 PM, Raby, John W USA CIV
via
> RT <
> > > > > > >> > met_help at ucar.edu> wrote:
> > > > > > >> >
> > > > > > >> > >
> > > > > > >> > >
> > <Caution-Caution-Caution-Caution-Caution-Caution-Caution-Caution-
Caution-url:
> Caution-
> > > > > > >> > Caution-Caution-
> > > > > > Caution-Caution-
> > > > Caution-Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > > > >> > >
> > > > > > >> > > Hi John -
> > > > > > >> > >
> > > > > > >> > > Yes. I got stuck on trying to figure out the cause
of that
> > > > error.
> > > > > > >> Seems
> > > > > > >> > > like if Grid-Stat can read the 2-m temperature from
the
> > > gridded
> > > > > obs,
> > > > > > >> then
> > > > > > >> > > why would Series-Analysis not be able to read it?
> > > > > > >> > >
> > > > > > >> > > Thanks.
> > > > > > >> > >
> > > > > > >> > > R/
> > > > > > >> > > John
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > -----Original Message-----
> > > > > > >> > > From: John Halley Gotway via RT
> > > > >
> [Caution-Caution-Caution-Caution-Caution-Caution-Caution-Caution-
Caution-mailto:
> > > > > > >> met_help at ucar.edu]
> > > > > > >> > > Sent: Wednesday, April 13, 2016 12:34 PM
> > > > > > >> > > To: Raby, John W CIV USARMY RDECOM ARL (US) <
> > > > > > >> john.w.raby2.civ at mail.mil>
> > > > > > >> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu
#75766]
> > > problem
> > > > > > >> running
> > > > > > >> > > series-analysis
> > > > > > >> > >
> > > > > > >> > > All active links contained in this email were
disabled.
> > > Please
> > > > > > verify
> > > > > > >> > the
> > > > > > >> > > identity of the sender, and confirm the
authenticity of
> all
> > > > links
> > > > > > >> > contained
> > > > > > >> > > within the message prior to copying and pasting the
> address
> > > to a
> > > > > Web
> > > > > > >> > > browser.
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > ----
> > > > > > >> > >
> > > > > > >> > > Hello John,
> > > > > > >> > >
> > > > > > >> > > I see that you were having trouble running series-
analysis
> > on
> > > > 2-m
> > > > > > >> > > temperature data last week.
> > > > > > >> > >
> > > > > > >> > > Is this still an issue? If so, I'd be happy to
take a
> look.
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > John Halley Gotway
> > > > > > >> > >
> > > > > > >> > > On Wed, Apr 13, 2016 at 9:38 AM, Randy Bullock via
RT <
> > > > > > >> met_help at ucar.edu
> > > > > > >> > >
> > > > > > >> > > wrote:
> > > > > > >> > >
> > > > > > >> > > >
> > > > > > >> > > > Wed Apr 13 09:38:10 2016: Request 75766 was acted
upon.
> > > > > > >> > > > Transaction: Given to johnhg (John Halley Gotway)
by
> > bullock
> > > > > > >> > > > Queue: met_help
> > > > > > >> > > > Subject: problem running series-analysis
> > > > > > >> > > > Owner: johnhg
> > > > > > >> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > > > >> > > > Status: open
> > > > > > >> > > > Ticket
> > >
> <Caution-Caution-Caution-Caution-Caution-Caution-Caution-Caution-
Caution-Caution-url:
> > > > > > >> > > > Caution-Caution-
> > > > > > >> > Caution-Caution-
> > > > > > Caution-Caution-
> > > > Caution-Caution-
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=75766 >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > This transaction appears to have no content
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
------------------------------------------------
More information about the Met_help
mailing list