[Met_help] [rt.rap.ucar.edu #86674] History for MET V7.0 Grid-Stat concern
John Halley Gotway via RT
met_help at ucar.edu
Fri Sep 7 16:47:46 MDT 2018
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
I am running on an HPC and the run duration for one fcst and observation input (radar corrected reflectivity) was appx 20 mins. This is taking significantly longer than a run I did last OCT. I have attached my config file ("180619B") and the log file for reference. I checked the usual suspects for causing long run times:
rank_corr_flag = FALSE;
n_rep = 0;
These are the settings in my config file. Is there anything else which may be causing the runs to take a long time?
For reference I also attached the config file (MET V5.2) for the run I did in OCT where it took appx 20 secs to generate the output for one set of inputs. This is the file which contains the string "170831A"
Thanks.
R/
John
Mr John W. Raby, Meteorologist
U.S. Army Research Laboratory
White Sands Missile Range, NM 88002
(575) 678-2004 DSN 258-2004
FAX (575) 678-1230 DSN 258-1230
Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: MET V7.0 Grid-Stat concern
From: John Halley Gotway
Time: Mon Aug 20 16:05:41 2018
John,
Can you please check your compilation of met-7.0 to see if it includes
the
bugfix posted on 07/24/18:
https://dtcenter.org/met/users/support/known_issues/METv7.0/index.php
If not, please recompile with that bugfix and retest.
Thanks,
John Halley Gotway
On Mon, Aug 20, 2018 at 1:30 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Mon Aug 20 13:30:01 2018: Request 86674 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: MET V7.0 Grid-Stat concern
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86674 >
>
>
> I am running on an HPC and the run duration for one fcst and
observation
> input (radar corrected reflectivity) was appx 20 mins. This is
taking
> significantly longer than a run I did last OCT. I have attached my
config
> file ("180619B") and the log file for reference. I checked the usual
> suspects for causing long run times:
> rank_corr_flag = FALSE;
> n_rep = 0;
>
> These are the settings in my config file. Is there anything else
which may
> be causing the runs to take a long time?
>
> For reference I also attached the config file (MET V5.2) for the run
I did
> in OCT where it took appx 20 secs to generate the output for one set
of
> inputs. This is the file which contains the string "170831A"
>
> Thanks.
>
> R/
> John
>
>
> Mr John W. Raby, Meteorologist
>
> U.S. Army Research Laboratory
>
> White Sands Missile Range, NM 88002
>
> (575) 678-2004 DSN 258-2004
>
> FAX (575) 678-1230 DSN 258-1230
>
> Email: john.w.raby2.civ at mail.mil<mailto:john.raby at us.army.mil>
>
>
------------------------------------------------
Subject: MET V7.0 Grid-Stat concern
From: Raby, John W USA CIV
Time: Tue Aug 21 16:28:42 2018
CLASSIFICATION: UNCLASSIFIED
John -
I used MET V7.0 with the bugfix by copying the installation directory
from a colleague and placing it in my home on the HPC and I still have
the issue with Grid-Stat running very slow. One fcst and observation
input takes about 20 mins to generate the output. I noticed that there
is a warning in the ncdump output of the attached radar observation
file which says the following:
Warning! ***HDF5 library version mismatched error***
I had to run MET Gen-Vx-Mask to produce the radar observation with no
missing values and the version I used for that was MET V7.0. If I use
MET V5.2 Gen-Vx-Mask, I don’t see that warning in the ncdump output
and then I can run V5.2 Grid-Stat which runs fine. See the attached
script which I use to run Gen-Vx-Mask.
Could you assess this situation to see what is the issue and what is
the best COA to resolve it? Let me know if you need other files or
info.
Thanks.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, August 20, 2018 4:06 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0 Grid-
Stat concern
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
Can you please check your compilation of met-7.0 to see if it includes
the bugfix posted on 07/24/18:
Caution-
https://dtcenter.org/met/users/support/known_issues/METv7.0/index.php
If not, please recompile with that bugfix and retest.
Thanks,
John Halley Gotway
On Mon, Aug 20, 2018 at 1:30 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Mon Aug 20 13:30:01 2018: Request 86674 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: MET V7.0 Grid-Stat concern
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86674 >
>
>
> I am running on an HPC and the run duration for one fcst and
> observation input (radar corrected reflectivity) was appx 20 mins.
> This is taking significantly longer than a run I did last OCT. I
have
> attached my config file ("180619B") and the log file for reference.
I
> checked the usual suspects for causing long run times:
> rank_corr_flag = FALSE;
> n_rep = 0;
>
> These are the settings in my config file. Is there anything else
which
> may be causing the runs to take a long time?
>
> For reference I also attached the config file (MET V5.2) for the run
I
> did in OCT where it took appx 20 secs to generate the output for one
> set of inputs. This is the file which contains the string "170831A"
>
> Thanks.
>
> R/
> John
>
>
> Mr John W. Raby, Meteorologist
>
> U.S. Army Research Laboratory
>
> White Sands Missile Range, NM 88002
>
> (575) 678-2004 DSN 258-2004
>
> FAX (575) 678-1230 DSN 258-1230
>
> Email: john.w.raby2.civ at mail.mil<Caution-
mailto:john.raby at us.army.mil>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: MET V7.0 Grid-Stat concern
From: John Halley Gotway
Time: Tue Aug 21 16:33:39 2018
John,
One more thing to check. Are you using a high NetCDF compression
level? A
user at NASA reported that it takes an exceptionally long time for MET
to
read NetCDF data that has been compressed to a high level.
I wonder also if perhaps the version “mismatch” is producing a similar
latency.
John
On Tue, Aug 21, 2018 at 4:29 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86674 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> I used MET V7.0 with the bugfix by copying the installation
directory from
> a colleague and placing it in my home on the HPC and I still have
the issue
> with Grid-Stat running very slow. One fcst and observation input
takes
> about 20 mins to generate the output. I noticed that there is a
warning in
> the ncdump output of the attached radar observation file which says
the
> following:
> Warning! ***HDF5 library version mismatched error***
>
> I had to run MET Gen-Vx-Mask to produce the radar observation with
no
> missing values and the version I used for that was MET V7.0. If I
use MET
> V5.2 Gen-Vx-Mask, I don’t see that warning in the ncdump output and
then I
> can run V5.2 Grid-Stat which runs fine. See the attached script
which I use
> to run Gen-Vx-Mask.
>
> Could you assess this situation to see what is the issue and what is
the
> best COA to resolve it? Let me know if you need other files or info.
>
> Thanks.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Monday, August 20, 2018 4:06 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0
Grid-Stat
> concern
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Can you please check your compilation of met-7.0 to see if it
includes the
> bugfix posted on 07/24/18:
> Caution-
>
https://dtcenter.org/met/users/support/known_issues/METv7.0/index.php
>
> If not, please recompile with that bugfix and retest.
>
> Thanks,
> John Halley Gotway
>
> On Mon, Aug 20, 2018 at 1:30 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Aug 20 13:30:01 2018: Request 86674 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: MET V7.0 Grid-Stat concern
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86674 >
> >
> >
> > I am running on an HPC and the run duration for one fcst and
> > observation input (radar corrected reflectivity) was appx 20 mins.
> > This is taking significantly longer than a run I did last OCT. I
have
> > attached my config file ("180619B") and the log file for
reference. I
> > checked the usual suspects for causing long run times:
> > rank_corr_flag = FALSE;
> > n_rep = 0;
> >
> > These are the settings in my config file. Is there anything else
which
> > may be causing the runs to take a long time?
> >
> > For reference I also attached the config file (MET V5.2) for the
run I
> > did in OCT where it took appx 20 secs to generate the output for
one
> > set of inputs. This is the file which contains the string
"170831A"
> >
> > Thanks.
> >
> > R/
> > John
> >
> >
> > Mr John W. Raby, Meteorologist
> >
> > U.S. Army Research Laboratory
> >
> > White Sands Missile Range, NM 88002
> >
> > (575) 678-2004 DSN 258-2004
> >
> > FAX (575) 678-1230 DSN 258-1230
> >
> > Email: john.w.raby2.civ at mail.mil<Caution-
mailto:john.raby at us.army.mil>
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0 Grid-Stat concern (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Aug 21 16:40:21 2018
CLASSIFICATION: UNCLASSIFIED
John -
I don't know. I forwarded your question to Huaqing Cai upstairs. I'll
send his reply when he responds.
I used IDV to view the radar ob and it visualizes it fine. There
doesn't seem to be a problem with the data despite the warning which
appears when I use ncdump to display the header info.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, August 21, 2018 4:34 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0
Grid-Stat concern (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
One more thing to check. Are you using a high NetCDF compression
level? A user at NASA reported that it takes an exceptionally long
time for MET to read NetCDF data that has been compressed to a high
level.
I wonder also if perhaps the version “mismatch” is producing a similar
latency.
John
On Tue, Aug 21, 2018 at 4:29 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86674 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> I used MET V7.0 with the bugfix by copying the installation
directory
> from a colleague and placing it in my home on the HPC and I still
have
> the issue with Grid-Stat running very slow. One fcst and observation
> input takes about 20 mins to generate the output. I noticed that
there
> is a warning in the ncdump output of the attached radar observation
> file which says the
> following:
> Warning! ***HDF5 library version mismatched error***
>
> I had to run MET Gen-Vx-Mask to produce the radar observation with
no
> missing values and the version I used for that was MET V7.0. If I
use
> MET
> V5.2 Gen-Vx-Mask, I don’t see that warning in the ncdump output and
> then I can run V5.2 Grid-Stat which runs fine. See the attached
script
> which I use to run Gen-Vx-Mask.
>
> Could you assess this situation to see what is the issue and what is
> the best COA to resolve it? Let me know if you need other files or
info.
>
> Thanks.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Monday, August 20, 2018 4:06 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0
> Grid-Stat concern
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> Can you please check your compilation of met-7.0 to see if it
includes
> the bugfix posted on 07/24/18:
> Caution-
> Caution-
https://dtcenter.org/met/users/support/known_issues/METv7.0/in
> dex.php
>
> If not, please recompile with that bugfix and retest.
>
> Thanks,
> John Halley Gotway
>
> On Mon, Aug 20, 2018 at 1:30 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Aug 20 13:30:01 2018: Request 86674 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: MET V7.0 Grid-Stat concern
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86
> > 674 >
> >
> >
> > I am running on an HPC and the run duration for one fcst and
> > observation input (radar corrected reflectivity) was appx 20 mins.
> > This is taking significantly longer than a run I did last OCT. I
> > have attached my config file ("180619B") and the log file for
> > reference. I checked the usual suspects for causing long run
times:
> > rank_corr_flag = FALSE;
> > n_rep = 0;
> >
> > These are the settings in my config file. Is there anything else
> > which may be causing the runs to take a long time?
> >
> > For reference I also attached the config file (MET V5.2) for the
run
> > I did in OCT where it took appx 20 secs to generate the output for
> > one set of inputs. This is the file which contains the string
"170831A"
> >
> > Thanks.
> >
> > R/
> > John
> >
> >
> > Mr John W. Raby, Meteorologist
> >
> > U.S. Army Research Laboratory
> >
> > White Sands Missile Range, NM 88002
> >
> > (575) 678-2004 DSN 258-2004
> >
> > FAX (575) 678-1230 DSN 258-1230
> >
> > Email:
> > john.w.raby2.civ at mail.mil<Caution-Caution-
mailto:john.raby at us.army.m
> > il>
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0 Grid-Stat concern (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Aug 21 16:43:02 2018
CLASSIFICATION: UNCLASSIFIED
John -
Huaqing says he used level 4 which is not very high.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, August 21, 2018 4:34 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0
Grid-Stat concern (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
One more thing to check. Are you using a high NetCDF compression
level? A user at NASA reported that it takes an exceptionally long
time for MET to read NetCDF data that has been compressed to a high
level.
I wonder also if perhaps the version “mismatch” is producing a similar
latency.
John
On Tue, Aug 21, 2018 at 4:29 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86674 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> I used MET V7.0 with the bugfix by copying the installation
directory
> from a colleague and placing it in my home on the HPC and I still
have
> the issue with Grid-Stat running very slow. One fcst and observation
> input takes about 20 mins to generate the output. I noticed that
there
> is a warning in the ncdump output of the attached radar observation
> file which says the
> following:
> Warning! ***HDF5 library version mismatched error***
>
> I had to run MET Gen-Vx-Mask to produce the radar observation with
no
> missing values and the version I used for that was MET V7.0. If I
use
> MET
> V5.2 Gen-Vx-Mask, I don’t see that warning in the ncdump output and
> then I can run V5.2 Grid-Stat which runs fine. See the attached
script
> which I use to run Gen-Vx-Mask.
>
> Could you assess this situation to see what is the issue and what is
> the best COA to resolve it? Let me know if you need other files or
info.
>
> Thanks.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Monday, August 20, 2018 4:06 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0
> Grid-Stat concern
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> Can you please check your compilation of met-7.0 to see if it
includes
> the bugfix posted on 07/24/18:
> Caution-
> Caution-
https://dtcenter.org/met/users/support/known_issues/METv7.0/in
> dex.php
>
> If not, please recompile with that bugfix and retest.
>
> Thanks,
> John Halley Gotway
>
> On Mon, Aug 20, 2018 at 1:30 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Aug 20 13:30:01 2018: Request 86674 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: MET V7.0 Grid-Stat concern
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86
> > 674 >
> >
> >
> > I am running on an HPC and the run duration for one fcst and
> > observation input (radar corrected reflectivity) was appx 20 mins.
> > This is taking significantly longer than a run I did last OCT. I
> > have attached my config file ("180619B") and the log file for
> > reference. I checked the usual suspects for causing long run
times:
> > rank_corr_flag = FALSE;
> > n_rep = 0;
> >
> > These are the settings in my config file. Is there anything else
> > which may be causing the runs to take a long time?
> >
> > For reference I also attached the config file (MET V5.2) for the
run
> > I did in OCT where it took appx 20 secs to generate the output for
> > one set of inputs. This is the file which contains the string
"170831A"
> >
> > Thanks.
> >
> > R/
> > John
> >
> >
> > Mr John W. Raby, Meteorologist
> >
> > U.S. Army Research Laboratory
> >
> > White Sands Missile Range, NM 88002
> >
> > (575) 678-2004 DSN 258-2004
> >
> > FAX (575) 678-1230 DSN 258-1230
> >
> > Email:
> > john.w.raby2.civ at mail.mil<Caution-Caution-
mailto:john.raby at us.army.m
> > il>
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0 Grid-Stat concern (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Fri Aug 31 14:04:39 2018
CLASSIFICATION: UNCLASSIFIED
John -
My ARL colleague in Adelphi, MD compiled MET V7.0 with the bugfilx
posted on 07/24/18. He is running MET Grid-Stat on his system and he
is still getting very long run times. For the same input data V5.2
Grid-Stat runs in about 26 mins. Based on testing, I estimate that
V7.0 Grid-Stat would take between 14 and 16 hours to complete. Huaqing
Cai confirmed that the compression level we are using for the NetCDF
radar observations is level 4 (not high). The fcst data is GRIB from
UPP.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, August 21, 2018 4:34 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0
Grid-Stat concern (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
One more thing to check. Are you using a high NetCDF compression
level? A user at NASA reported that it takes an exceptionally long
time for MET to read NetCDF data that has been compressed to a high
level.
I wonder also if perhaps the version “mismatch” is producing a similar
latency.
John
On Tue, Aug 21, 2018 at 4:29 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86674 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> I used MET V7.0 with the bugfix by copying the installation
directory
> from a colleague and placing it in my home on the HPC and I still
have
> the issue with Grid-Stat running very slow. One fcst and observation
> input takes about 20 mins to generate the output. I noticed that
there
> is a warning in the ncdump output of the attached radar observation
> file which says the
> following:
> Warning! ***HDF5 library version mismatched error***
>
> I had to run MET Gen-Vx-Mask to produce the radar observation with
no
> missing values and the version I used for that was MET V7.0. If I
use
> MET
> V5.2 Gen-Vx-Mask, I don’t see that warning in the ncdump output and
> then I can run V5.2 Grid-Stat which runs fine. See the attached
script
> which I use to run Gen-Vx-Mask.
>
> Could you assess this situation to see what is the issue and what is
> the best COA to resolve it? Let me know if you need other files or
info.
>
> Thanks.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Monday, August 20, 2018 4:06 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0
> Grid-Stat concern
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> Can you please check your compilation of met-7.0 to see if it
includes
> the bugfix posted on 07/24/18:
> Caution-
> Caution-
https://dtcenter.org/met/users/support/known_issues/METv7.0/in
> dex.php
>
> If not, please recompile with that bugfix and retest.
>
> Thanks,
> John Halley Gotway
>
> On Mon, Aug 20, 2018 at 1:30 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Mon Aug 20 13:30:01 2018: Request 86674 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: MET V7.0 Grid-Stat concern
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86
> > 674 >
> >
> >
> > I am running on an HPC and the run duration for one fcst and
> > observation input (radar corrected reflectivity) was appx 20 mins.
> > This is taking significantly longer than a run I did last OCT. I
> > have attached my config file ("180619B") and the log file for
> > reference. I checked the usual suspects for causing long run
times:
> > rank_corr_flag = FALSE;
> > n_rep = 0;
> >
> > These are the settings in my config file. Is there anything else
> > which may be causing the runs to take a long time?
> >
> > For reference I also attached the config file (MET V5.2) for the
run
> > I did in OCT where it took appx 20 secs to generate the output for
> > one set of inputs. This is the file which contains the string
"170831A"
> >
> > Thanks.
> >
> > R/
> > John
> >
> >
> > Mr John W. Raby, Meteorologist
> >
> > U.S. Army Research Laboratory
> >
> > White Sands Missile Range, NM 88002
> >
> > (575) 678-2004 DSN 258-2004
> >
> > FAX (575) 678-1230 DSN 258-1230
> >
> > Email:
> > john.w.raby2.civ at mail.mil<Caution-Caution-
mailto:john.raby at us.army.m
> > il>
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: MET V7.0 Grid-Stat concern
From: John Halley Gotway
Time: Tue Sep 04 16:11:31 2018
John,
I went hunting back through this ticket looking for the config file
you're
using, but don't see it. I'd like check to see if you're computing
fractions skill score... i.e. are you requesting the NBRCNT, NBRCTC,
or
NBRCTS line types?
It's come to our attention over the last week that Grid-Stat in met-
7.0 is
much slower in computing the fractional coverage fields used in Grid-
Stat.
This issue arose when adding support for circular neighborhoods. The
switch to using different data structures for computing the fractional
coverage led to a huge slowdown.
I wonder if this would explain the behavior you're seeing?
In the development version of the code, I've refactored that algorithm
to
speed it back up. For comparison, a particular configuration of grid-
stat
in met-6.0 takes 1:30 to complete. In met-7.0, it takes 10:30. In
the
current development version, it takes 1:50. I do have an idea of how
we
might further speed it up, but it won't get much faster than 1:45 or
so.
Thanks,
John
On Fri, Aug 31, 2018 at 2:05 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86674 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> My ARL colleague in Adelphi, MD compiled MET V7.0 with the bugfilx
posted
> on 07/24/18. He is running MET Grid-Stat on his system and he is
still
> getting very long run times. For the same input data V5.2 Grid-Stat
runs in
> about 26 mins. Based on testing, I estimate that V7.0 Grid-Stat
would take
> between 14 and 16 hours to complete. Huaqing Cai confirmed that the
> compression level we are using for the NetCDF radar observations is
level 4
> (not high). The fcst data is GRIB from UPP.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, August 21, 2018 4:34 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0
> Grid-Stat concern (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> One more thing to check. Are you using a high NetCDF compression
level?
> A user at NASA reported that it takes an exceptionally long time for
MET to
> read NetCDF data that has been compressed to a high level.
>
> I wonder also if perhaps the version “mismatch” is producing a
similar
> latency.
>
> John
>
> On Tue, Aug 21, 2018 at 4:29 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86674 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > I used MET V7.0 with the bugfix by copying the installation
directory
> > from a colleague and placing it in my home on the HPC and I still
have
> > the issue with Grid-Stat running very slow. One fcst and
observation
> > input takes about 20 mins to generate the output. I noticed that
there
> > is a warning in the ncdump output of the attached radar
observation
> > file which says the
> > following:
> > Warning! ***HDF5 library version mismatched error***
> >
> > I had to run MET Gen-Vx-Mask to produce the radar observation with
no
> > missing values and the version I used for that was MET V7.0. If I
use
> > MET
> > V5.2 Gen-Vx-Mask, I don’t see that warning in the ncdump output
and
> > then I can run V5.2 Grid-Stat which runs fine. See the attached
script
> > which I use to run Gen-Vx-Mask.
> >
> > Could you assess this situation to see what is the issue and what
is
> > the best COA to resolve it? Let me know if you need other files or
info.
> >
> > Thanks.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> > Sent: Monday, August 20, 2018 4:06 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0
> > Grid-Stat concern
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Can you please check your compilation of met-7.0 to see if it
includes
> > the bugfix posted on 07/24/18:
> > Caution-
> > Caution-
https://dtcenter.org/met/users/support/known_issues/METv7.0/in
> > dex.php
> >
> > If not, please recompile with that bugfix and retest.
> >
> > Thanks,
> > John Halley Gotway
> >
> > On Mon, Aug 20, 2018 at 1:30 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Mon Aug 20 13:30:01 2018: Request 86674 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > Queue: met_help
> > > Subject: MET V7.0 Grid-Stat concern
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: new
> > > Ticket <Caution-Caution-url:
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86
> > > 674 >
> > >
> > >
> > > I am running on an HPC and the run duration for one fcst and
> > > observation input (radar corrected reflectivity) was appx 20
mins.
> > > This is taking significantly longer than a run I did last OCT. I
> > > have attached my config file ("180619B") and the log file for
> > > reference. I checked the usual suspects for causing long run
times:
> > > rank_corr_flag = FALSE;
> > > n_rep = 0;
> > >
> > > These are the settings in my config file. Is there anything else
> > > which may be causing the runs to take a long time?
> > >
> > > For reference I also attached the config file (MET V5.2) for the
run
> > > I did in OCT where it took appx 20 secs to generate the output
for
> > > one set of inputs. This is the file which contains the string
"170831A"
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > >
> > > Mr John W. Raby, Meteorologist
> > >
> > > U.S. Army Research Laboratory
> > >
> > > White Sands Missile Range, NM 88002
> > >
> > > (575) 678-2004 DSN 258-2004
> > >
> > > FAX (575) 678-1230 DSN 258-1230
> > >
> > > Email:
> > > john.w.raby2.civ at mail.mil<Caution-Caution-
mailto:john.raby at us.army.m
> > > il>
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0 Grid-Stat concern (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Tue Sep 04 16:52:10 2018
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for following up. Yes, we are specifying generation of FSS and
the line types you list below. I have to depart for the day right now,
but I can go into more detail about what we are observing tomorrow.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, September 4, 2018 4:12 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0
Grid-Stat concern (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
I went hunting back through this ticket looking for the config file
you're using, but don't see it. I'd like check to see if you're
computing fractions skill score... i.e. are you requesting the NBRCNT,
NBRCTC, or NBRCTS line types?
It's come to our attention over the last week that Grid-Stat in met-
7.0 is much slower in computing the fractional coverage fields used in
Grid-Stat.
This issue arose when adding support for circular neighborhoods. The
switch to using different data structures for computing the fractional
coverage led to a huge slowdown.
I wonder if this would explain the behavior you're seeing?
In the development version of the code, I've refactored that algorithm
to speed it back up. For comparison, a particular configuration of
grid-stat in met-6.0 takes 1:30 to complete. In met-7.0, it takes
10:30. In the current development version, it takes 1:50. I do have
an idea of how we might further speed it up, but it won't get much
faster than 1:45 or so.
Thanks,
John
On Fri, Aug 31, 2018 at 2:05 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86674 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> My ARL colleague in Adelphi, MD compiled MET V7.0 with the bugfilx
> posted on 07/24/18. He is running MET Grid-Stat on his system and he
> is still getting very long run times. For the same input data V5.2
> Grid-Stat runs in about 26 mins. Based on testing, I estimate that
> V7.0 Grid-Stat would take between 14 and 16 hours to complete.
Huaqing
> Cai confirmed that the compression level we are using for the NetCDF
> radar observations is level 4 (not high). The fcst data is GRIB from
UPP.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Tuesday, August 21, 2018 4:34 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0
> Grid-Stat concern (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> One more thing to check. Are you using a high NetCDF compression
level?
> A user at NASA reported that it takes an exceptionally long time for
> MET to read NetCDF data that has been compressed to a high level.
>
> I wonder also if perhaps the version “mismatch” is producing a
similar
> latency.
>
> John
>
> On Tue, Aug 21, 2018 at 4:29 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86
> > 674 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > I used MET V7.0 with the bugfix by copying the installation
> > directory from a colleague and placing it in my home on the HPC
and
> > I still have the issue with Grid-Stat running very slow. One fcst
> > and observation input takes about 20 mins to generate the output.
I
> > noticed that there is a warning in the ncdump output of the
attached
> > radar observation file which says the
> > following:
> > Warning! ***HDF5 library version mismatched error***
> >
> > I had to run MET Gen-Vx-Mask to produce the radar observation with
> > no missing values and the version I used for that was MET V7.0. If
I
> > use MET
> > V5.2 Gen-Vx-Mask, I don’t see that warning in the ncdump output
and
> > then I can run V5.2 Grid-Stat which runs fine. See the attached
> > script which I use to run Gen-Vx-Mask.
> >
> > Could you assess this situation to see what is the issue and what
is
> > the best COA to resolve it? Let me know if you need other files or
info.
> >
> > Thanks.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT
> > [Caution-Caution-mailto:met_help at ucar.edu]
> > Sent: Monday, August 20, 2018 4:06 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0
> > Grid-Stat concern
> >
> > All active links contained in this email were disabled. Please
> > verify the identity of the sender, and confirm the authenticity of
> > all links contained within the message prior to copying and
pasting
> > the address to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Can you please check your compilation of met-7.0 to see if it
> > includes the bugfix posted on 07/24/18:
> > Caution-
> > Caution-Caution-
https://dtcenter.org/met/users/support/known_issues/
> > METv7.0/in
> > dex.php
> >
> > If not, please recompile with that bugfix and retest.
> >
> > Thanks,
> > John Halley Gotway
> >
> > On Mon, Aug 20, 2018 at 1:30 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Mon Aug 20 13:30:01 2018: Request 86674 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > Queue: met_help
> > > Subject: MET V7.0 Grid-Stat concern
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: new
> > > Ticket <Caution-Caution-Caution-url:
> > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > html?id=86
> > > 674 >
> > >
> > >
> > > I am running on an HPC and the run duration for one fcst and
> > > observation input (radar corrected reflectivity) was appx 20
mins.
> > > This is taking significantly longer than a run I did last OCT. I
> > > have attached my config file ("180619B") and the log file for
> > > reference. I checked the usual suspects for causing long run
times:
> > > rank_corr_flag = FALSE;
> > > n_rep = 0;
> > >
> > > These are the settings in my config file. Is there anything else
> > > which may be causing the runs to take a long time?
> > >
> > > For reference I also attached the config file (MET V5.2) for the
> > > run I did in OCT where it took appx 20 secs to generate the
output
> > > for one set of inputs. This is the file which contains the
string "170831A"
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > >
> > > Mr John W. Raby, Meteorologist
> > >
> > > U.S. Army Research Laboratory
> > >
> > > White Sands Missile Range, NM 88002
> > >
> > > (575) 678-2004 DSN 258-2004
> > >
> > > FAX (575) 678-1230 DSN 258-1230
> > >
> > > Email:
> > > john.w.raby2.civ at mail.mil<Caution-Caution-Caution-
mailto:john.raby
> > > @us.army.m
> > > il>
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: MET V7.0 Grid-Stat concern
From: John Halley Gotway
Time: Fri Sep 07 16:36:51 2018
John,
We've heard this feedback about met-7.0 being very slow in computing
neighborhood methods from a few different users via met-help. So we
decided to post a met-7.0 bugfix for this:
https://dtcenter.org/met/users/support/known_issues/METv7.0/index.php
On Tue, Sep 4, 2018 at 4:52 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86674 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for following up. Yes, we are specifying generation of FSS
and the
> line types you list below. I have to depart for the day right now,
but I
> can go into more detail about what we are observing tomorrow.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, September 4, 2018 4:12 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0
> Grid-Stat concern (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> I went hunting back through this ticket looking for the config file
you're
> using, but don't see it. I'd like check to see if you're computing
> fractions skill score... i.e. are you requesting the NBRCNT, NBRCTC,
or
> NBRCTS line types?
>
> It's come to our attention over the last week that Grid-Stat in met-
7.0 is
> much slower in computing the fractional coverage fields used in
Grid-Stat.
> This issue arose when adding support for circular neighborhoods.
The
> switch to using different data structures for computing the
fractional
> coverage led to a huge slowdown.
>
> I wonder if this would explain the behavior you're seeing?
>
> In the development version of the code, I've refactored that
algorithm to
> speed it back up. For comparison, a particular configuration of
grid-stat
> in met-6.0 takes 1:30 to complete. In met-7.0, it takes 10:30. In
the
> current development version, it takes 1:50. I do have an idea of
how we
> might further speed it up, but it won't get much faster than 1:45 or
so.
>
> Thanks,
> John
>
> On Fri, Aug 31, 2018 at 2:05 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86674 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > My ARL colleague in Adelphi, MD compiled MET V7.0 with the bugfilx
> > posted on 07/24/18. He is running MET Grid-Stat on his system and
he
> > is still getting very long run times. For the same input data V5.2
> > Grid-Stat runs in about 26 mins. Based on testing, I estimate that
> > V7.0 Grid-Stat would take between 14 and 16 hours to complete.
Huaqing
> > Cai confirmed that the compression level we are using for the
NetCDF
> > radar observations is level 4 (not high). The fcst data is GRIB
from UPP.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> > Sent: Tuesday, August 21, 2018 4:34 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET
V7.0
> > Grid-Stat concern (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > One more thing to check. Are you using a high NetCDF compression
level?
> > A user at NASA reported that it takes an exceptionally long time
for
> > MET to read NetCDF data that has been compressed to a high level.
> >
> > I wonder also if perhaps the version “mismatch” is producing a
similar
> > latency.
> >
> > John
> >
> > On Tue, Aug 21, 2018 at 4:29 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-url:
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86
> > > 674 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > John -
> > >
> > > I used MET V7.0 with the bugfix by copying the installation
> > > directory from a colleague and placing it in my home on the HPC
and
> > > I still have the issue with Grid-Stat running very slow. One
fcst
> > > and observation input takes about 20 mins to generate the
output. I
> > > noticed that there is a warning in the ncdump output of the
attached
> > > radar observation file which says the
> > > following:
> > > Warning! ***HDF5 library version mismatched error***
> > >
> > > I had to run MET Gen-Vx-Mask to produce the radar observation
with
> > > no missing values and the version I used for that was MET V7.0.
If I
> > > use MET
> > > V5.2 Gen-Vx-Mask, I don’t see that warning in the ncdump output
and
> > > then I can run V5.2 Grid-Stat which runs fine. See the attached
> > > script which I use to run Gen-Vx-Mask.
> > >
> > > Could you assess this situation to see what is the issue and
what is
> > > the best COA to resolve it? Let me know if you need other files
or
> info.
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT
> > > [Caution-Caution-mailto:met_help at ucar.edu]
> > > Sent: Monday, August 20, 2018 4:06 PM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0
> > > Grid-Stat concern
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
pasting
> > > the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > Can you please check your compilation of met-7.0 to see if it
> > > includes the bugfix posted on 07/24/18:
> > > Caution-
> > > Caution-Caution-
https://dtcenter.org/met/users/support/known_issues/
> > > METv7.0/in
> > > dex.php
> > >
> > > If not, please recompile with that bugfix and retest.
> > >
> > > Thanks,
> > > John Halley Gotway
> > >
> > > On Mon, Aug 20, 2018 at 1:30 PM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Mon Aug 20 13:30:01 2018: Request 86674 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > Queue: met_help
> > > > Subject: MET V7.0 Grid-Stat concern
> > > > Owner: Nobody
> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > Status: new
> > > > Ticket <Caution-Caution-Caution-url:
> > > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > html?id=86
> > > > 674 >
> > > >
> > > >
> > > > I am running on an HPC and the run duration for one fcst and
> > > > observation input (radar corrected reflectivity) was appx 20
mins.
> > > > This is taking significantly longer than a run I did last OCT.
I
> > > > have attached my config file ("180619B") and the log file for
> > > > reference. I checked the usual suspects for causing long run
times:
> > > > rank_corr_flag = FALSE;
> > > > n_rep = 0;
> > > >
> > > > These are the settings in my config file. Is there anything
else
> > > > which may be causing the runs to take a long time?
> > > >
> > > > For reference I also attached the config file (MET V5.2) for
the
> > > > run I did in OCT where it took appx 20 secs to generate the
output
> > > > for one set of inputs. This is the file which contains the
string
> "170831A"
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > >
> > > >
> > > > Mr John W. Raby, Meteorologist
> > > >
> > > > U.S. Army Research Laboratory
> > > >
> > > > White Sands Missile Range, NM 88002
> > > >
> > > > (575) 678-2004 DSN 258-2004
> > > >
> > > > FAX (575) 678-1230 DSN 258-1230
> > > >
> > > > Email:
> > > > john.w.raby2.civ at mail.mil<Caution-Caution-Caution-
mailto:john.raby
> > > > @us.army.m
> > > > il>
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
------------------------------------------------
Subject: RE: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0 Grid-Stat concern (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Fri Sep 07 16:45:40 2018
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for resolving this issue.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, September 7, 2018 4:37 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0
Grid-Stat concern (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
We've heard this feedback about met-7.0 being very slow in computing
neighborhood methods from a few different users via met-help. So we
decided to post a met-7.0 bugfix for this:
Caution-
https://dtcenter.org/met/users/support/known_issues/METv7.0/index.php
On Tue, Sep 4, 2018 at 4:52 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86674 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for following up. Yes, we are specifying generation of FSS
and
> the line types you list below. I have to depart for the day right
now,
> but I can go into more detail about what we are observing tomorrow.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Tuesday, September 4, 2018 4:12 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0
> Grid-Stat concern (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> I went hunting back through this ticket looking for the config file
> you're using, but don't see it. I'd like check to see if you're
> computing fractions skill score... i.e. are you requesting the
NBRCNT,
> NBRCTC, or NBRCTS line types?
>
> It's come to our attention over the last week that Grid-Stat in
> met-7.0 is much slower in computing the fractional coverage fields
used in Grid-Stat.
> This issue arose when adding support for circular neighborhoods.
The
> switch to using different data structures for computing the
fractional
> coverage led to a huge slowdown.
>
> I wonder if this would explain the behavior you're seeing?
>
> In the development version of the code, I've refactored that
algorithm
> to speed it back up. For comparison, a particular configuration of
> grid-stat in met-6.0 takes 1:30 to complete. In met-7.0, it takes
> 10:30. In the current development version, it takes 1:50. I do
have
> an idea of how we might further speed it up, but it won't get much
faster than 1:45 or so.
>
> Thanks,
> John
>
> On Fri, Aug 31, 2018 at 2:05 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=86
> > 674 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > My ARL colleague in Adelphi, MD compiled MET V7.0 with the bugfilx
> > posted on 07/24/18. He is running MET Grid-Stat on his system and
he
> > is still getting very long run times. For the same input data V5.2
> > Grid-Stat runs in about 26 mins. Based on testing, I estimate that
> > V7.0 Grid-Stat would take between 14 and 16 hours to complete.
> > Huaqing Cai confirmed that the compression level we are using for
> > the NetCDF radar observations is level 4 (not high). The fcst data
is GRIB from UPP.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT
> > [Caution-Caution-mailto:met_help at ucar.edu]
> > Sent: Tuesday, August 21, 2018 4:34 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET
V7.0
> > Grid-Stat concern (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
> > verify the identity of the sender, and confirm the authenticity of
> > all links contained within the message prior to copying and
pasting
> > the address to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > One more thing to check. Are you using a high NetCDF compression
level?
> > A user at NASA reported that it takes an exceptionally long time
for
> > MET to read NetCDF data that has been compressed to a high level.
> >
> > I wonder also if perhaps the version “mismatch” is producing a
> > similar latency.
> >
> > John
> >
> > On Tue, Aug 21, 2018 at 4:29 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url:
> > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > html?id=86
> > > 674 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > John -
> > >
> > > I used MET V7.0 with the bugfix by copying the installation
> > > directory from a colleague and placing it in my home on the HPC
> > > and I still have the issue with Grid-Stat running very slow. One
> > > fcst and observation input takes about 20 mins to generate the
> > > output. I noticed that there is a warning in the ncdump output
of
> > > the attached radar observation file which says the
> > > following:
> > > Warning! ***HDF5 library version mismatched error***
> > >
> > > I had to run MET Gen-Vx-Mask to produce the radar observation
with
> > > no missing values and the version I used for that was MET V7.0.
If
> > > I use MET
> > > V5.2 Gen-Vx-Mask, I don’t see that warning in the ncdump output
> > > and then I can run V5.2 Grid-Stat which runs fine. See the
> > > attached script which I use to run Gen-Vx-Mask.
> > >
> > > Could you assess this situation to see what is the issue and
what
> > > is the best COA to resolve it? Let me know if you need other
files
> > > or
> info.
> > >
> > > Thanks.
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT
> > > [Caution-Caution-Caution-mailto:met_help at ucar.edu]
> > > Sent: Monday, August 20, 2018 4:06 PM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #86674] MET V7.0
> > > Grid-Stat concern
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
> > > pasting the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > Can you please check your compilation of met-7.0 to see if it
> > > includes the bugfix posted on 07/24/18:
> > > Caution-
> > > Caution-Caution-Caution-
https://dtcenter.org/met/users/support/kno
> > > wn_issues/
> > > METv7.0/in
> > > dex.php
> > >
> > > If not, please recompile with that bugfix and retest.
> > >
> > > Thanks,
> > > John Halley Gotway
> > >
> > > On Mon, Aug 20, 2018 at 1:30 PM Raby, John W USA CIV via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Mon Aug 20 13:30:01 2018: Request 86674 was acted upon.
> > > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > > Queue: met_help
> > > > Subject: MET V7.0 Grid-Stat concern
> > > > Owner: Nobody
> > > > Requestors: john.w.raby2.civ at mail.mil
> > > > Status: new
> > > > Ticket <Caution-Caution-Caution-Caution-url:
> > > > Caution-Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > > html?id=86
> > > > 674 >
> > > >
> > > >
> > > > I am running on an HPC and the run duration for one fcst and
> > > > observation input (radar corrected reflectivity) was appx 20
mins.
> > > > This is taking significantly longer than a run I did last OCT.
I
> > > > have attached my config file ("180619B") and the log file for
> > > > reference. I checked the usual suspects for causing long run
times:
> > > > rank_corr_flag = FALSE;
> > > > n_rep = 0;
> > > >
> > > > These are the settings in my config file. Is there anything
else
> > > > which may be causing the runs to take a long time?
> > > >
> > > > For reference I also attached the config file (MET V5.2) for
the
> > > > run I did in OCT where it took appx 20 secs to generate the
> > > > output for one set of inputs. This is the file which contains
> > > > the string
> "170831A"
> > > >
> > > > Thanks.
> > > >
> > > > R/
> > > > John
> > > >
> > > >
> > > > Mr John W. Raby, Meteorologist
> > > >
> > > > U.S. Army Research Laboratory
> > > >
> > > > White Sands Missile Range, NM 88002
> > > >
> > > > (575) 678-2004 DSN 258-2004
> > > >
> > > > FAX (575) 678-1230 DSN 258-1230
> > > >
> > > > Email:
> > > > john.w.raby2.civ at mail.mil<Caution-Caution-Caution-Caution-
mailto
> > > > :john.raby
> > > > @us.army.m
> > > > il>
> > > >
> > > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
More information about the Met_help
mailing list