[Met_help] [rt.rap.ucar.edu #80202] History for question about Grid-Stat (UNCLASSIFIED)
John Halley Gotway via RT
met_help at ucar.edu
Fri May 12 09:15:36 MDT 2017
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
CLASSIFICATION: UNCLASSIFIED
I'm running MET V5.2 Grid-Stat on a high performance computer and
encountered an "out of memory" error. See attached log file generated using
verbosity level 4.
I tried adding the block_size into the config file and setting it to
3,000,000 to accommodate the gridded observations grid I'm using which has
dimensions of 2145X1377 (larger than the fcst grid), but I still have the
same error.
Is there a setting I can change to resolve this error? Any other
suggestions?
Thanks.
R/
John
Mr John W. Raby, Meteorologist
U.S. Army Research Laboratory
White Sands Missile Range, NM 88002
(575) 678-2004 DSN 258-2004
FAX (575) 678-1230 DSN 258-1230
Email: john.w.raby2.civ at mail.mil
CLASSIFICATION: UNCLASSIFIED
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: question about Grid-Stat (UNCLASSIFIED)
From: John Halley Gotway
Time: Tue Apr 18 22:43:04 2017
John,
I see you're having a memory issue with grid_stat in version 5.2.
Unfortunately, the block_size config option doesn't apply to
grid_stat.
Instead it's used by series_analysis to define how many grid points to
process concurrently. Setting it lowers in series_analysis uses less
memory, but it would have no effect in the grid_stat confit file.
If possible, it'd be worth trying this same run using version 6.0 of
grid_stat. We found and fixed some very inefficient memory usage in
grid_stat, as listed in the release notes:
http://www.dtcenter.org/met/users/support/release_notes/METv6.0_release_notes.php
These changes won't necessarily fix your memory issue but will
certainly
make it run faster.
Often when you submit a job to a batch queuing system, there is an
option
to request more memory. I'd suggest looking into that and requesting
more
memory, if possible.
You might also test to see if using the "regrid" option to evaluate on
a
more coarse grid enables it to actually run.
Hope that helps you get started.
Thanks,
John
On Tue, Apr 18, 2017 at 3:55 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Tue Apr 18 15:55:16 2017: Request 80202 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: question about Grid-Stat (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80202 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
> I'm running MET V5.2 Grid-Stat on a high performance computer and
> encountered an "out of memory" error. See attached log file
generated using
> verbosity level 4.
>
> I tried adding the block_size into the config file and setting it to
> 3,000,000 to accommodate the gridded observations grid I'm using
which has
> dimensions of 2145X1377 (larger than the fcst grid), but I still
have the
> same error.
>
> Is there a setting I can change to resolve this error? Any other
> suggestions?
>
> Thanks.
>
> R/
>
> John
>
> Mr John W. Raby, Meteorologist
>
> U.S. Army Research Laboratory
>
> White Sands Missile Range, NM 88002
>
> (575) 678-2004 DSN 258-2004
>
> FAX (575) 678-1230 DSN 258-1230
>
> Email: john.w.raby2.civ at mail.mil
>
>
>
>
>
>
>
>
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
------------------------------------------------
Subject: question about Grid-Stat (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed Apr 19 08:57:03 2017
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for your suggestions. I'll start looking into trying them. Will
installing V6.0 be easier because I have V5.2 installed or is it an
independent set of libraries and code which requires the same process
I went
through for V5.2?
Is the large size of the observation grid (2145X1377) alone causing
the
problem or is it the combination of the fcst grid (278X278) and the
observation grids together that result in the exceedance of the
available
memory size?
Is there a way to relate available memory size to the grid sizes I
need to
process?
I ran a command called "free -g" yesterday to see if this would show
the
available memory and I received the following:
Free - 13GB
Swap - 61GB
The HPC User's says that you shouldn't use more than 8 GB at a time,
so even
though 13 is "free", they may be restricting me to 8. Is this as a
potential
limitation?
In my config file (attached) I did use the "regrid" option to set the
verification grid to "FCST" so that the verification would occur on
that grid
(9 km spacing). How does the grid matching occur? Will the OBS grid
(2.5 km
spacing) be interpolated to the FCST grid (9 km spacing)?
Maybe I have used other config file settings improperly somewhere
which might
cause a problem. I have n_rep set to 1000 which was cited in the past
as a
process which slows things down.
If I don't want to use smoothing prior to verification, can I use some
null
settings to prevent it from running while leaving that section in the
config
file for future reference?
I had not envisioned having to run MET using a batch queuing system,
and I'm
not very familiar with it, but I know that my colleagues here use it
for
running the WRF model, so I should be able to tap that experience to
explore
the possibility that I can allocate more memory for running Grid-Stat.
The
compute nodes can only be used through batch job submission. Good
suggestion.
I thought of the possibility of using the options in "wgrib2" to
window out a
subset of the URMA gridded observations to pare down the grid size
some. I've
never done this. Have you tried this and would this cut down on the
memory
resources needed to process this data?
Can I anticipate this same problem if I use the same input fields into
MODE
and Series-Analysis?
Thanks.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, April 18, 2017 10:43 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #80202] question about
Grid-Stat (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
John,
I see you're having a memory issue with grid_stat in version 5.2.
Unfortunately, the block_size config option doesn't apply to
grid_stat.
Instead it's used by series_analysis to define how many grid points to
process
concurrently. Setting it lowers in series_analysis uses less memory,
but it
would have no effect in the grid_stat confit file.
If possible, it'd be worth trying this same run using version 6.0 of
grid_stat. We found and fixed some very inefficient memory usage in
grid_stat, as listed in the release notes:
Caution-
http://www.dtcenter.org/met/users/support/release_notes/METv6.0_release_notes.php
These changes won't necessarily fix your memory issue but will
certainly make
it run faster.
Often when you submit a job to a batch queuing system, there is an
option to
request more memory. I'd suggest looking into that and requesting
more
memory, if possible.
You might also test to see if using the "regrid" option to evaluate on
a more
coarse grid enables it to actually run.
Hope that helps you get started.
Thanks,
John
On Tue, Apr 18, 2017 at 3:55 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> Tue Apr 18 15:55:16 2017: Request 80202 was acted upon.
> Transaction: Ticket created by john.w.raby2.civ at mail.mil
> Queue: met_help
> Subject: question about Grid-Stat (UNCLASSIFIED)
> Owner: Nobody
> Requestors: john.w.raby2.civ at mail.mil
> Status: new
> Ticket <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80202 >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
> I'm running MET V5.2 Grid-Stat on a high performance computer and
> encountered an "out of memory" error. See attached log file
generated
> using verbosity level 4.
>
> I tried adding the block_size into the config file and setting it to
> 3,000,000 to accommodate the gridded observations grid I'm using
which
> has dimensions of 2145X1377 (larger than the fcst grid), but I still
> have the same error.
>
> Is there a setting I can change to resolve this error? Any other
> suggestions?
>
> Thanks.
>
> R/
>
> John
>
> Mr John W. Raby, Meteorologist
>
> U.S. Army Research Laboratory
>
> White Sands Missile Range, NM 88002
>
> (575) 678-2004 DSN 258-2004
>
> FAX (575) 678-1230 DSN 258-1230
>
> Email: john.w.raby2.civ at mail.mil
>
>
>
>
>
>
>
>
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: question about Grid-Stat (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu Apr 20 16:15:30 2017
John,
Just curious if you've made any progress resolving these memory
issues?
Thanks,
John
On Wed, Apr 19, 2017 at 8:57 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80202 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for your suggestions. I'll start looking into trying them.
Will
> installing V6.0 be easier because I have V5.2 installed or is it an
> independent set of libraries and code which requires the same
process I
> went
> through for V5.2?
>
> Is the large size of the observation grid (2145X1377) alone causing
the
> problem or is it the combination of the fcst grid (278X278) and the
> observation grids together that result in the exceedance of the
available
> memory size?
>
> Is there a way to relate available memory size to the grid sizes I
need to
> process?
>
> I ran a command called "free -g" yesterday to see if this would show
the
> available memory and I received the following:
>
> Free - 13GB
> Swap - 61GB
>
> The HPC User's says that you shouldn't use more than 8 GB at a time,
so
> even
> though 13 is "free", they may be restricting me to 8. Is this as a
> potential
> limitation?
>
> In my config file (attached) I did use the "regrid" option to set
the
> verification grid to "FCST" so that the verification would occur on
that
> grid
> (9 km spacing). How does the grid matching occur? Will the OBS grid
(2.5 km
> spacing) be interpolated to the FCST grid (9 km spacing)?
>
> Maybe I have used other config file settings improperly somewhere
which
> might
> cause a problem. I have n_rep set to 1000 which was cited in the
past as a
> process which slows things down.
>
> If I don't want to use smoothing prior to verification, can I use
some null
> settings to prevent it from running while leaving that section in
the
> config
> file for future reference?
>
> I had not envisioned having to run MET using a batch queuing system,
and
> I'm
> not very familiar with it, but I know that my colleagues here use it
for
> running the WRF model, so I should be able to tap that experience to
> explore
> the possibility that I can allocate more memory for running Grid-
Stat. The
> compute nodes can only be used through batch job submission. Good
> suggestion.
>
> I thought of the possibility of using the options in "wgrib2" to
window
> out a
> subset of the URMA gridded observations to pare down the grid size
some.
> I've
> never done this. Have you tried this and would this cut down on the
memory
> resources needed to process this data?
>
> Can I anticipate this same problem if I use the same input fields
into MODE
> and Series-Analysis?
>
> Thanks.
>
> R/
> John
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, April 18, 2017 10:43 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #80202] question
about
> Grid-Stat (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> I see you're having a memory issue with grid_stat in version 5.2.
> Unfortunately, the block_size config option doesn't apply to
grid_stat.
> Instead it's used by series_analysis to define how many grid points
to
> process
> concurrently. Setting it lowers in series_analysis uses less
memory, but
> it
> would have no effect in the grid_stat confit file.
>
> If possible, it'd be worth trying this same run using version 6.0 of
> grid_stat. We found and fixed some very inefficient memory usage in
> grid_stat, as listed in the release notes:
>
> Caution-http://www.dtcenter.org/met/users/support/release_
> notes/METv6.0_release_notes.php
>
> These changes won't necessarily fix your memory issue but will
certainly
> make
> it run faster.
>
> Often when you submit a job to a batch queuing system, there is an
option
> to
> request more memory. I'd suggest looking into that and requesting
more
> memory, if possible.
>
> You might also test to see if using the "regrid" option to evaluate
on a
> more
> coarse grid enables it to actually run.
>
> Hope that helps you get started.
>
> Thanks,
> John
>
>
> On Tue, Apr 18, 2017 at 3:55 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Apr 18 15:55:16 2017: Request 80202 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: question about Grid-Stat (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80202 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> > I'm running MET V5.2 Grid-Stat on a high performance computer and
> > encountered an "out of memory" error. See attached log file
generated
> > using verbosity level 4.
> >
> > I tried adding the block_size into the config file and setting it
to
> > 3,000,000 to accommodate the gridded observations grid I'm using
which
> > has dimensions of 2145X1377 (larger than the fcst grid), but I
still
> > have the same error.
> >
> > Is there a setting I can change to resolve this error? Any other
> > suggestions?
> >
> > Thanks.
> >
> > R/
> >
> > John
> >
> > Mr John W. Raby, Meteorologist
> >
> > U.S. Army Research Laboratory
> >
> > White Sands Missile Range, NM 88002
> >
> > (575) 678-2004 DSN 258-2004
> >
> > FAX (575) 678-1230 DSN 258-1230
> >
> > Email: john.w.raby2.civ at mail.mil
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: question about Grid-Stat (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu Apr 20 16:32:17 2017
CLASSIFICATION: UNCLASSIFIED
John -
I haven't been able to get back to that. I have begun looking into the
Portable Batch system (PBS) which is used on the HPC and I'm told that
you can
access a much larger memory on the compute nodes than where I was
running
Grid-Stat before. The login node has 2GB available while the compute
node has
128 GB. So, I'm going to try that approach which you suggested.
Would running regrid as a utility to pare down the size of the larger
grid
prior to running Grid-Stat be problematic in the same way? (i.e.
likely to run
out of memory)
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, April 20, 2017 4:16 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80202] question
about
Grid-Stat (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
John,
Just curious if you've made any progress resolving these memory
issues?
Thanks,
John
On Wed, Apr 19, 2017 at 8:57 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80202 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for your suggestions. I'll start looking into trying them.
Will
> installing V6.0 be easier because I have V5.2 installed or is it an
> independent set of libraries and code which requires the same
process
> I went through for V5.2?
>
> Is the large size of the observation grid (2145X1377) alone causing
> the problem or is it the combination of the fcst grid (278X278) and
> the observation grids together that result in the exceedance of the
> available memory size?
>
> Is there a way to relate available memory size to the grid sizes I
> need to process?
>
> I ran a command called "free -g" yesterday to see if this would show
> the available memory and I received the following:
>
> Free - 13GB
> Swap - 61GB
>
> The HPC User's says that you shouldn't use more than 8 GB at a time,
> so even though 13 is "free", they may be restricting me to 8. Is
this
> as a potential limitation?
>
> In my config file (attached) I did use the "regrid" option to set
the
> verification grid to "FCST" so that the verification would occur on
> that grid
> (9 km spacing). How does the grid matching occur? Will the OBS grid
> (2.5 km
> spacing) be interpolated to the FCST grid (9 km spacing)?
>
> Maybe I have used other config file settings improperly somewhere
> which might cause a problem. I have n_rep set to 1000 which was
cited
> in the past as a process which slows things down.
>
> If I don't want to use smoothing prior to verification, can I use
some
> null settings to prevent it from running while leaving that section
in
> the config file for future reference?
>
> I had not envisioned having to run MET using a batch queuing system,
> and I'm not very familiar with it, but I know that my colleagues
here
> use it for running the WRF model, so I should be able to tap that
> experience to explore the possibility that I can allocate more
memory
> for running Grid-Stat. The compute nodes can only be used through
> batch job submission. Good suggestion.
>
> I thought of the possibility of using the options in "wgrib2" to
> window out a subset of the URMA gridded observations to pare down
the
> grid size some.
> I've
> never done this. Have you tried this and would this cut down on the
> memory resources needed to process this data?
>
> Can I anticipate this same problem if I use the same input fields
into
> MODE and Series-Analysis?
>
> Thanks.
>
> R/
> John
>
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Tuesday, April 18, 2017 10:43 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #80202] question
about
> Grid-Stat (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> I see you're having a memory issue with grid_stat in version 5.2.
> Unfortunately, the block_size config option doesn't apply to
grid_stat.
> Instead it's used by series_analysis to define how many grid points
to
> process concurrently. Setting it lowers in series_analysis uses
less
> memory, but it would have no effect in the grid_stat confit file.
>
> If possible, it'd be worth trying this same run using version 6.0 of
> grid_stat. We found and fixed some very inefficient memory usage in
> grid_stat, as listed in the release notes:
>
> Caution-Caution-http://www.dtcenter.org/met/users/support/release_
> notes/METv6.0_release_notes.php
>
> These changes won't necessarily fix your memory issue but will
> certainly make it run faster.
>
> Often when you submit a job to a batch queuing system, there is an
> option to request more memory. I'd suggest looking into that and
> requesting more memory, if possible.
>
> You might also test to see if using the "regrid" option to evaluate
on
> a more coarse grid enables it to actually run.
>
> Hope that helps you get started.
>
> Thanks,
> John
>
>
> On Tue, Apr 18, 2017 at 3:55 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Apr 18 15:55:16 2017: Request 80202 was acted upon.
> > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > Queue: met_help
> > Subject: question about Grid-Stat (UNCLASSIFIED)
> > Owner: Nobody
> > Requestors: john.w.raby2.civ at mail.mil
> > Status: new
> > Ticket <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80
> > 202 >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> > I'm running MET V5.2 Grid-Stat on a high performance computer and
> > encountered an "out of memory" error. See attached log file
> > generated using verbosity level 4.
> >
> > I tried adding the block_size into the config file and setting it
to
> > 3,000,000 to accommodate the gridded observations grid I'm using
> > which has dimensions of 2145X1377 (larger than the fcst grid), but
I
> > still have the same error.
> >
> > Is there a setting I can change to resolve this error? Any other
> > suggestions?
> >
> > Thanks.
> >
> > R/
> >
> > John
> >
> > Mr John W. Raby, Meteorologist
> >
> > U.S. Army Research Laboratory
> >
> > White Sands Missile Range, NM 88002
> >
> > (575) 678-2004 DSN 258-2004
> >
> > FAX (575) 678-1230 DSN 258-1230
> >
> > Email: john.w.raby2.civ at mail.mil
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
> >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: question about Grid-Stat (UNCLASSIFIED)
From: John Halley Gotway
Time: Fri Apr 21 09:58:19 2017
John,
Unfortunately, I'm not sure if you'll run into more memory issues with
regrid_data_plane. I'd just suggest testing it out.
John
On Thu, Apr 20, 2017 at 4:32 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80202 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> I haven't been able to get back to that. I have begun looking into
the
> Portable Batch system (PBS) which is used on the HPC and I'm told
that you
> can
> access a much larger memory on the compute nodes than where I was
running
> Grid-Stat before. The login node has 2GB available while the compute
node
> has
> 128 GB. So, I'm going to try that approach which you suggested.
>
> Would running regrid as a utility to pare down the size of the
larger grid
> prior to running Grid-Stat be problematic in the same way? (i.e.
likely to
> run
> out of memory)
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, April 20, 2017 4:16 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] Re: [rt.rap.ucar.edu #80202] question
about
> Grid-Stat (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Just curious if you've made any progress resolving these memory
issues?
>
> Thanks,
> John
>
> On Wed, Apr 19, 2017 at 8:57 AM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80202 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > Thanks for your suggestions. I'll start looking into trying them.
Will
> > installing V6.0 be easier because I have V5.2 installed or is it
an
> > independent set of libraries and code which requires the same
process
> > I went through for V5.2?
> >
> > Is the large size of the observation grid (2145X1377) alone
causing
> > the problem or is it the combination of the fcst grid (278X278)
and
> > the observation grids together that result in the exceedance of
the
> > available memory size?
> >
> > Is there a way to relate available memory size to the grid sizes I
> > need to process?
> >
> > I ran a command called "free -g" yesterday to see if this would
show
> > the available memory and I received the following:
> >
> > Free - 13GB
> > Swap - 61GB
> >
> > The HPC User's says that you shouldn't use more than 8 GB at a
time,
> > so even though 13 is "free", they may be restricting me to 8. Is
this
> > as a potential limitation?
> >
> > In my config file (attached) I did use the "regrid" option to set
the
> > verification grid to "FCST" so that the verification would occur
on
> > that grid
> > (9 km spacing). How does the grid matching occur? Will the OBS
grid
> > (2.5 km
> > spacing) be interpolated to the FCST grid (9 km spacing)?
> >
> > Maybe I have used other config file settings improperly somewhere
> > which might cause a problem. I have n_rep set to 1000 which was
cited
> > in the past as a process which slows things down.
> >
> > If I don't want to use smoothing prior to verification, can I use
some
> > null settings to prevent it from running while leaving that
section in
> > the config file for future reference?
> >
> > I had not envisioned having to run MET using a batch queuing
system,
> > and I'm not very familiar with it, but I know that my colleagues
here
> > use it for running the WRF model, so I should be able to tap that
> > experience to explore the possibility that I can allocate more
memory
> > for running Grid-Stat. The compute nodes can only be used through
> > batch job submission. Good suggestion.
> >
> > I thought of the possibility of using the options in "wgrib2" to
> > window out a subset of the URMA gridded observations to pare down
the
> > grid size some.
> > I've
> > never done this. Have you tried this and would this cut down on
the
> > memory resources needed to process this data?
> >
> > Can I anticipate this same problem if I use the same input fields
into
> > MODE and Series-Analysis?
> >
> > Thanks.
> >
> > R/
> > John
> >
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> > Sent: Tuesday, April 18, 2017 10:43 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: [Non-DoD Source] Re: [rt.rap.ucar.edu #80202] question
about
> > Grid-Stat (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > I see you're having a memory issue with grid_stat in version 5.2.
> > Unfortunately, the block_size config option doesn't apply to
grid_stat.
> > Instead it's used by series_analysis to define how many grid
points to
> > process concurrently. Setting it lowers in series_analysis uses
less
> > memory, but it would have no effect in the grid_stat confit file.
> >
> > If possible, it'd be worth trying this same run using version 6.0
of
> > grid_stat. We found and fixed some very inefficient memory usage
in
> > grid_stat, as listed in the release notes:
> >
> > Caution-Caution-http://www.dtcenter.org/met/users/support/release_
> > notes/METv6.0_release_notes.php
> >
> > These changes won't necessarily fix your memory issue but will
> > certainly make it run faster.
> >
> > Often when you submit a job to a batch queuing system, there is an
> > option to request more memory. I'd suggest looking into that and
> > requesting more memory, if possible.
> >
> > You might also test to see if using the "regrid" option to
evaluate on
> > a more coarse grid enables it to actually run.
> >
> > Hope that helps you get started.
> >
> > Thanks,
> > John
> >
> >
> > On Tue, Apr 18, 2017 at 3:55 PM Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Apr 18 15:55:16 2017: Request 80202 was acted upon.
> > > Transaction: Ticket created by john.w.raby2.civ at mail.mil
> > > Queue: met_help
> > > Subject: question about Grid-Stat (UNCLASSIFIED)
> > > Owner: Nobody
> > > Requestors: john.w.raby2.civ at mail.mil
> > > Status: new
> > > Ticket <Caution-Caution-url:
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80
> > > 202 >
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > > I'm running MET V5.2 Grid-Stat on a high performance computer
and
> > > encountered an "out of memory" error. See attached log file
> > > generated using verbosity level 4.
> > >
> > > I tried adding the block_size into the config file and setting
it to
> > > 3,000,000 to accommodate the gridded observations grid I'm using
> > > which has dimensions of 2145X1377 (larger than the fcst grid),
but I
> > > still have the same error.
> > >
> > > Is there a setting I can change to resolve this error? Any other
> > > suggestions?
> > >
> > > Thanks.
> > >
> > > R/
> > >
> > > John
> > >
> > > Mr John W. Raby, Meteorologist
> > >
> > > U.S. Army Research Laboratory
> > >
> > > White Sands Missile Range, NM 88002
> > >
> > > (575) 678-2004 DSN 258-2004
> > >
> > > FAX (575) 678-1230 DSN 258-1230
> > >
> > > Email: john.w.raby2.civ at mail.mil
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> > >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: question about Grid-Stat (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu May 04 15:54:40 2017
CLASSIFICATION: UNCLASSIFIED
Follow-up on this ticket.
I just ran MET Grid-Stat (single call) on the HPC using a batch
request which
failed due to "out of memory". My request specified use of the
following
resources:
32 cores on 1 node with 128 GB of user-accessible shared memory.
I am investigating ways to allocate more memory using the online
User's Guide
for the Portable Batch System (PBS). For a serial process, I think my
system
can only send to one compute node.
Based on the MET V5.2 User's Guide I assume that MET is compiled as a
serial
application and is designed to run on a single processor.
Any suggestions on what I can try given the new info above?
I plan to also try regrid_data_plane to see if I can reduce the size
of the
observations grid.
Thanks.
R/
John
Mr John W. Raby, Meteorologist
U.S. Army Research Laboratory
White Sands Missile Range, NM 88002
(575) 678-2004 DSN 258-2004
FAX (575) 678-1230 DSN 258-1230
Email: john.w.raby2.civ at mail.mil
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, May 01, 2017 1:53 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved: question
about
Grid-Stat (UNCLASSIFIED)
According to our records, your request has been resolved. If you have
any
further questions or concerns, please respond to this message.
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: question about Grid-Stat (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu May 04 16:49:32 2017
CLASSIFICATION: UNCLASSIFIED
More follow-up on this issue:
Attempted to run regrid_data_plane (MET V5.2) to regrid the URMA
observations
file to the lower resolution grid of the forecast file (2.5 km to 9km)
and
from URMA domain over entire CONUS to the forecast grid over SW US
only by
using "to_grid" to provide the path to the forecast grid and the same
error
occurred. (out of memory). This was executed from my home on the HPC
on the
login node and not submitted as a batch job. Next step will be to
repeat this
test on the compute node using a batch request.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Monday, May 01, 2017 1:53 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved: question
about
Grid-Stat (UNCLASSIFIED)
According to our records, your request has been resolved. If you have
any
further questions or concerns, please respond to this message.
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: question about Grid-Stat (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu May 04 17:27:17 2017
John,
We had another user at NCEP report similar behavior when processing
GRIB2
data on an HPC. The problem was fixed by reconfiguring/recompiling
MET
using the -D__64BIT__ flag.
Please read the note about that on this page:
http://www.dtcenter.org/met/users/downloads/index.php
Another user with this issue received a segfault when trying to run
MET on
GRIB2 data.
Please let me know if that resolves the issue and we'll try to update
the
compilation instructions to make them more clear.
Thanks
John
On Thu, May 4, 2017 at 4:49 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80202 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> More follow-up on this issue:
>
> Attempted to run regrid_data_plane (MET V5.2) to regrid the URMA
> observations
> file to the lower resolution grid of the forecast file (2.5 km to
9km) and
> from URMA domain over entire CONUS to the forecast grid over SW US
only by
> using "to_grid" to provide the path to the forecast grid and the
same error
> occurred. (out of memory). This was executed from my home on the HPC
on the
> login node and not submitted as a batch job. Next step will be to
repeat
> this
> test on the compute node using a batch request.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Monday, May 01, 2017 1:53 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
question
> about
> Grid-Stat (UNCLASSIFIED)
>
> According to our records, your request has been resolved. If you
have any
> further questions or concerns, please respond to this message.
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: question about Grid-Stat (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu May 04 19:21:37 2017
.EmailQuote { PADDING-LEFT: 4pt; MARGIN-LEFT: 1pt; BORDER-LEFT:
#800000 2px solid } P { MARGIN-BOTTOM: 0px; MARGIN-TOP: 0px }
John -
Thanks for researching similar issues. I compiled MET on the HPC using
-D__64BIT__ flag.
R/
John
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Thursday, May 04, 2017 5:27 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
question about Grid-Stat (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
We had another user at NCEP report similar behavior when processing
GRIB2
data on an HPC. The problem was fixed by reconfiguring/recompiling
MET
using the -D__64BIT__ flag.
Please read the note about that on this page:
Caution-http://www.dtcenter.org/met/users/downloads/index.php
Another user with this issue received a segfault when trying to run
MET on
GRIB2 data.
Please let me know if that resolves the issue and we'll try to update
the
compilation instructions to make them more clear.
Thanks
John
On Thu, May 4, 2017 at 4:49 PM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80202 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> More follow-up on this issue:
>
> Attempted to run regrid_data_plane (MET V5.2) to regrid the URMA
> observations
> file to the lower resolution grid of the forecast file (2.5 km to
9km) and
> from URMA domain over entire CONUS to the forecast grid over SW US
only by
> using "to_grid" to provide the path to the forecast grid and the
same error
> occurred. (out of memory). This was executed from my home on the HPC
on the
> login node and not submitted as a batch job. Next step will be to
repeat
> this
> test on the compute node using a batch request.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Monday, May 01, 2017 1:53 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
question
> about
> Grid-Stat (UNCLASSIFIED)
>
> According to our records, your request has been resolved. If you
have any
> further questions or concerns, please respond to this message.
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: question about Grid-Stat (UNCLASSIFIED)
From: John Halley Gotway
Time: Fri May 05 12:52:17 2017
John,
So you're saying that you configured MET using -D__64BIT__ and the
your
version of the grib2c library uses that flag as well? Yet, you're
still
getting the out of memory error.
Here's what I'd recommend:
(1) There is still the possibility that this is related to
-D__64BIT__. To
rule that out, you could send to me:
- The "makefile" from the top-level directory of the GRIB2C library.
- From the top-level MET directory, the "config.log" file as well as
the
output of make, assuming you wrote it to a log file, like
"make_install.log".
I can check those files to make sure that -D__64BIT__ is used
consistently.
(2) It could really be a memory issue. You mentioned that you ran it
on a
login node, not one of the compute nodes. Often when you submit a
batch
job, you can request the amount of memory required for your job by
passing
a command line option. But those options differ for different queuing
systems. You might ask an administrator what the default amount of
memory
is, and how to request more.
Thanks,
John
On Thu, May 4, 2017 at 7:21 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80202 >
>
> .EmailQuote { PADDING-LEFT: 4pt; MARGIN-LEFT: 1pt; BORDER-LEFT:
#800000
> 2px solid } P { MARGIN-BOTTOM: 0px; MARGIN-TOP: 0px }
>
> John -
>
> Thanks for researching similar issues. I compiled MET on the HPC
using
> -D__64BIT__ flag.
>
> R/
>
> John
>
>
>
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, May 04, 2017 5:27 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
question
> about Grid-Stat (UNCLASSIFIED)
>
>
>
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> We had another user at NCEP report similar behavior when processing
GRIB2
> data on an HPC. The problem was fixed by reconfiguring/recompiling
MET
> using the -D__64BIT__ flag.
>
> Please read the note about that on this page:
> Caution-http://www.dtcenter.org/met/users/downloads/index.php
>
> Another user with this issue received a segfault when trying to run
MET on
> GRIB2 data.
>
> Please let me know if that resolves the issue and we'll try to
update the
> compilation instructions to make them more clear.
>
> Thanks
> John
>
> On Thu, May 4, 2017 at 4:49 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url: Caution-https://rt.rap.ucar.ed
> u/rt/Ticket/Display.html?id=80202 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > More follow-up on this issue:
> >
> > Attempted to run regrid_data_plane (MET V5.2) to regrid the URMA
> > observations
> > file to the lower resolution grid of the forecast file (2.5 km to
9km)
> and
> > from URMA domain over entire CONUS to the forecast grid over SW US
only
> by
> > using "to_grid" to provide the path to the forecast grid and the
same
> error
> > occurred. (out of memory). This was executed from my home on the
HPC on
> the
> > login node and not submitted as a batch job. Next step will be to
repeat
> > this
> > test on the compute node using a batch request.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> > Sent: Monday, May 01, 2017 1:53 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> > Subject: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
question
> > about
> > Grid-Stat (UNCLASSIFIED)
> >
> > According to our records, your request has been resolved. If you
have any
> > further questions or concerns, please respond to this message.
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
------------------------------------------------
Subject: question about Grid-Stat (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Fri May 05 18:50:58 2017
.EmailQuote { PADDING-LEFT: 4pt; MARGIN-LEFT: 1pt; BORDER-LEFT:
#800000 2px solid } P { MARGIN-BOTTOM: 0px; MARGIN-TOP: 0px }
John -
Thanks for your offer to investigate this further. I'll work on
gathering the files you mentioned and send them to you. I'm pretty
sure I compiled GRIB2C library with the 64 bit flag as well. Grid-Stat
was run on a login node and on a compute node (as a batch job using
PBS) and both runs has the same problem. I ran plot_data_plane on a
login node only so far with the same error. I haven't had a chance to
run it on a compute node.
R/
John
From: John Halley Gotway via RT [met_help at ucar.edu]
Sent: Friday, May 05, 2017 12:52 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
question about Grid-Stat (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the identity of the sender, and confirm the authenticity of all links
contained within the message prior to copying and pasting the address
to a Web browser.
----
John,
So you're saying that you configured MET using -D__64BIT__ and the
your
version of the grib2c library uses that flag as well? Yet, you're
still
getting the out of memory error.
Here's what I'd recommend:
(1) There is still the possibility that this is related to
-D__64BIT__. To
rule that out, you could send to me:
- The "makefile" from the top-level directory of the GRIB2C library.
- From the top-level MET directory, the "config.log" file as well as
the
output of make, assuming you wrote it to a log file, like
"make_install.log".
I can check those files to make sure that -D__64BIT__ is used
consistently.
(2) It could really be a memory issue. You mentioned that you ran it
on a
login node, not one of the compute nodes. Often when you submit a
batch
job, you can request the amount of memory required for your job by
passing
a command line option. But those options differ for different queuing
systems. You might ask an administrator what the default amount of
memory
is, and how to request more.
Thanks,
John
On Thu, May 4, 2017 at 7:21 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url: Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80202 >
>
> .EmailQuote { PADDING-LEFT: 4pt; MARGIN-LEFT: 1pt; BORDER-LEFT:
#800000
> 2px solid } P { MARGIN-BOTTOM: 0px; MARGIN-TOP: 0px }
>
> John -
>
> Thanks for researching similar issues. I compiled MET on the HPC
using
> -D__64BIT__ flag.
>
> R/
>
> John
>
>
>
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, May 04, 2017 5:27 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
question
> about Grid-Stat (UNCLASSIFIED)
>
>
>
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> We had another user at NCEP report similar behavior when processing
GRIB2
> data on an HPC. The problem was fixed by reconfiguring/recompiling
MET
> using the -D__64BIT__ flag.
>
> Please read the note about that on this page:
> Caution-Caution-
http://www.dtcenter.org/met/users/downloads/index.php
>
> Another user with this issue received a segfault when trying to run
MET on
> GRIB2 data.
>
> Please let me know if that resolves the issue and we'll try to
update the
> compilation instructions to make them more clear.
>
> Thanks
> John
>
> On Thu, May 4, 2017 at 4:49 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url: Caution-Caution-https://rt.rap.ucar.ed
> u/rt/Ticket/Display.html?id=80202 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > More follow-up on this issue:
> >
> > Attempted to run regrid_data_plane (MET V5.2) to regrid the URMA
> > observations
> > file to the lower resolution grid of the forecast file (2.5 km to
9km)
> and
> > from URMA domain over entire CONUS to the forecast grid over SW US
only
> by
> > using "to_grid" to provide the path to the forecast grid and the
same
> error
> > occurred. (out of memory). This was executed from my home on the
HPC on
> the
> > login node and not submitted as a batch job. Next step will be to
repeat
> > this
> > test on the compute node using a batch request.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-Caution-
mailto:met_help at ucar.edu]
> > Sent: Monday, May 01, 2017 1:53 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> > Subject: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
question
> > about
> > Grid-Stat (UNCLASSIFIED)
> >
> > According to our records, your request has been resolved. If you
have any
> > further questions or concerns, please respond to this message.
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
------------------------------------------------
Subject: question about Grid-Stat (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Sat May 06 20:40:43 2017
John -
The files you asked for are attached. The make_install file was the
last one
which was generated with the successful installation of all possible
MET
tools.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Friday, May 05, 2017 12:52 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
question
about Grid-Stat (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
John,
So you're saying that you configured MET using -D__64BIT__ and the
your
version of the grib2c library uses that flag as well? Yet, you're
still
getting the out of memory error.
Here's what I'd recommend:
(1) There is still the possibility that this is related to
-D__64BIT__. To
rule that out, you could send to me:
- The "makefile" from the top-level directory of the GRIB2C library.
- From the top-level MET directory, the "config.log" file as well as
the
output of make, assuming you wrote it to a log file, like
"make_install.log".
I can check those files to make sure that -D__64BIT__ is used
consistently.
(2) It could really be a memory issue. You mentioned that you ran it
on a
login node, not one of the compute nodes. Often when you submit a
batch job,
you can request the amount of memory required for your job by passing
a
command line option. But those options differ for different queuing
systems.
You might ask an administrator what the default amount of memory is,
and how
to request more.
Thanks,
John
On Thu, May 4, 2017 at 7:21 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80202 >
>
> .EmailQuote { PADDING-LEFT: 4pt; MARGIN-LEFT: 1pt; BORDER-LEFT:
> #800000 2px solid } P { MARGIN-BOTTOM: 0px; MARGIN-TOP: 0px }
>
> John -
>
> Thanks for researching similar issues. I compiled MET on the HPC
using
> -D__64BIT__ flag.
>
> R/
>
> John
>
>
>
> From: John Halley Gotway via RT [met_help at ucar.edu]
> Sent: Thursday, May 04, 2017 5:27 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
> question about Grid-Stat (UNCLASSIFIED)
>
>
>
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> We had another user at NCEP report similar behavior when processing
> GRIB2 data on an HPC. The problem was fixed by
> reconfiguring/recompiling MET using the -D__64BIT__ flag.
>
> Please read the note about that on this page:
> Caution-Caution-
http://www.dtcenter.org/met/users/downloads/index.php
>
> Another user with this issue received a segfault when trying to run
> MET on
> GRIB2 data.
>
> Please let me know if that resolves the issue and we'll try to
update
> the compilation instructions to make them more clear.
>
> Thanks
> John
>
> On Thu, May 4, 2017 at 4:49 PM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url: Caution-Caution-https://rt.rap.ucar.ed
> u/rt/Ticket/Display.html?id=80202 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > More follow-up on this issue:
> >
> > Attempted to run regrid_data_plane (MET V5.2) to regrid the URMA
> > observations file to the lower resolution grid of the forecast
file
> > (2.5 km to 9km)
> and
> > from URMA domain over entire CONUS to the forecast grid over SW US
> > only
> by
> > using "to_grid" to provide the path to the forecast grid and the
> > same
> error
> > occurred. (out of memory). This was executed from my home on the
HPC
> > on
> the
> > login node and not submitted as a batch job. Next step will be to
> > repeat this test on the compute node using a batch request.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT
> > [Caution-Caution-mailto:met_help at ucar.edu]
> > Sent: Monday, May 01, 2017 1:53 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
> > question about Grid-Stat (UNCLASSIFIED)
> >
> > According to our records, your request has been resolved. If you
> > have any further questions or concerns, please respond to this
message.
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
------------------------------------------------
Subject: question about Grid-Stat (UNCLASSIFIED)
From: John Halley Gotway
Time: Wed May 10 12:00:57 2017
John,
Thanks for sending those files. I looked through them and think I've
found
the problem.
Take a look at this page of the MET online tutorial about the
discussion of
the -D__64BIT__ flag:
http://www.dtcenter.org/met/users/support/online_tutorial/METv6.0/tutorial.php?name=compilation&category=req_libs
This setting must be used consistently for the GRIB2C library and
MET...
either define it in neither or both. You've chosen to do it in both
and
should have configured MET using "CXXFLAGS=-D__64BIT__".
However, looking in config.log, I see that you've set CFLAGS instead
of
CXXFLAGS:
CFLAGS='-D__64BIT__'
CXXFLAGS='-g'
Please try reconfiguring/recompiling met-6.0, being sure to set
CXXFLAGS.
And then see if your jobs run through batch without the out of memory
error.
Please let me know how it goes.
Thanks,
John
------------------------------------------------
Subject: question about Grid-Stat (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Wed May 10 15:41:51 2017
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for diagnosing the problem. At the time, I was compiling MET
V5.2 and
intended to set CFLAGS in both since my system is 64-bit. The tutorial
showed
how to set the CFLAGS. I didn't see anything about CXXFLAGS. See the
page from
that tutorial:
http://www.dtcenter.org/met/users/support/online_tutorial/METv5.2/tutorial.php?name=compilation&category=req_libs
Is there any way I can resolve the issue without having to migrate to
MET
V6.0? I'm trying to evaluate the cost of performing the upgrade at a
time when
I have a lot of other things I'm working on. Obviously, if there is no
solution which will enable me to continue with MET V5.2, then I'll
have to
migrate.
Having installed V5.2, is there any savings in migrating to V6.0 such
as
re-use of the required libraries which might speed up the process or
is it
best to uninstall (delete everything, including the required
libraries, I
generated for V5.2) and start over using the new tutorial and User's
Guide as
references?
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, May 10, 2017 12:01 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
question
about Grid-Stat (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
John,
Thanks for sending those files. I looked through them and think I've
found
the problem.
Take a look at this page of the MET online tutorial about the
discussion of
the -D__64BIT__ flag:
Caution-
http://www.dtcenter.org/met/users/support/online_tutorial/METv6.0/tutorial.php?name=compilation&category=req_libs
This setting must be used consistently for the GRIB2C library and
MET...
either define it in neither or both. You've chosen to do it in both
and
should have configured MET using "CXXFLAGS=-D__64BIT__".
However, looking in config.log, I see that you've set CFLAGS instead
of
CXXFLAGS:
CFLAGS='-D__64BIT__'
CXXFLAGS='-g'
Please try reconfiguring/recompiling met-6.0, being sure to set
CXXFLAGS.
And then see if your jobs run through batch without the out of memory
error.
Please let me know how it goes.
Thanks,
John
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: question about Grid-Stat (UNCLASSIFIED)
From: John Halley Gotway
Time: Wed May 10 16:58:29 2017
John,
Sorry for the bad instructions in the 5.2 tutorial. That should
probably
have said CXXFLAGS instead of CFLAGS. We'll get that updated.
You could try recompiling MET 5.2 setting CXXFLAGS that way.
Or you could remove -D__64BIT__ from both MET and the grib2c library.
Either method should work from my experience.
Unfortunately moving form 5.2 to 6.0 is a big step because of the
difference in NetCDF. In 6.0, MET uses the enhanced data model of
NetCDF4
which is built upon HDF5. If you already have a NetCDF4 compiled and
built
with HDF5 on your system... it might be pretty easy. But if you have
to
compile it yourself, it's several steps.
Thanks,
John
On Wed, May 10, 2017 at 3:41 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80202 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for diagnosing the problem. At the time, I was compiling MET
V5.2
> and
> intended to set CFLAGS in both since my system is 64-bit. The
tutorial
> showed
> how to set the CFLAGS. I didn't see anything about CXXFLAGS. See the
page
> from
> that tutorial:
> http://www.dtcenter.org/met/users/support/online_tutorial/
> METv5.2/tutorial.php?name=compilation&category=req_libs
>
> Is there any way I can resolve the issue without having to migrate
to MET
> V6.0? I'm trying to evaluate the cost of performing the upgrade at a
time
> when
> I have a lot of other things I'm working on. Obviously, if there is
no
> solution which will enable me to continue with MET V5.2, then I'll
have to
> migrate.
>
> Having installed V5.2, is there any savings in migrating to V6.0
such as
> re-use of the required libraries which might speed up the process or
is it
> best to uninstall (delete everything, including the required
libraries, I
> generated for V5.2) and start over using the new tutorial and User's
Guide
> as
> references?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, May 10, 2017 12:01 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
question
> about Grid-Stat (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Thanks for sending those files. I looked through them and think
I've found
> the problem.
>
> Take a look at this page of the MET online tutorial about the
discussion of
> the -D__64BIT__ flag:
>
> Caution-http://www.dtcenter.org/met/users/support/online_
> tutorial/METv6.0/tutorial.php?name=compilation&category=req_libs
>
> This setting must be used consistently for the GRIB2C library and
MET...
> either define it in neither or both. You've chosen to do it in both
and
> should have configured MET using "CXXFLAGS=-D__64BIT__".
>
> However, looking in config.log, I see that you've set CFLAGS instead
of
> CXXFLAGS:
> CFLAGS='-D__64BIT__'
> CXXFLAGS='-g'
>
> Please try reconfiguring/recompiling met-6.0, being sure to set
CXXFLAGS.
> And then see if your jobs run through batch without the out of
memory
> error.
>
> Please let me know how it goes.
>
> Thanks,
> John
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: question about Grid-Stat (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu May 11 07:43:45 2017
CLASSIFICATION: UNCLASSIFIED
John -
I'd like to try your suggestion to reconfigure/recompile MET V5.2
using the
CXXFLAGS setting. In this case would I need to recompile the grib2c
library
as well?
Thanks.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, May 10, 2017 4:58 PM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
question
about Grid-Stat (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
John,
Sorry for the bad instructions in the 5.2 tutorial. That should
probably have
said CXXFLAGS instead of CFLAGS. We'll get that updated.
You could try recompiling MET 5.2 setting CXXFLAGS that way.
Or you could remove -D__64BIT__ from both MET and the grib2c library.
Either method should work from my experience.
Unfortunately moving form 5.2 to 6.0 is a big step because of the
difference
in NetCDF. In 6.0, MET uses the enhanced data model of NetCDF4 which
is built
upon HDF5. If you already have a NetCDF4 compiled and built with HDF5
on your
system... it might be pretty easy. But if you have to compile it
yourself,
it's several steps.
Thanks,
John
On Wed, May 10, 2017 at 3:41 PM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80202 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for diagnosing the problem. At the time, I was compiling MET
> V5.2 and intended to set CFLAGS in both since my system is 64-bit.
The
> tutorial showed how to set the CFLAGS. I didn't see anything about
> CXXFLAGS. See the page from that tutorial:
> Caution-http://www.dtcenter.org/met/users/support/online_tutorial/
> METv5.2/tutorial.php?name=compilation&category=req_libs
>
> Is there any way I can resolve the issue without having to migrate
to
> MET V6.0? I'm trying to evaluate the cost of performing the upgrade
at
> a time when I have a lot of other things I'm working on. Obviously,
if
> there is no solution which will enable me to continue with MET V5.2,
> then I'll have to migrate.
>
> Having installed V5.2, is there any savings in migrating to V6.0
such
> as re-use of the required libraries which might speed up the process
> or is it best to uninstall (delete everything, including the
required
> libraries, I generated for V5.2) and start over using the new
tutorial
> and User's Guide as references?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Wednesday, May 10, 2017 12:01 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
> question about Grid-Stat (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> Thanks for sending those files. I looked through them and think
I've
> found the problem.
>
> Take a look at this page of the MET online tutorial about the
> discussion of the -D__64BIT__ flag:
>
> Caution-Caution-http://www.dtcenter.org/met/users/support/online_
> tutorial/METv6.0/tutorial.php?name=compilation&category=req_libs
>
> This setting must be used consistently for the GRIB2C library and
MET...
> either define it in neither or both. You've chosen to do it in both
> and should have configured MET using "CXXFLAGS=-D__64BIT__".
>
> However, looking in config.log, I see that you've set CFLAGS instead
> of
> CXXFLAGS:
> CFLAGS='-D__64BIT__'
> CXXFLAGS='-g'
>
> Please try reconfiguring/recompiling met-6.0, being sure to set
CXXFLAGS.
> And then see if your jobs run through batch without the out of
memory
> error.
>
> Please let me know how it goes.
>
> Thanks,
> John
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: question about Grid-Stat (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu May 11 08:14:25 2017
No, the existing build should be fine.
On Thu, May 11, 2017 at 7:43 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80202 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> I'd like to try your suggestion to reconfigure/recompile MET V5.2
using the
> CXXFLAGS setting. In this case would I need to recompile the grib2c
> library
> as well?
>
> Thanks.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, May 10, 2017 4:58 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
question
> about Grid-Stat (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> John,
>
> Sorry for the bad instructions in the 5.2 tutorial. That should
probably
> have
> said CXXFLAGS instead of CFLAGS. We'll get that updated.
>
> You could try recompiling MET 5.2 setting CXXFLAGS that way.
>
> Or you could remove -D__64BIT__ from both MET and the grib2c
library.
>
> Either method should work from my experience.
>
> Unfortunately moving form 5.2 to 6.0 is a big step because of the
> difference
> in NetCDF. In 6.0, MET uses the enhanced data model of NetCDF4
which is
> built
> upon HDF5. If you already have a NetCDF4 compiled and built with
HDF5 on
> your
> system... it might be pretty easy. But if you have to compile it
yourself,
> it's several steps.
>
> Thanks,
> John
>
> On Wed, May 10, 2017 at 3:41 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80202 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > Thanks for diagnosing the problem. At the time, I was compiling
MET
> > V5.2 and intended to set CFLAGS in both since my system is 64-bit.
The
> > tutorial showed how to set the CFLAGS. I didn't see anything about
> > CXXFLAGS. See the page from that tutorial:
> > Caution-http://www.dtcenter.org/met/users/support/online_tutorial/
> > METv5.2/tutorial.php?name=compilation&category=req_libs
> >
> > Is there any way I can resolve the issue without having to
migrate to
> > MET V6.0? I'm trying to evaluate the cost of performing the
upgrade at
> > a time when I have a lot of other things I'm working on.
Obviously, if
> > there is no solution which will enable me to continue with MET
V5.2,
> > then I'll have to migrate.
> >
> > Having installed V5.2, is there any savings in migrating to V6.0
such
> > as re-use of the required libraries which might speed up the
process
> > or is it best to uninstall (delete everything, including the
required
> > libraries, I generated for V5.2) and start over using the new
tutorial
> > and User's Guide as references?
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> > Sent: Wednesday, May 10, 2017 12:01 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
> > question about Grid-Stat (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Thanks for sending those files. I looked through them and think
I've
> > found the problem.
> >
> > Take a look at this page of the MET online tutorial about the
> > discussion of the -D__64BIT__ flag:
> >
> > Caution-Caution-http://www.dtcenter.org/met/users/support/online_
> > tutorial/METv6.0/tutorial.php?name=compilation&category=req_libs
> >
> > This setting must be used consistently for the GRIB2C library and
MET...
> > either define it in neither or both. You've chosen to do it in
both
> > and should have configured MET using "CXXFLAGS=-D__64BIT__".
> >
> > However, looking in config.log, I see that you've set CFLAGS
instead
> > of
> > CXXFLAGS:
> > CFLAGS='-D__64BIT__'
> > CXXFLAGS='-g'
> >
> > Please try reconfiguring/recompiling met-6.0, being sure to set
CXXFLAGS.
> > And then see if your jobs run through batch without the out of
memory
> > error.
> >
> > Please let me know how it goes.
> >
> > Thanks,
> > John
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: question about Grid-Stat (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu May 11 09:43:02 2017
CLASSIFICATION: UNCLASSIFIED
John -
Thanks for confirming that. The following is my proposed procedure for
reconfigure/recompile:
I think that the first step is to make the change in my .bashrc file
as
follows to change the CFLAGS setting to CXXFLAGS:
Change the line export CFLAGS="-D__64BIT__" to export CXXFLAGS="-
D__64BIT__"
Then, source the modified .bashrc from the command line using .
.bashrc
Navigate to /p/home/jraby/MET/met-5.2 (MET install directory)
Then, just repeat the steps for configure using the guidance from the
tutorial:
http://www.dtcenter.org/met/users/support/online_tutorial/METv5.2/tutorial.php?name=compilation&category=configure
which I think is to run configure as follows:
./configure --prefix=/p/home/jraby/MET/met-5.2 --enable-grib2
--enable-modis --enable-mode_graphics
Does the above look correct to you?
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, May 11, 2017 8:14 AM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
question
about Grid-Stat (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
No, the existing build should be fine.
On Thu, May 11, 2017 at 7:43 AM Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80202 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> I'd like to try your suggestion to reconfigure/recompile MET V5.2
> using the CXXFLAGS setting. In this case would I need to recompile
> the grib2c library as well?
>
> Thanks.
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Wednesday, May 10, 2017 4:58 PM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
> question about Grid-Stat (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> John,
>
> Sorry for the bad instructions in the 5.2 tutorial. That should
> probably have said CXXFLAGS instead of CFLAGS. We'll get that
> updated.
>
> You could try recompiling MET 5.2 setting CXXFLAGS that way.
>
> Or you could remove -D__64BIT__ from both MET and the grib2c
library.
>
> Either method should work from my experience.
>
> Unfortunately moving form 5.2 to 6.0 is a big step because of the
> difference in NetCDF. In 6.0, MET uses the enhanced data model of
> NetCDF4 which is built upon HDF5. If you already have a NetCDF4
> compiled and built with HDF5 on your system... it might be pretty
> easy. But if you have to compile it yourself, it's several steps.
>
> Thanks,
> John
>
> On Wed, May 10, 2017 at 3:41 PM, Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80
> > 202 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > Thanks for diagnosing the problem. At the time, I was compiling
MET
> > V5.2 and intended to set CFLAGS in both since my system is 64-bit.
> > The tutorial showed how to set the CFLAGS. I didn't see anything
> > about CXXFLAGS. See the page from that tutorial:
> > Caution-Caution-
http://www.dtcenter.org/met/users/support/online_tut
> > orial/ METv5.2/tutorial.php?name=compilation&category=req_libs
> >
> > Is there any way I can resolve the issue without having to
migrate
> > to MET V6.0? I'm trying to evaluate the cost of performing the
> > upgrade at a time when I have a lot of other things I'm working
on.
> > Obviously, if there is no solution which will enable me to
continue
> > with MET V5.2, then I'll have to migrate.
> >
> > Having installed V5.2, is there any savings in migrating to V6.0
> > such as re-use of the required libraries which might speed up the
> > process or is it best to uninstall (delete everything, including
the
> > required libraries, I generated for V5.2) and start over using the
> > new tutorial and User's Guide as references?
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT
> > [Caution-Caution-mailto:met_help at ucar.edu]
> > Sent: Wednesday, May 10, 2017 12:01 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
> > question about Grid-Stat (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
> > verify the identity of the sender, and confirm the authenticity of
> > all links contained within the message prior to copying and
pasting
> > the address to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Thanks for sending those files. I looked through them and think
> > I've found the problem.
> >
> > Take a look at this page of the MET online tutorial about the
> > discussion of the -D__64BIT__ flag:
> >
> > Caution-Caution-Caution-
http://www.dtcenter.org/met/users/support/online_tutorial/METv6.0/tutorial.php?name=compilation&category=req_libs
> >
> > This setting must be used consistently for the GRIB2C library and
MET...
> > either define it in neither or both. You've chosen to do it in
both
> > and should have configured MET using "CXXFLAGS=-D__64BIT__".
> >
> > However, looking in config.log, I see that you've set CFLAGS
instead
> > of
> > CXXFLAGS:
> > CFLAGS='-D__64BIT__'
> > CXXFLAGS='-g'
> >
> > Please try reconfiguring/recompiling met-6.0, being sure to set
CXXFLAGS.
> > And then see if your jobs run through batch without the out of
> > memory error.
> >
> > Please let me know how it goes.
> >
> > Thanks,
> > John
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
Subject: question about Grid-Stat (UNCLASSIFIED)
From: John Halley Gotway
Time: Thu May 11 10:02:31 2017
Yes, please give it a shot.
John
On Thu, May 11, 2017 at 9:43 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80202 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for confirming that. The following is my proposed procedure
for
> reconfigure/recompile:
>
> I think that the first step is to make the change in my .bashrc file
as
> follows to change the CFLAGS setting to CXXFLAGS:
>
> Change the line export CFLAGS="-D__64BIT__" to export
> CXXFLAGS="-D__64BIT__"
>
> Then, source the modified .bashrc from the command line using .
.bashrc
>
> Navigate to /p/home/jraby/MET/met-5.2 (MET install directory)
>
> Then, just repeat the steps for configure using the guidance from
the
> tutorial:
> http://www.dtcenter.org/met/users/support/online_tutorial/
> METv5.2/tutorial.php?name=compilation&category=configure
>
> which I think is to run configure as follows:
> ./configure --prefix=/p/home/jraby/MET/met-5.2 --enable-grib2
> --enable-modis --enable-mode_graphics
>
> Does the above look correct to you?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, May 11, 2017 8:14 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
question
> about Grid-Stat (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify the
> identity of the sender, and confirm the authenticity of all links
contained
> within the message prior to copying and pasting the address to a Web
> browser.
>
>
>
>
> ----
>
> No, the existing build should be fine.
>
> On Thu, May 11, 2017 at 7:43 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-url:
> > Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80202 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > I'd like to try your suggestion to reconfigure/recompile MET V5.2
> > using the CXXFLAGS setting. In this case would I need to
recompile
> > the grib2c library as well?
> >
> > Thanks.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> > Sent: Wednesday, May 10, 2017 4:58 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
> > question about Grid-Stat (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
verify
> > the identity of the sender, and confirm the authenticity of all
links
> > contained within the message prior to copying and pasting the
address
> > to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Sorry for the bad instructions in the 5.2 tutorial. That should
> > probably have said CXXFLAGS instead of CFLAGS. We'll get that
> > updated.
> >
> > You could try recompiling MET 5.2 setting CXXFLAGS that way.
> >
> > Or you could remove -D__64BIT__ from both MET and the grib2c
library.
> >
> > Either method should work from my experience.
> >
> > Unfortunately moving form 5.2 to 6.0 is a big step because of the
> > difference in NetCDF. In 6.0, MET uses the enhanced data model of
> > NetCDF4 which is built upon HDF5. If you already have a NetCDF4
> > compiled and built with HDF5 on your system... it might be pretty
> > easy. But if you have to compile it yourself, it's several steps.
> >
> > Thanks,
> > John
> >
> > On Wed, May 10, 2017 at 3:41 PM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-url:
> > > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80
> > > 202 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > John -
> > >
> > > Thanks for diagnosing the problem. At the time, I was compiling
MET
> > > V5.2 and intended to set CFLAGS in both since my system is 64-
bit.
> > > The tutorial showed how to set the CFLAGS. I didn't see anything
> > > about CXXFLAGS. See the page from that tutorial:
> > > Caution-Caution-
http://www.dtcenter.org/met/users/support/online_tut
> > > orial/ METv5.2/tutorial.php?name=compilation&category=req_libs
> > >
> > > Is there any way I can resolve the issue without having to
migrate
> > > to MET V6.0? I'm trying to evaluate the cost of performing the
> > > upgrade at a time when I have a lot of other things I'm working
on.
> > > Obviously, if there is no solution which will enable me to
continue
> > > with MET V5.2, then I'll have to migrate.
> > >
> > > Having installed V5.2, is there any savings in migrating to V6.0
> > > such as re-use of the required libraries which might speed up
the
> > > process or is it best to uninstall (delete everything, including
the
> > > required libraries, I generated for V5.2) and start over using
the
> > > new tutorial and User's Guide as references?
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT
> > > [Caution-Caution-mailto:met_help at ucar.edu]
> > > Sent: Wednesday, May 10, 2017 12:01 PM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
> > > question about Grid-Stat (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
pasting
> > > the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > Thanks for sending those files. I looked through them and think
> > > I've found the problem.
> > >
> > > Take a look at this page of the MET online tutorial about the
> > > discussion of the -D__64BIT__ flag:
> > >
> > > Caution-Caution-Caution-http://www.dtcenter.org/met/users/
> support/online_tutorial/METv6.0/tutorial.php?name=
> compilation&category=req_libs
> > >
> > > This setting must be used consistently for the GRIB2C library
and
> MET...
> > > either define it in neither or both. You've chosen to do it in
both
> > > and should have configured MET using "CXXFLAGS=-D__64BIT__".
> > >
> > > However, looking in config.log, I see that you've set CFLAGS
instead
> > > of
> > > CXXFLAGS:
> > > CFLAGS='-D__64BIT__'
> > > CXXFLAGS='-g'
> > >
> > > Please try reconfiguring/recompiling met-6.0, being sure to set
> CXXFLAGS.
> > > And then see if your jobs run through batch without the out of
> > > memory error.
> > >
> > > Please let me know how it goes.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
------------------------------------------------
Subject: question about Grid-Stat (UNCLASSIFIED)
From: Raby, John W USA CIV
Time: Thu May 11 14:54:04 2017
CLASSIFICATION: UNCLASSIFIED
John -
The reconfigure/recompile went smoothly with no errors.
I was able to get MET Grid-Stat to run as a batch job without the "out
of
memory" error. Your diagnosis solved my problem!
Working on debugging some minor errors caused by my config file
settings, but
I don't see anything significant so far. I think it will eventually
run with
success.
Thanks for your help.
R/
John
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Thursday, May 11, 2017 10:03 AM
To: Raby, John W CIV USARMY RDECOM ARL (US)
<john.w.raby2.civ at mail.mil>
Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
question
about Grid-Stat (UNCLASSIFIED)
All active links contained in this email were disabled. Please verify
the
identity of the sender, and confirm the authenticity of all links
contained
within the message prior to copying and pasting the address to a Web
browser.
----
Yes, please give it a shot.
John
On Thu, May 11, 2017 at 9:43 AM, Raby, John W USA CIV via RT <
met_help at ucar.edu> wrote:
>
> <Caution-url:
> Caution-https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80202 >
>
> CLASSIFICATION: UNCLASSIFIED
>
> John -
>
> Thanks for confirming that. The following is my proposed procedure
for
> reconfigure/recompile:
>
> I think that the first step is to make the change in my .bashrc file
> as follows to change the CFLAGS setting to CXXFLAGS:
>
> Change the line export CFLAGS="-D__64BIT__" to export
> CXXFLAGS="-D__64BIT__"
>
> Then, source the modified .bashrc from the command line using .
> .bashrc
>
> Navigate to /p/home/jraby/MET/met-5.2 (MET install directory)
>
> Then, just repeat the steps for configure using the guidance from
the
> tutorial:
> Caution-http://www.dtcenter.org/met/users/support/online_tutorial/
> METv5.2/tutorial.php?name=compilation&category=configure
>
> which I think is to run configure as follows:
> ./configure --prefix=/p/home/jraby/MET/met-5.2 --enable-grib2
> --enable-modis --enable-mode_graphics
>
> Does the above look correct to you?
>
> R/
> John
>
> -----Original Message-----
> From: John Halley Gotway via RT [Caution-mailto:met_help at ucar.edu]
> Sent: Thursday, May 11, 2017 8:14 AM
> To: Raby, John W CIV USARMY RDECOM ARL (US)
> <john.w.raby2.civ at mail.mil>
> Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
> question about Grid-Stat (UNCLASSIFIED)
>
> All active links contained in this email were disabled. Please
verify
> the identity of the sender, and confirm the authenticity of all
links
> contained within the message prior to copying and pasting the
address
> to a Web browser.
>
>
>
>
> ----
>
> No, the existing build should be fine.
>
> On Thu, May 11, 2017 at 7:43 AM Raby, John W USA CIV via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <Caution-Caution-url:
> > Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=80
> > 202 >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> > John -
> >
> > I'd like to try your suggestion to reconfigure/recompile MET V5.2
> > using the CXXFLAGS setting. In this case would I need to
recompile
> > the grib2c library as well?
> >
> > Thanks.
> >
> > R/
> > John
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT
> > [Caution-Caution-mailto:met_help at ucar.edu]
> > Sent: Wednesday, May 10, 2017 4:58 PM
> > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > <john.w.raby2.civ at mail.mil>
> > Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
> > question about Grid-Stat (UNCLASSIFIED)
> >
> > All active links contained in this email were disabled. Please
> > verify the identity of the sender, and confirm the authenticity of
> > all links contained within the message prior to copying and
pasting
> > the address to a Web browser.
> >
> >
> >
> >
> > ----
> >
> > John,
> >
> > Sorry for the bad instructions in the 5.2 tutorial. That should
> > probably have said CXXFLAGS instead of CFLAGS. We'll get that
> > updated.
> >
> > You could try recompiling MET 5.2 setting CXXFLAGS that way.
> >
> > Or you could remove -D__64BIT__ from both MET and the grib2c
library.
> >
> > Either method should work from my experience.
> >
> > Unfortunately moving form 5.2 to 6.0 is a big step because of the
> > difference in NetCDF. In 6.0, MET uses the enhanced data model of
> > NetCDF4 which is built upon HDF5. If you already have a NetCDF4
> > compiled and built with HDF5 on your system... it might be pretty
> > easy. But if you have to compile it yourself, it's several steps.
> >
> > Thanks,
> > John
> >
> > On Wed, May 10, 2017 at 3:41 PM, Raby, John W USA CIV via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <Caution-Caution-Caution-url:
> > > Caution-Caution-Caution-
https://rt.rap.ucar.edu/rt/Ticket/Display.
> > > html?id=80
> > > 202 >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > > John -
> > >
> > > Thanks for diagnosing the problem. At the time, I was compiling
> > > MET
> > > V5.2 and intended to set CFLAGS in both since my system is 64-
bit.
> > > The tutorial showed how to set the CFLAGS. I didn't see anything
> > > about CXXFLAGS. See the page from that tutorial:
> > > Caution-Caution-Caution-
http://www.dtcenter.org/met/users/support/
> > > online_tut orial/
> > > METv5.2/tutorial.php?name=compilation&category=req_libs
> > >
> > > Is there any way I can resolve the issue without having to
> > > migrate to MET V6.0? I'm trying to evaluate the cost of
performing
> > > the upgrade at a time when I have a lot of other things I'm
working on.
> > > Obviously, if there is no solution which will enable me to
> > > continue with MET V5.2, then I'll have to migrate.
> > >
> > > Having installed V5.2, is there any savings in migrating to V6.0
> > > such as re-use of the required libraries which might speed up
the
> > > process or is it best to uninstall (delete everything, including
> > > the required libraries, I generated for V5.2) and start over
using
> > > the new tutorial and User's Guide as references?
> > >
> > > R/
> > > John
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT
> > > [Caution-Caution-Caution-mailto:met_help at ucar.edu]
> > > Sent: Wednesday, May 10, 2017 12:01 PM
> > > To: Raby, John W CIV USARMY RDECOM ARL (US)
> > > <john.w.raby2.civ at mail.mil>
> > > Subject: Re: [Non-DoD Source] [rt.rap.ucar.edu #80202] Resolved:
> > > question about Grid-Stat (UNCLASSIFIED)
> > >
> > > All active links contained in this email were disabled. Please
> > > verify the identity of the sender, and confirm the authenticity
of
> > > all links contained within the message prior to copying and
> > > pasting the address to a Web browser.
> > >
> > >
> > >
> > >
> > > ----
> > >
> > > John,
> > >
> > > Thanks for sending those files. I looked through them and think
> > > I've found the problem.
> > >
> > > Take a look at this page of the MET online tutorial about the
> > > discussion of the -D__64BIT__ flag:
> > >
> > > Caution-Caution-Caution-Caution-
http://www.dtcenter.org/met/users/
> support/online_tutorial/METv6.0/tutorial.php?name=
> compilation&category=req_libs
> > >
> > > This setting must be used consistently for the GRIB2C library
and
> MET...
> > > either define it in neither or both. You've chosen to do it in
> > > both and should have configured MET using "CXXFLAGS=-
D__64BIT__".
> > >
> > > However, looking in config.log, I see that you've set CFLAGS
> > > instead of
> > > CXXFLAGS:
> > > CFLAGS='-D__64BIT__'
> > > CXXFLAGS='-g'
> > >
> > > Please try reconfiguring/recompiling met-6.0, being sure to set
> CXXFLAGS.
> > > And then see if your jobs run through batch without the out of
> > > memory error.
> > >
> > > Please let me know how it goes.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > > CLASSIFICATION: UNCLASSIFIED
> > >
> > >
> >
> >
> > CLASSIFICATION: UNCLASSIFIED
> >
> >
>
>
> CLASSIFICATION: UNCLASSIFIED
>
>
CLASSIFICATION: UNCLASSIFIED
------------------------------------------------
More information about the Met_help
mailing list