[Met_help] [rt.rap.ucar.edu #57644] History for Processing time for point_stat with NCAR ds337 ncdf files
John Halley Gotway via RT
met_help at ucar.edu
Thu Aug 2 13:54:02 MDT 2012
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello again John,
I was wondering if you could comment on the following behavior of
point_stat when running with PrepBufr files from the NCAR ds337 dataset.
When I run point_stat with daily input ncdf observation files (generated
by pb2nc) that cover a relatively small model domain (say, the size of
Southern New England), with a corresponding small number of obs (several
tens of thousand individual obs valid over the entire day), point_stat
completes successfully in less than 10 s. When I try to verify on an
outer domain that covers CONUS, with a corresponding large number of obs
(1-2 million), point_stat requires about 60-70 minutes to complete.
Thus, to verify hourly a full model run of 24 valid times, over a day of
wallclock time is required.
I don't recall if this scaling behavior occurred for v2.0 of MET.
I welcome your thoughts.
Thanks.
Respectfully,
John Henderson
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #57644] Processing time for point_stat with NCAR ds337 ncdf files
From: John Halley Gotway
Time: Wed Aug 01 08:12:27 2012
John,
There are two aspects of point_stat that don't scale very well - the
computation of rank correlation coefficients and bootstrap confidence
intervals. Both of them are controlled by settings in the
config file and disabling them yields significant increases in runtime
performance. By default, they are both *enabled* in Point-Stat since
that tool is typically run with a relatively small number
of observations. And by default, they are both *disabled* in Grid-
Stat since that tool is typically run with a relatively large number
of observations.
I'd suggest turning both of them off and try rerunning to see how much
the runtime improves.
Disable rank correlation computations by setting "rank_corr_flag =
FALSE;" in METv4.0 (or by setting "rank_corr_flag = 0;" in previous
versions of MET).
Disable bootstrap confidence intervals by setting "n_rep = 0;" in the
"boot" section in METv4.0 (or by setting "n_boot_rep = 0;" in previous
versions of MET).
Please let me know if/how much those changes improve the runtime
performance.
Thanks,
John
On 07/30/2012 01:25 PM, jhenders at aer.com via RT wrote:
>
> Mon Jul 30 13:25:36 2012: Request 57644 was acted upon.
> Transaction: Ticket created by jhenders at aer.com
> Queue: met_help
> Subject: Processing time for point_stat with NCAR ds337 ncdf
files
> Owner: Nobody
> Requestors: jhenders at aer.com
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>
>
> Hello again John,
>
> I was wondering if you could comment on the following behavior of
> point_stat when running with PrepBufr files from the NCAR ds337
dataset.
>
> When I run point_stat with daily input ncdf observation files
(generated
> by pb2nc) that cover a relatively small model domain (say, the size
of
> Southern New England), with a corresponding small number of obs
(several
> tens of thousand individual obs valid over the entire day),
point_stat
> completes successfully in less than 10 s. When I try to verify on an
> outer domain that covers CONUS, with a corresponding large number of
obs
> (1-2 million), point_stat requires about 60-70 minutes to complete.
> Thus, to verify hourly a full model run of 24 valid times, over a
day of
> wallclock time is required.
>
> I don't recall if this scaling behavior occurred for v2.0 of MET.
>
> I welcome your thoughts.
>
> Thanks.
>
> Respectfully,
>
> John Henderson
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #57644] Processing time for point_stat with NCAR ds337 ncdf files
From: jhenders at aer.com
Time: Wed Aug 01 08:30:12 2012
Hello John,
It turns out that I already have both of those disabled. I wanted to
speed up my MET v2 runs a year ago, so at that time I set the flags
appropriately.
John
On 8/1/12 10:12 AM, John Halley Gotway via RT wrote:
> John,
>
> There are two aspects of point_stat that don't scale very well - the
computation of rank correlation coefficients and bootstrap confidence
intervals. Both of them are controlled by settings in the
> config file and disabling them yields significant increases in
runtime performance. By default, they are both *enabled* in Point-
Stat since that tool is typically run with a relatively small number
> of observations. And by default, they are both *disabled* in Grid-
Stat since that tool is typically run with a relatively large number
of observations.
>
> I'd suggest turning both of them off and try rerunning to see how
much the runtime improves.
>
> Disable rank correlation computations by setting "rank_corr_flag =
FALSE;" in METv4.0 (or by setting "rank_corr_flag = 0;" in previous
versions of MET).
> Disable bootstrap confidence intervals by setting "n_rep = 0;" in
the "boot" section in METv4.0 (or by setting "n_boot_rep = 0;" in
previous versions of MET).
>
> Please let me know if/how much those changes improve the runtime
performance.
>
> Thanks,
> John
>
> On 07/30/2012 01:25 PM, jhenders at aer.com via RT wrote:
>> Mon Jul 30 13:25:36 2012: Request 57644 was acted upon.
>> Transaction: Ticket created by jhenders at aer.com
>> Queue: met_help
>> Subject: Processing time for point_stat with NCAR ds337 ncdf
files
>> Owner: Nobody
>> Requestors: jhenders at aer.com
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>
>>
>> Hello again John,
>>
>> I was wondering if you could comment on the following behavior of
>> point_stat when running with PrepBufr files from the NCAR ds337
dataset.
>>
>> When I run point_stat with daily input ncdf observation files
(generated
>> by pb2nc) that cover a relatively small model domain (say, the size
of
>> Southern New England), with a corresponding small number of obs
(several
>> tens of thousand individual obs valid over the entire day),
point_stat
>> completes successfully in less than 10 s. When I try to verify on
an
>> outer domain that covers CONUS, with a corresponding large number
of obs
>> (1-2 million), point_stat requires about 60-70 minutes to complete.
>> Thus, to verify hourly a full model run of 24 valid times, over a
day of
>> wallclock time is required.
>>
>> I don't recall if this scaling behavior occurred for v2.0 of MET.
>>
>> I welcome your thoughts.
>>
>> Thanks.
>>
>> Respectfully,
>>
>> John Henderson
>>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #57644] Processing time for point_stat with NCAR ds337 ncdf files
From: John Halley Gotway
Time: Wed Aug 01 08:47:28 2012
John,
Uh oh, guess I'm out of tricks then!
Would you be able to send me sample data for a run that's taking 60-70
minutes? I could try running it here to see if there's any other
improvements that might help. I'd need the sample forecast
file, NetCDF point observation file, and the Point-Stat Config file.
You can post data to our anonymous ftp site following these
directions:
http://www.dtcenter.org/met/users/support/met_help.php#ftp
Thanks,
John
On 08/01/2012 08:30 AM, jhenders at aer.com via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>
> Hello John,
>
> It turns out that I already have both of those disabled. I wanted to
> speed up my MET v2 runs a year ago, so at that time I set the flags
> appropriately.
>
> John
>
> On 8/1/12 10:12 AM, John Halley Gotway via RT wrote:
>> John,
>>
>> There are two aspects of point_stat that don't scale very well -
the computation of rank correlation coefficients and bootstrap
confidence intervals. Both of them are controlled by settings in the
>> config file and disabling them yields significant increases in
runtime performance. By default, they are both *enabled* in Point-
Stat since that tool is typically run with a relatively small number
>> of observations. And by default, they are both *disabled* in Grid-
Stat since that tool is typically run with a relatively large number
of observations.
>>
>> I'd suggest turning both of them off and try rerunning to see how
much the runtime improves.
>>
>> Disable rank correlation computations by setting "rank_corr_flag =
FALSE;" in METv4.0 (or by setting "rank_corr_flag = 0;" in previous
versions of MET).
>> Disable bootstrap confidence intervals by setting "n_rep = 0;" in
the "boot" section in METv4.0 (or by setting "n_boot_rep = 0;" in
previous versions of MET).
>>
>> Please let me know if/how much those changes improve the runtime
performance.
>>
>> Thanks,
>> John
>>
>> On 07/30/2012 01:25 PM, jhenders at aer.com via RT wrote:
>>> Mon Jul 30 13:25:36 2012: Request 57644 was acted upon.
>>> Transaction: Ticket created by jhenders at aer.com
>>> Queue: met_help
>>> Subject: Processing time for point_stat with NCAR ds337
ncdf files
>>> Owner: Nobody
>>> Requestors: jhenders at aer.com
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>
>>>
>>> Hello again John,
>>>
>>> I was wondering if you could comment on the following behavior of
>>> point_stat when running with PrepBufr files from the NCAR ds337
dataset.
>>>
>>> When I run point_stat with daily input ncdf observation files
(generated
>>> by pb2nc) that cover a relatively small model domain (say, the
size of
>>> Southern New England), with a corresponding small number of obs
(several
>>> tens of thousand individual obs valid over the entire day),
point_stat
>>> completes successfully in less than 10 s. When I try to verify on
an
>>> outer domain that covers CONUS, with a corresponding large number
of obs
>>> (1-2 million), point_stat requires about 60-70 minutes to
complete.
>>> Thus, to verify hourly a full model run of 24 valid times, over a
day of
>>> wallclock time is required.
>>>
>>> I don't recall if this scaling behavior occurred for v2.0 of MET.
>>>
>>> I welcome your thoughts.
>>>
>>> Thanks.
>>>
>>> Respectfully,
>>>
>>> John Henderson
>>>
>>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #57644] Processing time for point_stat with NCAR ds337 ncdf files
From: jhenders at aer.com
Time: Wed Aug 01 11:10:36 2012
Hi John,
I've loaded the relevant files to henderson_data. Note that because of
the offset (with respect to UTC dates) for ds337's files, I call
point_stat with two daily ncdf files. This, of course, contributes to
the 60-70 minutes it requires for point-stat, but my question simply
is
whether it is normal for such a non-linear scaling.
Thanks.
John
On 8/1/12 10:47 AM, John Halley Gotway via RT wrote:
> John,
>
> Uh oh, guess I'm out of tricks then!
>
> Would you be able to send me sample data for a run that's taking 60-
70 minutes? I could try running it here to see if there's any other
improvements that might help. I'd need the sample forecast
> file, NetCDF point observation file, and the Point-Stat Config file.
>
> You can post data to our anonymous ftp site following these
directions:
> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Thanks,
> John
>
> On 08/01/2012 08:30 AM, jhenders at aer.com via RT wrote:
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>
>> Hello John,
>>
>> It turns out that I already have both of those disabled. I wanted
to
>> speed up my MET v2 runs a year ago, so at that time I set the flags
>> appropriately.
>>
>> John
>>
>> On 8/1/12 10:12 AM, John Halley Gotway via RT wrote:
>>> John,
>>>
>>> There are two aspects of point_stat that don't scale very well -
the computation of rank correlation coefficients and bootstrap
confidence intervals. Both of them are controlled by settings in the
>>> config file and disabling them yields significant increases in
runtime performance. By default, they are both *enabled* in Point-
Stat since that tool is typically run with a relatively small number
>>> of observations. And by default, they are both *disabled* in
Grid-Stat since that tool is typically run with a relatively large
number of observations.
>>>
>>> I'd suggest turning both of them off and try rerunning to see how
much the runtime improves.
>>>
>>> Disable rank correlation computations by setting "rank_corr_flag =
FALSE;" in METv4.0 (or by setting "rank_corr_flag = 0;" in previous
versions of MET).
>>> Disable bootstrap confidence intervals by setting "n_rep = 0;" in
the "boot" section in METv4.0 (or by setting "n_boot_rep = 0;" in
previous versions of MET).
>>>
>>> Please let me know if/how much those changes improve the runtime
performance.
>>>
>>> Thanks,
>>> John
>>>
>>> On 07/30/2012 01:25 PM, jhenders at aer.com via RT wrote:
>>>> Mon Jul 30 13:25:36 2012: Request 57644 was acted upon.
>>>> Transaction: Ticket created by jhenders at aer.com
>>>> Queue: met_help
>>>> Subject: Processing time for point_stat with NCAR ds337
ncdf files
>>>> Owner: Nobody
>>>> Requestors: jhenders at aer.com
>>>> Status: new
>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>
>>>>
>>>> Hello again John,
>>>>
>>>> I was wondering if you could comment on the following behavior of
>>>> point_stat when running with PrepBufr files from the NCAR ds337
dataset.
>>>>
>>>> When I run point_stat with daily input ncdf observation files
(generated
>>>> by pb2nc) that cover a relatively small model domain (say, the
size of
>>>> Southern New England), with a corresponding small number of obs
(several
>>>> tens of thousand individual obs valid over the entire day),
point_stat
>>>> completes successfully in less than 10 s. When I try to verify on
an
>>>> outer domain that covers CONUS, with a corresponding large number
of obs
>>>> (1-2 million), point_stat requires about 60-70 minutes to
complete.
>>>> Thus, to verify hourly a full model run of 24 valid times, over a
day of
>>>> wallclock time is required.
>>>>
>>>> I don't recall if this scaling behavior occurred for v2.0 of MET.
>>>>
>>>> I welcome your thoughts.
>>>>
>>>> Thanks.
>>>>
>>>> Respectfully,
>>>>
>>>> John Henderson
>>>>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #57644] Processing time for point_stat with NCAR ds337 ncdf files
From: John Halley Gotway
Time: Wed Aug 01 11:32:18 2012
John,
Something funny is going on here. I ran the exact data you sent me on
my desktop machine, and it took 24 seconds to complete - not 60-70
minutes.
My guess would be something to do with memory (perhaps it's running
out and using swap space) and/or disk I/O. On what system are you
running this?
Are you able to try running the same thing on a different system to
see what the runtime is?
John
On 08/01/2012 11:10 AM, jhenders at aer.com via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>
> Hi John,
>
> I've loaded the relevant files to henderson_data. Note that because
of
> the offset (with respect to UTC dates) for ds337's files, I call
> point_stat with two daily ncdf files. This, of course, contributes
to
> the 60-70 minutes it requires for point-stat, but my question simply
is
> whether it is normal for such a non-linear scaling.
>
> Thanks.
>
> John
>
> On 8/1/12 10:47 AM, John Halley Gotway via RT wrote:
>> John,
>>
>> Uh oh, guess I'm out of tricks then!
>>
>> Would you be able to send me sample data for a run that's taking
60-70 minutes? I could try running it here to see if there's any
other improvements that might help. I'd need the sample forecast
>> file, NetCDF point observation file, and the Point-Stat Config
file.
>>
>> You can post data to our anonymous ftp site following these
directions:
>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>
>> Thanks,
>> John
>>
>> On 08/01/2012 08:30 AM, jhenders at aer.com via RT wrote:
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>
>>> Hello John,
>>>
>>> It turns out that I already have both of those disabled. I wanted
to
>>> speed up my MET v2 runs a year ago, so at that time I set the
flags
>>> appropriately.
>>>
>>> John
>>>
>>> On 8/1/12 10:12 AM, John Halley Gotway via RT wrote:
>>>> John,
>>>>
>>>> There are two aspects of point_stat that don't scale very well -
the computation of rank correlation coefficients and bootstrap
confidence intervals. Both of them are controlled by settings in the
>>>> config file and disabling them yields significant increases in
runtime performance. By default, they are both *enabled* in Point-
Stat since that tool is typically run with a relatively small number
>>>> of observations. And by default, they are both *disabled* in
Grid-Stat since that tool is typically run with a relatively large
number of observations.
>>>>
>>>> I'd suggest turning both of them off and try rerunning to see how
much the runtime improves.
>>>>
>>>> Disable rank correlation computations by setting "rank_corr_flag
= FALSE;" in METv4.0 (or by setting "rank_corr_flag = 0;" in previous
versions of MET).
>>>> Disable bootstrap confidence intervals by setting "n_rep = 0;" in
the "boot" section in METv4.0 (or by setting "n_boot_rep = 0;" in
previous versions of MET).
>>>>
>>>> Please let me know if/how much those changes improve the runtime
performance.
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>> On 07/30/2012 01:25 PM, jhenders at aer.com via RT wrote:
>>>>> Mon Jul 30 13:25:36 2012: Request 57644 was acted upon.
>>>>> Transaction: Ticket created by jhenders at aer.com
>>>>> Queue: met_help
>>>>> Subject: Processing time for point_stat with NCAR
ds337 ncdf files
>>>>> Owner: Nobody
>>>>> Requestors: jhenders at aer.com
>>>>> Status: new
>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>
>>>>>
>>>>> Hello again John,
>>>>>
>>>>> I was wondering if you could comment on the following behavior
of
>>>>> point_stat when running with PrepBufr files from the NCAR ds337
dataset.
>>>>>
>>>>> When I run point_stat with daily input ncdf observation files
(generated
>>>>> by pb2nc) that cover a relatively small model domain (say, the
size of
>>>>> Southern New England), with a corresponding small number of obs
(several
>>>>> tens of thousand individual obs valid over the entire day),
point_stat
>>>>> completes successfully in less than 10 s. When I try to verify
on an
>>>>> outer domain that covers CONUS, with a corresponding large
number of obs
>>>>> (1-2 million), point_stat requires about 60-70 minutes to
complete.
>>>>> Thus, to verify hourly a full model run of 24 valid times, over
a day of
>>>>> wallclock time is required.
>>>>>
>>>>> I don't recall if this scaling behavior occurred for v2.0 of
MET.
>>>>>
>>>>> I welcome your thoughts.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Respectfully,
>>>>>
>>>>> John Henderson
>>>>>
>>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #57644] Processing time for point_stat with NCAR ds337 ncdf files
From: jhenders at aer.com
Time: Wed Aug 01 11:47:38 2012
Hi John,
Hmm, I have to admit that I am encouraged by your results and quite
(but
not completely) surprised. I am running on Pleiades, a supercomputer
at
NASA Ames. This is the machine on which I have done all my MET
v3.1-related work. (I ported MET to the machine last month for the
first
time specifically for these runs). Our past correspondence from a year
or so ago related to MET v2 runs done locally at AER.
Just to remove as many variables as possible, could you please include
the precise call you used? If I get a chance, I also will run the
point-stat job locally to get some timing.
In the meantime, I will investigate further on Pleiades. Thank you
very
much for identifying that the problem is on my end.
John
On 8/1/12 1:32 PM, John Halley Gotway via RT wrote:
> John,
>
> Something funny is going on here. I ran the exact data you sent me
on my desktop machine, and it took 24 seconds to complete - not 60-70
minutes.
>
> My guess would be something to do with memory (perhaps it's running
out and using swap space) and/or disk I/O. On what system are you
running this?
>
> Are you able to try running the same thing on a different system to
see what the runtime is?
>
> John
>
> On 08/01/2012 11:10 AM, jhenders at aer.com via RT wrote:
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>
>> Hi John,
>>
>> I've loaded the relevant files to henderson_data. Note that because
of
>> the offset (with respect to UTC dates) for ds337's files, I call
>> point_stat with two daily ncdf files. This, of course, contributes
to
>> the 60-70 minutes it requires for point-stat, but my question
simply is
>> whether it is normal for such a non-linear scaling.
>>
>> Thanks.
>>
>> John
>>
>> On 8/1/12 10:47 AM, John Halley Gotway via RT wrote:
>>> John,
>>>
>>> Uh oh, guess I'm out of tricks then!
>>>
>>> Would you be able to send me sample data for a run that's taking
60-70 minutes? I could try running it here to see if there's any
other improvements that might help. I'd need the sample forecast
>>> file, NetCDF point observation file, and the Point-Stat Config
file.
>>>
>>> You can post data to our anonymous ftp site following these
directions:
>>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>
>>> Thanks,
>>> John
>>>
>>> On 08/01/2012 08:30 AM, jhenders at aer.com via RT wrote:
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>
>>>> Hello John,
>>>>
>>>> It turns out that I already have both of those disabled. I wanted
to
>>>> speed up my MET v2 runs a year ago, so at that time I set the
flags
>>>> appropriately.
>>>>
>>>> John
>>>>
>>>> On 8/1/12 10:12 AM, John Halley Gotway via RT wrote:
>>>>> John,
>>>>>
>>>>> There are two aspects of point_stat that don't scale very well -
the computation of rank correlation coefficients and bootstrap
confidence intervals. Both of them are controlled by settings in the
>>>>> config file and disabling them yields significant increases in
runtime performance. By default, they are both *enabled* in Point-
Stat since that tool is typically run with a relatively small number
>>>>> of observations. And by default, they are both *disabled* in
Grid-Stat since that tool is typically run with a relatively large
number of observations.
>>>>>
>>>>> I'd suggest turning both of them off and try rerunning to see
how much the runtime improves.
>>>>>
>>>>> Disable rank correlation computations by setting "rank_corr_flag
= FALSE;" in METv4.0 (or by setting "rank_corr_flag = 0;" in previous
versions of MET).
>>>>> Disable bootstrap confidence intervals by setting "n_rep = 0;"
in the "boot" section in METv4.0 (or by setting "n_boot_rep = 0;" in
previous versions of MET).
>>>>>
>>>>> Please let me know if/how much those changes improve the runtime
performance.
>>>>>
>>>>> Thanks,
>>>>> John
>>>>>
>>>>> On 07/30/2012 01:25 PM, jhenders at aer.com via RT wrote:
>>>>>> Mon Jul 30 13:25:36 2012: Request 57644 was acted upon.
>>>>>> Transaction: Ticket created by jhenders at aer.com
>>>>>> Queue: met_help
>>>>>> Subject: Processing time for point_stat with NCAR
ds337 ncdf files
>>>>>> Owner: Nobody
>>>>>> Requestors: jhenders at aer.com
>>>>>> Status: new
>>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>
>>>>>>
>>>>>> Hello again John,
>>>>>>
>>>>>> I was wondering if you could comment on the following behavior
of
>>>>>> point_stat when running with PrepBufr files from the NCAR ds337
dataset.
>>>>>>
>>>>>> When I run point_stat with daily input ncdf observation files
(generated
>>>>>> by pb2nc) that cover a relatively small model domain (say, the
size of
>>>>>> Southern New England), with a corresponding small number of obs
(several
>>>>>> tens of thousand individual obs valid over the entire day),
point_stat
>>>>>> completes successfully in less than 10 s. When I try to verify
on an
>>>>>> outer domain that covers CONUS, with a corresponding large
number of obs
>>>>>> (1-2 million), point_stat requires about 60-70 minutes to
complete.
>>>>>> Thus, to verify hourly a full model run of 24 valid times, over
a day of
>>>>>> wallclock time is required.
>>>>>>
>>>>>> I don't recall if this scaling behavior occurred for v2.0 of
MET.
>>>>>>
>>>>>> I welcome your thoughts.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> Respectfully,
>>>>>>
>>>>>> John Henderson
>>>>>>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #57644] Processing time for point_stat with NCAR ds337 ncdf files
From: jhenders at aer.com
Time: Wed Aug 01 14:58:47 2012
Hello again John,
I reran my test case locally on a rather old linux cluster. This is
the
same one I used for MET v2 a year or so ago. This current job took
about
5 minutes and seems to have completed successfully.
I then reran on a newer cluster and it breezed through the reading of
the observations in a matter of seconds. This is the step that took a
very long time on Pleiades. However, close inspection of the output
files shows that computations involving the forecast (FBAR, e.g.) are
all Nans. Physical values appear for all observation-related
quantities.
I will keep looking...
John
On 8/1/12 1:32 PM, John Halley Gotway via RT wrote:
> John,
>
> Something funny is going on here. I ran the exact data you sent me
on my desktop machine, and it took 24 seconds to complete - not 60-70
minutes.
>
> My guess would be something to do with memory (perhaps it's running
out and using swap space) and/or disk I/O. On what system are you
running this?
>
> Are you able to try running the same thing on a different system to
see what the runtime is?
>
> John
>
> On 08/01/2012 11:10 AM, jhenders at aer.com via RT wrote:
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>
>> Hi John,
>>
>> I've loaded the relevant files to henderson_data. Note that because
of
>> the offset (with respect to UTC dates) for ds337's files, I call
>> point_stat with two daily ncdf files. This, of course, contributes
to
>> the 60-70 minutes it requires for point-stat, but my question
simply is
>> whether it is normal for such a non-linear scaling.
>>
>> Thanks.
>>
>> John
>>
>> On 8/1/12 10:47 AM, John Halley Gotway via RT wrote:
>>> John,
>>>
>>> Uh oh, guess I'm out of tricks then!
>>>
>>> Would you be able to send me sample data for a run that's taking
60-70 minutes? I could try running it here to see if there's any
other improvements that might help. I'd need the sample forecast
>>> file, NetCDF point observation file, and the Point-Stat Config
file.
>>>
>>> You can post data to our anonymous ftp site following these
directions:
>>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>
>>> Thanks,
>>> John
>>>
>>> On 08/01/2012 08:30 AM, jhenders at aer.com via RT wrote:
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>
>>>> Hello John,
>>>>
>>>> It turns out that I already have both of those disabled. I wanted
to
>>>> speed up my MET v2 runs a year ago, so at that time I set the
flags
>>>> appropriately.
>>>>
>>>> John
>>>>
>>>> On 8/1/12 10:12 AM, John Halley Gotway via RT wrote:
>>>>> John,
>>>>>
>>>>> There are two aspects of point_stat that don't scale very well -
the computation of rank correlation coefficients and bootstrap
confidence intervals. Both of them are controlled by settings in the
>>>>> config file and disabling them yields significant increases in
runtime performance. By default, they are both *enabled* in Point-
Stat since that tool is typically run with a relatively small number
>>>>> of observations. And by default, they are both *disabled* in
Grid-Stat since that tool is typically run with a relatively large
number of observations.
>>>>>
>>>>> I'd suggest turning both of them off and try rerunning to see
how much the runtime improves.
>>>>>
>>>>> Disable rank correlation computations by setting "rank_corr_flag
= FALSE;" in METv4.0 (or by setting "rank_corr_flag = 0;" in previous
versions of MET).
>>>>> Disable bootstrap confidence intervals by setting "n_rep = 0;"
in the "boot" section in METv4.0 (or by setting "n_boot_rep = 0;" in
previous versions of MET).
>>>>>
>>>>> Please let me know if/how much those changes improve the runtime
performance.
>>>>>
>>>>> Thanks,
>>>>> John
>>>>>
>>>>> On 07/30/2012 01:25 PM, jhenders at aer.com via RT wrote:
>>>>>> Mon Jul 30 13:25:36 2012: Request 57644 was acted upon.
>>>>>> Transaction: Ticket created by jhenders at aer.com
>>>>>> Queue: met_help
>>>>>> Subject: Processing time for point_stat with NCAR
ds337 ncdf files
>>>>>> Owner: Nobody
>>>>>> Requestors: jhenders at aer.com
>>>>>> Status: new
>>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>
>>>>>>
>>>>>> Hello again John,
>>>>>>
>>>>>> I was wondering if you could comment on the following behavior
of
>>>>>> point_stat when running with PrepBufr files from the NCAR ds337
dataset.
>>>>>>
>>>>>> When I run point_stat with daily input ncdf observation files
(generated
>>>>>> by pb2nc) that cover a relatively small model domain (say, the
size of
>>>>>> Southern New England), with a corresponding small number of obs
(several
>>>>>> tens of thousand individual obs valid over the entire day),
point_stat
>>>>>> completes successfully in less than 10 s. When I try to verify
on an
>>>>>> outer domain that covers CONUS, with a corresponding large
number of obs
>>>>>> (1-2 million), point_stat requires about 60-70 minutes to
complete.
>>>>>> Thus, to verify hourly a full model run of 24 valid times, over
a day of
>>>>>> wallclock time is required.
>>>>>>
>>>>>> I don't recall if this scaling behavior occurred for v2.0 of
MET.
>>>>>>
>>>>>> I welcome your thoughts.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> Respectfully,
>>>>>>
>>>>>> John Henderson
>>>>>>
>
------------------------------------------------
Subject: Processing time for point_stat with NCAR ds337 ncdf files
From: John Halley Gotway
Time: Thu Aug 02 08:54:40 2012
John,
Sorry for the delay in getting back to you. I was out of the office
yesterday afternoon.
Here's the exact command I ran with the data you sent:
time \
${MET_BASE}/bin/point_stat \
d1_2010121500_1200.grib \
obs-20101215.nc \
PointStatConfig_V3.1-p1652 \
-point_obs obs-20101216.nc \
-outdir out \
-v 4
And as I mentioned, it took about 24 seconds to execute on my desktop.
I've attached a tar file containing the ASCII output of this run. I'm
not seeing NaN's in the FBAR column (in
point_stat_120000L_20101215_120000V_cnt.txt). Instead I see
reasonable FBAR values ...
followed by reasonable values in FBAR_NCL and FBAR_NCU for the normal
confidence limits ... followed by NA's in the FBAR_BCL and FBAR_BCU
since we disabled bootstrap CI's in the config file.
Is it possible you were seeing the NA's in those columns rather than
the good values in the FBAR column?
Thanks,
John
On 08/01/2012 02:58 PM, jhenders at aer.com via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>
> Hello again John,
>
> I reran my test case locally on a rather old linux cluster. This is
the
> same one I used for MET v2 a year or so ago. This current job took
about
> 5 minutes and seems to have completed successfully.
>
> I then reran on a newer cluster and it breezed through the reading
of
> the observations in a matter of seconds. This is the step that took
a
> very long time on Pleiades. However, close inspection of the output
> files shows that computations involving the forecast (FBAR, e.g.)
are
> all Nans. Physical values appear for all observation-related
quantities.
>
> I will keep looking...
>
> John
>
> On 8/1/12 1:32 PM, John Halley Gotway via RT wrote:
>> John,
>>
>> Something funny is going on here. I ran the exact data you sent me
on my desktop machine, and it took 24 seconds to complete - not 60-70
minutes.
>>
>> My guess would be something to do with memory (perhaps it's running
out and using swap space) and/or disk I/O. On what system are you
running this?
>>
>> Are you able to try running the same thing on a different system to
see what the runtime is?
>>
>> John
>>
>> On 08/01/2012 11:10 AM, jhenders at aer.com via RT wrote:
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>
>>> Hi John,
>>>
>>> I've loaded the relevant files to henderson_data. Note that
because of
>>> the offset (with respect to UTC dates) for ds337's files, I call
>>> point_stat with two daily ncdf files. This, of course, contributes
to
>>> the 60-70 minutes it requires for point-stat, but my question
simply is
>>> whether it is normal for such a non-linear scaling.
>>>
>>> Thanks.
>>>
>>> John
>>>
>>> On 8/1/12 10:47 AM, John Halley Gotway via RT wrote:
>>>> John,
>>>>
>>>> Uh oh, guess I'm out of tricks then!
>>>>
>>>> Would you be able to send me sample data for a run that's taking
60-70 minutes? I could try running it here to see if there's any
other improvements that might help. I'd need the sample forecast
>>>> file, NetCDF point observation file, and the Point-Stat Config
file.
>>>>
>>>> You can post data to our anonymous ftp site following these
directions:
>>>>
http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>> On 08/01/2012 08:30 AM, jhenders at aer.com via RT wrote:
>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>
>>>>> Hello John,
>>>>>
>>>>> It turns out that I already have both of those disabled. I
wanted to
>>>>> speed up my MET v2 runs a year ago, so at that time I set the
flags
>>>>> appropriately.
>>>>>
>>>>> John
>>>>>
>>>>> On 8/1/12 10:12 AM, John Halley Gotway via RT wrote:
>>>>>> John,
>>>>>>
>>>>>> There are two aspects of point_stat that don't scale very well
- the computation of rank correlation coefficients and bootstrap
confidence intervals. Both of them are controlled by settings in the
>>>>>> config file and disabling them yields significant increases in
runtime performance. By default, they are both *enabled* in Point-
Stat since that tool is typically run with a relatively small number
>>>>>> of observations. And by default, they are both *disabled* in
Grid-Stat since that tool is typically run with a relatively large
number of observations.
>>>>>>
>>>>>> I'd suggest turning both of them off and try rerunning to see
how much the runtime improves.
>>>>>>
>>>>>> Disable rank correlation computations by setting
"rank_corr_flag = FALSE;" in METv4.0 (or by setting "rank_corr_flag =
0;" in previous versions of MET).
>>>>>> Disable bootstrap confidence intervals by setting "n_rep = 0;"
in the "boot" section in METv4.0 (or by setting "n_boot_rep = 0;" in
previous versions of MET).
>>>>>>
>>>>>> Please let me know if/how much those changes improve the
runtime performance.
>>>>>>
>>>>>> Thanks,
>>>>>> John
>>>>>>
>>>>>> On 07/30/2012 01:25 PM, jhenders at aer.com via RT wrote:
>>>>>>> Mon Jul 30 13:25:36 2012: Request 57644 was acted upon.
>>>>>>> Transaction: Ticket created by jhenders at aer.com
>>>>>>> Queue: met_help
>>>>>>> Subject: Processing time for point_stat with NCAR
ds337 ncdf files
>>>>>>> Owner: Nobody
>>>>>>> Requestors: jhenders at aer.com
>>>>>>> Status: new
>>>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>>
>>>>>>>
>>>>>>> Hello again John,
>>>>>>>
>>>>>>> I was wondering if you could comment on the following behavior
of
>>>>>>> point_stat when running with PrepBufr files from the NCAR
ds337 dataset.
>>>>>>>
>>>>>>> When I run point_stat with daily input ncdf observation files
(generated
>>>>>>> by pb2nc) that cover a relatively small model domain (say, the
size of
>>>>>>> Southern New England), with a corresponding small number of
obs (several
>>>>>>> tens of thousand individual obs valid over the entire day),
point_stat
>>>>>>> completes successfully in less than 10 s. When I try to verify
on an
>>>>>>> outer domain that covers CONUS, with a corresponding large
number of obs
>>>>>>> (1-2 million), point_stat requires about 60-70 minutes to
complete.
>>>>>>> Thus, to verify hourly a full model run of 24 valid times,
over a day of
>>>>>>> wallclock time is required.
>>>>>>>
>>>>>>> I don't recall if this scaling behavior occurred for v2.0 of
MET.
>>>>>>>
>>>>>>> I welcome your thoughts.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> Respectfully,
>>>>>>>
>>>>>>> John Henderson
>>>>>>>
>>
>
------------------------------------------------
Subject: Processing time for point_stat with NCAR ds337 ncdf files
From: jhenders at aer.com
Time: Thu Aug 02 09:05:30 2012
Hi John,
Thanks for providing the command. Mine is very similar, as might be
expected...
I've attached my point_stat output... You can see that I have Nans,
plus
the expected NAs.
John
On 8/2/12 10:54 AM, John Halley Gotway via RT wrote:
> John,
>
> Sorry for the delay in getting back to you. I was out of the office
yesterday afternoon.
>
> Here's the exact command I ran with the data you sent:
>
> time \
> ${MET_BASE}/bin/point_stat \
> d1_2010121500_1200.grib \
> obs-20101215.nc \
> PointStatConfig_V3.1-p1652 \
> -point_obs obs-20101216.nc \
> -outdir out \
> -v 4
>
> And as I mentioned, it took about 24 seconds to execute on my
desktop.
>
> I've attached a tar file containing the ASCII output of this run.
I'm not seeing NaN's in the FBAR column (in
point_stat_120000L_20101215_120000V_cnt.txt). Instead I see
reasonable FBAR values ...
> followed by reasonable values in FBAR_NCL and FBAR_NCU for the
normal confidence limits ... followed by NA's in the FBAR_BCL and
FBAR_BCU since we disabled bootstrap CI's in the config file.
>
> Is it possible you were seeing the NA's in those columns rather than
the good values in the FBAR column?
>
> Thanks,
> John
>
> On 08/01/2012 02:58 PM, jhenders at aer.com via RT wrote:
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>
>> Hello again John,
>>
>> I reran my test case locally on a rather old linux cluster. This is
the
>> same one I used for MET v2 a year or so ago. This current job took
about
>> 5 minutes and seems to have completed successfully.
>>
>> I then reran on a newer cluster and it breezed through the reading
of
>> the observations in a matter of seconds. This is the step that took
a
>> very long time on Pleiades. However, close inspection of the output
>> files shows that computations involving the forecast (FBAR, e.g.)
are
>> all Nans. Physical values appear for all observation-related
quantities.
>>
>> I will keep looking...
>>
>> John
>>
>> On 8/1/12 1:32 PM, John Halley Gotway via RT wrote:
>>> John,
>>>
>>> Something funny is going on here. I ran the exact data you sent
me on my desktop machine, and it took 24 seconds to complete - not 60-
70 minutes.
>>>
>>> My guess would be something to do with memory (perhaps it's
running out and using swap space) and/or disk I/O. On what system are
you running this?
>>>
>>> Are you able to try running the same thing on a different system
to see what the runtime is?
>>>
>>> John
>>>
>>> On 08/01/2012 11:10 AM, jhenders at aer.com via RT wrote:
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>
>>>> Hi John,
>>>>
>>>> I've loaded the relevant files to henderson_data. Note that
because of
>>>> the offset (with respect to UTC dates) for ds337's files, I call
>>>> point_stat with two daily ncdf files. This, of course,
contributes to
>>>> the 60-70 minutes it requires for point-stat, but my question
simply is
>>>> whether it is normal for such a non-linear scaling.
>>>>
>>>> Thanks.
>>>>
>>>> John
>>>>
>>>> On 8/1/12 10:47 AM, John Halley Gotway via RT wrote:
>>>>> John,
>>>>>
>>>>> Uh oh, guess I'm out of tricks then!
>>>>>
>>>>> Would you be able to send me sample data for a run that's taking
60-70 minutes? I could try running it here to see if there's any
other improvements that might help. I'd need the sample forecast
>>>>> file, NetCDF point observation file, and the Point-Stat Config
file.
>>>>>
>>>>> You can post data to our anonymous ftp site following these
directions:
>>>>>
http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>>>
>>>>> Thanks,
>>>>> John
>>>>>
>>>>> On 08/01/2012 08:30 AM, jhenders at aer.com via RT wrote:
>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>
>>>>>> Hello John,
>>>>>>
>>>>>> It turns out that I already have both of those disabled. I
wanted to
>>>>>> speed up my MET v2 runs a year ago, so at that time I set the
flags
>>>>>> appropriately.
>>>>>>
>>>>>> John
>>>>>>
>>>>>> On 8/1/12 10:12 AM, John Halley Gotway via RT wrote:
>>>>>>> John,
>>>>>>>
>>>>>>> There are two aspects of point_stat that don't scale very well
- the computation of rank correlation coefficients and bootstrap
confidence intervals. Both of them are controlled by settings in the
>>>>>>> config file and disabling them yields significant increases in
runtime performance. By default, they are both *enabled* in Point-
Stat since that tool is typically run with a relatively small number
>>>>>>> of observations. And by default, they are both *disabled* in
Grid-Stat since that tool is typically run with a relatively large
number of observations.
>>>>>>>
>>>>>>> I'd suggest turning both of them off and try rerunning to see
how much the runtime improves.
>>>>>>>
>>>>>>> Disable rank correlation computations by setting
"rank_corr_flag = FALSE;" in METv4.0 (or by setting "rank_corr_flag =
0;" in previous versions of MET).
>>>>>>> Disable bootstrap confidence intervals by setting "n_rep = 0;"
in the "boot" section in METv4.0 (or by setting "n_boot_rep = 0;" in
previous versions of MET).
>>>>>>>
>>>>>>> Please let me know if/how much those changes improve the
runtime performance.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> John
>>>>>>>
>>>>>>> On 07/30/2012 01:25 PM, jhenders at aer.com via RT wrote:
>>>>>>>> Mon Jul 30 13:25:36 2012: Request 57644 was acted upon.
>>>>>>>> Transaction: Ticket created by jhenders at aer.com
>>>>>>>> Queue: met_help
>>>>>>>> Subject: Processing time for point_stat with
NCAR ds337 ncdf files
>>>>>>>> Owner: Nobody
>>>>>>>> Requestors: jhenders at aer.com
>>>>>>>> Status: new
>>>>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>>>
>>>>>>>>
>>>>>>>> Hello again John,
>>>>>>>>
>>>>>>>> I was wondering if you could comment on the following
behavior of
>>>>>>>> point_stat when running with PrepBufr files from the NCAR
ds337 dataset.
>>>>>>>>
>>>>>>>> When I run point_stat with daily input ncdf observation files
(generated
>>>>>>>> by pb2nc) that cover a relatively small model domain (say,
the size of
>>>>>>>> Southern New England), with a corresponding small number of
obs (several
>>>>>>>> tens of thousand individual obs valid over the entire day),
point_stat
>>>>>>>> completes successfully in less than 10 s. When I try to
verify on an
>>>>>>>> outer domain that covers CONUS, with a corresponding large
number of obs
>>>>>>>> (1-2 million), point_stat requires about 60-70 minutes to
complete.
>>>>>>>> Thus, to verify hourly a full model run of 24 valid times,
over a day of
>>>>>>>> wallclock time is required.
>>>>>>>>
>>>>>>>> I don't recall if this scaling behavior occurred for v2.0 of
MET.
>>>>>>>>
>>>>>>>> I welcome your thoughts.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> Respectfully,
>>>>>>>>
>>>>>>>> John Henderson
>>>>>>>>
------------------------------------------------
Subject: Processing time for point_stat with NCAR ds337 ncdf files
From: jhenders at aer.com
Time: Thu Aug 02 09:05:30 2012
VERSION MODEL FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD
OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR OBS_LEV
OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH
COV_THRESH ALPHA LINE_TYPE TOTAL FBAR FBAR_NCL FBAR_NCU FBAR_BCL
FBAR_BCU FSTDEV FSTDEV_NCL FSTDEV_NCU FSTDEV_BCL FSTDEV_BCU OBAR
OBAR_NCL OBAR_NCU OBAR_BCL OBAR_BCU OSTDEV OSTDEV_NCL
OSTDEV_NCU OSTDEV_BCL OSTDEV_BCU PR_CORR PR_CORR_NCL PR_CORR_NCU
PR_CORR_BCL PR_CORR_BCU SP_CORR KT_CORR RANKS FRANK_TIES ORANK_TIES ME
ME_NCL ME_NCU ME_BCL ME_BCU ESTDEV ESTDEV_NCL ESTDEV_NCU ESTDEV_BCL
ESTDEV_BCU MBIAS MBIAS_BCL MBIAS_BCU MAE MAE_BCL MAE_BCU MSE MSE_BCL
MSE_BCU BCMSE BCMSE_BCL BCMSE_BCU RMSE RMSE_BCL RMSE_BCU E10 E10_BCL
E10_BCU E25 E25_BCL E25_BCU E50 E50_BCL E50_BCU E75 E75_BCL E75_BCU
E90 E90_BCL E90_BCU
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 TMP P500 TMP P500
ADPUPA FULL BILIN 4 NA NA NA
0.05000 CNT 95 -nan -nan -nan NA NA -nan
-nan -nan NA NA 252.47211 250.74679
254.19742 NA NA 8.57989 7.50917 10.00984 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 TMP P700 TMP P700
ADPUPA FULL BILIN 4 NA NA NA
0.05000 CNT 94 -nan -nan -nan NA NA -nan
-nan -nan NA NA 267.83404 266.00301
269.66507 NA NA 9.05757 7.92196 10.57656 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 TMP P850 TMP P850
ADPUPA FULL BILIN 4 NA NA NA
0.05000 CNT 87 -nan -nan -nan NA NA -nan
-nan -nan NA NA 273.18908 270.98588
275.39228 NA NA 10.48491 9.12488 12.32554 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 HGT P500 HGT P500
ADPUPA FULL BILIN 4 NA NA NA
0.05000 CNT 94 -nan -nan -nan NA NA -nan
-nan -nan NA NA 5507.91489 5470.40445
5545.42534 NA NA 185.55303 162.28882 216.67094 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 HGT P700 HGT P700
ADPUPA FULL BILIN 4 NA NA NA
0.05000 CNT 94 -nan -nan -nan NA NA -nan
-nan -nan NA NA 2942.98936 2921.11683
2964.86189 NA NA 108.19690 94.63143 126.34191 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 HGT P850 HGT P850
ADPUPA FULL BILIN 4 NA NA NA
0.05000 CNT 95 -nan -nan -nan NA NA -nan
-nan -nan NA NA 1398.11579 1383.63297
1412.59861 NA NA 72.02229 63.03428 84.02570 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 RH P500 RH P500
ADPUPA FULL BILIN 4 NA NA NA
0.05000 CNT 91 -nan -nan -nan NA NA -nan
-nan -nan NA NA 37.12063 31.65244
42.58881 NA NA 26.61435 23.22955 31.16385 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 RH P700 RH P700
ADPUPA FULL BILIN 4 NA NA NA
0.05000 CNT 93 -nan -nan -nan NA NA -nan
-nan -nan NA NA 42.35325 36.07625
48.63026 NA NA 30.88487 26.99433 36.09711 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 RH P850 RH P850
ADPUPA FULL BILIN 4 NA NA NA
0.05000 CNT 86 -nan -nan -nan NA NA -nan
-nan -nan NA NA 55.66194 49.90139
61.42249 NA NA 27.25617 23.70273 32.07394 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 SPFH P500 SPFH P500
ADPUPA FULL BILIN 4 NA NA NA
0.05000 CNT 92 -nan -nan -nan NA NA -nan
-nan -nan NA NA 0.00055 0.00045
0.00065 NA NA 0.00051 0.00045 0.00060 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 SPFH P700 SPFH P700
ADPUPA FULL BILIN 4 NA NA NA
0.05000 CNT 94 -nan -nan -nan NA NA -nan
-nan -nan NA NA 0.00143 0.00120
0.00167 NA NA 0.00116 0.00102 0.00136 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 SPFH P850 SPFH P850
ADPUPA FULL BILIN 4 NA NA NA
0.05000 CNT 87 -nan -nan -nan NA NA -nan
-nan -nan NA NA 0.00264 0.00225
0.00304 NA NA 0.00188 0.00163 0.00220 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 WIND P500 WIND P500
ADPUPA FULL BILIN 4 NA NA NA
0.05000 CNT 87 -nan -nan -nan NA NA -nan
-nan -nan NA NA 23.60987 21.55141
25.66834 NA NA 9.79613 8.52544 11.51584 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 WIND P700 WIND P700
ADPUPA FULL BILIN 4 NA NA NA
0.05000 CNT 87 -nan -nan -nan NA NA -nan
-nan -nan NA NA 14.05597 12.70735
15.40459 NA NA 6.41802 5.58552 7.54471 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 WIND P850 WIND P850
ADPUPA FULL BILIN 4 NA NA NA
0.05000 CNT 78 -nan -nan -nan NA NA -nan
-nan -nan NA NA 11.80406 10.52604
13.08208 NA NA 5.75887 4.97536 6.83786 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 UGRD P500 UGRD P500
ADPUPA FULL BILIN 4 NA NA NA
0.05000 CNT 87 -nan -nan -nan NA NA -nan
-nan -nan NA NA 17.05402 14.14947
19.95858 NA NA 13.82265 12.02967 16.24922 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 VGRD P500 VGRD P500
ADPUPA FULL BILIN 4 NA NA NA
0.05000 CNT 87 -nan -nan -nan NA NA -nan
-nan -nan NA NA -3.95172 -6.59959
-1.30386 NA NA 12.60107 10.96654 14.81319 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 UGRD P700 UGRD P700
ADPUPA FULL BILIN 4 NA NA NA
0.05000 CNT 87 -nan -nan -nan NA NA -nan
-nan -nan NA NA 9.28851 7.53683
11.04018 NA NA 8.33613 7.25482 9.79954 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 VGRD P700 VGRD P700
ADPUPA FULL BILIN 4 NA NA NA
0.05000 CNT 87 -nan -nan -nan NA NA -nan
-nan -nan NA NA -2.63218 -4.47941
-0.78496 NA NA 8.79086 7.65057 10.33409 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 UGRD P850 UGRD P850
ADPUPA FULL BILIN 4 NA NA NA
0.05000 CNT 78 -nan -nan -nan NA NA -nan
-nan -nan NA NA 6.07821 4.41052
7.74589 NA NA 7.51471 6.49231 8.92267 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 VGRD P850 VGRD P850
ADPUPA FULL BILIN 4 NA NA NA
0.05000 CNT 78 -nan -nan -nan NA NA -nan
-nan -nan NA NA -0.88205 -2.86235
1.09824 NA NA 8.92338 7.70933 10.59528 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 TMP Z002 TMP Z002
ADPSFC FULL BILIN 4 NA NA NA
0.05000 CNT 3405 -nan -nan -nan NA NA -nan
-nan -nan NA NA 270.48880 270.14994
270.82765 NA NA 10.08842 9.85438 10.33392 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 TMP Z002 TMP Z002
SFCSHP FULL BILIN 4 NA NA NA
0.05000 CNT 518 -nan -nan -nan NA NA -nan
-nan -nan NA NA 275.39402 274.52817
276.25986 NA NA 10.05446 9.47721 10.70717 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 DPT Z002 DPT Z002
ADPSFC FULL BILIN 4 NA NA NA
0.05000 CNT 3129 -nan -nan -nan NA NA -nan
-nan -nan NA NA 265.52380 265.18005
265.86755 NA NA 9.81056 9.57338 10.05987 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 DPT Z002 DPT Z002
SFCSHP FULL BILIN 4 NA NA NA
0.05000 CNT 184 -nan -nan -nan NA NA -nan
-nan -nan NA NA 273.67676 272.28714
275.06639 NA NA 9.61740 8.72487 10.71501 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 SPFH Z002 SPFH Z002
ADPSFC FULL BILIN 4 NA NA NA
0.05000 CNT 3133 -nan -nan -nan NA NA -nan
-nan -nan NA NA 0.00286 0.00279
0.00294 NA NA 0.00228 0.00223 0.00234 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 SPFH Z002 SPFH Z002
SFCSHP FULL BILIN 4 NA NA NA
0.05000 CNT 185 -nan -nan -nan NA NA -nan
-nan -nan NA NA 0.00480 0.00436
0.00525 NA NA 0.00310 0.00281 0.00345 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 RH Z002 RH Z002
ADPSFC FULL BILIN 4 NA NA NA
0.05000 CNT 3129 -nan -nan -nan NA NA -nan
-nan -nan NA NA 73.03918 72.50012
73.57825 NA NA 15.38498 15.01303 15.77595 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 RH Z002 RH Z002
SFCSHP FULL BILIN 4 NA NA NA
0.05000 CNT 184 -nan -nan -nan NA NA -nan
-nan -nan NA NA 66.57703 64.48119
68.67287 NA NA 14.50505 13.15893 16.16048 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 WIND Z010 WIND Z010
ADPSFC FULL BILIN 4 NA NA NA
0.05000 CNT 3111 -nan -nan -nan NA NA -nan
-nan -nan NA NA 3.62413 3.52362
3.72464 NA NA 2.86041 2.79107 2.93332 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 WIND Z010 WIND Z010
SFCSHP FULL BILIN 4 NA NA NA
0.05000 CNT 472 -nan -nan -nan NA NA -nan
-nan -nan NA NA 5.57961 5.23941
5.91982 NA NA 3.77104 3.54484 4.02831 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 UGRD Z010 UGRD Z010
ADPSFC FULL BILIN 4 NA NA NA
0.05000 CNT 3111 -nan -nan -nan NA NA -nan
-nan -nan NA NA 0.01925 -0.11039
0.14890 NA NA 3.68934 3.59990 3.78338 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 UGRD Z010 UGRD Z010
SFCSHP FULL BILIN 4 NA NA NA
0.05000 CNT 472 -nan -nan -nan NA NA -nan
-nan -nan NA NA 1.73432 1.32543
2.14321 NA NA 4.53243 4.26055 4.84165 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 VGRD Z010 VGRD Z010
ADPSFC FULL BILIN 4 NA NA NA
0.05000 CNT 3111 -nan -nan -nan NA NA -nan
-nan -nan NA NA 0.67158 0.57692
0.76625 NA NA 2.69401 2.62870 2.76268 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
V3.1 WRF 120000 20101215_120000 20101215_120000 000000
20101215_114500 20101215_121500 VGRD Z010 VGRD Z010
SFCSHP FULL BILIN 4 NA NA NA
0.05000 CNT 472 -nan -nan -nan NA NA -nan
-nan -nan NA NA -0.84110 -1.25602
-0.42619 NA NA 4.59922 4.32334 4.91299 NA
NA -nan -nan -nan NA NA NA
NA 0 0 0 -nan -nan -nan NA NA
-nan -nan -nan NA NA -nan NA NA
nan NA NA -nan NA NA -nan NA NA
-nan NA NA -nan NA NA -nan NA NA -nan
NA NA -nan NA NA -nan NA NA
------------------------------------------------
Subject: Processing time for point_stat with NCAR ds337 ncdf files
From: John Halley Gotway
Time: Thu Aug 02 09:24:52 2012
John,
Unfortunately, unless I'm able to reproduce that problem with NaN's
here, I wont' be able to help much in the debugging.
Since it's just the forecast values and the observation values look
fine, I suspect that the problem could be in the MET library code that
reads GRIB1 data.
You could try running the following command to just make a plot of
some data (TMP at 500mb) from that file. If the plot looks good, the
problem is likely not in that GRIB1 library code. If the plot
looks bad (all NA's), it probably is:
${MET_BASE}/bin/plot_data_plane d1_2010121500_1200.grib
tmp_p500.ps 'name="TMP"; level="P500";'
I've attached the resulting image (after converting to a png)
generated on my machine, which looks pretty good.
John
On 08/02/2012 09:05 AM, jhenders at aer.com via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>
> Hi John,
>
> Thanks for providing the command. Mine is very similar, as might be
> expected...
>
> I've attached my point_stat output... You can see that I have Nans,
plus
> the expected NAs.
>
> John
>
> On 8/2/12 10:54 AM, John Halley Gotway via RT wrote:
>> John,
>>
>> Sorry for the delay in getting back to you. I was out of the
office yesterday afternoon.
>>
>> Here's the exact command I ran with the data you sent:
>>
>> time \
>> ${MET_BASE}/bin/point_stat \
>> d1_2010121500_1200.grib \
>> obs-20101215.nc \
>> PointStatConfig_V3.1-p1652 \
>> -point_obs obs-20101216.nc \
>> -outdir out \
>> -v 4
>>
>> And as I mentioned, it took about 24 seconds to execute on my
desktop.
>>
>> I've attached a tar file containing the ASCII output of this run.
I'm not seeing NaN's in the FBAR column (in
point_stat_120000L_20101215_120000V_cnt.txt). Instead I see
reasonable FBAR values ...
>> followed by reasonable values in FBAR_NCL and FBAR_NCU for the
normal confidence limits ... followed by NA's in the FBAR_BCL and
FBAR_BCU since we disabled bootstrap CI's in the config file.
>>
>> Is it possible you were seeing the NA's in those columns rather
than the good values in the FBAR column?
>>
>> Thanks,
>> John
>>
>> On 08/01/2012 02:58 PM, jhenders at aer.com via RT wrote:
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>
>>> Hello again John,
>>>
>>> I reran my test case locally on a rather old linux cluster. This
is the
>>> same one I used for MET v2 a year or so ago. This current job took
about
>>> 5 minutes and seems to have completed successfully.
>>>
>>> I then reran on a newer cluster and it breezed through the reading
of
>>> the observations in a matter of seconds. This is the step that
took a
>>> very long time on Pleiades. However, close inspection of the
output
>>> files shows that computations involving the forecast (FBAR, e.g.)
are
>>> all Nans. Physical values appear for all observation-related
quantities.
>>>
>>> I will keep looking...
>>>
>>> John
>>>
>>> On 8/1/12 1:32 PM, John Halley Gotway via RT wrote:
>>>> John,
>>>>
>>>> Something funny is going on here. I ran the exact data you sent
me on my desktop machine, and it took 24 seconds to complete - not 60-
70 minutes.
>>>>
>>>> My guess would be something to do with memory (perhaps it's
running out and using swap space) and/or disk I/O. On what system are
you running this?
>>>>
>>>> Are you able to try running the same thing on a different system
to see what the runtime is?
>>>>
>>>> John
>>>>
>>>> On 08/01/2012 11:10 AM, jhenders at aer.com via RT wrote:
>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>
>>>>> Hi John,
>>>>>
>>>>> I've loaded the relevant files to henderson_data. Note that
because of
>>>>> the offset (with respect to UTC dates) for ds337's files, I call
>>>>> point_stat with two daily ncdf files. This, of course,
contributes to
>>>>> the 60-70 minutes it requires for point-stat, but my question
simply is
>>>>> whether it is normal for such a non-linear scaling.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> John
>>>>>
>>>>> On 8/1/12 10:47 AM, John Halley Gotway via RT wrote:
>>>>>> John,
>>>>>>
>>>>>> Uh oh, guess I'm out of tricks then!
>>>>>>
>>>>>> Would you be able to send me sample data for a run that's
taking 60-70 minutes? I could try running it here to see if there's
any other improvements that might help. I'd need the sample forecast
>>>>>> file, NetCDF point observation file, and the Point-Stat Config
file.
>>>>>>
>>>>>> You can post data to our anonymous ftp site following these
directions:
>>>>>>
http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>>>>
>>>>>> Thanks,
>>>>>> John
>>>>>>
>>>>>> On 08/01/2012 08:30 AM, jhenders at aer.com via RT wrote:
>>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644
>
>>>>>>>
>>>>>>> Hello John,
>>>>>>>
>>>>>>> It turns out that I already have both of those disabled. I
wanted to
>>>>>>> speed up my MET v2 runs a year ago, so at that time I set the
flags
>>>>>>> appropriately.
>>>>>>>
>>>>>>> John
>>>>>>>
>>>>>>> On 8/1/12 10:12 AM, John Halley Gotway via RT wrote:
>>>>>>>> John,
>>>>>>>>
>>>>>>>> There are two aspects of point_stat that don't scale very
well - the computation of rank correlation coefficients and bootstrap
confidence intervals. Both of them are controlled by settings in the
>>>>>>>> config file and disabling them yields significant increases
in runtime performance. By default, they are both *enabled* in Point-
Stat since that tool is typically run with a relatively small number
>>>>>>>> of observations. And by default, they are both *disabled* in
Grid-Stat since that tool is typically run with a relatively large
number of observations.
>>>>>>>>
>>>>>>>> I'd suggest turning both of them off and try rerunning to see
how much the runtime improves.
>>>>>>>>
>>>>>>>> Disable rank correlation computations by setting
"rank_corr_flag = FALSE;" in METv4.0 (or by setting "rank_corr_flag =
0;" in previous versions of MET).
>>>>>>>> Disable bootstrap confidence intervals by setting "n_rep =
0;" in the "boot" section in METv4.0 (or by setting "n_boot_rep = 0;"
in previous versions of MET).
>>>>>>>>
>>>>>>>> Please let me know if/how much those changes improve the
runtime performance.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> John
>>>>>>>>
>>>>>>>> On 07/30/2012 01:25 PM, jhenders at aer.com via RT wrote:
>>>>>>>>> Mon Jul 30 13:25:36 2012: Request 57644 was acted upon.
>>>>>>>>> Transaction: Ticket created by jhenders at aer.com
>>>>>>>>> Queue: met_help
>>>>>>>>> Subject: Processing time for point_stat with
NCAR ds337 ncdf files
>>>>>>>>> Owner: Nobody
>>>>>>>>> Requestors: jhenders at aer.com
>>>>>>>>> Status: new
>>>>>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hello again John,
>>>>>>>>>
>>>>>>>>> I was wondering if you could comment on the following
behavior of
>>>>>>>>> point_stat when running with PrepBufr files from the NCAR
ds337 dataset.
>>>>>>>>>
>>>>>>>>> When I run point_stat with daily input ncdf observation
files (generated
>>>>>>>>> by pb2nc) that cover a relatively small model domain (say,
the size of
>>>>>>>>> Southern New England), with a corresponding small number of
obs (several
>>>>>>>>> tens of thousand individual obs valid over the entire day),
point_stat
>>>>>>>>> completes successfully in less than 10 s. When I try to
verify on an
>>>>>>>>> outer domain that covers CONUS, with a corresponding large
number of obs
>>>>>>>>> (1-2 million), point_stat requires about 60-70 minutes to
complete.
>>>>>>>>> Thus, to verify hourly a full model run of 24 valid times,
over a day of
>>>>>>>>> wallclock time is required.
>>>>>>>>>
>>>>>>>>> I don't recall if this scaling behavior occurred for v2.0 of
MET.
>>>>>>>>>
>>>>>>>>> I welcome your thoughts.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> Respectfully,
>>>>>>>>>
>>>>>>>>> John Henderson
>>>>>>>>>
>
>
------------------------------------------------
Subject: Processing time for point_stat with NCAR ds337 ncdf files
From: jhenders at aer.com
Time: Thu Aug 02 10:15:05 2012
John,
I seemed only to get the plot_data_plane call to work when I specified
the magic name like this: 'TMP/P500'. I trust that this is the proper
syntax. Then, however, the code said 'ColorTable::interp(double) const
-> confused!', which indicates to me that the grib reading is hosed.
I then closely inspected the outputs from the NCAR-provided test
scripts
that are to be run at installation. I've attached the CNT file, which
shows that the forecast values are bad. I don't see any errors in the
make.out log file and I've never experienced grib1 libraries
exhibiting
such flaky behavior.
John
On 8/2/12 11:24 AM, John Halley Gotway via RT wrote:
> John,
>
> Unfortunately, unless I'm able to reproduce that problem with NaN's
here, I wont' be able to help much in the debugging.
>
> Since it's just the forecast values and the observation values look
fine, I suspect that the problem could be in the MET library code that
reads GRIB1 data.
>
> You could try running the following command to just make a plot of
some data (TMP at 500mb) from that file. If the plot looks good, the
problem is likely not in that GRIB1 library code. If the plot
> looks bad (all NA's), it probably is:
>
> ${MET_BASE}/bin/plot_data_plane d1_2010121500_1200.grib
tmp_p500.ps 'name="TMP"; level="P500";'
>
> I've attached the resulting image (after converting to a png)
generated on my machine, which looks pretty good.
>
> John
>
> On 08/02/2012 09:05 AM, jhenders at aer.com via RT wrote:
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>
>> Hi John,
>>
>> Thanks for providing the command. Mine is very similar, as might be
>> expected...
>>
>> I've attached my point_stat output... You can see that I have Nans,
plus
>> the expected NAs.
>>
>> John
>>
>> On 8/2/12 10:54 AM, John Halley Gotway via RT wrote:
>>> John,
>>>
>>> Sorry for the delay in getting back to you. I was out of the
office yesterday afternoon.
>>>
>>> Here's the exact command I ran with the data you sent:
>>>
>>> time \
>>> ${MET_BASE}/bin/point_stat \
>>> d1_2010121500_1200.grib \
>>> obs-20101215.nc \
>>> PointStatConfig_V3.1-p1652 \
>>> -point_obs obs-20101216.nc \
>>> -outdir out \
>>> -v 4
>>>
>>> And as I mentioned, it took about 24 seconds to execute on my
desktop.
>>>
>>> I've attached a tar file containing the ASCII output of this run.
I'm not seeing NaN's in the FBAR column (in
point_stat_120000L_20101215_120000V_cnt.txt). Instead I see
reasonable FBAR values ...
>>> followed by reasonable values in FBAR_NCL and FBAR_NCU for the
normal confidence limits ... followed by NA's in the FBAR_BCL and
FBAR_BCU since we disabled bootstrap CI's in the config file.
>>>
>>> Is it possible you were seeing the NA's in those columns rather
than the good values in the FBAR column?
>>>
>>> Thanks,
>>> John
>>>
>>> On 08/01/2012 02:58 PM, jhenders at aer.com via RT wrote:
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>
>>>> Hello again John,
>>>>
>>>> I reran my test case locally on a rather old linux cluster. This
is the
>>>> same one I used for MET v2 a year or so ago. This current job
took about
>>>> 5 minutes and seems to have completed successfully.
>>>>
>>>> I then reran on a newer cluster and it breezed through the
reading of
>>>> the observations in a matter of seconds. This is the step that
took a
>>>> very long time on Pleiades. However, close inspection of the
output
>>>> files shows that computations involving the forecast (FBAR, e.g.)
are
>>>> all Nans. Physical values appear for all observation-related
quantities.
>>>>
>>>> I will keep looking...
>>>>
>>>> John
>>>>
>>>> On 8/1/12 1:32 PM, John Halley Gotway via RT wrote:
>>>>> John,
>>>>>
>>>>> Something funny is going on here. I ran the exact data you sent
me on my desktop machine, and it took 24 seconds to complete - not 60-
70 minutes.
>>>>>
>>>>> My guess would be something to do with memory (perhaps it's
running out and using swap space) and/or disk I/O. On what system are
you running this?
>>>>>
>>>>> Are you able to try running the same thing on a different system
to see what the runtime is?
>>>>>
>>>>> John
>>>>>
>>>>> On 08/01/2012 11:10 AM, jhenders at aer.com via RT wrote:
>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>
>>>>>> Hi John,
>>>>>>
>>>>>> I've loaded the relevant files to henderson_data. Note that
because of
>>>>>> the offset (with respect to UTC dates) for ds337's files, I
call
>>>>>> point_stat with two daily ncdf files. This, of course,
contributes to
>>>>>> the 60-70 minutes it requires for point-stat, but my question
simply is
>>>>>> whether it is normal for such a non-linear scaling.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> John
>>>>>>
>>>>>> On 8/1/12 10:47 AM, John Halley Gotway via RT wrote:
>>>>>>> John,
>>>>>>>
>>>>>>> Uh oh, guess I'm out of tricks then!
>>>>>>>
>>>>>>> Would you be able to send me sample data for a run that's
taking 60-70 minutes? I could try running it here to see if there's
any other improvements that might help. I'd need the sample forecast
>>>>>>> file, NetCDF point observation file, and the Point-Stat Config
file.
>>>>>>>
>>>>>>> You can post data to our anonymous ftp site following these
directions:
>>>>>>>
http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>>>>>
>>>>>>> Thanks,
>>>>>>> John
>>>>>>>
>>>>>>> On 08/01/2012 08:30 AM, jhenders at aer.com via RT wrote:
>>>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644
>
>>>>>>>>
>>>>>>>> Hello John,
>>>>>>>>
>>>>>>>> It turns out that I already have both of those disabled. I
wanted to
>>>>>>>> speed up my MET v2 runs a year ago, so at that time I set the
flags
>>>>>>>> appropriately.
>>>>>>>>
>>>>>>>> John
>>>>>>>>
>>>>>>>> On 8/1/12 10:12 AM, John Halley Gotway via RT wrote:
>>>>>>>>> John,
>>>>>>>>>
>>>>>>>>> There are two aspects of point_stat that don't scale very
well - the computation of rank correlation coefficients and bootstrap
confidence intervals. Both of them are controlled by settings in the
>>>>>>>>> config file and disabling them yields significant increases
in runtime performance. By default, they are both *enabled* in Point-
Stat since that tool is typically run with a relatively small number
>>>>>>>>> of observations. And by default, they are both *disabled*
in Grid-Stat since that tool is typically run with a relatively large
number of observations.
>>>>>>>>>
>>>>>>>>> I'd suggest turning both of them off and try rerunning to
see how much the runtime improves.
>>>>>>>>>
>>>>>>>>> Disable rank correlation computations by setting
"rank_corr_flag = FALSE;" in METv4.0 (or by setting "rank_corr_flag =
0;" in previous versions of MET).
>>>>>>>>> Disable bootstrap confidence intervals by setting "n_rep =
0;" in the "boot" section in METv4.0 (or by setting "n_boot_rep = 0;"
in previous versions of MET).
>>>>>>>>>
>>>>>>>>> Please let me know if/how much those changes improve the
runtime performance.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> John
>>>>>>>>>
>>>>>>>>> On 07/30/2012 01:25 PM, jhenders at aer.com via RT wrote:
>>>>>>>>>> Mon Jul 30 13:25:36 2012: Request 57644 was acted upon.
>>>>>>>>>> Transaction: Ticket created by jhenders at aer.com
>>>>>>>>>> Queue: met_help
>>>>>>>>>> Subject: Processing time for point_stat with
NCAR ds337 ncdf files
>>>>>>>>>> Owner: Nobody
>>>>>>>>>> Requestors: jhenders at aer.com
>>>>>>>>>> Status: new
>>>>>>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hello again John,
>>>>>>>>>>
>>>>>>>>>> I was wondering if you could comment on the following
behavior of
>>>>>>>>>> point_stat when running with PrepBufr files from the NCAR
ds337 dataset.
>>>>>>>>>>
>>>>>>>>>> When I run point_stat with daily input ncdf observation
files (generated
>>>>>>>>>> by pb2nc) that cover a relatively small model domain (say,
the size of
>>>>>>>>>> Southern New England), with a corresponding small number of
obs (several
>>>>>>>>>> tens of thousand individual obs valid over the entire day),
point_stat
>>>>>>>>>> completes successfully in less than 10 s. When I try to
verify on an
>>>>>>>>>> outer domain that covers CONUS, with a corresponding large
number of obs
>>>>>>>>>> (1-2 million), point_stat requires about 60-70 minutes to
complete.
>>>>>>>>>> Thus, to verify hourly a full model run of 24 valid times,
over a day of
>>>>>>>>>> wallclock time is required.
>>>>>>>>>>
>>>>>>>>>> I don't recall if this scaling behavior occurred for v2.0
of MET.
>>>>>>>>>>
>>>>>>>>>> I welcome your thoughts.
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>> Respectfully,
>>>>>>>>>>
>>>>>>>>>> John Henderson
>>>>>>>>>>
>>
>
------------------------------------------------
Subject: Processing time for point_stat with NCAR ds337 ncdf files
From: jhenders at aer.com
Time: Thu Aug 02 10:15:05 2012
VERSION MODEL FCST_LEAD FCST_VALID_BEG FCST_VALID_END OBS_LEAD
OBS_VALID_BEG OBS_VALID_END FCST_VAR FCST_LEV OBS_VAR OBS_LEV
OBTYPE VX_MASK INTERP_MTHD INTERP_PNTS FCST_THRESH OBS_THRESH
COV_THRESH ALPHA LINE_TYPE TOTAL FBAR FBAR_NCL FBAR_NCU FBAR_BCL
FBAR_BCU FSTDEV FSTDEV_NCL FSTDEV_NCU FSTDEV_BCL FSTDEV_BCU OBAR
OBAR_NCL OBAR_NCU OBAR_BCL OBAR_BCU OSTDEV OSTDEV_NCL OSTDEV_NCU
OSTDEV_BCL OSTDEV_BCU PR_CORR PR_CORR_NCL PR_CORR_NCU PR_CORR_BCL
PR_CORR_BCU SP_CORR KT_CORR RANKS FRANK_TIES ORANK_TIES ME
ME_NCL ME_NCU ME_BCL ME_BCU ESTDEV ESTDEV_NCL
ESTDEV_NCU ESTDEV_BCL ESTDEV_BCU MBIAS MBIAS_BCL MBIAS_BCU MAE
MAE_BCL MAE_BCU MSE MSE_BCL MSE_BCU BCMSE
BCMSE_BCL BCMSE_BCU RMSE RMSE_BCL RMSE_BCU E10 E10_BCL
E10_BCU E25 E25_BCL E25_BCU E50 E50_BCL
E50_BCU E75 E75_BCL E75_BCU E90 E90_BCL
E90_BCU
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 TMP P900-750 TMP P900-750
ADPUPA DTC165 UW_MEAN 1 NA NA NA
0.05000 CNT 155 24.49594 23.53027 25.46160 23.55037 25.40368
6.13400 5.51872 6.90498 5.30749 6.84249 275.57000
274.62036 276.51964 274.60289 276.50344 6.03220 5.42713 6.79039
5.40013 6.59530 0.77269 0.70036 0.82932 0.68190
0.84115 0.67858 0.51198 155 26 47 -251.07406
-251.71994 -250.42819 -251.66736 -250.43320 4.10266 3.69114 4.61832
3.55398 4.62609 0.08889 0.08574 0.09193 251.07406
250.43320 251.66736 63054.90827 62732.32422 63353.40175 16.72323
12.54929 21.26267 251.10736 250.46422 251.70102 -255.59196
-256.75382 -254.65378 -253.66579 -254.24528 -252.85992 -251.18712
-251.55508 -250.80765 -248.35990 -249.15379 -247.37084 -246.37562
-247.05378 -245.69069
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 TMP P900-750 TMP P900-750
ADPUPA DTC165 MEDIAN 9 NA NA NA
0.05000 CNT 155 25.08716 24.19435 25.97998 24.18144 26.03999
5.67126 5.10239 6.38408 4.82642 6.39446 275.57000
274.62036 276.51964 274.56661 276.55365 6.03220 5.42713 6.79039
5.44587 6.56097 0.86106 0.81389 0.89695 0.80745
0.90207 0.80715 0.63284 155 25 47 -250.48284
-250.97155 -249.99413 -250.93939 -249.98270 3.10434 2.79295 3.49452
2.71150 3.50688 0.09104 0.08801 0.09417 250.48284
249.98270 250.93939 62751.22739 62502.98584 62979.38942 9.57473
7.30481 12.21884 250.50195 250.00597 250.95695 -254.41384
-254.89891 -253.49504 -252.58995 -253.12584 -252.15382 -251.07047
-251.52880 -250.30835 -248.49328 -249.00380 -247.94575 -246.66741
-247.60208 -246.09174
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 TMP P900-750 TMP P900-750
ADPUPA DTC165 DW_MEAN 9 NA NA NA
0.05000 CNT 155 24.40996 23.45373 25.36619 23.45684 25.42489
6.07405 5.46478 6.83750 5.29826 6.85268 275.57000
274.62036 276.51964 274.63276 276.54508 6.03220 5.42713 6.79039
5.45301 6.60857 0.83505 0.78010 0.87722 0.76904
0.88417 0.78122 0.59677 155 0 47 -251.16004
-251.70741 -250.61267 -251.69355 -250.57514 3.47693 3.12817 3.91394
3.03242 3.89845 0.08858 0.08534 0.09203 251.16004
250.57514 251.69355 63093.37645 62796.85264 63359.59893 12.01105
9.13624 15.09985 251.18395 250.59300 251.71333 -255.71261
-256.47923 -254.52760 -253.62229 -254.23910 -252.87670 -251.36133
-252.11927 -250.24047 -248.95109 -249.30793 -248.22487 -247.24014
-247.80125 -246.86921
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 TMP P900-750 TMP P900-750
ADPUPA DTC166 UW_MEAN 1 NA NA NA
0.05000 CNT 334 28.80964 27.99666 29.62261 27.97257 29.65292
7.58062 7.04601 8.20371 6.82217 8.26158 279.23892
278.41172 280.06612 278.40165 280.13937 7.71320 7.16924 8.34719
6.90481 8.47148 0.90594 0.88463 0.92348 0.87765
0.92790 0.88217 0.70135 334 50 152 -250.42929
-250.78525 -250.07332 -250.76541 -250.07836 3.31917 3.08509 3.59199
3.02369 3.63327 0.10317 0.10047 0.10582 250.42929
250.07836 250.76541 62725.81145 62549.15312 62893.91620 10.98388
9.11530 13.16115 250.45122 250.09829 250.78660 -254.34094
-254.59239 -253.97131 -252.62137 -253.25115 -251.95385 -250.56572
-250.95385 -250.17882 -248.33407 -248.69167 -247.85380 -246.51382
-247.05382 -245.75383
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 TMP P900-750 TMP P900-750
ADPUPA DTC166 MEDIAN 9 NA NA NA
0.05000 CNT 334 28.44933 27.67112 29.22753 27.73959 29.26169
7.25634 6.74459 7.85278 6.54812 7.87305 279.23892
278.41172 280.06612 278.48420 280.09763 7.71320 7.16924 8.34719
6.91764 8.40002 0.94906 0.93719 0.95873 0.93428
0.96038 0.92276 0.76215 334 42 152 -250.78960
-251.05034 -250.52885 -251.05805 -250.54363 2.43134 2.25987 2.63119
2.23171 2.60907 0.10188 0.09960 0.10448 250.78960
250.54363 251.05805 62901.31553 62778.38676 63035.82910 5.89372
4.96560 6.78687 250.80135 250.55615 251.06937 -253.99155
-254.13250 -253.72954 -252.47535 -252.80067 -252.25152 -250.83672
-251.25380 -250.56893 -249.36719 -249.67151 -248.97168 -247.58785
-248.24621 -246.86405
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 TMP P900-750 TMP P900-750
ADPUPA DTC166 DW_MEAN 9 NA NA NA
0.05000 CNT 334 28.66424 27.85829 29.47018 27.82547 29.45214
7.51505 6.98506 8.13276 6.76744 8.20934 279.23892
278.41172 280.06612 278.42058 280.03633 7.71320 7.16924 8.34719
6.94060 8.44246 0.94312 0.92993 0.95389 0.92583
0.95600 0.92374 0.76351 334 6 152 -250.57469
-250.85089 -250.29849 -250.86276 -250.30376 2.57542 2.39379 2.78711
2.35902 2.78254 0.10265 0.09990 0.10516 250.57469
250.30376 250.86276 62794.28627 62657.97437 62938.78423 6.61295
5.54829 7.71933 250.58788 250.31575 250.87603 -253.77631
-254.10428 -253.31837 -252.26585 -252.48217 -252.09715 -250.59898
-250.82329 -250.14634 -248.99999 -249.36488 -248.56298 -247.51789
-248.09794 -246.87894
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 TMP P900-750 TMP P900-750
ADPUPA LMV UW_MEAN 1 NA NA NA
0.05000 CNT 13 33.05428 29.88759 36.22097 30.37809 35.70681
5.24032 3.75776 8.65037 3.04424 6.64470 283.58846
281.55642 285.62051 281.94981 285.25000 3.36267 2.41132 5.55088
2.29031 3.99574 0.87736 0.58838 0.96760 0.74121
0.95947 0.85756 0.73710 13 3 0 -250.53419
-252.22704 -248.84133 -251.96648 -249.01884 2.80138 2.00883 4.62434
1.53141 3.68945 0.11656 0.10763 0.12523 250.53419
249.01884 251.96648 62774.62225 62013.49712 63496.52128 7.24408
2.16483 12.56498 250.54864 249.02509 251.98516 -253.54235
-256.70324 -250.95264 -251.33116 -253.61449 -250.22578 -250.43646
-251.33116 -248.25383 -248.25383 -250.43646 -247.76047 -247.85363
-250.22578 -246.35384
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 TMP P900-750 TMP P900-750
ADPUPA LMV MEDIAN 9 NA NA NA
0.05000 CNT 13 32.94339 31.19583 34.69095 31.55037 34.51465
2.89190 2.07374 4.77376 1.60615 3.70127 283.58846
281.55642 285.62051 281.96520 285.35000 3.36267 2.41132 5.55088
2.32616 3.93605 0.91819 0.71057 0.97873 0.84125
0.97554 0.93372 0.79754 13 2 0 -250.64508
-251.45871 -249.83145 -251.38609 -249.96119 1.34642 0.96550 2.22258
0.88660 1.67247 0.11617 0.11188 0.12111 250.64508
249.96119 251.38609 62824.62754 62481.85391 63197.03953 1.67339
0.72559 2.58200 250.64841 249.96371 251.39021 -252.17384
-253.25383 -251.24558 -251.35384 -252.25383 -249.90259 -250.88476
-251.35384 -249.41252 -249.41252 -250.88648 -249.16897 -249.18177
-249.87345 -248.93032
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 TMP P900-750 TMP P900-750
ADPUPA LMV DW_MEAN 9 NA NA NA
0.05000 CNT 13 33.41331 30.65799 36.16863 30.92692 35.83043
4.55957 3.26960 7.52664 2.72389 5.67196 283.58846
281.55642 285.62051 281.90366 285.45038 3.36267 2.41132 5.55088
2.33685 3.89586 0.92805 0.74210 0.98136 0.87406
0.97353 0.92857 0.79487 13 0 0 -250.17515
-251.32790 -249.02240 -251.22101 -249.17797 1.90760 1.36792 3.14895
0.96610 2.63969 0.11782 0.10972 0.12555 250.17515
249.17797 251.22101 62590.96422 62091.01809 63115.61055 3.35903
0.86156 6.43198 250.18186 249.18069 251.22820 -251.70070
-254.96865 -250.30765 -250.89881 -251.77753 -250.03183 -250.08037
-250.89881 -249.07091 -249.07091 -250.08037 -248.26465 -248.26967
-249.56393 -247.51493
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 TMP P900-750 TMP P900-750
ADPSFC DTC165 UW_MEAN 1 NA NA NA
0.05000 CNT 312 26.43761 25.94891 26.92631 25.93685 26.96411
4.40424 4.08364 4.77990 4.00920 4.77830 273.45929
272.98129 273.93729 272.97882 273.95837 4.30781 3.99423 4.67525
3.90608 4.70509 0.59646 0.51985 0.66355 0.48767
0.68923 0.56454 0.41534 312 64 213 -247.02168
-247.45602 -246.58735 -247.47392 -246.61733 3.91430 3.62936 4.24817
3.48444 4.25996 0.09668 0.09496 0.09850 247.02168
246.61733 247.47392 61034.98495 60832.68530 61257.86232 15.27261
12.10240 18.08907 247.05260 246.64283 247.50326 -252.26014
-253.30570 -251.24495 -248.56321 -249.41195 -248.12371 -246.97713
-247.21376 -246.63906 -244.66666 -245.51558 -244.26467 -242.65382
-243.27907 -242.01181
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 TMP P900-750 TMP P900-750
ADPSFC DTC165 MEDIAN 9 NA NA NA
0.05000 CNT 312 26.80708 26.33912 27.27505 26.34251 27.25497
4.21739 3.91039 4.57712 3.81328 4.62485 273.45929
272.98129 273.93729 272.97074 273.93659 4.30781 3.99423 4.67525
3.86526 4.72526 0.72211 0.66434 0.77131 0.64767
0.78655 0.68846 0.50758 312 64 213 -246.65221
-247.00494 -246.29948 -247.01345 -246.29374 3.17889 2.94749 3.45003
2.87316 3.46333 0.09803 0.09645 0.09952 246.65221
246.29374 247.01345 60847.38577 60671.60010 61026.26323 10.07294
8.22857 11.95623 246.67263 246.31606 247.03494 -250.72307
-251.33333 -250.15642 -248.37122 -248.88679 -247.95172 -246.75383
-247.08740 -246.50085 -245.10330 -245.55379 -244.64953 -242.88629
-243.44053 -241.92981
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 TMP P900-750 TMP P900-750
ADPSFC DTC165 DW_MEAN 9 NA NA NA
0.05000 CNT 312 26.44878 25.99278 26.90479 26.03158 26.90958
4.10960 3.81045 4.46013 3.72057 4.50883 273.45929
272.98129 273.93729 273.02396 273.91675 4.30781 3.99423 4.67525
3.88435 4.74322 0.69081 0.62793 0.74472 0.60274
0.75910 0.66261 0.48490 312 63 213 -247.01051
-247.37831 -246.64271 -247.39839 -246.64772 3.31464 3.07335 3.59736
2.99931 3.61231 0.09672 0.09529 0.09828 247.01051
246.64772 247.39839 61025.14392 60846.27460 61217.11151 10.95161
8.96704 13.00699 247.03268 246.67038 247.42092 -251.11657
-251.52352 -250.42343 -249.15522 -249.73058 -248.54139 -246.95039
-247.43904 -246.74280 -245.31382 -245.85381 -244.74543 -242.77569
-243.50606 -241.88011
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 TMP P900-750 TMP P900-750
ADPSFC DTC166 UW_MEAN 1 NA NA NA
0.05000 CNT 8 31.75638 26.85339 36.65938 27.77793 35.54262
5.86469 3.87758 11.93623 2.78717 7.57607 281.52500
275.72965 287.32035 277.01157 286.27500 6.93206 4.58330 14.10864
4.33460 7.44017 0.93760 0.57860 0.99226 0.88064
0.99994 0.90699 0.80064 8 3 2 -249.76862
-251.85243 -247.68481 -251.26793 -248.07505 2.49253 1.64800 5.07298
1.41644 2.83578 0.11280 0.10044 0.12415 249.76862
248.07505 251.26793 62389.79794 61544.79447 63135.57502 5.43613
1.76684 7.03644 249.77950 248.08223 251.26793 -251.76233
-252.55380 -251.23898 -251.30081 -252.55380 -248.19674 -251.21005
-251.42313 -247.17565 -247.17565 -251.26004 -246.14058 -246.86513
-250.06285 -246.14058
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 TMP P900-750 TMP P900-750
ADPSFC DTC166 MEDIAN 9 NA NA NA
0.05000 CNT 8 31.36583 27.02957 35.70208 27.84269 34.49034
5.18678 3.42936 10.55650 2.81106 6.83996 281.52500
275.72965 287.32035 276.71250 286.07718 6.93206 4.58330 14.10864
4.35711 7.48877 0.93144 0.54529 0.99147 0.84916
0.99998 0.90699 0.80064 8 3 2 -250.15917
-252.52028 -247.79806 -251.83465 -248.23101 2.82422 1.86730 5.74806
1.71526 3.20722 0.11141 0.10045 0.12047 250.15917
248.23101 251.83465 62586.59062 61623.41335 63423.92459 6.97921
2.57449 9.00049 250.17312 248.24064 251.84107 -252.69129
-252.69129 -251.55380 -252.61630 -252.69129 -248.27018 -251.40381
-252.64130 -247.17565 -247.17565 -251.55380 -246.14058 -246.86513
-250.25075 -246.14058
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 TMP P900-750 TMP P900-750
ADPSFC DTC166 DW_MEAN 9 NA NA NA
0.05000 CNT 8 31.08418 26.43958 35.72878 27.37103 34.41402
5.55561 3.67322 11.30717 3.34353 6.91706 281.52500
275.72965 287.32035 276.96251 286.00155 6.93206 4.58330 14.10864
4.43671 7.40728 0.95727 0.69355 0.99475 0.90987
0.99994 0.90699 0.80064 8 3 2 -250.44082
-252.34459 -248.53705 -251.77861 -248.97246 2.27718 1.50561 4.63468
1.35183 2.60089 0.11041 0.09852 0.12066 250.44082
248.97246 251.77861 62725.14257 61990.63437 63394.38933 4.53736
1.59915 5.91904 250.44988 248.97919 251.78242 -252.45132
-252.45132 -251.55060 -252.37634 -252.45132 -249.16454 -251.48757
-252.45132 -248.10678 -248.10678 -251.81056 -247.08389 -247.79991
-250.77111 -247.08389
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 UGRD Z10 UGRD Z10
ADPSFC DTC165 UW_MEAN 1 NA NA NA
0.05000 CNT 934 12.54977 12.02964 13.06991 12.01902 13.05374
8.11036 7.75851 8.49590 7.93143 8.27141 0.97002 0.79613
1.14391 0.80512 1.14022 2.71145 2.59382 2.84034 2.40389
3.07385 0.14271 0.07928 0.20498 0.07935 0.19621
0.15210 0.10484 934 414 824 11.57975 11.05538
12.10412 11.01917 12.10041 8.17640 7.82168 8.56508 7.97923
8.36277 12.93762 10.92349 15.41020 11.69915 11.16106
12.21517 200.87258 187.89199 213.57659 66.78195 63.59993
69.86111 14.17295 13.70737 14.61426 1.31396 0.94275 1.58167
3.84763 3.46264 4.47627 12.48782 8.84457 14.45905
18.91756 18.57599 19.44043 21.54975 21.28398 21.85907
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 UGRD Z10 UGRD Z10
ADPSFC DTC165 MEDIAN 9 NA NA NA
0.05000 CNT 934 12.82725 12.38168 13.27282 12.37676 13.29502
6.94769 6.64628 7.27796 6.78754 7.08900 0.97002 0.79613
1.14391 0.80222 1.14337 2.71145 2.59382 2.84034 2.40172
3.05972 0.12676 0.06312 0.18936 0.06450 0.17985
0.15194 0.10549 934 424 824 11.85723 11.39993
12.31453 11.41424 12.29944 7.13068 6.82133 7.46965 6.96390
7.29564 13.22368 11.21727 15.97761 11.89222 11.45060
12.33101 191.38609 180.56262 202.44920 50.79218 48.44395
53.16932 13.83424 13.43736 14.22846 2.63936 2.27782 2.97494
5.27161 4.93282 5.55817 13.20962 10.62520 14.69317
18.43281 18.07515 18.73960 20.39357 20.10954 20.81161
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 UGRD Z10 UGRD Z10
ADPSFC DTC165 DW_MEAN 9 NA NA NA
0.05000 CNT 934 12.70276 12.31734 13.08819 12.33749 13.12834
6.00987 5.74914 6.29556 5.82193 6.18911 0.97002 0.79613
1.14391 0.80748 1.14309 2.71145 2.59382 2.84034 2.38031
3.05836 0.16073 0.09759 0.22258 0.09458 0.21848
0.12741 0.08799 934 316 824 11.73274 11.33620
12.12928 11.35116 12.18631 6.18321 5.91496 6.47714 5.98603
6.36764 13.09534 11.13117 15.82979 11.76060 11.38113
12.20507 175.84840 166.47733 185.87153 38.19120 35.79420
40.50349 13.26078 12.90261 13.63347 3.29884 2.95950 3.81944
6.79032 6.05236 7.18086 12.03406 11.22365 12.93637
16.80863 16.48210 17.42946 19.60820 19.25262 19.87991
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 UGRD Z10 UGRD Z10
ADPSFC DTC166 UW_MEAN 1 NA NA NA
0.05000 CNT 2955 13.66345 13.35484 13.97206 13.35299 13.96286
8.55926 8.34648 8.78325 8.46498 8.65592 -1.01039 -1.11899
-0.90179 -1.11649 -0.89611 3.01202 2.93714 3.09084 2.91690
3.10697 0.12366 0.08800 0.15901 0.08666 0.15887
0.22728 0.16132 2955 1575 2806 14.67384 14.35960
14.98807 14.36785 14.98533 8.71533 8.49867 8.94341 8.60017
8.83202 -13.52296 -15.29237 -12.22181 14.74441 14.42971
15.05040 291.25286 281.57232 300.56341 75.93133 73.93781
77.97826 17.06613 16.78012 17.33676 4.28044 4.09562 4.45795
6.46275 6.26542 6.72380 15.69471 12.54244 17.25351
22.70308 22.50572 22.92079 25.02422 24.83370 25.30152
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 UGRD Z10 UGRD Z10
ADPSFC DTC166 MEDIAN 9 NA NA NA
0.05000 CNT 2955 13.57321 13.29373 13.85268 13.28303 13.84679
7.75122 7.55853 7.95407 7.66814 7.83309 -1.01039 -1.11899
-0.90179 -1.12907 -0.90194 3.01202 2.93714 3.09084 2.91251
3.09992 0.13190 0.09630 0.16716 0.09798 0.16881
0.27667 0.19886 2955 1599 2806 14.58360 14.29743
14.86977 14.30742 14.87687 7.93692 7.73961 8.14463 7.81810
8.03207 -13.43364 -15.09261 -11.97927 14.63223 14.35481
14.91840 275.65473 267.21067 283.92265 62.97344 61.10208
64.49232 16.60285 16.34658 16.85000 5.21102 4.99379 5.49984
7.25808 7.08392 7.42780 13.54629 12.29788 16.28796
22.35112 22.20870 22.48326 24.24164 23.99122 24.40023
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 UGRD Z10 UGRD Z10
ADPSFC DTC166 DW_MEAN 9 NA NA NA
0.05000 CNT 2955 13.64950 13.38845 13.91056 13.39472 13.88682
7.24037 7.06038 7.42985 7.14581 7.33448 -1.01039 -1.11899
-0.90179 -1.11453 -0.89632 3.01202 2.93714 3.09084 2.91942
3.10439 0.14727 0.11181 0.18236 0.11433 0.18210
0.20421 0.14216 2955 1462 2806 14.65989 14.39233
14.92746 14.38446 14.91352 7.42104 7.23655 7.61525 7.30489
7.53083 -13.50916 -15.27774 -12.15509 14.70260 14.42998
14.95055 269.96565 261.78574 277.55264 55.05319 53.34342
56.69414 16.43063 16.17979 16.65991 5.54912 5.35607 5.76215
7.71488 7.52554 7.94852 14.59774 13.59703 15.29271
21.61228 21.34423 21.92420 24.13504 23.91611 24.35432
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 UGRD Z10 UGRD Z10
ADPSFC LMV UW_MEAN 1 NA NA NA
0.05000 CNT 364 14.71994 13.85173 15.58814 13.86228 15.53519
8.45133 7.87874 9.11438 8.20709 8.67826 0.50687 0.21111
0.80263 0.19010 0.80165 2.87899 2.68393 3.10486 2.66438
3.07142 -0.13605 -0.23555 -0.03373 -0.23655 -0.03410
-0.09473 -0.06644 364 196 270 14.21307 13.25854
15.16760 13.25870 15.13028 9.29161 8.66209 10.02058 8.91454
9.64373 29.04096 18.42838 77.34725 14.45927 13.55643
15.34057 288.10818 260.92875 312.65166 86.09683 79.25074
92.74612 16.97375 16.15329 17.68196 2.67798 1.05212 3.81956
5.33331 4.59837 6.11707 17.28069 14.82844 19.08386
22.43178 21.98438 23.08460 24.54608 23.85522 25.11170
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 UGRD Z10 UGRD Z10
ADPSFC LMV MEDIAN 9 NA NA NA
0.05000 CNT 364 14.26807 13.51784 15.01829 13.51330 15.01613
7.30286 6.80807 7.87580 7.14817 7.42178 0.50687 0.21111
0.80263 0.23148 0.79483 2.87899 2.68393 3.10486 2.66304
3.07423 -0.13543 -0.23495 -0.03310 -0.23550 -0.03329
-0.06423 -0.03844 364 201 270 13.76120 12.91834
14.60405 12.93265 14.56159 8.20457 7.64869 8.84825 7.88871
8.47805 28.14947 17.88949 63.24427 13.84069 13.01102
14.62513 256.50057 234.77241 278.60042 67.12997 62.06076
71.67994 16.01564 15.32229 16.69133 3.17145 2.45605 4.19294
6.46356 5.95714 6.98924 15.70413 12.12409 16.85901
22.10275 21.66557 22.34363 23.21300 22.87624 23.80853
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 UGRD Z10 UGRD Z10
ADPSFC LMV DW_MEAN 9 NA NA NA
0.05000 CNT 364 14.60408 13.87168 15.33649 13.83565 15.33823
7.12939 6.64636 7.68873 6.90402 7.33297 0.50687 0.21111
0.80263 0.22487 0.83170 2.87899 2.68393 3.10486 2.66634
3.07632 -0.16010 -0.25863 -0.05826 -0.25692 -0.05749
-0.16777 -0.10560 364 193 270 14.09722 13.26460
14.92983 13.22398 14.89841 8.10487 7.55575 8.74073 7.73911
8.43599 28.81239 17.36843 63.65506 14.20411 13.34317
14.98571 264.23985 239.52240 287.04129 65.50838 59.72928
70.97037 16.25546 15.47651 16.94229 4.08922 3.42600 4.73392
6.99182 5.93017 7.35494 15.11459 11.44697 16.48564
21.84520 21.20963 22.40348 23.94191 23.47815 24.31761
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 VGRD Z10 VGRD Z10
ADPSFC DTC165 UW_MEAN 1 NA NA NA
0.05000 CNT 934 16.13358 15.62210 16.64506 15.63114 16.65094
7.97543 7.62943 8.35456 7.81307 8.13711 -0.04604 -0.20332
0.11124 -0.20399 0.11703 2.45249 2.34609 2.56907 2.29435
2.61748 0.04816 -0.01604 0.11196 -0.01509 0.11164
0.07273 0.05006 934 414 816 16.17962 15.65179
16.70744 15.66430 16.70975 8.23032 7.87326 8.62156 8.02370
8.42519 -350.43630 -3143.74635 2146.65376 16.19599 15.67963
16.72885 329.44565 312.90849 346.43958 67.66566 64.31082
70.90780 18.15064 17.68922 18.61289 5.67525 5.34680 6.12576
8.61117 8.27218 9.13238 17.69632 13.46519 19.05747
23.76030 23.23512 24.17777 26.25252 25.85985 26.68968
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 VGRD Z10 VGRD Z10
ADPSFC DTC165 MEDIAN 9 NA NA NA
0.05000 CNT 934 16.17807 15.74118 16.61496 15.73988 16.59536
6.81236 6.51681 7.13619 6.69055 6.93607 -0.04604 -0.20332
0.11124 -0.19598 0.10373 2.45249 2.34609 2.56907 2.29928
2.59729 0.08331 0.01926 0.14667 0.01855 0.14901
0.10950 0.07772 934 424 816 16.22411 15.77226
16.67595 15.76302 16.63929 7.04552 6.73986 7.38044 6.86546
7.21852 -351.40268 -2855.60155 2365.32940 16.22411 15.76302
16.63929 312.80781 298.02871 326.40361 49.58615 47.08412
52.05126 17.68637 17.26351 18.06664 7.12513 6.76099 7.57988
9.27982 8.96041 9.67241 18.38246 16.14192 19.25825
22.59505 22.13053 22.88947 24.14179 23.93591 24.69283
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 VGRD Z10 VGRD Z10
ADPSFC DTC165 DW_MEAN 9 NA NA NA
0.05000 CNT 934 15.96358 15.58639 16.34077 15.56709 16.32694
5.88149 5.62633 6.16108 5.71655 6.04652 -0.04604 -0.20332
0.11124 -0.21296 0.10391 2.45249 2.34609 2.56907 2.30408
2.60754 0.04863 -0.01556 0.11243 -0.01473 0.11609
0.02918 0.01982 934 316 816 16.00962 15.60807
16.41117 15.60932 16.40410 6.26129 5.98965 6.55893 6.04398
6.46072 -346.74375 -2862.12649 1960.08057 16.00962 15.60932
16.40410 295.46956 282.70102 308.38625 39.16175 36.49063
41.69619 17.18923 16.81372 17.56093 7.85046 7.37799 8.22776
10.55603 10.18341 11.03427 16.25956 15.37213 17.14736
21.22704 20.90715 21.75727 24.17617 23.80143 24.40827
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 VGRD Z10 VGRD Z10
ADPSFC DTC166 UW_MEAN 1 NA NA NA
0.05000 CNT 2955 13.61009 13.29544 13.92474 13.32158 13.92390
8.72686 8.50992 8.95524 8.59905 8.84002 -0.03557 -0.13819
0.06706 -0.14308 0.06796 2.84632 2.77556 2.92080 2.74564
2.94363 0.21330 0.17862 0.24746 0.17666 0.24607
0.26132 0.18527 2955 1575 2803 13.64566 13.33621
13.95511 13.36703 13.95659 8.58272 8.36936 8.80733 8.46274
8.70299 -382.66248 -4155.22673 2600.09596 13.70651 13.42930
14.01066 259.84228 251.99692 268.73009 73.63823 71.59370
75.71633 16.11962 15.87441 16.39299 2.84588 2.60245 3.17456
5.71603 5.46328 6.03126 15.71978 12.80772 16.57105
21.06528 20.91580 21.29076 24.01217 23.61241 24.26102
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 VGRD Z10 VGRD Z10
ADPSFC DTC166 MEDIAN 9 NA NA NA
0.05000 CNT 2955 13.45531 13.17588 13.73473 13.16854 13.75058
7.74987 7.55721 7.95268 7.65650 7.83740 -0.03557 -0.13819
0.06706 -0.14048 0.07342 2.84632 2.77556 2.92080 2.74193
2.94208 0.23833 0.20403 0.27205 0.20677 0.26948
0.31004 0.22290 2955 1608 2803 13.49087 13.21712
13.76463 13.21914 13.77983 7.59259 7.40384 7.79129 7.50160
7.68817 -378.31049 -3158.27808 3281.24914 13.49949 13.22712
13.78890 239.63158 232.31091 247.66515 57.62790 56.25491
59.08800 15.48004 15.24175 15.73738 4.29235 4.09092 4.43235
6.20893 6.01998 6.37564 13.89951 11.62476 15.68314
20.59402 20.42120 20.81842 22.58313 22.32771 22.81499
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 VGRD Z10 VGRD Z10
ADPSFC DTC166 DW_MEAN 9 NA NA NA
0.05000 CNT 2955 13.57728 13.31314 13.84142 13.32688 13.82071
7.32597 7.14384 7.51769 7.22741 7.42673 -0.03557 -0.13819
0.06706 -0.14339 0.06072 2.84632 2.77556 2.92080 2.74936
2.94082 0.24438 0.21018 0.27799 0.20691 0.27465
0.27351 0.19286 2955 1462 2803 13.61284 13.35390
13.87179 13.35103 13.86102 7.18189 7.00335 7.36984 7.07294
7.29160 -381.73983 -3190.96663 4574.39122 13.62771 13.36617
13.87770 236.87162 229.76610 243.42575 51.56208 50.00956
53.14938 15.39063 15.15804 15.60211 4.51083 4.34513 4.72654
6.85764 6.64793 7.08726 13.97556 13.23332 14.73277
20.08818 19.88051 20.21805 22.33641 22.09251 22.57411
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 VGRD Z10 VGRD Z10
ADPSFC LMV UW_MEAN 1 NA NA NA
0.05000 CNT 364 16.71701 15.87360 17.56042 15.88131 17.55913
8.20994 7.65370 8.85405 7.94459 8.43890 2.33956 2.08857
2.59055 2.07031 2.58874 2.44319 2.27766 2.63487 2.25711
2.62234 0.10270 -0.00010 0.20334 -0.00061 0.20133
0.08695 0.05113 364 196 277 14.37745 13.52255
15.23235 13.54494 15.19099 8.32180 7.75798 8.97469 7.99429
8.61424 7.14536 6.42847 8.11028 14.40244 13.56049
15.22183 275.77313 251.20233 300.55114 69.06209 63.73315
74.00121 16.60642 15.84936 17.33641 3.86571 3.67405 4.43871
6.61295 5.65918 7.36989 15.28521 11.39675 18.68190
21.43505 20.96834 22.02937 24.71332 24.05384 25.49357
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 VGRD Z10 VGRD Z10
ADPSFC LMV MEDIAN 9 NA NA NA
0.05000 CNT 364 15.74453 14.99653 16.49253 15.03552 16.50714
7.28121 6.78790 7.85246 7.09039 7.45660 2.33956 2.08857
2.59055 2.07905 2.59919 2.44319 2.27766 2.63487 2.26077
2.62795 0.12606 0.02357 0.22592 0.02089 0.23027
0.09651 0.06485 364 204 277 13.40497 12.64657
14.16336 12.68253 14.21643 7.38243 6.88225 7.96162 7.07958
7.66477 6.72969 6.01018 7.56187 13.40497 12.68253
14.21643 234.04363 212.77108 256.89222 54.35051 49.98275
58.58734 15.29848 14.58667 16.02786 4.58904 4.35152 4.77151
6.25578 5.49401 7.21058 11.84034 10.45301 14.68576
20.43220 19.85339 20.84928 22.23905 21.48889 23.47307
V3.1 WRF 360000 20070331_120000 20070331_120000 000000
20070331_103000 20070331_133000 VGRD Z10 VGRD Z10
ADPSFC LMV DW_MEAN 9 NA NA NA
0.05000 CNT 364 16.58198 15.88963 17.27433 15.92894 17.25808
6.73949 6.28287 7.26823 6.44288 6.99009 2.33956 2.08857
2.59055 2.07497 2.60193 2.44319 2.27766 2.63487 2.24883
2.61409 0.10892 0.00620 0.20937 0.01146 0.20732
0.10785 0.07503 364 193 277 14.24242 13.53215
14.95269 13.56073 14.95023 6.91395 6.44552 7.45639 6.58257
7.20412 7.08765 6.35265 7.96279 14.24562 13.56401
14.95299 250.51795 230.75720 270.38501 47.67143 43.21125
51.75676 15.82776 15.19069 16.44339 4.86282 4.49573 5.70418
7.35009 6.73136 8.56723 15.24293 13.18712 16.83007
20.36643 19.59761 20.94083 22.29647 21.73797 22.95054
------------------------------------------------
Subject: Processing time for point_stat with NCAR ds337 ncdf files
From: John Halley Gotway
Time: Thu Aug 02 10:42:52 2012
John,
Ah yes, my apologies. I sent you the syntax for METv4.0 - instead of
METv3.1, that you're using. The switch for METv4.0 is due to the
updates to the config file language.
Yes, it appears the problem is in the reading of data from GRIB1
files. I've never seen that "confused" error message before, but I
was able to see it by passing a NaN to that
"ColorTable::interp(double) const" function. So it is reading the
GRIB1 data as all NaN's.
My suspicion is that the sizes/alignments of things on your system
differ from any system on which we're run/tested MET in the past.
I've attached a small test program to this email that I'm hoping
you can compile and run, and then send us the output.
You can compile it like this:
g++ test_program.cc -o test_program
Thanks,
John
On 08/02/2012 10:15 AM, jhenders at aer.com via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>
> John,
>
> I seemed only to get the plot_data_plane call to work when I
specified
> the magic name like this: 'TMP/P500'. I trust that this is the
proper
> syntax. Then, however, the code said 'ColorTable::interp(double)
const
> -> confused!', which indicates to me that the grib reading is hosed.
>
> I then closely inspected the outputs from the NCAR-provided test
scripts
> that are to be run at installation. I've attached the CNT file,
which
> shows that the forecast values are bad. I don't see any errors in
the
> make.out log file and I've never experienced grib1 libraries
exhibiting
> such flaky behavior.
>
> John
>
>
>
> On 8/2/12 11:24 AM, John Halley Gotway via RT wrote:
>> John,
>>
>> Unfortunately, unless I'm able to reproduce that problem with NaN's
here, I wont' be able to help much in the debugging.
>>
>> Since it's just the forecast values and the observation values look
fine, I suspect that the problem could be in the MET library code that
reads GRIB1 data.
>>
>> You could try running the following command to just make a plot of
some data (TMP at 500mb) from that file. If the plot looks good, the
problem is likely not in that GRIB1 library code. If the plot
>> looks bad (all NA's), it probably is:
>>
>> ${MET_BASE}/bin/plot_data_plane d1_2010121500_1200.grib
tmp_p500.ps 'name="TMP"; level="P500";'
>>
>> I've attached the resulting image (after converting to a png)
generated on my machine, which looks pretty good.
>>
>> John
>>
>> On 08/02/2012 09:05 AM, jhenders at aer.com via RT wrote:
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>
>>> Hi John,
>>>
>>> Thanks for providing the command. Mine is very similar, as might
be
>>> expected...
>>>
>>> I've attached my point_stat output... You can see that I have
Nans, plus
>>> the expected NAs.
>>>
>>> John
>>>
>>> On 8/2/12 10:54 AM, John Halley Gotway via RT wrote:
>>>> John,
>>>>
>>>> Sorry for the delay in getting back to you. I was out of the
office yesterday afternoon.
>>>>
>>>> Here's the exact command I ran with the data you sent:
>>>>
>>>> time \
>>>> ${MET_BASE}/bin/point_stat \
>>>> d1_2010121500_1200.grib \
>>>> obs-20101215.nc \
>>>> PointStatConfig_V3.1-p1652 \
>>>> -point_obs obs-20101216.nc \
>>>> -outdir out \
>>>> -v 4
>>>>
>>>> And as I mentioned, it took about 24 seconds to execute on my
desktop.
>>>>
>>>> I've attached a tar file containing the ASCII output of this run.
I'm not seeing NaN's in the FBAR column (in
point_stat_120000L_20101215_120000V_cnt.txt). Instead I see
reasonable FBAR values ...
>>>> followed by reasonable values in FBAR_NCL and FBAR_NCU for the
normal confidence limits ... followed by NA's in the FBAR_BCL and
FBAR_BCU since we disabled bootstrap CI's in the config file.
>>>>
>>>> Is it possible you were seeing the NA's in those columns rather
than the good values in the FBAR column?
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>> On 08/01/2012 02:58 PM, jhenders at aer.com via RT wrote:
>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>
>>>>> Hello again John,
>>>>>
>>>>> I reran my test case locally on a rather old linux cluster. This
is the
>>>>> same one I used for MET v2 a year or so ago. This current job
took about
>>>>> 5 minutes and seems to have completed successfully.
>>>>>
>>>>> I then reran on a newer cluster and it breezed through the
reading of
>>>>> the observations in a matter of seconds. This is the step that
took a
>>>>> very long time on Pleiades. However, close inspection of the
output
>>>>> files shows that computations involving the forecast (FBAR,
e.g.) are
>>>>> all Nans. Physical values appear for all observation-related
quantities.
>>>>>
>>>>> I will keep looking...
>>>>>
>>>>> John
>>>>>
>>>>> On 8/1/12 1:32 PM, John Halley Gotway via RT wrote:
>>>>>> John,
>>>>>>
>>>>>> Something funny is going on here. I ran the exact data you
sent me on my desktop machine, and it took 24 seconds to complete -
not 60-70 minutes.
>>>>>>
>>>>>> My guess would be something to do with memory (perhaps it's
running out and using swap space) and/or disk I/O. On what system are
you running this?
>>>>>>
>>>>>> Are you able to try running the same thing on a different
system to see what the runtime is?
>>>>>>
>>>>>> John
>>>>>>
>>>>>> On 08/01/2012 11:10 AM, jhenders at aer.com via RT wrote:
>>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644
>
>>>>>>>
>>>>>>> Hi John,
>>>>>>>
>>>>>>> I've loaded the relevant files to henderson_data. Note that
because of
>>>>>>> the offset (with respect to UTC dates) for ds337's files, I
call
>>>>>>> point_stat with two daily ncdf files. This, of course,
contributes to
>>>>>>> the 60-70 minutes it requires for point-stat, but my question
simply is
>>>>>>> whether it is normal for such a non-linear scaling.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> John
>>>>>>>
>>>>>>> On 8/1/12 10:47 AM, John Halley Gotway via RT wrote:
>>>>>>>> John,
>>>>>>>>
>>>>>>>> Uh oh, guess I'm out of tricks then!
>>>>>>>>
>>>>>>>> Would you be able to send me sample data for a run that's
taking 60-70 minutes? I could try running it here to see if there's
any other improvements that might help. I'd need the sample forecast
>>>>>>>> file, NetCDF point observation file, and the Point-Stat
Config file.
>>>>>>>>
>>>>>>>> You can post data to our anonymous ftp site following these
directions:
>>>>>>>>
http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> John
>>>>>>>>
>>>>>>>> On 08/01/2012 08:30 AM, jhenders at aer.com via RT wrote:
>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>>>>
>>>>>>>>> Hello John,
>>>>>>>>>
>>>>>>>>> It turns out that I already have both of those disabled. I
wanted to
>>>>>>>>> speed up my MET v2 runs a year ago, so at that time I set
the flags
>>>>>>>>> appropriately.
>>>>>>>>>
>>>>>>>>> John
>>>>>>>>>
>>>>>>>>> On 8/1/12 10:12 AM, John Halley Gotway via RT wrote:
>>>>>>>>>> John,
>>>>>>>>>>
>>>>>>>>>> There are two aspects of point_stat that don't scale very
well - the computation of rank correlation coefficients and bootstrap
confidence intervals. Both of them are controlled by settings in the
>>>>>>>>>> config file and disabling them yields significant increases
in runtime performance. By default, they are both *enabled* in Point-
Stat since that tool is typically run with a relatively small number
>>>>>>>>>> of observations. And by default, they are both *disabled*
in Grid-Stat since that tool is typically run with a relatively large
number of observations.
>>>>>>>>>>
>>>>>>>>>> I'd suggest turning both of them off and try rerunning to
see how much the runtime improves.
>>>>>>>>>>
>>>>>>>>>> Disable rank correlation computations by setting
"rank_corr_flag = FALSE;" in METv4.0 (or by setting "rank_corr_flag =
0;" in previous versions of MET).
>>>>>>>>>> Disable bootstrap confidence intervals by setting "n_rep =
0;" in the "boot" section in METv4.0 (or by setting "n_boot_rep = 0;"
in previous versions of MET).
>>>>>>>>>>
>>>>>>>>>> Please let me know if/how much those changes improve the
runtime performance.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>> On 07/30/2012 01:25 PM, jhenders at aer.com via RT wrote:
>>>>>>>>>>> Mon Jul 30 13:25:36 2012: Request 57644 was acted upon.
>>>>>>>>>>> Transaction: Ticket created by jhenders at aer.com
>>>>>>>>>>> Queue: met_help
>>>>>>>>>>> Subject: Processing time for point_stat
with NCAR ds337 ncdf files
>>>>>>>>>>> Owner: Nobody
>>>>>>>>>>> Requestors: jhenders at aer.com
>>>>>>>>>>> Status: new
>>>>>>>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hello again John,
>>>>>>>>>>>
>>>>>>>>>>> I was wondering if you could comment on the following
behavior of
>>>>>>>>>>> point_stat when running with PrepBufr files from the NCAR
ds337 dataset.
>>>>>>>>>>>
>>>>>>>>>>> When I run point_stat with daily input ncdf observation
files (generated
>>>>>>>>>>> by pb2nc) that cover a relatively small model domain (say,
the size of
>>>>>>>>>>> Southern New England), with a corresponding small number
of obs (several
>>>>>>>>>>> tens of thousand individual obs valid over the entire
day), point_stat
>>>>>>>>>>> completes successfully in less than 10 s. When I try to
verify on an
>>>>>>>>>>> outer domain that covers CONUS, with a corresponding large
number of obs
>>>>>>>>>>> (1-2 million), point_stat requires about 60-70 minutes to
complete.
>>>>>>>>>>> Thus, to verify hourly a full model run of 24 valid times,
over a day of
>>>>>>>>>>> wallclock time is required.
>>>>>>>>>>>
>>>>>>>>>>> I don't recall if this scaling behavior occurred for v2.0
of MET.
>>>>>>>>>>>
>>>>>>>>>>> I welcome your thoughts.
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>> Respectfully,
>>>>>>>>>>>
>>>>>>>>>>> John Henderson
>>>>>>>>>>>
>>>
>>
>
>
------------------------------------------------
Subject: Processing time for point_stat with NCAR ds337 ncdf files
From: jhenders at aer.com
Time: Thu Aug 02 10:49:07 2012
John,
Here's the output (attached). I agree that likely the automated
configure/compile-time code is making some bad assumptions as to what
we
have on our machine.
I hope this helps. I suppose the hands-off approach to compiling
(without getting my hands dirty under the hood) can have its
drawbacks...
John
On 8/2/12 12:42 PM, John Halley Gotway via RT wrote:
> John,
>
> Ah yes, my apologies. I sent you the syntax for METv4.0 - instead
of METv3.1, that you're using. The switch for METv4.0 is due to the
updates to the config file language.
>
> Yes, it appears the problem is in the reading of data from GRIB1
files. I've never seen that "confused" error message before, but I
was able to see it by passing a NaN to that
> "ColorTable::interp(double) const" function. So it is reading the
GRIB1 data as all NaN's.
>
> My suspicion is that the sizes/alignments of things on your system
differ from any system on which we're run/tested MET in the past.
I've attached a small test program to this email that I'm hoping
> you can compile and run, and then send us the output.
>
> You can compile it like this:
> g++ test_program.cc -o test_program
>
> Thanks,
> John
>
> On 08/02/2012 10:15 AM, jhenders at aer.com via RT wrote:
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>
>> John,
>>
>> I seemed only to get the plot_data_plane call to work when I
specified
>> the magic name like this: 'TMP/P500'. I trust that this is the
proper
>> syntax. Then, however, the code said 'ColorTable::interp(double)
const
>> -> confused!', which indicates to me that the grib reading is
hosed.
>>
>> I then closely inspected the outputs from the NCAR-provided test
scripts
>> that are to be run at installation. I've attached the CNT file,
which
>> shows that the forecast values are bad. I don't see any errors in
the
>> make.out log file and I've never experienced grib1 libraries
exhibiting
>> such flaky behavior.
>>
>> John
>>
>>
>>
>> On 8/2/12 11:24 AM, John Halley Gotway via RT wrote:
>>> John,
>>>
>>> Unfortunately, unless I'm able to reproduce that problem with
NaN's here, I wont' be able to help much in the debugging.
>>>
>>> Since it's just the forecast values and the observation values
look fine, I suspect that the problem could be in the MET library code
that reads GRIB1 data.
>>>
>>> You could try running the following command to just make a plot of
some data (TMP at 500mb) from that file. If the plot looks good, the
problem is likely not in that GRIB1 library code. If the plot
>>> looks bad (all NA's), it probably is:
>>>
>>> ${MET_BASE}/bin/plot_data_plane d1_2010121500_1200.grib
tmp_p500.ps 'name="TMP"; level="P500";'
>>>
>>> I've attached the resulting image (after converting to a png)
generated on my machine, which looks pretty good.
>>>
>>> John
>>>
>>> On 08/02/2012 09:05 AM, jhenders at aer.com via RT wrote:
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>
>>>> Hi John,
>>>>
>>>> Thanks for providing the command. Mine is very similar, as might
be
>>>> expected...
>>>>
>>>> I've attached my point_stat output... You can see that I have
Nans, plus
>>>> the expected NAs.
>>>>
>>>> John
>>>>
>>>> On 8/2/12 10:54 AM, John Halley Gotway via RT wrote:
>>>>> John,
>>>>>
>>>>> Sorry for the delay in getting back to you. I was out of the
office yesterday afternoon.
>>>>>
>>>>> Here's the exact command I ran with the data you sent:
>>>>>
>>>>> time \
>>>>> ${MET_BASE}/bin/point_stat \
>>>>> d1_2010121500_1200.grib \
>>>>> obs-20101215.nc \
>>>>> PointStatConfig_V3.1-p1652 \
>>>>> -point_obs obs-20101216.nc \
>>>>> -outdir out \
>>>>> -v 4
>>>>>
>>>>> And as I mentioned, it took about 24 seconds to execute on my
desktop.
>>>>>
>>>>> I've attached a tar file containing the ASCII output of this
run. I'm not seeing NaN's in the FBAR column (in
point_stat_120000L_20101215_120000V_cnt.txt). Instead I see
reasonable FBAR values ...
>>>>> followed by reasonable values in FBAR_NCL and FBAR_NCU for the
normal confidence limits ... followed by NA's in the FBAR_BCL and
FBAR_BCU since we disabled bootstrap CI's in the config file.
>>>>>
>>>>> Is it possible you were seeing the NA's in those columns rather
than the good values in the FBAR column?
>>>>>
>>>>> Thanks,
>>>>> John
>>>>>
>>>>> On 08/01/2012 02:58 PM, jhenders at aer.com via RT wrote:
>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>
>>>>>> Hello again John,
>>>>>>
>>>>>> I reran my test case locally on a rather old linux cluster.
This is the
>>>>>> same one I used for MET v2 a year or so ago. This current job
took about
>>>>>> 5 minutes and seems to have completed successfully.
>>>>>>
>>>>>> I then reran on a newer cluster and it breezed through the
reading of
>>>>>> the observations in a matter of seconds. This is the step that
took a
>>>>>> very long time on Pleiades. However, close inspection of the
output
>>>>>> files shows that computations involving the forecast (FBAR,
e.g.) are
>>>>>> all Nans. Physical values appear for all observation-related
quantities.
>>>>>>
>>>>>> I will keep looking...
>>>>>>
>>>>>> John
>>>>>>
>>>>>> On 8/1/12 1:32 PM, John Halley Gotway via RT wrote:
>>>>>>> John,
>>>>>>>
>>>>>>> Something funny is going on here. I ran the exact data you
sent me on my desktop machine, and it took 24 seconds to complete -
not 60-70 minutes.
>>>>>>>
>>>>>>> My guess would be something to do with memory (perhaps it's
running out and using swap space) and/or disk I/O. On what system are
you running this?
>>>>>>>
>>>>>>> Are you able to try running the same thing on a different
system to see what the runtime is?
>>>>>>>
>>>>>>> John
>>>>>>>
>>>>>>> On 08/01/2012 11:10 AM, jhenders at aer.com via RT wrote:
>>>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644
>
>>>>>>>>
>>>>>>>> Hi John,
>>>>>>>>
>>>>>>>> I've loaded the relevant files to henderson_data. Note that
because of
>>>>>>>> the offset (with respect to UTC dates) for ds337's files, I
call
>>>>>>>> point_stat with two daily ncdf files. This, of course,
contributes to
>>>>>>>> the 60-70 minutes it requires for point-stat, but my question
simply is
>>>>>>>> whether it is normal for such a non-linear scaling.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> John
>>>>>>>>
>>>>>>>> On 8/1/12 10:47 AM, John Halley Gotway via RT wrote:
>>>>>>>>> John,
>>>>>>>>>
>>>>>>>>> Uh oh, guess I'm out of tricks then!
>>>>>>>>>
>>>>>>>>> Would you be able to send me sample data for a run that's
taking 60-70 minutes? I could try running it here to see if there's
any other improvements that might help. I'd need the sample forecast
>>>>>>>>> file, NetCDF point observation file, and the Point-Stat
Config file.
>>>>>>>>>
>>>>>>>>> You can post data to our anonymous ftp site following these
directions:
>>>>>>>>>
http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> John
>>>>>>>>>
>>>>>>>>> On 08/01/2012 08:30 AM, jhenders at aer.com via RT wrote:
>>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>>>>>
>>>>>>>>>> Hello John,
>>>>>>>>>>
>>>>>>>>>> It turns out that I already have both of those disabled. I
wanted to
>>>>>>>>>> speed up my MET v2 runs a year ago, so at that time I set
the flags
>>>>>>>>>> appropriately.
>>>>>>>>>>
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>> On 8/1/12 10:12 AM, John Halley Gotway via RT wrote:
>>>>>>>>>>> John,
>>>>>>>>>>>
>>>>>>>>>>> There are two aspects of point_stat that don't scale very
well - the computation of rank correlation coefficients and bootstrap
confidence intervals. Both of them are controlled by settings in the
>>>>>>>>>>> config file and disabling them yields significant
increases in runtime performance. By default, they are both *enabled*
in Point-Stat since that tool is typically run with a relatively small
number
>>>>>>>>>>> of observations. And by default, they are both *disabled*
in Grid-Stat since that tool is typically run with a relatively large
number of observations.
>>>>>>>>>>>
>>>>>>>>>>> I'd suggest turning both of them off and try rerunning to
see how much the runtime improves.
>>>>>>>>>>>
>>>>>>>>>>> Disable rank correlation computations by setting
"rank_corr_flag = FALSE;" in METv4.0 (or by setting "rank_corr_flag =
0;" in previous versions of MET).
>>>>>>>>>>> Disable bootstrap confidence intervals by setting "n_rep =
0;" in the "boot" section in METv4.0 (or by setting "n_boot_rep = 0;"
in previous versions of MET).
>>>>>>>>>>>
>>>>>>>>>>> Please let me know if/how much those changes improve the
runtime performance.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> John
>>>>>>>>>>>
>>>>>>>>>>> On 07/30/2012 01:25 PM, jhenders at aer.com via RT wrote:
>>>>>>>>>>>> Mon Jul 30 13:25:36 2012: Request 57644 was acted upon.
>>>>>>>>>>>> Transaction: Ticket created by jhenders at aer.com
>>>>>>>>>>>> Queue: met_help
>>>>>>>>>>>> Subject: Processing time for point_stat
with NCAR ds337 ncdf files
>>>>>>>>>>>> Owner: Nobody
>>>>>>>>>>>> Requestors: jhenders at aer.com
>>>>>>>>>>>> Status: new
>>>>>>>>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hello again John,
>>>>>>>>>>>>
>>>>>>>>>>>> I was wondering if you could comment on the following
behavior of
>>>>>>>>>>>> point_stat when running with PrepBufr files from the NCAR
ds337 dataset.
>>>>>>>>>>>>
>>>>>>>>>>>> When I run point_stat with daily input ncdf observation
files (generated
>>>>>>>>>>>> by pb2nc) that cover a relatively small model domain
(say, the size of
>>>>>>>>>>>> Southern New England), with a corresponding small number
of obs (several
>>>>>>>>>>>> tens of thousand individual obs valid over the entire
day), point_stat
>>>>>>>>>>>> completes successfully in less than 10 s. When I try to
verify on an
>>>>>>>>>>>> outer domain that covers CONUS, with a corresponding
large number of obs
>>>>>>>>>>>> (1-2 million), point_stat requires about 60-70 minutes to
complete.
>>>>>>>>>>>> Thus, to verify hourly a full model run of 24 valid
times, over a day of
>>>>>>>>>>>> wallclock time is required.
>>>>>>>>>>>>
>>>>>>>>>>>> I don't recall if this scaling behavior occurred for v2.0
of MET.
>>>>>>>>>>>>
>>>>>>>>>>>> I welcome your thoughts.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>> Respectfully,
>>>>>>>>>>>>
>>>>>>>>>>>> John Henderson
>>>>>>>>>>>>
>>
>
------------------------------------------------
Subject: Processing time for point_stat with NCAR ds337 ncdf files
From: jhenders at aer.com
Time: Thu Aug 02 10:49:07 2012
=========================================
Host ...
rotor
=========================================
C++ Compiler Version ...
g++ (SUSE Linux) 4.3.4 [gcc-4_3-branch revision 152973]
Copyright (C) 2008 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There
is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
PURPOSE.
=========================================
Sizes of basic types ...
Sizeof char = 1
Sizeof short = 2
Sizeof int = 4
Sizeof long = 8
Sizeof long long = 8
Sizeof float = 4
Sizeof double = 8
Sizeof long double = 16
Sizeof char * = 8
Sizeof short * = 8
Sizeof int * = 8
Sizeof long * = 8
Sizeof long long * = 8
Sizeof float * = 8
Sizeof double * = 8
Sizeof long double * = 8
Sizeof void * = 8
=========================================
Alignments of basic types ...
char alignment = 1 bytes
short alignment = 2 bytes
int alignment = 4 bytes
long alignment = 8 bytes
long long alignment = 8 bytes
float alignment = 4 bytes
double alignment = 8 bytes
long double alignment = 16 bytes
void * alignment = 8 bytes
=========================================
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #57644] Processing time for point_stat with NCAR ds337 ncdf files
From: John Halley Gotway
Time: Thu Aug 02 10:56:31 2012
John,
The sizes and alignments of the basic types do not differ at all from
my desktop machine. I should have asked though - are you compiling
MET using GNU? If not, please compile the test program using
the C++ compiler you've specified in the CXX line of
"METv3.1/user_defs.mk".
Also if you haven't already done so, please make sure you have the
latest set of patches from:
http://www.dtcenter.org/met/users/support/known_issues/METv3.1/index.php
Although I doubt any of those posted patches will have any effect on
this behavior.
John
On 08/02/2012 10:49 AM, jhenders at aer.com via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>
> John,
>
> Here's the output (attached). I agree that likely the automated
> configure/compile-time code is making some bad assumptions as to
what we
> have on our machine.
>
> I hope this helps. I suppose the hands-off approach to compiling
> (without getting my hands dirty under the hood) can have its
drawbacks...
>
> John
>
> On 8/2/12 12:42 PM, John Halley Gotway via RT wrote:
>> John,
>>
>> Ah yes, my apologies. I sent you the syntax for METv4.0 - instead
of METv3.1, that you're using. The switch for METv4.0 is due to the
updates to the config file language.
>>
>> Yes, it appears the problem is in the reading of data from GRIB1
files. I've never seen that "confused" error message before, but I
was able to see it by passing a NaN to that
>> "ColorTable::interp(double) const" function. So it is reading the
GRIB1 data as all NaN's.
>>
>> My suspicion is that the sizes/alignments of things on your system
differ from any system on which we're run/tested MET in the past.
I've attached a small test program to this email that I'm hoping
>> you can compile and run, and then send us the output.
>>
>> You can compile it like this:
>> g++ test_program.cc -o test_program
>>
>> Thanks,
>> John
>>
>> On 08/02/2012 10:15 AM, jhenders at aer.com via RT wrote:
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>
>>> John,
>>>
>>> I seemed only to get the plot_data_plane call to work when I
specified
>>> the magic name like this: 'TMP/P500'. I trust that this is the
proper
>>> syntax. Then, however, the code said 'ColorTable::interp(double)
const
>>> -> confused!', which indicates to me that the grib reading is
hosed.
>>>
>>> I then closely inspected the outputs from the NCAR-provided test
scripts
>>> that are to be run at installation. I've attached the CNT file,
which
>>> shows that the forecast values are bad. I don't see any errors in
the
>>> make.out log file and I've never experienced grib1 libraries
exhibiting
>>> such flaky behavior.
>>>
>>> John
>>>
>>>
>>>
>>> On 8/2/12 11:24 AM, John Halley Gotway via RT wrote:
>>>> John,
>>>>
>>>> Unfortunately, unless I'm able to reproduce that problem with
NaN's here, I wont' be able to help much in the debugging.
>>>>
>>>> Since it's just the forecast values and the observation values
look fine, I suspect that the problem could be in the MET library code
that reads GRIB1 data.
>>>>
>>>> You could try running the following command to just make a plot
of some data (TMP at 500mb) from that file. If the plot looks good,
the problem is likely not in that GRIB1 library code. If the plot
>>>> looks bad (all NA's), it probably is:
>>>>
>>>> ${MET_BASE}/bin/plot_data_plane d1_2010121500_1200.grib
tmp_p500.ps 'name="TMP"; level="P500";'
>>>>
>>>> I've attached the resulting image (after converting to a png)
generated on my machine, which looks pretty good.
>>>>
>>>> John
>>>>
>>>> On 08/02/2012 09:05 AM, jhenders at aer.com via RT wrote:
>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>
>>>>> Hi John,
>>>>>
>>>>> Thanks for providing the command. Mine is very similar, as might
be
>>>>> expected...
>>>>>
>>>>> I've attached my point_stat output... You can see that I have
Nans, plus
>>>>> the expected NAs.
>>>>>
>>>>> John
>>>>>
>>>>> On 8/2/12 10:54 AM, John Halley Gotway via RT wrote:
>>>>>> John,
>>>>>>
>>>>>> Sorry for the delay in getting back to you. I was out of the
office yesterday afternoon.
>>>>>>
>>>>>> Here's the exact command I ran with the data you sent:
>>>>>>
>>>>>> time \
>>>>>> ${MET_BASE}/bin/point_stat \
>>>>>> d1_2010121500_1200.grib \
>>>>>> obs-20101215.nc \
>>>>>> PointStatConfig_V3.1-p1652 \
>>>>>> -point_obs obs-20101216.nc \
>>>>>> -outdir out \
>>>>>> -v 4
>>>>>>
>>>>>> And as I mentioned, it took about 24 seconds to execute on my
desktop.
>>>>>>
>>>>>> I've attached a tar file containing the ASCII output of this
run. I'm not seeing NaN's in the FBAR column (in
point_stat_120000L_20101215_120000V_cnt.txt). Instead I see
reasonable FBAR values ...
>>>>>> followed by reasonable values in FBAR_NCL and FBAR_NCU for the
normal confidence limits ... followed by NA's in the FBAR_BCL and
FBAR_BCU since we disabled bootstrap CI's in the config file.
>>>>>>
>>>>>> Is it possible you were seeing the NA's in those columns rather
than the good values in the FBAR column?
>>>>>>
>>>>>> Thanks,
>>>>>> John
>>>>>>
>>>>>> On 08/01/2012 02:58 PM, jhenders at aer.com via RT wrote:
>>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644
>
>>>>>>>
>>>>>>> Hello again John,
>>>>>>>
>>>>>>> I reran my test case locally on a rather old linux cluster.
This is the
>>>>>>> same one I used for MET v2 a year or so ago. This current job
took about
>>>>>>> 5 minutes and seems to have completed successfully.
>>>>>>>
>>>>>>> I then reran on a newer cluster and it breezed through the
reading of
>>>>>>> the observations in a matter of seconds. This is the step that
took a
>>>>>>> very long time on Pleiades. However, close inspection of the
output
>>>>>>> files shows that computations involving the forecast (FBAR,
e.g.) are
>>>>>>> all Nans. Physical values appear for all observation-related
quantities.
>>>>>>>
>>>>>>> I will keep looking...
>>>>>>>
>>>>>>> John
>>>>>>>
>>>>>>> On 8/1/12 1:32 PM, John Halley Gotway via RT wrote:
>>>>>>>> John,
>>>>>>>>
>>>>>>>> Something funny is going on here. I ran the exact data you
sent me on my desktop machine, and it took 24 seconds to complete -
not 60-70 minutes.
>>>>>>>>
>>>>>>>> My guess would be something to do with memory (perhaps it's
running out and using swap space) and/or disk I/O. On what system are
you running this?
>>>>>>>>
>>>>>>>> Are you able to try running the same thing on a different
system to see what the runtime is?
>>>>>>>>
>>>>>>>> John
>>>>>>>>
>>>>>>>> On 08/01/2012 11:10 AM, jhenders at aer.com via RT wrote:
>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>>>>
>>>>>>>>> Hi John,
>>>>>>>>>
>>>>>>>>> I've loaded the relevant files to henderson_data. Note that
because of
>>>>>>>>> the offset (with respect to UTC dates) for ds337's files, I
call
>>>>>>>>> point_stat with two daily ncdf files. This, of course,
contributes to
>>>>>>>>> the 60-70 minutes it requires for point-stat, but my
question simply is
>>>>>>>>> whether it is normal for such a non-linear scaling.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>> John
>>>>>>>>>
>>>>>>>>> On 8/1/12 10:47 AM, John Halley Gotway via RT wrote:
>>>>>>>>>> John,
>>>>>>>>>>
>>>>>>>>>> Uh oh, guess I'm out of tricks then!
>>>>>>>>>>
>>>>>>>>>> Would you be able to send me sample data for a run that's
taking 60-70 minutes? I could try running it here to see if there's
any other improvements that might help. I'd need the sample forecast
>>>>>>>>>> file, NetCDF point observation file, and the Point-Stat
Config file.
>>>>>>>>>>
>>>>>>>>>> You can post data to our anonymous ftp site following these
directions:
>>>>>>>>>>
http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>> On 08/01/2012 08:30 AM, jhenders at aer.com via RT wrote:
>>>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>>>>>>
>>>>>>>>>>> Hello John,
>>>>>>>>>>>
>>>>>>>>>>> It turns out that I already have both of those disabled. I
wanted to
>>>>>>>>>>> speed up my MET v2 runs a year ago, so at that time I set
the flags
>>>>>>>>>>> appropriately.
>>>>>>>>>>>
>>>>>>>>>>> John
>>>>>>>>>>>
>>>>>>>>>>> On 8/1/12 10:12 AM, John Halley Gotway via RT wrote:
>>>>>>>>>>>> John,
>>>>>>>>>>>>
>>>>>>>>>>>> There are two aspects of point_stat that don't scale very
well - the computation of rank correlation coefficients and bootstrap
confidence intervals. Both of them are controlled by settings in the
>>>>>>>>>>>> config file and disabling them yields significant
increases in runtime performance. By default, they are both *enabled*
in Point-Stat since that tool is typically run with a relatively small
number
>>>>>>>>>>>> of observations. And by default, they are both
*disabled* in Grid-Stat since that tool is typically run with a
relatively large number of observations.
>>>>>>>>>>>>
>>>>>>>>>>>> I'd suggest turning both of them off and try rerunning to
see how much the runtime improves.
>>>>>>>>>>>>
>>>>>>>>>>>> Disable rank correlation computations by setting
"rank_corr_flag = FALSE;" in METv4.0 (or by setting "rank_corr_flag =
0;" in previous versions of MET).
>>>>>>>>>>>> Disable bootstrap confidence intervals by setting "n_rep
= 0;" in the "boot" section in METv4.0 (or by setting "n_boot_rep =
0;" in previous versions of MET).
>>>>>>>>>>>>
>>>>>>>>>>>> Please let me know if/how much those changes improve the
runtime performance.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> John
>>>>>>>>>>>>
>>>>>>>>>>>> On 07/30/2012 01:25 PM, jhenders at aer.com via RT wrote:
>>>>>>>>>>>>> Mon Jul 30 13:25:36 2012: Request 57644 was acted upon.
>>>>>>>>>>>>> Transaction: Ticket created by jhenders at aer.com
>>>>>>>>>>>>> Queue: met_help
>>>>>>>>>>>>> Subject: Processing time for
point_stat with NCAR ds337 ncdf files
>>>>>>>>>>>>> Owner: Nobody
>>>>>>>>>>>>> Requestors: jhenders at aer.com
>>>>>>>>>>>>> Status: new
>>>>>>>>>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hello again John,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I was wondering if you could comment on the following
behavior of
>>>>>>>>>>>>> point_stat when running with PrepBufr files from the
NCAR ds337 dataset.
>>>>>>>>>>>>>
>>>>>>>>>>>>> When I run point_stat with daily input ncdf observation
files (generated
>>>>>>>>>>>>> by pb2nc) that cover a relatively small model domain
(say, the size of
>>>>>>>>>>>>> Southern New England), with a corresponding small number
of obs (several
>>>>>>>>>>>>> tens of thousand individual obs valid over the entire
day), point_stat
>>>>>>>>>>>>> completes successfully in less than 10 s. When I try to
verify on an
>>>>>>>>>>>>> outer domain that covers CONUS, with a corresponding
large number of obs
>>>>>>>>>>>>> (1-2 million), point_stat requires about 60-70 minutes
to complete.
>>>>>>>>>>>>> Thus, to verify hourly a full model run of 24 valid
times, over a day of
>>>>>>>>>>>>> wallclock time is required.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I don't recall if this scaling behavior occurred for
v2.0 of MET.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I welcome your thoughts.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Respectfully,
>>>>>>>>>>>>>
>>>>>>>>>>>>> John Henderson
>>>>>>>>>>>>>
>>>
>>
>
>
------------------------------------------------
Subject: Processing time for point_stat with NCAR ds337 ncdf files
From: jhenders at aer.com
Time: Thu Aug 02 11:01:11 2012
John,
I am using GNU (see attached user_defs.mk) and I have all patches
through 20120501.
John
On 8/2/12 12:56 PM, John Halley Gotway via RT wrote:
> John,
>
> The sizes and alignments of the basic types do not differ at all
from my desktop machine. I should have asked though - are you
compiling MET using GNU? If not, please compile the test program
using
> the C++ compiler you've specified in the CXX line of
"METv3.1/user_defs.mk".
>
> Also if you haven't already done so, please make sure you have the
latest set of patches from:
>
http://www.dtcenter.org/met/users/support/known_issues/METv3.1/index.php
>
> Although I doubt any of those posted patches will have any effect on
this behavior.
>
> John
>
> On 08/02/2012 10:49 AM, jhenders at aer.com via RT wrote:
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>
>> John,
>>
>> Here's the output (attached). I agree that likely the automated
>> configure/compile-time code is making some bad assumptions as to
what we
>> have on our machine.
>>
>> I hope this helps. I suppose the hands-off approach to compiling
>> (without getting my hands dirty under the hood) can have its
drawbacks...
>>
>> John
>>
>> On 8/2/12 12:42 PM, John Halley Gotway via RT wrote:
>>> John,
>>>
>>> Ah yes, my apologies. I sent you the syntax for METv4.0 - instead
of METv3.1, that you're using. The switch for METv4.0 is due to the
updates to the config file language.
>>>
>>> Yes, it appears the problem is in the reading of data from GRIB1
files. I've never seen that "confused" error message before, but I
was able to see it by passing a NaN to that
>>> "ColorTable::interp(double) const" function. So it is reading the
GRIB1 data as all NaN's.
>>>
>>> My suspicion is that the sizes/alignments of things on your system
differ from any system on which we're run/tested MET in the past.
I've attached a small test program to this email that I'm hoping
>>> you can compile and run, and then send us the output.
>>>
>>> You can compile it like this:
>>> g++ test_program.cc -o test_program
>>>
>>> Thanks,
>>> John
>>>
>>> On 08/02/2012 10:15 AM, jhenders at aer.com via RT wrote:
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>
>>>> John,
>>>>
>>>> I seemed only to get the plot_data_plane call to work when I
specified
>>>> the magic name like this: 'TMP/P500'. I trust that this is the
proper
>>>> syntax. Then, however, the code said 'ColorTable::interp(double)
const
>>>> -> confused!', which indicates to me that the grib reading is
hosed.
>>>>
>>>> I then closely inspected the outputs from the NCAR-provided test
scripts
>>>> that are to be run at installation. I've attached the CNT file,
which
>>>> shows that the forecast values are bad. I don't see any errors in
the
>>>> make.out log file and I've never experienced grib1 libraries
exhibiting
>>>> such flaky behavior.
>>>>
>>>> John
>>>>
>>>>
>>>>
>>>> On 8/2/12 11:24 AM, John Halley Gotway via RT wrote:
>>>>> John,
>>>>>
>>>>> Unfortunately, unless I'm able to reproduce that problem with
NaN's here, I wont' be able to help much in the debugging.
>>>>>
>>>>> Since it's just the forecast values and the observation values
look fine, I suspect that the problem could be in the MET library code
that reads GRIB1 data.
>>>>>
>>>>> You could try running the following command to just make a plot
of some data (TMP at 500mb) from that file. If the plot looks good,
the problem is likely not in that GRIB1 library code. If the plot
>>>>> looks bad (all NA's), it probably is:
>>>>>
>>>>> ${MET_BASE}/bin/plot_data_plane d1_2010121500_1200.grib
tmp_p500.ps 'name="TMP"; level="P500";'
>>>>>
>>>>> I've attached the resulting image (after converting to a png)
generated on my machine, which looks pretty good.
>>>>>
>>>>> John
>>>>>
>>>>> On 08/02/2012 09:05 AM, jhenders at aer.com via RT wrote:
>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>
>>>>>> Hi John,
>>>>>>
>>>>>> Thanks for providing the command. Mine is very similar, as
might be
>>>>>> expected...
>>>>>>
>>>>>> I've attached my point_stat output... You can see that I have
Nans, plus
>>>>>> the expected NAs.
>>>>>>
>>>>>> John
>>>>>>
>>>>>> On 8/2/12 10:54 AM, John Halley Gotway via RT wrote:
>>>>>>> John,
>>>>>>>
>>>>>>> Sorry for the delay in getting back to you. I was out of the
office yesterday afternoon.
>>>>>>>
>>>>>>> Here's the exact command I ran with the data you sent:
>>>>>>>
>>>>>>> time \
>>>>>>> ${MET_BASE}/bin/point_stat \
>>>>>>> d1_2010121500_1200.grib \
>>>>>>> obs-20101215.nc \
>>>>>>> PointStatConfig_V3.1-p1652 \
>>>>>>> -point_obs obs-20101216.nc \
>>>>>>> -outdir out \
>>>>>>> -v 4
>>>>>>>
>>>>>>> And as I mentioned, it took about 24 seconds to execute on my
desktop.
>>>>>>>
>>>>>>> I've attached a tar file containing the ASCII output of this
run. I'm not seeing NaN's in the FBAR column (in
point_stat_120000L_20101215_120000V_cnt.txt). Instead I see
reasonable FBAR values ...
>>>>>>> followed by reasonable values in FBAR_NCL and FBAR_NCU for the
normal confidence limits ... followed by NA's in the FBAR_BCL and
FBAR_BCU since we disabled bootstrap CI's in the config file.
>>>>>>>
>>>>>>> Is it possible you were seeing the NA's in those columns
rather than the good values in the FBAR column?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> John
>>>>>>>
>>>>>>> On 08/01/2012 02:58 PM, jhenders at aer.com via RT wrote:
>>>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644
>
>>>>>>>>
>>>>>>>> Hello again John,
>>>>>>>>
>>>>>>>> I reran my test case locally on a rather old linux cluster.
This is the
>>>>>>>> same one I used for MET v2 a year or so ago. This current job
took about
>>>>>>>> 5 minutes and seems to have completed successfully.
>>>>>>>>
>>>>>>>> I then reran on a newer cluster and it breezed through the
reading of
>>>>>>>> the observations in a matter of seconds. This is the step
that took a
>>>>>>>> very long time on Pleiades. However, close inspection of the
output
>>>>>>>> files shows that computations involving the forecast (FBAR,
e.g.) are
>>>>>>>> all Nans. Physical values appear for all observation-related
quantities.
>>>>>>>>
>>>>>>>> I will keep looking...
>>>>>>>>
>>>>>>>> John
>>>>>>>>
>>>>>>>> On 8/1/12 1:32 PM, John Halley Gotway via RT wrote:
>>>>>>>>> John,
>>>>>>>>>
>>>>>>>>> Something funny is going on here. I ran the exact data you
sent me on my desktop machine, and it took 24 seconds to complete -
not 60-70 minutes.
>>>>>>>>>
>>>>>>>>> My guess would be something to do with memory (perhaps it's
running out and using swap space) and/or disk I/O. On what system are
you running this?
>>>>>>>>>
>>>>>>>>> Are you able to try running the same thing on a different
system to see what the runtime is?
>>>>>>>>>
>>>>>>>>> John
>>>>>>>>>
>>>>>>>>> On 08/01/2012 11:10 AM, jhenders at aer.com via RT wrote:
>>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>>>>>
>>>>>>>>>> Hi John,
>>>>>>>>>>
>>>>>>>>>> I've loaded the relevant files to henderson_data. Note that
because of
>>>>>>>>>> the offset (with respect to UTC dates) for ds337's files, I
call
>>>>>>>>>> point_stat with two daily ncdf files. This, of course,
contributes to
>>>>>>>>>> the 60-70 minutes it requires for point-stat, but my
question simply is
>>>>>>>>>> whether it is normal for such a non-linear scaling.
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>> On 8/1/12 10:47 AM, John Halley Gotway via RT wrote:
>>>>>>>>>>> John,
>>>>>>>>>>>
>>>>>>>>>>> Uh oh, guess I'm out of tricks then!
>>>>>>>>>>>
>>>>>>>>>>> Would you be able to send me sample data for a run that's
taking 60-70 minutes? I could try running it here to see if there's
any other improvements that might help. I'd need the sample forecast
>>>>>>>>>>> file, NetCDF point observation file, and the Point-Stat
Config file.
>>>>>>>>>>>
>>>>>>>>>>> You can post data to our anonymous ftp site following
these directions:
>>>>>>>>>>>
http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> John
>>>>>>>>>>>
>>>>>>>>>>> On 08/01/2012 08:30 AM, jhenders at aer.com via RT wrote:
>>>>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>>>>>>>
>>>>>>>>>>>> Hello John,
>>>>>>>>>>>>
>>>>>>>>>>>> It turns out that I already have both of those disabled.
I wanted to
>>>>>>>>>>>> speed up my MET v2 runs a year ago, so at that time I set
the flags
>>>>>>>>>>>> appropriately.
>>>>>>>>>>>>
>>>>>>>>>>>> John
>>>>>>>>>>>>
>>>>>>>>>>>> On 8/1/12 10:12 AM, John Halley Gotway via RT wrote:
>>>>>>>>>>>>> John,
>>>>>>>>>>>>>
>>>>>>>>>>>>> There are two aspects of point_stat that don't scale
very well - the computation of rank correlation coefficients and
bootstrap confidence intervals. Both of them are controlled by
settings in the
>>>>>>>>>>>>> config file and disabling them yields significant
increases in runtime performance. By default, they are both *enabled*
in Point-Stat since that tool is typically run with a relatively small
number
>>>>>>>>>>>>> of observations. And by default, they are both
*disabled* in Grid-Stat since that tool is typically run with a
relatively large number of observations.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'd suggest turning both of them off and try rerunning
to see how much the runtime improves.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Disable rank correlation computations by setting
"rank_corr_flag = FALSE;" in METv4.0 (or by setting "rank_corr_flag =
0;" in previous versions of MET).
>>>>>>>>>>>>> Disable bootstrap confidence intervals by setting "n_rep
= 0;" in the "boot" section in METv4.0 (or by setting "n_boot_rep =
0;" in previous versions of MET).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please let me know if/how much those changes improve the
runtime performance.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> John
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 07/30/2012 01:25 PM, jhenders at aer.com via RT wrote:
>>>>>>>>>>>>>> Mon Jul 30 13:25:36 2012: Request 57644 was acted upon.
>>>>>>>>>>>>>> Transaction: Ticket created by jhenders at aer.com
>>>>>>>>>>>>>> Queue: met_help
>>>>>>>>>>>>>> Subject: Processing time for
point_stat with NCAR ds337 ncdf files
>>>>>>>>>>>>>> Owner: Nobody
>>>>>>>>>>>>>> Requestors: jhenders at aer.com
>>>>>>>>>>>>>> Status: new
>>>>>>>>>>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello again John,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I was wondering if you could comment on the following
behavior of
>>>>>>>>>>>>>> point_stat when running with PrepBufr files from the
NCAR ds337 dataset.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> When I run point_stat with daily input ncdf observation
files (generated
>>>>>>>>>>>>>> by pb2nc) that cover a relatively small model domain
(say, the size of
>>>>>>>>>>>>>> Southern New England), with a corresponding small
number of obs (several
>>>>>>>>>>>>>> tens of thousand individual obs valid over the entire
day), point_stat
>>>>>>>>>>>>>> completes successfully in less than 10 s. When I try to
verify on an
>>>>>>>>>>>>>> outer domain that covers CONUS, with a corresponding
large number of obs
>>>>>>>>>>>>>> (1-2 million), point_stat requires about 60-70 minutes
to complete.
>>>>>>>>>>>>>> Thus, to verify hourly a full model run of 24 valid
times, over a day of
>>>>>>>>>>>>>> wallclock time is required.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I don't recall if this scaling behavior occurred for
v2.0 of MET.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I welcome your thoughts.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Respectfully,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> John Henderson
>>>>>>>>>>>>>>
>>
>
------------------------------------------------
Subject: Processing time for point_stat with NCAR ds337 ncdf files
From: jhenders at aer.com
Time: Thu Aug 02 11:01:11 2012
###############################################################################
##
## Begin Variables to be modified before building
## Default settings for GNU compilers
##
###############################################################################
# Path to GNU Make command
MAKE = /usr/bin/make
# Architecture flags
ARCH_FLAGS = -DBIGENDIAN -DBLOCK4
# Path to the C++ Compiler
# C++ compiler flags
# Any additional required libraries
CXX = /usr/bin/g++
CXX_FLAGS = -Wall -Wshadow -static #-g -m32
CXX_LIBS = -L/usr/lib
# Path to the Fortran Compiler
# Fortran compiler flags
# Any additional required libraries
FC = /usr/bin/gfortran #/usr/bin/g77
FC_FLAGS = -Wall -Wshadow -static -ff2c #-g -m32
FC_LIBS = -L/usr/lib -lgfortran
# Make print options
PRINT_OPTS = --no-print-directory
# Top level directory for the NetCDF library
# NetCDF include directory specified as: -I/your/include/path
# NetCDF library directory specified as: -L/your/library/path
NETCDF_BASE = /rotor/data/nwp-installs/netcdf-3.6.3/installed/gnu
NETCDF_INCS = -I$(NETCDF_BASE)/include
NETCDF_LIBS = -L$(NETCDF_BASE)/lib
# Top level directory for BUFRLIB
# BUFRLIB include directory specified as: -I/your/include/path
# BUFRLIB library directory specified as: -L/your/library/path
BUFR_BASE = /home/jhenders/NWP/MET/BUFRLIB
BUFR_INCS = -I$(BUFR_BASE)
BUFR_LIBS = -L$(BUFR_BASE)
# Top level directory for the GNU Scientific Library (GSL) if it's not
# installed in a standard location.
# GSL include directory specified as: -I/your/include/path
# GSL library directory specified as: -L/your/library/path
GSL_BASE = /home/jhenders/NWP/MET/gsl-1.13
GSL_INCS = -I$(GSL_BASE)
GSL_LIBS = -L$(GSL_BASE)/.libs -L$(GSL_BASE)/cblas/.libs
# Top level directory for the F2C or G2C Library if it's not installed
in a
# standard location.
# F2C include directory specified as: -I/your/include/path
# F2C library directory containing libf2c.a or libg2c.a and specified
as:
# -L/your/library/path
# Name of the library to be used: -lf2c or -lg2c
# NOTE: Only required for the GNU g77 Fortran compiler
F2C_BASE =
F2C_INCS =
F2C_LIBS =
F2C_LIBNAME =
# Optional flags to disable the compilation of MET tools
# Specify a non-zero value to enable the compilation of the tool
ENABLE_ASCII2NC = 1
ENABLE_ENSEMBLE_STAT = 1
ENABLE_GEN_POLY_MASK = 1
ENABLE_GRID_STAT = 1
ENABLE_MADIS2NC = 1
ENABLE_MODE = 1
ENABLE_MODE_ANALYSIS = 1
ENABLE_PB2NC = 1
ENABLE_PCP_COMBINE = 1
ENABLE_PLOT_DATA_PLANE = 1
ENABLE_PLOT_POINT_OBS = 1
ENABLE_POINT_STAT = 1
ENABLE_STAT_ANALYSIS = 1
ENABLE_WAVELET_STAT = 1
ENABLE_WWMCA = 1
###############################################################################
##
## End Variables to be modified before building
##
###############################################################################
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #57644] Processing time for point_stat with NCAR ds337 ncdf files
From: John Halley Gotway
Time: Thu Aug 02 11:09:26 2012
John,
Please try removing -DBIGENDIAN from the ARCH_FLAGS setting in
user_defs.mk. Then recompile, being sure to do a "make clean" first:
make clean
make >& make_met.log
Then try rerunning your test case to see if the behavior differs.
When I compiled METv3.1 using the -DBIGENDIAN flag on my local machine
and reran the test case you sent, I was able to reproduce the NaN's
you're seeing in the output. So I'm very hopeful this fix
will solve the problem.
This issue really underscores our need for a better configure/compile
script so that users don't have to muck around in the build details
like this.
John
On 08/02/2012 11:01 AM, jhenders at aer.com via RT wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>
> John,
>
> I am using GNU (see attached user_defs.mk) and I have all patches
> through 20120501.
>
> John
>
> On 8/2/12 12:56 PM, John Halley Gotway via RT wrote:
>> John,
>>
>> The sizes and alignments of the basic types do not differ at all
from my desktop machine. I should have asked though - are you
compiling MET using GNU? If not, please compile the test program
using
>> the C++ compiler you've specified in the CXX line of
"METv3.1/user_defs.mk".
>>
>> Also if you haven't already done so, please make sure you have the
latest set of patches from:
>>
http://www.dtcenter.org/met/users/support/known_issues/METv3.1/index.php
>>
>> Although I doubt any of those posted patches will have any effect
on this behavior.
>>
>> John
>>
>> On 08/02/2012 10:49 AM, jhenders at aer.com via RT wrote:
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>
>>> John,
>>>
>>> Here's the output (attached). I agree that likely the automated
>>> configure/compile-time code is making some bad assumptions as to
what we
>>> have on our machine.
>>>
>>> I hope this helps. I suppose the hands-off approach to compiling
>>> (without getting my hands dirty under the hood) can have its
drawbacks...
>>>
>>> John
>>>
>>> On 8/2/12 12:42 PM, John Halley Gotway via RT wrote:
>>>> John,
>>>>
>>>> Ah yes, my apologies. I sent you the syntax for METv4.0 -
instead of METv3.1, that you're using. The switch for METv4.0 is due
to the updates to the config file language.
>>>>
>>>> Yes, it appears the problem is in the reading of data from GRIB1
files. I've never seen that "confused" error message before, but I
was able to see it by passing a NaN to that
>>>> "ColorTable::interp(double) const" function. So it is reading
the GRIB1 data as all NaN's.
>>>>
>>>> My suspicion is that the sizes/alignments of things on your
system differ from any system on which we're run/tested MET in the
past. I've attached a small test program to this email that I'm
hoping
>>>> you can compile and run, and then send us the output.
>>>>
>>>> You can compile it like this:
>>>> g++ test_program.cc -o test_program
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>> On 08/02/2012 10:15 AM, jhenders at aer.com via RT wrote:
>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>
>>>>> John,
>>>>>
>>>>> I seemed only to get the plot_data_plane call to work when I
specified
>>>>> the magic name like this: 'TMP/P500'. I trust that this is the
proper
>>>>> syntax. Then, however, the code said 'ColorTable::interp(double)
const
>>>>> -> confused!', which indicates to me that the grib reading is
hosed.
>>>>>
>>>>> I then closely inspected the outputs from the NCAR-provided test
scripts
>>>>> that are to be run at installation. I've attached the CNT file,
which
>>>>> shows that the forecast values are bad. I don't see any errors
in the
>>>>> make.out log file and I've never experienced grib1 libraries
exhibiting
>>>>> such flaky behavior.
>>>>>
>>>>> John
>>>>>
>>>>>
>>>>>
>>>>> On 8/2/12 11:24 AM, John Halley Gotway via RT wrote:
>>>>>> John,
>>>>>>
>>>>>> Unfortunately, unless I'm able to reproduce that problem with
NaN's here, I wont' be able to help much in the debugging.
>>>>>>
>>>>>> Since it's just the forecast values and the observation values
look fine, I suspect that the problem could be in the MET library code
that reads GRIB1 data.
>>>>>>
>>>>>> You could try running the following command to just make a plot
of some data (TMP at 500mb) from that file. If the plot looks good,
the problem is likely not in that GRIB1 library code. If the plot
>>>>>> looks bad (all NA's), it probably is:
>>>>>>
>>>>>> ${MET_BASE}/bin/plot_data_plane
d1_2010121500_1200.grib tmp_p500.ps 'name="TMP"; level="P500";'
>>>>>>
>>>>>> I've attached the resulting image (after converting to a png)
generated on my machine, which looks pretty good.
>>>>>>
>>>>>> John
>>>>>>
>>>>>> On 08/02/2012 09:05 AM, jhenders at aer.com via RT wrote:
>>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644
>
>>>>>>>
>>>>>>> Hi John,
>>>>>>>
>>>>>>> Thanks for providing the command. Mine is very similar, as
might be
>>>>>>> expected...
>>>>>>>
>>>>>>> I've attached my point_stat output... You can see that I have
Nans, plus
>>>>>>> the expected NAs.
>>>>>>>
>>>>>>> John
>>>>>>>
>>>>>>> On 8/2/12 10:54 AM, John Halley Gotway via RT wrote:
>>>>>>>> John,
>>>>>>>>
>>>>>>>> Sorry for the delay in getting back to you. I was out of the
office yesterday afternoon.
>>>>>>>>
>>>>>>>> Here's the exact command I ran with the data you sent:
>>>>>>>>
>>>>>>>> time \
>>>>>>>> ${MET_BASE}/bin/point_stat \
>>>>>>>> d1_2010121500_1200.grib \
>>>>>>>> obs-20101215.nc \
>>>>>>>> PointStatConfig_V3.1-p1652 \
>>>>>>>> -point_obs obs-20101216.nc \
>>>>>>>> -outdir out \
>>>>>>>> -v 4
>>>>>>>>
>>>>>>>> And as I mentioned, it took about 24 seconds to execute on my
desktop.
>>>>>>>>
>>>>>>>> I've attached a tar file containing the ASCII output of this
run. I'm not seeing NaN's in the FBAR column (in
point_stat_120000L_20101215_120000V_cnt.txt). Instead I see
reasonable FBAR values ...
>>>>>>>> followed by reasonable values in FBAR_NCL and FBAR_NCU for
the normal confidence limits ... followed by NA's in the FBAR_BCL and
FBAR_BCU since we disabled bootstrap CI's in the config file.
>>>>>>>>
>>>>>>>> Is it possible you were seeing the NA's in those columns
rather than the good values in the FBAR column?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> John
>>>>>>>>
>>>>>>>> On 08/01/2012 02:58 PM, jhenders at aer.com via RT wrote:
>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>>>>
>>>>>>>>> Hello again John,
>>>>>>>>>
>>>>>>>>> I reran my test case locally on a rather old linux cluster.
This is the
>>>>>>>>> same one I used for MET v2 a year or so ago. This current
job took about
>>>>>>>>> 5 minutes and seems to have completed successfully.
>>>>>>>>>
>>>>>>>>> I then reran on a newer cluster and it breezed through the
reading of
>>>>>>>>> the observations in a matter of seconds. This is the step
that took a
>>>>>>>>> very long time on Pleiades. However, close inspection of the
output
>>>>>>>>> files shows that computations involving the forecast (FBAR,
e.g.) are
>>>>>>>>> all Nans. Physical values appear for all observation-related
quantities.
>>>>>>>>>
>>>>>>>>> I will keep looking...
>>>>>>>>>
>>>>>>>>> John
>>>>>>>>>
>>>>>>>>> On 8/1/12 1:32 PM, John Halley Gotway via RT wrote:
>>>>>>>>>> John,
>>>>>>>>>>
>>>>>>>>>> Something funny is going on here. I ran the exact data you
sent me on my desktop machine, and it took 24 seconds to complete -
not 60-70 minutes.
>>>>>>>>>>
>>>>>>>>>> My guess would be something to do with memory (perhaps it's
running out and using swap space) and/or disk I/O. On what system are
you running this?
>>>>>>>>>>
>>>>>>>>>> Are you able to try running the same thing on a different
system to see what the runtime is?
>>>>>>>>>>
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>> On 08/01/2012 11:10 AM, jhenders at aer.com via RT wrote:
>>>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>>>>>>
>>>>>>>>>>> Hi John,
>>>>>>>>>>>
>>>>>>>>>>> I've loaded the relevant files to henderson_data. Note
that because of
>>>>>>>>>>> the offset (with respect to UTC dates) for ds337's files,
I call
>>>>>>>>>>> point_stat with two daily ncdf files. This, of course,
contributes to
>>>>>>>>>>> the 60-70 minutes it requires for point-stat, but my
question simply is
>>>>>>>>>>> whether it is normal for such a non-linear scaling.
>>>>>>>>>>>
>>>>>>>>>>> Thanks.
>>>>>>>>>>>
>>>>>>>>>>> John
>>>>>>>>>>>
>>>>>>>>>>> On 8/1/12 10:47 AM, John Halley Gotway via RT wrote:
>>>>>>>>>>>> John,
>>>>>>>>>>>>
>>>>>>>>>>>> Uh oh, guess I'm out of tricks then!
>>>>>>>>>>>>
>>>>>>>>>>>> Would you be able to send me sample data for a run that's
taking 60-70 minutes? I could try running it here to see if there's
any other improvements that might help. I'd need the sample forecast
>>>>>>>>>>>> file, NetCDF point observation file, and the Point-Stat
Config file.
>>>>>>>>>>>>
>>>>>>>>>>>> You can post data to our anonymous ftp site following
these directions:
>>>>>>>>>>>>
http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> John
>>>>>>>>>>>>
>>>>>>>>>>>> On 08/01/2012 08:30 AM, jhenders at aer.com via RT wrote:
>>>>>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hello John,
>>>>>>>>>>>>>
>>>>>>>>>>>>> It turns out that I already have both of those disabled.
I wanted to
>>>>>>>>>>>>> speed up my MET v2 runs a year ago, so at that time I
set the flags
>>>>>>>>>>>>> appropriately.
>>>>>>>>>>>>>
>>>>>>>>>>>>> John
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 8/1/12 10:12 AM, John Halley Gotway via RT wrote:
>>>>>>>>>>>>>> John,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> There are two aspects of point_stat that don't scale
very well - the computation of rank correlation coefficients and
bootstrap confidence intervals. Both of them are controlled by
settings in the
>>>>>>>>>>>>>> config file and disabling them yields significant
increases in runtime performance. By default, they are both *enabled*
in Point-Stat since that tool is typically run with a relatively small
number
>>>>>>>>>>>>>> of observations. And by default, they are both
*disabled* in Grid-Stat since that tool is typically run with a
relatively large number of observations.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'd suggest turning both of them off and try rerunning
to see how much the runtime improves.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Disable rank correlation computations by setting
"rank_corr_flag = FALSE;" in METv4.0 (or by setting "rank_corr_flag =
0;" in previous versions of MET).
>>>>>>>>>>>>>> Disable bootstrap confidence intervals by setting
"n_rep = 0;" in the "boot" section in METv4.0 (or by setting
"n_boot_rep = 0;" in previous versions of MET).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Please let me know if/how much those changes improve
the runtime performance.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 07/30/2012 01:25 PM, jhenders at aer.com via RT wrote:
>>>>>>>>>>>>>>> Mon Jul 30 13:25:36 2012: Request 57644 was acted
upon.
>>>>>>>>>>>>>>> Transaction: Ticket created by jhenders at aer.com
>>>>>>>>>>>>>>> Queue: met_help
>>>>>>>>>>>>>>> Subject: Processing time for
point_stat with NCAR ds337 ncdf files
>>>>>>>>>>>>>>> Owner: Nobody
>>>>>>>>>>>>>>> Requestors: jhenders at aer.com
>>>>>>>>>>>>>>> Status: new
>>>>>>>>>>>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hello again John,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I was wondering if you could comment on the following
behavior of
>>>>>>>>>>>>>>> point_stat when running with PrepBufr files from the
NCAR ds337 dataset.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When I run point_stat with daily input ncdf
observation files (generated
>>>>>>>>>>>>>>> by pb2nc) that cover a relatively small model domain
(say, the size of
>>>>>>>>>>>>>>> Southern New England), with a corresponding small
number of obs (several
>>>>>>>>>>>>>>> tens of thousand individual obs valid over the entire
day), point_stat
>>>>>>>>>>>>>>> completes successfully in less than 10 s. When I try
to verify on an
>>>>>>>>>>>>>>> outer domain that covers CONUS, with a corresponding
large number of obs
>>>>>>>>>>>>>>> (1-2 million), point_stat requires about 60-70 minutes
to complete.
>>>>>>>>>>>>>>> Thus, to verify hourly a full model run of 24 valid
times, over a day of
>>>>>>>>>>>>>>> wallclock time is required.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I don't recall if this scaling behavior occurred for
v2.0 of MET.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I welcome your thoughts.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Respectfully,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> John Henderson
>>>>>>>>>>>>>>>
>>>
>>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #57644] Processing time for point_stat with NCAR ds337 ncdf files
From: jhenders at aer.com
Time: Thu Aug 02 13:47:24 2012
Hello again,
Well...complete success!
Thank you for pointing out that I had inadvertently included what
turned
out in effect to be a byte-swapping flag. A quick check of your
recommended user_defs.mk files did not show it present, so it must
have
been an artifact of my upgrade on another local machine (the old one
that ran v2) that has a different endianness. Sorry to have caused
your
in-depth investigation.
The other bit of good news is that it ran in 27 seconds. A number of
sensitivity tests ongoing still on Pleiades give run times of between
20
minutes and 68 minutes depending on the type of processor I use, plus
where I stage the data. All in all, I shouldn't be happy at all with
even 20 minutes. I will keep testing on that machine...
I very much appreciate all your prompt help. Thanks again.
Regards,
John
On 8/2/12 1:09 PM, John Halley Gotway via RT wrote:
> John,
>
> Please try removing -DBIGENDIAN from the ARCH_FLAGS setting in
user_defs.mk. Then recompile, being sure to do a "make clean" first:
> make clean
> make >& make_met.log
>
> Then try rerunning your test case to see if the behavior differs.
>
> When I compiled METv3.1 using the -DBIGENDIAN flag on my local
machine and reran the test case you sent, I was able to reproduce the
NaN's you're seeing in the output. So I'm very hopeful this fix
> will solve the problem.
>
> This issue really underscores our need for a better
configure/compile script so that users don't have to muck around in
the build details like this.
>
> John
>
> On 08/02/2012 11:01 AM, jhenders at aer.com via RT wrote:
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>
>> John,
>>
>> I am using GNU (see attached user_defs.mk) and I have all patches
>> through 20120501.
>>
>> John
>>
>> On 8/2/12 12:56 PM, John Halley Gotway via RT wrote:
>>> John,
>>>
>>> The sizes and alignments of the basic types do not differ at all
from my desktop machine. I should have asked though - are you
compiling MET using GNU? If not, please compile the test program
using
>>> the C++ compiler you've specified in the CXX line of
"METv3.1/user_defs.mk".
>>>
>>> Also if you haven't already done so, please make sure you have the
latest set of patches from:
>>>
http://www.dtcenter.org/met/users/support/known_issues/METv3.1/index.php
>>>
>>> Although I doubt any of those posted patches will have any effect
on this behavior.
>>>
>>> John
>>>
>>> On 08/02/2012 10:49 AM, jhenders at aer.com via RT wrote:
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>
>>>> John,
>>>>
>>>> Here's the output (attached). I agree that likely the automated
>>>> configure/compile-time code is making some bad assumptions as to
what we
>>>> have on our machine.
>>>>
>>>> I hope this helps. I suppose the hands-off approach to compiling
>>>> (without getting my hands dirty under the hood) can have its
drawbacks...
>>>>
>>>> John
>>>>
>>>> On 8/2/12 12:42 PM, John Halley Gotway via RT wrote:
>>>>> John,
>>>>>
>>>>> Ah yes, my apologies. I sent you the syntax for METv4.0 -
instead of METv3.1, that you're using. The switch for METv4.0 is due
to the updates to the config file language.
>>>>>
>>>>> Yes, it appears the problem is in the reading of data from GRIB1
files. I've never seen that "confused" error message before, but I
was able to see it by passing a NaN to that
>>>>> "ColorTable::interp(double) const" function. So it is reading
the GRIB1 data as all NaN's.
>>>>>
>>>>> My suspicion is that the sizes/alignments of things on your
system differ from any system on which we're run/tested MET in the
past. I've attached a small test program to this email that I'm
hoping
>>>>> you can compile and run, and then send us the output.
>>>>>
>>>>> You can compile it like this:
>>>>> g++ test_program.cc -o test_program
>>>>>
>>>>> Thanks,
>>>>> John
>>>>>
>>>>> On 08/02/2012 10:15 AM, jhenders at aer.com via RT wrote:
>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>
>>>>>> John,
>>>>>>
>>>>>> I seemed only to get the plot_data_plane call to work when I
specified
>>>>>> the magic name like this: 'TMP/P500'. I trust that this is the
proper
>>>>>> syntax. Then, however, the code said
'ColorTable::interp(double) const
>>>>>> -> confused!', which indicates to me that the grib reading is
hosed.
>>>>>>
>>>>>> I then closely inspected the outputs from the NCAR-provided
test scripts
>>>>>> that are to be run at installation. I've attached the CNT file,
which
>>>>>> shows that the forecast values are bad. I don't see any errors
in the
>>>>>> make.out log file and I've never experienced grib1 libraries
exhibiting
>>>>>> such flaky behavior.
>>>>>>
>>>>>> John
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 8/2/12 11:24 AM, John Halley Gotway via RT wrote:
>>>>>>> John,
>>>>>>>
>>>>>>> Unfortunately, unless I'm able to reproduce that problem with
NaN's here, I wont' be able to help much in the debugging.
>>>>>>>
>>>>>>> Since it's just the forecast values and the observation values
look fine, I suspect that the problem could be in the MET library code
that reads GRIB1 data.
>>>>>>>
>>>>>>> You could try running the following command to just make a
plot of some data (TMP at 500mb) from that file. If the plot looks
good, the problem is likely not in that GRIB1 library code. If the
plot
>>>>>>> looks bad (all NA's), it probably is:
>>>>>>>
>>>>>>> ${MET_BASE}/bin/plot_data_plane
d1_2010121500_1200.grib tmp_p500.ps 'name="TMP"; level="P500";'
>>>>>>>
>>>>>>> I've attached the resulting image (after converting to a png)
generated on my machine, which looks pretty good.
>>>>>>>
>>>>>>> John
>>>>>>>
>>>>>>> On 08/02/2012 09:05 AM, jhenders at aer.com via RT wrote:
>>>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644
>
>>>>>>>>
>>>>>>>> Hi John,
>>>>>>>>
>>>>>>>> Thanks for providing the command. Mine is very similar, as
might be
>>>>>>>> expected...
>>>>>>>>
>>>>>>>> I've attached my point_stat output... You can see that I have
Nans, plus
>>>>>>>> the expected NAs.
>>>>>>>>
>>>>>>>> John
>>>>>>>>
>>>>>>>> On 8/2/12 10:54 AM, John Halley Gotway via RT wrote:
>>>>>>>>> John,
>>>>>>>>>
>>>>>>>>> Sorry for the delay in getting back to you. I was out of
the office yesterday afternoon.
>>>>>>>>>
>>>>>>>>> Here's the exact command I ran with the data you sent:
>>>>>>>>>
>>>>>>>>> time \
>>>>>>>>> ${MET_BASE}/bin/point_stat \
>>>>>>>>> d1_2010121500_1200.grib \
>>>>>>>>> obs-20101215.nc \
>>>>>>>>> PointStatConfig_V3.1-p1652 \
>>>>>>>>> -point_obs obs-20101216.nc \
>>>>>>>>> -outdir out \
>>>>>>>>> -v 4
>>>>>>>>>
>>>>>>>>> And as I mentioned, it took about 24 seconds to execute on
my desktop.
>>>>>>>>>
>>>>>>>>> I've attached a tar file containing the ASCII output of this
run. I'm not seeing NaN's in the FBAR column (in
point_stat_120000L_20101215_120000V_cnt.txt). Instead I see
reasonable FBAR values ...
>>>>>>>>> followed by reasonable values in FBAR_NCL and FBAR_NCU for
the normal confidence limits ... followed by NA's in the FBAR_BCL and
FBAR_BCU since we disabled bootstrap CI's in the config file.
>>>>>>>>>
>>>>>>>>> Is it possible you were seeing the NA's in those columns
rather than the good values in the FBAR column?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> John
>>>>>>>>>
>>>>>>>>> On 08/01/2012 02:58 PM, jhenders at aer.com via RT wrote:
>>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>>>>>
>>>>>>>>>> Hello again John,
>>>>>>>>>>
>>>>>>>>>> I reran my test case locally on a rather old linux cluster.
This is the
>>>>>>>>>> same one I used for MET v2 a year or so ago. This current
job took about
>>>>>>>>>> 5 minutes and seems to have completed successfully.
>>>>>>>>>>
>>>>>>>>>> I then reran on a newer cluster and it breezed through the
reading of
>>>>>>>>>> the observations in a matter of seconds. This is the step
that took a
>>>>>>>>>> very long time on Pleiades. However, close inspection of
the output
>>>>>>>>>> files shows that computations involving the forecast (FBAR,
e.g.) are
>>>>>>>>>> all Nans. Physical values appear for all observation-
related quantities.
>>>>>>>>>>
>>>>>>>>>> I will keep looking...
>>>>>>>>>>
>>>>>>>>>> John
>>>>>>>>>>
>>>>>>>>>> On 8/1/12 1:32 PM, John Halley Gotway via RT wrote:
>>>>>>>>>>> John,
>>>>>>>>>>>
>>>>>>>>>>> Something funny is going on here. I ran the exact data
you sent me on my desktop machine, and it took 24 seconds to complete
- not 60-70 minutes.
>>>>>>>>>>>
>>>>>>>>>>> My guess would be something to do with memory (perhaps
it's running out and using swap space) and/or disk I/O. On what
system are you running this?
>>>>>>>>>>>
>>>>>>>>>>> Are you able to try running the same thing on a different
system to see what the runtime is?
>>>>>>>>>>>
>>>>>>>>>>> John
>>>>>>>>>>>
>>>>>>>>>>> On 08/01/2012 11:10 AM, jhenders at aer.com via RT wrote:
>>>>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>>>>>>>
>>>>>>>>>>>> Hi John,
>>>>>>>>>>>>
>>>>>>>>>>>> I've loaded the relevant files to henderson_data. Note
that because of
>>>>>>>>>>>> the offset (with respect to UTC dates) for ds337's files,
I call
>>>>>>>>>>>> point_stat with two daily ncdf files. This, of course,
contributes to
>>>>>>>>>>>> the 60-70 minutes it requires for point-stat, but my
question simply is
>>>>>>>>>>>> whether it is normal for such a non-linear scaling.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>> John
>>>>>>>>>>>>
>>>>>>>>>>>> On 8/1/12 10:47 AM, John Halley Gotway via RT wrote:
>>>>>>>>>>>>> John,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Uh oh, guess I'm out of tricks then!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Would you be able to send me sample data for a run
that's taking 60-70 minutes? I could try running it here to see if
there's any other improvements that might help. I'd need the sample
forecast
>>>>>>>>>>>>> file, NetCDF point observation file, and the Point-Stat
Config file.
>>>>>>>>>>>>>
>>>>>>>>>>>>> You can post data to our anonymous ftp site following
these directions:
>>>>>>>>>>>>>
http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> John
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 08/01/2012 08:30 AM, jhenders at aer.com via RT wrote:
>>>>>>>>>>>>>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello John,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It turns out that I already have both of those
disabled. I wanted to
>>>>>>>>>>>>>> speed up my MET v2 runs a year ago, so at that time I
set the flags
>>>>>>>>>>>>>> appropriately.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 8/1/12 10:12 AM, John Halley Gotway via RT wrote:
>>>>>>>>>>>>>>> John,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> There are two aspects of point_stat that don't scale
very well - the computation of rank correlation coefficients and
bootstrap confidence intervals. Both of them are controlled by
settings in the
>>>>>>>>>>>>>>> config file and disabling them yields significant
increases in runtime performance. By default, they are both *enabled*
in Point-Stat since that tool is typically run with a relatively small
number
>>>>>>>>>>>>>>> of observations. And by default, they are both
*disabled* in Grid-Stat since that tool is typically run with a
relatively large number of observations.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'd suggest turning both of them off and try rerunning
to see how much the runtime improves.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Disable rank correlation computations by setting
"rank_corr_flag = FALSE;" in METv4.0 (or by setting "rank_corr_flag =
0;" in previous versions of MET).
>>>>>>>>>>>>>>> Disable bootstrap confidence intervals by setting
"n_rep = 0;" in the "boot" section in METv4.0 (or by setting
"n_boot_rep = 0;" in previous versions of MET).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Please let me know if/how much those changes improve
the runtime performance.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> John
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 07/30/2012 01:25 PM, jhenders at aer.com via RT wrote:
>>>>>>>>>>>>>>>> Mon Jul 30 13:25:36 2012: Request 57644 was acted
upon.
>>>>>>>>>>>>>>>> Transaction: Ticket created by jhenders at aer.com
>>>>>>>>>>>>>>>> Queue: met_help
>>>>>>>>>>>>>>>> Subject: Processing time for
point_stat with NCAR ds337 ncdf files
>>>>>>>>>>>>>>>> Owner: Nobody
>>>>>>>>>>>>>>>> Requestors: jhenders at aer.com
>>>>>>>>>>>>>>>> Status: new
>>>>>>>>>>>>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=57644 >
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hello again John,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I was wondering if you could comment on the following
behavior of
>>>>>>>>>>>>>>>> point_stat when running with PrepBufr files from the
NCAR ds337 dataset.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> When I run point_stat with daily input ncdf
observation files (generated
>>>>>>>>>>>>>>>> by pb2nc) that cover a relatively small model domain
(say, the size of
>>>>>>>>>>>>>>>> Southern New England), with a corresponding small
number of obs (several
>>>>>>>>>>>>>>>> tens of thousand individual obs valid over the entire
day), point_stat
>>>>>>>>>>>>>>>> completes successfully in less than 10 s. When I try
to verify on an
>>>>>>>>>>>>>>>> outer domain that covers CONUS, with a corresponding
large number of obs
>>>>>>>>>>>>>>>> (1-2 million), point_stat requires about 60-70
minutes to complete.
>>>>>>>>>>>>>>>> Thus, to verify hourly a full model run of 24 valid
times, over a day of
>>>>>>>>>>>>>>>> wallclock time is required.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I don't recall if this scaling behavior occurred for
v2.0 of MET.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I welcome your thoughts.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Respectfully,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> John Henderson
>>>>>>>>>>>>>>>>
>>
>
------------------------------------------------
More information about the Met_help
mailing list