[Met_help] [rt.rap.ucar.edu #45433] History for probably bug for pb2nc when compiled with intel
RAL HelpDesk {for John Halley Gotway}
met_help at ucar.edu
Fri Mar 18 14:24:04 MDT 2011
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Dear Sir/Madam:
When i compile the MET use the intel compiler on the NOAA Jet machine, it takes too long time, mostly an hour or so, for the pb2nc to complete. My prepbufr file was about 50M. And the number of points was around 200,000.
The sample file provided in the MET subdirectory data did complete very qucik. It seems the code didn't work very well with the larger files.
Here is the test data location on the Jet machine (I thought you have the account on the Jet Machine, so i pasted the path below):
/mnt/pan1/projects/wrfruc/tg1/kefeng//verification/METv3.0//bin/pb2nc \
/lfs0/projects/wrfruc/rr_retro/obs/prepbufr/20101321200.ruc2a.t12z.prepbufr.tm00.20100512 \
/mnt/lfs1/projects/wrfruc/kefeng/enkfdfiv32/pb2nc/pb2010051212.nc \
/mnt/pan1/projects/wrfruc/tg1/kefeng/enkfwrf32//rrscript/config/PB2NCConfig_defaultv3 -v 3
If it could be completed more efficiency, please let me know. Normally, the output for an size of 30M file cann't be take so long.
Thanks!
kefeng
2011-03-17
kefeng
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #45433] probably bug for pb2nc when compiled with intel
From: John Halley Gotway
Time: Thu Mar 17 15:17:00 2011
Kefeng,
I logged on to jet to take a look at this issue you've raised.
However, when I tried to run using your data, I ran into permissions
problems. Specifically, I don't have access to your config file in:
/mnt/pan1/projects/wrfruc/tg1/kefeng/enkfwrf32//rrscript/config/PB2NCConfig_defaultv3
The permissions are set to restrictively for this directory:
/mnt/pan1/projects/wrfruc/tg1/kefeng/enkfwrf32//rrscript/config
Could you please open up the permissions or email me the config file
you're using?
Thanks,
John Halley Gotway
met_help at ucar.edu
On 03/17/2011 10:35 AM, RAL HelpDesk {for kefeng} wrote:
>
> Thu Mar 17 10:35:14 2011: Request 45433 was acted upon.
> Transaction: Ticket created by kefeng at ou.edu
> Queue: met_help
> Subject: probably bug for pb2nc when compiled with intel
> Owner: Nobody
> Requestors: kefeng at ou.edu
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45433 >
>
>
> Dear Sir/Madam:
>
> When i compile the MET use the intel compiler on the NOAA Jet
machine, it takes too long time, mostly an hour or so, for the pb2nc
to complete. My prepbufr file was about 50M. And the number of points
was around 200,000.
>
> The sample file provided in the MET subdirectory data did
complete very qucik. It seems the code didn't work very well with the
larger files.
>
> Here is the test data location on the Jet machine (I thought
you have the account on the Jet Machine, so i pasted the path below):
>
>
/mnt/pan1/projects/wrfruc/tg1/kefeng//verification/METv3.0//bin/pb2nc
\
>
/lfs0/projects/wrfruc/rr_retro/obs/prepbufr/20101321200.ruc2a.t12z.prepbufr.tm00.20100512
\
> /mnt/lfs1/projects/wrfruc/kefeng/enkfdfiv32/pb2nc/pb2010051212.nc \
>
/mnt/pan1/projects/wrfruc/tg1/kefeng/enkfwrf32//rrscript/config/PB2NCConfig_defaultv3
-v 3
>
> If it could be completed more efficiency, please let me know.
Normally, the output for an size of 30M file cann't be take so long.
>
> Thanks!
>
>
>
> kefeng
>
>
>
>
>
>
> 2011-03-17
>
>
>
> kefeng
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #45433] probably bug for pb2nc when compiled with intel
From: John Halley Gotway
Time: Thu Mar 17 15:28:12 2011
Kefeng,
Since I didn't immediately have access to your config file, I tried
running using the default PB2NC config file
(METv3.0/data/config/PB2NCConfig_default). When I did, I found this
PB2NC command took
approximately 1.5 minutes to run. Please see below.
If your command is taking a lot longer than that, then there's
something funny going on. We should take a look in your config file
to see if you've made some choices that are impacting the run time.
In particular, I'm wondering if you're using a complex polyline to
define a masking region for which observations to retain. If so, that
could slow it down a lot. If you'd like to use a polyline
mask in PB2NC to specify which observations are retained, I'd suggest
using a very simple mask - perhaps even just 4 points at the corners.
And then if you'd like to verify over a specific region
using Point-Stat, you can specify the complex polyline in the Point-
Stat config file. Or better yet, run the complex polyline through
gen_poly_mask and use it's output to define the verification area
in the config file for Point-Stat.
But I'm getting ahead of myself. We need to start by looking at the
config file you're using.
Thanks,
John
[johnhg at fe3 kefeng_data_20110317]$ ./run_pb2nc.sh
Reading Config File: PB2NCConfig_default
Creating NetCDF File: pb2010051212.nc
Processing PrepBufr File:
20101321200.ruc2a.t12z.prepbufr.tm00.20100512
Blocking PrepBufr file to: /tmp/tmp_pb2nc_blk_30030_0
PrepBufr Time Center: 20100512_120000
Searching Time Window: 20100512_103000 to 20100512_133000
Processing 738456 PrepBufr messages...
5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90%
95%
Total PrepBufr Messages processed = 738456
Rejected based on message type = 0
Rejected based on station id = 0
Rejected based on valid time = 0
Rejected based on masking grid = 0
Rejected based on masking polygon = 0
Rejected based on elevation = 0
Rejected based on pb report type = 0
Rejected based on input report type = 0
Rejected based on instrument type = 0
Rejected based on zero observations = 470305
Total PrepBufr Messages retained = 268151
Total observations retained or derived = 699413
real 1m28.451s
user 1m22.867s
sys 0m4.874s
On 03/17/2011 03:17 PM, RAL HelpDesk {for John Halley Gotway} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45433 >
>
> Kefeng,
>
> I logged on to jet to take a look at this issue you've raised.
However, when I tried to run using your data, I ran into permissions
problems. Specifically, I don't have access to your config file in:
>
/mnt/pan1/projects/wrfruc/tg1/kefeng/enkfwrf32//rrscript/config/PB2NCConfig_defaultv3
>
> The permissions are set to restrictively for this directory:
> /mnt/pan1/projects/wrfruc/tg1/kefeng/enkfwrf32//rrscript/config
>
> Could you please open up the permissions or email me the config file
you're using?
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu
>
> On 03/17/2011 10:35 AM, RAL HelpDesk {for kefeng} wrote:
>>
>> Thu Mar 17 10:35:14 2011: Request 45433 was acted upon.
>> Transaction: Ticket created by kefeng at ou.edu
>> Queue: met_help
>> Subject: probably bug for pb2nc when compiled with intel
>> Owner: Nobody
>> Requestors: kefeng at ou.edu
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45433 >
>>
>>
>> Dear Sir/Madam:
>>
>> When i compile the MET use the intel compiler on the NOAA Jet
machine, it takes too long time, mostly an hour or so, for the pb2nc
to complete. My prepbufr file was about 50M. And the number of points
was around 200,000.
>>
>> The sample file provided in the MET subdirectory data did
complete very qucik. It seems the code didn't work very well with the
larger files.
>>
>> Here is the test data location on the Jet machine (I thought
you have the account on the Jet Machine, so i pasted the path below):
>>
>>
/mnt/pan1/projects/wrfruc/tg1/kefeng//verification/METv3.0//bin/pb2nc
\
>>
/lfs0/projects/wrfruc/rr_retro/obs/prepbufr/20101321200.ruc2a.t12z.prepbufr.tm00.20100512
\
>> /mnt/lfs1/projects/wrfruc/kefeng/enkfdfiv32/pb2nc/pb2010051212.nc \
>>
/mnt/pan1/projects/wrfruc/tg1/kefeng/enkfwrf32//rrscript/config/PB2NCConfig_defaultv3
-v 3
>>
>> If it could be completed more efficiency, please let me know.
Normally, the output for an size of 30M file cann't be take so long.
>>
>> Thanks!
>>
>>
>>
>> kefeng
>>
>>
>>
>>
>>
>>
>> 2011-03-17
>>
>>
>>
>> kefeng
------------------------------------------------
Subject: probably bug for pb2nc when compiled with intel
From: kefeng
Time: Thu Mar 17 16:12:11 2011
I have changed the properties now. And here is the config file
also in case you couldn't access it.
Which compiler did you use? It
seems it related with the compiler not the configure. Because i have a
previous version built by g77 in our local machine. It completed very
quick.
By the way, if it could be built by g77 in the JET machine,
could you pass me the Makefile. I have tried once, but it reported an
error says "could not found the stdc++"
Thanks!
kefeng
2011-03-17
kefeng
发件人： RAL HelpDesk {for John Halley
Gotway}
发送时间： 2011-03-17 16:28:14
收件人： kefeng at ou.edu
抄送：
met_help at ucar.edu
主题： Re: [rt.rap.ucar.edu #45433] probably bug for
pb2nc when compiled with intel
Kefeng,
Since I didn't
immediately have access to your config file, I tried running using the
default PB2NC config file (METv3.0/data/config/PB2NCConfig_default).
When I did, I found this PB2NC command took
approximately 1.5 minutes
to run. Please see below.
If your command is taking a lot longer
than that, then there's something funny going on. We should take a
look in your config file to see if you've made some choices that are
impacting the run time.
In particular, I'm wondering if you're using
a complex polyline to define a masking region for which observations
to retain. If so, that could slow it down a lot. If you'd like to
use a polyline
mask in PB2NC to specify which observations are
retained, I'd suggest using a very simple mask - perhaps even just 4
points at the corners. And then if you'd like to verify over a
specific region
using Point-Stat, you can specify the complex
polyline in the Point-Stat config file. Or better yet, run the
complex polyline through gen_poly_mask and use it's output to define
the verification area
in the config file for Point-Stat.
But I'm
getting ahead of myself. We need to start by looking at the config
file you're using.
Thanks,
John
[johnhg at fe3 kefeng_data_20110317]$
./run_pb2nc.sh
Reading Config File: PB2NCConfig_default
Creating
NetCDF File: pb2010051212.nc
Processing PrepBufr File:
20101321200.ruc2a.t12z.prepbufr.tm00.20100512
Blocking PrepBufr file
to: /tmp/tmp_pb2nc_blk_30030_0
PrepBufr Time Center:
20100512_120000
Searching Time Window: 20100512_103000 to
20100512_133000
Processing 738456 PrepBufr messages...
5% 10% 15%
20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95%
Total
PrepBufr Messages processed = 738456
Rejected based on message
type = 0
Rejected based on station id = 0
Rejected based on valid time = 0
Rejected based on masking
grid = 0
Rejected based on masking polygon = 0
Rejected based on elevation = 0
Rejected based on pb
report type = 0
Rejected based on input report type = 0
Rejected based on instrument type = 0
Rejected based on zero
observations = 470305
Total PrepBufr Messages retained =
268151
Total observations retained or derived = 699413
real
1m28.451s
user 1m22.867s
sys 0m4.874s
On 03/17/2011 03:17
PM, RAL HelpDesk {for John Halley Gotway} wrote:
>
> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45433 >
>
>
Kefeng,
>
> I logged on to jet to take a look at this issue you've
raised. However, when I tried to run using your data, I ran into
permissions problems. Specifically, I don't have access to your
config file in:
>
/mnt/pan1/projects/wrfruc/tg1/kefeng/enkfwrf32//rrscript/config/PB2NCConfig_defaultv3
>
> The permissions are set to restrictively for this directory:
>
/mnt/pan1/projects/wrfruc/tg1/kefeng/enkfwrf32//rrscript/config
>
>
Could you please open up the permissions or email me the config file
you're using?
>
> Thanks,
> John Halley Gotway
>
met_help at ucar.edu
>
> On 03/17/2011 10:35 AM, RAL HelpDesk {for
kefeng} wrote:
>>
>> Thu Mar 17 10:35:14 2011: Request 45433 was
acted upon.
>> Transaction: Ticket created by kefeng at ou.edu
>>
Queue: met_help
>> Subject: probably bug for pb2nc when compiled
with intel
>> Owner: Nobody
>> Requestors: kefeng at ou.edu
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45433 >
>>
>>
>>
Dear Sir/Madam:
>>
>> When i compile the MET use the intel
compiler on the NOAA Jet machine, it takes too long time, mostly an
hour or so, for the pb2nc to complete. My prepbufr file was about 50M.
And the number of points was around 200,000.
>>
>> The sample
file provided in the MET subdirectory data did complete very qucik. It
seems the code didn't work very well with the larger files.
>>
>>
Here is the test data location on the Jet machine (I thought you have
the account on the Jet Machine, so i pasted the path below):
>>
>>
/mnt/pan1/projects/wrfruc/tg1/kefeng//verification/METv3.0//bin/pb2nc
\
>>
/lfs0/projects/wrfruc/rr_retro/obs/prepbufr/20101321200.ruc2a.t12z.prepbufr.tm00.20100512
\
>>
/mnt/lfs1/projects/wrfruc/kefeng/enkfdfiv32/pb2nc/pb2010051212.nc \
>>
/mnt/pan1/projects/wrfruc/tg1/kefeng/enkfwrf32//rrscript/config/PB2NCConfig_defaultv3
-v 3
>>
>> If it could be completed more efficiency, please
let me know. Normally, the output for an size of 30M file cann't be
take so long.
>>
>> Thanks!
>>
>>
>>
>> kefeng
>>
>>
>>
>>
>>
>>
>> 2011-03-17
>>
>>
>>
>> kefeng
------------------------------------------------
Subject: probably bug for pb2nc when compiled with intel
From: kefeng
Time: Thu Mar 17 16:21:54 2011
And i have tracked the code. It is the "write_netcdf_hdr_data"
that cause the problem.
I have added some output in that
function.
You can copy my executable file
/mnt/pan1/projects/wrfruc/tg1/kefeng/verification/METv3.0/bin/pb2nc to
test it.
Kefeng
2011-03-17
kefeng
发件人： RAL HelpDesk {for John Halley Gotway}
发送时间： 2011-03-17
16:28:14
收件人： kefeng at ou.edu
抄送： met_help at ucar.edu
主题： Re:
[rt.rap.ucar.edu #45433] probably bug for pb2nc when compiled with
intel
Kefeng,
Since I didn't immediately have access to your
config file, I tried running using the default PB2NC config file
(METv3.0/data/config/PB2NCConfig_default). When I did, I found this
PB2NC command took
approximately 1.5 minutes to run. Please see
below.
If your command is taking a lot longer than that, then there's
something funny going on. We should take a look in your config file
to see if you've made some choices that are impacting the run time.
In particular, I'm wondering if you're using a complex polyline to
define a masking region for which observations to retain. If so, that
could slow it down a lot. If you'd like to use a polyline
mask in
PB2NC to specify which observations are retained, I'd suggest using a
very simple mask - perhaps even just 4 points at the corners. And
then if you'd like to verify over a specific region
using Point-Stat,
you can specify the complex polyline in the Point-Stat config file.
Or better yet, run the complex polyline through gen_poly_mask and use
it's output to define the verification area
in the config file for
Point-Stat.
But I'm getting ahead of myself. We need to start by
looking at the config file you're using.
Thanks,
John
[johnhg at fe3
kefeng_data_20110317]$ ./run_pb2nc.sh
Reading Config File:
PB2NCConfig_default
Creating NetCDF File: pb2010051212.nc
Processing PrepBufr File:
20101321200.ruc2a.t12z.prepbufr.tm00.20100512
Blocking PrepBufr file
to: /tmp/tmp_pb2nc_blk_30030_0
PrepBufr Time Center:
20100512_120000
Searching Time Window: 20100512_103000 to
20100512_133000
Processing 738456 PrepBufr messages...
5% 10% 15%
20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95%
Total
PrepBufr Messages processed = 738456
Rejected based on message
type = 0
Rejected based on station id = 0
Rejected based on valid time = 0
Rejected based on masking
grid = 0
Rejected based on masking polygon = 0
Rejected based on elevation = 0
Rejected based on pb
report type = 0
Rejected based on input report type = 0
Rejected based on instrument type = 0
Rejected based on zero
observations = 470305
Total PrepBufr Messages retained =
268151
Total observations retained or derived = 699413
real
1m28.451s
user 1m22.867s
sys 0m4.874s
On 03/17/2011 03:17
PM, RAL HelpDesk {for John Halley Gotway} wrote:
>
> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45433 >
>
>
Kefeng,
>
> I logged on to jet to take a look at this issue you've
raised. However, when I tried to run using your data, I ran into
permissions problems. Specifically, I don't have access to your
config file in:
>
/mnt/pan1/projects/wrfruc/tg1/kefeng/enkfwrf32//rrscript/config/PB2NCConfig_defaultv3
>
> The permissions are set to restrictively for this directory:
>
/mnt/pan1/projects/wrfruc/tg1/kefeng/enkfwrf32//rrscript/config
>
>
Could you please open up the permissions or email me the config file
you're using?
>
> Thanks,
> John Halley Gotway
>
met_help at ucar.edu
>
> On 03/17/2011 10:35 AM, RAL HelpDesk {for
kefeng} wrote:
>>
>> Thu Mar 17 10:35:14 2011: Request 45433 was
acted upon.
>> Transaction: Ticket created by kefeng at ou.edu
>>
Queue: met_help
>> Subject: probably bug for pb2nc when compiled
with intel
>> Owner: Nobody
>> Requestors: kefeng at ou.edu
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45433 >
>>
>>
>>
Dear Sir/Madam:
>>
>> When i compile the MET use the intel
compiler on the NOAA Jet machine, it takes too long time, mostly an
hour or so, for the pb2nc to complete. My prepbufr file was about 50M.
And the number of points was around 200,000.
>>
>> The sample
file provided in the MET subdirectory data did complete very qucik. It
seems the code didn't work very well with the larger files.
>>
>>
Here is the test data location on the Jet machine (I thought you have
the account on the Jet Machine, so i pasted the path below):
>>
>>
/mnt/pan1/projects/wrfruc/tg1/kefeng//verification/METv3.0//bin/pb2nc
\
>>
/lfs0/projects/wrfruc/rr_retro/obs/prepbufr/20101321200.ruc2a.t12z.prepbufr.tm00.20100512
\
>>
/mnt/lfs1/projects/wrfruc/kefeng/enkfdfiv32/pb2nc/pb2010051212.nc \
>>
/mnt/pan1/projects/wrfruc/tg1/kefeng/enkfwrf32//rrscript/config/PB2NCConfig_defaultv3
-v 3
>>
>> If it could be completed more efficiency, please
let me know. Normally, the output for an size of 30M file cann't be
take so long.
>>
>> Thanks!
>>
>>
>>
>> kefeng
>>
>>
>>
>>
>>
>>
>> 2011-03-17
>>
>>
>>
>> kefeng
------------------------------------------------
Subject: probably bug for pb2nc when compiled with intel
From: kefeng
Time: Fri Mar 18 09:52:43 2011
Dose the setting of fcst_thresh in the point_stat analysis
affect the continous statistic? I have seen the text of *mpr*, it
seems not. I just want to make for sure it dose not used as a
threshold that get rid of the observations.
Thanks!
Kefeng
2011-03-18
kefeng
发件人： RAL HelpDesk {for
kefeng}
发送时间： 2011-03-17 17:21:56
收件人： kefeng at ou.edu
抄送：
met_help at ucar.edu
主题： Re: Re: [rt.rap.ucar.edu #45433] probably bug
for pb2nc when compiled with intel
And i have tracked the
code. It is the "write_netcdf_hdr_data" that cause the problem.
I have added some output in that function.
You can copy my
executable file
/mnt/pan1/projects/wrfruc/tg1/kefeng/verification/METv3.0/bin/pb2nc to
test it.
Kefeng
2011-03-17
kefeng
发件人： RAL HelpDesk
{for John Halley Gotway}
发送时间： 2011-03-17 16:28:14
收件人：
kefeng at ou.edu
抄送： met_help at ucar.edu
主题： Re: [rt.rap.ucar.edu
#45433] probably bug for pb2nc when compiled with intel
Kefeng,
Since I didn't immediately have access to your config file, I tried
running using the default PB2NC config file
(METv3.0/data/config/PB2NCConfig_default). When I did, I found this
PB2NC command took
approximately 1.5 minutes to run. Please see
below.
If your command is taking a lot longer than that, then there's
something funny going on. We should take a look in your config file
to see if you've made some choices that are impacting the run time.
In particular, I'm wondering if you're using a complex polyline to
define a masking region for which observations to retain. If so, that
could slow it down a lot. If you'd like to use a polyline
mask in
PB2NC to specify which observations are retained, I'd suggest using a
very simple mask - perhaps even just 4 points at the corners. And
then if you'd like to verify over a specific region
using Point-Stat,
you can specify the complex polyline in the Point-Stat config file.
Or better yet, run the complex polyline through gen_poly_mask and use
it's output to define the verification area
in the config file for
Point-Stat.
But I'm getting ahead of myself. We need to start by
looking at the config file you're using.
Thanks,
John
[johnhg at fe3
kefeng_data_20110317]$ ./run_pb2nc.sh
Reading Config File:
PB2NCConfig_default
Creating NetCDF File: pb2010051212.nc
Processing PrepBufr File:
20101321200.ruc2a.t12z.prepbufr.tm00.20100512
Blocking PrepBufr file
to: /tmp/tmp_pb2nc_blk_30030_0
PrepBufr Time Center:
20100512_120000
Searching Time Window: 20100512_103000 to
20100512_133000
Processing 738456 PrepBufr messages...
5% 10% 15%
20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95%
Total
PrepBufr Messages processed = 738456
Rejected based on message
type = 0
Rejected based on station id = 0
Rejected based on valid time = 0
Rejected based on masking
grid = 0
Rejected based on masking polygon = 0
Rejected based on elevation = 0
Rejected based on pb
report type = 0
Rejected based on input report type = 0
Rejected based on instrument type = 0
Rejected based on zero
observations = 470305
Total PrepBufr Messages retained =
268151
Total observations retained or derived = 699413
real
1m28.451s
user 1m22.867s
sys 0m4.874s
On 03/17/2011 03:17
PM, RAL HelpDesk {for John Halley Gotway} wrote:
>
> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45433 >
>
>
Kefeng,
>
> I logged on to jet to take a look at this issue you've
raised. However, when I tried to run using your data, I ran into
permissions problems. Specifically, I don't have access to your
config file in:
>
/mnt/pan1/projects/wrfruc/tg1/kefeng/enkfwrf32//rrscript/config/PB2NCConfig_defaultv3
>
> The permissions are set to restrictively for this directory:
>
/mnt/pan1/projects/wrfruc/tg1/kefeng/enkfwrf32//rrscript/config
>
>
Could you please open up the permissions or email me the config file
you're using?
>
> Thanks,
> John Halley Gotway
>
met_help at ucar.edu
>
> On 03/17/2011 10:35 AM, RAL HelpDesk {for
kefeng} wrote:
>>
>> Thu Mar 17 10:35:14 2011: Request 45433 was
acted upon.
>> Transaction: Ticket created by kefeng at ou.edu
>>
Queue: met_help
>> Subject: probably bug for pb2nc when compiled
with intel
>> Owner: Nobody
>> Requestors: kefeng at ou.edu
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45433 >
>>
>>
>>
Dear Sir/Madam:
>>
>> When i compile the MET use the intel
compiler on the NOAA Jet machine, it takes too long time, mostly an
hour or so, for the pb2nc to complete. My prepbufr file was about 50M.
And the number of points was around 200,000.
>>
>> The sample
file provided in the MET subdirectory data did complete very qucik. It
seems the code didn't work very well with the larger files.
>>
>>
Here is the test data location on the Jet machine (I thought you have
the account on the Jet Machine, so i pasted the path below):
>>
>>
/mnt/pan1/projects/wrfruc/tg1/kefeng//verification/METv3.0//bin/pb2nc
\
>>
/lfs0/projects/wrfruc/rr_retro/obs/prepbufr/20101321200.ruc2a.t12z.prepbufr.tm00.20100512
\
>>
/mnt/lfs1/projects/wrfruc/kefeng/enkfdfiv32/pb2nc/pb2010051212.nc \
>>
/mnt/pan1/projects/wrfruc/tg1/kefeng/enkfwrf32//rrscript/config/PB2NCConfig_defaultv3
-v 3
>>
>> If it could be completed more efficiency, please
let me know. Normally, the output for an size of 30M file cann't be
take so long.
>>
>> Thanks!
>>
>>
>>
>> kefeng
>>
>>
>>
>>
>>
>>
>> 2011-03-17
>>
>>
>>
>> kefeng
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #45433] probably bug for pb2nc when compiled with intel
From: John Halley Gotway
Time: Fri Mar 18 10:22:47 2011
Kefeng,
You are correct. The "fcst_thresh" field is used on in the
computation of contingency table counts and statistics. It does *NOT*
affect the continuous statistics (CNT) line type or the contents of
the matched pair (MPR) line type. In fact, the header column for
"FCST_THRESH" and "OBS_THRESH" should contain NA in the MPR line
types, since no thresholding is applied to them.
John
On 03/18/2011 09:52 AM, RAL HelpDesk {for kefeng} wrote:
> Dose the setting of fcst_thresh in the point_stat analysis
affect the continous statistic? I have seen the text of *mpr*, it
seems not. I just want to make for sure it dose not used as a
threshold that get rid of the observations.
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #45433] probably bug for pb2nc when compiled with intel
From: John Halley Gotway
Time: Fri Mar 18 10:57:31 2011
Kefeng,
Actually, the runtime I reported to you yesterday was using the
executable you're using.
Now, I've rerun this case using the PB2NC Config file you sent me. I
ran once using your executable that includes a lot of debug output.
And I ran a second time linking to the version of MET in my
home directory. In both cases, the total runtime was only 1.5
minutes.
You can see the output of the tests I've done by looking in:
/whome/johnhg/MET_Help/kefeng_data_20110317/run_pb2nc.sh
/whome/johnhg/MET_Help/kefeng_data_20110317/run_pb2nc.log
As for which compiler to use on jet, I've always used the intel
compilers. I haven't tried building with GNU, but comparing the two,
intel should actually do a better job compiling and running code
than the GNU compilers. And really, I don't think the issue has to do
with the compiler.
In the command you're running, it appears that the MET executable is
located on "pan1", the input prepbufr file is on "lfs0", and you're
writing the output to "lfs1". Since you've identified the
bottleneck in the code as "write_netcdf_hdr_data", I suspect that this
is an I/O problem. I'm guessing that the problem really is in how
your cross-mounts are set up to these other machines.
Could you please try running a test to see if this is the problem?
Before running PB2NC linking to input on "lfs0" and writing output to
"lfs1", please try *copying* the input file over to "pan1".
And then from "pan1", run pb2nc and write the output to pan1 as well.
Just do this for a single case and see how long it takes to run pb2nc.
I'm guessing it'll run in under 2 minutes.
Please let me know how it goes.
Thanks,
John
On 03/18/2011 10:22 AM, RAL HelpDesk {for John Halley Gotway} wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45433 >
>
> Kefeng,
>
> You are correct. The "fcst_thresh" field is used on in the
computation of contingency table counts and statistics. It does *NOT*
affect the continuous statistics (CNT) line type or the contents of
> the matched pair (MPR) line type. In fact, the header column for
"FCST_THRESH" and "OBS_THRESH" should contain NA in the MPR line
types, since no thresholding is applied to them.
>
> John
>
> On 03/18/2011 09:52 AM, RAL HelpDesk {for kefeng} wrote:
>> Dose the setting of fcst_thresh in the point_stat analysis
affect the continous statistic? I have seen the text of *mpr*, it
seems not. I just want to make for sure it dose not used as a
threshold that get rid of the observations.
------------------------------------------------
Subject: probably bug for pb2nc when compiled with intel
From: kefeng
Time: Fri Mar 18 14:09:56 2011
Hi, John:
I have tested it. You're right! when i redirect the
output to the /pan1. It completed quickly. However, when i run pb2nc
in the /lfs1, even i copy everything to the same directory and use
relative path, it is still slow. It seems the program only act normal
in the /pan1 where i bulit.
Thanks!
Kefeng
2011-
03-18
kefeng
发件人： RAL HelpDesk {for John Halley
Gotway}
发送时间： 2011-03-18 11:57:32
收件人： kefeng at ou.edu
抄送：
met_help at ucar.edu
主题： Re: [rt.rap.ucar.edu #45433] probably bug for
pb2nc when compiled with intel
Kefeng,
Actually, the runtime I
reported to you yesterday was using the executable you're using.
Now,
I've rerun this case using the PB2NC Config file you sent me. I ran
once using your executable that includes a lot of debug output. And I
ran a second time linking to the version of MET in my
home directory.
In both cases, the total runtime was only 1.5 minutes.
You can see
the output of the tests I've done by looking in:
/whome/johnhg/MET_Help/kefeng_data_20110317/run_pb2nc.sh
/whome/johnhg/MET_Help/kefeng_data_20110317/run_pb2nc.log
As for
which compiler to use on jet, I've always used the intel compilers. I
haven't tried building with GNU, but comparing the two, intel should
actually do a better job compiling and running code
than the GNU
compilers. And really, I don't think the issue has to do with the
compiler.
In the command you're running, it appears that the MET
executable is located on "pan1", the input prepbufr file is on "lfs0",
and you're writing the output to "lfs1". Since you've identified the
bottleneck in the code as "write_netcdf_hdr_data", I suspect that this
is an I/O problem. I'm guessing that the problem really is in how
your cross-mounts are set up to these other machines.
Could you
please try running a test to see if this is the problem? Before
running PB2NC linking to input on "lfs0" and writing output to "lfs1",
please try *copying* the input file over to "pan1".
And then from
"pan1", run pb2nc and write the output to pan1 as well. Just do this
for a single case and see how long it takes to run pb2nc. I'm
guessing it'll run in under 2 minutes.
Please let me know how it
goes.
Thanks,
John
On 03/18/2011 10:22 AM, RAL HelpDesk {for John
Halley Gotway} wrote:
>
> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=45433 >
>
>
Kefeng,
>
> You are correct. The "fcst_thresh" field is used on in
the computation of contingency table counts and statistics. It does
*NOT* affect the continuous statistics (CNT) line type or the contents
of
> the matched pair (MPR) line type. In fact, the header column
for "FCST_THRESH" and "OBS_THRESH" should contain NA in the MPR line
types, since no thresholding is applied to them.
>
> John
>
> On
03/18/2011 09:52 AM, RAL HelpDesk {for kefeng} wrote:
>> Dose
the setting of fcst_thresh in the point_stat analysis affect the
continous statistic? I have seen the text of *mpr*, it seems not. I
just want to make for sure it dose not used as a threshold that get
rid of the observations.
------------------------------------------------
More information about the Met_help
mailing list