[Met_help] [rt.rap.ucar.edu #71632] History for tc_pairs execution timing
John Halley Gotway via RT
met_help at ucar.edu
Mon Apr 27 12:21:11 MDT 2015
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi there,
I'm finding that tc_pairs takes quite a long time to execute, even if there
are only a few times in the adeck file. When I thin
an adeck down to just one forecast (only one initial time), it's only
slightly faster than including all forecasts (27 seconds vs. 29
seconds), even though the adeck file size shrinks from 7.6MB to 2.1KB. This
doesn't seem right. Is there any way that tc_pairs can
be sped up? I thought it might be my compiler, so I also tried compiling on
another machine using a different compiler, but it didn't
make much difference.
Thanks,
Andy
Andrew Penny
Hurricane Model Diagnostician
SRG / National Hurricane Center
11691 SW 17th Street
Miami, FL 33165-2149
phone: 305.229.4457
email: andrew.penny at noaa.gov
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: tc_pairs execution timing
From: John Halley Gotway
Time: Fri Apr 24 17:17:13 2015
Hello Andrew,
Could you please send me your sample 7.6MB and 2.1KB files for the
ADECK
and corresponding BDECK data along with your configuration file and
your
command line calls?
I'd like to replicate the timing behavior you're seeing and that'll
help me
provide you with suggestions.
You can post it to our anonymous ftp site following these
instructions:
http://www.dtcenter.org/met/users/support/met_help.php#ftp
If you're using a very high resolution distance to land file, perhaps
it's
spending a 26.5 seconds reading that data in?
Thanks,
John Halley Gotway
met_help at ucar.edu
On Fri, Apr 24, 2015 at 3:51 PM, Andrew Penny - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> Fri Apr 24 15:51:54 2015: Request 71632 was acted upon.
> Transaction: Ticket created by andrew.penny at noaa.gov
> Queue: met_help
> Subject: tc_pairs execution timing
> Owner: Nobody
> Requestors: andrew.penny at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71632 >
>
>
> Hi there,
>
> I'm finding that tc_pairs takes quite a long time to execute, even
if there
> are only a few times in the adeck file. When I thin
> an adeck down to just one forecast (only one initial time), it's
only
> slightly faster than including all forecasts (27 seconds vs. 29
> seconds), even though the adeck file size shrinks from 7.6MB to
2.1KB. This
> doesn't seem right. Is there any way that tc_pairs can
> be sped up? I thought it might be my compiler, so I also tried
compiling on
> another machine using a different compiler, but it didn't
> make much difference.
>
> Thanks,
> Andy
>
> Andrew Penny
> Hurricane Model Diagnostician
> SRG / National Hurricane Center
> 11691 SW 17th Street
> Miami, FL 33165-2149
> phone: 305.229.4457
> email: andrew.penny at noaa.gov
>
>
------------------------------------------------
Subject: tc_pairs execution timing
From: Andrew Penny - NOAA Affiliate
Time: Mon Apr 27 05:30:59 2015
Hi John,
I think you may be on to something in suggesting the distance to land
file,
as that step does seem to take a long time. Are there low-resolution
land
files that I could also try?
I had trouble moving my files to the ftp site, so I've attached them
in
this email. The command I'm using is: time tc_pairs -adeck
aal082014_thin.dat -bdeck bal082014.dat -config TCPairsConfig -out
test
Thanks for the help,
Andy
On Fri, Apr 24, 2015 at 7:17 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Hello Andrew,
>
> Could you please send me your sample 7.6MB and 2.1KB files for the
ADECK
> and corresponding BDECK data along with your configuration file and
your
> command line calls?
>
> I'd like to replicate the timing behavior you're seeing and that'll
help me
> provide you with suggestions.
>
> You can post it to our anonymous ftp site following these
instructions:
> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> If you're using a very high resolution distance to land file,
perhaps it's
> spending a 26.5 seconds reading that data in?
>
> Thanks,
> John Halley Gotway
> met_help at ucar.edu
>
> On Fri, Apr 24, 2015 at 3:51 PM, Andrew Penny - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Fri Apr 24 15:51:54 2015: Request 71632 was acted upon.
> > Transaction: Ticket created by andrew.penny at noaa.gov
> > Queue: met_help
> > Subject: tc_pairs execution timing
> > Owner: Nobody
> > Requestors: andrew.penny at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71632 >
> >
> >
> > Hi there,
> >
> > I'm finding that tc_pairs takes quite a long time to execute, even
if
> there
> > are only a few times in the adeck file. When I thin
> > an adeck down to just one forecast (only one initial time), it's
only
> > slightly faster than including all forecasts (27 seconds vs. 29
> > seconds), even though the adeck file size shrinks from 7.6MB to
2.1KB.
> This
> > doesn't seem right. Is there any way that tc_pairs can
> > be sped up? I thought it might be my compiler, so I also tried
compiling
> on
> > another machine using a different compiler, but it didn't
> > make much difference.
> >
> > Thanks,
> > Andy
> >
> > Andrew Penny
> > Hurricane Model Diagnostician
> > SRG / National Hurricane Center
> > 11691 SW 17th Street
> > Miami, FL 33165-2149
> > phone: 305.229.4457
> > email: andrew.penny at noaa.gov
> >
> >
>
>
--
Andrew Penny
Hurricane Model Diagnostician
SRG / National Hurricane Center
11691 SW 17th Street
Miami, FL 33165-2149
phone: 305.229.4457
email: andrew.penny at noaa.gov
------------------------------------------------
Subject: tc_pairs execution timing
From: John Halley Gotway
Time: Mon Apr 27 09:50:40 2015
Andy,
You can easily test this out. Just edit your TC-Pairs configuration
file
by setting "dland_file" to an empty string:
dland_file = "";
Then rerun and compare the timing.
With no distance to land file specified, the "ADLAND" and "BDLAND"
columns
in the output will contain missing data (NA). If you don't care about
the
distance to land, you can leave that as-is. If you do, you could add
in a
more coarse distance to land file. The met-5.0 release includes two
1/10th
degree files in met-5.0/data/tc_data/dland_nw_hem_tenth_degree.nc and
dland_global_tenth_degree.nc. The former is for the northwest
hemisphere
and the latter is the for the whole globe.
This is just the output of the tc_dland tool. Here's an example of
running
the tc_dland tool to generate a global 0.5 degree distance to land
file.
You could try using that in your tc_pairs runs to see how it affects
the
timing:
tc_dland dland_global_half_degree.nc -grid 0.0 -180.0 0.5 0.5 121
361
Please let me know how it goes.
Thanks,
John
On Mon, Apr 27, 2015 at 5:31 AM, Andrew Penny - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71632 >
>
> Hi John,
>
> I think you may be on to something in suggesting the distance to
land file,
> as that step does seem to take a long time. Are there low-resolution
land
> files that I could also try?
>
> I had trouble moving my files to the ftp site, so I've attached them
in
> this email. The command I'm using is: time tc_pairs -adeck
> aal082014_thin.dat -bdeck bal082014.dat -config TCPairsConfig -out
test
>
> Thanks for the help,
> Andy
>
> On Fri, Apr 24, 2015 at 7:17 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Hello Andrew,
> >
> > Could you please send me your sample 7.6MB and 2.1KB files for the
ADECK
> > and corresponding BDECK data along with your configuration file
and your
> > command line calls?
> >
> > I'd like to replicate the timing behavior you're seeing and
that'll help
> me
> > provide you with suggestions.
> >
> > You can post it to our anonymous ftp site following these
instructions:
> > http://www.dtcenter.org/met/users/support/met_help.php#ftp
> >
> > If you're using a very high resolution distance to land file,
perhaps
> it's
> > spending a 26.5 seconds reading that data in?
> >
> > Thanks,
> > John Halley Gotway
> > met_help at ucar.edu
> >
> > On Fri, Apr 24, 2015 at 3:51 PM, Andrew Penny - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Fri Apr 24 15:51:54 2015: Request 71632 was acted upon.
> > > Transaction: Ticket created by andrew.penny at noaa.gov
> > > Queue: met_help
> > > Subject: tc_pairs execution timing
> > > Owner: Nobody
> > > Requestors: andrew.penny at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71632
> >
> > >
> > >
> > > Hi there,
> > >
> > > I'm finding that tc_pairs takes quite a long time to execute,
even if
> > there
> > > are only a few times in the adeck file. When I thin
> > > an adeck down to just one forecast (only one initial time), it's
only
> > > slightly faster than including all forecasts (27 seconds vs. 29
> > > seconds), even though the adeck file size shrinks from 7.6MB to
2.1KB.
> > This
> > > doesn't seem right. Is there any way that tc_pairs can
> > > be sped up? I thought it might be my compiler, so I also tried
> compiling
> > on
> > > another machine using a different compiler, but it didn't
> > > make much difference.
> > >
> > > Thanks,
> > > Andy
> > >
> > > Andrew Penny
> > > Hurricane Model Diagnostician
> > > SRG / National Hurricane Center
> > > 11691 SW 17th Street
> > > Miami, FL 33165-2149
> > > phone: 305.229.4457
> > > email: andrew.penny at noaa.gov
> > >
> > >
> >
> >
>
>
> --
> Andrew Penny
> Hurricane Model Diagnostician
> SRG / National Hurricane Center
> 11691 SW 17th Street
> Miami, FL 33165-2149
> phone: 305.229.4457
> email: andrew.penny at noaa.gov
>
>
------------------------------------------------
Subject: tc_pairs execution timing
From: John Halley Gotway
Time: Mon Apr 27 10:08:23 2015
Andy,
Sorry about that - I sent you the tc_dland command for only the
northwest
hemisphere. Here's a global one:
tc_dland dland_global_half_degree.nc -grid -90.0 -180.0 0.5 0.5 361
721
-v 3
It takes long time to run (about 13 min on my machine), but you only
need
to run it once. The resulting file size is 3 Mb instead of 75 Mb for
the
global 1/10th degree one.
Thanks,
John
On Mon, Apr 27, 2015 at 9:50 AM, John Halley Gotway <johnhg at ucar.edu>
wrote:
> Andy,
>
> You can easily test this out. Just edit your TC-Pairs configuration
file
> by setting "dland_file" to an empty string:
> dland_file = "";
>
> Then rerun and compare the timing.
>
> With no distance to land file specified, the "ADLAND" and "BDLAND"
columns
> in the output will contain missing data (NA). If you don't care
about the
> distance to land, you can leave that as-is. If you do, you could
add in a
> more coarse distance to land file. The met-5.0 release includes two
1/10th
> degree files in met-5.0/data/tc_data/dland_nw_hem_tenth_degree.nc
and
> dland_global_tenth_degree.nc. The former is for the northwest
hemisphere
> and the latter is the for the whole globe.
>
> This is just the output of the tc_dland tool. Here's an example of
> running the tc_dland tool to generate a global 0.5 degree distance
to land
> file. You could try using that in your tc_pairs runs to see how it
affects
> the timing:
> tc_dland dland_global_half_degree.nc -grid 0.0 -180.0 0.5 0.5
121 361
>
> Please let me know how it goes.
>
> Thanks,
> John
>
> On Mon, Apr 27, 2015 at 5:31 AM, Andrew Penny - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71632 >
>>
>> Hi John,
>>
>> I think you may be on to something in suggesting the distance to
land
>> file,
>> as that step does seem to take a long time. Are there low-
resolution land
>> files that I could also try?
>>
>> I had trouble moving my files to the ftp site, so I've attached
them in
>> this email. The command I'm using is: time tc_pairs -adeck
>> aal082014_thin.dat -bdeck bal082014.dat -config TCPairsConfig -out
test
>>
>> Thanks for the help,
>> Andy
>>
>> On Fri, Apr 24, 2015 at 7:17 PM, John Halley Gotway via RT <
>> met_help at ucar.edu> wrote:
>>
>> > Hello Andrew,
>> >
>> > Could you please send me your sample 7.6MB and 2.1KB files for
the ADECK
>> > and corresponding BDECK data along with your configuration file
and your
>> > command line calls?
>> >
>> > I'd like to replicate the timing behavior you're seeing and
that'll
>> help me
>> > provide you with suggestions.
>> >
>> > You can post it to our anonymous ftp site following these
instructions:
>> > http://www.dtcenter.org/met/users/support/met_help.php#ftp
>> >
>> > If you're using a very high resolution distance to land file,
perhaps
>> it's
>> > spending a 26.5 seconds reading that data in?
>> >
>> > Thanks,
>> > John Halley Gotway
>> > met_help at ucar.edu
>> >
>> > On Fri, Apr 24, 2015 at 3:51 PM, Andrew Penny - NOAA Affiliate
via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > Fri Apr 24 15:51:54 2015: Request 71632 was acted upon.
>> > > Transaction: Ticket created by andrew.penny at noaa.gov
>> > > Queue: met_help
>> > > Subject: tc_pairs execution timing
>> > > Owner: Nobody
>> > > Requestors: andrew.penny at noaa.gov
>> > > Status: new
>> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71632
>> >
>> > >
>> > >
>> > > Hi there,
>> > >
>> > > I'm finding that tc_pairs takes quite a long time to execute,
even if
>> > there
>> > > are only a few times in the adeck file. When I thin
>> > > an adeck down to just one forecast (only one initial time),
it's only
>> > > slightly faster than including all forecasts (27 seconds vs. 29
>> > > seconds), even though the adeck file size shrinks from 7.6MB to
2.1KB.
>> > This
>> > > doesn't seem right. Is there any way that tc_pairs can
>> > > be sped up? I thought it might be my compiler, so I also tried
>> compiling
>> > on
>> > > another machine using a different compiler, but it didn't
>> > > make much difference.
>> > >
>> > > Thanks,
>> > > Andy
>> > >
>> > > Andrew Penny
>> > > Hurricane Model Diagnostician
>> > > SRG / National Hurricane Center
>> > > 11691 SW 17th Street
>> > > Miami, FL 33165-2149
>> > > phone: 305.229.4457
>> > > email: andrew.penny at noaa.gov
>> > >
>> > >
>> >
>> >
>>
>>
>> --
>> Andrew Penny
>> Hurricane Model Diagnostician
>> SRG / National Hurricane Center
>> 11691 SW 17th Street
>> Miami, FL 33165-2149
>> phone: 305.229.4457
>> email: andrew.penny at noaa.gov
>>
>>
>
------------------------------------------------
Subject: tc_pairs execution timing
From: Andrew Penny - NOAA Affiliate
Time: Mon Apr 27 12:00:36 2015
Thanks John. The distance to land file was the culprit. Another timing
test
revealed the following:
global: 31 seconds
nw_hem: 8.5 seconds
no distance to land file: 4.7 seconds
Having the distance to land is not that important for what I'm doing
now,
so I'll set the string to blank.
Thanks for your help,
Andy
On Mon, Apr 27, 2015 at 12:08 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Andy,
>
> Sorry about that - I sent you the tc_dland command for only the
northwest
> hemisphere. Here's a global one:
> tc_dland dland_global_half_degree.nc -grid -90.0 -180.0 0.5 0.5
361 721
> -v 3
>
> It takes long time to run (about 13 min on my machine), but you only
need
> to run it once. The resulting file size is 3 Mb instead of 75 Mb
for the
> global 1/10th degree one.
>
> Thanks,
> John
>
> On Mon, Apr 27, 2015 at 9:50 AM, John Halley Gotway
<johnhg at ucar.edu>
> wrote:
>
> > Andy,
> >
> > You can easily test this out. Just edit your TC-Pairs
configuration file
> > by setting "dland_file" to an empty string:
> > dland_file = "";
> >
> > Then rerun and compare the timing.
> >
> > With no distance to land file specified, the "ADLAND" and "BDLAND"
> columns
> > in the output will contain missing data (NA). If you don't care
about
> the
> > distance to land, you can leave that as-is. If you do, you could
add in
> a
> > more coarse distance to land file. The met-5.0 release includes
two
> 1/10th
> > degree files in met-5.0/data/tc_data/dland_nw_hem_tenth_degree.nc
and
> > dland_global_tenth_degree.nc. The former is for the northwest
> hemisphere
> > and the latter is the for the whole globe.
> >
> > This is just the output of the tc_dland tool. Here's an example
of
> > running the tc_dland tool to generate a global 0.5 degree distance
to
> land
> > file. You could try using that in your tc_pairs runs to see how
it
> affects
> > the timing:
> > tc_dland dland_global_half_degree.nc -grid 0.0 -180.0 0.5 0.5
121
> 361
> >
> > Please let me know how it goes.
> >
> > Thanks,
> > John
> >
> > On Mon, Apr 27, 2015 at 5:31 AM, Andrew Penny - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71632 >
> >>
> >> Hi John,
> >>
> >> I think you may be on to something in suggesting the distance to
land
> >> file,
> >> as that step does seem to take a long time. Are there low-
resolution
> land
> >> files that I could also try?
> >>
> >> I had trouble moving my files to the ftp site, so I've attached
them in
> >> this email. The command I'm using is: time tc_pairs -adeck
> >> aal082014_thin.dat -bdeck bal082014.dat -config TCPairsConfig
-out test
> >>
> >> Thanks for the help,
> >> Andy
> >>
> >> On Fri, Apr 24, 2015 at 7:17 PM, John Halley Gotway via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> > Hello Andrew,
> >> >
> >> > Could you please send me your sample 7.6MB and 2.1KB files for
the
> ADECK
> >> > and corresponding BDECK data along with your configuration file
and
> your
> >> > command line calls?
> >> >
> >> > I'd like to replicate the timing behavior you're seeing and
that'll
> >> help me
> >> > provide you with suggestions.
> >> >
> >> > You can post it to our anonymous ftp site following these
> instructions:
> >> > http://www.dtcenter.org/met/users/support/met_help.php#ftp
> >> >
> >> > If you're using a very high resolution distance to land file,
perhaps
> >> it's
> >> > spending a 26.5 seconds reading that data in?
> >> >
> >> > Thanks,
> >> > John Halley Gotway
> >> > met_help at ucar.edu
> >> >
> >> > On Fri, Apr 24, 2015 at 3:51 PM, Andrew Penny - NOAA Affiliate
via RT
> <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > Fri Apr 24 15:51:54 2015: Request 71632 was acted upon.
> >> > > Transaction: Ticket created by andrew.penny at noaa.gov
> >> > > Queue: met_help
> >> > > Subject: tc_pairs execution timing
> >> > > Owner: Nobody
> >> > > Requestors: andrew.penny at noaa.gov
> >> > > Status: new
> >> > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=71632
> >> >
> >> > >
> >> > >
> >> > > Hi there,
> >> > >
> >> > > I'm finding that tc_pairs takes quite a long time to execute,
even
> if
> >> > there
> >> > > are only a few times in the adeck file. When I thin
> >> > > an adeck down to just one forecast (only one initial time),
it's
> only
> >> > > slightly faster than including all forecasts (27 seconds vs.
29
> >> > > seconds), even though the adeck file size shrinks from 7.6MB
to
> 2.1KB.
> >> > This
> >> > > doesn't seem right. Is there any way that tc_pairs can
> >> > > be sped up? I thought it might be my compiler, so I also
tried
> >> compiling
> >> > on
> >> > > another machine using a different compiler, but it didn't
> >> > > make much difference.
> >> > >
> >> > > Thanks,
> >> > > Andy
> >> > >
> >> > > Andrew Penny
> >> > > Hurricane Model Diagnostician
> >> > > SRG / National Hurricane Center
> >> > > 11691 SW 17th Street
> >> > > Miami, FL 33165-2149
> >> > > phone: 305.229.4457
> >> > > email: andrew.penny at noaa.gov
> >> > >
> >> > >
> >> >
> >> >
> >>
> >>
> >> --
> >> Andrew Penny
> >> Hurricane Model Diagnostician
> >> SRG / National Hurricane Center
> >> 11691 SW 17th Street
> >> Miami, FL 33165-2149
> >> phone: 305.229.4457
> >> email: andrew.penny at noaa.gov
> >>
> >>
> >
>
>
--
Andrew Penny
Hurricane Model Diagnostician
SRG / National Hurricane Center
11691 SW 17th Street
Miami, FL 33165-2149
phone: 305.229.4457
email: andrew.penny at noaa.gov
------------------------------------------------
More information about the Met_help
mailing list