[Met_help] [rt.rap.ucar.edu #76891] History for observation count?
Julie Prestopnik via RT
met_help at ucar.edu
Fri Sep 2 10:53:14 MDT 2016
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
I have a question. I am running the ascii2nc tool, and was wondering if there is an amount of observations limit? For some reason it seems as though once I get above ~85000 obs, the system takes an extremely long time. For example, below that it takes less than 5 seconds to create the netcdf file, once I exceed that amount of observations it suddenly take 5 minutes or longer. The same goes for using the point_stat tool.
V/R,
STEVEN D. BLIUJUS, GS-07, DAFC
16 WS/WXE
DSN: 272-7151
COMM: 402-232-7151
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #76891] observation count?
From: Julie Prestopnik
Time: Thu Jun 30 09:57:25 2016
Hi Steven.
Unfortunately, John is out of the office this week and the beginning
of next week. He is more knowledgeable and familiar with issues like
this one. I would guess that there is not an observation limit, but
do not know of a reason for the slow down after you get above ~85000
obs.
Could you please post your data to our ftp site?
You can follow the instructions here:
http://www.dtcenter.org/met/users/support/met_help.php#ftp
Please also send me that command that you are using in running
ascii2nc and let me know what version of MET you are using. I'll take
a look and see what I can figure out.
Thanks,
Julie
On Thu, Jun 30, 2016 at 8:43 AM, BLIUJUS, STEVEN D GS-07 USAF ACC 16
WS/WXE via RT <met_help at ucar.edu> wrote:
>
>
> Thu Jun 30 08:43:21 2016: Request 76891 was acted upon.
> Transaction: Ticket created by steven.bliujus.1 at us.af.mil
> Queue: met_help
> Subject: observation count?
> Owner: Nobody
> Requestors: steven.bliujus.1 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891 >
>
>
> I have a question. I am running the ascii2nc tool, and was wondering
if there is an amount of observations limit? For some reason it seems
as though once I get above ~85000 obs, the system takes an extremely
long time. For example, below that it takes less than 5 seconds to
create the netcdf file, once I exceed that amount of observations it
suddenly take 5 minutes or longer. The same goes for using the
point_stat tool.
>
> V/R,
>
> STEVEN D. BLIUJUS, GS-07, DAFC
> 16 WS/WXE
> DSN: 272-7151
> COMM: 402-232-7151
>
>
>
>
--
Julie Prestopnik
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #76891] observation count?
From: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
Time: Thu Jun 30 10:54:41 2016
I put in two files and two config files I used (one for the ascii
portion and the other the point stat...the first one is one under
85000, the other is more, much more (~385000) points but if you want
to hack it down to like 90000 it would do the same. The command I'm
using is:
Ascii2nc Modis.txt Modis.nx -format "met_point"
The version is 5.1
Steve
-----Original Message-----
From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
Sent: Thursday, June 30, 2016 10:57 AM
To: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
<steven.bliujus.1 at us.af.mil>
Subject: Re: [rt.rap.ucar.edu #76891] observation count?
Hi Steven.
Unfortunately, John is out of the office this week and the beginning
of next week. He is more knowledgeable and familiar with issues like
this one. I would guess that there is not an observation limit, but
do not know of a reason for the slow down after you get above ~85000
obs.
Could you please post your data to our ftp site?
You can follow the instructions here:
http://www.dtcenter.org/met/users/support/met_help.php#ftp
Please also send me that command that you are using in running
ascii2nc and let me know what version of MET you are using. I'll take
a look and see what I can figure out.
Thanks,
Julie
On Thu, Jun 30, 2016 at 8:43 AM, BLIUJUS, STEVEN D GS-07 USAF ACC 16
WS/WXE via RT <met_help at ucar.edu> wrote:
>
>
> Thu Jun 30 08:43:21 2016: Request 76891 was acted upon.
> Transaction: Ticket created by steven.bliujus.1 at us.af.mil
> Queue: met_help
> Subject: observation count?
> Owner: Nobody
> Requestors: steven.bliujus.1 at us.af.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891
> >
>
>
> I have a question. I am running the ascii2nc tool, and was wondering
if there is an amount of observations limit? For some reason it seems
as though once I get above ~85000 obs, the system takes an extremely
long time. For example, below that it takes less than 5 seconds to
create the netcdf file, once I exceed that amount of observations it
suddenly take 5 minutes or longer. The same goes for using the
point_stat tool.
>
> V/R,
>
> STEVEN D. BLIUJUS, GS-07, DAFC
> 16 WS/WXE
> DSN: 272-7151
> COMM: 402-232-7151
>
>
>
>
--
Julie Prestopnik
National Center for Atmospheric Research Research Applications
Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #76891] observation count?
From: Julie Prestopnik
Time: Thu Jun 30 11:59:56 2016
Hi Steve.
I ran ascii2nc using varied numbers of observations from Modis.txt. I
used 50000, 80000, 90000, 389751 (full file). I ran the code using
the "time" command to get the time it took to run.
So, for example, I ran:
/usr/bin/time --verbose --output ascii2nc_time_50000.txt ascii2nc
Modis_50000.txt Modis_50000.nc -format "met_point" -config
./Ascii2NcConfig_default
for each file.
The times were the following:
50000 obs:
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:07.63
80000 obs:
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:13.08
90000 obs:
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.02
389751 obs:
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:59.76
Those numbers seem reasonable and in line with the number of
observations (389751 / 50000 = 7.79502; 7.79502 * 7.63 seconds =
59.476 seconds ) You mentioned that the system was taking an
"extremely long time". Can you please tell me how long it was taking
for the 389751 obs on your machine?
Thanks,
Julie
On Thu, Jun 30, 2016 at 10:54 AM, BLIUJUS, STEVEN D GS-07 USAF ACC 16
WS/WXE via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891 >
>
> I put in two files and two config files I used (one for the ascii
portion and the other the point stat...the first one is one under
85000, the other is more, much more (~385000) points but if you want
to hack it down to like 90000 it would do the same. The command I'm
using is:
>
> Ascii2nc Modis.txt Modis.nx -format "met_point"
>
> The version is 5.1
>
> Steve
>
> -----Original Message-----
> From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, June 30, 2016 10:57 AM
> To: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
<steven.bliujus.1 at us.af.mil>
> Subject: Re: [rt.rap.ucar.edu #76891] observation count?
>
> Hi Steven.
>
> Unfortunately, John is out of the office this week and the beginning
of next week. He is more knowledgeable and familiar with issues like
this one. I would guess that there is not an observation limit, but
do not know of a reason for the slow down after you get above ~85000
obs.
>
> Could you please post your data to our ftp site?
>
> You can follow the instructions here:
> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Please also send me that command that you are using in running
ascii2nc and let me know what version of MET you are using. I'll take
a look and see what I can figure out.
>
> Thanks,
> Julie
>
> On Thu, Jun 30, 2016 at 8:43 AM, BLIUJUS, STEVEN D GS-07 USAF ACC 16
WS/WXE via RT <met_help at ucar.edu> wrote:
>>
>>
>> Thu Jun 30 08:43:21 2016: Request 76891 was acted upon.
>> Transaction: Ticket created by steven.bliujus.1 at us.af.mil
>> Queue: met_help
>> Subject: observation count?
>> Owner: Nobody
>> Requestors: steven.bliujus.1 at us.af.mil
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891
>> >
>>
>>
>> I have a question. I am running the ascii2nc tool, and was
wondering if there is an amount of observations limit? For some reason
it seems as though once I get above ~85000 obs, the system takes an
extremely long time. For example, below that it takes less than 5
seconds to create the netcdf file, once I exceed that amount of
observations it suddenly take 5 minutes or longer. The same goes for
using the point_stat tool.
>>
>> V/R,
>>
>> STEVEN D. BLIUJUS, GS-07, DAFC
>> 16 WS/WXE
>> DSN: 272-7151
>> COMM: 402-232-7151
>>
>>
>>
>>
>
>
>
> --
> Julie Prestopnik
> National Center for Atmospheric Research Research Applications
Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
>
>
>
--
Julie Prestopnik
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #76891] observation count?
From: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
Time: Thu Jun 30 12:42:25 2016
For 50000: Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.59
For 80000: Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.61
For 90000: Elapsed (wall clock) time (h:mm:ss or m:ss): 1:34.41
For 100000: Elapsed (wall clock) time (h:mm:ss or m:ss): 7:22.46
So as you can see it is exponentially growing in length of time which
is why I wont run the 300000 file again. I also noted that the
percent of CPU this job got kept dropping.
-----Original Message-----
From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
Sent: Thursday, June 30, 2016 1:00 PM
To: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
<steven.bliujus.1 at us.af.mil>
Subject: Re: [rt.rap.ucar.edu #76891] observation count?
Hi Steve.
I ran ascii2nc using varied numbers of observations from Modis.txt. I
used 50000, 80000, 90000, 389751 (full file). I ran the code using
the "time" command to get the time it took to run.
So, for example, I ran:
/usr/bin/time --verbose --output ascii2nc_time_50000.txt ascii2nc
Modis_50000.txt Modis_50000.nc -format "met_point" -config
./Ascii2NcConfig_default
for each file.
The times were the following:
50000 obs:
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:07.63
80000 obs:
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:13.08
90000 obs:
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.02
389751 obs:
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:59.76
Those numbers seem reasonable and in line with the number of
observations (389751 / 50000 = 7.79502; 7.79502 * 7.63 seconds =
59.476 seconds ) You mentioned that the system was taking an
"extremely long time". Can you please tell me how long it was taking
for the 389751 obs on your machine?
Thanks,
Julie
On Thu, Jun 30, 2016 at 10:54 AM, BLIUJUS, STEVEN D GS-07 USAF ACC 16
WS/WXE via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891 >
>
> I put in two files and two config files I used (one for the ascii
portion and the other the point stat...the first one is one under
85000, the other is more, much more (~385000) points but if you want
to hack it down to like 90000 it would do the same. The command I'm
using is:
>
> Ascii2nc Modis.txt Modis.nx -format "met_point"
>
> The version is 5.1
>
> Steve
>
> -----Original Message-----
> From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, June 30, 2016 10:57 AM
> To: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
<steven.bliujus.1 at us.af.mil>
> Subject: Re: [rt.rap.ucar.edu #76891] observation count?
>
> Hi Steven.
>
> Unfortunately, John is out of the office this week and the beginning
of next week. He is more knowledgeable and familiar with issues like
this one. I would guess that there is not an observation limit, but
do not know of a reason for the slow down after you get above ~85000
obs.
>
> Could you please post your data to our ftp site?
>
> You can follow the instructions here:
> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>
> Please also send me that command that you are using in running
ascii2nc and let me know what version of MET you are using. I'll take
a look and see what I can figure out.
>
> Thanks,
> Julie
>
> On Thu, Jun 30, 2016 at 8:43 AM, BLIUJUS, STEVEN D GS-07 USAF ACC 16
WS/WXE via RT <met_help at ucar.edu> wrote:
>>
>>
>> Thu Jun 30 08:43:21 2016: Request 76891 was acted upon.
>> Transaction: Ticket created by steven.bliujus.1 at us.af.mil
>> Queue: met_help
>> Subject: observation count?
>> Owner: Nobody
>> Requestors: steven.bliujus.1 at us.af.mil
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891
>> >
>>
>>
>> I have a question. I am running the ascii2nc tool, and was
wondering if there is an amount of observations limit? For some reason
it seems as though once I get above ~85000 obs, the system takes an
extremely long time. For example, below that it takes less than 5
seconds to create the netcdf file, once I exceed that amount of
observations it suddenly take 5 minutes or longer. The same goes for
using the point_stat tool.
>>
>> V/R,
>>
>> STEVEN D. BLIUJUS, GS-07, DAFC
>> 16 WS/WXE
>> DSN: 272-7151
>> COMM: 402-232-7151
>>
>>
>>
>>
>
>
>
> --
> Julie Prestopnik
> National Center for Atmospheric Research Research Applications
Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
>
>
>
--
Julie Prestopnik
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #76891] observation count?
From: Julie Prestopnik
Time: Thu Jun 30 14:08:47 2016
Thank you for sending these times.
I was thinking that maybe getting the latest set of patches for
METv5.1 would help. However, I was just looking over the "Summary of
Recommended Updates" on the "Known Issues and Fixes for METv5.1" page:
http://www.dtcenter.org/met/users/support/known_issues/METv5.1/index.php
and I don't see anything there that might help with the problem you
are experiencing. Unfortunately, I think I need to defer to John on
this issue. I will ask him to take a look when he gets back.
Thank you for your patience regarding this issue.
Julie
On Thu, Jun 30, 2016 at 12:42 PM, BLIUJUS, STEVEN D GS-07 USAF ACC 16
WS/WXE via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891 >
>
> For 50000: Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.59
> For 80000: Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.61
> For 90000: Elapsed (wall clock) time (h:mm:ss or m:ss): 1:34.41
> For 100000: Elapsed (wall clock) time (h:mm:ss or m:ss): 7:22.46
>
> So as you can see it is exponentially growing in length of time
which is why I wont run the 300000 file again. I also noted that the
percent of CPU this job got kept dropping.
>
>
>
> -----Original Message-----
> From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, June 30, 2016 1:00 PM
> To: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
<steven.bliujus.1 at us.af.mil>
> Subject: Re: [rt.rap.ucar.edu #76891] observation count?
>
> Hi Steve.
>
> I ran ascii2nc using varied numbers of observations from Modis.txt.
I
> used 50000, 80000, 90000, 389751 (full file). I ran the code using
> the "time" command to get the time it took to run.
>
> So, for example, I ran:
>
> /usr/bin/time --verbose --output ascii2nc_time_50000.txt ascii2nc
> Modis_50000.txt Modis_50000.nc -format "met_point" -config
> ./Ascii2NcConfig_default
>
> for each file.
>
> The times were the following:
>
> 50000 obs:
> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:07.63
>
> 80000 obs:
> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:13.08
>
> 90000 obs:
> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.02
>
> 389751 obs:
> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:59.76
>
>
> Those numbers seem reasonable and in line with the number of
> observations (389751 / 50000 = 7.79502; 7.79502 * 7.63 seconds =
> 59.476 seconds ) You mentioned that the system was taking an
> "extremely long time". Can you please tell me how long it was
taking
> for the 389751 obs on your machine?
>
> Thanks,
> Julie
>
> On Thu, Jun 30, 2016 at 10:54 AM, BLIUJUS, STEVEN D GS-07 USAF ACC
16
> WS/WXE via RT <met_help at ucar.edu> wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891 >
>>
>> I put in two files and two config files I used (one for the ascii
portion and the other the point stat...the first one is one under
85000, the other is more, much more (~385000) points but if you want
to hack it down to like 90000 it would do the same. The command I'm
using is:
>>
>> Ascii2nc Modis.txt Modis.nx -format "met_point"
>>
>> The version is 5.1
>>
>> Steve
>>
>> -----Original Message-----
>> From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
>> Sent: Thursday, June 30, 2016 10:57 AM
>> To: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
<steven.bliujus.1 at us.af.mil>
>> Subject: Re: [rt.rap.ucar.edu #76891] observation count?
>>
>> Hi Steven.
>>
>> Unfortunately, John is out of the office this week and the
beginning of next week. He is more knowledgeable and familiar with
issues like this one. I would guess that there is not an observation
limit, but do not know of a reason for the slow down after you get
above ~85000 obs.
>>
>> Could you please post your data to our ftp site?
>>
>> You can follow the instructions here:
>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>
>> Please also send me that command that you are using in running
ascii2nc and let me know what version of MET you are using. I'll take
a look and see what I can figure out.
>>
>> Thanks,
>> Julie
>>
>> On Thu, Jun 30, 2016 at 8:43 AM, BLIUJUS, STEVEN D GS-07 USAF ACC
16 WS/WXE via RT <met_help at ucar.edu> wrote:
>>>
>>>
>>> Thu Jun 30 08:43:21 2016: Request 76891 was acted upon.
>>> Transaction: Ticket created by steven.bliujus.1 at us.af.mil
>>> Queue: met_help
>>> Subject: observation count?
>>> Owner: Nobody
>>> Requestors: steven.bliujus.1 at us.af.mil
>>> Status: new
>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891
>>> >
>>>
>>>
>>> I have a question. I am running the ascii2nc tool, and was
wondering if there is an amount of observations limit? For some reason
it seems as though once I get above ~85000 obs, the system takes an
extremely long time. For example, below that it takes less than 5
seconds to create the netcdf file, once I exceed that amount of
observations it suddenly take 5 minutes or longer. The same goes for
using the point_stat tool.
>>>
>>> V/R,
>>>
>>> STEVEN D. BLIUJUS, GS-07, DAFC
>>> 16 WS/WXE
>>> DSN: 272-7151
>>> COMM: 402-232-7151
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> Julie Prestopnik
>> National Center for Atmospheric Research Research Applications
Laboratory
>> Phone: 303.497.8399
>> Email: jpresto at ucar.edu
>>
>>
>>
>>
>
>
>
> --
> Julie Prestopnik
> National Center for Atmospheric Research
> Research Applications Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
>
>
>
--
Julie Prestopnik
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #76891] observation count?
From: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
Time: Thu Jun 30 14:11:55 2016
Thank you for your assistance Julie.
-----Original Message-----
From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
Sent: Thursday, June 30, 2016 3:09 PM
To: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
<steven.bliujus.1 at us.af.mil>
Subject: Re: [rt.rap.ucar.edu #76891] observation count?
Thank you for sending these times.
I was thinking that maybe getting the latest set of patches for
METv5.1 would help. However, I was just looking over the "Summary of
Recommended Updates" on the "Known Issues and Fixes for METv5.1" page:
http://www.dtcenter.org/met/users/support/known_issues/METv5.1/index.php
and I don't see anything there that might help with the problem you
are experiencing. Unfortunately, I think I need to defer to John on
this issue. I will ask him to take a look when he gets back.
Thank you for your patience regarding this issue.
Julie
On Thu, Jun 30, 2016 at 12:42 PM, BLIUJUS, STEVEN D GS-07 USAF ACC 16
WS/WXE via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891 >
>
> For 50000: Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.59 For
> 80000: Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.61 For
90000:
> Elapsed (wall clock) time (h:mm:ss or m:ss): 1:34.41 For 100000:
> Elapsed (wall clock) time (h:mm:ss or m:ss): 7:22.46
>
> So as you can see it is exponentially growing in length of time
which is why I wont run the 300000 file again. I also noted that the
percent of CPU this job got kept dropping.
>
>
>
> -----Original Message-----
> From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, June 30, 2016 1:00 PM
> To: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
> <steven.bliujus.1 at us.af.mil>
> Subject: Re: [rt.rap.ucar.edu #76891] observation count?
>
> Hi Steve.
>
> I ran ascii2nc using varied numbers of observations from Modis.txt.
I
> used 50000, 80000, 90000, 389751 (full file). I ran the code using
> the "time" command to get the time it took to run.
>
> So, for example, I ran:
>
> /usr/bin/time --verbose --output ascii2nc_time_50000.txt ascii2nc
> Modis_50000.txt Modis_50000.nc -format "met_point" -config
> ./Ascii2NcConfig_default
>
> for each file.
>
> The times were the following:
>
> 50000 obs:
> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:07.63
>
> 80000 obs:
> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:13.08
>
> 90000 obs:
> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.02
>
> 389751 obs:
> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:59.76
>
>
> Those numbers seem reasonable and in line with the number of
> observations (389751 / 50000 = 7.79502; 7.79502 * 7.63 seconds =
> 59.476 seconds ) You mentioned that the system was taking an
> "extremely long time". Can you please tell me how long it was
taking
> for the 389751 obs on your machine?
>
> Thanks,
> Julie
>
> On Thu, Jun 30, 2016 at 10:54 AM, BLIUJUS, STEVEN D GS-07 USAF ACC
16
> WS/WXE via RT <met_help at ucar.edu> wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891 >
>>
>> I put in two files and two config files I used (one for the ascii
portion and the other the point stat...the first one is one under
85000, the other is more, much more (~385000) points but if you want
to hack it down to like 90000 it would do the same. The command I'm
using is:
>>
>> Ascii2nc Modis.txt Modis.nx -format "met_point"
>>
>> The version is 5.1
>>
>> Steve
>>
>> -----Original Message-----
>> From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
>> Sent: Thursday, June 30, 2016 10:57 AM
>> To: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
>> <steven.bliujus.1 at us.af.mil>
>> Subject: Re: [rt.rap.ucar.edu #76891] observation count?
>>
>> Hi Steven.
>>
>> Unfortunately, John is out of the office this week and the
beginning of next week. He is more knowledgeable and familiar with
issues like this one. I would guess that there is not an observation
limit, but do not know of a reason for the slow down after you get
above ~85000 obs.
>>
>> Could you please post your data to our ftp site?
>>
>> You can follow the instructions here:
>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>
>> Please also send me that command that you are using in running
ascii2nc and let me know what version of MET you are using. I'll take
a look and see what I can figure out.
>>
>> Thanks,
>> Julie
>>
>> On Thu, Jun 30, 2016 at 8:43 AM, BLIUJUS, STEVEN D GS-07 USAF ACC
16 WS/WXE via RT <met_help at ucar.edu> wrote:
>>>
>>>
>>> Thu Jun 30 08:43:21 2016: Request 76891 was acted upon.
>>> Transaction: Ticket created by steven.bliujus.1 at us.af.mil
>>> Queue: met_help
>>> Subject: observation count?
>>> Owner: Nobody
>>> Requestors: steven.bliujus.1 at us.af.mil
>>> Status: new
>>> Ticket <URL:
>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891
>>> >
>>>
>>>
>>> I have a question. I am running the ascii2nc tool, and was
wondering if there is an amount of observations limit? For some reason
it seems as though once I get above ~85000 obs, the system takes an
extremely long time. For example, below that it takes less than 5
seconds to create the netcdf file, once I exceed that amount of
observations it suddenly take 5 minutes or longer. The same goes for
using the point_stat tool.
>>>
>>> V/R,
>>>
>>> STEVEN D. BLIUJUS, GS-07, DAFC
>>> 16 WS/WXE
>>> DSN: 272-7151
>>> COMM: 402-232-7151
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> Julie Prestopnik
>> National Center for Atmospheric Research Research Applications
>> Laboratory
>> Phone: 303.497.8399
>> Email: jpresto at ucar.edu
>>
>>
>>
>>
>
>
>
> --
> Julie Prestopnik
> National Center for Atmospheric Research Research Applications
> Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
>
>
>
--
Julie Prestopnik
National Center for Atmospheric Research Research Applications
Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #76891] observation count?
From: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
Time: Mon Jul 11 08:28:33 2016
Julie,
Did you have any recommendations or thoughts as to why my ascii2nc is
taking so long?
Steve
-----Original Message-----
From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
Sent: Thursday, June 30, 2016 3:09 PM
To: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
<steven.bliujus.1 at us.af.mil>
Subject: Re: [rt.rap.ucar.edu #76891] observation count?
Thank you for sending these times.
I was thinking that maybe getting the latest set of patches for
METv5.1 would help. However, I was just looking over the "Summary of
Recommended Updates" on the "Known Issues and Fixes for METv5.1" page:
http://www.dtcenter.org/met/users/support/known_issues/METv5.1/index.php
and I don't see anything there that might help with the problem you
are experiencing. Unfortunately, I think I need to defer to John on
this issue. I will ask him to take a look when he gets back.
Thank you for your patience regarding this issue.
Julie
On Thu, Jun 30, 2016 at 12:42 PM, BLIUJUS, STEVEN D GS-07 USAF ACC 16
WS/WXE via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891 >
>
> For 50000: Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.59 For
> 80000: Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.61 For
90000:
> Elapsed (wall clock) time (h:mm:ss or m:ss): 1:34.41 For 100000:
> Elapsed (wall clock) time (h:mm:ss or m:ss): 7:22.46
>
> So as you can see it is exponentially growing in length of time
which is why I wont run the 300000 file again. I also noted that the
percent of CPU this job got kept dropping.
>
>
>
> -----Original Message-----
> From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, June 30, 2016 1:00 PM
> To: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
> <steven.bliujus.1 at us.af.mil>
> Subject: Re: [rt.rap.ucar.edu #76891] observation count?
>
> Hi Steve.
>
> I ran ascii2nc using varied numbers of observations from Modis.txt.
I
> used 50000, 80000, 90000, 389751 (full file). I ran the code using
> the "time" command to get the time it took to run.
>
> So, for example, I ran:
>
> /usr/bin/time --verbose --output ascii2nc_time_50000.txt ascii2nc
> Modis_50000.txt Modis_50000.nc -format "met_point" -config
> ./Ascii2NcConfig_default
>
> for each file.
>
> The times were the following:
>
> 50000 obs:
> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:07.63
>
> 80000 obs:
> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:13.08
>
> 90000 obs:
> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.02
>
> 389751 obs:
> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:59.76
>
>
> Those numbers seem reasonable and in line with the number of
> observations (389751 / 50000 = 7.79502; 7.79502 * 7.63 seconds =
> 59.476 seconds ) You mentioned that the system was taking an
> "extremely long time". Can you please tell me how long it was
taking
> for the 389751 obs on your machine?
>
> Thanks,
> Julie
>
> On Thu, Jun 30, 2016 at 10:54 AM, BLIUJUS, STEVEN D GS-07 USAF ACC
16
> WS/WXE via RT <met_help at ucar.edu> wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891 >
>>
>> I put in two files and two config files I used (one for the ascii
portion and the other the point stat...the first one is one under
85000, the other is more, much more (~385000) points but if you want
to hack it down to like 90000 it would do the same. The command I'm
using is:
>>
>> Ascii2nc Modis.txt Modis.nx -format "met_point"
>>
>> The version is 5.1
>>
>> Steve
>>
>> -----Original Message-----
>> From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
>> Sent: Thursday, June 30, 2016 10:57 AM
>> To: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
>> <steven.bliujus.1 at us.af.mil>
>> Subject: Re: [rt.rap.ucar.edu #76891] observation count?
>>
>> Hi Steven.
>>
>> Unfortunately, John is out of the office this week and the
beginning of next week. He is more knowledgeable and familiar with
issues like this one. I would guess that there is not an observation
limit, but do not know of a reason for the slow down after you get
above ~85000 obs.
>>
>> Could you please post your data to our ftp site?
>>
>> You can follow the instructions here:
>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>
>> Please also send me that command that you are using in running
ascii2nc and let me know what version of MET you are using. I'll take
a look and see what I can figure out.
>>
>> Thanks,
>> Julie
>>
>> On Thu, Jun 30, 2016 at 8:43 AM, BLIUJUS, STEVEN D GS-07 USAF ACC
16 WS/WXE via RT <met_help at ucar.edu> wrote:
>>>
>>>
>>> Thu Jun 30 08:43:21 2016: Request 76891 was acted upon.
>>> Transaction: Ticket created by steven.bliujus.1 at us.af.mil
>>> Queue: met_help
>>> Subject: observation count?
>>> Owner: Nobody
>>> Requestors: steven.bliujus.1 at us.af.mil
>>> Status: new
>>> Ticket <URL:
>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891
>>> >
>>>
>>>
>>> I have a question. I am running the ascii2nc tool, and was
wondering if there is an amount of observations limit? For some reason
it seems as though once I get above ~85000 obs, the system takes an
extremely long time. For example, below that it takes less than 5
seconds to create the netcdf file, once I exceed that amount of
observations it suddenly take 5 minutes or longer. The same goes for
using the point_stat tool.
>>>
>>> V/R,
>>>
>>> STEVEN D. BLIUJUS, GS-07, DAFC
>>> 16 WS/WXE
>>> DSN: 272-7151
>>> COMM: 402-232-7151
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> Julie Prestopnik
>> National Center for Atmospheric Research Research Applications
>> Laboratory
>> Phone: 303.497.8399
>> Email: jpresto at ucar.edu
>>
>>
>>
>>
>
>
>
> --
> Julie Prestopnik
> National Center for Atmospheric Research Research Applications
> Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
>
>
>
--
Julie Prestopnik
National Center for Atmospheric Research Research Applications
Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #76891] observation count?
From: Julie Prestopnik
Time: Tue Jul 12 13:24:57 2016
Unfortunately, Steve, I do not. Hopefully, John will be able to
provide some assistance.
Julie
On Mon, Jul 11, 2016 at 8:28 AM, BLIUJUS, STEVEN D GS-07 USAF ACC 16
WS/WXE via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891 >
>
> Julie,
>
> Did you have any recommendations or thoughts as to why my ascii2nc
is taking so long?
>
> Steve
>
> -----Original Message-----
> From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
> Sent: Thursday, June 30, 2016 3:09 PM
> To: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
<steven.bliujus.1 at us.af.mil>
> Subject: Re: [rt.rap.ucar.edu #76891] observation count?
>
> Thank you for sending these times.
>
> I was thinking that maybe getting the latest set of patches for
> METv5.1 would help. However, I was just looking over the "Summary
of Recommended Updates" on the "Known Issues and Fixes for METv5.1"
page:
>
>
http://www.dtcenter.org/met/users/support/known_issues/METv5.1/index.php
>
> and I don't see anything there that might help with the problem you
are experiencing. Unfortunately, I think I need to defer to John on
this issue. I will ask him to take a look when he gets back.
>
> Thank you for your patience regarding this issue.
>
> Julie
>
> On Thu, Jun 30, 2016 at 12:42 PM, BLIUJUS, STEVEN D GS-07 USAF ACC
16 WS/WXE via RT <met_help at ucar.edu> wrote:
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891 >
>>
>> For 50000: Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.59 For
>> 80000: Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.61 For
90000:
>> Elapsed (wall clock) time (h:mm:ss or m:ss): 1:34.41 For 100000:
>> Elapsed (wall clock) time (h:mm:ss or m:ss): 7:22.46
>>
>> So as you can see it is exponentially growing in length of time
which is why I wont run the 300000 file again. I also noted that the
percent of CPU this job got kept dropping.
>>
>>
>>
>> -----Original Message-----
>> From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
>> Sent: Thursday, June 30, 2016 1:00 PM
>> To: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
>> <steven.bliujus.1 at us.af.mil>
>> Subject: Re: [rt.rap.ucar.edu #76891] observation count?
>>
>> Hi Steve.
>>
>> I ran ascii2nc using varied numbers of observations from Modis.txt.
I
>> used 50000, 80000, 90000, 389751 (full file). I ran the code using
>> the "time" command to get the time it took to run.
>>
>> So, for example, I ran:
>>
>> /usr/bin/time --verbose --output ascii2nc_time_50000.txt ascii2nc
>> Modis_50000.txt Modis_50000.nc -format "met_point" -config
>> ./Ascii2NcConfig_default
>>
>> for each file.
>>
>> The times were the following:
>>
>> 50000 obs:
>> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:07.63
>>
>> 80000 obs:
>> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:13.08
>>
>> 90000 obs:
>> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.02
>>
>> 389751 obs:
>> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:59.76
>>
>>
>> Those numbers seem reasonable and in line with the number of
>> observations (389751 / 50000 = 7.79502; 7.79502 * 7.63 seconds =
>> 59.476 seconds ) You mentioned that the system was taking an
>> "extremely long time". Can you please tell me how long it was
taking
>> for the 389751 obs on your machine?
>>
>> Thanks,
>> Julie
>>
>> On Thu, Jun 30, 2016 at 10:54 AM, BLIUJUS, STEVEN D GS-07 USAF ACC
16
>> WS/WXE via RT <met_help at ucar.edu> wrote:
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891 >
>>>
>>> I put in two files and two config files I used (one for the ascii
portion and the other the point stat...the first one is one under
85000, the other is more, much more (~385000) points but if you want
to hack it down to like 90000 it would do the same. The command I'm
using is:
>>>
>>> Ascii2nc Modis.txt Modis.nx -format "met_point"
>>>
>>> The version is 5.1
>>>
>>> Steve
>>>
>>> -----Original Message-----
>>> From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
>>> Sent: Thursday, June 30, 2016 10:57 AM
>>> To: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
>>> <steven.bliujus.1 at us.af.mil>
>>> Subject: Re: [rt.rap.ucar.edu #76891] observation count?
>>>
>>> Hi Steven.
>>>
>>> Unfortunately, John is out of the office this week and the
beginning of next week. He is more knowledgeable and familiar with
issues like this one. I would guess that there is not an observation
limit, but do not know of a reason for the slow down after you get
above ~85000 obs.
>>>
>>> Could you please post your data to our ftp site?
>>>
>>> You can follow the instructions here:
>>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
>>>
>>> Please also send me that command that you are using in running
ascii2nc and let me know what version of MET you are using. I'll take
a look and see what I can figure out.
>>>
>>> Thanks,
>>> Julie
>>>
>>> On Thu, Jun 30, 2016 at 8:43 AM, BLIUJUS, STEVEN D GS-07 USAF ACC
16 WS/WXE via RT <met_help at ucar.edu> wrote:
>>>>
>>>>
>>>> Thu Jun 30 08:43:21 2016: Request 76891 was acted upon.
>>>> Transaction: Ticket created by steven.bliujus.1 at us.af.mil
>>>> Queue: met_help
>>>> Subject: observation count?
>>>> Owner: Nobody
>>>> Requestors: steven.bliujus.1 at us.af.mil
>>>> Status: new
>>>> Ticket <URL:
>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891
>>>> >
>>>>
>>>>
>>>> I have a question. I am running the ascii2nc tool, and was
wondering if there is an amount of observations limit? For some reason
it seems as though once I get above ~85000 obs, the system takes an
extremely long time. For example, below that it takes less than 5
seconds to create the netcdf file, once I exceed that amount of
observations it suddenly take 5 minutes or longer. The same goes for
using the point_stat tool.
>>>>
>>>> V/R,
>>>>
>>>> STEVEN D. BLIUJUS, GS-07, DAFC
>>>> 16 WS/WXE
>>>> DSN: 272-7151
>>>> COMM: 402-232-7151
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Julie Prestopnik
>>> National Center for Atmospheric Research Research Applications
>>> Laboratory
>>> Phone: 303.497.8399
>>> Email: jpresto at ucar.edu
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> Julie Prestopnik
>> National Center for Atmospheric Research Research Applications
>> Laboratory
>> Phone: 303.497.8399
>> Email: jpresto at ucar.edu
>>
>>
>>
>>
>
>
>
> --
> Julie Prestopnik
> National Center for Atmospheric Research Research Applications
Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
>
>
>
--
Julie Prestopnik
National Center for Atmospheric Research
Research Applications Laboratory
Phone: 303.497.8399
Email: jpresto at ucar.edu
------------------------------------------------
Subject: observation count?
From: John Halley Gotway
Time: Wed Jul 13 11:46:17 2016
Hello Steve,
Sorry for the delay in getting back to you on this.
I just ran a similar test on my machine, running ascii2nc many times
by
doubling the observations and checking the runtime. I found a very
linear
progression in run times. See below...
DEBUG 2: Finished processing 50000 observations for 85 headers.
0.556u 0.008s 0:00.56 98.2% 0+0k 0+6016io 0pf+0w
DEBUG 2: Finished processing 100000 observations for 170 headers.
0.936u 0.012s 0:00.95 98.9% 0+0k 0+11816io 0pf+0w
DEBUG 2: Finished processing 200000 observations for 340 headers.
1.664u 0.032s 0:01.69 100.0% 0+0k 0+23568io 0pf+0w
DEBUG 2: Finished processing 400000 observations for 680 headers.
3.200u 0.056s 0:03.25 100.0% 0+0k 0+47176io 0pf+0w
DEBUG 2: Finished processing 800000 observations for 1360 headers.
5.888u 0.112s 0:06.00 99.8% 0+0k 8+94216io 0pf+0w
DEBUG 2: Finished processing 1600000 observations for 2720 headers.
11.788u 0.184s 0:11.97 99.9% 0+0k 0+188400io 0pf+0w
DEBUG 2: Finished processing 3200000 observations for 5440 headers.
23.472u 0.492s 0:23.98 99.9% 0+0k 16+376864io 0pf+0w
DEBUG 2: Finished processing 6400000 observations for 10880 headers.
47.328u 0.892s 0:48.25 99.9% 0+0k 24+753648io 0pf+0w
DEBUG 2: Finished processing 12800000 observations for 21760 headers.
94.060u 2.124s 1:36.24 99.9% 0+0k 48+1512432io 0pf+0w
Based on this, I would expect a similar linear progression in runtimes
on
your machine. But you're obviously not seeing that. Instead, you see
a
big jump in run times between 80k and 90k. My guess would be that
that's
due to memory.
As I ran my tests, I used the "top" command to watch the amount of
memory
being consumed. It never exceeded 10%. When a process has consumed
all
the available memory, it switches to using swap space which is
extremely
slow! If this is occurring, that would explain the symptoms you're
seeing.
I would suggest the following...
(1) Use one window to run "top" on your machine.
(2) In another window, run the 80000 observation test case and closely
watch the total memory being used.
(3) Then run the 90000 observation test case and watch the total
memory
consumption. Is it close to 100%?
If memory turns out to be the issue, then you're limited by your
hardware.
I'd suggest asking a sys admin to see if there's anyway of increasing
the
memory available - either through a system configuration setting or by
adding more physical memory.
Alternatively, you could write a wrapper script for ascii2nc to split
up
the ascii input files into smaller pieces and process them in chunks.
Theoretically, the ascii2nc tool itself could be modified to process
chunks
of data in this way rather than doing it via a script, but that would
take
development work.
Please let us know what you find.
Thanks,
John
On Tue, Jul 12, 2016 at 1:24 PM, Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891 >
>
> Unfortunately, Steve, I do not. Hopefully, John will be able to
> provide some assistance.
>
> Julie
>
> On Mon, Jul 11, 2016 at 8:28 AM, BLIUJUS, STEVEN D GS-07 USAF ACC 16
> WS/WXE via RT <met_help at ucar.edu> wrote:
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891 >
> >
> > Julie,
> >
> > Did you have any recommendations or thoughts as to why my ascii2nc
is
> taking so long?
> >
> > Steve
> >
> > -----Original Message-----
> > From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
> > Sent: Thursday, June 30, 2016 3:09 PM
> > To: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE <
> steven.bliujus.1 at us.af.mil>
> > Subject: Re: [rt.rap.ucar.edu #76891] observation count?
> >
> > Thank you for sending these times.
> >
> > I was thinking that maybe getting the latest set of patches for
> > METv5.1 would help. However, I was just looking over the "Summary
of
> Recommended Updates" on the "Known Issues and Fixes for METv5.1"
page:
> >
> >
http://www.dtcenter.org/met/users/support/known_issues/METv5.1/index.php
> >
> > and I don't see anything there that might help with the problem
you are
> experiencing. Unfortunately, I think I need to defer to John on
this
> issue. I will ask him to take a look when he gets back.
> >
> > Thank you for your patience regarding this issue.
> >
> > Julie
> >
> > On Thu, Jun 30, 2016 at 12:42 PM, BLIUJUS, STEVEN D GS-07 USAF ACC
16
> WS/WXE via RT <met_help at ucar.edu> wrote:
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891 >
> >>
> >> For 50000: Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.59
For
> >> 80000: Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.61 For
90000:
> >> Elapsed (wall clock) time (h:mm:ss or m:ss): 1:34.41 For 100000:
> >> Elapsed (wall clock) time (h:mm:ss or m:ss): 7:22.46
> >>
> >> So as you can see it is exponentially growing in length of time
which
> is why I wont run the 300000 file again. I also noted that the
percent of
> CPU this job got kept dropping.
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
> >> Sent: Thursday, June 30, 2016 1:00 PM
> >> To: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
> >> <steven.bliujus.1 at us.af.mil>
> >> Subject: Re: [rt.rap.ucar.edu #76891] observation count?
> >>
> >> Hi Steve.
> >>
> >> I ran ascii2nc using varied numbers of observations from
Modis.txt. I
> >> used 50000, 80000, 90000, 389751 (full file). I ran the code
using
> >> the "time" command to get the time it took to run.
> >>
> >> So, for example, I ran:
> >>
> >> /usr/bin/time --verbose --output ascii2nc_time_50000.txt ascii2nc
> >> Modis_50000.txt Modis_50000.nc -format "met_point" -config
> >> ./Ascii2NcConfig_default
> >>
> >> for each file.
> >>
> >> The times were the following:
> >>
> >> 50000 obs:
> >> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:07.63
> >>
> >> 80000 obs:
> >> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:13.08
> >>
> >> 90000 obs:
> >> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.02
> >>
> >> 389751 obs:
> >> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:59.76
> >>
> >>
> >> Those numbers seem reasonable and in line with the number of
> >> observations (389751 / 50000 = 7.79502; 7.79502 * 7.63 seconds =
> >> 59.476 seconds ) You mentioned that the system was taking an
> >> "extremely long time". Can you please tell me how long it was
taking
> >> for the 389751 obs on your machine?
> >>
> >> Thanks,
> >> Julie
> >>
> >> On Thu, Jun 30, 2016 at 10:54 AM, BLIUJUS, STEVEN D GS-07 USAF
ACC 16
> >> WS/WXE via RT <met_help at ucar.edu> wrote:
> >>>
> >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891 >
> >>>
> >>> I put in two files and two config files I used (one for the
ascii
> portion and the other the point stat...the first one is one under
85000,
> the other is more, much more (~385000) points but if you want to
hack it
> down to like 90000 it would do the same. The command I'm using is:
> >>>
> >>> Ascii2nc Modis.txt Modis.nx -format "met_point"
> >>>
> >>> The version is 5.1
> >>>
> >>> Steve
> >>>
> >>> -----Original Message-----
> >>> From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
> >>> Sent: Thursday, June 30, 2016 10:57 AM
> >>> To: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
> >>> <steven.bliujus.1 at us.af.mil>
> >>> Subject: Re: [rt.rap.ucar.edu #76891] observation count?
> >>>
> >>> Hi Steven.
> >>>
> >>> Unfortunately, John is out of the office this week and the
beginning
> of next week. He is more knowledgeable and familiar with issues like
this
> one. I would guess that there is not an observation limit, but do
not know
> of a reason for the slow down after you get above ~85000 obs.
> >>>
> >>> Could you please post your data to our ftp site?
> >>>
> >>> You can follow the instructions here:
> >>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
> >>>
> >>> Please also send me that command that you are using in running
> ascii2nc and let me know what version of MET you are using. I'll
take a
> look and see what I can figure out.
> >>>
> >>> Thanks,
> >>> Julie
> >>>
> >>> On Thu, Jun 30, 2016 at 8:43 AM, BLIUJUS, STEVEN D GS-07 USAF
ACC 16
> WS/WXE via RT <met_help at ucar.edu> wrote:
> >>>>
> >>>>
> >>>> Thu Jun 30 08:43:21 2016: Request 76891 was acted upon.
> >>>> Transaction: Ticket created by steven.bliujus.1 at us.af.mil
> >>>> Queue: met_help
> >>>> Subject: observation count?
> >>>> Owner: Nobody
> >>>> Requestors: steven.bliujus.1 at us.af.mil
> >>>> Status: new
> >>>> Ticket <URL:
> >>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891
> >>>> >
> >>>>
> >>>>
> >>>> I have a question. I am running the ascii2nc tool, and was
wondering
> if there is an amount of observations limit? For some reason it
seems as
> though once I get above ~85000 obs, the system takes an extremely
long
> time. For example, below that it takes less than 5 seconds to create
the
> netcdf file, once I exceed that amount of observations it suddenly
take 5
> minutes or longer. The same goes for using the point_stat tool.
> >>>>
> >>>> V/R,
> >>>>
> >>>> STEVEN D. BLIUJUS, GS-07, DAFC
> >>>> 16 WS/WXE
> >>>> DSN: 272-7151
> >>>> COMM: 402-232-7151
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Julie Prestopnik
> >>> National Center for Atmospheric Research Research Applications
> >>> Laboratory
> >>> Phone: 303.497.8399
> >>> Email: jpresto at ucar.edu
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> Julie Prestopnik
> >> National Center for Atmospheric Research Research Applications
> >> Laboratory
> >> Phone: 303.497.8399
> >> Email: jpresto at ucar.edu
> >>
> >>
> >>
> >>
> >
> >
> >
> > --
> > Julie Prestopnik
> > National Center for Atmospheric Research Research Applications
Laboratory
> > Phone: 303.497.8399
> > Email: jpresto at ucar.edu
> >
> >
> >
> >
>
>
>
> --
> Julie Prestopnik
> National Center for Atmospheric Research
> Research Applications Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #76891] observation count?
From: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
Time: Wed Jul 13 12:16:57 2016
So I did think about breaking it up, and creating a new NetCDF file
for a month at a time. I just didn’t want to risk the chance of
something messing up during the point stat phase (I didn’t want any
files getting overwritten). I have been tasked to do some other stuff
now, so it may be a couple weeks or so until I can dive back into it.
I guess I can break up the point stat too, and then the stat_analysis
will look at all the files.
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, July 13, 2016 12:46 PM
To: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
<steven.bliujus.1 at us.af.mil>
Subject: Re: [rt.rap.ucar.edu #76891] observation count?
Hello Steve,
Sorry for the delay in getting back to you on this.
I just ran a similar test on my machine, running ascii2nc many times
by doubling the observations and checking the runtime. I found a very
linear progression in run times. See below...
DEBUG 2: Finished processing 50000 observations for 85 headers.
0.556u 0.008s 0:00.56 98.2% 0+0k 0+6016io 0pf+0w
DEBUG 2: Finished processing 100000 observations for 170 headers.
0.936u 0.012s 0:00.95 98.9% 0+0k 0+11816io 0pf+0w
DEBUG 2: Finished processing 200000 observations for 340 headers.
1.664u 0.032s 0:01.69 100.0% 0+0k 0+23568io 0pf+0w
DEBUG 2: Finished processing 400000 observations for 680 headers.
3.200u 0.056s 0:03.25 100.0% 0+0k 0+47176io 0pf+0w
DEBUG 2: Finished processing 800000 observations for 1360 headers.
5.888u 0.112s 0:06.00 99.8% 0+0k 8+94216io 0pf+0w
DEBUG 2: Finished processing 1600000 observations for 2720 headers.
11.788u 0.184s 0:11.97 99.9% 0+0k 0+188400io 0pf+0w
DEBUG 2: Finished processing 3200000 observations for 5440 headers.
23.472u 0.492s 0:23.98 99.9% 0+0k 16+376864io 0pf+0w
DEBUG 2: Finished processing 6400000 observations for 10880 headers.
47.328u 0.892s 0:48.25 99.9% 0+0k 24+753648io 0pf+0w
DEBUG 2: Finished processing 12800000 observations for 21760 headers.
94.060u 2.124s 1:36.24 99.9% 0+0k 48+1512432io 0pf+0w
Based on this, I would expect a similar linear progression in runtimes
on your machine. But you're obviously not seeing that. Instead, you
see a big jump in run times between 80k and 90k. My guess would be
that that's due to memory.
As I ran my tests, I used the "top" command to watch the amount of
memory being consumed. It never exceeded 10%. When a process has
consumed all the available memory, it switches to using swap space
which is extremely slow! If this is occurring, that would explain the
symptoms you're seeing.
I would suggest the following...
(1) Use one window to run "top" on your machine.
(2) In another window, run the 80000 observation test case and closely
watch the total memory being used.
(3) Then run the 90000 observation test case and watch the total
memory consumption. Is it close to 100%?
If memory turns out to be the issue, then you're limited by your
hardware.
I'd suggest asking a sys admin to see if there's anyway of increasing
the memory available - either through a system configuration setting
or by adding more physical memory.
Alternatively, you could write a wrapper script for ascii2nc to split
up the ascii input files into smaller pieces and process them in
chunks.
Theoretically, the ascii2nc tool itself could be modified to process
chunks of data in this way rather than doing it via a script, but that
would take development work.
Please let us know what you find.
Thanks,
John
On Tue, Jul 12, 2016 at 1:24 PM, Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891 >
>
> Unfortunately, Steve, I do not. Hopefully, John will be able to
> provide some assistance.
>
> Julie
>
> On Mon, Jul 11, 2016 at 8:28 AM, BLIUJUS, STEVEN D GS-07 USAF ACC 16
> WS/WXE via RT <met_help at ucar.edu> wrote:
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891 >
> >
> > Julie,
> >
> > Did you have any recommendations or thoughts as to why my ascii2nc
> > is
> taking so long?
> >
> > Steve
> >
> > -----Original Message-----
> > From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
> > Sent: Thursday, June 30, 2016 3:09 PM
> > To: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE <
> steven.bliujus.1 at us.af.mil>
> > Subject: Re: [rt.rap.ucar.edu #76891] observation count?
> >
> > Thank you for sending these times.
> >
> > I was thinking that maybe getting the latest set of patches for
> > METv5.1 would help. However, I was just looking over the "Summary
> > of
> Recommended Updates" on the "Known Issues and Fixes for METv5.1"
page:
> >
> >
http://www.dtcenter.org/met/users/support/known_issues/METv5.1/index
> > .php
> >
> > and I don't see anything there that might help with the problem
you
> > are
> experiencing. Unfortunately, I think I need to defer to John on
this
> issue. I will ask him to take a look when he gets back.
> >
> > Thank you for your patience regarding this issue.
> >
> > Julie
> >
> > On Thu, Jun 30, 2016 at 12:42 PM, BLIUJUS, STEVEN D GS-07 USAF ACC
> > 16
> WS/WXE via RT <met_help at ucar.edu> wrote:
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891 >
> >>
> >> For 50000: Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.59
For
> >> 80000: Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.61 For
90000:
> >> Elapsed (wall clock) time (h:mm:ss or m:ss): 1:34.41 For 100000:
> >> Elapsed (wall clock) time (h:mm:ss or m:ss): 7:22.46
> >>
> >> So as you can see it is exponentially growing in length of time
> >> which
> is why I wont run the 300000 file again. I also noted that the
> percent of CPU this job got kept dropping.
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
> >> Sent: Thursday, June 30, 2016 1:00 PM
> >> To: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
> >> <steven.bliujus.1 at us.af.mil>
> >> Subject: Re: [rt.rap.ucar.edu #76891] observation count?
> >>
> >> Hi Steve.
> >>
> >> I ran ascii2nc using varied numbers of observations from
Modis.txt.
> >> I used 50000, 80000, 90000, 389751 (full file). I ran the code
> >> using the "time" command to get the time it took to run.
> >>
> >> So, for example, I ran:
> >>
> >> /usr/bin/time --verbose --output ascii2nc_time_50000.txt ascii2nc
> >> Modis_50000.txt Modis_50000.nc -format "met_point" -config
> >> ./Ascii2NcConfig_default
> >>
> >> for each file.
> >>
> >> The times were the following:
> >>
> >> 50000 obs:
> >> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:07.63
> >>
> >> 80000 obs:
> >> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:13.08
> >>
> >> 90000 obs:
> >> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:15.02
> >>
> >> 389751 obs:
> >> Elapsed (wall clock) time (h:mm:ss or m:ss): 0:59.76
> >>
> >>
> >> Those numbers seem reasonable and in line with the number of
> >> observations (389751 / 50000 = 7.79502; 7.79502 * 7.63 seconds =
> >> 59.476 seconds ) You mentioned that the system was taking an
> >> "extremely long time". Can you please tell me how long it was
> >> taking for the 389751 obs on your machine?
> >>
> >> Thanks,
> >> Julie
> >>
> >> On Thu, Jun 30, 2016 at 10:54 AM, BLIUJUS, STEVEN D GS-07 USAF
ACC
> >> 16 WS/WXE via RT <met_help at ucar.edu> wrote:
> >>>
> >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891 >
> >>>
> >>> I put in two files and two config files I used (one for the
ascii
> portion and the other the point stat...the first one is one under
> 85000, the other is more, much more (~385000) points but if you want
> to hack it down to like 90000 it would do the same. The command I'm
using is:
> >>>
> >>> Ascii2nc Modis.txt Modis.nx -format "met_point"
> >>>
> >>> The version is 5.1
> >>>
> >>> Steve
> >>>
> >>> -----Original Message-----
> >>> From: Julie Prestopnik via RT [mailto:met_help at ucar.edu]
> >>> Sent: Thursday, June 30, 2016 10:57 AM
> >>> To: BLIUJUS, STEVEN D GS-07 USAF ACC 16 WS/WXE
> >>> <steven.bliujus.1 at us.af.mil>
> >>> Subject: Re: [rt.rap.ucar.edu #76891] observation count?
> >>>
> >>> Hi Steven.
> >>>
> >>> Unfortunately, John is out of the office this week and the
> >>> beginning
> of next week. He is more knowledgeable and familiar with issues like
> this one. I would guess that there is not an observation limit, but
> do not know of a reason for the slow down after you get above ~85000
obs.
> >>>
> >>> Could you please post your data to our ftp site?
> >>>
> >>> You can follow the instructions here:
> >>> http://www.dtcenter.org/met/users/support/met_help.php#ftp
> >>>
> >>> Please also send me that command that you are using in running
> ascii2nc and let me know what version of MET you are using. I'll
take
> a look and see what I can figure out.
> >>>
> >>> Thanks,
> >>> Julie
> >>>
> >>> On Thu, Jun 30, 2016 at 8:43 AM, BLIUJUS, STEVEN D GS-07 USAF
ACC
> >>> 16
> WS/WXE via RT <met_help at ucar.edu> wrote:
> >>>>
> >>>>
> >>>> Thu Jun 30 08:43:21 2016: Request 76891 was acted upon.
> >>>> Transaction: Ticket created by steven.bliujus.1 at us.af.mil
> >>>> Queue: met_help
> >>>> Subject: observation count?
> >>>> Owner: Nobody
> >>>> Requestors: steven.bliujus.1 at us.af.mil
> >>>> Status: new
> >>>> Ticket <URL:
> >>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=76891
> >>>> >
> >>>>
> >>>>
> >>>> I have a question. I am running the ascii2nc tool, and was
> >>>> wondering
> if there is an amount of observations limit? For some reason it
seems
> as though once I get above ~85000 obs, the system takes an extremely
> long time. For example, below that it takes less than 5 seconds to
> create the netcdf file, once I exceed that amount of observations it
> suddenly take 5 minutes or longer. The same goes for using the
point_stat tool.
> >>>>
> >>>> V/R,
> >>>>
> >>>> STEVEN D. BLIUJUS, GS-07, DAFC
> >>>> 16 WS/WXE
> >>>> DSN: 272-7151
> >>>> COMM: 402-232-7151
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Julie Prestopnik
> >>> National Center for Atmospheric Research Research Applications
> >>> Laboratory
> >>> Phone: 303.497.8399
> >>> Email: jpresto at ucar.edu
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> Julie Prestopnik
> >> National Center for Atmospheric Research Research Applications
> >> Laboratory
> >> Phone: 303.497.8399
> >> Email: jpresto at ucar.edu
> >>
> >>
> >>
> >>
> >
> >
> >
> > --
> > Julie Prestopnik
> > National Center for Atmospheric Research Research Applications
> > Laboratory
> > Phone: 303.497.8399
> > Email: jpresto at ucar.edu
> >
> >
> >
> >
>
>
>
> --
> Julie Prestopnik
> National Center for Atmospheric Research Research Applications
> Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
>
>
------------------------------------------------
More information about the Met_help
mailing list