[Met_help] [rt.rap.ucar.edu #41012] History for Run grid_stat faster

Tue Sep 21 13:27:58 MDT 2010

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hi John,

I am not sure whether to send this to you or to the met help desk. Let me know if I should email */met_help at ucar.edu instead./*

It is taking a really long time (e.g, ~2-3 minutes per day of data) to run grid_stat.  (I have to run 10 years! of data that I need to run through grid_stat.) Is there some parameter in my config file that I can change to make grid_stat run faster? Or perhaps the CNRFC poly line I am using (see attached) causing grid_stat to run slowly?  I am attaching my config file and CNRFC poly line with this with this email.  At this point the output I am looking for is counts of hits, misses, false alarms (CTC tables), stats (CTS tables), and matched pairs if that is possible.

Thanks,
Ellen

----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Re: [rt.rap.ucar.edu #41012] Run grid_stat faster
From: John Halley Gotway
Time: Tue Sep 21 10:24:45 2010

Ellen,

I went ahead and created a met_help ticket for this.  I took a look at
the config file you sent and saw that:
   - bootstrapping is turned off (n_boot_rep = 0)
   - the rank correlation flag is turned off (rank_corr_flag = 0)

I think you're right about how to speed it up.  I took a look in that
CNRFC_Ring1.poly and see that it contains 15274 points!  Each time you
run Grid-Stat, it's checking to see if each grid point is
inside that polyline of lenght 15274.  I suspect that's taking up the
majority of the run time.  This is the exact reason we created the
gen_poly_mask tool in the first place - so you can apply the
polyline to your grid once rather than once per run.  So just run the
polyline and a gridded data file through gen_poly_mask.  See the test
script, METv2.0/scripts/test_gen_poly_mask.sh, for an
example of running it.  Then in your Grid-Stat config file, replace
CNRFC_Ring1.poly with the NetCDF output file from gen_poly_mask.

I'm curious to find out how much that speeds it up.

Here's a few other things to mention:
(1) I see that you're using both the FULL and CNRFC masking regions.
And that's fine to do, but if you really are only interested in CNRFC,
you can remove FULL.  It's fine for mask_grid to be set as
an empty list.
(2) I see that you're dumping out FHO and CTC lines.  Those contain
redundant information - counts of the 2x2 contingency table.  I'd
suggest turning off FHO, unless you have a good reason to keep them.
(3) I see that you're using a flag value of 2 - so the STAT data will
be dumped to a ".stat" file as well as a ".txt" file.  If you're
running everyday for 10 years, that'll be a lot of duplicate
data.  You could consider changing the 2's to 1's so that it only goes
into a ".stat" file.
(4) You mentioned that you want matched pairs.  Grid-Stat doesn't dump
out MPR lines in the same way that Point-Stat does.  Instead it can
write out a NetCDF file containing the
forecast/observation/difference fields.  If you'd like this NetCDF
"pairs" output file, you'll need to set the last value in the
output_flag to 1.

Hope that helps.

John

RAL HelpDesk {for John Halley Gotway} wrote:
> Tue Sep 21 10:11:24 2010: Request 41012 was acted upon.
> Transaction: Ticket created by johnhg
>        Queue: met_help
>      Subject: Run grid_stat faster
>        Owner: johnhg
>   Requestors: Ellen.Sukovich at noaa.gov
>       Status: new
>  Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=41012 >
>
>
> Hi John,
>
> I am not sure whether to send this to you or to the met help desk.
Let me know if I should email */met_help at ucar.edu instead./*
>
> It is taking a really long time (e.g, ~2-3 minutes per day of data)
to run grid_stat.  (I have to run 10 years! of data that I need to run
through grid_stat.) Is there some parameter in my config file that I
can change to make grid_stat run faster? Or perhaps the CNRFC poly
line I am using (see attached) causing grid_stat to run slowly?  I am
attaching my config file and CNRFC poly line with this with this
email.  At this point the output I am looking for is counts of hits,
misses, false alarms (CTC tables), stats (CTS tables), and matched
pairs if that is possible.
>
> Thanks,
> Ellen

------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #41012] Resolved: Run grid_stat faster
From: Ellen.Sukovich at noaa.gov
Time: Tue Sep 21 13:26:37 2010

  Met's grid_stat is running MUCH faster now (e.g., < 1 min). Thank
you!

On 9/21/2010 10:42 AM, RAL HelpDesk {for John Halley Gotway} wrote:
> According to our records, your request has been resolved. If you
have any
> further questions or concerns, please respond to this message.

--
Ellen Sukovich
University of Colorado,
Cooperative Institute for Research in Environmental Sciences (CIRES),
and NOAA Earth System Research Laboratory (ESRL)
Mail Code: R/PSD2
325 Broadway Street
Boulder, CO 80305
Phone: 303.497.4811
Fax: 303.497.6101
Email: ellen.sukovich at noaa.gov

------------------------------------------------
Subject: Run grid_stat faster
From: John Halley Gotway
Time: Tue Sep 21 13:27:57 2010

Ellen,

Great, glad to hear it.  Just remember that if you change the domain,
you'll need to rerun the gen_poly_mask.

Thanks,
John

------------------------------------------------