[Met_help] [rt.rap.ucar.edu #79242] History for Using Grib2 data with MET5.2 Grid-Stat
John Halley Gotway via RT
met_help at ucar.edu
Fri Jan 27 09:24:00 MST 2017
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
I recently updated my MET package from 5.1 to 5.2 so I could use some grib2 observations in gridstat. I am not experienced using grib data and I am getting a memory error.
"DEBUG 1: Default Config File: /home/s4m/met-5.2/met-5.2_bugfix/share/met/config/GridStatConfig_default
DEBUG 1: User Config File: /work/s4m/MET5.1/scripts/Gridstat//config/GridStatConfig
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_NcCF".
DEBUG 4: NcCfFile::open() -> parsing units for the time variable "hours since 2013-05-18 00:00:00"
DEBUG 4: parse_cf_time_string() -> parsed NetCDF CF convention time unit string "hours since 2013-05-18 00:00:00" as a reference time of 20130518_000000 and 3600 second(s) per time step.
DEBUG 4: NcCfFile::open() -> could not extract init time from the "forecast_reference_time" variable.
DEBUG 4: NcCfFile::open() -> could not extract init time from file name.
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_Gb2".
ERROR :
ERROR : out of memory! Exiting!"
The data I am using is RTMA 2m temperature (ftp://nomads.ncdc.noaa.gov/NDGD/) and the file size is 7.4M. I renamed the files so that they end with ".grb2". I did compile both g2clib and MET5.2 as 64 bit.
I am able to get reproducible gridstat results with other variables from netCDF files. I have tried regridding the grib2 file to the forecast grid within the gridstat config file as well as using regrid_data_plane tool but in both cases I get the same memory error. I also tried converting the grib2 file to netCDF with wgrib2 and I get this gridstat error:
"ERROR : NcCfFile::read_netcdf_grid() -> Couldn't figure out projection from information in netCDF file."
The original grib2 data uses a Lambert Conformal projection. Do you know what I could be doing wrong? I have attached my config file, one observation file (grib2 and netcdf), and one forecast file.
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Using Grib2 data with MET5.2 Grid-Stat
From: John Halley Gotway
Time: Mon Jan 23 13:06:24 2017
Hi Shannon,
Thanks for sending your sample data files. I used your email to
create a met_help support ticket, which enables us to track the
support we provide.
I ran your data to compare the NetCDF forecast file (fcst.nc) to the
GRIB2 file (LTIA98_KWBR_201305180600.grb2) over the FULL model domain.
While Grid-Stat *eventually* ran to completion, it took over 36
minutes to do so! I'd like to debug to figure out what's taking so
long. You already had bootstrapping disabled (n_rep = 0) and rank
correlation statistics turned off (rank_corr_flag = FALSE)... which
are the usual suspects.
Granted, computing statistics over 1,774,484 matched pairs is a lot of
work... but 36 minutes is pretty ridiculous.
I wonder if the issue is related to compiling things with 64 bit
flags. I think our initial goal should be ensuring that you can get
MET to read GRIB2 data. Please try simply plotting data from that
GRIB2 file by running:
met-5.2/bin/plot_data_plane LTIA98_KWBR_201305180600.grb2 tmp_z2.ps
'name="TMP"; level="R2";
Notice that I said "R2" which tells MET to plot record number 2.
Record numbers 1 and 2 both contain temperature data and 2-meters.
Here's some wgrib2 output:
1:0:d=2013051806:TMP:2 m above ground:anl:analysis/forecast error
2:3323062:d=2013051806:TMP:2 m above ground:anl:
So far, all the GRIB id info has been the same between records 1 and
2... but presumably there's a difference somewhere.
Please let me know how the plot_data_plane command goes.
Thanks,
John Halley Gotway
------------------------------------------------
Subject: Using Grib2 data with MET5.2 Grid-Stat
From: Shannon Rees - NOAA Affiliate
Time: Mon Jan 23 13:26:54 2017
Thanks John, I tried the plot_data_plane tool but I got an error:
~/met-5.2/met-5.2_bugfix/bin/plot_data_plane
LTIA98_KWBR_201305180600.grb2
tmp_z2.ps 'name="TMP"; level="R2" ; '
DEBUG 1: Opening data file: LTIA98_KWBR_201305180600.grb2
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Abort
On Mon, Jan 23, 2017 at 3:06 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Hi Shannon,
>
> Thanks for sending your sample data files. I used your email to
create a
> met_help support ticket, which enables us to track the support we
provide.
>
> I ran your data to compare the NetCDF forecast file (fcst.nc) to the
> GRIB2 file (LTIA98_KWBR_201305180600.grb2) over the FULL model
domain.
> While Grid-Stat *eventually* ran to completion, it took over 36
minutes to
> do so! I'd like to debug to figure out what's taking so long. You
already
> had bootstrapping disabled (n_rep = 0) and rank correlation
statistics
> turned off (rank_corr_flag = FALSE)... which are the usual suspects.
>
> Granted, computing statistics over 1,774,484 matched pairs is a lot
of
> work... but 36 minutes is pretty ridiculous.
>
> I wonder if the issue is related to compiling things with 64 bit
flags. I
> think our initial goal should be ensuring that you can get MET to
read
> GRIB2 data. Please try simply plotting data from that GRIB2 file by
> running:
> met-5.2/bin/plot_data_plane LTIA98_KWBR_201305180600.grb2
tmp_z2.ps
> 'name="TMP"; level="R2";
>
> Notice that I said "R2" which tells MET to plot record number 2.
Record
> numbers 1 and 2 both contain temperature data and 2-meters. Here's
some
> wgrib2 output:
>
> 1:0:d=2013051806:TMP:2 m above ground:anl:analysis/forecast error
> 2:3323062:d=2013051806:TMP:2 m above ground:anl:
>
> So far, all the GRIB id info has been the same between records 1 and
2...
> but presumably there's a difference somewhere.
>
> Please let me know how the plot_data_plane command goes.
>
> Thanks,
> John Halley Gotway
>
>
>
--
Shannon Rees
Programming Scientist
Engility
Geophysical Fluid Dynamics Lab
201 Forrestal Rd Princeton, NJ
609-452-5384
------------------------------------------------
Subject: Using Grib2 data with MET5.2 Grid-Stat
From: John Halley Gotway
Time: Mon Jan 23 14:33:03 2017
OK, thanks for letting me know. Let's try to solve this problem and
get
plot_data_plane working on GRIB2 data.
You mentioned that you'd compiled it using 64-bit flags... as I'm sure
you
read from the MET online tutorial. Let's try removing those. Are you
able
to recompile both the GRIB2C library and MET, removing those 64 bit
flags?
And then try that plot_data_plane command again. Please let me know
how it
goes.
Thanks,
John
On Mon, Jan 23, 2017 at 1:26 PM, Shannon Rees - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
>
> Thanks John, I tried the plot_data_plane tool but I got an error:
>
> ~/met-5.2/met-5.2_bugfix/bin/plot_data_plane
LTIA98_KWBR_201305180600.grb2
> tmp_z2.ps 'name="TMP"; level="R2" ; '
> DEBUG 1: Opening data file: LTIA98_KWBR_201305180600.grb2
> terminate called after throwing an instance of 'std::bad_alloc'
> what(): std::bad_alloc
> Abort
>
>
> On Mon, Jan 23, 2017 at 3:06 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Hi Shannon,
> >
> > Thanks for sending your sample data files. I used your email to
create a
> > met_help support ticket, which enables us to track the support we
> provide.
> >
> > I ran your data to compare the NetCDF forecast file (fcst.nc) to
the
> > GRIB2 file (LTIA98_KWBR_201305180600.grb2) over the FULL model
domain.
> > While Grid-Stat *eventually* ran to completion, it took over 36
minutes
> to
> > do so! I'd like to debug to figure out what's taking so long.
You
> already
> > had bootstrapping disabled (n_rep = 0) and rank correlation
statistics
> > turned off (rank_corr_flag = FALSE)... which are the usual
suspects.
> >
> > Granted, computing statistics over 1,774,484 matched pairs is a
lot of
> > work... but 36 minutes is pretty ridiculous.
> >
> > I wonder if the issue is related to compiling things with 64 bit
flags.
> I
> > think our initial goal should be ensuring that you can get MET to
read
> > GRIB2 data. Please try simply plotting data from that GRIB2 file
by
> > running:
> > met-5.2/bin/plot_data_plane LTIA98_KWBR_201305180600.grb2
tmp_z2.ps
> > 'name="TMP"; level="R2";
> >
> > Notice that I said "R2" which tells MET to plot record number 2.
Record
> > numbers 1 and 2 both contain temperature data and 2-meters.
Here's some
> > wgrib2 output:
> >
> > 1:0:d=2013051806:TMP:2 m above ground:anl:analysis/forecast error
> > 2:3323062:d=2013051806 <(201)%20305-1806>:TMP:2 m above
ground:anl:
> >
> > So far, all the GRIB id info has been the same between records 1
and 2...
> > but presumably there's a difference somewhere.
> >
> > Please let me know how the plot_data_plane command goes.
> >
> > Thanks,
> > John Halley Gotway
> >
> >
> >
>
>
> --
> Shannon Rees
> Programming Scientist
> Engility
> Geophysical Fluid Dynamics Lab
> 201 Forrestal Rd Princeton, NJ
> 609-452-5384 <(609)%20452-5384>
>
>
------------------------------------------------
Subject: Using Grib2 data with MET5.2 Grid-Stat
From: John Halley Gotway
Time: Mon Jan 23 14:34:34 2017
Also, when you recompile MET, please save the output to a log file:
make install >& make_install.log
If you continue to have problems, please send me that make_install.log
file
along with the config.log file from the top-level MET directory.
Thanks,
John
On Mon, Jan 23, 2017 at 2:32 PM, John Halley Gotway <johnhg at ucar.edu>
wrote:
> OK, thanks for letting me know. Let's try to solve this problem and
get
> plot_data_plane working on GRIB2 data.
>
> You mentioned that you'd compiled it using 64-bit flags... as I'm
sure you
> read from the MET online tutorial. Let's try removing those. Are
you able
> to recompile both the GRIB2C library and MET, removing those 64 bit
flags?
>
> And then try that plot_data_plane command again. Please let me know
how
> it goes.
>
> Thanks,
> John
>
>
> On Mon, Jan 23, 2017 at 1:26 PM, Shannon Rees - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
>>
>> Thanks John, I tried the plot_data_plane tool but I got an error:
>>
>> ~/met-5.2/met-5.2_bugfix/bin/plot_data_plane
>> LTIA98_KWBR_201305180600.grb2
>> tmp_z2.ps 'name="TMP"; level="R2" ; '
>> DEBUG 1: Opening data file: LTIA98_KWBR_201305180600.grb2
>> terminate called after throwing an instance of 'std::bad_alloc'
>> what(): std::bad_alloc
>> Abort
>>
>>
>> On Mon, Jan 23, 2017 at 3:06 PM, John Halley Gotway via RT <
>> met_help at ucar.edu> wrote:
>>
>> > Hi Shannon,
>> >
>> > Thanks for sending your sample data files. I used your email to
create
>> a
>> > met_help support ticket, which enables us to track the support we
>> provide.
>> >
>> > I ran your data to compare the NetCDF forecast file (fcst.nc) to
the
>> > GRIB2 file (LTIA98_KWBR_201305180600.grb2) over the FULL model
domain.
>> > While Grid-Stat *eventually* ran to completion, it took over 36
minutes
>> to
>> > do so! I'd like to debug to figure out what's taking so long.
You
>> already
>> > had bootstrapping disabled (n_rep = 0) and rank correlation
statistics
>> > turned off (rank_corr_flag = FALSE)... which are the usual
suspects.
>> >
>> > Granted, computing statistics over 1,774,484 matched pairs is a
lot of
>> > work... but 36 minutes is pretty ridiculous.
>> >
>> > I wonder if the issue is related to compiling things with 64 bit
>> flags. I
>> > think our initial goal should be ensuring that you can get MET to
read
>> > GRIB2 data. Please try simply plotting data from that GRIB2 file
by
>> > running:
>> > met-5.2/bin/plot_data_plane LTIA98_KWBR_201305180600.grb2
tmp_z2.ps
>> > 'name="TMP"; level="R2";
>> >
>> > Notice that I said "R2" which tells MET to plot record number 2.
Record
>> > numbers 1 and 2 both contain temperature data and 2-meters.
Here's some
>> > wgrib2 output:
>> >
>> > 1:0:d=2013051806:TMP:2 m above ground:anl:analysis/forecast error
>> > 2:3323062:d=2013051806 <(201)%20305-1806>:TMP:2 m above
ground:anl:
>> >
>> > So far, all the GRIB id info has been the same between records 1
and
>> 2...
>> > but presumably there's a difference somewhere.
>> >
>> > Please let me know how the plot_data_plane command goes.
>> >
>> > Thanks,
>> > John Halley Gotway
>> >
>> >
>> >
>>
>>
>> --
>> Shannon Rees
>> Programming Scientist
>> Engility
>> Geophysical Fluid Dynamics Lab
>> 201 Forrestal Rd Princeton, NJ
>> 609-452-5384 <(609)%20452-5384>
>>
>>
>
------------------------------------------------
Subject: Using Grib2 data with MET5.2 Grid-Stat
From: John Halley Gotway
Time: Mon Jan 23 16:48:04 2017
Hi Shannon,
Just an update. Looking more closely at the GRIB2 file you sent, I
found
the difference between the two temperature records in the "process"
id:
wgrib2 -process LTIA98_KWBR_201305180600.grb2
1:0:code table 4.3=7 anl err
2:3323062:code table 4.3=0 anl
So in the development version of MET, I added support for specifying
"GRIB2_process" (as well as GRIB2_pdt, GRIB2_ens_type, and
GRIB2_der_type). In version 6.0, you'll be able to run this command:
plot_data_plane LTIA98_KWBR_201305180600.grb2 tmp_z2.ps 'name="TMP";
level="Z2"; GRIB2_process=0;'
The "GRIB2_process=0;" option says only match GRIB records whose
process
value is 0 (or change to 7 for the 1st record). In MET version 5.2,
you're
stuck with specifying the exact record number to use:
plot_data_plane LTIA98_KWBR_201305180600.grb2 tmp_z2.ps 'name="TMP";
level="R2";'
Still no word yet on why it's taking so long to run.
John
On Mon, Jan 23, 2017 at 2:34 PM, John Halley Gotway <johnhg at ucar.edu>
wrote:
> Also, when you recompile MET, please save the output to a log file:
> make install >& make_install.log
>
> If you continue to have problems, please send me that
make_install.log
> file along with the config.log file from the top-level MET
directory.
>
> Thanks,
> John
>
> On Mon, Jan 23, 2017 at 2:32 PM, John Halley Gotway
<johnhg at ucar.edu>
> wrote:
>
>> OK, thanks for letting me know. Let's try to solve this problem
and get
>> plot_data_plane working on GRIB2 data.
>>
>> You mentioned that you'd compiled it using 64-bit flags... as I'm
sure
>> you read from the MET online tutorial. Let's try removing those.
Are you
>> able to recompile both the GRIB2C library and MET, removing those
64 bit
>> flags?
>>
>> And then try that plot_data_plane command again. Please let me
know how
>> it goes.
>>
>> Thanks,
>> John
>>
>>
>> On Mon, Jan 23, 2017 at 1:26 PM, Shannon Rees - NOAA Affiliate via
RT <
>> met_help at ucar.edu> wrote:
>>
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
>>>
>>> Thanks John, I tried the plot_data_plane tool but I got an error:
>>>
>>> ~/met-5.2/met-5.2_bugfix/bin/plot_data_plane
>>> LTIA98_KWBR_201305180600.grb2
>>> tmp_z2.ps 'name="TMP"; level="R2" ; '
>>> DEBUG 1: Opening data file: LTIA98_KWBR_201305180600.grb2
>>> terminate called after throwing an instance of 'std::bad_alloc'
>>> what(): std::bad_alloc
>>> Abort
>>>
>>>
>>> On Mon, Jan 23, 2017 at 3:06 PM, John Halley Gotway via RT <
>>> met_help at ucar.edu> wrote:
>>>
>>> > Hi Shannon,
>>> >
>>> > Thanks for sending your sample data files. I used your email to
>>> create a
>>> > met_help support ticket, which enables us to track the support
we
>>> provide.
>>> >
>>> > I ran your data to compare the NetCDF forecast file (fcst.nc) to
the
>>> > GRIB2 file (LTIA98_KWBR_201305180600.grb2) over the FULL model
domain.
>>> > While Grid-Stat *eventually* ran to completion, it took over 36
>>> minutes to
>>> > do so! I'd like to debug to figure out what's taking so long.
You
>>> already
>>> > had bootstrapping disabled (n_rep = 0) and rank correlation
statistics
>>> > turned off (rank_corr_flag = FALSE)... which are the usual
suspects.
>>> >
>>> > Granted, computing statistics over 1,774,484 matched pairs is a
lot of
>>> > work... but 36 minutes is pretty ridiculous.
>>> >
>>> > I wonder if the issue is related to compiling things with 64 bit
>>> flags. I
>>> > think our initial goal should be ensuring that you can get MET
to read
>>> > GRIB2 data. Please try simply plotting data from that GRIB2
file by
>>> > running:
>>> > met-5.2/bin/plot_data_plane LTIA98_KWBR_201305180600.grb2
tmp_z2.ps
>>> > 'name="TMP"; level="R2";
>>> >
>>> > Notice that I said "R2" which tells MET to plot record number 2.
>>> Record
>>> > numbers 1 and 2 both contain temperature data and 2-meters.
Here's
>>> some
>>> > wgrib2 output:
>>> >
>>> > 1:0:d=2013051806:TMP:2 m above ground:anl:analysis/forecast
error
>>> > 2:3323062:d=2013051806 <(201)%20305-1806>:TMP:2 m above
ground:anl:
>>> >
>>> > So far, all the GRIB id info has been the same between records 1
and
>>> 2...
>>> > but presumably there's a difference somewhere.
>>> >
>>> > Please let me know how the plot_data_plane command goes.
>>> >
>>> > Thanks,
>>> > John Halley Gotway
>>> >
>>> >
>>> >
>>>
>>>
>>> --
>>> Shannon Rees
>>> Programming Scientist
>>> Engility
>>> Geophysical Fluid Dynamics Lab
>>> 201 Forrestal Rd Princeton, NJ
>>> 609-452-5384 <(609)%20452-5384>
>>>
>>>
>>
>
------------------------------------------------
Subject: Using Grib2 data with MET5.2 Grid-Stat
From: Shannon Rees - NOAA Affiliate
Time: Tue Jan 24 07:54:20 2017
Hi John,
I recompiled both without 64 bit turned on and the plot_data_plane
command
worked. I plotted both R1 and R2 (attached). I am not sure why R1 is
there or what it is representing. I ran grid-stat again and it
processed
the files and the results look reasonable. It does take a long time.
I
thought this was because of the amount of points, but if you say that
it is
still slower than usual that is promising. It would be very useful if
we
could speed it up. I tried regridding to 13 km first, and it ran much
more
quickly.
My main concern now is how will running without 64 bit affect things?
Will
it hurt my ability to process very large files?
Thanks,
Shannon
On Mon, Jan 23, 2017 at 6:48 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Hi Shannon,
>
> Just an update. Looking more closely at the GRIB2 file you sent, I
found
> the difference between the two temperature records in the "process"
id:
>
> wgrib2 -process LTIA98_KWBR_201305180600.grb2
> 1:0:code table 4.3=7 anl err
> 2:3323062:code table 4.3=0 anl
>
> So in the development version of MET, I added support for specifying
> "GRIB2_process" (as well as GRIB2_pdt, GRIB2_ens_type, and
> GRIB2_der_type). In version 6.0, you'll be able to run this
command:
>
> plot_data_plane LTIA98_KWBR_201305180600.grb2 tmp_z2.ps 'name="TMP";
> level="Z2"; GRIB2_process=0;'
>
> The "GRIB2_process=0;" option says only match GRIB records whose
process
> value is 0 (or change to 7 for the 1st record). In MET version 5.2,
you're
> stuck with specifying the exact record number to use:
>
> plot_data_plane LTIA98_KWBR_201305180600.grb2 tmp_z2.ps 'name="TMP";
> level="R2";'
>
> Still no word yet on why it's taking so long to run.
>
> John
>
>
>
>
> On Mon, Jan 23, 2017 at 2:34 PM, John Halley Gotway
<johnhg at ucar.edu>
> wrote:
>
> > Also, when you recompile MET, please save the output to a log
file:
> > make install >& make_install.log
> >
> > If you continue to have problems, please send me that
make_install.log
> > file along with the config.log file from the top-level MET
directory.
> >
> > Thanks,
> > John
> >
> > On Mon, Jan 23, 2017 at 2:32 PM, John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> >
> >> OK, thanks for letting me know. Let's try to solve this problem
and get
> >> plot_data_plane working on GRIB2 data.
> >>
> >> You mentioned that you'd compiled it using 64-bit flags... as I'm
sure
> >> you read from the MET online tutorial. Let's try removing those.
Are
> you
> >> able to recompile both the GRIB2C library and MET, removing those
64 bit
> >> flags?
> >>
> >> And then try that plot_data_plane command again. Please let me
know how
> >> it goes.
> >>
> >> Thanks,
> >> John
> >>
> >>
> >> On Mon, Jan 23, 2017 at 1:26 PM, Shannon Rees - NOAA Affiliate
via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >>>
> >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
> >>>
> >>> Thanks John, I tried the plot_data_plane tool but I got an
error:
> >>>
> >>> ~/met-5.2/met-5.2_bugfix/bin/plot_data_plane
> >>> LTIA98_KWBR_201305180600.grb2
> >>> tmp_z2.ps 'name="TMP"; level="R2" ; '
> >>> DEBUG 1: Opening data file: LTIA98_KWBR_201305180600.grb2
> >>> terminate called after throwing an instance of 'std::bad_alloc'
> >>> what(): std::bad_alloc
> >>> Abort
> >>>
> >>>
> >>> On Mon, Jan 23, 2017 at 3:06 PM, John Halley Gotway via RT <
> >>> met_help at ucar.edu> wrote:
> >>>
> >>> > Hi Shannon,
> >>> >
> >>> > Thanks for sending your sample data files. I used your email
to
> >>> create a
> >>> > met_help support ticket, which enables us to track the support
we
> >>> provide.
> >>> >
> >>> > I ran your data to compare the NetCDF forecast file (fcst.nc)
to the
> >>> > GRIB2 file (LTIA98_KWBR_201305180600.grb2) over the FULL model
> domain.
> >>> > While Grid-Stat *eventually* ran to completion, it took over
36
> >>> minutes to
> >>> > do so! I'd like to debug to figure out what's taking so long.
You
> >>> already
> >>> > had bootstrapping disabled (n_rep = 0) and rank correlation
> statistics
> >>> > turned off (rank_corr_flag = FALSE)... which are the usual
suspects.
> >>> >
> >>> > Granted, computing statistics over 1,774,484 matched pairs is
a lot
> of
> >>> > work... but 36 minutes is pretty ridiculous.
> >>> >
> >>> > I wonder if the issue is related to compiling things with 64
bit
> >>> flags. I
> >>> > think our initial goal should be ensuring that you can get MET
to
> read
> >>> > GRIB2 data. Please try simply plotting data from that GRIB2
file by
> >>> > running:
> >>> > met-5.2/bin/plot_data_plane LTIA98_KWBR_201305180600.grb2
> tmp_z2.ps
> >>> > 'name="TMP"; level="R2";
> >>> >
> >>> > Notice that I said "R2" which tells MET to plot record number
2.
> >>> Record
> >>> > numbers 1 and 2 both contain temperature data and 2-meters.
Here's
> >>> some
> >>> > wgrib2 output:
> >>> >
> >>> > 1:0:d=2013051806:TMP:2 m above ground:anl:analysis/forecast
error
> >>> > 2:3323062:d=2013051806 <(201)%20305-1806>:TMP:2 m above
ground:anl:
> >>> >
> >>> > So far, all the GRIB id info has been the same between records
1 and
> >>> 2...
> >>> > but presumably there's a difference somewhere.
> >>> >
> >>> > Please let me know how the plot_data_plane command goes.
> >>> >
> >>> > Thanks,
> >>> > John Halley Gotway
> >>> >
> >>> >
> >>> >
> >>>
> >>>
> >>> --
> >>> Shannon Rees
> >>> Programming Scientist
> >>> Engility
> >>> Geophysical Fluid Dynamics Lab
> >>> 201 Forrestal Rd Princeton, NJ
> >>> 609-452-5384 <(609)%20452-5384>
> >>>
> >>>
> >>
> >
>
>
--
Shannon Rees
Programming Scientist
Engility
Geophysical Fluid Dynamics Lab
201 Forrestal Rd Princeton, NJ
609-452-5384 <(609)%20452-5384>
------------------------------------------------
Subject: Using Grib2 data with MET5.2 Grid-Stat
From: John Halley Gotway
Time: Tue Jan 24 09:27:21 2017
Shannon,
I really wouldn't worry about the 64-bit flag setting. I'm just glad
you
were able to run successfully.
I'll try to figure out what's taking so long in Grid-Stat to see if
there's
any way to speed it up.
Thanks,
John
On Tue, Jan 24, 2017 at 7:54 AM, Shannon Rees - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
>
> Hi John,
>
> I recompiled both without 64 bit turned on and the plot_data_plane
command
> worked. I plotted both R1 and R2 (attached). I am not sure why R1
is
> there or what it is representing. I ran grid-stat again and it
processed
> the files and the results look reasonable. It does take a long
time. I
> thought this was because of the amount of points, but if you say
that it is
> still slower than usual that is promising. It would be very useful
if we
> could speed it up. I tried regridding to 13 km first, and it ran
much more
> quickly.
>
> My main concern now is how will running without 64 bit affect
things? Will
> it hurt my ability to process very large files?
>
> Thanks,
> Shannon
>
> On Mon, Jan 23, 2017 at 6:48 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Hi Shannon,
> >
> > Just an update. Looking more closely at the GRIB2 file you sent,
I found
> > the difference between the two temperature records in the
"process" id:
> >
> > wgrib2 -process LTIA98_KWBR_201305180600.grb2
> > 1:0:code table 4.3=7 anl err
> > 2:3323062:code table 4.3=0 anl
> >
> > So in the development version of MET, I added support for
specifying
> > "GRIB2_process" (as well as GRIB2_pdt, GRIB2_ens_type, and
> > GRIB2_der_type). In version 6.0, you'll be able to run this
command:
> >
> > plot_data_plane LTIA98_KWBR_201305180600.grb2 tmp_z2.ps
'name="TMP";
> > level="Z2"; GRIB2_process=0;'
> >
> > The "GRIB2_process=0;" option says only match GRIB records whose
process
> > value is 0 (or change to 7 for the 1st record). In MET version
5.2,
> you're
> > stuck with specifying the exact record number to use:
> >
> > plot_data_plane LTIA98_KWBR_201305180600.grb2 tmp_z2.ps
'name="TMP";
> > level="R2";'
> >
> > Still no word yet on why it's taking so long to run.
> >
> > John
> >
> >
> >
> >
> > On Mon, Jan 23, 2017 at 2:34 PM, John Halley Gotway
<johnhg at ucar.edu>
> > wrote:
> >
> > > Also, when you recompile MET, please save the output to a log
file:
> > > make install >& make_install.log
> > >
> > > If you continue to have problems, please send me that
make_install.log
> > > file along with the config.log file from the top-level MET
directory.
> > >
> > > Thanks,
> > > John
> > >
> > > On Mon, Jan 23, 2017 at 2:32 PM, John Halley Gotway
<johnhg at ucar.edu>
> > > wrote:
> > >
> > >> OK, thanks for letting me know. Let's try to solve this
problem and
> get
> > >> plot_data_plane working on GRIB2 data.
> > >>
> > >> You mentioned that you'd compiled it using 64-bit flags... as
I'm sure
> > >> you read from the MET online tutorial. Let's try removing
those. Are
> > you
> > >> able to recompile both the GRIB2C library and MET, removing
those 64
> bit
> > >> flags?
> > >>
> > >> And then try that plot_data_plane command again. Please let me
know
> how
> > >> it goes.
> > >>
> > >> Thanks,
> > >> John
> > >>
> > >>
> > >> On Mon, Jan 23, 2017 at 1:26 PM, Shannon Rees - NOAA Affiliate
via RT
> <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >>>
> > >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242
>
> > >>>
> > >>> Thanks John, I tried the plot_data_plane tool but I got an
error:
> > >>>
> > >>> ~/met-5.2/met-5.2_bugfix/bin/plot_data_plane
> > >>> LTIA98_KWBR_201305180600.grb2
> > >>> tmp_z2.ps 'name="TMP"; level="R2" ; '
> > >>> DEBUG 1: Opening data file: LTIA98_KWBR_201305180600.grb2
> > >>> terminate called after throwing an instance of
'std::bad_alloc'
> > >>> what(): std::bad_alloc
> > >>> Abort
> > >>>
> > >>>
> > >>> On Mon, Jan 23, 2017 at 3:06 PM, John Halley Gotway via RT <
> > >>> met_help at ucar.edu> wrote:
> > >>>
> > >>> > Hi Shannon,
> > >>> >
> > >>> > Thanks for sending your sample data files. I used your
email to
> > >>> create a
> > >>> > met_help support ticket, which enables us to track the
support we
> > >>> provide.
> > >>> >
> > >>> > I ran your data to compare the NetCDF forecast file
(fcst.nc) to
> the
> > >>> > GRIB2 file (LTIA98_KWBR_201305180600.grb2) over the FULL
model
> > domain.
> > >>> > While Grid-Stat *eventually* ran to completion, it took over
36
> > >>> minutes to
> > >>> > do so! I'd like to debug to figure out what's taking so
long. You
> > >>> already
> > >>> > had bootstrapping disabled (n_rep = 0) and rank correlation
> > statistics
> > >>> > turned off (rank_corr_flag = FALSE)... which are the usual
> suspects.
> > >>> >
> > >>> > Granted, computing statistics over 1,774,484 matched pairs
is a lot
> > of
> > >>> > work... but 36 minutes is pretty ridiculous.
> > >>> >
> > >>> > I wonder if the issue is related to compiling things with 64
bit
> > >>> flags. I
> > >>> > think our initial goal should be ensuring that you can get
MET to
> > read
> > >>> > GRIB2 data. Please try simply plotting data from that GRIB2
file
> by
> > >>> > running:
> > >>> > met-5.2/bin/plot_data_plane LTIA98_KWBR_201305180600.grb2
> > tmp_z2.ps
> > >>> > 'name="TMP"; level="R2";
> > >>> >
> > >>> > Notice that I said "R2" which tells MET to plot record
number 2.
> > >>> Record
> > >>> > numbers 1 and 2 both contain temperature data and 2-meters.
Here's
> > >>> some
> > >>> > wgrib2 output:
> > >>> >
> > >>> > 1:0:d=2013051806:TMP:2 m above ground:anl:analysis/forecast
error
> > >>> > 2:3323062:d=2013051806 <(201)%20305-1806>:TMP:2 m above
> ground:anl:
> > >>> >
> > >>> > So far, all the GRIB id info has been the same between
records 1
> and
> > >>> 2...
> > >>> > but presumably there's a difference somewhere.
> > >>> >
> > >>> > Please let me know how the plot_data_plane command goes.
> > >>> >
> > >>> > Thanks,
> > >>> > John Halley Gotway
> > >>> >
> > >>> >
> > >>> >
> > >>>
> > >>>
> > >>> --
> > >>> Shannon Rees
> > >>> Programming Scientist
> > >>> Engility
> > >>> Geophysical Fluid Dynamics Lab
> > >>> 201 Forrestal Rd Princeton, NJ
> > >>> 609-452-5384 <(609)%20452-5384>
> > >>>
> > >>>
> > >>
> > >
> >
> >
>
>
> --
> Shannon Rees
> Programming Scientist
> Engility
> Geophysical Fluid Dynamics Lab
> 201 Forrestal Rd Princeton, NJ
> 609-452-5384 <(609)%20452-5384>
>
>
------------------------------------------------
Subject: Using Grib2 data with MET5.2 Grid-Stat
From: Shannon Rees - NOAA Affiliate
Time: Tue Jan 24 09:28:45 2017
Great, thanks for your help!
On Tue, Jan 24, 2017 at 11:27 AM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Shannon,
>
> I really wouldn't worry about the 64-bit flag setting. I'm just
glad you
> were able to run successfully.
>
> I'll try to figure out what's taking so long in Grid-Stat to see if
there's
> any way to speed it up.
>
> Thanks,
> John
>
> On Tue, Jan 24, 2017 at 7:54 AM, Shannon Rees - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
> >
> > Hi John,
> >
> > I recompiled both without 64 bit turned on and the plot_data_plane
> command
> > worked. I plotted both R1 and R2 (attached). I am not sure why
R1 is
> > there or what it is representing. I ran grid-stat again and it
processed
> > the files and the results look reasonable. It does take a long
time. I
> > thought this was because of the amount of points, but if you say
that it
> is
> > still slower than usual that is promising. It would be very
useful if we
> > could speed it up. I tried regridding to 13 km first, and it ran
much
> more
> > quickly.
> >
> > My main concern now is how will running without 64 bit affect
things?
> Will
> > it hurt my ability to process very large files?
> >
> > Thanks,
> > Shannon
> >
> > On Mon, Jan 23, 2017 at 6:48 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Hi Shannon,
> > >
> > > Just an update. Looking more closely at the GRIB2 file you
sent, I
> found
> > > the difference between the two temperature records in the
"process" id:
> > >
> > > wgrib2 -process LTIA98_KWBR_201305180600.grb2
> > > 1:0:code table 4.3=7 anl err
> > > 2:3323062:code table 4.3=0 anl
> > >
> > > So in the development version of MET, I added support for
specifying
> > > "GRIB2_process" (as well as GRIB2_pdt, GRIB2_ens_type, and
> > > GRIB2_der_type). In version 6.0, you'll be able to run this
command:
> > >
> > > plot_data_plane LTIA98_KWBR_201305180600.grb2 tmp_z2.ps
'name="TMP";
> > > level="Z2"; GRIB2_process=0;'
> > >
> > > The "GRIB2_process=0;" option says only match GRIB records whose
> process
> > > value is 0 (or change to 7 for the 1st record). In MET version
5.2,
> > you're
> > > stuck with specifying the exact record number to use:
> > >
> > > plot_data_plane LTIA98_KWBR_201305180600.grb2 tmp_z2.ps
'name="TMP";
> > > level="R2";'
> > >
> > > Still no word yet on why it's taking so long to run.
> > >
> > > John
> > >
> > >
> > >
> > >
> > > On Mon, Jan 23, 2017 at 2:34 PM, John Halley Gotway
<johnhg at ucar.edu>
> > > wrote:
> > >
> > > > Also, when you recompile MET, please save the output to a log
file:
> > > > make install >& make_install.log
> > > >
> > > > If you continue to have problems, please send me that
> make_install.log
> > > > file along with the config.log file from the top-level MET
directory.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Mon, Jan 23, 2017 at 2:32 PM, John Halley Gotway
<johnhg at ucar.edu
> >
> > > > wrote:
> > > >
> > > >> OK, thanks for letting me know. Let's try to solve this
problem and
> > get
> > > >> plot_data_plane working on GRIB2 data.
> > > >>
> > > >> You mentioned that you'd compiled it using 64-bit flags... as
I'm
> sure
> > > >> you read from the MET online tutorial. Let's try removing
those.
> Are
> > > you
> > > >> able to recompile both the GRIB2C library and MET, removing
those 64
> > bit
> > > >> flags?
> > > >>
> > > >> And then try that plot_data_plane command again. Please let
me know
> > how
> > > >> it goes.
> > > >>
> > > >> Thanks,
> > > >> John
> > > >>
> > > >>
> > > >> On Mon, Jan 23, 2017 at 1:26 PM, Shannon Rees - NOAA
Affiliate via
> RT
> > <
> > > >> met_help at ucar.edu> wrote:
> > > >>
> > > >>>
> > > >>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
> > > >>>
> > > >>> Thanks John, I tried the plot_data_plane tool but I got an
error:
> > > >>>
> > > >>> ~/met-5.2/met-5.2_bugfix/bin/plot_data_plane
> > > >>> LTIA98_KWBR_201305180600.grb2
> > > >>> tmp_z2.ps 'name="TMP"; level="R2" ; '
> > > >>> DEBUG 1: Opening data file: LTIA98_KWBR_201305180600.grb2
> > > >>> terminate called after throwing an instance of
'std::bad_alloc'
> > > >>> what(): std::bad_alloc
> > > >>> Abort
> > > >>>
> > > >>>
> > > >>> On Mon, Jan 23, 2017 at 3:06 PM, John Halley Gotway via RT <
> > > >>> met_help at ucar.edu> wrote:
> > > >>>
> > > >>> > Hi Shannon,
> > > >>> >
> > > >>> > Thanks for sending your sample data files. I used your
email to
> > > >>> create a
> > > >>> > met_help support ticket, which enables us to track the
support we
> > > >>> provide.
> > > >>> >
> > > >>> > I ran your data to compare the NetCDF forecast file
(fcst.nc) to
> > the
> > > >>> > GRIB2 file (LTIA98_KWBR_201305180600.grb2) over the FULL
model
> > > domain.
> > > >>> > While Grid-Stat *eventually* ran to completion, it took
over 36
> > > >>> minutes to
> > > >>> > do so! I'd like to debug to figure out what's taking so
long.
> You
> > > >>> already
> > > >>> > had bootstrapping disabled (n_rep = 0) and rank
correlation
> > > statistics
> > > >>> > turned off (rank_corr_flag = FALSE)... which are the usual
> > suspects.
> > > >>> >
> > > >>> > Granted, computing statistics over 1,774,484 matched pairs
is a
> lot
> > > of
> > > >>> > work... but 36 minutes is pretty ridiculous.
> > > >>> >
> > > >>> > I wonder if the issue is related to compiling things with
64 bit
> > > >>> flags. I
> > > >>> > think our initial goal should be ensuring that you can get
MET to
> > > read
> > > >>> > GRIB2 data. Please try simply plotting data from that
GRIB2 file
> > by
> > > >>> > running:
> > > >>> > met-5.2/bin/plot_data_plane
LTIA98_KWBR_201305180600.grb2
> > > tmp_z2.ps
> > > >>> > 'name="TMP"; level="R2";
> > > >>> >
> > > >>> > Notice that I said "R2" which tells MET to plot record
number 2.
> > > >>> Record
> > > >>> > numbers 1 and 2 both contain temperature data and 2-
meters.
> Here's
> > > >>> some
> > > >>> > wgrib2 output:
> > > >>> >
> > > >>> > 1:0:d=2013051806:TMP:2 m above
ground:anl:analysis/forecast
> error
> > > >>> > 2:3323062:d=2013051806 <(201)%20305-1806>:TMP:2 m above
> > ground:anl:
> > > >>> >
> > > >>> > So far, all the GRIB id info has been the same between
records 1
> > and
> > > >>> 2...
> > > >>> > but presumably there's a difference somewhere.
> > > >>> >
> > > >>> > Please let me know how the plot_data_plane command goes.
> > > >>> >
> > > >>> > Thanks,
> > > >>> > John Halley Gotway
> > > >>> >
> > > >>> >
> > > >>> >
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Shannon Rees
> > > >>> Programming Scientist
> > > >>> Engility
> > > >>> Geophysical Fluid Dynamics Lab
> > > >>> 201 Forrestal Rd Princeton, NJ
> > > >>> 609-452-5384 <(609)%20452-5384>
> > > >>>
> > > >>>
> > > >>
> > > >
> > >
> > >
> >
> >
> > --
> > Shannon Rees
> > Programming Scientist
> > Engility
> > Geophysical Fluid Dynamics Lab
> > 201 Forrestal Rd Princeton, NJ
> > 609-452-5384 <(609)%20452-5384>
> >
> >
>
>
--
Shannon Rees
Programming Scientist
Engility
Geophysical Fluid Dynamics Lab
201 Forrestal Rd Princeton, NJ
609-452-5384
------------------------------------------------
Subject: Using Grib2 data with MET5.2 Grid-Stat
From: John Halley Gotway
Time: Wed Jan 25 16:16:50 2017
Shannon,
I've been digging into this Grid-Stat timing issue.
When I run Grid-Stat to compute continuous stats on your sample case
it
takes about 36 minutes. When I set the configuration to only do
categorical counts, it takes about 5 seconds!
That's a pretty big difference. Running it through the profiler, I've
found that Grid-Stat is doing some pretty "dumb" memory allocation.
And
those types of problems only become evident on large grids.
The good news is that using the development version of Grid-Stat the
I've
gotten the continuous statistics run time down to 19 seconds.
Now I need to figure out which of the many little changes I made are
the
ones necessary for the improvement.
Thanks,
John
On Tue, Jan 24, 2017 at 9:28 AM, Shannon Rees - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
>
> Great, thanks for your help!
>
> On Tue, Jan 24, 2017 at 11:27 AM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Shannon,
> >
> > I really wouldn't worry about the 64-bit flag setting. I'm just
glad you
> > were able to run successfully.
> >
> > I'll try to figure out what's taking so long in Grid-Stat to see
if
> there's
> > any way to speed it up.
> >
> > Thanks,
> > John
> >
> > On Tue, Jan 24, 2017 at 7:54 AM, Shannon Rees - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
> > >
> > > Hi John,
> > >
> > > I recompiled both without 64 bit turned on and the
plot_data_plane
> > command
> > > worked. I plotted both R1 and R2 (attached). I am not sure why
R1 is
> > > there or what it is representing. I ran grid-stat again and it
> processed
> > > the files and the results look reasonable. It does take a long
time.
> I
> > > thought this was because of the amount of points, but if you say
that
> it
> > is
> > > still slower than usual that is promising. It would be very
useful if
> we
> > > could speed it up. I tried regridding to 13 km first, and it
ran much
> > more
> > > quickly.
> > >
> > > My main concern now is how will running without 64 bit affect
things?
> > Will
> > > it hurt my ability to process very large files?
> > >
> > > Thanks,
> > > Shannon
> > >
> > > On Mon, Jan 23, 2017 at 6:48 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Hi Shannon,
> > > >
> > > > Just an update. Looking more closely at the GRIB2 file you
sent, I
> > found
> > > > the difference between the two temperature records in the
"process"
> id:
> > > >
> > > > wgrib2 -process LTIA98_KWBR_201305180600.grb2
> > > > 1:0:code table 4.3=7 anl err
> > > > 2:3323062:code table 4.3=0 anl
> > > >
> > > > So in the development version of MET, I added support for
specifying
> > > > "GRIB2_process" (as well as GRIB2_pdt, GRIB2_ens_type, and
> > > > GRIB2_der_type). In version 6.0, you'll be able to run this
command:
> > > >
> > > > plot_data_plane LTIA98_KWBR_201305180600.grb2 tmp_z2.ps
'name="TMP";
> > > > level="Z2"; GRIB2_process=0;'
> > > >
> > > > The "GRIB2_process=0;" option says only match GRIB records
whose
> > process
> > > > value is 0 (or change to 7 for the 1st record). In MET
version 5.2,
> > > you're
> > > > stuck with specifying the exact record number to use:
> > > >
> > > > plot_data_plane LTIA98_KWBR_201305180600.grb2 tmp_z2.ps
'name="TMP";
> > > > level="R2";'
> > > >
> > > > Still no word yet on why it's taking so long to run.
> > > >
> > > > John
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Jan 23, 2017 at 2:34 PM, John Halley Gotway
<johnhg at ucar.edu
> >
> > > > wrote:
> > > >
> > > > > Also, when you recompile MET, please save the output to a
log file:
> > > > > make install >& make_install.log
> > > > >
> > > > > If you continue to have problems, please send me that
> > make_install.log
> > > > > file along with the config.log file from the top-level MET
> directory.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Mon, Jan 23, 2017 at 2:32 PM, John Halley Gotway <
> johnhg at ucar.edu
> > >
> > > > > wrote:
> > > > >
> > > > >> OK, thanks for letting me know. Let's try to solve this
problem
> and
> > > get
> > > > >> plot_data_plane working on GRIB2 data.
> > > > >>
> > > > >> You mentioned that you'd compiled it using 64-bit flags...
as I'm
> > sure
> > > > >> you read from the MET online tutorial. Let's try removing
those.
> > Are
> > > > you
> > > > >> able to recompile both the GRIB2C library and MET, removing
those
> 64
> > > bit
> > > > >> flags?
> > > > >>
> > > > >> And then try that plot_data_plane command again. Please
let me
> know
> > > how
> > > > >> it goes.
> > > > >>
> > > > >> Thanks,
> > > > >> John
> > > > >>
> > > > >>
> > > > >> On Mon, Jan 23, 2017 at 1:26 PM, Shannon Rees - NOAA
Affiliate via
> > RT
> > > <
> > > > >> met_help at ucar.edu> wrote:
> > > > >>
> > > > >>>
> > > > >>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
> > > > >>>
> > > > >>> Thanks John, I tried the plot_data_plane tool but I got an
error:
> > > > >>>
> > > > >>> ~/met-5.2/met-5.2_bugfix/bin/plot_data_plane
> > > > >>> LTIA98_KWBR_201305180600.grb2
> > > > >>> tmp_z2.ps 'name="TMP"; level="R2" ; '
> > > > >>> DEBUG 1: Opening data file: LTIA98_KWBR_201305180600.grb2
> > > > >>> terminate called after throwing an instance of
'std::bad_alloc'
> > > > >>> what(): std::bad_alloc
> > > > >>> Abort
> > > > >>>
> > > > >>>
> > > > >>> On Mon, Jan 23, 2017 at 3:06 PM, John Halley Gotway via RT
<
> > > > >>> met_help at ucar.edu> wrote:
> > > > >>>
> > > > >>> > Hi Shannon,
> > > > >>> >
> > > > >>> > Thanks for sending your sample data files. I used your
email
> to
> > > > >>> create a
> > > > >>> > met_help support ticket, which enables us to track the
support
> we
> > > > >>> provide.
> > > > >>> >
> > > > >>> > I ran your data to compare the NetCDF forecast file
(fcst.nc)
> to
> > > the
> > > > >>> > GRIB2 file (LTIA98_KWBR_201305180600.grb2) over the FULL
model
> > > > domain.
> > > > >>> > While Grid-Stat *eventually* ran to completion, it took
over 36
> > > > >>> minutes to
> > > > >>> > do so! I'd like to debug to figure out what's taking so
long.
> > You
> > > > >>> already
> > > > >>> > had bootstrapping disabled (n_rep = 0) and rank
correlation
> > > > statistics
> > > > >>> > turned off (rank_corr_flag = FALSE)... which are the
usual
> > > suspects.
> > > > >>> >
> > > > >>> > Granted, computing statistics over 1,774,484 matched
pairs is a
> > lot
> > > > of
> > > > >>> > work... but 36 minutes is pretty ridiculous.
> > > > >>> >
> > > > >>> > I wonder if the issue is related to compiling things
with 64
> bit
> > > > >>> flags. I
> > > > >>> > think our initial goal should be ensuring that you can
get MET
> to
> > > > read
> > > > >>> > GRIB2 data. Please try simply plotting data from that
GRIB2
> file
> > > by
> > > > >>> > running:
> > > > >>> > met-5.2/bin/plot_data_plane
LTIA98_KWBR_201305180600.grb2
> > > > tmp_z2.ps
> > > > >>> > 'name="TMP"; level="R2";
> > > > >>> >
> > > > >>> > Notice that I said "R2" which tells MET to plot record
number
> 2.
> > > > >>> Record
> > > > >>> > numbers 1 and 2 both contain temperature data and 2-
meters.
> > Here's
> > > > >>> some
> > > > >>> > wgrib2 output:
> > > > >>> >
> > > > >>> > 1:0:d=2013051806:TMP:2 m above
ground:anl:analysis/forecast
> > error
> > > > >>> > 2:3323062:d=2013051806 <(201)%20305-1806>
> <(201)%20305-1806>:TMP:2 m above
> > > ground:anl:
> > > > >>> >
> > > > >>> > So far, all the GRIB id info has been the same between
records
> 1
> > > and
> > > > >>> 2...
> > > > >>> > but presumably there's a difference somewhere.
> > > > >>> >
> > > > >>> > Please let me know how the plot_data_plane command goes.
> > > > >>> >
> > > > >>> > Thanks,
> > > > >>> > John Halley Gotway
> > > > >>> >
> > > > >>> >
> > > > >>> >
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Shannon Rees
> > > > >>> Programming Scientist
> > > > >>> Engility
> > > > >>> Geophysical Fluid Dynamics Lab
> > > > >>> 201 Forrestal Rd Princeton, NJ
> > > > >>> 609-452-5384 <(609)%20452-5384> <(609)%20452-5384>
> > > > >>>
> > > > >>>
> > > > >>
> > > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Shannon Rees
> > > Programming Scientist
> > > Engility
> > > Geophysical Fluid Dynamics Lab
> > > 201 Forrestal Rd Princeton, NJ
> > > 609-452-5384 <(609)%20452-5384> <(609)%20452-5384>
> > >
> > >
> >
> >
>
>
> --
> Shannon Rees
> Programming Scientist
> Engility
> Geophysical Fluid Dynamics Lab
> 201 Forrestal Rd Princeton, NJ
> 609-452-5384 <(609)%20452-5384>
>
>
------------------------------------------------
Subject: Using Grib2 data with MET5.2 Grid-Stat
From: Shannon Rees - NOAA Affiliate
Time: Thu Jan 26 06:57:20 2017
John,
That is good to hear! We are planning on using the MET package to
help us
do a lot of model tuning for the Spring Experiments this year. This
will
require even higher resolutions over CONUS and both continuous and
categorical statistics. We are planning on starting this testing
shortly
and it will speed up the whole process if MET moves faster.
Thanks,
Shannon
On Wed, Jan 25, 2017 at 6:16 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Shannon,
>
> I've been digging into this Grid-Stat timing issue.
>
> When I run Grid-Stat to compute continuous stats on your sample case
it
> takes about 36 minutes. When I set the configuration to only do
> categorical counts, it takes about 5 seconds!
>
> That's a pretty big difference. Running it through the profiler,
I've
> found that Grid-Stat is doing some pretty "dumb" memory allocation.
And
> those types of problems only become evident on large grids.
>
> The good news is that using the development version of Grid-Stat the
I've
> gotten the continuous statistics run time down to 19 seconds.
>
> Now I need to figure out which of the many little changes I made are
the
> ones necessary for the improvement.
>
> Thanks,
> John
>
> On Tue, Jan 24, 2017 at 9:28 AM, Shannon Rees - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
> >
> > Great, thanks for your help!
> >
> > On Tue, Jan 24, 2017 at 11:27 AM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Shannon,
> > >
> > > I really wouldn't worry about the 64-bit flag setting. I'm just
glad
> you
> > > were able to run successfully.
> > >
> > > I'll try to figure out what's taking so long in Grid-Stat to see
if
> > there's
> > > any way to speed it up.
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Jan 24, 2017 at 7:54 AM, Shannon Rees - NOAA Affiliate
via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242
>
> > > >
> > > > Hi John,
> > > >
> > > > I recompiled both without 64 bit turned on and the
plot_data_plane
> > > command
> > > > worked. I plotted both R1 and R2 (attached). I am not sure
why R1
> is
> > > > there or what it is representing. I ran grid-stat again and
it
> > processed
> > > > the files and the results look reasonable. It does take a
long time.
> > I
> > > > thought this was because of the amount of points, but if you
say that
> > it
> > > is
> > > > still slower than usual that is promising. It would be very
useful
> if
> > we
> > > > could speed it up. I tried regridding to 13 km first, and it
ran
> much
> > > more
> > > > quickly.
> > > >
> > > > My main concern now is how will running without 64 bit affect
things?
> > > Will
> > > > it hurt my ability to process very large files?
> > > >
> > > > Thanks,
> > > > Shannon
> > > >
> > > > On Mon, Jan 23, 2017 at 6:48 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Hi Shannon,
> > > > >
> > > > > Just an update. Looking more closely at the GRIB2 file you
sent, I
> > > found
> > > > > the difference between the two temperature records in the
"process"
> > id:
> > > > >
> > > > > wgrib2 -process LTIA98_KWBR_201305180600.grb2
> > > > > 1:0:code table 4.3=7 anl err
> > > > > 2:3323062:code table 4.3=0 anl
> > > > >
> > > > > So in the development version of MET, I added support for
> specifying
> > > > > "GRIB2_process" (as well as GRIB2_pdt, GRIB2_ens_type, and
> > > > > GRIB2_der_type). In version 6.0, you'll be able to run this
> command:
> > > > >
> > > > > plot_data_plane LTIA98_KWBR_201305180600.grb2 tmp_z2.ps
> 'name="TMP";
> > > > > level="Z2"; GRIB2_process=0;'
> > > > >
> > > > > The "GRIB2_process=0;" option says only match GRIB records
whose
> > > process
> > > > > value is 0 (or change to 7 for the 1st record). In MET
version
> 5.2,
> > > > you're
> > > > > stuck with specifying the exact record number to use:
> > > > >
> > > > > plot_data_plane LTIA98_KWBR_201305180600.grb2 tmp_z2.ps
> 'name="TMP";
> > > > > level="R2";'
> > > > >
> > > > > Still no word yet on why it's taking so long to run.
> > > > >
> > > > > John
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Jan 23, 2017 at 2:34 PM, John Halley Gotway <
> johnhg at ucar.edu
> > >
> > > > > wrote:
> > > > >
> > > > > > Also, when you recompile MET, please save the output to a
log
> file:
> > > > > > make install >& make_install.log
> > > > > >
> > > > > > If you continue to have problems, please send me that
> > > make_install.log
> > > > > > file along with the config.log file from the top-level MET
> > directory.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Mon, Jan 23, 2017 at 2:32 PM, John Halley Gotway <
> > johnhg at ucar.edu
> > > >
> > > > > > wrote:
> > > > > >
> > > > > >> OK, thanks for letting me know. Let's try to solve this
problem
> > and
> > > > get
> > > > > >> plot_data_plane working on GRIB2 data.
> > > > > >>
> > > > > >> You mentioned that you'd compiled it using 64-bit
flags... as
> I'm
> > > sure
> > > > > >> you read from the MET online tutorial. Let's try
removing
> those.
> > > Are
> > > > > you
> > > > > >> able to recompile both the GRIB2C library and MET,
removing
> those
> > 64
> > > > bit
> > > > > >> flags?
> > > > > >>
> > > > > >> And then try that plot_data_plane command again. Please
let me
> > know
> > > > how
> > > > > >> it goes.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> John
> > > > > >>
> > > > > >>
> > > > > >> On Mon, Jan 23, 2017 at 1:26 PM, Shannon Rees - NOAA
Affiliate
> via
> > > RT
> > > > <
> > > > > >> met_help at ucar.edu> wrote:
> > > > > >>
> > > > > >>>
> > > > > >>> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242
> >
> > > > > >>>
> > > > > >>> Thanks John, I tried the plot_data_plane tool but I got
an
> error:
> > > > > >>>
> > > > > >>> ~/met-5.2/met-5.2_bugfix/bin/plot_data_plane
> > > > > >>> LTIA98_KWBR_201305180600.grb2
> > > > > >>> tmp_z2.ps 'name="TMP"; level="R2" ; '
> > > > > >>> DEBUG 1: Opening data file:
LTIA98_KWBR_201305180600.grb2
> > > > > >>> terminate called after throwing an instance of
'std::bad_alloc'
> > > > > >>> what(): std::bad_alloc
> > > > > >>> Abort
> > > > > >>>
> > > > > >>>
> > > > > >>> On Mon, Jan 23, 2017 at 3:06 PM, John Halley Gotway via
RT <
> > > > > >>> met_help at ucar.edu> wrote:
> > > > > >>>
> > > > > >>> > Hi Shannon,
> > > > > >>> >
> > > > > >>> > Thanks for sending your sample data files. I used
your email
> > to
> > > > > >>> create a
> > > > > >>> > met_help support ticket, which enables us to track the
> support
> > we
> > > > > >>> provide.
> > > > > >>> >
> > > > > >>> > I ran your data to compare the NetCDF forecast file
(fcst.nc
> )
> > to
> > > > the
> > > > > >>> > GRIB2 file (LTIA98_KWBR_201305180600.grb2) over the
FULL
> model
> > > > > domain.
> > > > > >>> > While Grid-Stat *eventually* ran to completion, it
took over
> 36
> > > > > >>> minutes to
> > > > > >>> > do so! I'd like to debug to figure out what's taking
so
> long.
> > > You
> > > > > >>> already
> > > > > >>> > had bootstrapping disabled (n_rep = 0) and rank
correlation
> > > > > statistics
> > > > > >>> > turned off (rank_corr_flag = FALSE)... which are the
usual
> > > > suspects.
> > > > > >>> >
> > > > > >>> > Granted, computing statistics over 1,774,484 matched
pairs
> is a
> > > lot
> > > > > of
> > > > > >>> > work... but 36 minutes is pretty ridiculous.
> > > > > >>> >
> > > > > >>> > I wonder if the issue is related to compiling things
with 64
> > bit
> > > > > >>> flags. I
> > > > > >>> > think our initial goal should be ensuring that you can
get
> MET
> > to
> > > > > read
> > > > > >>> > GRIB2 data. Please try simply plotting data from that
GRIB2
> > file
> > > > by
> > > > > >>> > running:
> > > > > >>> > met-5.2/bin/plot_data_plane
LTIA98_KWBR_201305180600.grb2
> > > > > tmp_z2.ps
> > > > > >>> > 'name="TMP"; level="R2";
> > > > > >>> >
> > > > > >>> > Notice that I said "R2" which tells MET to plot record
number
> > 2.
> > > > > >>> Record
> > > > > >>> > numbers 1 and 2 both contain temperature data and 2-
meters.
> > > Here's
> > > > > >>> some
> > > > > >>> > wgrib2 output:
> > > > > >>> >
> > > > > >>> > 1:0:d=2013051806:TMP:2 m above
ground:anl:analysis/forecast
> > > error
> > > > > >>> > 2:3323062:d=2013051806 <(201)%20305-1806>
> > <(201)%20305-1806>:TMP:2 m above
> > > > ground:anl:
> > > > > >>> >
> > > > > >>> > So far, all the GRIB id info has been the same between
> records
> > 1
> > > > and
> > > > > >>> 2...
> > > > > >>> > but presumably there's a difference somewhere.
> > > > > >>> >
> > > > > >>> > Please let me know how the plot_data_plane command
goes.
> > > > > >>> >
> > > > > >>> > Thanks,
> > > > > >>> > John Halley Gotway
> > > > > >>> >
> > > > > >>> >
> > > > > >>> >
> > > > > >>>
> > > > > >>>
> > > > > >>> --
> > > > > >>> Shannon Rees
> > > > > >>> Programming Scientist
> > > > > >>> Engility
> > > > > >>> Geophysical Fluid Dynamics Lab
> > > > > >>> 201 Forrestal Rd Princeton, NJ
> > > > > >>> 609-452-5384 <(609)%20452-5384> <(609)%20452-5384>
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Shannon Rees
> > > > Programming Scientist
> > > > Engility
> > > > Geophysical Fluid Dynamics Lab
> > > > 201 Forrestal Rd Princeton, NJ
> > > > 609-452-5384 <(609)%20452-5384> <(609)%20452-5384>
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > Shannon Rees
> > Programming Scientist
> > Engility
> > Geophysical Fluid Dynamics Lab
> > 201 Forrestal Rd Princeton, NJ
> > 609-452-5384 <(609)%20452-5384>
> >
> >
>
>
--
Shannon Rees
Programming Scientist
Engility
Geophysical Fluid Dynamics Lab
201 Forrestal Rd Princeton, NJ
609-452-5384
------------------------------------------------
Subject: Using Grib2 data with MET5.2 Grid-Stat
From: John Halley Gotway
Time: Thu Jan 26 09:58:35 2017
Shannon,
I have a question for you. The changes I'm making will definitely be
included in the next version of MET, version 6.0 due out in March.
Is that sufficient or do you really need a patch for version 5.2 for
your
work? If so, I'll need to copy the updates into the release branch
and
post a bugfix.
Thanks,
John
On Thu, Jan 26, 2017 at 6:57 AM, Shannon Rees - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
>
> John,
>
> That is good to hear! We are planning on using the MET package to
help us
> do a lot of model tuning for the Spring Experiments this year. This
will
> require even higher resolutions over CONUS and both continuous and
> categorical statistics. We are planning on starting this testing
shortly
> and it will speed up the whole process if MET moves faster.
>
> Thanks,
> Shannon
>
> On Wed, Jan 25, 2017 at 6:16 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Shannon,
> >
> > I've been digging into this Grid-Stat timing issue.
> >
> > When I run Grid-Stat to compute continuous stats on your sample
case it
> > takes about 36 minutes. When I set the configuration to only do
> > categorical counts, it takes about 5 seconds!
> >
> > That's a pretty big difference. Running it through the profiler,
I've
> > found that Grid-Stat is doing some pretty "dumb" memory
allocation. And
> > those types of problems only become evident on large grids.
> >
> > The good news is that using the development version of Grid-Stat
the I've
> > gotten the continuous statistics run time down to 19 seconds.
> >
> > Now I need to figure out which of the many little changes I made
are the
> > ones necessary for the improvement.
> >
> > Thanks,
> > John
> >
> > On Tue, Jan 24, 2017 at 9:28 AM, Shannon Rees - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
> > >
> > > Great, thanks for your help!
> > >
> > > On Tue, Jan 24, 2017 at 11:27 AM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Shannon,
> > > >
> > > > I really wouldn't worry about the 64-bit flag setting. I'm
just glad
> > you
> > > > were able to run successfully.
> > > >
> > > > I'll try to figure out what's taking so long in Grid-Stat to
see if
> > > there's
> > > > any way to speed it up.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Tue, Jan 24, 2017 at 7:54 AM, Shannon Rees - NOAA Affiliate
via
> RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
> > > > >
> > > > > Hi John,
> > > > >
> > > > > I recompiled both without 64 bit turned on and the
plot_data_plane
> > > > command
> > > > > worked. I plotted both R1 and R2 (attached). I am not sure
why R1
> > is
> > > > > there or what it is representing. I ran grid-stat again and
it
> > > processed
> > > > > the files and the results look reasonable. It does take a
long
> time.
> > > I
> > > > > thought this was because of the amount of points, but if you
say
> that
> > > it
> > > > is
> > > > > still slower than usual that is promising. It would be very
useful
> > if
> > > we
> > > > > could speed it up. I tried regridding to 13 km first, and
it ran
> > much
> > > > more
> > > > > quickly.
> > > > >
> > > > > My main concern now is how will running without 64 bit
affect
> things?
> > > > Will
> > > > > it hurt my ability to process very large files?
> > > > >
> > > > > Thanks,
> > > > > Shannon
> > > > >
> > > > > On Mon, Jan 23, 2017 at 6:48 PM, John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > > Hi Shannon,
> > > > > >
> > > > > > Just an update. Looking more closely at the GRIB2 file
you
> sent, I
> > > > found
> > > > > > the difference between the two temperature records in the
> "process"
> > > id:
> > > > > >
> > > > > > wgrib2 -process LTIA98_KWBR_201305180600.grb2
> > > > > > 1:0:code table 4.3=7 anl err
> > > > > > 2:3323062:code table 4.3=0 anl
> > > > > >
> > > > > > So in the development version of MET, I added support for
> > specifying
> > > > > > "GRIB2_process" (as well as GRIB2_pdt, GRIB2_ens_type, and
> > > > > > GRIB2_der_type). In version 6.0, you'll be able to run
this
> > command:
> > > > > >
> > > > > > plot_data_plane LTIA98_KWBR_201305180600.grb2 tmp_z2.ps
> > 'name="TMP";
> > > > > > level="Z2"; GRIB2_process=0;'
> > > > > >
> > > > > > The "GRIB2_process=0;" option says only match GRIB records
whose
> > > > process
> > > > > > value is 0 (or change to 7 for the 1st record). In MET
version
> > 5.2,
> > > > > you're
> > > > > > stuck with specifying the exact record number to use:
> > > > > >
> > > > > > plot_data_plane LTIA98_KWBR_201305180600.grb2 tmp_z2.ps
> > 'name="TMP";
> > > > > > level="R2";'
> > > > > >
> > > > > > Still no word yet on why it's taking so long to run.
> > > > > >
> > > > > > John
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Jan 23, 2017 at 2:34 PM, John Halley Gotway <
> > johnhg at ucar.edu
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Also, when you recompile MET, please save the output to
a log
> > file:
> > > > > > > make install >& make_install.log
> > > > > > >
> > > > > > > If you continue to have problems, please send me that
> > > > make_install.log
> > > > > > > file along with the config.log file from the top-level
MET
> > > directory.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Mon, Jan 23, 2017 at 2:32 PM, John Halley Gotway <
> > > johnhg at ucar.edu
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> OK, thanks for letting me know. Let's try to solve
this
> problem
> > > and
> > > > > get
> > > > > > >> plot_data_plane working on GRIB2 data.
> > > > > > >>
> > > > > > >> You mentioned that you'd compiled it using 64-bit
flags... as
> > I'm
> > > > sure
> > > > > > >> you read from the MET online tutorial. Let's try
removing
> > those.
> > > > Are
> > > > > > you
> > > > > > >> able to recompile both the GRIB2C library and MET,
removing
> > those
> > > 64
> > > > > bit
> > > > > > >> flags?
> > > > > > >>
> > > > > > >> And then try that plot_data_plane command again.
Please let
> me
> > > know
> > > > > how
> > > > > > >> it goes.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> John
> > > > > > >>
> > > > > > >>
> > > > > > >> On Mon, Jan 23, 2017 at 1:26 PM, Shannon Rees - NOAA
Affiliate
> > via
> > > > RT
> > > > > <
> > > > > > >> met_help at ucar.edu> wrote:
> > > > > > >>
> > > > > > >>>
> > > > > > >>> <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=79242
> > >
> > > > > > >>>
> > > > > > >>> Thanks John, I tried the plot_data_plane tool but I
got an
> > error:
> > > > > > >>>
> > > > > > >>> ~/met-5.2/met-5.2_bugfix/bin/plot_data_plane
> > > > > > >>> LTIA98_KWBR_201305180600.grb2
> > > > > > >>> tmp_z2.ps 'name="TMP"; level="R2" ; '
> > > > > > >>> DEBUG 1: Opening data file:
LTIA98_KWBR_201305180600.grb2
> > > > > > >>> terminate called after throwing an instance of
> 'std::bad_alloc'
> > > > > > >>> what(): std::bad_alloc
> > > > > > >>> Abort
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> On Mon, Jan 23, 2017 at 3:06 PM, John Halley Gotway
via RT <
> > > > > > >>> met_help at ucar.edu> wrote:
> > > > > > >>>
> > > > > > >>> > Hi Shannon,
> > > > > > >>> >
> > > > > > >>> > Thanks for sending your sample data files. I used
your
> email
> > > to
> > > > > > >>> create a
> > > > > > >>> > met_help support ticket, which enables us to track
the
> > support
> > > we
> > > > > > >>> provide.
> > > > > > >>> >
> > > > > > >>> > I ran your data to compare the NetCDF forecast file
(
> fcst.nc
> > )
> > > to
> > > > > the
> > > > > > >>> > GRIB2 file (LTIA98_KWBR_201305180600.grb2) over the
FULL
> > model
> > > > > > domain.
> > > > > > >>> > While Grid-Stat *eventually* ran to completion, it
took
> over
> > 36
> > > > > > >>> minutes to
> > > > > > >>> > do so! I'd like to debug to figure out what's
taking so
> > long.
> > > > You
> > > > > > >>> already
> > > > > > >>> > had bootstrapping disabled (n_rep = 0) and rank
correlation
> > > > > > statistics
> > > > > > >>> > turned off (rank_corr_flag = FALSE)... which are the
usual
> > > > > suspects.
> > > > > > >>> >
> > > > > > >>> > Granted, computing statistics over 1,774,484 matched
pairs
> > is a
> > > > lot
> > > > > > of
> > > > > > >>> > work... but 36 minutes is pretty ridiculous.
> > > > > > >>> >
> > > > > > >>> > I wonder if the issue is related to compiling things
with
> 64
> > > bit
> > > > > > >>> flags. I
> > > > > > >>> > think our initial goal should be ensuring that you
can get
> > MET
> > > to
> > > > > > read
> > > > > > >>> > GRIB2 data. Please try simply plotting data from
that
> GRIB2
> > > file
> > > > > by
> > > > > > >>> > running:
> > > > > > >>> > met-5.2/bin/plot_data_plane
> LTIA98_KWBR_201305180600.grb2
> > > > > > tmp_z2.ps
> > > > > > >>> > 'name="TMP"; level="R2";
> > > > > > >>> >
> > > > > > >>> > Notice that I said "R2" which tells MET to plot
record
> number
> > > 2.
> > > > > > >>> Record
> > > > > > >>> > numbers 1 and 2 both contain temperature data and 2-
meters.
> > > > Here's
> > > > > > >>> some
> > > > > > >>> > wgrib2 output:
> > > > > > >>> >
> > > > > > >>> > 1:0:d=2013051806:TMP:2 m above
> ground:anl:analysis/forecast
> > > > error
> > > > > > >>> > 2:3323062:d=2013051806 <(201)%20305-1806>
> > > <(201)%20305-1806>:TMP:2 m above
> > > > > ground:anl:
> > > > > > >>> >
> > > > > > >>> > So far, all the GRIB id info has been the same
between
> > records
> > > 1
> > > > > and
> > > > > > >>> 2...
> > > > > > >>> > but presumably there's a difference somewhere.
> > > > > > >>> >
> > > > > > >>> > Please let me know how the plot_data_plane command
goes.
> > > > > > >>> >
> > > > > > >>> > Thanks,
> > > > > > >>> > John Halley Gotway
> > > > > > >>> >
> > > > > > >>> >
> > > > > > >>> >
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> --
> > > > > > >>> Shannon Rees
> > > > > > >>> Programming Scientist
> > > > > > >>> Engility
> > > > > > >>> Geophysical Fluid Dynamics Lab
> > > > > > >>> 201 Forrestal Rd Princeton, NJ
> > > > > > >>> 609-452-5384 <(609)%20452-5384> <(609)%20452-5384>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Shannon Rees
> > > > > Programming Scientist
> > > > > Engility
> > > > > Geophysical Fluid Dynamics Lab
> > > > > 201 Forrestal Rd Princeton, NJ
> > > > > 609-452-5384 <(609)%20452-5384> <(609)%20452-5384>
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Shannon Rees
> > > Programming Scientist
> > > Engility
> > > Geophysical Fluid Dynamics Lab
> > > 201 Forrestal Rd Princeton, NJ
> > > 609-452-5384 <(609)%20452-5384>
> > >
> > >
> >
> >
>
>
> --
> Shannon Rees
> Programming Scientist
> Engility
> Geophysical Fluid Dynamics Lab
> 201 Forrestal Rd Princeton, NJ
> 609-452-5384
>
>
------------------------------------------------
Subject: Using Grib2 data with MET5.2 Grid-Stat
From: Shannon Rees - NOAA Affiliate
Time: Thu Jan 26 10:17:15 2017
Hi John,
It would definitely be preferable to have the bugfix as soon as
possible.
We will likely start the testing next week, since we have a lot of
work to
do to be ready for the Spring Experiments. Let me know if an earlier
bugfix
is possible. Do you know if this is only a problem with grid-stat or
does
it affect other tools like point-stat?
Thanks so much,
Shannon
On Thu, Jan 26, 2017 at 11:58 AM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Shannon,
>
> I have a question for you. The changes I'm making will definitely
be
> included in the next version of MET, version 6.0 due out in March.
>
> Is that sufficient or do you really need a patch for version 5.2 for
your
> work? If so, I'll need to copy the updates into the release branch
and
> post a bugfix.
>
> Thanks,
> John
>
>
>
> On Thu, Jan 26, 2017 at 6:57 AM, Shannon Rees - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
> >
> > John,
> >
> > That is good to hear! We are planning on using the MET package to
help
> us
> > do a lot of model tuning for the Spring Experiments this year.
This will
> > require even higher resolutions over CONUS and both continuous and
> > categorical statistics. We are planning on starting this testing
shortly
> > and it will speed up the whole process if MET moves faster.
> >
> > Thanks,
> > Shannon
> >
> > On Wed, Jan 25, 2017 at 6:16 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Shannon,
> > >
> > > I've been digging into this Grid-Stat timing issue.
> > >
> > > When I run Grid-Stat to compute continuous stats on your sample
case it
> > > takes about 36 minutes. When I set the configuration to only do
> > > categorical counts, it takes about 5 seconds!
> > >
> > > That's a pretty big difference. Running it through the
profiler, I've
> > > found that Grid-Stat is doing some pretty "dumb" memory
allocation.
> And
> > > those types of problems only become evident on large grids.
> > >
> > > The good news is that using the development version of Grid-Stat
the
> I've
> > > gotten the continuous statistics run time down to 19 seconds.
> > >
> > > Now I need to figure out which of the many little changes I made
are
> the
> > > ones necessary for the improvement.
> > >
> > > Thanks,
> > > John
> > >
> > > On Tue, Jan 24, 2017 at 9:28 AM, Shannon Rees - NOAA Affiliate
via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242
>
> > > >
> > > > Great, thanks for your help!
> > > >
> > > > On Tue, Jan 24, 2017 at 11:27 AM, John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Shannon,
> > > > >
> > > > > I really wouldn't worry about the 64-bit flag setting. I'm
just
> glad
> > > you
> > > > > were able to run successfully.
> > > > >
> > > > > I'll try to figure out what's taking so long in Grid-Stat to
see if
> > > > there's
> > > > > any way to speed it up.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Tue, Jan 24, 2017 at 7:54 AM, Shannon Rees - NOAA
Affiliate via
> > RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
> > > > > >
> > > > > > Hi John,
> > > > > >
> > > > > > I recompiled both without 64 bit turned on and the
> plot_data_plane
> > > > > command
> > > > > > worked. I plotted both R1 and R2 (attached). I am not
sure why
> R1
> > > is
> > > > > > there or what it is representing. I ran grid-stat again
and it
> > > > processed
> > > > > > the files and the results look reasonable. It does take a
long
> > time.
> > > > I
> > > > > > thought this was because of the amount of points, but if
you say
> > that
> > > > it
> > > > > is
> > > > > > still slower than usual that is promising. It would be
very
> useful
> > > if
> > > > we
> > > > > > could speed it up. I tried regridding to 13 km first, and
it ran
> > > much
> > > > > more
> > > > > > quickly.
> > > > > >
> > > > > > My main concern now is how will running without 64 bit
affect
> > things?
> > > > > Will
> > > > > > it hurt my ability to process very large files?
> > > > > >
> > > > > > Thanks,
> > > > > > Shannon
> > > > > >
> > > > > > On Mon, Jan 23, 2017 at 6:48 PM, John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > > Hi Shannon,
> > > > > > >
> > > > > > > Just an update. Looking more closely at the GRIB2 file
you
> > sent, I
> > > > > found
> > > > > > > the difference between the two temperature records in
the
> > "process"
> > > > id:
> > > > > > >
> > > > > > > wgrib2 -process LTIA98_KWBR_201305180600.grb2
> > > > > > > 1:0:code table 4.3=7 anl err
> > > > > > > 2:3323062:code table 4.3=0 anl
> > > > > > >
> > > > > > > So in the development version of MET, I added support
for
> > > specifying
> > > > > > > "GRIB2_process" (as well as GRIB2_pdt, GRIB2_ens_type,
and
> > > > > > > GRIB2_der_type). In version 6.0, you'll be able to run
this
> > > command:
> > > > > > >
> > > > > > > plot_data_plane LTIA98_KWBR_201305180600.grb2 tmp_z2.ps
> > > 'name="TMP";
> > > > > > > level="Z2"; GRIB2_process=0;'
> > > > > > >
> > > > > > > The "GRIB2_process=0;" option says only match GRIB
records
> whose
> > > > > process
> > > > > > > value is 0 (or change to 7 for the 1st record). In MET
version
> > > 5.2,
> > > > > > you're
> > > > > > > stuck with specifying the exact record number to use:
> > > > > > >
> > > > > > > plot_data_plane LTIA98_KWBR_201305180600.grb2 tmp_z2.ps
> > > 'name="TMP";
> > > > > > > level="R2";'
> > > > > > >
> > > > > > > Still no word yet on why it's taking so long to run.
> > > > > > >
> > > > > > > John
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jan 23, 2017 at 2:34 PM, John Halley Gotway <
> > > johnhg at ucar.edu
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Also, when you recompile MET, please save the output
to a log
> > > file:
> > > > > > > > make install >& make_install.log
> > > > > > > >
> > > > > > > > If you continue to have problems, please send me that
> > > > > make_install.log
> > > > > > > > file along with the config.log file from the top-level
MET
> > > > directory.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Mon, Jan 23, 2017 at 2:32 PM, John Halley Gotway <
> > > > johnhg at ucar.edu
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> OK, thanks for letting me know. Let's try to solve
this
> > problem
> > > > and
> > > > > > get
> > > > > > > >> plot_data_plane working on GRIB2 data.
> > > > > > > >>
> > > > > > > >> You mentioned that you'd compiled it using 64-bit
flags...
> as
> > > I'm
> > > > > sure
> > > > > > > >> you read from the MET online tutorial. Let's try
removing
> > > those.
> > > > > Are
> > > > > > > you
> > > > > > > >> able to recompile both the GRIB2C library and MET,
removing
> > > those
> > > > 64
> > > > > > bit
> > > > > > > >> flags?
> > > > > > > >>
> > > > > > > >> And then try that plot_data_plane command again.
Please let
> > me
> > > > know
> > > > > > how
> > > > > > > >> it goes.
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >> John
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Mon, Jan 23, 2017 at 1:26 PM, Shannon Rees - NOAA
> Affiliate
> > > via
> > > > > RT
> > > > > > <
> > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > >>
> > > > > > > >>>
> > > > > > > >>> <URL: https://rt.rap.ucar.edu/rt/
> > Ticket/Display.html?id=79242
> > > >
> > > > > > > >>>
> > > > > > > >>> Thanks John, I tried the plot_data_plane tool but I
got an
> > > error:
> > > > > > > >>>
> > > > > > > >>> ~/met-5.2/met-5.2_bugfix/bin/plot_data_plane
> > > > > > > >>> LTIA98_KWBR_201305180600.grb2
> > > > > > > >>> tmp_z2.ps 'name="TMP"; level="R2" ; '
> > > > > > > >>> DEBUG 1: Opening data file:
LTIA98_KWBR_201305180600.grb2
> > > > > > > >>> terminate called after throwing an instance of
> > 'std::bad_alloc'
> > > > > > > >>> what(): std::bad_alloc
> > > > > > > >>> Abort
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> On Mon, Jan 23, 2017 at 3:06 PM, John Halley Gotway
via RT
> <
> > > > > > > >>> met_help at ucar.edu> wrote:
> > > > > > > >>>
> > > > > > > >>> > Hi Shannon,
> > > > > > > >>> >
> > > > > > > >>> > Thanks for sending your sample data files. I used
your
> > email
> > > > to
> > > > > > > >>> create a
> > > > > > > >>> > met_help support ticket, which enables us to track
the
> > > support
> > > > we
> > > > > > > >>> provide.
> > > > > > > >>> >
> > > > > > > >>> > I ran your data to compare the NetCDF forecast
file (
> > fcst.nc
> > > )
> > > > to
> > > > > > the
> > > > > > > >>> > GRIB2 file (LTIA98_KWBR_201305180600.grb2) over
the FULL
> > > model
> > > > > > > domain.
> > > > > > > >>> > While Grid-Stat *eventually* ran to completion, it
took
> > over
> > > 36
> > > > > > > >>> minutes to
> > > > > > > >>> > do so! I'd like to debug to figure out what's
taking so
> > > long.
> > > > > You
> > > > > > > >>> already
> > > > > > > >>> > had bootstrapping disabled (n_rep = 0) and rank
> correlation
> > > > > > > statistics
> > > > > > > >>> > turned off (rank_corr_flag = FALSE)... which are
the
> usual
> > > > > > suspects.
> > > > > > > >>> >
> > > > > > > >>> > Granted, computing statistics over 1,774,484
matched
> pairs
> > > is a
> > > > > lot
> > > > > > > of
> > > > > > > >>> > work... but 36 minutes is pretty ridiculous.
> > > > > > > >>> >
> > > > > > > >>> > I wonder if the issue is related to compiling
things with
> > 64
> > > > bit
> > > > > > > >>> flags. I
> > > > > > > >>> > think our initial goal should be ensuring that you
can
> get
> > > MET
> > > > to
> > > > > > > read
> > > > > > > >>> > GRIB2 data. Please try simply plotting data from
that
> > GRIB2
> > > > file
> > > > > > by
> > > > > > > >>> > running:
> > > > > > > >>> > met-5.2/bin/plot_data_plane
> > LTIA98_KWBR_201305180600.grb2
> > > > > > > tmp_z2.ps
> > > > > > > >>> > 'name="TMP"; level="R2";
> > > > > > > >>> >
> > > > > > > >>> > Notice that I said "R2" which tells MET to plot
record
> > number
> > > > 2.
> > > > > > > >>> Record
> > > > > > > >>> > numbers 1 and 2 both contain temperature data and
> 2-meters.
> > > > > Here's
> > > > > > > >>> some
> > > > > > > >>> > wgrib2 output:
> > > > > > > >>> >
> > > > > > > >>> > 1:0:d=2013051806:TMP:2 m above
> > ground:anl:analysis/forecast
> > > > > error
> > > > > > > >>> > 2:3323062:d=2013051806 <(201)%20305-1806>
> > > > <(201)%20305-1806>:TMP:2 m above
> > > > > > ground:anl:
> > > > > > > >>> >
> > > > > > > >>> > So far, all the GRIB id info has been the same
between
> > > records
> > > > 1
> > > > > > and
> > > > > > > >>> 2...
> > > > > > > >>> > but presumably there's a difference somewhere.
> > > > > > > >>> >
> > > > > > > >>> > Please let me know how the plot_data_plane command
goes.
> > > > > > > >>> >
> > > > > > > >>> > Thanks,
> > > > > > > >>> > John Halley Gotway
> > > > > > > >>> >
> > > > > > > >>> >
> > > > > > > >>> >
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> --
> > > > > > > >>> Shannon Rees
> > > > > > > >>> Programming Scientist
> > > > > > > >>> Engility
> > > > > > > >>> Geophysical Fluid Dynamics Lab
> > > > > > > >>> 201 Forrestal Rd Princeton, NJ
> > > > > > > >>> 609-452-5384 <(609)%20452-5384> <(609)%20452-5384>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Shannon Rees
> > > > > > Programming Scientist
> > > > > > Engility
> > > > > > Geophysical Fluid Dynamics Lab
> > > > > > 201 Forrestal Rd Princeton, NJ
> > > > > > 609-452-5384 <(609)%20452-5384> <(609)%20452-5384>
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Shannon Rees
> > > > Programming Scientist
> > > > Engility
> > > > Geophysical Fluid Dynamics Lab
> > > > 201 Forrestal Rd Princeton, NJ
> > > > 609-452-5384 <(609)%20452-5384>
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > Shannon Rees
> > Programming Scientist
> > Engility
> > Geophysical Fluid Dynamics Lab
> > 201 Forrestal Rd Princeton, NJ
> > 609-452-5384
> >
> >
>
>
--
Shannon Rees
Programming Scientist
Engility
Geophysical Fluid Dynamics Lab
201 Forrestal Rd Princeton, NJ
609-452-5384
------------------------------------------------
Subject: Using Grib2 data with MET5.2 Grid-Stat
From: John Halley Gotway
Time: Thu Jan 26 10:35:09 2017
Shannon,
OK, I'll work on it today and will let you know when I have a patch.
I looked over Point-Stat as well and didn't see an obvious opportunity
to
do better memory allocation.
Thanks,
John
On Thu, Jan 26, 2017 at 10:17 AM, Shannon Rees - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
>
> Hi John,
>
> It would definitely be preferable to have the bugfix as soon as
possible.
> We will likely start the testing next week, since we have a lot of
work to
> do to be ready for the Spring Experiments. Let me know if an earlier
bugfix
> is possible. Do you know if this is only a problem with grid-stat
or does
> it affect other tools like point-stat?
>
> Thanks so much,
> Shannon
>
> On Thu, Jan 26, 2017 at 11:58 AM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Shannon,
> >
> > I have a question for you. The changes I'm making will definitely
be
> > included in the next version of MET, version 6.0 due out in March.
> >
> > Is that sufficient or do you really need a patch for version 5.2
for your
> > work? If so, I'll need to copy the updates into the release
branch and
> > post a bugfix.
> >
> > Thanks,
> > John
> >
> >
> >
> > On Thu, Jan 26, 2017 at 6:57 AM, Shannon Rees - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
> > >
> > > John,
> > >
> > > That is good to hear! We are planning on using the MET package
to help
> > us
> > > do a lot of model tuning for the Spring Experiments this year.
This
> will
> > > require even higher resolutions over CONUS and both continuous
and
> > > categorical statistics. We are planning on starting this
testing
> shortly
> > > and it will speed up the whole process if MET moves faster.
> > >
> > > Thanks,
> > > Shannon
> > >
> > > On Wed, Jan 25, 2017 at 6:16 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Shannon,
> > > >
> > > > I've been digging into this Grid-Stat timing issue.
> > > >
> > > > When I run Grid-Stat to compute continuous stats on your
sample case
> it
> > > > takes about 36 minutes. When I set the configuration to only
do
> > > > categorical counts, it takes about 5 seconds!
> > > >
> > > > That's a pretty big difference. Running it through the
profiler,
> I've
> > > > found that Grid-Stat is doing some pretty "dumb" memory
allocation.
> > And
> > > > those types of problems only become evident on large grids.
> > > >
> > > > The good news is that using the development version of Grid-
Stat the
> > I've
> > > > gotten the continuous statistics run time down to 19 seconds.
> > > >
> > > > Now I need to figure out which of the many little changes I
made are
> > the
> > > > ones necessary for the improvement.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Tue, Jan 24, 2017 at 9:28 AM, Shannon Rees - NOAA Affiliate
via
> RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
> > > > >
> > > > > Great, thanks for your help!
> > > > >
> > > > > On Tue, Jan 24, 2017 at 11:27 AM, John Halley Gotway via RT
<
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > > Shannon,
> > > > > >
> > > > > > I really wouldn't worry about the 64-bit flag setting.
I'm just
> > glad
> > > > you
> > > > > > were able to run successfully.
> > > > > >
> > > > > > I'll try to figure out what's taking so long in Grid-Stat
to see
> if
> > > > > there's
> > > > > > any way to speed it up.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Tue, Jan 24, 2017 at 7:54 AM, Shannon Rees - NOAA
Affiliate
> via
> > > RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242
> >
> > > > > > >
> > > > > > > Hi John,
> > > > > > >
> > > > > > > I recompiled both without 64 bit turned on and the
> > plot_data_plane
> > > > > > command
> > > > > > > worked. I plotted both R1 and R2 (attached). I am not
sure
> why
> > R1
> > > > is
> > > > > > > there or what it is representing. I ran grid-stat again
and it
> > > > > processed
> > > > > > > the files and the results look reasonable. It does take
a long
> > > time.
> > > > > I
> > > > > > > thought this was because of the amount of points, but if
you
> say
> > > that
> > > > > it
> > > > > > is
> > > > > > > still slower than usual that is promising. It would be
very
> > useful
> > > > if
> > > > > we
> > > > > > > could speed it up. I tried regridding to 13 km first,
and it
> ran
> > > > much
> > > > > > more
> > > > > > > quickly.
> > > > > > >
> > > > > > > My main concern now is how will running without 64 bit
affect
> > > things?
> > > > > > Will
> > > > > > > it hurt my ability to process very large files?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Shannon
> > > > > > >
> > > > > > > On Mon, Jan 23, 2017 at 6:48 PM, John Halley Gotway via
RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > > Hi Shannon,
> > > > > > > >
> > > > > > > > Just an update. Looking more closely at the GRIB2
file you
> > > sent, I
> > > > > > found
> > > > > > > > the difference between the two temperature records in
the
> > > "process"
> > > > > id:
> > > > > > > >
> > > > > > > > wgrib2 -process LTIA98_KWBR_201305180600.grb2
> > > > > > > > 1:0:code table 4.3=7 anl err
> > > > > > > > 2:3323062:code table 4.3=0 anl
> > > > > > > >
> > > > > > > > So in the development version of MET, I added support
for
> > > > specifying
> > > > > > > > "GRIB2_process" (as well as GRIB2_pdt, GRIB2_ens_type,
and
> > > > > > > > GRIB2_der_type). In version 6.0, you'll be able to
run this
> > > > command:
> > > > > > > >
> > > > > > > > plot_data_plane LTIA98_KWBR_201305180600.grb2
tmp_z2.ps
> > > > 'name="TMP";
> > > > > > > > level="Z2"; GRIB2_process=0;'
> > > > > > > >
> > > > > > > > The "GRIB2_process=0;" option says only match GRIB
records
> > whose
> > > > > > process
> > > > > > > > value is 0 (or change to 7 for the 1st record). In
MET
> version
> > > > 5.2,
> > > > > > > you're
> > > > > > > > stuck with specifying the exact record number to use:
> > > > > > > >
> > > > > > > > plot_data_plane LTIA98_KWBR_201305180600.grb2
tmp_z2.ps
> > > > 'name="TMP";
> > > > > > > > level="R2";'
> > > > > > > >
> > > > > > > > Still no word yet on why it's taking so long to run.
> > > > > > > >
> > > > > > > > John
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Jan 23, 2017 at 2:34 PM, John Halley Gotway <
> > > > johnhg at ucar.edu
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Also, when you recompile MET, please save the output
to a
> log
> > > > file:
> > > > > > > > > make install >& make_install.log
> > > > > > > > >
> > > > > > > > > If you continue to have problems, please send me
that
> > > > > > make_install.log
> > > > > > > > > file along with the config.log file from the top-
level MET
> > > > > directory.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > > On Mon, Jan 23, 2017 at 2:32 PM, John Halley Gotway
<
> > > > > johnhg at ucar.edu
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> OK, thanks for letting me know. Let's try to solve
this
> > > problem
> > > > > and
> > > > > > > get
> > > > > > > > >> plot_data_plane working on GRIB2 data.
> > > > > > > > >>
> > > > > > > > >> You mentioned that you'd compiled it using 64-bit
flags...
> > as
> > > > I'm
> > > > > > sure
> > > > > > > > >> you read from the MET online tutorial. Let's try
removing
> > > > those.
> > > > > > Are
> > > > > > > > you
> > > > > > > > >> able to recompile both the GRIB2C library and MET,
> removing
> > > > those
> > > > > 64
> > > > > > > bit
> > > > > > > > >> flags?
> > > > > > > > >>
> > > > > > > > >> And then try that plot_data_plane command again.
Please
> let
> > > me
> > > > > know
> > > > > > > how
> > > > > > > > >> it goes.
> > > > > > > > >>
> > > > > > > > >> Thanks,
> > > > > > > > >> John
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Mon, Jan 23, 2017 at 1:26 PM, Shannon Rees -
NOAA
> > Affiliate
> > > > via
> > > > > > RT
> > > > > > > <
> > > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > > >>
> > > > > > > > >>>
> > > > > > > > >>> <URL: https://rt.rap.ucar.edu/rt/
> > > Ticket/Display.html?id=79242
> > > > >
> > > > > > > > >>>
> > > > > > > > >>> Thanks John, I tried the plot_data_plane tool but
I got
> an
> > > > error:
> > > > > > > > >>>
> > > > > > > > >>> ~/met-5.2/met-5.2_bugfix/bin/plot_data_plane
> > > > > > > > >>> LTIA98_KWBR_201305180600.grb2
> > > > > > > > >>> tmp_z2.ps 'name="TMP"; level="R2" ; '
> > > > > > > > >>> DEBUG 1: Opening data file:
LTIA98_KWBR_201305180600.grb2
> > > > > > > > >>> terminate called after throwing an instance of
> > > 'std::bad_alloc'
> > > > > > > > >>> what(): std::bad_alloc
> > > > > > > > >>> Abort
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> On Mon, Jan 23, 2017 at 3:06 PM, John Halley
Gotway via
> RT
> > <
> > > > > > > > >>> met_help at ucar.edu> wrote:
> > > > > > > > >>>
> > > > > > > > >>> > Hi Shannon,
> > > > > > > > >>> >
> > > > > > > > >>> > Thanks for sending your sample data files. I
used your
> > > email
> > > > > to
> > > > > > > > >>> create a
> > > > > > > > >>> > met_help support ticket, which enables us to
track the
> > > > support
> > > > > we
> > > > > > > > >>> provide.
> > > > > > > > >>> >
> > > > > > > > >>> > I ran your data to compare the NetCDF forecast
file (
> > > fcst.nc
> > > > )
> > > > > to
> > > > > > > the
> > > > > > > > >>> > GRIB2 file (LTIA98_KWBR_201305180600.grb2) over
the
> FULL
> > > > model
> > > > > > > > domain.
> > > > > > > > >>> > While Grid-Stat *eventually* ran to completion,
it took
> > > over
> > > > 36
> > > > > > > > >>> minutes to
> > > > > > > > >>> > do so! I'd like to debug to figure out what's
taking
> so
> > > > long.
> > > > > > You
> > > > > > > > >>> already
> > > > > > > > >>> > had bootstrapping disabled (n_rep = 0) and rank
> > correlation
> > > > > > > > statistics
> > > > > > > > >>> > turned off (rank_corr_flag = FALSE)... which are
the
> > usual
> > > > > > > suspects.
> > > > > > > > >>> >
> > > > > > > > >>> > Granted, computing statistics over 1,774,484
matched
> > pairs
> > > > is a
> > > > > > lot
> > > > > > > > of
> > > > > > > > >>> > work... but 36 minutes is pretty ridiculous.
> > > > > > > > >>> >
> > > > > > > > >>> > I wonder if the issue is related to compiling
things
> with
> > > 64
> > > > > bit
> > > > > > > > >>> flags. I
> > > > > > > > >>> > think our initial goal should be ensuring that
you can
> > get
> > > > MET
> > > > > to
> > > > > > > > read
> > > > > > > > >>> > GRIB2 data. Please try simply plotting data
from that
> > > GRIB2
> > > > > file
> > > > > > > by
> > > > > > > > >>> > running:
> > > > > > > > >>> > met-5.2/bin/plot_data_plane
> > > LTIA98_KWBR_201305180600.grb2
> > > > > > > > tmp_z2.ps
> > > > > > > > >>> > 'name="TMP"; level="R2";
> > > > > > > > >>> >
> > > > > > > > >>> > Notice that I said "R2" which tells MET to plot
record
> > > number
> > > > > 2.
> > > > > > > > >>> Record
> > > > > > > > >>> > numbers 1 and 2 both contain temperature data
and
> > 2-meters.
> > > > > > Here's
> > > > > > > > >>> some
> > > > > > > > >>> > wgrib2 output:
> > > > > > > > >>> >
> > > > > > > > >>> > 1:0:d=2013051806:TMP:2 m above
> > > ground:anl:analysis/forecast
> > > > > > error
> > > > > > > > >>> > 2:3323062:d=2013051806 <(201)%20305-1806>
> > > > > <(201)%20305-1806>:TMP:2 m above
> > > > > > > ground:anl:
> > > > > > > > >>> >
> > > > > > > > >>> > So far, all the GRIB id info has been the same
between
> > > > records
> > > > > 1
> > > > > > > and
> > > > > > > > >>> 2...
> > > > > > > > >>> > but presumably there's a difference somewhere.
> > > > > > > > >>> >
> > > > > > > > >>> > Please let me know how the plot_data_plane
command
> goes.
> > > > > > > > >>> >
> > > > > > > > >>> > Thanks,
> > > > > > > > >>> > John Halley Gotway
> > > > > > > > >>> >
> > > > > > > > >>> >
> > > > > > > > >>> >
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> --
> > > > > > > > >>> Shannon Rees
> > > > > > > > >>> Programming Scientist
> > > > > > > > >>> Engility
> > > > > > > > >>> Geophysical Fluid Dynamics Lab
> > > > > > > > >>> 201 Forrestal Rd Princeton, NJ
> > > > > > > > >>> 609-452-5384 <(609)%20452-5384> <(609)%20452-5384>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Shannon Rees
> > > > > > > Programming Scientist
> > > > > > > Engility
> > > > > > > Geophysical Fluid Dynamics Lab
> > > > > > > 201 Forrestal Rd Princeton, NJ
> > > > > > > 609-452-5384 <(609)%20452-5384> <(609)%20452-5384>
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Shannon Rees
> > > > > Programming Scientist
> > > > > Engility
> > > > > Geophysical Fluid Dynamics Lab
> > > > > 201 Forrestal Rd Princeton, NJ
> > > > > 609-452-5384 <(609)%20452-5384>
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Shannon Rees
> > > Programming Scientist
> > > Engility
> > > Geophysical Fluid Dynamics Lab
> > > 201 Forrestal Rd Princeton, NJ
> > > 609-452-5384
> > >
> > >
> >
> >
>
>
> --
> Shannon Rees
> Programming Scientist
> Engility
> Geophysical Fluid Dynamics Lab
> 201 Forrestal Rd Princeton, NJ
> 609-452-5384
>
>
------------------------------------------------
Subject: Using Grib2 data with MET5.2 Grid-Stat
From: Shannon Rees - NOAA Affiliate
Time: Thu Jan 26 10:40:09 2017
OK, wonderful, thanks!
On Thu, Jan 26, 2017 at 12:35 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Shannon,
>
> OK, I'll work on it today and will let you know when I have a patch.
>
> I looked over Point-Stat as well and didn't see an obvious
opportunity to
> do better memory allocation.
>
> Thanks,
> John
>
> On Thu, Jan 26, 2017 at 10:17 AM, Shannon Rees - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
> >
> > Hi John,
> >
> > It would definitely be preferable to have the bugfix as soon as
possible.
> > We will likely start the testing next week, since we have a lot of
work
> to
> > do to be ready for the Spring Experiments. Let me know if an
earlier
> bugfix
> > is possible. Do you know if this is only a problem with grid-stat
or
> does
> > it affect other tools like point-stat?
> >
> > Thanks so much,
> > Shannon
> >
> > On Thu, Jan 26, 2017 at 11:58 AM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Shannon,
> > >
> > > I have a question for you. The changes I'm making will
definitely be
> > > included in the next version of MET, version 6.0 due out in
March.
> > >
> > > Is that sufficient or do you really need a patch for version 5.2
for
> your
> > > work? If so, I'll need to copy the updates into the release
branch and
> > > post a bugfix.
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > > On Thu, Jan 26, 2017 at 6:57 AM, Shannon Rees - NOAA Affiliate
via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242
>
> > > >
> > > > John,
> > > >
> > > > That is good to hear! We are planning on using the MET
package to
> help
> > > us
> > > > do a lot of model tuning for the Spring Experiments this year.
This
> > will
> > > > require even higher resolutions over CONUS and both continuous
and
> > > > categorical statistics. We are planning on starting this
testing
> > shortly
> > > > and it will speed up the whole process if MET moves faster.
> > > >
> > > > Thanks,
> > > > Shannon
> > > >
> > > > On Wed, Jan 25, 2017 at 6:16 PM, John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Shannon,
> > > > >
> > > > > I've been digging into this Grid-Stat timing issue.
> > > > >
> > > > > When I run Grid-Stat to compute continuous stats on your
sample
> case
> > it
> > > > > takes about 36 minutes. When I set the configuration to
only do
> > > > > categorical counts, it takes about 5 seconds!
> > > > >
> > > > > That's a pretty big difference. Running it through the
profiler,
> > I've
> > > > > found that Grid-Stat is doing some pretty "dumb" memory
allocation.
> > > And
> > > > > those types of problems only become evident on large grids.
> > > > >
> > > > > The good news is that using the development version of Grid-
Stat
> the
> > > I've
> > > > > gotten the continuous statistics run time down to 19
seconds.
> > > > >
> > > > > Now I need to figure out which of the many little changes I
made
> are
> > > the
> > > > > ones necessary for the improvement.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Tue, Jan 24, 2017 at 9:28 AM, Shannon Rees - NOAA
Affiliate via
> > RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
> > > > > >
> > > > > > Great, thanks for your help!
> > > > > >
> > > > > > On Tue, Jan 24, 2017 at 11:27 AM, John Halley Gotway via
RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > > Shannon,
> > > > > > >
> > > > > > > I really wouldn't worry about the 64-bit flag setting.
I'm
> just
> > > glad
> > > > > you
> > > > > > > were able to run successfully.
> > > > > > >
> > > > > > > I'll try to figure out what's taking so long in Grid-
Stat to
> see
> > if
> > > > > > there's
> > > > > > > any way to speed it up.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Tue, Jan 24, 2017 at 7:54 AM, Shannon Rees - NOAA
Affiliate
> > via
> > > > RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=79242
> > >
> > > > > > > >
> > > > > > > > Hi John,
> > > > > > > >
> > > > > > > > I recompiled both without 64 bit turned on and the
> > > plot_data_plane
> > > > > > > command
> > > > > > > > worked. I plotted both R1 and R2 (attached). I am
not sure
> > why
> > > R1
> > > > > is
> > > > > > > > there or what it is representing. I ran grid-stat
again and
> it
> > > > > > processed
> > > > > > > > the files and the results look reasonable. It does
take a
> long
> > > > time.
> > > > > > I
> > > > > > > > thought this was because of the amount of points, but
if you
> > say
> > > > that
> > > > > > it
> > > > > > > is
> > > > > > > > still slower than usual that is promising. It would
be very
> > > useful
> > > > > if
> > > > > > we
> > > > > > > > could speed it up. I tried regridding to 13 km first,
and it
> > ran
> > > > > much
> > > > > > > more
> > > > > > > > quickly.
> > > > > > > >
> > > > > > > > My main concern now is how will running without 64 bit
affect
> > > > things?
> > > > > > > Will
> > > > > > > > it hurt my ability to process very large files?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Shannon
> > > > > > > >
> > > > > > > > On Mon, Jan 23, 2017 at 6:48 PM, John Halley Gotway
via RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > > Hi Shannon,
> > > > > > > > >
> > > > > > > > > Just an update. Looking more closely at the GRIB2
file you
> > > > sent, I
> > > > > > > found
> > > > > > > > > the difference between the two temperature records
in the
> > > > "process"
> > > > > > id:
> > > > > > > > >
> > > > > > > > > wgrib2 -process LTIA98_KWBR_201305180600.grb2
> > > > > > > > > 1:0:code table 4.3=7 anl err
> > > > > > > > > 2:3323062:code table 4.3=0 anl
> > > > > > > > >
> > > > > > > > > So in the development version of MET, I added
support for
> > > > > specifying
> > > > > > > > > "GRIB2_process" (as well as GRIB2_pdt,
GRIB2_ens_type, and
> > > > > > > > > GRIB2_der_type). In version 6.0, you'll be able to
run
> this
> > > > > command:
> > > > > > > > >
> > > > > > > > > plot_data_plane LTIA98_KWBR_201305180600.grb2
tmp_z2.ps
> > > > > 'name="TMP";
> > > > > > > > > level="Z2"; GRIB2_process=0;'
> > > > > > > > >
> > > > > > > > > The "GRIB2_process=0;" option says only match GRIB
records
> > > whose
> > > > > > > process
> > > > > > > > > value is 0 (or change to 7 for the 1st record). In
MET
> > version
> > > > > 5.2,
> > > > > > > > you're
> > > > > > > > > stuck with specifying the exact record number to
use:
> > > > > > > > >
> > > > > > > > > plot_data_plane LTIA98_KWBR_201305180600.grb2
tmp_z2.ps
> > > > > 'name="TMP";
> > > > > > > > > level="R2";'
> > > > > > > > >
> > > > > > > > > Still no word yet on why it's taking so long to run.
> > > > > > > > >
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Jan 23, 2017 at 2:34 PM, John Halley Gotway
<
> > > > > johnhg at ucar.edu
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Also, when you recompile MET, please save the
output to a
> > log
> > > > > file:
> > > > > > > > > > make install >& make_install.log
> > > > > > > > > >
> > > > > > > > > > If you continue to have problems, please send me
that
> > > > > > > make_install.log
> > > > > > > > > > file along with the config.log file from the top-
level
> MET
> > > > > > directory.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > John
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 23, 2017 at 2:32 PM, John Halley
Gotway <
> > > > > > johnhg at ucar.edu
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> OK, thanks for letting me know. Let's try to
solve this
> > > > problem
> > > > > > and
> > > > > > > > get
> > > > > > > > > >> plot_data_plane working on GRIB2 data.
> > > > > > > > > >>
> > > > > > > > > >> You mentioned that you'd compiled it using 64-bit
> flags...
> > > as
> > > > > I'm
> > > > > > > sure
> > > > > > > > > >> you read from the MET online tutorial. Let's try
> removing
> > > > > those.
> > > > > > > Are
> > > > > > > > > you
> > > > > > > > > >> able to recompile both the GRIB2C library and
MET,
> > removing
> > > > > those
> > > > > > 64
> > > > > > > > bit
> > > > > > > > > >> flags?
> > > > > > > > > >>
> > > > > > > > > >> And then try that plot_data_plane command again.
Please
> > let
> > > > me
> > > > > > know
> > > > > > > > how
> > > > > > > > > >> it goes.
> > > > > > > > > >>
> > > > > > > > > >> Thanks,
> > > > > > > > > >> John
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> On Mon, Jan 23, 2017 at 1:26 PM, Shannon Rees -
NOAA
> > > Affiliate
> > > > > via
> > > > > > > RT
> > > > > > > > <
> > > > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > > > >>
> > > > > > > > > >>>
> > > > > > > > > >>> <URL: https://rt.rap.ucar.edu/rt/
> > > > Ticket/Display.html?id=79242
> > > > > >
> > > > > > > > > >>>
> > > > > > > > > >>> Thanks John, I tried the plot_data_plane tool
but I got
> > an
> > > > > error:
> > > > > > > > > >>>
> > > > > > > > > >>> ~/met-5.2/met-5.2_bugfix/bin/plot_data_plane
> > > > > > > > > >>> LTIA98_KWBR_201305180600.grb2
> > > > > > > > > >>> tmp_z2.ps 'name="TMP"; level="R2" ; '
> > > > > > > > > >>> DEBUG 1: Opening data file:
> LTIA98_KWBR_201305180600.grb2
> > > > > > > > > >>> terminate called after throwing an instance of
> > > > 'std::bad_alloc'
> > > > > > > > > >>> what(): std::bad_alloc
> > > > > > > > > >>> Abort
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> On Mon, Jan 23, 2017 at 3:06 PM, John Halley
Gotway via
> > RT
> > > <
> > > > > > > > > >>> met_help at ucar.edu> wrote:
> > > > > > > > > >>>
> > > > > > > > > >>> > Hi Shannon,
> > > > > > > > > >>> >
> > > > > > > > > >>> > Thanks for sending your sample data files. I
used
> your
> > > > email
> > > > > > to
> > > > > > > > > >>> create a
> > > > > > > > > >>> > met_help support ticket, which enables us to
track
> the
> > > > > support
> > > > > > we
> > > > > > > > > >>> provide.
> > > > > > > > > >>> >
> > > > > > > > > >>> > I ran your data to compare the NetCDF forecast
file (
> > > > fcst.nc
> > > > > )
> > > > > > to
> > > > > > > > the
> > > > > > > > > >>> > GRIB2 file (LTIA98_KWBR_201305180600.grb2)
over the
> > FULL
> > > > > model
> > > > > > > > > domain.
> > > > > > > > > >>> > While Grid-Stat *eventually* ran to
completion, it
> took
> > > > over
> > > > > 36
> > > > > > > > > >>> minutes to
> > > > > > > > > >>> > do so! I'd like to debug to figure out what's
taking
> > so
> > > > > long.
> > > > > > > You
> > > > > > > > > >>> already
> > > > > > > > > >>> > had bootstrapping disabled (n_rep = 0) and
rank
> > > correlation
> > > > > > > > > statistics
> > > > > > > > > >>> > turned off (rank_corr_flag = FALSE)... which
are the
> > > usual
> > > > > > > > suspects.
> > > > > > > > > >>> >
> > > > > > > > > >>> > Granted, computing statistics over 1,774,484
matched
> > > pairs
> > > > > is a
> > > > > > > lot
> > > > > > > > > of
> > > > > > > > > >>> > work... but 36 minutes is pretty ridiculous.
> > > > > > > > > >>> >
> > > > > > > > > >>> > I wonder if the issue is related to compiling
things
> > with
> > > > 64
> > > > > > bit
> > > > > > > > > >>> flags. I
> > > > > > > > > >>> > think our initial goal should be ensuring that
you
> can
> > > get
> > > > > MET
> > > > > > to
> > > > > > > > > read
> > > > > > > > > >>> > GRIB2 data. Please try simply plotting data
from
> that
> > > > GRIB2
> > > > > > file
> > > > > > > > by
> > > > > > > > > >>> > running:
> > > > > > > > > >>> > met-5.2/bin/plot_data_plane
> > > > LTIA98_KWBR_201305180600.grb2
> > > > > > > > > tmp_z2.ps
> > > > > > > > > >>> > 'name="TMP"; level="R2";
> > > > > > > > > >>> >
> > > > > > > > > >>> > Notice that I said "R2" which tells MET to
plot
> record
> > > > number
> > > > > > 2.
> > > > > > > > > >>> Record
> > > > > > > > > >>> > numbers 1 and 2 both contain temperature data
and
> > > 2-meters.
> > > > > > > Here's
> > > > > > > > > >>> some
> > > > > > > > > >>> > wgrib2 output:
> > > > > > > > > >>> >
> > > > > > > > > >>> > 1:0:d=2013051806:TMP:2 m above
> > > > ground:anl:analysis/forecast
> > > > > > > error
> > > > > > > > > >>> > 2:3323062:d=2013051806 <(201)%20305-1806>
> > > > > > <(201)%20305-1806>:TMP:2 m above
> > > > > > > > ground:anl:
> > > > > > > > > >>> >
> > > > > > > > > >>> > So far, all the GRIB id info has been the same
> between
> > > > > records
> > > > > > 1
> > > > > > > > and
> > > > > > > > > >>> 2...
> > > > > > > > > >>> > but presumably there's a difference somewhere.
> > > > > > > > > >>> >
> > > > > > > > > >>> > Please let me know how the plot_data_plane
command
> > goes.
> > > > > > > > > >>> >
> > > > > > > > > >>> > Thanks,
> > > > > > > > > >>> > John Halley Gotway
> > > > > > > > > >>> >
> > > > > > > > > >>> >
> > > > > > > > > >>> >
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> --
> > > > > > > > > >>> Shannon Rees
> > > > > > > > > >>> Programming Scientist
> > > > > > > > > >>> Engility
> > > > > > > > > >>> Geophysical Fluid Dynamics Lab
> > > > > > > > > >>> 201 Forrestal Rd Princeton, NJ
> > > > > > > > > >>> 609-452-5384 <(609)%20452-5384> <(609)%20452-
5384>
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Shannon Rees
> > > > > > > > Programming Scientist
> > > > > > > > Engility
> > > > > > > > Geophysical Fluid Dynamics Lab
> > > > > > > > 201 Forrestal Rd Princeton, NJ
> > > > > > > > 609-452-5384 <(609)%20452-5384> <(609)%20452-5384>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Shannon Rees
> > > > > > Programming Scientist
> > > > > > Engility
> > > > > > Geophysical Fluid Dynamics Lab
> > > > > > 201 Forrestal Rd Princeton, NJ
> > > > > > 609-452-5384 <(609)%20452-5384>
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Shannon Rees
> > > > Programming Scientist
> > > > Engility
> > > > Geophysical Fluid Dynamics Lab
> > > > 201 Forrestal Rd Princeton, NJ
> > > > 609-452-5384
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > Shannon Rees
> > Programming Scientist
> > Engility
> > Geophysical Fluid Dynamics Lab
> > 201 Forrestal Rd Princeton, NJ
> > 609-452-5384
> >
> >
>
>
--
Shannon Rees
Programming Scientist
Engility
Geophysical Fluid Dynamics Lab
201 Forrestal Rd Princeton, NJ
609-452-5384
------------------------------------------------
Subject: Using Grib2 data with MET5.2 Grid-Stat
From: John Halley Gotway
Time: Thu Jan 26 15:13:13 2017
Shannon,
OK, I just posted the relevant bugfix. You can find it here:
http://www.dtcenter.org/met/users/support/known_issues/METv5.2/index.php
It ended up being pretty small changes to many files. Please follow
the
instructions on that page, recompile met-5.2, and let me know how much
the
Grid-Stat run time improves for you test case.
Thanks,
John
On Thu, Jan 26, 2017 at 10:40 AM, Shannon Rees - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
>
> OK, wonderful, thanks!
>
> On Thu, Jan 26, 2017 at 12:35 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Shannon,
> >
> > OK, I'll work on it today and will let you know when I have a
patch.
> >
> > I looked over Point-Stat as well and didn't see an obvious
opportunity to
> > do better memory allocation.
> >
> > Thanks,
> > John
> >
> > On Thu, Jan 26, 2017 at 10:17 AM, Shannon Rees - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
> > >
> > > Hi John,
> > >
> > > It would definitely be preferable to have the bugfix as soon as
> possible.
> > > We will likely start the testing next week, since we have a lot
of work
> > to
> > > do to be ready for the Spring Experiments. Let me know if an
earlier
> > bugfix
> > > is possible. Do you know if this is only a problem with grid-
stat or
> > does
> > > it affect other tools like point-stat?
> > >
> > > Thanks so much,
> > > Shannon
> > >
> > > On Thu, Jan 26, 2017 at 11:58 AM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Shannon,
> > > >
> > > > I have a question for you. The changes I'm making will
definitely be
> > > > included in the next version of MET, version 6.0 due out in
March.
> > > >
> > > > Is that sufficient or do you really need a patch for version
5.2 for
> > your
> > > > work? If so, I'll need to copy the updates into the release
branch
> and
> > > > post a bugfix.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > >
> > > > On Thu, Jan 26, 2017 at 6:57 AM, Shannon Rees - NOAA Affiliate
via
> RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
> > > > >
> > > > > John,
> > > > >
> > > > > That is good to hear! We are planning on using the MET
package to
> > help
> > > > us
> > > > > do a lot of model tuning for the Spring Experiments this
year.
> This
> > > will
> > > > > require even higher resolutions over CONUS and both
continuous and
> > > > > categorical statistics. We are planning on starting this
testing
> > > shortly
> > > > > and it will speed up the whole process if MET moves faster.
> > > > >
> > > > > Thanks,
> > > > > Shannon
> > > > >
> > > > > On Wed, Jan 25, 2017 at 6:16 PM, John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > > Shannon,
> > > > > >
> > > > > > I've been digging into this Grid-Stat timing issue.
> > > > > >
> > > > > > When I run Grid-Stat to compute continuous stats on your
sample
> > case
> > > it
> > > > > > takes about 36 minutes. When I set the configuration to
only do
> > > > > > categorical counts, it takes about 5 seconds!
> > > > > >
> > > > > > That's a pretty big difference. Running it through the
profiler,
> > > I've
> > > > > > found that Grid-Stat is doing some pretty "dumb" memory
> allocation.
> > > > And
> > > > > > those types of problems only become evident on large
grids.
> > > > > >
> > > > > > The good news is that using the development version of
Grid-Stat
> > the
> > > > I've
> > > > > > gotten the continuous statistics run time down to 19
seconds.
> > > > > >
> > > > > > Now I need to figure out which of the many little changes
I made
> > are
> > > > the
> > > > > > ones necessary for the improvement.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Tue, Jan 24, 2017 at 9:28 AM, Shannon Rees - NOAA
Affiliate
> via
> > > RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242
> >
> > > > > > >
> > > > > > > Great, thanks for your help!
> > > > > > >
> > > > > > > On Tue, Jan 24, 2017 at 11:27 AM, John Halley Gotway via
RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > > Shannon,
> > > > > > > >
> > > > > > > > I really wouldn't worry about the 64-bit flag setting.
I'm
> > just
> > > > glad
> > > > > > you
> > > > > > > > were able to run successfully.
> > > > > > > >
> > > > > > > > I'll try to figure out what's taking so long in Grid-
Stat to
> > see
> > > if
> > > > > > > there's
> > > > > > > > any way to speed it up.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Tue, Jan 24, 2017 at 7:54 AM, Shannon Rees - NOAA
> Affiliate
> > > via
> > > > > RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > Ticket/Display.html?id=79242
> > > >
> > > > > > > > >
> > > > > > > > > Hi John,
> > > > > > > > >
> > > > > > > > > I recompiled both without 64 bit turned on and the
> > > > plot_data_plane
> > > > > > > > command
> > > > > > > > > worked. I plotted both R1 and R2 (attached). I am
not
> sure
> > > why
> > > > R1
> > > > > > is
> > > > > > > > > there or what it is representing. I ran grid-stat
again
> and
> > it
> > > > > > > processed
> > > > > > > > > the files and the results look reasonable. It does
take a
> > long
> > > > > time.
> > > > > > > I
> > > > > > > > > thought this was because of the amount of points,
but if
> you
> > > say
> > > > > that
> > > > > > > it
> > > > > > > > is
> > > > > > > > > still slower than usual that is promising. It would
be
> very
> > > > useful
> > > > > > if
> > > > > > > we
> > > > > > > > > could speed it up. I tried regridding to 13 km
first, and
> it
> > > ran
> > > > > > much
> > > > > > > > more
> > > > > > > > > quickly.
> > > > > > > > >
> > > > > > > > > My main concern now is how will running without 64
bit
> affect
> > > > > things?
> > > > > > > > Will
> > > > > > > > > it hurt my ability to process very large files?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Shannon
> > > > > > > > >
> > > > > > > > > On Mon, Jan 23, 2017 at 6:48 PM, John Halley Gotway
via RT
> <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > > Hi Shannon,
> > > > > > > > > >
> > > > > > > > > > Just an update. Looking more closely at the GRIB2
file
> you
> > > > > sent, I
> > > > > > > > found
> > > > > > > > > > the difference between the two temperature records
in the
> > > > > "process"
> > > > > > > id:
> > > > > > > > > >
> > > > > > > > > > wgrib2 -process LTIA98_KWBR_201305180600.grb2
> > > > > > > > > > 1:0:code table 4.3=7 anl err
> > > > > > > > > > 2:3323062:code table 4.3=0 anl
> > > > > > > > > >
> > > > > > > > > > So in the development version of MET, I added
support for
> > > > > > specifying
> > > > > > > > > > "GRIB2_process" (as well as GRIB2_pdt,
GRIB2_ens_type,
> and
> > > > > > > > > > GRIB2_der_type). In version 6.0, you'll be able
to run
> > this
> > > > > > command:
> > > > > > > > > >
> > > > > > > > > > plot_data_plane LTIA98_KWBR_201305180600.grb2
tmp_z2.ps
> > > > > > 'name="TMP";
> > > > > > > > > > level="Z2"; GRIB2_process=0;'
> > > > > > > > > >
> > > > > > > > > > The "GRIB2_process=0;" option says only match GRIB
> records
> > > > whose
> > > > > > > > process
> > > > > > > > > > value is 0 (or change to 7 for the 1st record).
In MET
> > > version
> > > > > > 5.2,
> > > > > > > > > you're
> > > > > > > > > > stuck with specifying the exact record number to
use:
> > > > > > > > > >
> > > > > > > > > > plot_data_plane LTIA98_KWBR_201305180600.grb2
tmp_z2.ps
> > > > > > 'name="TMP";
> > > > > > > > > > level="R2";'
> > > > > > > > > >
> > > > > > > > > > Still no word yet on why it's taking so long to
run.
> > > > > > > > > >
> > > > > > > > > > John
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 23, 2017 at 2:34 PM, John Halley
Gotway <
> > > > > > johnhg at ucar.edu
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Also, when you recompile MET, please save the
output
> to a
> > > log
> > > > > > file:
> > > > > > > > > > > make install >& make_install.log
> > > > > > > > > > >
> > > > > > > > > > > If you continue to have problems, please send me
that
> > > > > > > > make_install.log
> > > > > > > > > > > file along with the config.log file from the
top-level
> > MET
> > > > > > > directory.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > John
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jan 23, 2017 at 2:32 PM, John Halley
Gotway <
> > > > > > > johnhg at ucar.edu
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> OK, thanks for letting me know. Let's try to
solve
> this
> > > > > problem
> > > > > > > and
> > > > > > > > > get
> > > > > > > > > > >> plot_data_plane working on GRIB2 data.
> > > > > > > > > > >>
> > > > > > > > > > >> You mentioned that you'd compiled it using 64-
bit
> > flags...
> > > > as
> > > > > > I'm
> > > > > > > > sure
> > > > > > > > > > >> you read from the MET online tutorial. Let's
try
> > removing
> > > > > > those.
> > > > > > > > Are
> > > > > > > > > > you
> > > > > > > > > > >> able to recompile both the GRIB2C library and
MET,
> > > removing
> > > > > > those
> > > > > > > 64
> > > > > > > > > bit
> > > > > > > > > > >> flags?
> > > > > > > > > > >>
> > > > > > > > > > >> And then try that plot_data_plane command
again.
> Please
> > > let
> > > > > me
> > > > > > > know
> > > > > > > > > how
> > > > > > > > > > >> it goes.
> > > > > > > > > > >>
> > > > > > > > > > >> Thanks,
> > > > > > > > > > >> John
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> On Mon, Jan 23, 2017 at 1:26 PM, Shannon Rees -
NOAA
> > > > Affiliate
> > > > > > via
> > > > > > > > RT
> > > > > > > > > <
> > > > > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >>>
> > > > > > > > > > >>> <URL: https://rt.rap.ucar.edu/rt/
> > > > > Ticket/Display.html?id=79242
> > > > > > >
> > > > > > > > > > >>>
> > > > > > > > > > >>> Thanks John, I tried the plot_data_plane tool
but I
> got
> > > an
> > > > > > error:
> > > > > > > > > > >>>
> > > > > > > > > > >>> ~/met-5.2/met-5.2_bugfix/bin/plot_data_plane
> > > > > > > > > > >>> LTIA98_KWBR_201305180600.grb2
> > > > > > > > > > >>> tmp_z2.ps 'name="TMP"; level="R2" ; '
> > > > > > > > > > >>> DEBUG 1: Opening data file:
> > LTIA98_KWBR_201305180600.grb2
> > > > > > > > > > >>> terminate called after throwing an instance of
> > > > > 'std::bad_alloc'
> > > > > > > > > > >>> what(): std::bad_alloc
> > > > > > > > > > >>> Abort
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>> On Mon, Jan 23, 2017 at 3:06 PM, John Halley
Gotway
> via
> > > RT
> > > > <
> > > > > > > > > > >>> met_help at ucar.edu> wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>> > Hi Shannon,
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > Thanks for sending your sample data files.
I used
> > your
> > > > > email
> > > > > > > to
> > > > > > > > > > >>> create a
> > > > > > > > > > >>> > met_help support ticket, which enables us to
track
> > the
> > > > > > support
> > > > > > > we
> > > > > > > > > > >>> provide.
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > I ran your data to compare the NetCDF
forecast
> file (
> > > > > fcst.nc
> > > > > > )
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > >>> > GRIB2 file (LTIA98_KWBR_201305180600.grb2)
over
> the
> > > FULL
> > > > > > model
> > > > > > > > > > domain.
> > > > > > > > > > >>> > While Grid-Stat *eventually* ran to
completion, it
> > took
> > > > > over
> > > > > > 36
> > > > > > > > > > >>> minutes to
> > > > > > > > > > >>> > do so! I'd like to debug to figure out
what's
> taking
> > > so
> > > > > > long.
> > > > > > > > You
> > > > > > > > > > >>> already
> > > > > > > > > > >>> > had bootstrapping disabled (n_rep = 0) and
rank
> > > > correlation
> > > > > > > > > > statistics
> > > > > > > > > > >>> > turned off (rank_corr_flag = FALSE)... which
are
> the
> > > > usual
> > > > > > > > > suspects.
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > Granted, computing statistics over 1,774,484
> matched
> > > > pairs
> > > > > > is a
> > > > > > > > lot
> > > > > > > > > > of
> > > > > > > > > > >>> > work... but 36 minutes is pretty ridiculous.
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > I wonder if the issue is related to
compiling
> things
> > > with
> > > > > 64
> > > > > > > bit
> > > > > > > > > > >>> flags. I
> > > > > > > > > > >>> > think our initial goal should be ensuring
that you
> > can
> > > > get
> > > > > > MET
> > > > > > > to
> > > > > > > > > > read
> > > > > > > > > > >>> > GRIB2 data. Please try simply plotting data
from
> > that
> > > > > GRIB2
> > > > > > > file
> > > > > > > > > by
> > > > > > > > > > >>> > running:
> > > > > > > > > > >>> > met-5.2/bin/plot_data_plane
> > > > > LTIA98_KWBR_201305180600.grb2
> > > > > > > > > > tmp_z2.ps
> > > > > > > > > > >>> > 'name="TMP"; level="R2";
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > Notice that I said "R2" which tells MET to
plot
> > record
> > > > > number
> > > > > > > 2.
> > > > > > > > > > >>> Record
> > > > > > > > > > >>> > numbers 1 and 2 both contain temperature
data and
> > > > 2-meters.
> > > > > > > > Here's
> > > > > > > > > > >>> some
> > > > > > > > > > >>> > wgrib2 output:
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > 1:0:d=2013051806:TMP:2 m above
> > > > > ground:anl:analysis/forecast
> > > > > > > > error
> > > > > > > > > > >>> > 2:3323062:d=2013051806 <(201)%20305-1806>
> > > > > > > <(201)%20305-1806>:TMP:2 m above
> > > > > > > > > ground:anl:
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > So far, all the GRIB id info has been the
same
> > between
> > > > > > records
> > > > > > > 1
> > > > > > > > > and
> > > > > > > > > > >>> 2...
> > > > > > > > > > >>> > but presumably there's a difference
somewhere.
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > Please let me know how the plot_data_plane
command
> > > goes.
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > Thanks,
> > > > > > > > > > >>> > John Halley Gotway
> > > > > > > > > > >>> >
> > > > > > > > > > >>> >
> > > > > > > > > > >>> >
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>> --
> > > > > > > > > > >>> Shannon Rees
> > > > > > > > > > >>> Programming Scientist
> > > > > > > > > > >>> Engility
> > > > > > > > > > >>> Geophysical Fluid Dynamics Lab
> > > > > > > > > > >>> 201 Forrestal Rd Princeton, NJ
> > > > > > > > > > >>> 609-452-5384 <(609)%20452-5384> <(609)%20452-
5384>
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Shannon Rees
> > > > > > > > > Programming Scientist
> > > > > > > > > Engility
> > > > > > > > > Geophysical Fluid Dynamics Lab
> > > > > > > > > 201 Forrestal Rd Princeton, NJ
> > > > > > > > > 609-452-5384 <(609)%20452-5384> <(609)%20452-5384>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Shannon Rees
> > > > > > > Programming Scientist
> > > > > > > Engility
> > > > > > > Geophysical Fluid Dynamics Lab
> > > > > > > 201 Forrestal Rd Princeton, NJ
> > > > > > > 609-452-5384 <(609)%20452-5384>
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Shannon Rees
> > > > > Programming Scientist
> > > > > Engility
> > > > > Geophysical Fluid Dynamics Lab
> > > > > 201 Forrestal Rd Princeton, NJ
> > > > > 609-452-5384
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Shannon Rees
> > > Programming Scientist
> > > Engility
> > > Geophysical Fluid Dynamics Lab
> > > 201 Forrestal Rd Princeton, NJ
> > > 609-452-5384
> > >
> > >
> >
> >
>
>
> --
> Shannon Rees
> Programming Scientist
> Engility
> Geophysical Fluid Dynamics Lab
> 201 Forrestal Rd Princeton, NJ
> 609-452-5384
>
>
------------------------------------------------
Subject: Using Grib2 data with MET5.2 Grid-Stat
From: Shannon Rees - NOAA Affiliate
Time: Fri Jan 27 08:50:42 2017
Hi John,
This worked out well. My grid-stat test time was cut down from 13
minutes
to 54 seconds, so big improvement. My only issue was with compiling
the
bugfix at first since it doesn't contain the met-5.2 subdirectory. I
just
had to move a few things around.
Thanks for the quick work!
Shannon
On Thu, Jan 26, 2017 at 5:13 PM, John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Shannon,
>
> OK, I just posted the relevant bugfix. You can find it here:
> http://www.dtcenter.org/met/users/support/known_issues/
> METv5.2/index.php
>
> It ended up being pretty small changes to many files. Please follow
the
> instructions on that page, recompile met-5.2, and let me know how
much the
> Grid-Stat run time improves for you test case.
>
> Thanks,
> John
>
> On Thu, Jan 26, 2017 at 10:40 AM, Shannon Rees - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
> >
> > OK, wonderful, thanks!
> >
> > On Thu, Jan 26, 2017 at 12:35 PM, John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Shannon,
> > >
> > > OK, I'll work on it today and will let you know when I have a
patch.
> > >
> > > I looked over Point-Stat as well and didn't see an obvious
opportunity
> to
> > > do better memory allocation.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Jan 26, 2017 at 10:17 AM, Shannon Rees - NOAA Affiliate
via RT
> <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242
>
> > > >
> > > > Hi John,
> > > >
> > > > It would definitely be preferable to have the bugfix as soon
as
> > possible.
> > > > We will likely start the testing next week, since we have a
lot of
> work
> > > to
> > > > do to be ready for the Spring Experiments. Let me know if an
earlier
> > > bugfix
> > > > is possible. Do you know if this is only a problem with grid-
stat or
> > > does
> > > > it affect other tools like point-stat?
> > > >
> > > > Thanks so much,
> > > > Shannon
> > > >
> > > > On Thu, Jan 26, 2017 at 11:58 AM, John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Shannon,
> > > > >
> > > > > I have a question for you. The changes I'm making will
definitely
> be
> > > > > included in the next version of MET, version 6.0 due out in
March.
> > > > >
> > > > > Is that sufficient or do you really need a patch for version
5.2
> for
> > > your
> > > > > work? If so, I'll need to copy the updates into the release
branch
> > and
> > > > > post a bugfix.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jan 26, 2017 at 6:57 AM, Shannon Rees - NOAA
Affiliate via
> > RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
> > > > > >
> > > > > > John,
> > > > > >
> > > > > > That is good to hear! We are planning on using the MET
package
> to
> > > help
> > > > > us
> > > > > > do a lot of model tuning for the Spring Experiments this
year.
> > This
> > > > will
> > > > > > require even higher resolutions over CONUS and both
continuous
> and
> > > > > > categorical statistics. We are planning on starting this
testing
> > > > shortly
> > > > > > and it will speed up the whole process if MET moves
faster.
> > > > > >
> > > > > > Thanks,
> > > > > > Shannon
> > > > > >
> > > > > > On Wed, Jan 25, 2017 at 6:16 PM, John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > > Shannon,
> > > > > > >
> > > > > > > I've been digging into this Grid-Stat timing issue.
> > > > > > >
> > > > > > > When I run Grid-Stat to compute continuous stats on your
sample
> > > case
> > > > it
> > > > > > > takes about 36 minutes. When I set the configuration to
only
> do
> > > > > > > categorical counts, it takes about 5 seconds!
> > > > > > >
> > > > > > > That's a pretty big difference. Running it through the
> profiler,
> > > > I've
> > > > > > > found that Grid-Stat is doing some pretty "dumb" memory
> > allocation.
> > > > > And
> > > > > > > those types of problems only become evident on large
grids.
> > > > > > >
> > > > > > > The good news is that using the development version of
> Grid-Stat
> > > the
> > > > > I've
> > > > > > > gotten the continuous statistics run time down to 19
seconds.
> > > > > > >
> > > > > > > Now I need to figure out which of the many little
changes I
> made
> > > are
> > > > > the
> > > > > > > ones necessary for the improvement.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Tue, Jan 24, 2017 at 9:28 AM, Shannon Rees - NOAA
Affiliate
> > via
> > > > RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> Ticket/Display.html?id=79242
> > >
> > > > > > > >
> > > > > > > > Great, thanks for your help!
> > > > > > > >
> > > > > > > > On Tue, Jan 24, 2017 at 11:27 AM, John Halley Gotway
via RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > > Shannon,
> > > > > > > > >
> > > > > > > > > I really wouldn't worry about the 64-bit flag
setting. I'm
> > > just
> > > > > glad
> > > > > > > you
> > > > > > > > > were able to run successfully.
> > > > > > > > >
> > > > > > > > > I'll try to figure out what's taking so long in
Grid-Stat
> to
> > > see
> > > > if
> > > > > > > > there's
> > > > > > > > > any way to speed it up.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > > On Tue, Jan 24, 2017 at 7:54 AM, Shannon Rees - NOAA
> > Affiliate
> > > > via
> > > > > > RT <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > > Ticket/Display.html?id=79242
> > > > >
> > > > > > > > > >
> > > > > > > > > > Hi John,
> > > > > > > > > >
> > > > > > > > > > I recompiled both without 64 bit turned on and the
> > > > > plot_data_plane
> > > > > > > > > command
> > > > > > > > > > worked. I plotted both R1 and R2 (attached). I
am not
> > sure
> > > > why
> > > > > R1
> > > > > > > is
> > > > > > > > > > there or what it is representing. I ran grid-stat
again
> > and
> > > it
> > > > > > > > processed
> > > > > > > > > > the files and the results look reasonable. It
does take
> a
> > > long
> > > > > > time.
> > > > > > > > I
> > > > > > > > > > thought this was because of the amount of points,
but if
> > you
> > > > say
> > > > > > that
> > > > > > > > it
> > > > > > > > > is
> > > > > > > > > > still slower than usual that is promising. It
would be
> > very
> > > > > useful
> > > > > > > if
> > > > > > > > we
> > > > > > > > > > could speed it up. I tried regridding to 13 km
first,
> and
> > it
> > > > ran
> > > > > > > much
> > > > > > > > > more
> > > > > > > > > > quickly.
> > > > > > > > > >
> > > > > > > > > > My main concern now is how will running without 64
bit
> > affect
> > > > > > things?
> > > > > > > > > Will
> > > > > > > > > > it hurt my ability to process very large files?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Shannon
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 23, 2017 at 6:48 PM, John Halley
Gotway via
> RT
> > <
> > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Shannon,
> > > > > > > > > > >
> > > > > > > > > > > Just an update. Looking more closely at the
GRIB2 file
> > you
> > > > > > sent, I
> > > > > > > > > found
> > > > > > > > > > > the difference between the two temperature
records in
> the
> > > > > > "process"
> > > > > > > > id:
> > > > > > > > > > >
> > > > > > > > > > > wgrib2 -process LTIA98_KWBR_201305180600.grb2
> > > > > > > > > > > 1:0:code table 4.3=7 anl err
> > > > > > > > > > > 2:3323062:code table 4.3=0 anl
> > > > > > > > > > >
> > > > > > > > > > > So in the development version of MET, I added
support
> for
> > > > > > > specifying
> > > > > > > > > > > "GRIB2_process" (as well as GRIB2_pdt,
GRIB2_ens_type,
> > and
> > > > > > > > > > > GRIB2_der_type). In version 6.0, you'll be able
to run
> > > this
> > > > > > > command:
> > > > > > > > > > >
> > > > > > > > > > > plot_data_plane LTIA98_KWBR_201305180600.grb2
> tmp_z2.ps
> > > > > > > 'name="TMP";
> > > > > > > > > > > level="Z2"; GRIB2_process=0;'
> > > > > > > > > > >
> > > > > > > > > > > The "GRIB2_process=0;" option says only match
GRIB
> > records
> > > > > whose
> > > > > > > > > process
> > > > > > > > > > > value is 0 (or change to 7 for the 1st record).
In MET
> > > > version
> > > > > > > 5.2,
> > > > > > > > > > you're
> > > > > > > > > > > stuck with specifying the exact record number to
use:
> > > > > > > > > > >
> > > > > > > > > > > plot_data_plane LTIA98_KWBR_201305180600.grb2
> tmp_z2.ps
> > > > > > > 'name="TMP";
> > > > > > > > > > > level="R2";'
> > > > > > > > > > >
> > > > > > > > > > > Still no word yet on why it's taking so long to
run.
> > > > > > > > > > >
> > > > > > > > > > > John
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jan 23, 2017 at 2:34 PM, John Halley
Gotway <
> > > > > > > johnhg at ucar.edu
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Also, when you recompile MET, please save the
output
> > to a
> > > > log
> > > > > > > file:
> > > > > > > > > > > > make install >& make_install.log
> > > > > > > > > > > >
> > > > > > > > > > > > If you continue to have problems, please send
me that
> > > > > > > > > make_install.log
> > > > > > > > > > > > file along with the config.log file from the
> top-level
> > > MET
> > > > > > > > directory.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > John
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Jan 23, 2017 at 2:32 PM, John Halley
Gotway <
> > > > > > > > johnhg at ucar.edu
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> OK, thanks for letting me know. Let's try to
solve
> > this
> > > > > > problem
> > > > > > > > and
> > > > > > > > > > get
> > > > > > > > > > > >> plot_data_plane working on GRIB2 data.
> > > > > > > > > > > >>
> > > > > > > > > > > >> You mentioned that you'd compiled it using
64-bit
> > > flags...
> > > > > as
> > > > > > > I'm
> > > > > > > > > sure
> > > > > > > > > > > >> you read from the MET online tutorial. Let's
try
> > > removing
> > > > > > > those.
> > > > > > > > > Are
> > > > > > > > > > > you
> > > > > > > > > > > >> able to recompile both the GRIB2C library and
MET,
> > > > removing
> > > > > > > those
> > > > > > > > 64
> > > > > > > > > > bit
> > > > > > > > > > > >> flags?
> > > > > > > > > > > >>
> > > > > > > > > > > >> And then try that plot_data_plane command
again.
> > Please
> > > > let
> > > > > > me
> > > > > > > > know
> > > > > > > > > > how
> > > > > > > > > > > >> it goes.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Thanks,
> > > > > > > > > > > >> John
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Mon, Jan 23, 2017 at 1:26 PM, Shannon Rees
- NOAA
> > > > > Affiliate
> > > > > > > via
> > > > > > > > > RT
> > > > > > > > > > <
> > > > > > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> <URL: https://rt.rap.ucar.edu/rt/
> > > > > > Ticket/Display.html?id=79242
> > > > > > > >
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> Thanks John, I tried the plot_data_plane
tool but I
> > got
> > > > an
> > > > > > > error:
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> ~/met-5.2/met-5.2_bugfix/bin/plot_data_plane
> > > > > > > > > > > >>> LTIA98_KWBR_201305180600.grb2
> > > > > > > > > > > >>> tmp_z2.ps 'name="TMP"; level="R2" ; '
> > > > > > > > > > > >>> DEBUG 1: Opening data file:
> > > LTIA98_KWBR_201305180600.grb2
> > > > > > > > > > > >>> terminate called after throwing an instance
of
> > > > > > 'std::bad_alloc'
> > > > > > > > > > > >>> what(): std::bad_alloc
> > > > > > > > > > > >>> Abort
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> On Mon, Jan 23, 2017 at 3:06 PM, John Halley
Gotway
> > via
> > > > RT
> > > > > <
> > > > > > > > > > > >>> met_help at ucar.edu> wrote:
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> > Hi Shannon,
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > Thanks for sending your sample data files.
I
> used
> > > your
> > > > > > email
> > > > > > > > to
> > > > > > > > > > > >>> create a
> > > > > > > > > > > >>> > met_help support ticket, which enables us
to
> track
> > > the
> > > > > > > support
> > > > > > > > we
> > > > > > > > > > > >>> provide.
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > I ran your data to compare the NetCDF
forecast
> > file (
> > > > > > fcst.nc
> > > > > > > )
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > >>> > GRIB2 file (LTIA98_KWBR_201305180600.grb2)
over
> > the
> > > > FULL
> > > > > > > model
> > > > > > > > > > > domain.
> > > > > > > > > > > >>> > While Grid-Stat *eventually* ran to
completion,
> it
> > > took
> > > > > > over
> > > > > > > 36
> > > > > > > > > > > >>> minutes to
> > > > > > > > > > > >>> > do so! I'd like to debug to figure out
what's
> > taking
> > > > so
> > > > > > > long.
> > > > > > > > > You
> > > > > > > > > > > >>> already
> > > > > > > > > > > >>> > had bootstrapping disabled (n_rep = 0) and
rank
> > > > > correlation
> > > > > > > > > > > statistics
> > > > > > > > > > > >>> > turned off (rank_corr_flag = FALSE)...
which are
> > the
> > > > > usual
> > > > > > > > > > suspects.
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > Granted, computing statistics over
1,774,484
> > matched
> > > > > pairs
> > > > > > > is a
> > > > > > > > > lot
> > > > > > > > > > > of
> > > > > > > > > > > >>> > work... but 36 minutes is pretty
ridiculous.
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > I wonder if the issue is related to
compiling
> > things
> > > > with
> > > > > > 64
> > > > > > > > bit
> > > > > > > > > > > >>> flags. I
> > > > > > > > > > > >>> > think our initial goal should be ensuring
that
> you
> > > can
> > > > > get
> > > > > > > MET
> > > > > > > > to
> > > > > > > > > > > read
> > > > > > > > > > > >>> > GRIB2 data. Please try simply plotting
data from
> > > that
> > > > > > GRIB2
> > > > > > > > file
> > > > > > > > > > by
> > > > > > > > > > > >>> > running:
> > > > > > > > > > > >>> > met-5.2/bin/plot_data_plane
> > > > > > LTIA98_KWBR_201305180600.grb2
> > > > > > > > > > > tmp_z2.ps
> > > > > > > > > > > >>> > 'name="TMP"; level="R2";
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > Notice that I said "R2" which tells MET to
plot
> > > record
> > > > > > number
> > > > > > > > 2.
> > > > > > > > > > > >>> Record
> > > > > > > > > > > >>> > numbers 1 and 2 both contain temperature
data and
> > > > > 2-meters.
> > > > > > > > > Here's
> > > > > > > > > > > >>> some
> > > > > > > > > > > >>> > wgrib2 output:
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > 1:0:d=2013051806:TMP:2 m above
> > > > > > ground:anl:analysis/forecast
> > > > > > > > > error
> > > > > > > > > > > >>> > 2:3323062:d=2013051806 <(201)%20305-1806>
> > > > > > > > <(201)%20305-1806>:TMP:2 m above
> > > > > > > > > > ground:anl:
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > So far, all the GRIB id info has been the
same
> > > between
> > > > > > > records
> > > > > > > > 1
> > > > > > > > > > and
> > > > > > > > > > > >>> 2...
> > > > > > > > > > > >>> > but presumably there's a difference
somewhere.
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > Please let me know how the plot_data_plane
> command
> > > > goes.
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> > Thanks,
> > > > > > > > > > > >>> > John Halley Gotway
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>> >
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> --
> > > > > > > > > > > >>> Shannon Rees
> > > > > > > > > > > >>> Programming Scientist
> > > > > > > > > > > >>> Engility
> > > > > > > > > > > >>> Geophysical Fluid Dynamics Lab
> > > > > > > > > > > >>> 201 Forrestal Rd Princeton, NJ
> > > > > > > > > > > >>> 609-452-5384 <(609)%20452-5384>
<(609)%20452-5384>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Shannon Rees
> > > > > > > > > > Programming Scientist
> > > > > > > > > > Engility
> > > > > > > > > > Geophysical Fluid Dynamics Lab
> > > > > > > > > > 201 Forrestal Rd Princeton, NJ
> > > > > > > > > > 609-452-5384 <(609)%20452-5384> <(609)%20452-5384>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Shannon Rees
> > > > > > > > Programming Scientist
> > > > > > > > Engility
> > > > > > > > Geophysical Fluid Dynamics Lab
> > > > > > > > 201 Forrestal Rd Princeton, NJ
> > > > > > > > 609-452-5384 <(609)%20452-5384>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Shannon Rees
> > > > > > Programming Scientist
> > > > > > Engility
> > > > > > Geophysical Fluid Dynamics Lab
> > > > > > 201 Forrestal Rd Princeton, NJ
> > > > > > 609-452-5384
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Shannon Rees
> > > > Programming Scientist
> > > > Engility
> > > > Geophysical Fluid Dynamics Lab
> > > > 201 Forrestal Rd Princeton, NJ
> > > > 609-452-5384
> > > >
> > > >
> > >
> > >
> >
> >
> > --
> > Shannon Rees
> > Programming Scientist
> > Engility
> > Geophysical Fluid Dynamics Lab
> > 201 Forrestal Rd Princeton, NJ
> > 609-452-5384
> >
> >
>
>
--
Shannon Rees
Programming Scientist
Engility
Geophysical Fluid Dynamics Lab
201 Forrestal Rd Princeton, NJ
609-452-5384
------------------------------------------------
Subject: Using Grib2 data with MET5.2 Grid-Stat
From: John Halley Gotway
Time: Fri Jan 27 09:23:55 2017
Shannon,
Thanks for following up and letting me know. As you apply MET to even
higher resolution datasets, if you run into suspicious performance
issues,
please let us know. Once we're able to replicate the behavior,
profiling
the code to identify the bottlenecks really isn't that difficult.
And these updates for Grid-Stat will benefit at least two other
groups. So
please keep the feedback coming.
I'll go ahead and resolve this ticket.
Thanks,
John
On Fri, Jan 27, 2017 at 8:50 AM, Shannon Rees - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
>
> Hi John,
>
> This worked out well. My grid-stat test time was cut down from 13
minutes
> to 54 seconds, so big improvement. My only issue was with compiling
the
> bugfix at first since it doesn't contain the met-5.2 subdirectory.
I just
> had to move a few things around.
>
> Thanks for the quick work!
> Shannon
>
> On Thu, Jan 26, 2017 at 5:13 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Shannon,
> >
> > OK, I just posted the relevant bugfix. You can find it here:
> > http://www.dtcenter.org/met/users/support/known_issues/
> > METv5.2/index.php
> >
> > It ended up being pretty small changes to many files. Please
follow the
> > instructions on that page, recompile met-5.2, and let me know how
much
> the
> > Grid-Stat run time improves for you test case.
> >
> > Thanks,
> > John
> >
> > On Thu, Jan 26, 2017 at 10:40 AM, Shannon Rees - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
> > >
> > > OK, wonderful, thanks!
> > >
> > > On Thu, Jan 26, 2017 at 12:35 PM, John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Shannon,
> > > >
> > > > OK, I'll work on it today and will let you know when I have a
patch.
> > > >
> > > > I looked over Point-Stat as well and didn't see an obvious
> opportunity
> > to
> > > > do better memory allocation.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Thu, Jan 26, 2017 at 10:17 AM, Shannon Rees - NOAA
Affiliate via
> RT
> > <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242 >
> > > > >
> > > > > Hi John,
> > > > >
> > > > > It would definitely be preferable to have the bugfix as soon
as
> > > possible.
> > > > > We will likely start the testing next week, since we have a
lot of
> > work
> > > > to
> > > > > do to be ready for the Spring Experiments. Let me know if an
> earlier
> > > > bugfix
> > > > > is possible. Do you know if this is only a problem with
grid-stat
> or
> > > > does
> > > > > it affect other tools like point-stat?
> > > > >
> > > > > Thanks so much,
> > > > > Shannon
> > > > >
> > > > > On Thu, Jan 26, 2017 at 11:58 AM, John Halley Gotway via RT
<
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > > Shannon,
> > > > > >
> > > > > > I have a question for you. The changes I'm making will
> definitely
> > be
> > > > > > included in the next version of MET, version 6.0 due out
in
> March.
> > > > > >
> > > > > > Is that sufficient or do you really need a patch for
version 5.2
> > for
> > > > your
> > > > > > work? If so, I'll need to copy the updates into the
release
> branch
> > > and
> > > > > > post a bugfix.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Jan 26, 2017 at 6:57 AM, Shannon Rees - NOAA
Affiliate
> via
> > > RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=79242
> >
> > > > > > >
> > > > > > > John,
> > > > > > >
> > > > > > > That is good to hear! We are planning on using the MET
package
> > to
> > > > help
> > > > > > us
> > > > > > > do a lot of model tuning for the Spring Experiments this
year.
> > > This
> > > > > will
> > > > > > > require even higher resolutions over CONUS and both
continuous
> > and
> > > > > > > categorical statistics. We are planning on starting
this
> testing
> > > > > shortly
> > > > > > > and it will speed up the whole process if MET moves
faster.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Shannon
> > > > > > >
> > > > > > > On Wed, Jan 25, 2017 at 6:16 PM, John Halley Gotway via
RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > > Shannon,
> > > > > > > >
> > > > > > > > I've been digging into this Grid-Stat timing issue.
> > > > > > > >
> > > > > > > > When I run Grid-Stat to compute continuous stats on
your
> sample
> > > > case
> > > > > it
> > > > > > > > takes about 36 minutes. When I set the configuration
to only
> > do
> > > > > > > > categorical counts, it takes about 5 seconds!
> > > > > > > >
> > > > > > > > That's a pretty big difference. Running it through
the
> > profiler,
> > > > > I've
> > > > > > > > found that Grid-Stat is doing some pretty "dumb"
memory
> > > allocation.
> > > > > > And
> > > > > > > > those types of problems only become evident on large
grids.
> > > > > > > >
> > > > > > > > The good news is that using the development version of
> > Grid-Stat
> > > > the
> > > > > > I've
> > > > > > > > gotten the continuous statistics run time down to 19
seconds.
> > > > > > > >
> > > > > > > > Now I need to figure out which of the many little
changes I
> > made
> > > > are
> > > > > > the
> > > > > > > > ones necessary for the improvement.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John
> > > > > > > >
> > > > > > > > On Tue, Jan 24, 2017 at 9:28 AM, Shannon Rees - NOAA
> Affiliate
> > > via
> > > > > RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > Ticket/Display.html?id=79242
> > > >
> > > > > > > > >
> > > > > > > > > Great, thanks for your help!
> > > > > > > > >
> > > > > > > > > On Tue, Jan 24, 2017 at 11:27 AM, John Halley Gotway
via
> RT <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > > Shannon,
> > > > > > > > > >
> > > > > > > > > > I really wouldn't worry about the 64-bit flag
setting.
> I'm
> > > > just
> > > > > > glad
> > > > > > > > you
> > > > > > > > > > were able to run successfully.
> > > > > > > > > >
> > > > > > > > > > I'll try to figure out what's taking so long in
Grid-Stat
> > to
> > > > see
> > > > > if
> > > > > > > > > there's
> > > > > > > > > > any way to speed it up.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > John
> > > > > > > > > >
> > > > > > > > > > On Tue, Jan 24, 2017 at 7:54 AM, Shannon Rees -
NOAA
> > > Affiliate
> > > > > via
> > > > > > > RT <
> > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > <URL: https://rt.rap.ucar.edu/rt/
> > > > Ticket/Display.html?id=79242
> > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi John,
> > > > > > > > > > >
> > > > > > > > > > > I recompiled both without 64 bit turned on and
the
> > > > > > plot_data_plane
> > > > > > > > > > command
> > > > > > > > > > > worked. I plotted both R1 and R2 (attached). I
am not
> > > sure
> > > > > why
> > > > > > R1
> > > > > > > > is
> > > > > > > > > > > there or what it is representing. I ran grid-
stat
> again
> > > and
> > > > it
> > > > > > > > > processed
> > > > > > > > > > > the files and the results look reasonable. It
does
> take
> > a
> > > > long
> > > > > > > time.
> > > > > > > > > I
> > > > > > > > > > > thought this was because of the amount of
points, but
> if
> > > you
> > > > > say
> > > > > > > that
> > > > > > > > > it
> > > > > > > > > > is
> > > > > > > > > > > still slower than usual that is promising. It
would be
> > > very
> > > > > > useful
> > > > > > > > if
> > > > > > > > > we
> > > > > > > > > > > could speed it up. I tried regridding to 13 km
first,
> > and
> > > it
> > > > > ran
> > > > > > > > much
> > > > > > > > > > more
> > > > > > > > > > > quickly.
> > > > > > > > > > >
> > > > > > > > > > > My main concern now is how will running without
64 bit
> > > affect
> > > > > > > things?
> > > > > > > > > > Will
> > > > > > > > > > > it hurt my ability to process very large files?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Shannon
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jan 23, 2017 at 6:48 PM, John Halley
Gotway via
> > RT
> > > <
> > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Shannon,
> > > > > > > > > > > >
> > > > > > > > > > > > Just an update. Looking more closely at the
GRIB2
> file
> > > you
> > > > > > > sent, I
> > > > > > > > > > found
> > > > > > > > > > > > the difference between the two temperature
records in
> > the
> > > > > > > "process"
> > > > > > > > > id:
> > > > > > > > > > > >
> > > > > > > > > > > > wgrib2 -process LTIA98_KWBR_201305180600.grb2
> > > > > > > > > > > > 1:0:code table 4.3=7 anl err
> > > > > > > > > > > > 2:3323062:code table 4.3=0 anl
> > > > > > > > > > > >
> > > > > > > > > > > > So in the development version of MET, I added
support
> > for
> > > > > > > > specifying
> > > > > > > > > > > > "GRIB2_process" (as well as GRIB2_pdt,
> GRIB2_ens_type,
> > > and
> > > > > > > > > > > > GRIB2_der_type). In version 6.0, you'll be
able to
> run
> > > > this
> > > > > > > > command:
> > > > > > > > > > > >
> > > > > > > > > > > > plot_data_plane LTIA98_KWBR_201305180600.grb2
> > tmp_z2.ps
> > > > > > > > 'name="TMP";
> > > > > > > > > > > > level="Z2"; GRIB2_process=0;'
> > > > > > > > > > > >
> > > > > > > > > > > > The "GRIB2_process=0;" option says only match
GRIB
> > > records
> > > > > > whose
> > > > > > > > > > process
> > > > > > > > > > > > value is 0 (or change to 7 for the 1st
record). In
> MET
> > > > > version
> > > > > > > > 5.2,
> > > > > > > > > > > you're
> > > > > > > > > > > > stuck with specifying the exact record number
to use:
> > > > > > > > > > > >
> > > > > > > > > > > > plot_data_plane LTIA98_KWBR_201305180600.grb2
> > tmp_z2.ps
> > > > > > > > 'name="TMP";
> > > > > > > > > > > > level="R2";'
> > > > > > > > > > > >
> > > > > > > > > > > > Still no word yet on why it's taking so long
to run.
> > > > > > > > > > > >
> > > > > > > > > > > > John
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Jan 23, 2017 at 2:34 PM, John Halley
Gotway <
> > > > > > > > johnhg at ucar.edu
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Also, when you recompile MET, please save
the
> output
> > > to a
> > > > > log
> > > > > > > > file:
> > > > > > > > > > > > > make install >& make_install.log
> > > > > > > > > > > > >
> > > > > > > > > > > > > If you continue to have problems, please
send me
> that
> > > > > > > > > > make_install.log
> > > > > > > > > > > > > file along with the config.log file from the
> > top-level
> > > > MET
> > > > > > > > > directory.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > John
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Jan 23, 2017 at 2:32 PM, John Halley
> Gotway <
> > > > > > > > > johnhg at ucar.edu
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> OK, thanks for letting me know. Let's try
to
> solve
> > > this
> > > > > > > problem
> > > > > > > > > and
> > > > > > > > > > > get
> > > > > > > > > > > > >> plot_data_plane working on GRIB2 data.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> You mentioned that you'd compiled it using
64-bit
> > > > flags...
> > > > > > as
> > > > > > > > I'm
> > > > > > > > > > sure
> > > > > > > > > > > > >> you read from the MET online tutorial.
Let's try
> > > > removing
> > > > > > > > those.
> > > > > > > > > > Are
> > > > > > > > > > > > you
> > > > > > > > > > > > >> able to recompile both the GRIB2C library
and MET,
> > > > > removing
> > > > > > > > those
> > > > > > > > > 64
> > > > > > > > > > > bit
> > > > > > > > > > > > >> flags?
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> And then try that plot_data_plane command
again.
> > > Please
> > > > > let
> > > > > > > me
> > > > > > > > > know
> > > > > > > > > > > how
> > > > > > > > > > > > >> it goes.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > >> John
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> On Mon, Jan 23, 2017 at 1:26 PM, Shannon
Rees -
> NOAA
> > > > > > Affiliate
> > > > > > > > via
> > > > > > > > > > RT
> > > > > > > > > > > <
> > > > > > > > > > > > >> met_help at ucar.edu> wrote:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> <URL: https://rt.rap.ucar.edu/rt/
> > > > > > > Ticket/Display.html?id=79242
> > > > > > > > >
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> Thanks John, I tried the plot_data_plane
tool
> but I
> > > got
> > > > > an
> > > > > > > > error:
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> ~/met-5.2/met-
5.2_bugfix/bin/plot_data_plane
> > > > > > > > > > > > >>> LTIA98_KWBR_201305180600.grb2
> > > > > > > > > > > > >>> tmp_z2.ps 'name="TMP"; level="R2" ; '
> > > > > > > > > > > > >>> DEBUG 1: Opening data file:
> > > > LTIA98_KWBR_201305180600.grb2
> > > > > > > > > > > > >>> terminate called after throwing an
instance of
> > > > > > > 'std::bad_alloc'
> > > > > > > > > > > > >>> what(): std::bad_alloc
> > > > > > > > > > > > >>> Abort
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> On Mon, Jan 23, 2017 at 3:06 PM, John
Halley
> Gotway
> > > via
> > > > > RT
> > > > > > <
> > > > > > > > > > > > >>> met_help at ucar.edu> wrote:
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> > Hi Shannon,
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > Thanks for sending your sample data
files. I
> > used
> > > > your
> > > > > > > email
> > > > > > > > > to
> > > > > > > > > > > > >>> create a
> > > > > > > > > > > > >>> > met_help support ticket, which enables
us to
> > track
> > > > the
> > > > > > > > support
> > > > > > > > > we
> > > > > > > > > > > > >>> provide.
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > I ran your data to compare the NetCDF
forecast
> > > file (
> > > > > > > fcst.nc
> > > > > > > > )
> > > > > > > > > to
> > > > > > > > > > > the
> > > > > > > > > > > > >>> > GRIB2 file
(LTIA98_KWBR_201305180600.grb2)
> over
> > > the
> > > > > FULL
> > > > > > > > model
> > > > > > > > > > > > domain.
> > > > > > > > > > > > >>> > While Grid-Stat *eventually* ran to
completion,
> > it
> > > > took
> > > > > > > over
> > > > > > > > 36
> > > > > > > > > > > > >>> minutes to
> > > > > > > > > > > > >>> > do so! I'd like to debug to figure out
what's
> > > taking
> > > > > so
> > > > > > > > long.
> > > > > > > > > > You
> > > > > > > > > > > > >>> already
> > > > > > > > > > > > >>> > had bootstrapping disabled (n_rep = 0)
and rank
> > > > > > correlation
> > > > > > > > > > > > statistics
> > > > > > > > > > > > >>> > turned off (rank_corr_flag = FALSE)...
which
> are
> > > the
> > > > > > usual
> > > > > > > > > > > suspects.
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > Granted, computing statistics over
1,774,484
> > > matched
> > > > > > pairs
> > > > > > > > is a
> > > > > > > > > > lot
> > > > > > > > > > > > of
> > > > > > > > > > > > >>> > work... but 36 minutes is pretty
ridiculous.
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > I wonder if the issue is related to
compiling
> > > things
> > > > > with
> > > > > > > 64
> > > > > > > > > bit
> > > > > > > > > > > > >>> flags. I
> > > > > > > > > > > > >>> > think our initial goal should be
ensuring that
> > you
> > > > can
> > > > > > get
> > > > > > > > MET
> > > > > > > > > to
> > > > > > > > > > > > read
> > > > > > > > > > > > >>> > GRIB2 data. Please try simply plotting
data
> from
> > > > that
> > > > > > > GRIB2
> > > > > > > > > file
> > > > > > > > > > > by
> > > > > > > > > > > > >>> > running:
> > > > > > > > > > > > >>> > met-5.2/bin/plot_data_plane
> > > > > > > LTIA98_KWBR_201305180600.grb2
> > > > > > > > > > > > tmp_z2.ps
> > > > > > > > > > > > >>> > 'name="TMP"; level="R2";
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > Notice that I said "R2" which tells MET
to plot
> > > > record
> > > > > > > number
> > > > > > > > > 2.
> > > > > > > > > > > > >>> Record
> > > > > > > > > > > > >>> > numbers 1 and 2 both contain temperature
data
> and
> > > > > > 2-meters.
> > > > > > > > > > Here's
> > > > > > > > > > > > >>> some
> > > > > > > > > > > > >>> > wgrib2 output:
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > 1:0:d=2013051806:TMP:2 m above
> > > > > > > ground:anl:analysis/forecast
> > > > > > > > > > error
> > > > > > > > > > > > >>> > 2:3323062:d=2013051806 <(201)%20305-
1806>
> > > > > > > > > <(201)%20305-1806>:TMP:2 m above
> > > > > > > > > > > ground:anl:
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > So far, all the GRIB id info has been
the same
> > > > between
> > > > > > > > records
> > > > > > > > > 1
> > > > > > > > > > > and
> > > > > > > > > > > > >>> 2...
> > > > > > > > > > > > >>> > but presumably there's a difference
somewhere.
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > Please let me know how the
plot_data_plane
> > command
> > > > > goes.
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> > Thanks,
> > > > > > > > > > > > >>> > John Halley Gotway
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>> >
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> --
> > > > > > > > > > > > >>> Shannon Rees
> > > > > > > > > > > > >>> Programming Scientist
> > > > > > > > > > > > >>> Engility
> > > > > > > > > > > > >>> Geophysical Fluid Dynamics Lab
> > > > > > > > > > > > >>> 201 Forrestal Rd Princeton, NJ
> > > > > > > > > > > > >>> 609-452-5384 <(609)%20452-5384>
> <(609)%20452-5384>
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Shannon Rees
> > > > > > > > > > > Programming Scientist
> > > > > > > > > > > Engility
> > > > > > > > > > > Geophysical Fluid Dynamics Lab
> > > > > > > > > > > 201 Forrestal Rd Princeton, NJ
> > > > > > > > > > > 609-452-5384 <(609)%20452-5384> <(609)%20452-
5384>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Shannon Rees
> > > > > > > > > Programming Scientist
> > > > > > > > > Engility
> > > > > > > > > Geophysical Fluid Dynamics Lab
> > > > > > > > > 201 Forrestal Rd Princeton, NJ
> > > > > > > > > 609-452-5384 <(609)%20452-5384>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Shannon Rees
> > > > > > > Programming Scientist
> > > > > > > Engility
> > > > > > > Geophysical Fluid Dynamics Lab
> > > > > > > 201 Forrestal Rd Princeton, NJ
> > > > > > > 609-452-5384
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Shannon Rees
> > > > > Programming Scientist
> > > > > Engility
> > > > > Geophysical Fluid Dynamics Lab
> > > > > 201 Forrestal Rd Princeton, NJ
> > > > > 609-452-5384
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Shannon Rees
> > > Programming Scientist
> > > Engility
> > > Geophysical Fluid Dynamics Lab
> > > 201 Forrestal Rd Princeton, NJ
> > > 609-452-5384
> > >
> > >
> >
> >
>
>
> --
> Shannon Rees
> Programming Scientist
> Engility
> Geophysical Fluid Dynamics Lab
> 201 Forrestal Rd Princeton, NJ
> 609-452-5384
>
>
------------------------------------------------
More information about the Met_help
mailing list