[Met_help] [rt.rap.ucar.edu #96126] History for Compression and grid_stat run time

[John Halley Gotway via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Good morning MET help desk
I'm trying to figure out what compression levels we should be using for
regrid_data_plane with the latest version on WCOSS (V9.0.3) and how to get
the fastest run times out of grid_stat.  We did see this ticket
<https://github.com/NCAR/MET/issues/1363> in the latest release notes.  I
wasn't sure if this change would explain the results we were seeing.
For reference, an uncompressed netCDF file that comes out of
regrid_data_plane for most models for the CONUS is ~44MB.  With -compress
level 9 turned on, it compresses to ~30MB using MET V9.0.3.  Using MET
V8.1, it compresses to ~23MB.  With the number of models and projections
that we work with, 7MB per file adds up quickly.  Is it possible to get
more compression with the newer version of MET?
As for grid_stat, it is currently taking 8-10 minutes to run grid_stat for
a single case (or ~40 minutes for all models).  I don't recall if this
started when we started using V9.0.2 or V9.0.3, but it used to take only a
minute or two to run grid_stat for a single case (or ~5 minutes for all
models).   I'm not sure if the compression of the netCDF files is the issue
(likely not) or if something else changed in grid_stat to increase the run
times.  If you know of any ways to speed this up to the previous run time,
please let me know.  If not, we will likely revert back to v8.1.  I have
not tested the V9.1 betas yet.
Thanks
John
-- 
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Compression and grid_stat run time
From: George McCabe
Time: Wed Aug 05 08:36:09 2020

Hi John,

I see you have questions about compression in regards to run time
length and file sizes. I have assigned this ticket to Howard Soh.
Please allow a few business days for a response.

Thanks,
George

------------------------------------------------
Subject: Compression and grid_stat run time
From: Howard Soh
Time: Wed Aug 05 22:29:00 2020

The compression is done by NetCDF library. The NetCDF output (size
difference) is caused by the NetCDF library, not by MET code. Please
check the NetCDF library version:
   ldd regrid_data_plane | grep netcdf

The issue 1363 (applied 9.0.3 and 9.1 beta 2) improves the speed of
reading NetCDF input, and no update on writing NetCDF output.

Please let us know if 9.0.3 runs slower than 9.0.2 with using the same
NetCDF library.

Cheers,
Howard

Here are examples to check NetCDF library:
ldd met-8.1/bin/regrid_data_plane | grep netcdf
        libnetcdf_c++4.so.1 =>
/usr/local/netcdf/lib/libnetcdf_c++4.so.1 (0x00007fee87e70000)
        libnetcdf.so.7 => not found
        libnetcdf.so.13 => /usr/local/netcdf/lib/libnetcdf.so.13
(0x00007fee83f05000)

ldd met-9.0.2/bin/regrid_data_plane | grep netcdf
        libnetcdf_c++4.so.1 => /usr/local/netcdf-4.7.0/gcc-
6.3.0/lib/libnetcdf_c++4.so.1 (0x00007fdf5d1b2000)
        libnetcdf.so.15 => /usr/local/netcdf-4.7.0/gcc-
6.3.0/lib/libnetcdf.so.15 (0x00007fdf5ce31000)


ldd met-9.0.3/bin/regrid_data_plane | grep netcdf
        libnetcdf_c++4.so.1 => /usr/local/netcdf-4.7.0/gcc-
6.3.0/lib/libnetcdf_c++4.so.1 (0x00007f7dfe91d000)
        libnetcdf.so.15 => /usr/local/netcdf-4.7.0/gcc-
6.3.0/lib/libnetcdf.so.15 (0x00007f7dfe59c000)


On Wed Aug 05 08:36:09 2020, mccabe wrote:
> Hi John,
>
> I see you have questions about compression in regards to run time
> length and file sizes. I have assigned this ticket to Howard Soh.
> Please allow a few business days for a response.
>
> Thanks,
> George



------------------------------------------------
Subject: Compression and grid_stat run time
From: John L Wagner - NOAA Federal
Time: Mon Aug 10 10:10:44 2020

Thanks Julie.  I did submit this to the help desk last week.  That is
where
Howard mentioned checking the netcdf library.  I'll submit again to
get
John's thoughts.
John, please see my email below.  If you need any more information
please
let me know.
Thanks
John

On Mon, Aug 10, 2020 at 12:05 PM Julie Prestopnik <jpresto at ucar.edu>
wrote:

> Hi John.
>
> I am not the best person to address these issues about compression.
I can
> confirm that I started linking to v4.5.0 of the NetCDF library for
the
> reason you indicated - to use the standard WCOSS libraries, as that
is how
> it will be done for the operational installation of MET.
>
> Could you please submit your questions about compression to
> met_help at ucar.edu so that your questions can be addressed by the
right
> person (likely John, who also happens to be on helpdesk today)?
>
> Thanks!
>
> Julie
>
> On Mon, Aug 10, 2020 at 8:12 AM John L Wagner - NOAA Federal <
> john.l.wagner at noaa.gov> wrote:
>
>> Good morning Julie
>> Over the past few weeks, we've been seeing less compression with
>> regrid_data_plane and significantly longer run times with
grid_stat.  For
>> regrid_data_plane, our CONUS files used to compress from ~44MB to
~23MB.
>> Now they only compress to ~30MB.  For grid_stat, a single model and
>> analysis grid would run in a minute or two.  It now takes 8-10
minutes to
>> run.  We have been testing mainly with MET V9.0.2 and V9.0.3 out of
your
>> directory on mars.
>> Howard mentioned that the compression is controlled by the netcdf
>> library.  Starting with V9.0.1, you started linking to V4.5.0.
Prior to
>> that, it looks like you had your own version of the library.  I
assume
>> the change was made just to use the standard WCOSS libraries.  I
was just
>> wondering if anyone else had experienced the same issues that we
have after
>> this change?
>> I ran some tests over the weekend reverting back to MET V8.1.  I
didn't
>> get in early enough this morning to check the file sizes of the
>> regrid_data_plane output, but the grid_stat run times were still
longer
>> than expected.  I suspect the netcdf library is not the issue
there.  If
>> you know of any issues that might be causing longer grid_stat run
times or
>> have any suggestions for other tests we can try, please let me
know.
>> Thanks
>> John
>>
>> ---------- Forwarded message ---------
>> From: Howard Soh via RT <met_help at ucar.edu>
>> Date: Thu, Aug 6, 2020 at 12:29 AM
>> Subject: [rt.rap.ucar.edu #96126] Compression and grid_stat run
time
>> To: <john.l.wagner at noaa.gov>
>> Cc: <dana.strom at noaa.gov>, <tamarah.curtis at noaa.gov>
>>
>>
>> The compression is done by NetCDF library. The NetCDF output (size
>> difference) is caused by the NetCDF library, not by MET code.
Please check
>> the NetCDF library version:
>>    ldd regrid_data_plane | grep netcdf
>>
>> The issue 1363 (applied 9.0.3 and 9.1 beta 2) improves the speed of
>> reading NetCDF input, and no update on writing NetCDF output.
>>
>> Please let us know if 9.0.3 runs slower than 9.0.2 with using the
same
>> NetCDF library.
>>
>> Cheers,
>> Howard
>>
>> Here are examples to check NetCDF library:
>> ldd met-8.1/bin/regrid_data_plane | grep netcdf
>>         libnetcdf_c++4.so.1 =>
/usr/local/netcdf/lib/libnetcdf_c++4.so.1
>> (0x00007fee87e70000)
>>         libnetcdf.so.7 => not found
>>         libnetcdf.so.13 => /usr/local/netcdf/lib/libnetcdf.so.13
>> (0x00007fee83f05000)
>>
>> ldd met-9.0.2/bin/regrid_data_plane | grep netcdf
>>         libnetcdf_c++4.so.1 =>
>> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
>> (0x00007fdf5d1b2000)
>>         libnetcdf.so.15 =>
>> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
(0x00007fdf5ce31000)
>>
>>
>> ldd met-9.0.3/bin/regrid_data_plane | grep netcdf
>>         libnetcdf_c++4.so.1 =>
>> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
>> (0x00007f7dfe91d000)
>>         libnetcdf.so.15 =>
>> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
(0x00007f7dfe59c000)
>>
>>
>> On Wed Aug 05 08:36:09 2020, mccabe wrote:
>> > Hi John,
>> >
>> > I see you have questions about compression in regards to run time
>> > length and file sizes. I have assigned this ticket to Howard Soh.
>> > Please allow a few business days for a response.
>> >
>> > Thanks,
>> > George
>>
>>
>>
>>
>>
>> --
>> John Wagner
>> Verification Task Lead
>> NOAA/National Weather Service
>> Meteorological Development Laboratory
>> Digital Forecast Services Division
>> SSMC2 Room 10106
>> Silver Spring, MD 20910
>> (301) 427-9471 (office)
>> (908) 902-4155 (cell/text)
>>
>
>
> --
> Julie Prestopnik (she/her/hers)
> Software Engineer
> National Center for Atmospheric Research
> Research Applications Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
> My working day may not be your working day.  Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>


--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)

------------------------------------------------
Subject: Compression and grid_stat run time
From: John Halley Gotway
Time: Tue Aug 11 09:57:04 2020

John,

I did see your email come through met-help last week. Howard is the
person
who's worked the most on NetCDF compression, so I was glad he was able
to
respond. Unfortunately, I don't have any quick and obvious answers for
you.
We just posted the met-9.1 release yesterday. It sounds like you're
weighing 2 main considerations... the output file size of
regrid_data_plane
and the runtime of grid_stat.

Seems like we should do some comparison tests between met-8.1 and met-
9.1.
What I'd like to figure out is whether the longer runtimes are due
solely
to the difference in NetCDF compression levels. Howard has told me
that
writing highly compressed NetCDF files is much, much slower.

We could pick a single case and run it through
regrid_data_plane/grid_stat
6 times... with no/medium/high compression and met-8.1/9.1. Then for
each
configuration, note the runtime and file sizes. And I also want to
confirm
that the output generated is the same (i.e. met-9.1 isn't writing any
extra
output variables). WCOSS is down this week, but I could do this
testing on
a project machine at NCAR. But I'd like to approximate the datasets
you're
using on WCOSS.

Can you point me to sample data files, the regrid_data_plane command
you're
running, and the Grid-Stat configuration files you're using?

Thanks,
John

On Mon, Aug 10, 2020 at 10:11 AM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126 >
>
> Thanks Julie.  I did submit this to the help desk last week.  That
is where
> Howard mentioned checking the netcdf library.  I'll submit again to
get
> John's thoughts.
> John, please see my email below.  If you need any more information
please
> let me know.
> Thanks
> John
>
> On Mon, Aug 10, 2020 at 12:05 PM Julie Prestopnik <jpresto at ucar.edu>
> wrote:
>
> > Hi John.
> >
> > I am not the best person to address these issues about
compression.  I
> can
> > confirm that I started linking to v4.5.0 of the NetCDF library for
the
> > reason you indicated - to use the standard WCOSS libraries, as
that is
> how
> > it will be done for the operational installation of MET.
> >
> > Could you please submit your questions about compression to
> > met_help at ucar.edu so that your questions can be addressed by the
right
> > person (likely John, who also happens to be on helpdesk today)?
> >
> > Thanks!
> >
> > Julie
> >
> > On Mon, Aug 10, 2020 at 8:12 AM John L Wagner - NOAA Federal <
> > john.l.wagner at noaa.gov> wrote:
> >
> >> Good morning Julie
> >> Over the past few weeks, we've been seeing less compression with
> >> regrid_data_plane and significantly longer run times with
grid_stat.
> For
> >> regrid_data_plane, our CONUS files used to compress from ~44MB to
~23MB.
> >> Now they only compress to ~30MB.  For grid_stat, a single model
and
> >> analysis grid would run in a minute or two.  It now takes 8-10
minutes
> to
> >> run.  We have been testing mainly with MET V9.0.2 and V9.0.3 out
of your
> >> directory on mars.
> >> Howard mentioned that the compression is controlled by the netcdf
> >> library.  Starting with V9.0.1, you started linking to V4.5.0.
Prior to
> >> that, it looks like you had your own version of the library.  I
assume
> >> the change was made just to use the standard WCOSS libraries.  I
was
> just
> >> wondering if anyone else had experienced the same issues that we
have
> after
> >> this change?
> >> I ran some tests over the weekend reverting back to MET V8.1.  I
didn't
> >> get in early enough this morning to check the file sizes of the
> >> regrid_data_plane output, but the grid_stat run times were still
longer
> >> than expected.  I suspect the netcdf library is not the issue
there.  If
> >> you know of any issues that might be causing longer grid_stat run
times
> or
> >> have any suggestions for other tests we can try, please let me
know.
> >> Thanks
> >> John
> >>
> >> ---------- Forwarded message ---------
> >> From: Howard Soh via RT <met_help at ucar.edu>
> >> Date: Thu, Aug 6, 2020 at 12:29 AM
> >> Subject: [rt.rap.ucar.edu #96126] Compression and grid_stat run
time
> >> To: <john.l.wagner at noaa.gov>
> >> Cc: <dana.strom at noaa.gov>, <tamarah.curtis at noaa.gov>
> >>
> >>
> >> The compression is done by NetCDF library. The NetCDF output
(size
> >> difference) is caused by the NetCDF library, not by MET code.
Please
> check
> >> the NetCDF library version:
> >>    ldd regrid_data_plane | grep netcdf
> >>
> >> The issue 1363 (applied 9.0.3 and 9.1 beta 2) improves the speed
of
> >> reading NetCDF input, and no update on writing NetCDF output.
> >>
> >> Please let us know if 9.0.3 runs slower than 9.0.2 with using the
same
> >> NetCDF library.
> >>
> >> Cheers,
> >> Howard
> >>
> >> Here are examples to check NetCDF library:
> >> ldd met-8.1/bin/regrid_data_plane | grep netcdf
> >>         libnetcdf_c++4.so.1 =>
/usr/local/netcdf/lib/libnetcdf_c++4.so.1
> >> (0x00007fee87e70000)
> >>         libnetcdf.so.7 => not found
> >>         libnetcdf.so.13 => /usr/local/netcdf/lib/libnetcdf.so.13
> >> (0x00007fee83f05000)
> >>
> >> ldd met-9.0.2/bin/regrid_data_plane | grep netcdf
> >>         libnetcdf_c++4.so.1 =>
> >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
> >> (0x00007fdf5d1b2000)
> >>         libnetcdf.so.15 =>
> >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
> (0x00007fdf5ce31000)
> >>
> >>
> >> ldd met-9.0.3/bin/regrid_data_plane | grep netcdf
> >>         libnetcdf_c++4.so.1 =>
> >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
> >> (0x00007f7dfe91d000)
> >>         libnetcdf.so.15 =>
> >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
> (0x00007f7dfe59c000)
> >>
> >>
> >> On Wed Aug 05 08:36:09 2020, mccabe wrote:
> >> > Hi John,
> >> >
> >> > I see you have questions about compression in regards to run
time
> >> > length and file sizes. I have assigned this ticket to Howard
Soh.
> >> > Please allow a few business days for a response.
> >> >
> >> > Thanks,
> >> > George
> >>
> >>
> >>
> >>
> >>
> >> --
> >> John Wagner
> >> Verification Task Lead
> >> NOAA/National Weather Service
> >> Meteorological Development Laboratory
> >> Digital Forecast Services Division
> >> SSMC2 Room 10106
> >> Silver Spring, MD 20910
> >> (301) 427-9471 (office)
> >> (908) 902-4155 (cell/text)
> >>
> >
> >
> > --
> > Julie Prestopnik (she/her/hers)
> > Software Engineer
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > Phone: 303.497.8399
> > Email: jpresto at ucar.edu
> >
> > My working day may not be your working day.  Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.
> >
>
>
> --
> John Wagner
> Verification Task Lead
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Division
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>

------------------------------------------------
Subject: Compression and grid_stat run time
From: John L Wagner - NOAA Federal
Time: Tue Aug 11 12:03:36 2020

Thanks John.  Attached are the NDFD and URMA grib2 files that we are
starting with and the grid_stat config file we are using.  The
regrid_data_plane command we are using is

regrid_data_plane <grib2 file> <CONUS mask> <output netCDF file>
-method
NEAREST -width 1 -v 0 -name "tt" -compress 9 -field "name=tt;
level=Z2;"

I have access to the prod machine (venus) so I have been running some
tests
today with grid_stat.  I've run tests with V8.1 out of NCO's area and
with
V8.1, V9.0.1, V9.0.2, and V9.0.3 in Julie's area.  The shortest and
longest
run times I've seen today were both with Julie's V8.1.  This morning,
it
ran in about 5 minutes.  At lunch time, it ran in about 40 minutes.
All
other runs took about 10 minutes.

Tests I am going to run this afternoon:
1.  We are currently running with masks for all of the CONUS WFOs and
regions (about 135 masks).  I'm going to test with just the CONUS
mask.
2.  I'll need to look at what elements we use this grid_stat_config
file
for.  For temperature, we only need the cnt file.  I'll turn off the
other
files.
3.  I am going to try on the Cray instead of the Dell.  We've run into
several problems where the only logical problem is the read and write
speeds on the Dell (phase 3).  If your tests on the project machine at
NCAR
don't show any issues, then this will continue to be what I think the
real
problem is.

If you need anything else from me, let me know.
Thanks
John

On Tue, Aug 11, 2020 at 11:57 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:

> John,
>
> I did see your email come through met-help last week. Howard is the
person
> who's worked the most on NetCDF compression, so I was glad he was
able to
> respond. Unfortunately, I don't have any quick and obvious answers
for you.
> We just posted the met-9.1 release yesterday. It sounds like you're
> weighing 2 main considerations... the output file size of
regrid_data_plane
> and the runtime of grid_stat.
>
> Seems like we should do some comparison tests between met-8.1 and
met-9.1.
> What I'd like to figure out is whether the longer runtimes are due
solely
> to the difference in NetCDF compression levels. Howard has told me
that
> writing highly compressed NetCDF files is much, much slower.
>
> We could pick a single case and run it through
regrid_data_plane/grid_stat
> 6 times... with no/medium/high compression and met-8.1/9.1. Then for
each
> configuration, note the runtime and file sizes. And I also want to
confirm
> that the output generated is the same (i.e. met-9.1 isn't writing
any extra
> output variables). WCOSS is down this week, but I could do this
testing on
> a project machine at NCAR. But I'd like to approximate the datasets
you're
> using on WCOSS.
>
> Can you point me to sample data files, the regrid_data_plane command
you're
> running, and the Grid-Stat configuration files you're using?
>
> Thanks,
> John
>
> On Mon, Aug 10, 2020 at 10:11 AM John L Wagner - NOAA Federal via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126 >
> >
> > Thanks Julie.  I did submit this to the help desk last week.  That
is
> where
> > Howard mentioned checking the netcdf library.  I'll submit again
to get
> > John's thoughts.
> > John, please see my email below.  If you need any more information
please
> > let me know.
> > Thanks
> > John
> >
> > On Mon, Aug 10, 2020 at 12:05 PM Julie Prestopnik
<jpresto at ucar.edu>
> > wrote:
> >
> > > Hi John.
> > >
> > > I am not the best person to address these issues about
compression.  I
> > can
> > > confirm that I started linking to v4.5.0 of the NetCDF library
for the
> > > reason you indicated - to use the standard WCOSS libraries, as
that is
> > how
> > > it will be done for the operational installation of MET.
> > >
> > > Could you please submit your questions about compression to
> > > met_help at ucar.edu so that your questions can be addressed by the
right
> > > person (likely John, who also happens to be on helpdesk today)?
> > >
> > > Thanks!
> > >
> > > Julie
> > >
> > > On Mon, Aug 10, 2020 at 8:12 AM John L Wagner - NOAA Federal <
> > > john.l.wagner at noaa.gov> wrote:
> > >
> > >> Good morning Julie
> > >> Over the past few weeks, we've been seeing less compression
with
> > >> regrid_data_plane and significantly longer run times with
grid_stat.
> > For
> > >> regrid_data_plane, our CONUS files used to compress from ~44MB
to
> ~23MB.
> > >> Now they only compress to ~30MB.  For grid_stat, a single model
and
> > >> analysis grid would run in a minute or two.  It now takes 8-10
minutes
> > to
> > >> run.  We have been testing mainly with MET V9.0.2 and V9.0.3
out of
> your
> > >> directory on mars.
> > >> Howard mentioned that the compression is controlled by the
netcdf
> > >> library.  Starting with V9.0.1, you started linking to V4.5.0.
Prior
> to
> > >> that, it looks like you had your own version of the library.  I
assume
> > >> the change was made just to use the standard WCOSS libraries.
I was
> > just
> > >> wondering if anyone else had experienced the same issues that
we have
> > after
> > >> this change?
> > >> I ran some tests over the weekend reverting back to MET V8.1.
I
> didn't
> > >> get in early enough this morning to check the file sizes of the
> > >> regrid_data_plane output, but the grid_stat run times were
still
> longer
> > >> than expected.  I suspect the netcdf library is not the issue
there.
> If
> > >> you know of any issues that might be causing longer grid_stat
run
> times
> > or
> > >> have any suggestions for other tests we can try, please let me
know.
> > >> Thanks
> > >> John
> > >>
> > >> ---------- Forwarded message ---------
> > >> From: Howard Soh via RT <met_help at ucar.edu>
> > >> Date: Thu, Aug 6, 2020 at 12:29 AM
> > >> Subject: [rt.rap.ucar.edu #96126] Compression and grid_stat run
time
> > >> To: <john.l.wagner at noaa.gov>
> > >> Cc: <dana.strom at noaa.gov>, <tamarah.curtis at noaa.gov>
> > >>
> > >>
> > >> The compression is done by NetCDF library. The NetCDF output
(size
> > >> difference) is caused by the NetCDF library, not by MET code.
Please
> > check
> > >> the NetCDF library version:
> > >>    ldd regrid_data_plane | grep netcdf
> > >>
> > >> The issue 1363 (applied 9.0.3 and 9.1 beta 2) improves the
speed of
> > >> reading NetCDF input, and no update on writing NetCDF output.
> > >>
> > >> Please let us know if 9.0.3 runs slower than 9.0.2 with using
the same
> > >> NetCDF library.
> > >>
> > >> Cheers,
> > >> Howard
> > >>
> > >> Here are examples to check NetCDF library:
> > >> ldd met-8.1/bin/regrid_data_plane | grep netcdf
> > >>         libnetcdf_c++4.so.1 =>
> /usr/local/netcdf/lib/libnetcdf_c++4.so.1
> > >> (0x00007fee87e70000)
> > >>         libnetcdf.so.7 => not found
> > >>         libnetcdf.so.13 =>
/usr/local/netcdf/lib/libnetcdf.so.13
> > >> (0x00007fee83f05000)
> > >>
> > >> ldd met-9.0.2/bin/regrid_data_plane | grep netcdf
> > >>         libnetcdf_c++4.so.1 =>
> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
> > >> (0x00007fdf5d1b2000)
> > >>         libnetcdf.so.15 =>
> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
> > (0x00007fdf5ce31000)
> > >>
> > >>
> > >> ldd met-9.0.3/bin/regrid_data_plane | grep netcdf
> > >>         libnetcdf_c++4.so.1 =>
> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
> > >> (0x00007f7dfe91d000)
> > >>         libnetcdf.so.15 =>
> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
> > (0x00007f7dfe59c000)
> > >>
> > >>
> > >> On Wed Aug 05 08:36:09 2020, mccabe wrote:
> > >> > Hi John,
> > >> >
> > >> > I see you have questions about compression in regards to run
time
> > >> > length and file sizes. I have assigned this ticket to Howard
Soh.
> > >> > Please allow a few business days for a response.
> > >> >
> > >> > Thanks,
> > >> > George
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> --
> > >> John Wagner
> > >> Verification Task Lead
> > >> NOAA/National Weather Service
> > >> Meteorological Development Laboratory
> > >> Digital Forecast Services Division
> > >> SSMC2 Room 10106
> > >> Silver Spring, MD 20910
> > >> (301) 427-9471 (office)
> > >> (908) 902-4155 (cell/text)
> > >>
> > >
> > >
> > > --
> > > Julie Prestopnik (she/her/hers)
> > > Software Engineer
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > Phone: 303.497.8399
> > > Email: jpresto at ucar.edu
> > >
> > > My working day may not be your working day.  Please do not feel
obliged
> > to
> > > reply to this email outside of your normal working hours.
> > >
> >
> >
> > --
> > John Wagner
> > Verification Task Lead
> > NOAA/National Weather Service
> > Meteorological Development Laboratory
> > Digital Forecast Services Division
> > SSMC2 Room 10106
> > Silver Spring, MD 20910
> > (301) 427-9471 (office)
> > (908) 902-4155 (cell/text)
> >
> >
>
>

--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)

------------------------------------------------
Subject: Compression and grid_stat run time
From: John L Wagner - NOAA Federal
Time: Wed Aug 12 06:42:42 2020

Good morning John
I ran the tests yesterday afternoon that I mentioned in my previous
email.
Running for just the CNT output and using the Cray's ptmp to read and
write
data had no effect on the run times.  Running for just the CONUS mask
initially took about 57 seconds to run with compressed files.
Switching to
uncompressed files, it took about 38 seconds to run for just the CONUS
mask
and 8 minutes and 40 seconds for all masks.
This morning, I tested running with just the CONUS mask again and used
the
uncompressed files.  For some reason this morning, grid_stat runs in
about
5-6 seconds, which is what I was seeing a few weeks ago.  I made no
other
changes to my scripts or modules other than to limit the runs to just
the
CONUS mask again.  The start and stop times from my logs are below.
I'm really at a loss as to what could be causing these fluctuations in
run
times.
Thanks
John

/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_ndfdconc2p519110112ttf006
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
-v 0 -outdir /gpfs/dell2/ptmp/John.L.Wagner/ndfd/co/dbtx/1911/01/18/tt
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_ndfd

*GRID STAT STARTWed Aug 12 12:17:20 UTC 2020*
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=18446744072918741855

*GRID STAT STOPWed Aug 12 12:17:26 UTC 2020*
missedTracking not set.  Running grid_stat for blend 201911017
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_blendconc2p519110107ttf011
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
-v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/blend/co/dbtx/1911/01/18/tt
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_blend

*GRID STAT STARTWed Aug 12 12:17:26 UTC 2020*
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=18446744073019405151

*GRID STAT STOPWed Aug 12 12:17:31 UTC 2020*
missedTracking not set.  Running grid_stat for blendp 201911017
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_blendpconc2p519110107ttf011
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
-v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/blendp/co/dbtx/1911/01/18/tt
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_blendp

*GRID STAT STARTWed Aug 12 12:17:31 UTC 2020*
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=18446744073103291231

*GRID STAT STOPWed Aug 12 12:17:36 UTC 2020*
missedTracking not set.  Running grid_stat for gmos 201911010
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_gmosconc2p519110100ttf018
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
-v 0 -outdir /gpfs/dell2/ptmp/John.L.Wagner/gmos/co/dbtx/1911/01/18/tt
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_gmos

*GRID STAT STARTWed Aug 12 12:17:37 UTC 2020*
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=18446744073203954527

*GRID STAT STOPWed Aug 12 12:17:42 UTC 2020*

On Tue, Aug 11, 2020 at 2:03 PM John L Wagner - NOAA Federal <
john.l.wagner at noaa.gov> wrote:

> Thanks John.  Attached are the NDFD and URMA grib2 files that we are
> starting with and the grid_stat config file we are using.  The
> regrid_data_plane command we are using is
>
> regrid_data_plane <grib2 file> <CONUS mask> <output netCDF file>
-method
> NEAREST -width 1 -v 0 -name "tt" -compress 9 -field "name=tt;
level=Z2;"
>
> I have access to the prod machine (venus) so I have been running
some
> tests today with grid_stat.  I've run tests with V8.1 out of NCO's
area and
> with V8.1, V9.0.1, V9.0.2, and V9.0.3 in Julie's area.  The shortest
and
> longest run times I've seen today were both with Julie's V8.1.  This
> morning, it ran in about 5 minutes.  At lunch time, it ran in about
40
> minutes.  All other runs took about 10 minutes.
>
> Tests I am going to run this afternoon:
> 1.  We are currently running with masks for all of the CONUS WFOs
and
> regions (about 135 masks).  I'm going to test with just the CONUS
mask.
> 2.  I'll need to look at what elements we use this grid_stat_config
file
> for.  For temperature, we only need the cnt file.  I'll turn off the
other
> files.
> 3.  I am going to try on the Cray instead of the Dell.  We've run
into
> several problems where the only logical problem is the read and
write
> speeds on the Dell (phase 3).  If your tests on the project machine
at NCAR
> don't show any issues, then this will continue to be what I think
the real
> problem is.
>
> If you need anything else from me, let me know.
> Thanks
> John
>
> On Tue, Aug 11, 2020 at 11:57 AM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> John,
>>
>> I did see your email come through met-help last week. Howard is the
person
>> who's worked the most on NetCDF compression, so I was glad he was
able to
>> respond. Unfortunately, I don't have any quick and obvious answers
for
>> you.
>> We just posted the met-9.1 release yesterday. It sounds like you're
>> weighing 2 main considerations... the output file size of
>> regrid_data_plane
>> and the runtime of grid_stat.
>>
>> Seems like we should do some comparison tests between met-8.1 and
met-9.1.
>> What I'd like to figure out is whether the longer runtimes are due
solely
>> to the difference in NetCDF compression levels. Howard has told me
that
>> writing highly compressed NetCDF files is much, much slower.
>>
>> We could pick a single case and run it through
regrid_data_plane/grid_stat
>> 6 times... with no/medium/high compression and met-8.1/9.1. Then
for each
>> configuration, note the runtime and file sizes. And I also want to
confirm
>> that the output generated is the same (i.e. met-9.1 isn't writing
any
>> extra
>> output variables). WCOSS is down this week, but I could do this
testing on
>> a project machine at NCAR. But I'd like to approximate the datasets
you're
>> using on WCOSS.
>>
>> Can you point me to sample data files, the regrid_data_plane
command
>> you're
>> running, and the Grid-Stat configuration files you're using?
>>
>> Thanks,
>> John
>>
>> On Mon, Aug 10, 2020 at 10:11 AM John L Wagner - NOAA Federal via
RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126 >
>> >
>> > Thanks Julie.  I did submit this to the help desk last week.
That is
>> where
>> > Howard mentioned checking the netcdf library.  I'll submit again
to get
>> > John's thoughts.
>> > John, please see my email below.  If you need any more
information
>> please
>> > let me know.
>> > Thanks
>> > John
>> >
>> > On Mon, Aug 10, 2020 at 12:05 PM Julie Prestopnik
<jpresto at ucar.edu>
>> > wrote:
>> >
>> > > Hi John.
>> > >
>> > > I am not the best person to address these issues about
compression.  I
>> > can
>> > > confirm that I started linking to v4.5.0 of the NetCDF library
for the
>> > > reason you indicated - to use the standard WCOSS libraries, as
that is
>> > how
>> > > it will be done for the operational installation of MET.
>> > >
>> > > Could you please submit your questions about compression to
>> > > met_help at ucar.edu so that your questions can be addressed by
the
>> right
>> > > person (likely John, who also happens to be on helpdesk today)?
>> > >
>> > > Thanks!
>> > >
>> > > Julie
>> > >
>> > > On Mon, Aug 10, 2020 at 8:12 AM John L Wagner - NOAA Federal <
>> > > john.l.wagner at noaa.gov> wrote:
>> > >
>> > >> Good morning Julie
>> > >> Over the past few weeks, we've been seeing less compression
with
>> > >> regrid_data_plane and significantly longer run times with
grid_stat.
>> > For
>> > >> regrid_data_plane, our CONUS files used to compress from ~44MB
to
>> ~23MB.
>> > >> Now they only compress to ~30MB.  For grid_stat, a single
model and
>> > >> analysis grid would run in a minute or two.  It now takes 8-10
>> minutes
>> > to
>> > >> run.  We have been testing mainly with MET V9.0.2 and V9.0.3
out of
>> your
>> > >> directory on mars.
>> > >> Howard mentioned that the compression is controlled by the
netcdf
>> > >> library.  Starting with V9.0.1, you started linking to V4.5.0.
>> Prior to
>> > >> that, it looks like you had your own version of the library.
I
>> assume
>> > >> the change was made just to use the standard WCOSS libraries.
I was
>> > just
>> > >> wondering if anyone else had experienced the same issues that
we have
>> > after
>> > >> this change?
>> > >> I ran some tests over the weekend reverting back to MET V8.1.
I
>> didn't
>> > >> get in early enough this morning to check the file sizes of
the
>> > >> regrid_data_plane output, but the grid_stat run times were
still
>> longer
>> > >> than expected.  I suspect the netcdf library is not the issue
>> there.  If
>> > >> you know of any issues that might be causing longer grid_stat
run
>> times
>> > or
>> > >> have any suggestions for other tests we can try, please let me
know.
>> > >> Thanks
>> > >> John
>> > >>
>> > >> ---------- Forwarded message ---------
>> > >> From: Howard Soh via RT <met_help at ucar.edu>
>> > >> Date: Thu, Aug 6, 2020 at 12:29 AM
>> > >> Subject: [rt.rap.ucar.edu #96126] Compression and grid_stat
run time
>> > >> To: <john.l.wagner at noaa.gov>
>> > >> Cc: <dana.strom at noaa.gov>, <tamarah.curtis at noaa.gov>
>> > >>
>> > >>
>> > >> The compression is done by NetCDF library. The NetCDF output
(size
>> > >> difference) is caused by the NetCDF library, not by MET code.
Please
>> > check
>> > >> the NetCDF library version:
>> > >>    ldd regrid_data_plane | grep netcdf
>> > >>
>> > >> The issue 1363 (applied 9.0.3 and 9.1 beta 2) improves the
speed of
>> > >> reading NetCDF input, and no update on writing NetCDF output.
>> > >>
>> > >> Please let us know if 9.0.3 runs slower than 9.0.2 with using
the
>> same
>> > >> NetCDF library.
>> > >>
>> > >> Cheers,
>> > >> Howard
>> > >>
>> > >> Here are examples to check NetCDF library:
>> > >> ldd met-8.1/bin/regrid_data_plane | grep netcdf
>> > >>         libnetcdf_c++4.so.1 =>
>> /usr/local/netcdf/lib/libnetcdf_c++4.so.1
>> > >> (0x00007fee87e70000)
>> > >>         libnetcdf.so.7 => not found
>> > >>         libnetcdf.so.13 =>
/usr/local/netcdf/lib/libnetcdf.so.13
>> > >> (0x00007fee83f05000)
>> > >>
>> > >> ldd met-9.0.2/bin/regrid_data_plane | grep netcdf
>> > >>         libnetcdf_c++4.so.1 =>
>> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
>> > >> (0x00007fdf5d1b2000)
>> > >>         libnetcdf.so.15 =>
>> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
>> > (0x00007fdf5ce31000)
>> > >>
>> > >>
>> > >> ldd met-9.0.3/bin/regrid_data_plane | grep netcdf
>> > >>         libnetcdf_c++4.so.1 =>
>> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
>> > >> (0x00007f7dfe91d000)
>> > >>         libnetcdf.so.15 =>
>> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
>> > (0x00007f7dfe59c000)
>> > >>
>> > >>
>> > >> On Wed Aug 05 08:36:09 2020, mccabe wrote:
>> > >> > Hi John,
>> > >> >
>> > >> > I see you have questions about compression in regards to run
time
>> > >> > length and file sizes. I have assigned this ticket to Howard
Soh.
>> > >> > Please allow a few business days for a response.
>> > >> >
>> > >> > Thanks,
>> > >> > George
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> John Wagner
>> > >> Verification Task Lead
>> > >> NOAA/National Weather Service
>> > >> Meteorological Development Laboratory
>> > >> Digital Forecast Services Division
>> > >> SSMC2 Room 10106
>> > >> Silver Spring, MD 20910
>> > >> (301) 427-9471 (office)
>> > >> (908) 902-4155 (cell/text)
>> > >>
>> > >
>> > >
>> > > --
>> > > Julie Prestopnik (she/her/hers)
>> > > Software Engineer
>> > > National Center for Atmospheric Research
>> > > Research Applications Laboratory
>> > > Phone: 303.497.8399
>> > > Email: jpresto at ucar.edu
>> > >
>> > > My working day may not be your working day.  Please do not feel
>> obliged
>> > to
>> > > reply to this email outside of your normal working hours.
>> > >
>> >
>> >
>> > --
>> > John Wagner
>> > Verification Task Lead
>> > NOAA/National Weather Service
>> > Meteorological Development Laboratory
>> > Digital Forecast Services Division
>> > SSMC2 Room 10106
>> > Silver Spring, MD 20910
>> > (301) 427-9471 (office)
>> > (908) 902-4155 (cell/text)
>> >
>> >
>>
>>
>
> --
> John Wagner
> Verification Task Lead
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Division
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>


--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)

------------------------------------------------
Subject: Compression and grid_stat run time
From: John Halley Gotway
Time: Thu Aug 13 11:53:20 2020

John,

Thanks for sending this data. Unfortunately, I didn't get a chance to
dig
into the details before heading out of town. But I'll be back from PTO
on
Monday.

Thanks,
John

On Wed, Aug 12, 2020 at 6:42 AM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126 >
>
> Good morning John
> I ran the tests yesterday afternoon that I mentioned in my previous
email.
> Running for just the CNT output and using the Cray's ptmp to read
and write
> data had no effect on the run times.  Running for just the CONUS
mask
> initially took about 57 seconds to run with compressed files.
Switching to
> uncompressed files, it took about 38 seconds to run for just the
CONUS mask
> and 8 minutes and 40 seconds for all masks.
> This morning, I tested running with just the CONUS mask again and
used the
> uncompressed files.  For some reason this morning, grid_stat runs in
about
> 5-6 seconds, which is what I was seeing a few weeks ago.  I made no
other
> changes to my scripts or modules other than to limit the runs to
just the
> CONUS mask again.  The start and stop times from my logs are below.
> I'm really at a loss as to what could be causing these fluctuations
in run
> times.
> Thanks
> John
>
>
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_ndfdconc2p519110112ttf006
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/ndfd/co/dbtx/1911/01/18/tt
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_ndfd
>
> *GRID STAT STARTWed Aug 12 12:17:20 UTC 2020*
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744072918741855
>
> *GRID STAT STOPWed Aug 12 12:17:26 UTC 2020*
> missedTracking not set.  Running grid_stat for blend 201911017
>
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_blendconc2p519110107ttf011
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/blend/co/dbtx/1911/01/18/tt
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_blend
>
> *GRID STAT STARTWed Aug 12 12:17:26 UTC 2020*
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744073019405151
>
> *GRID STAT STOPWed Aug 12 12:17:31 UTC 2020*
> missedTracking not set.  Running grid_stat for blendp 201911017
>
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_blendpconc2p519110107ttf011
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/blendp/co/dbtx/1911/01/18/tt
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_blendp
>
> *GRID STAT STARTWed Aug 12 12:17:31 UTC 2020*
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744073103291231
>
> *GRID STAT STOPWed Aug 12 12:17:36 UTC 2020*
> missedTracking not set.  Running grid_stat for gmos 201911010
>
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_gmosconc2p519110100ttf018
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/gmos/co/dbtx/1911/01/18/tt
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_gmos
>
> *GRID STAT STARTWed Aug 12 12:17:37 UTC 2020*
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744073203954527
>
> *GRID STAT STOPWed Aug 12 12:17:42 UTC 2020*
>
> On Tue, Aug 11, 2020 at 2:03 PM John L Wagner - NOAA Federal <
> john.l.wagner at noaa.gov> wrote:
>
> > Thanks John.  Attached are the NDFD and URMA grib2 files that we
are
> > starting with and the grid_stat config file we are using.  The
> > regrid_data_plane command we are using is
> >
> > regrid_data_plane <grib2 file> <CONUS mask> <output netCDF file>
-method
> > NEAREST -width 1 -v 0 -name "tt" -compress 9 -field "name=tt;
level=Z2;"
> >
> > I have access to the prod machine (venus) so I have been running
some
> > tests today with grid_stat.  I've run tests with V8.1 out of NCO's
area
> and
> > with V8.1, V9.0.1, V9.0.2, and V9.0.3 in Julie's area.  The
shortest and
> > longest run times I've seen today were both with Julie's V8.1.
This
> > morning, it ran in about 5 minutes.  At lunch time, it ran in
about 40
> > minutes.  All other runs took about 10 minutes.
> >
> > Tests I am going to run this afternoon:
> > 1.  We are currently running with masks for all of the CONUS WFOs
and
> > regions (about 135 masks).  I'm going to test with just the CONUS
mask.
> > 2.  I'll need to look at what elements we use this
grid_stat_config file
> > for.  For temperature, we only need the cnt file.  I'll turn off
the
> other
> > files.
> > 3.  I am going to try on the Cray instead of the Dell.  We've run
into
> > several problems where the only logical problem is the read and
write
> > speeds on the Dell (phase 3).  If your tests on the project
machine at
> NCAR
> > don't show any issues, then this will continue to be what I think
the
> real
> > problem is.
> >
> > If you need anything else from me, let me know.
> > Thanks
> > John
> >
> > On Tue, Aug 11, 2020 at 11:57 AM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> John,
> >>
> >> I did see your email come through met-help last week. Howard is
the
> person
> >> who's worked the most on NetCDF compression, so I was glad he was
able
> to
> >> respond. Unfortunately, I don't have any quick and obvious
answers for
> >> you.
> >> We just posted the met-9.1 release yesterday. It sounds like
you're
> >> weighing 2 main considerations... the output file size of
> >> regrid_data_plane
> >> and the runtime of grid_stat.
> >>
> >> Seems like we should do some comparison tests between met-8.1 and
> met-9.1.
> >> What I'd like to figure out is whether the longer runtimes are
due
> solely
> >> to the difference in NetCDF compression levels. Howard has told
me that
> >> writing highly compressed NetCDF files is much, much slower.
> >>
> >> We could pick a single case and run it through
> regrid_data_plane/grid_stat
> >> 6 times... with no/medium/high compression and met-8.1/9.1. Then
for
> each
> >> configuration, note the runtime and file sizes. And I also want
to
> confirm
> >> that the output generated is the same (i.e. met-9.1 isn't writing
any
> >> extra
> >> output variables). WCOSS is down this week, but I could do this
testing
> on
> >> a project machine at NCAR. But I'd like to approximate the
datasets
> you're
> >> using on WCOSS.
> >>
> >> Can you point me to sample data files, the regrid_data_plane
command
> >> you're
> >> running, and the Grid-Stat configuration files you're using?
> >>
> >> Thanks,
> >> John
> >>
> >> On Mon, Aug 10, 2020 at 10:11 AM John L Wagner - NOAA Federal via
RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126 >
> >> >
> >> > Thanks Julie.  I did submit this to the help desk last week.
That is
> >> where
> >> > Howard mentioned checking the netcdf library.  I'll submit
again to
> get
> >> > John's thoughts.
> >> > John, please see my email below.  If you need any more
information
> >> please
> >> > let me know.
> >> > Thanks
> >> > John
> >> >
> >> > On Mon, Aug 10, 2020 at 12:05 PM Julie Prestopnik
<jpresto at ucar.edu>
> >> > wrote:
> >> >
> >> > > Hi John.
> >> > >
> >> > > I am not the best person to address these issues about
> compression.  I
> >> > can
> >> > > confirm that I started linking to v4.5.0 of the NetCDF
library for
> the
> >> > > reason you indicated - to use the standard WCOSS libraries,
as that
> is
> >> > how
> >> > > it will be done for the operational installation of MET.
> >> > >
> >> > > Could you please submit your questions about compression to
> >> > > met_help at ucar.edu so that your questions can be addressed by
the
> >> right
> >> > > person (likely John, who also happens to be on helpdesk
today)?
> >> > >
> >> > > Thanks!
> >> > >
> >> > > Julie
> >> > >
> >> > > On Mon, Aug 10, 2020 at 8:12 AM John L Wagner - NOAA Federal
<
> >> > > john.l.wagner at noaa.gov> wrote:
> >> > >
> >> > >> Good morning Julie
> >> > >> Over the past few weeks, we've been seeing less compression
with
> >> > >> regrid_data_plane and significantly longer run times with
> grid_stat.
> >> > For
> >> > >> regrid_data_plane, our CONUS files used to compress from
~44MB to
> >> ~23MB.
> >> > >> Now they only compress to ~30MB.  For grid_stat, a single
model and
> >> > >> analysis grid would run in a minute or two.  It now takes 8-
10
> >> minutes
> >> > to
> >> > >> run.  We have been testing mainly with MET V9.0.2 and V9.0.3
out of
> >> your
> >> > >> directory on mars.
> >> > >> Howard mentioned that the compression is controlled by the
netcdf
> >> > >> library.  Starting with V9.0.1, you started linking to
V4.5.0.
> >> Prior to
> >> > >> that, it looks like you had your own version of the library.
I
> >> assume
> >> > >> the change was made just to use the standard WCOSS
libraries.  I
> was
> >> > just
> >> > >> wondering if anyone else had experienced the same issues
that we
> have
> >> > after
> >> > >> this change?
> >> > >> I ran some tests over the weekend reverting back to MET
V8.1.  I
> >> didn't
> >> > >> get in early enough this morning to check the file sizes of
the
> >> > >> regrid_data_plane output, but the grid_stat run times were
still
> >> longer
> >> > >> than expected.  I suspect the netcdf library is not the
issue
> >> there.  If
> >> > >> you know of any issues that might be causing longer
grid_stat run
> >> times
> >> > or
> >> > >> have any suggestions for other tests we can try, please let
me
> know.
> >> > >> Thanks
> >> > >> John
> >> > >>
> >> > >> ---------- Forwarded message ---------
> >> > >> From: Howard Soh via RT <met_help at ucar.edu>
> >> > >> Date: Thu, Aug 6, 2020 at 12:29 AM
> >> > >> Subject: [rt.rap.ucar.edu #96126] Compression and grid_stat
run
> time
> >> > >> To: <john.l.wagner at noaa.gov>
> >> > >> Cc: <dana.strom at noaa.gov>, <tamarah.curtis at noaa.gov>
> >> > >>
> >> > >>
> >> > >> The compression is done by NetCDF library. The NetCDF output
(size
> >> > >> difference) is caused by the NetCDF library, not by MET
code.
> Please
> >> > check
> >> > >> the NetCDF library version:
> >> > >>    ldd regrid_data_plane | grep netcdf
> >> > >>
> >> > >> The issue 1363 (applied 9.0.3 and 9.1 beta 2) improves the
speed of
> >> > >> reading NetCDF input, and no update on writing NetCDF
output.
> >> > >>
> >> > >> Please let us know if 9.0.3 runs slower than 9.0.2 with
using the
> >> same
> >> > >> NetCDF library.
> >> > >>
> >> > >> Cheers,
> >> > >> Howard
> >> > >>
> >> > >> Here are examples to check NetCDF library:
> >> > >> ldd met-8.1/bin/regrid_data_plane | grep netcdf
> >> > >>         libnetcdf_c++4.so.1 =>
> >> /usr/local/netcdf/lib/libnetcdf_c++4.so.1
> >> > >> (0x00007fee87e70000)
> >> > >>         libnetcdf.so.7 => not found
> >> > >>         libnetcdf.so.13 =>
/usr/local/netcdf/lib/libnetcdf.so.13
> >> > >> (0x00007fee83f05000)
> >> > >>
> >> > >> ldd met-9.0.2/bin/regrid_data_plane | grep netcdf
> >> > >>         libnetcdf_c++4.so.1 =>
> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
> >> > >> (0x00007fdf5d1b2000)
> >> > >>         libnetcdf.so.15 =>
> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
> >> > (0x00007fdf5ce31000)
> >> > >>
> >> > >>
> >> > >> ldd met-9.0.3/bin/regrid_data_plane | grep netcdf
> >> > >>         libnetcdf_c++4.so.1 =>
> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
> >> > >> (0x00007f7dfe91d000)
> >> > >>         libnetcdf.so.15 =>
> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
> >> > (0x00007f7dfe59c000)
> >> > >>
> >> > >>
> >> > >> On Wed Aug 05 08:36:09 2020, mccabe wrote:
> >> > >> > Hi John,
> >> > >> >
> >> > >> > I see you have questions about compression in regards to
run time
> >> > >> > length and file sizes. I have assigned this ticket to
Howard Soh.
> >> > >> > Please allow a few business days for a response.
> >> > >> >
> >> > >> > Thanks,
> >> > >> > George
> >> > >>
> >> > >>
> >> > >>
> >> > >>
> >> > >>
> >> > >> --
> >> > >> John Wagner
> >> > >> Verification Task Lead
> >> > >> NOAA/National Weather Service
> >> > >> Meteorological Development Laboratory
> >> > >> Digital Forecast Services Division
> >> > >> SSMC2 Room 10106
> >> > >> Silver Spring, MD 20910
> >> > >> (301) 427-9471 (office)
> >> > >> (908) 902-4155 (cell/text)
> >> > >>
> >> > >
> >> > >
> >> > > --
> >> > > Julie Prestopnik (she/her/hers)
> >> > > Software Engineer
> >> > > National Center for Atmospheric Research
> >> > > Research Applications Laboratory
> >> > > Phone: 303.497.8399
> >> > > Email: jpresto at ucar.edu
> >> > >
> >> > > My working day may not be your working day.  Please do not
feel
> >> obliged
> >> > to
> >> > > reply to this email outside of your normal working hours.
> >> > >
> >> >
> >> >
> >> > --
> >> > John Wagner
> >> > Verification Task Lead
> >> > NOAA/National Weather Service
> >> > Meteorological Development Laboratory
> >> > Digital Forecast Services Division
> >> > SSMC2 Room 10106
> >> > Silver Spring, MD 20910
> >> > (301) 427-9471 (office)
> >> > (908) 902-4155 (cell/text)
> >> >
> >> >
> >>
> >>
> >
> > --
> > John Wagner
> > Verification Task Lead
> > NOAA/National Weather Service
> > Meteorological Development Laboratory
> > Digital Forecast Services Division
> > SSMC2 Room 10106
> > Silver Spring, MD 20910
> > (301) 427-9471 (office)
> > (908) 902-4155 (cell/text)
> >
>
>
> --
> John Wagner
> Verification Task Lead
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Division
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>

------------------------------------------------
Subject: Compression and grid_stat run time
From: John Halley Gotway
Time: Thu Aug 27 14:04:02 2020

John,

I'm going to go ahead and resolve this old met-help ticket. As we
discussed
at the NOAA METplus telecon earlier this week, you're going to try
switching to using GRIB input files and check to see what impact that
has
on runtimes.

Thanks,
John HG

On Thu, Aug 13, 2020 at 11:53 AM John Halley Gotway <johnhg at ucar.edu>
wrote:

> John,
>
> Thanks for sending this data. Unfortunately, I didn't get a chance
to dig
> into the details before heading out of town. But I'll be back from
PTO on
> Monday.
>
> Thanks,
> John
>
> On Wed, Aug 12, 2020 at 6:42 AM John L Wagner - NOAA Federal via RT
<
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126 >
>>
>> Good morning John
>> I ran the tests yesterday afternoon that I mentioned in my previous
email.
>> Running for just the CNT output and using the Cray's ptmp to read
and
>> write
>> data had no effect on the run times.  Running for just the CONUS
mask
>> initially took about 57 seconds to run with compressed files.
Switching
>> to
>> uncompressed files, it took about 38 seconds to run for just the
CONUS
>> mask
>> and 8 minutes and 40 seconds for all masks.
>> This morning, I tested running with just the CONUS mask again and
used the
>> uncompressed files.  For some reason this morning, grid_stat runs
in about
>> 5-6 seconds, which is what I was seeing a few weeks ago.  I made no
other
>> changes to my scripts or modules other than to limit the runs to
just the
>> CONUS mask again.  The start and stop times from my logs are below.
>> I'm really at a loss as to what could be causing these fluctuations
in run
>> times.
>> Thanks
>> John
>>
>>
>>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_ndfdconc2p519110112ttf006
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
>> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/ndfd/co/dbtx/1911/01/18/tt
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_ndfd
>>
>> *GRID STAT STARTWed Aug 12 12:17:20 UTC 2020*
>> GSL_RNG_TYPE=mt19937
>> GSL_RNG_SEED=18446744072918741855
>>
>> *GRID STAT STOPWed Aug 12 12:17:26 UTC 2020*
>> missedTracking not set.  Running grid_stat for blend 201911017
>>
>>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_blendconc2p519110107ttf011
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
>> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/blend/co/dbtx/1911/01/18/tt
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_blend
>>
>> *GRID STAT STARTWed Aug 12 12:17:26 UTC 2020*
>> GSL_RNG_TYPE=mt19937
>> GSL_RNG_SEED=18446744073019405151
>>
>> *GRID STAT STOPWed Aug 12 12:17:31 UTC 2020*
>> missedTracking not set.  Running grid_stat for blendp 201911017
>>
>>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_blendpconc2p519110107ttf011
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
>> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/blendp/co/dbtx/1911/01/18/tt
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_blendp
>>
>> *GRID STAT STARTWed Aug 12 12:17:31 UTC 2020*
>> GSL_RNG_TYPE=mt19937
>> GSL_RNG_SEED=18446744073103291231
>>
>> *GRID STAT STOPWed Aug 12 12:17:36 UTC 2020*
>> missedTracking not set.  Running grid_stat for gmos 201911010
>>
>>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_gmosconc2p519110100ttf018
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
>> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/gmos/co/dbtx/1911/01/18/tt
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_gmos
>>
>> *GRID STAT STARTWed Aug 12 12:17:37 UTC 2020*
>> GSL_RNG_TYPE=mt19937
>> GSL_RNG_SEED=18446744073203954527
>>
>> *GRID STAT STOPWed Aug 12 12:17:42 UTC 2020*
>>
>> On Tue, Aug 11, 2020 at 2:03 PM John L Wagner - NOAA Federal <
>> john.l.wagner at noaa.gov> wrote:
>>
>> > Thanks John.  Attached are the NDFD and URMA grib2 files that we
are
>> > starting with and the grid_stat config file we are using.  The
>> > regrid_data_plane command we are using is
>> >
>> > regrid_data_plane <grib2 file> <CONUS mask> <output netCDF file>
-method
>> > NEAREST -width 1 -v 0 -name "tt" -compress 9 -field "name=tt;
level=Z2;"
>> >
>> > I have access to the prod machine (venus) so I have been running
some
>> > tests today with grid_stat.  I've run tests with V8.1 out of
NCO's area
>> and
>> > with V8.1, V9.0.1, V9.0.2, and V9.0.3 in Julie's area.  The
shortest and
>> > longest run times I've seen today were both with Julie's V8.1.
This
>> > morning, it ran in about 5 minutes.  At lunch time, it ran in
about 40
>> > minutes.  All other runs took about 10 minutes.
>> >
>> > Tests I am going to run this afternoon:
>> > 1.  We are currently running with masks for all of the CONUS WFOs
and
>> > regions (about 135 masks).  I'm going to test with just the CONUS
mask.
>> > 2.  I'll need to look at what elements we use this
grid_stat_config file
>> > for.  For temperature, we only need the cnt file.  I'll turn off
the
>> other
>> > files.
>> > 3.  I am going to try on the Cray instead of the Dell.  We've run
into
>> > several problems where the only logical problem is the read and
write
>> > speeds on the Dell (phase 3).  If your tests on the project
machine at
>> NCAR
>> > don't show any issues, then this will continue to be what I think
the
>> real
>> > problem is.
>> >
>> > If you need anything else from me, let me know.
>> > Thanks
>> > John
>> >
>> > On Tue, Aug 11, 2020 at 11:57 AM John Halley Gotway via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> >> John,
>> >>
>> >> I did see your email come through met-help last week. Howard is
the
>> person
>> >> who's worked the most on NetCDF compression, so I was glad he
was able
>> to
>> >> respond. Unfortunately, I don't have any quick and obvious
answers for
>> >> you.
>> >> We just posted the met-9.1 release yesterday. It sounds like
you're
>> >> weighing 2 main considerations... the output file size of
>> >> regrid_data_plane
>> >> and the runtime of grid_stat.
>> >>
>> >> Seems like we should do some comparison tests between met-8.1
and
>> met-9.1.
>> >> What I'd like to figure out is whether the longer runtimes are
due
>> solely
>> >> to the difference in NetCDF compression levels. Howard has told
me that
>> >> writing highly compressed NetCDF files is much, much slower.
>> >>
>> >> We could pick a single case and run it through
>> regrid_data_plane/grid_stat
>> >> 6 times... with no/medium/high compression and met-8.1/9.1. Then
for
>> each
>> >> configuration, note the runtime and file sizes. And I also want
to
>> confirm
>> >> that the output generated is the same (i.e. met-9.1 isn't
writing any
>> >> extra
>> >> output variables). WCOSS is down this week, but I could do this
>> testing on
>> >> a project machine at NCAR. But I'd like to approximate the
datasets
>> you're
>> >> using on WCOSS.
>> >>
>> >> Can you point me to sample data files, the regrid_data_plane
command
>> >> you're
>> >> running, and the Grid-Stat configuration files you're using?
>> >>
>> >> Thanks,
>> >> John
>> >>
>> >> On Mon, Aug 10, 2020 at 10:11 AM John L Wagner - NOAA Federal
via RT <
>> >> met_help at ucar.edu> wrote:
>> >>
>> >> >
>> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126
>
>> >> >
>> >> > Thanks Julie.  I did submit this to the help desk last week.
That is
>> >> where
>> >> > Howard mentioned checking the netcdf library.  I'll submit
again to
>> get
>> >> > John's thoughts.
>> >> > John, please see my email below.  If you need any more
information
>> >> please
>> >> > let me know.
>> >> > Thanks
>> >> > John
>> >> >
>> >> > On Mon, Aug 10, 2020 at 12:05 PM Julie Prestopnik
<jpresto at ucar.edu>
>> >> > wrote:
>> >> >
>> >> > > Hi John.
>> >> > >
>> >> > > I am not the best person to address these issues about
>> compression.  I
>> >> > can
>> >> > > confirm that I started linking to v4.5.0 of the NetCDF
library for
>> the
>> >> > > reason you indicated - to use the standard WCOSS libraries,
as
>> that is
>> >> > how
>> >> > > it will be done for the operational installation of MET.
>> >> > >
>> >> > > Could you please submit your questions about compression to
>> >> > > met_help at ucar.edu so that your questions can be addressed by
the
>> >> right
>> >> > > person (likely John, who also happens to be on helpdesk
today)?
>> >> > >
>> >> > > Thanks!
>> >> > >
>> >> > > Julie
>> >> > >
>> >> > > On Mon, Aug 10, 2020 at 8:12 AM John L Wagner - NOAA Federal
<
>> >> > > john.l.wagner at noaa.gov> wrote:
>> >> > >
>> >> > >> Good morning Julie
>> >> > >> Over the past few weeks, we've been seeing less compression
with
>> >> > >> regrid_data_plane and significantly longer run times with
>> grid_stat.
>> >> > For
>> >> > >> regrid_data_plane, our CONUS files used to compress from
~44MB to
>> >> ~23MB.
>> >> > >> Now they only compress to ~30MB.  For grid_stat, a single
model
>> and
>> >> > >> analysis grid would run in a minute or two.  It now takes
8-10
>> >> minutes
>> >> > to
>> >> > >> run.  We have been testing mainly with MET V9.0.2 and
V9.0.3 out
>> of
>> >> your
>> >> > >> directory on mars.
>> >> > >> Howard mentioned that the compression is controlled by the
netcdf
>> >> > >> library.  Starting with V9.0.1, you started linking to
V4.5.0.
>> >> Prior to
>> >> > >> that, it looks like you had your own version of the
library.  I
>> >> assume
>> >> > >> the change was made just to use the standard WCOSS
libraries.  I
>> was
>> >> > just
>> >> > >> wondering if anyone else had experienced the same issues
that we
>> have
>> >> > after
>> >> > >> this change?
>> >> > >> I ran some tests over the weekend reverting back to MET
V8.1.  I
>> >> didn't
>> >> > >> get in early enough this morning to check the file sizes of
the
>> >> > >> regrid_data_plane output, but the grid_stat run times were
still
>> >> longer
>> >> > >> than expected.  I suspect the netcdf library is not the
issue
>> >> there.  If
>> >> > >> you know of any issues that might be causing longer
grid_stat run
>> >> times
>> >> > or
>> >> > >> have any suggestions for other tests we can try, please let
me
>> know.
>> >> > >> Thanks
>> >> > >> John
>> >> > >>
>> >> > >> ---------- Forwarded message ---------
>> >> > >> From: Howard Soh via RT <met_help at ucar.edu>
>> >> > >> Date: Thu, Aug 6, 2020 at 12:29 AM
>> >> > >> Subject: [rt.rap.ucar.edu #96126] Compression and grid_stat
run
>> time
>> >> > >> To: <john.l.wagner at noaa.gov>
>> >> > >> Cc: <dana.strom at noaa.gov>, <tamarah.curtis at noaa.gov>
>> >> > >>
>> >> > >>
>> >> > >> The compression is done by NetCDF library. The NetCDF
output (size
>> >> > >> difference) is caused by the NetCDF library, not by MET
code.
>> Please
>> >> > check
>> >> > >> the NetCDF library version:
>> >> > >>    ldd regrid_data_plane | grep netcdf
>> >> > >>
>> >> > >> The issue 1363 (applied 9.0.3 and 9.1 beta 2) improves the
speed
>> of
>> >> > >> reading NetCDF input, and no update on writing NetCDF
output.
>> >> > >>
>> >> > >> Please let us know if 9.0.3 runs slower than 9.0.2 with
using the
>> >> same
>> >> > >> NetCDF library.
>> >> > >>
>> >> > >> Cheers,
>> >> > >> Howard
>> >> > >>
>> >> > >> Here are examples to check NetCDF library:
>> >> > >> ldd met-8.1/bin/regrid_data_plane | grep netcdf
>> >> > >>         libnetcdf_c++4.so.1 =>
>> >> /usr/local/netcdf/lib/libnetcdf_c++4.so.1
>> >> > >> (0x00007fee87e70000)
>> >> > >>         libnetcdf.so.7 => not found
>> >> > >>         libnetcdf.so.13 =>
/usr/local/netcdf/lib/libnetcdf.so.13
>> >> > >> (0x00007fee83f05000)
>> >> > >>
>> >> > >> ldd met-9.0.2/bin/regrid_data_plane | grep netcdf
>> >> > >>         libnetcdf_c++4.so.1 =>
>> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
>> >> > >> (0x00007fdf5d1b2000)
>> >> > >>         libnetcdf.so.15 =>
>> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
>> >> > (0x00007fdf5ce31000)
>> >> > >>
>> >> > >>
>> >> > >> ldd met-9.0.3/bin/regrid_data_plane | grep netcdf
>> >> > >>         libnetcdf_c++4.so.1 =>
>> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
>> >> > >> (0x00007f7dfe91d000)
>> >> > >>         libnetcdf.so.15 =>
>> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
>> >> > (0x00007f7dfe59c000)
>> >> > >>
>> >> > >>
>> >> > >> On Wed Aug 05 08:36:09 2020, mccabe wrote:
>> >> > >> > Hi John,
>> >> > >> >
>> >> > >> > I see you have questions about compression in regards to
run
>> time
>> >> > >> > length and file sizes. I have assigned this ticket to
Howard
>> Soh.
>> >> > >> > Please allow a few business days for a response.
>> >> > >> >
>> >> > >> > Thanks,
>> >> > >> > George
>> >> > >>
>> >> > >>
>> >> > >>
>> >> > >>
>> >> > >>
>> >> > >> --
>> >> > >> John Wagner
>> >> > >> Verification Task Lead
>> >> > >> NOAA/National Weather Service
>> >> > >> Meteorological Development Laboratory
>> >> > >> Digital Forecast Services Division
>> >> > >> SSMC2 Room 10106
>> >> > >> Silver Spring, MD 20910
>> >> > >> (301) 427-9471 (office)
>> >> > >> (908) 902-4155 (cell/text)
>> >> > >>
>> >> > >
>> >> > >
>> >> > > --
>> >> > > Julie Prestopnik (she/her/hers)
>> >> > > Software Engineer
>> >> > > National Center for Atmospheric Research
>> >> > > Research Applications Laboratory
>> >> > > Phone: 303.497.8399
>> >> > > Email: jpresto at ucar.edu
>> >> > >
>> >> > > My working day may not be your working day.  Please do not
feel
>> >> obliged
>> >> > to
>> >> > > reply to this email outside of your normal working hours.
>> >> > >
>> >> >
>> >> >
>> >> > --
>> >> > John Wagner
>> >> > Verification Task Lead
>> >> > NOAA/National Weather Service
>> >> > Meteorological Development Laboratory
>> >> > Digital Forecast Services Division
>> >> > SSMC2 Room 10106
>> >> > Silver Spring, MD 20910
>> >> > (301) 427-9471 (office)
>> >> > (908) 902-4155 (cell/text)
>> >> >
>> >> >
>> >>
>> >>
>> >
>> > --
>> > John Wagner
>> > Verification Task Lead
>> > NOAA/National Weather Service
>> > Meteorological Development Laboratory
>> > Digital Forecast Services Division
>> > SSMC2 Room 10106
>> > Silver Spring, MD 20910
>> > (301) 427-9471 (office)
>> > (908) 902-4155 (cell/text)
>> >
>>
>>
>> --
>> John Wagner
>> Verification Task Lead
>> NOAA/National Weather Service
>> Meteorological Development Laboratory
>> Digital Forecast Services Division
>> SSMC2 Room 10106
>> Silver Spring, MD 20910
>> (301) 427-9471 (office)
>> (908) 902-4155 (cell/text)
>>
>>

------------------------------------------------
Subject: Compression and grid_stat run time
From: John L Wagner - NOAA Federal
Time: Thu Aug 27 14:31:38 2020

Thanks John.  I noted earlier that grid_stat run times were as long as
25
minutes.  We have more files on disk now, so I think I/O is the real
issue.  We'll know for sure on Monday, after the prod switch when we
have
to start from scratch essentially.
Switching to grib2 files will give us smaller file sizes and fewer
files
that we need to keep. Hoping that it alleviates the I/O issues.

On Thu, Aug 27, 2020 at 4:07 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:

> John,
>
> I'm going to go ahead and resolve this old met-help ticket. As we
discussed
> at the NOAA METplus telecon earlier this week, you're going to try
> switching to using GRIB input files and check to see what impact
that has
> on runtimes.
>
> Thanks,
> John HG
>
> On Thu, Aug 13, 2020 at 11:53 AM John Halley Gotway
<johnhg at ucar.edu>
> wrote:
>
> > John,
> >
> > Thanks for sending this data. Unfortunately, I didn't get a chance
to dig
> > into the details before heading out of town. But I'll be back from
PTO on
> > Monday.
> >
> > Thanks,
> > John
> >
> > On Wed, Aug 12, 2020 at 6:42 AM John L Wagner - NOAA Federal via
RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126 >
> >>
> >> Good morning John
> >> I ran the tests yesterday afternoon that I mentioned in my
previous
> email.
> >> Running for just the CNT output and using the Cray's ptmp to read
and
> >> write
> >> data had no effect on the run times.  Running for just the CONUS
mask
> >> initially took about 57 seconds to run with compressed files.
Switching
> >> to
> >> uncompressed files, it took about 38 seconds to run for just the
CONUS
> >> mask
> >> and 8 minutes and 40 seconds for all masks.
> >> This morning, I tested running with just the CONUS mask again and
used
> the
> >> uncompressed files.  For some reason this morning, grid_stat runs
in
> about
> >> 5-6 seconds, which is what I was seeing a few weeks ago.  I made
no
> other
> >> changes to my scripts or modules other than to limit the runs to
just
> the
> >> CONUS mask again.  The start and stop times from my logs are
below.
> >> I'm really at a loss as to what could be causing these
fluctuations in
> run
> >> times.
> >> Thanks
> >> John
> >>
> >>
> >>
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_ndfdconc2p519110112ttf006
> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
> >> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/ndfd/co/dbtx/1911/01/18/tt
> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_ndfd
> >>
> >> *GRID STAT STARTWed Aug 12 12:17:20 UTC 2020*
> >> GSL_RNG_TYPE=mt19937
> >> GSL_RNG_SEED=18446744072918741855
> >>
> >> *GRID STAT STOPWed Aug 12 12:17:26 UTC 2020*
> >> missedTracking not set.  Running grid_stat for blend 201911017
> >>
> >>
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_blendconc2p519110107ttf011
> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
> >> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/blend/co/dbtx/1911/01/18/tt
> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_blend
> >>
> >> *GRID STAT STARTWed Aug 12 12:17:26 UTC 2020*
> >> GSL_RNG_TYPE=mt19937
> >> GSL_RNG_SEED=18446744073019405151
> >>
> >> *GRID STAT STOPWed Aug 12 12:17:31 UTC 2020*
> >> missedTracking not set.  Running grid_stat for blendp 201911017
> >>
> >>
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_blendpconc2p519110107ttf011
> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
> >> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/blendp/co/dbtx/1911/01/18/tt
> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_blendp
> >>
> >> *GRID STAT STARTWed Aug 12 12:17:31 UTC 2020*
> >> GSL_RNG_TYPE=mt19937
> >> GSL_RNG_SEED=18446744073103291231
> >>
> >> *GRID STAT STOPWed Aug 12 12:17:36 UTC 2020*
> >> missedTracking not set.  Running grid_stat for gmos 201911010
> >>
> >>
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_gmosconc2p519110100ttf018
> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
> >> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/gmos/co/dbtx/1911/01/18/tt
> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_gmos
> >>
> >> *GRID STAT STARTWed Aug 12 12:17:37 UTC 2020*
> >> GSL_RNG_TYPE=mt19937
> >> GSL_RNG_SEED=18446744073203954527
> >>
> >> *GRID STAT STOPWed Aug 12 12:17:42 UTC 2020*
> >>
> >> On Tue, Aug 11, 2020 at 2:03 PM John L Wagner - NOAA Federal <
> >> john.l.wagner at noaa.gov> wrote:
> >>
> >> > Thanks John.  Attached are the NDFD and URMA grib2 files that
we are
> >> > starting with and the grid_stat config file we are using.  The
> >> > regrid_data_plane command we are using is
> >> >
> >> > regrid_data_plane <grib2 file> <CONUS mask> <output netCDF
file>
> -method
> >> > NEAREST -width 1 -v 0 -name "tt" -compress 9 -field "name=tt;
> level=Z2;"
> >> >
> >> > I have access to the prod machine (venus) so I have been
running some
> >> > tests today with grid_stat.  I've run tests with V8.1 out of
NCO's
> area
> >> and
> >> > with V8.1, V9.0.1, V9.0.2, and V9.0.3 in Julie's area.  The
shortest
> and
> >> > longest run times I've seen today were both with Julie's V8.1.
This
> >> > morning, it ran in about 5 minutes.  At lunch time, it ran in
about 40
> >> > minutes.  All other runs took about 10 minutes.
> >> >
> >> > Tests I am going to run this afternoon:
> >> > 1.  We are currently running with masks for all of the CONUS
WFOs and
> >> > regions (about 135 masks).  I'm going to test with just the
CONUS
> mask.
> >> > 2.  I'll need to look at what elements we use this
grid_stat_config
> file
> >> > for.  For temperature, we only need the cnt file.  I'll turn
off the
> >> other
> >> > files.
> >> > 3.  I am going to try on the Cray instead of the Dell.  We've
run into
> >> > several problems where the only logical problem is the read and
write
> >> > speeds on the Dell (phase 3).  If your tests on the project
machine at
> >> NCAR
> >> > don't show any issues, then this will continue to be what I
think the
> >> real
> >> > problem is.
> >> >
> >> > If you need anything else from me, let me know.
> >> > Thanks
> >> > John
> >> >
> >> > On Tue, Aug 11, 2020 at 11:57 AM John Halley Gotway via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> >> John,
> >> >>
> >> >> I did see your email come through met-help last week. Howard
is the
> >> person
> >> >> who's worked the most on NetCDF compression, so I was glad he
was
> able
> >> to
> >> >> respond. Unfortunately, I don't have any quick and obvious
answers
> for
> >> >> you.
> >> >> We just posted the met-9.1 release yesterday. It sounds like
you're
> >> >> weighing 2 main considerations... the output file size of
> >> >> regrid_data_plane
> >> >> and the runtime of grid_stat.
> >> >>
> >> >> Seems like we should do some comparison tests between met-8.1
and
> >> met-9.1.
> >> >> What I'd like to figure out is whether the longer runtimes are
due
> >> solely
> >> >> to the difference in NetCDF compression levels. Howard has
told me
> that
> >> >> writing highly compressed NetCDF files is much, much slower.
> >> >>
> >> >> We could pick a single case and run it through
> >> regrid_data_plane/grid_stat
> >> >> 6 times... with no/medium/high compression and met-8.1/9.1.
Then for
> >> each
> >> >> configuration, note the runtime and file sizes. And I also
want to
> >> confirm
> >> >> that the output generated is the same (i.e. met-9.1 isn't
writing any
> >> >> extra
> >> >> output variables). WCOSS is down this week, but I could do
this
> >> testing on
> >> >> a project machine at NCAR. But I'd like to approximate the
datasets
> >> you're
> >> >> using on WCOSS.
> >> >>
> >> >> Can you point me to sample data files, the regrid_data_plane
command
> >> >> you're
> >> >> running, and the Grid-Stat configuration files you're using?
> >> >>
> >> >> Thanks,
> >> >> John
> >> >>
> >> >> On Mon, Aug 10, 2020 at 10:11 AM John L Wagner - NOAA Federal
via RT
> <
> >> >> met_help at ucar.edu> wrote:
> >> >>
> >> >> >
> >> >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126 >
> >> >> >
> >> >> > Thanks Julie.  I did submit this to the help desk last week.
That
> is
> >> >> where
> >> >> > Howard mentioned checking the netcdf library.  I'll submit
again to
> >> get
> >> >> > John's thoughts.
> >> >> > John, please see my email below.  If you need any more
information
> >> >> please
> >> >> > let me know.
> >> >> > Thanks
> >> >> > John
> >> >> >
> >> >> > On Mon, Aug 10, 2020 at 12:05 PM Julie Prestopnik <
> jpresto at ucar.edu>
> >> >> > wrote:
> >> >> >
> >> >> > > Hi John.
> >> >> > >
> >> >> > > I am not the best person to address these issues about
> >> compression.  I
> >> >> > can
> >> >> > > confirm that I started linking to v4.5.0 of the NetCDF
library
> for
> >> the
> >> >> > > reason you indicated - to use the standard WCOSS
libraries, as
> >> that is
> >> >> > how
> >> >> > > it will be done for the operational installation of MET.
> >> >> > >
> >> >> > > Could you please submit your questions about compression
to
> >> >> > > met_help at ucar.edu so that your questions can be addressed
by the
> >> >> right
> >> >> > > person (likely John, who also happens to be on helpdesk
today)?
> >> >> > >
> >> >> > > Thanks!
> >> >> > >
> >> >> > > Julie
> >> >> > >
> >> >> > > On Mon, Aug 10, 2020 at 8:12 AM John L Wagner - NOAA
Federal <
> >> >> > > john.l.wagner at noaa.gov> wrote:
> >> >> > >
> >> >> > >> Good morning Julie
> >> >> > >> Over the past few weeks, we've been seeing less
compression with
> >> >> > >> regrid_data_plane and significantly longer run times with
> >> grid_stat.
> >> >> > For
> >> >> > >> regrid_data_plane, our CONUS files used to compress from
~44MB
> to
> >> >> ~23MB.
> >> >> > >> Now they only compress to ~30MB.  For grid_stat, a single
model
> >> and
> >> >> > >> analysis grid would run in a minute or two.  It now takes
8-10
> >> >> minutes
> >> >> > to
> >> >> > >> run.  We have been testing mainly with MET V9.0.2 and
V9.0.3 out
> >> of
> >> >> your
> >> >> > >> directory on mars.
> >> >> > >> Howard mentioned that the compression is controlled by
the
> netcdf
> >> >> > >> library.  Starting with V9.0.1, you started linking to
V4.5.0.
> >> >> Prior to
> >> >> > >> that, it looks like you had your own version of the
library.  I
> >> >> assume
> >> >> > >> the change was made just to use the standard WCOSS
libraries.  I
> >> was
> >> >> > just
> >> >> > >> wondering if anyone else had experienced the same issues
that we
> >> have
> >> >> > after
> >> >> > >> this change?
> >> >> > >> I ran some tests over the weekend reverting back to MET
V8.1.  I
> >> >> didn't
> >> >> > >> get in early enough this morning to check the file sizes
of the
> >> >> > >> regrid_data_plane output, but the grid_stat run times
were still
> >> >> longer
> >> >> > >> than expected.  I suspect the netcdf library is not the
issue
> >> >> there.  If
> >> >> > >> you know of any issues that might be causing longer
grid_stat
> run
> >> >> times
> >> >> > or
> >> >> > >> have any suggestions for other tests we can try, please
let me
> >> know.
> >> >> > >> Thanks
> >> >> > >> John
> >> >> > >>
> >> >> > >> ---------- Forwarded message ---------
> >> >> > >> From: Howard Soh via RT <met_help at ucar.edu>
> >> >> > >> Date: Thu, Aug 6, 2020 at 12:29 AM
> >> >> > >> Subject: [rt.rap.ucar.edu #96126] Compression and
grid_stat run
> >> time
> >> >> > >> To: <john.l.wagner at noaa.gov>
> >> >> > >> Cc: <dana.strom at noaa.gov>, <tamarah.curtis at noaa.gov>
> >> >> > >>
> >> >> > >>
> >> >> > >> The compression is done by NetCDF library. The NetCDF
output
> (size
> >> >> > >> difference) is caused by the NetCDF library, not by MET
code.
> >> Please
> >> >> > check
> >> >> > >> the NetCDF library version:
> >> >> > >>    ldd regrid_data_plane | grep netcdf
> >> >> > >>
> >> >> > >> The issue 1363 (applied 9.0.3 and 9.1 beta 2) improves
the speed
> >> of
> >> >> > >> reading NetCDF input, and no update on writing NetCDF
output.
> >> >> > >>
> >> >> > >> Please let us know if 9.0.3 runs slower than 9.0.2 with
using
> the
> >> >> same
> >> >> > >> NetCDF library.
> >> >> > >>
> >> >> > >> Cheers,
> >> >> > >> Howard
> >> >> > >>
> >> >> > >> Here are examples to check NetCDF library:
> >> >> > >> ldd met-8.1/bin/regrid_data_plane | grep netcdf
> >> >> > >>         libnetcdf_c++4.so.1 =>
> >> >> /usr/local/netcdf/lib/libnetcdf_c++4.so.1
> >> >> > >> (0x00007fee87e70000)
> >> >> > >>         libnetcdf.so.7 => not found
> >> >> > >>         libnetcdf.so.13 =>
/usr/local/netcdf/lib/libnetcdf.so.13
> >> >> > >> (0x00007fee83f05000)
> >> >> > >>
> >> >> > >> ldd met-9.0.2/bin/regrid_data_plane | grep netcdf
> >> >> > >>         libnetcdf_c++4.so.1 =>
> >> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
> >> >> > >> (0x00007fdf5d1b2000)
> >> >> > >>         libnetcdf.so.15 =>
> >> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
> >> >> > (0x00007fdf5ce31000)
> >> >> > >>
> >> >> > >>
> >> >> > >> ldd met-9.0.3/bin/regrid_data_plane | grep netcdf
> >> >> > >>         libnetcdf_c++4.so.1 =>
> >> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
> >> >> > >> (0x00007f7dfe91d000)
> >> >> > >>         libnetcdf.so.15 =>
> >> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
> >> >> > (0x00007f7dfe59c000)
> >> >> > >>
> >> >> > >>
> >> >> > >> On Wed Aug 05 08:36:09 2020, mccabe wrote:
> >> >> > >> > Hi John,
> >> >> > >> >
> >> >> > >> > I see you have questions about compression in regards
to run
> >> time
> >> >> > >> > length and file sizes. I have assigned this ticket to
Howard
> >> Soh.
> >> >> > >> > Please allow a few business days for a response.
> >> >> > >> >
> >> >> > >> > Thanks,
> >> >> > >> > George
> >> >> > >>
> >> >> > >>
> >> >> > >>
> >> >> > >>
> >> >> > >>
> >> >> > >> --
> >> >> > >> John Wagner
> >> >> > >> Verification Task Lead
> >> >> > >> NOAA/National Weather Service
> >> >> > >> Meteorological Development Laboratory
> >> >> > >> Digital Forecast Services Division
> >> >> > >> SSMC2 Room 10106
> >> >> > >> Silver Spring, MD 20910
> >> >> > >> (301) 427-9471 (office)
> >> >> > >> (908) 902-4155 (cell/text)
> >> >> > >>
> >> >> > >
> >> >> > >
> >> >> > > --
> >> >> > > Julie Prestopnik (she/her/hers)
> >> >> > > Software Engineer
> >> >> > > National Center for Atmospheric Research
> >> >> > > Research Applications Laboratory
> >> >> > > Phone: 303.497.8399
> >> >> > > Email: jpresto at ucar.edu
> >> >> > >
> >> >> > > My working day may not be your working day.  Please do not
feel
> >> >> obliged
> >> >> > to
> >> >> > > reply to this email outside of your normal working hours.
> >> >> > >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > John Wagner
> >> >> > Verification Task Lead
> >> >> > NOAA/National Weather Service
> >> >> > Meteorological Development Laboratory
> >> >> > Digital Forecast Services Division
> >> >> > SSMC2 Room 10106
> >> >> > Silver Spring, MD 20910
> >> >> > (301) 427-9471 (office)
> >> >> > (908) 902-4155 (cell/text)
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >
> >> > --
> >> > John Wagner
> >> > Verification Task Lead
> >> > NOAA/National Weather Service
> >> > Meteorological Development Laboratory
> >> > Digital Forecast Services Division
> >> > SSMC2 Room 10106
> >> > Silver Spring, MD 20910
> >> > (301) 427-9471 (office)
> >> > (908) 902-4155 (cell/text)
> >> >
> >>
> >>
> >> --
> >> John Wagner
> >> Verification Task Lead
> >> NOAA/National Weather Service
> >> Meteorological Development Laboratory
> >> Digital Forecast Services Division
> >> SSMC2 Room 10106
> >> Silver Spring, MD 20910
> >> (301) 427-9471 (office)
> >> (908) 902-4155 (cell/text)
> >>
> >>
>
>

--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)

------------------------------------------------
Subject: Compression and grid_stat run time
From: John L Wagner - NOAA Federal
Time: Fri Sep 11 10:15:35 2020

Hey John
I meant to mention that I figured out what was going on last week.
Our
mask that we were using for regrid_data_plane and the subdomain masks
that
we were using in grid_stat were not set up using the same grid
coordinates.  The mask in regrid_data_plane was the NBM grid
(2345x1597
gridpoints) and the grid_stat subdomains (WFOs, regions, RFCs) were
all
from the URMA grid (2145x1377 gridpoints).  This is why
regrid_data_plane
wasn't compressing to the level I expected.  When grid_stat ran, it
was
regridding for each subdomain, which led to the longer run times.
Once I
regridded the subdomains to the NBM grid, the run times went back to
normal
for everything except PoP12.  I'm still looking into that one.
Just wanted you to have the solution in case anyone else ran into
this issue.
Thanks
John

On Thu, Aug 27, 2020 at 4:31 PM John L Wagner - NOAA Federal <
john.l.wagner at noaa.gov> wrote:

> Thanks John.  I noted earlier that grid_stat run times were as long
as 25
> minutes.  We have more files on disk now, so I think I/O is the real
> issue.  We'll know for sure on Monday, after the prod switch when we
have
> to start from scratch essentially.
> Switching to grib2 files will give us smaller file sizes and fewer
files
> that we need to keep. Hoping that it alleviates the I/O issues.
>
> On Thu, Aug 27, 2020 at 4:07 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> John,
>>
>> I'm going to go ahead and resolve this old met-help ticket. As we
>> discussed
>> at the NOAA METplus telecon earlier this week, you're going to try
>> switching to using GRIB input files and check to see what impact
that has
>> on runtimes.
>>
>> Thanks,
>> John HG
>>
>> On Thu, Aug 13, 2020 at 11:53 AM John Halley Gotway
<johnhg at ucar.edu>
>> wrote:
>>
>> > John,
>> >
>> > Thanks for sending this data. Unfortunately, I didn't get a
chance to
>> dig
>> > into the details before heading out of town. But I'll be back
from PTO
>> on
>> > Monday.
>> >
>> > Thanks,
>> > John
>> >
>> > On Wed, Aug 12, 2020 at 6:42 AM John L Wagner - NOAA Federal via
RT <
>> > met_help at ucar.edu> wrote:
>> >
>> >>
>> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126 >
>> >>
>> >> Good morning John
>> >> I ran the tests yesterday afternoon that I mentioned in my
previous
>> email.
>> >> Running for just the CNT output and using the Cray's ptmp to
read and
>> >> write
>> >> data had no effect on the run times.  Running for just the CONUS
mask
>> >> initially took about 57 seconds to run with compressed files.
>> Switching
>> >> to
>> >> uncompressed files, it took about 38 seconds to run for just the
CONUS
>> >> mask
>> >> and 8 minutes and 40 seconds for all masks.
>> >> This morning, I tested running with just the CONUS mask again
and used
>> the
>> >> uncompressed files.  For some reason this morning, grid_stat
runs in
>> about
>> >> 5-6 seconds, which is what I was seeing a few weeks ago.  I made
no
>> other
>> >> changes to my scripts or modules other than to limit the runs to
just
>> the
>> >> CONUS mask again.  The start and stop times from my logs are
below.
>> >> I'm really at a loss as to what could be causing these
fluctuations in
>> run
>> >> times.
>> >> Thanks
>> >> John
>> >>
>> >>
>> >>
>>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
>> >>
>> >>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_ndfdconc2p519110112ttf006
>> >>
>> >>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
>> >> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/ndfd/co/dbtx/1911/01/18/tt
>> >>
>> >>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_ndfd
>> >>
>> >> *GRID STAT STARTWed Aug 12 12:17:20 UTC 2020*
>> >> GSL_RNG_TYPE=mt19937
>> >> GSL_RNG_SEED=18446744072918741855
>> >>
>> >> *GRID STAT STOPWed Aug 12 12:17:26 UTC 2020*
>> >> missedTracking not set.  Running grid_stat for blend 201911017
>> >>
>> >>
>>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
>> >>
>> >>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_blendconc2p519110107ttf011
>> >>
>> >>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
>> >> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/blend/co/dbtx/1911/01/18/tt
>> >>
>> >>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_blend
>> >>
>> >> *GRID STAT STARTWed Aug 12 12:17:26 UTC 2020*
>> >> GSL_RNG_TYPE=mt19937
>> >> GSL_RNG_SEED=18446744073019405151
>> >>
>> >> *GRID STAT STOPWed Aug 12 12:17:31 UTC 2020*
>> >> missedTracking not set.  Running grid_stat for blendp 201911017
>> >>
>> >>
>>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
>> >>
>> >>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_blendpconc2p519110107ttf011
>> >>
>> >>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
>> >> -v 0 -outdir
>> /gpfs/dell2/ptmp/John.L.Wagner/blendp/co/dbtx/1911/01/18/tt
>> >>
>> >>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_blendp
>> >>
>> >> *GRID STAT STARTWed Aug 12 12:17:31 UTC 2020*
>> >> GSL_RNG_TYPE=mt19937
>> >> GSL_RNG_SEED=18446744073103291231
>> >>
>> >> *GRID STAT STOPWed Aug 12 12:17:36 UTC 2020*
>> >> missedTracking not set.  Running grid_stat for gmos 201911010
>> >>
>> >>
>>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
>> >>
>> >>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_gmosconc2p519110100ttf018
>> >>
>> >>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
>> >> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/gmos/co/dbtx/1911/01/18/tt
>> >>
>> >>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_gmos
>> >>
>> >> *GRID STAT STARTWed Aug 12 12:17:37 UTC 2020*
>> >> GSL_RNG_TYPE=mt19937
>> >> GSL_RNG_SEED=18446744073203954527
>> >>
>> >> *GRID STAT STOPWed Aug 12 12:17:42 UTC 2020*
>> >>
>> >> On Tue, Aug 11, 2020 at 2:03 PM John L Wagner - NOAA Federal <
>> >> john.l.wagner at noaa.gov> wrote:
>> >>
>> >> > Thanks John.  Attached are the NDFD and URMA grib2 files that
we are
>> >> > starting with and the grid_stat config file we are using.  The
>> >> > regrid_data_plane command we are using is
>> >> >
>> >> > regrid_data_plane <grib2 file> <CONUS mask> <output netCDF
file>
>> -method
>> >> > NEAREST -width 1 -v 0 -name "tt" -compress 9 -field "name=tt;
>> level=Z2;"
>> >> >
>> >> > I have access to the prod machine (venus) so I have been
running some
>> >> > tests today with grid_stat.  I've run tests with V8.1 out of
NCO's
>> area
>> >> and
>> >> > with V8.1, V9.0.1, V9.0.2, and V9.0.3 in Julie's area.  The
shortest
>> and
>> >> > longest run times I've seen today were both with Julie's V8.1.
This
>> >> > morning, it ran in about 5 minutes.  At lunch time, it ran in
about
>> 40
>> >> > minutes.  All other runs took about 10 minutes.
>> >> >
>> >> > Tests I am going to run this afternoon:
>> >> > 1.  We are currently running with masks for all of the CONUS
WFOs and
>> >> > regions (about 135 masks).  I'm going to test with just the
CONUS
>> mask.
>> >> > 2.  I'll need to look at what elements we use this
grid_stat_config
>> file
>> >> > for.  For temperature, we only need the cnt file.  I'll turn
off the
>> >> other
>> >> > files.
>> >> > 3.  I am going to try on the Cray instead of the Dell.  We've
run
>> into
>> >> > several problems where the only logical problem is the read
and write
>> >> > speeds on the Dell (phase 3).  If your tests on the project
machine
>> at
>> >> NCAR
>> >> > don't show any issues, then this will continue to be what I
think the
>> >> real
>> >> > problem is.
>> >> >
>> >> > If you need anything else from me, let me know.
>> >> > Thanks
>> >> > John
>> >> >
>> >> > On Tue, Aug 11, 2020 at 11:57 AM John Halley Gotway via RT <
>> >> > met_help at ucar.edu> wrote:
>> >> >
>> >> >> John,
>> >> >>
>> >> >> I did see your email come through met-help last week. Howard
is the
>> >> person
>> >> >> who's worked the most on NetCDF compression, so I was glad he
was
>> able
>> >> to
>> >> >> respond. Unfortunately, I don't have any quick and obvious
answers
>> for
>> >> >> you.
>> >> >> We just posted the met-9.1 release yesterday. It sounds like
you're
>> >> >> weighing 2 main considerations... the output file size of
>> >> >> regrid_data_plane
>> >> >> and the runtime of grid_stat.
>> >> >>
>> >> >> Seems like we should do some comparison tests between met-8.1
and
>> >> met-9.1.
>> >> >> What I'd like to figure out is whether the longer runtimes
are due
>> >> solely
>> >> >> to the difference in NetCDF compression levels. Howard has
told me
>> that
>> >> >> writing highly compressed NetCDF files is much, much slower.
>> >> >>
>> >> >> We could pick a single case and run it through
>> >> regrid_data_plane/grid_stat
>> >> >> 6 times... with no/medium/high compression and met-8.1/9.1.
Then for
>> >> each
>> >> >> configuration, note the runtime and file sizes. And I also
want to
>> >> confirm
>> >> >> that the output generated is the same (i.e. met-9.1 isn't
writing
>> any
>> >> >> extra
>> >> >> output variables). WCOSS is down this week, but I could do
this
>> >> testing on
>> >> >> a project machine at NCAR. But I'd like to approximate the
datasets
>> >> you're
>> >> >> using on WCOSS.
>> >> >>
>> >> >> Can you point me to sample data files, the regrid_data_plane
command
>> >> >> you're
>> >> >> running, and the Grid-Stat configuration files you're using?
>> >> >>
>> >> >> Thanks,
>> >> >> John
>> >> >>
>> >> >> On Mon, Aug 10, 2020 at 10:11 AM John L Wagner - NOAA Federal
via
>> RT <
>> >> >> met_help at ucar.edu> wrote:
>> >> >>
>> >> >> >
>> >> >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126 >
>> >> >> >
>> >> >> > Thanks Julie.  I did submit this to the help desk last
week.
>> That is
>> >> >> where
>> >> >> > Howard mentioned checking the netcdf library.  I'll submit
again
>> to
>> >> get
>> >> >> > John's thoughts.
>> >> >> > John, please see my email below.  If you need any more
information
>> >> >> please
>> >> >> > let me know.
>> >> >> > Thanks
>> >> >> > John
>> >> >> >
>> >> >> > On Mon, Aug 10, 2020 at 12:05 PM Julie Prestopnik <
>> jpresto at ucar.edu>
>> >> >> > wrote:
>> >> >> >
>> >> >> > > Hi John.
>> >> >> > >
>> >> >> > > I am not the best person to address these issues about
>> >> compression.  I
>> >> >> > can
>> >> >> > > confirm that I started linking to v4.5.0 of the NetCDF
library
>> for
>> >> the
>> >> >> > > reason you indicated - to use the standard WCOSS
libraries, as
>> >> that is
>> >> >> > how
>> >> >> > > it will be done for the operational installation of MET.
>> >> >> > >
>> >> >> > > Could you please submit your questions about compression
to
>> >> >> > > met_help at ucar.edu so that your questions can be addressed
by
>> the
>> >> >> right
>> >> >> > > person (likely John, who also happens to be on helpdesk
today)?
>> >> >> > >
>> >> >> > > Thanks!
>> >> >> > >
>> >> >> > > Julie
>> >> >> > >
>> >> >> > > On Mon, Aug 10, 2020 at 8:12 AM John L Wagner - NOAA
Federal <
>> >> >> > > john.l.wagner at noaa.gov> wrote:
>> >> >> > >
>> >> >> > >> Good morning Julie
>> >> >> > >> Over the past few weeks, we've been seeing less
compression
>> with
>> >> >> > >> regrid_data_plane and significantly longer run times
with
>> >> grid_stat.
>> >> >> > For
>> >> >> > >> regrid_data_plane, our CONUS files used to compress from
~44MB
>> to
>> >> >> ~23MB.
>> >> >> > >> Now they only compress to ~30MB.  For grid_stat, a
single model
>> >> and
>> >> >> > >> analysis grid would run in a minute or two.  It now
takes 8-10
>> >> >> minutes
>> >> >> > to
>> >> >> > >> run.  We have been testing mainly with MET V9.0.2 and
V9.0.3
>> out
>> >> of
>> >> >> your
>> >> >> > >> directory on mars.
>> >> >> > >> Howard mentioned that the compression is controlled by
the
>> netcdf
>> >> >> > >> library.  Starting with V9.0.1, you started linking to
V4.5.0.
>> >> >> Prior to
>> >> >> > >> that, it looks like you had your own version of the
library.  I
>> >> >> assume
>> >> >> > >> the change was made just to use the standard WCOSS
libraries.
>> I
>> >> was
>> >> >> > just
>> >> >> > >> wondering if anyone else had experienced the same issues
that
>> we
>> >> have
>> >> >> > after
>> >> >> > >> this change?
>> >> >> > >> I ran some tests over the weekend reverting back to MET
V8.1.
>> I
>> >> >> didn't
>> >> >> > >> get in early enough this morning to check the file sizes
of the
>> >> >> > >> regrid_data_plane output, but the grid_stat run times
were
>> still
>> >> >> longer
>> >> >> > >> than expected.  I suspect the netcdf library is not the
issue
>> >> >> there.  If
>> >> >> > >> you know of any issues that might be causing longer
grid_stat
>> run
>> >> >> times
>> >> >> > or
>> >> >> > >> have any suggestions for other tests we can try, please
let me
>> >> know.
>> >> >> > >> Thanks
>> >> >> > >> John
>> >> >> > >>
>> >> >> > >> ---------- Forwarded message ---------
>> >> >> > >> From: Howard Soh via RT <met_help at ucar.edu>
>> >> >> > >> Date: Thu, Aug 6, 2020 at 12:29 AM
>> >> >> > >> Subject: [rt.rap.ucar.edu #96126] Compression and
grid_stat
>> run
>> >> time
>> >> >> > >> To: <john.l.wagner at noaa.gov>
>> >> >> > >> Cc: <dana.strom at noaa.gov>, <tamarah.curtis at noaa.gov>
>> >> >> > >>
>> >> >> > >>
>> >> >> > >> The compression is done by NetCDF library. The NetCDF
output
>> (size
>> >> >> > >> difference) is caused by the NetCDF library, not by MET
code.
>> >> Please
>> >> >> > check
>> >> >> > >> the NetCDF library version:
>> >> >> > >>    ldd regrid_data_plane | grep netcdf
>> >> >> > >>
>> >> >> > >> The issue 1363 (applied 9.0.3 and 9.1 beta 2) improves
the
>> speed
>> >> of
>> >> >> > >> reading NetCDF input, and no update on writing NetCDF
output.
>> >> >> > >>
>> >> >> > >> Please let us know if 9.0.3 runs slower than 9.0.2 with
using
>> the
>> >> >> same
>> >> >> > >> NetCDF library.
>> >> >> > >>
>> >> >> > >> Cheers,
>> >> >> > >> Howard
>> >> >> > >>
>> >> >> > >> Here are examples to check NetCDF library:
>> >> >> > >> ldd met-8.1/bin/regrid_data_plane | grep netcdf
>> >> >> > >>         libnetcdf_c++4.so.1 =>
>> >> >> /usr/local/netcdf/lib/libnetcdf_c++4.so.1
>> >> >> > >> (0x00007fee87e70000)
>> >> >> > >>         libnetcdf.so.7 => not found
>> >> >> > >>         libnetcdf.so.13 =>
>> /usr/local/netcdf/lib/libnetcdf.so.13
>> >> >> > >> (0x00007fee83f05000)
>> >> >> > >>
>> >> >> > >> ldd met-9.0.2/bin/regrid_data_plane | grep netcdf
>> >> >> > >>         libnetcdf_c++4.so.1 =>
>> >> >> > >> /usr/local/netcdf-4.7.0/gcc-
6.3.0/lib/libnetcdf_c++4.so.1
>> >> >> > >> (0x00007fdf5d1b2000)
>> >> >> > >>         libnetcdf.so.15 =>
>> >> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
>> >> >> > (0x00007fdf5ce31000)
>> >> >> > >>
>> >> >> > >>
>> >> >> > >> ldd met-9.0.3/bin/regrid_data_plane | grep netcdf
>> >> >> > >>         libnetcdf_c++4.so.1 =>
>> >> >> > >> /usr/local/netcdf-4.7.0/gcc-
6.3.0/lib/libnetcdf_c++4.so.1
>> >> >> > >> (0x00007f7dfe91d000)
>> >> >> > >>         libnetcdf.so.15 =>
>> >> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
>> >> >> > (0x00007f7dfe59c000)
>> >> >> > >>
>> >> >> > >>
>> >> >> > >> On Wed Aug 05 08:36:09 2020, mccabe wrote:
>> >> >> > >> > Hi John,
>> >> >> > >> >
>> >> >> > >> > I see you have questions about compression in regards
to run
>> >> time
>> >> >> > >> > length and file sizes. I have assigned this ticket to
Howard
>> >> Soh.
>> >> >> > >> > Please allow a few business days for a response.
>> >> >> > >> >
>> >> >> > >> > Thanks,
>> >> >> > >> > George
>> >> >> > >>
>> >> >> > >>
>> >> >> > >>
>> >> >> > >>
>> >> >> > >>
>> >> >> > >> --
>> >> >> > >> John Wagner
>> >> >> > >> Verification Task Lead
>> >> >> > >> NOAA/National Weather Service
>> >> >> > >> Meteorological Development Laboratory
>> >> >> > >> Digital Forecast Services Division
>> >> >> > >> SSMC2 Room 10106
>> >> >> > >> Silver Spring, MD 20910
>> >> >> > >> (301) 427-9471 (office)
>> >> >> > >> (908) 902-4155 (cell/text)
>> >> >> > >>
>> >> >> > >
>> >> >> > >
>> >> >> > > --
>> >> >> > > Julie Prestopnik (she/her/hers)
>> >> >> > > Software Engineer
>> >> >> > > National Center for Atmospheric Research
>> >> >> > > Research Applications Laboratory
>> >> >> > > Phone: 303.497.8399
>> >> >> > > Email: jpresto at ucar.edu
>> >> >> > >
>> >> >> > > My working day may not be your working day.  Please do
not feel
>> >> >> obliged
>> >> >> > to
>> >> >> > > reply to this email outside of your normal working hours.
>> >> >> > >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > John Wagner
>> >> >> > Verification Task Lead
>> >> >> > NOAA/National Weather Service
>> >> >> > Meteorological Development Laboratory
>> >> >> > Digital Forecast Services Division
>> >> >> > SSMC2 Room 10106
>> >> >> > Silver Spring, MD 20910
>> >> >> > (301) 427-9471 (office)
>> >> >> > (908) 902-4155 (cell/text)
>> >> >> >
>> >> >> >
>> >> >>
>> >> >>
>> >> >
>> >> > --
>> >> > John Wagner
>> >> > Verification Task Lead
>> >> > NOAA/National Weather Service
>> >> > Meteorological Development Laboratory
>> >> > Digital Forecast Services Division
>> >> > SSMC2 Room 10106
>> >> > Silver Spring, MD 20910
>> >> > (301) 427-9471 (office)
>> >> > (908) 902-4155 (cell/text)
>> >> >
>> >>
>> >>
>> >> --
>> >> John Wagner
>> >> Verification Task Lead
>> >> NOAA/National Weather Service
>> >> Meteorological Development Laboratory
>> >> Digital Forecast Services Division
>> >> SSMC2 Room 10106
>> >> Silver Spring, MD 20910
>> >> (301) 427-9471 (office)
>> >> (908) 902-4155 (cell/text)
>> >>
>> >>
>>
>>
>
> --
> John Wagner
> Verification Task Lead
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Division
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>


--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)

------------------------------------------------
Subject: Compression and grid_stat run time
From: John Halley Gotway
Time: Fri Sep 11 12:02:52 2020

John,

Great, thanks for the update! You know, in earlier versions of MET,
when
the grid of the masking region did not match the verification domain,
the
tools errored out stating that fact. At the request of EMC, we updated
the
logic to do the automatic regridding using nearest neighbor
interpolation.

But I've always been a bit uncomfortable about this "feature". In my
opinion, it's better to define masking regions separately for each
domain
rather than doing automated regridding. I know that there is a log
message
stating that the regridding is occurring, but I wonder if you have any
other recommendations about this? What sort of functionality/log
messages/warnings/config file options would make the behavior of the
software more transparent?

Thanks,
John

On Fri, Sep 11, 2020 at 10:16 AM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:

>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126 >
>
> Hey John
> I meant to mention that I figured out what was going on last week.
Our
> mask that we were using for regrid_data_plane and the subdomain
masks that
> we were using in grid_stat were not set up using the same grid
> coordinates.  The mask in regrid_data_plane was the NBM grid
(2345x1597
> gridpoints) and the grid_stat subdomains (WFOs, regions, RFCs) were
all
> from the URMA grid (2145x1377 gridpoints).  This is why
regrid_data_plane
> wasn't compressing to the level I expected.  When grid_stat ran, it
was
> regridding for each subdomain, which led to the longer run times.
Once I
> regridded the subdomains to the NBM grid, the run times went back to
normal
> for everything except PoP12.  I'm still looking into that one.
> Just wanted you to have the solution in case anyone else ran into
> this issue.
> Thanks
> John
>
> On Thu, Aug 27, 2020 at 4:31 PM John L Wagner - NOAA Federal <
> john.l.wagner at noaa.gov> wrote:
>
> > Thanks John.  I noted earlier that grid_stat run times were as
long as 25
> > minutes.  We have more files on disk now, so I think I/O is the
real
> > issue.  We'll know for sure on Monday, after the prod switch when
we have
> > to start from scratch essentially.
> > Switching to grib2 files will give us smaller file sizes and fewer
files
> > that we need to keep. Hoping that it alleviates the I/O issues.
> >
> > On Thu, Aug 27, 2020 at 4:07 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> John,
> >>
> >> I'm going to go ahead and resolve this old met-help ticket. As we
> >> discussed
> >> at the NOAA METplus telecon earlier this week, you're going to
try
> >> switching to using GRIB input files and check to see what impact
that
> has
> >> on runtimes.
> >>
> >> Thanks,
> >> John HG
> >>
> >> On Thu, Aug 13, 2020 at 11:53 AM John Halley Gotway
<johnhg at ucar.edu>
> >> wrote:
> >>
> >> > John,
> >> >
> >> > Thanks for sending this data. Unfortunately, I didn't get a
chance to
> >> dig
> >> > into the details before heading out of town. But I'll be back
from PTO
> >> on
> >> > Monday.
> >> >
> >> > Thanks,
> >> > John
> >> >
> >> > On Wed, Aug 12, 2020 at 6:42 AM John L Wagner - NOAA Federal
via RT <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> >>
> >> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126
>
> >> >>
> >> >> Good morning John
> >> >> I ran the tests yesterday afternoon that I mentioned in my
previous
> >> email.
> >> >> Running for just the CNT output and using the Cray's ptmp to
read and
> >> >> write
> >> >> data had no effect on the run times.  Running for just the
CONUS mask
> >> >> initially took about 57 seconds to run with compressed files.
> >> Switching
> >> >> to
> >> >> uncompressed files, it took about 38 seconds to run for just
the
> CONUS
> >> >> mask
> >> >> and 8 minutes and 40 seconds for all masks.
> >> >> This morning, I tested running with just the CONUS mask again
and
> used
> >> the
> >> >> uncompressed files.  For some reason this morning, grid_stat
runs in
> >> about
> >> >> 5-6 seconds, which is what I was seeing a few weeks ago.  I
made no
> >> other
> >> >> changes to my scripts or modules other than to limit the runs
to just
> >> the
> >> >> CONUS mask again.  The start and stop times from my logs are
below.
> >> >> I'm really at a loss as to what could be causing these
fluctuations
> in
> >> run
> >> >> times.
> >> >> Thanks
> >> >> John
> >> >>
> >> >>
> >> >>
> >>
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
> >> >>
> >> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_ndfdconc2p519110112ttf006
> >> >>
> >> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
> >> >> -v 0 -outdir
> /gpfs/dell2/ptmp/John.L.Wagner/ndfd/co/dbtx/1911/01/18/tt
> >> >>
> >> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_ndfd
> >> >>
> >> >> *GRID STAT STARTWed Aug 12 12:17:20 UTC 2020*
> >> >> GSL_RNG_TYPE=mt19937
> >> >> GSL_RNG_SEED=18446744072918741855
> >> >>
> >> >> *GRID STAT STOPWed Aug 12 12:17:26 UTC 2020*
> >> >> missedTracking not set.  Running grid_stat for blend 201911017
> >> >>
> >> >>
> >>
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
> >> >>
> >> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_blendconc2p519110107ttf011
> >> >>
> >> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
> >> >> -v 0 -outdir
> /gpfs/dell2/ptmp/John.L.Wagner/blend/co/dbtx/1911/01/18/tt
> >> >>
> >> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_blend
> >> >>
> >> >> *GRID STAT STARTWed Aug 12 12:17:26 UTC 2020*
> >> >> GSL_RNG_TYPE=mt19937
> >> >> GSL_RNG_SEED=18446744073019405151
> >> >>
> >> >> *GRID STAT STOPWed Aug 12 12:17:31 UTC 2020*
> >> >> missedTracking not set.  Running grid_stat for blendp
201911017
> >> >>
> >> >>
> >>
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
> >> >>
> >> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_blendpconc2p519110107ttf011
> >> >>
> >> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
> >> >> -v 0 -outdir
> >> /gpfs/dell2/ptmp/John.L.Wagner/blendp/co/dbtx/1911/01/18/tt
> >> >>
> >> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_blendp
> >> >>
> >> >> *GRID STAT STARTWed Aug 12 12:17:31 UTC 2020*
> >> >> GSL_RNG_TYPE=mt19937
> >> >> GSL_RNG_SEED=18446744073103291231
> >> >>
> >> >> *GRID STAT STOPWed Aug 12 12:17:36 UTC 2020*
> >> >> missedTracking not set.  Running grid_stat for gmos 201911010
> >> >>
> >> >>
> >>
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
> >> >>
> >> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_gmosconc2p519110100ttf018
> >> >>
> >> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
> >> >> -v 0 -outdir
> /gpfs/dell2/ptmp/John.L.Wagner/gmos/co/dbtx/1911/01/18/tt
> >> >>
> >> >>
> >>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_gmos
> >> >>
> >> >> *GRID STAT STARTWed Aug 12 12:17:37 UTC 2020*
> >> >> GSL_RNG_TYPE=mt19937
> >> >> GSL_RNG_SEED=18446744073203954527
> >> >>
> >> >> *GRID STAT STOPWed Aug 12 12:17:42 UTC 2020*
> >> >>
> >> >> On Tue, Aug 11, 2020 at 2:03 PM John L Wagner - NOAA Federal <
> >> >> john.l.wagner at noaa.gov> wrote:
> >> >>
> >> >> > Thanks John.  Attached are the NDFD and URMA grib2 files
that we
> are
> >> >> > starting with and the grid_stat config file we are using.
The
> >> >> > regrid_data_plane command we are using is
> >> >> >
> >> >> > regrid_data_plane <grib2 file> <CONUS mask> <output netCDF
file>
> >> -method
> >> >> > NEAREST -width 1 -v 0 -name "tt" -compress 9 -field
"name=tt;
> >> level=Z2;"
> >> >> >
> >> >> > I have access to the prod machine (venus) so I have been
running
> some
> >> >> > tests today with grid_stat.  I've run tests with V8.1 out of
NCO's
> >> area
> >> >> and
> >> >> > with V8.1, V9.0.1, V9.0.2, and V9.0.3 in Julie's area.  The
> shortest
> >> and
> >> >> > longest run times I've seen today were both with Julie's
V8.1.
> This
> >> >> > morning, it ran in about 5 minutes.  At lunch time, it ran
in about
> >> 40
> >> >> > minutes.  All other runs took about 10 minutes.
> >> >> >
> >> >> > Tests I am going to run this afternoon:
> >> >> > 1.  We are currently running with masks for all of the CONUS
WFOs
> and
> >> >> > regions (about 135 masks).  I'm going to test with just the
CONUS
> >> mask.
> >> >> > 2.  I'll need to look at what elements we use this
grid_stat_config
> >> file
> >> >> > for.  For temperature, we only need the cnt file.  I'll turn
off
> the
> >> >> other
> >> >> > files.
> >> >> > 3.  I am going to try on the Cray instead of the Dell.
We've run
> >> into
> >> >> > several problems where the only logical problem is the read
and
> write
> >> >> > speeds on the Dell (phase 3).  If your tests on the project
machine
> >> at
> >> >> NCAR
> >> >> > don't show any issues, then this will continue to be what I
think
> the
> >> >> real
> >> >> > problem is.
> >> >> >
> >> >> > If you need anything else from me, let me know.
> >> >> > Thanks
> >> >> > John
> >> >> >
> >> >> > On Tue, Aug 11, 2020 at 11:57 AM John Halley Gotway via RT <
> >> >> > met_help at ucar.edu> wrote:
> >> >> >
> >> >> >> John,
> >> >> >>
> >> >> >> I did see your email come through met-help last week.
Howard is
> the
> >> >> person
> >> >> >> who's worked the most on NetCDF compression, so I was glad
he was
> >> able
> >> >> to
> >> >> >> respond. Unfortunately, I don't have any quick and obvious
answers
> >> for
> >> >> >> you.
> >> >> >> We just posted the met-9.1 release yesterday. It sounds
like
> you're
> >> >> >> weighing 2 main considerations... the output file size of
> >> >> >> regrid_data_plane
> >> >> >> and the runtime of grid_stat.
> >> >> >>
> >> >> >> Seems like we should do some comparison tests between met-
8.1 and
> >> >> met-9.1.
> >> >> >> What I'd like to figure out is whether the longer runtimes
are due
> >> >> solely
> >> >> >> to the difference in NetCDF compression levels. Howard has
told me
> >> that
> >> >> >> writing highly compressed NetCDF files is much, much
slower.
> >> >> >>
> >> >> >> We could pick a single case and run it through
> >> >> regrid_data_plane/grid_stat
> >> >> >> 6 times... with no/medium/high compression and met-8.1/9.1.
Then
> for
> >> >> each
> >> >> >> configuration, note the runtime and file sizes. And I also
want to
> >> >> confirm
> >> >> >> that the output generated is the same (i.e. met-9.1 isn't
writing
> >> any
> >> >> >> extra
> >> >> >> output variables). WCOSS is down this week, but I could do
this
> >> >> testing on
> >> >> >> a project machine at NCAR. But I'd like to approximate the
> datasets
> >> >> you're
> >> >> >> using on WCOSS.
> >> >> >>
> >> >> >> Can you point me to sample data files, the
regrid_data_plane
> command
> >> >> >> you're
> >> >> >> running, and the Grid-Stat configuration files you're
using?
> >> >> >>
> >> >> >> Thanks,
> >> >> >> John
> >> >> >>
> >> >> >> On Mon, Aug 10, 2020 at 10:11 AM John L Wagner - NOAA
Federal via
> >> RT <
> >> >> >> met_help at ucar.edu> wrote:
> >> >> >>
> >> >> >> >
> >> >> >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126 >
> >> >> >> >
> >> >> >> > Thanks Julie.  I did submit this to the help desk last
week.
> >> That is
> >> >> >> where
> >> >> >> > Howard mentioned checking the netcdf library.  I'll
submit again
> >> to
> >> >> get
> >> >> >> > John's thoughts.
> >> >> >> > John, please see my email below.  If you need any more
> information
> >> >> >> please
> >> >> >> > let me know.
> >> >> >> > Thanks
> >> >> >> > John
> >> >> >> >
> >> >> >> > On Mon, Aug 10, 2020 at 12:05 PM Julie Prestopnik <
> >> jpresto at ucar.edu>
> >> >> >> > wrote:
> >> >> >> >
> >> >> >> > > Hi John.
> >> >> >> > >
> >> >> >> > > I am not the best person to address these issues about
> >> >> compression.  I
> >> >> >> > can
> >> >> >> > > confirm that I started linking to v4.5.0 of the NetCDF
library
> >> for
> >> >> the
> >> >> >> > > reason you indicated - to use the standard WCOSS
libraries, as
> >> >> that is
> >> >> >> > how
> >> >> >> > > it will be done for the operational installation of
MET.
> >> >> >> > >
> >> >> >> > > Could you please submit your questions about
compression to
> >> >> >> > > met_help at ucar.edu so that your questions can be
addressed by
> >> the
> >> >> >> right
> >> >> >> > > person (likely John, who also happens to be on helpdesk
> today)?
> >> >> >> > >
> >> >> >> > > Thanks!
> >> >> >> > >
> >> >> >> > > Julie
> >> >> >> > >
> >> >> >> > > On Mon, Aug 10, 2020 at 8:12 AM John L Wagner - NOAA
Federal <
> >> >> >> > > john.l.wagner at noaa.gov> wrote:
> >> >> >> > >
> >> >> >> > >> Good morning Julie
> >> >> >> > >> Over the past few weeks, we've been seeing less
compression
> >> with
> >> >> >> > >> regrid_data_plane and significantly longer run times
with
> >> >> grid_stat.
> >> >> >> > For
> >> >> >> > >> regrid_data_plane, our CONUS files used to compress
from
> ~44MB
> >> to
> >> >> >> ~23MB.
> >> >> >> > >> Now they only compress to ~30MB.  For grid_stat, a
single
> model
> >> >> and
> >> >> >> > >> analysis grid would run in a minute or two.  It now
takes
> 8-10
> >> >> >> minutes
> >> >> >> > to
> >> >> >> > >> run.  We have been testing mainly with MET V9.0.2 and
V9.0.3
> >> out
> >> >> of
> >> >> >> your
> >> >> >> > >> directory on mars.
> >> >> >> > >> Howard mentioned that the compression is controlled by
the
> >> netcdf
> >> >> >> > >> library.  Starting with V9.0.1, you started linking to
> V4.5.0.
> >> >> >> Prior to
> >> >> >> > >> that, it looks like you had your own version of the
> library.  I
> >> >> >> assume
> >> >> >> > >> the change was made just to use the standard WCOSS
libraries.
> >> I
> >> >> was
> >> >> >> > just
> >> >> >> > >> wondering if anyone else had experienced the same
issues that
> >> we
> >> >> have
> >> >> >> > after
> >> >> >> > >> this change?
> >> >> >> > >> I ran some tests over the weekend reverting back to
MET V8.1.
> >> I
> >> >> >> didn't
> >> >> >> > >> get in early enough this morning to check the file
sizes of
> the
> >> >> >> > >> regrid_data_plane output, but the grid_stat run times
were
> >> still
> >> >> >> longer
> >> >> >> > >> than expected.  I suspect the netcdf library is not
the issue
> >> >> >> there.  If
> >> >> >> > >> you know of any issues that might be causing longer
grid_stat
> >> run
> >> >> >> times
> >> >> >> > or
> >> >> >> > >> have any suggestions for other tests we can try,
please let
> me
> >> >> know.
> >> >> >> > >> Thanks
> >> >> >> > >> John
> >> >> >> > >>
> >> >> >> > >> ---------- Forwarded message ---------
> >> >> >> > >> From: Howard Soh via RT <met_help at ucar.edu>
> >> >> >> > >> Date: Thu, Aug 6, 2020 at 12:29 AM
> >> >> >> > >> Subject: [rt.rap.ucar.edu #96126] Compression and
grid_stat
> >> run
> >> >> time
> >> >> >> > >> To: <john.l.wagner at noaa.gov>
> >> >> >> > >> Cc: <dana.strom at noaa.gov>, <tamarah.curtis at noaa.gov>
> >> >> >> > >>
> >> >> >> > >>
> >> >> >> > >> The compression is done by NetCDF library. The NetCDF
output
> >> (size
> >> >> >> > >> difference) is caused by the NetCDF library, not by
MET code.
> >> >> Please
> >> >> >> > check
> >> >> >> > >> the NetCDF library version:
> >> >> >> > >>    ldd regrid_data_plane | grep netcdf
> >> >> >> > >>
> >> >> >> > >> The issue 1363 (applied 9.0.3 and 9.1 beta 2) improves
the
> >> speed
> >> >> of
> >> >> >> > >> reading NetCDF input, and no update on writing NetCDF
output.
> >> >> >> > >>
> >> >> >> > >> Please let us know if 9.0.3 runs slower than 9.0.2
with using
> >> the
> >> >> >> same
> >> >> >> > >> NetCDF library.
> >> >> >> > >>
> >> >> >> > >> Cheers,
> >> >> >> > >> Howard
> >> >> >> > >>
> >> >> >> > >> Here are examples to check NetCDF library:
> >> >> >> > >> ldd met-8.1/bin/regrid_data_plane | grep netcdf
> >> >> >> > >>         libnetcdf_c++4.so.1 =>
> >> >> >> /usr/local/netcdf/lib/libnetcdf_c++4.so.1
> >> >> >> > >> (0x00007fee87e70000)
> >> >> >> > >>         libnetcdf.so.7 => not found
> >> >> >> > >>         libnetcdf.so.13 =>
> >> /usr/local/netcdf/lib/libnetcdf.so.13
> >> >> >> > >> (0x00007fee83f05000)
> >> >> >> > >>
> >> >> >> > >> ldd met-9.0.2/bin/regrid_data_plane | grep netcdf
> >> >> >> > >>         libnetcdf_c++4.so.1 =>
> >> >> >> > >> /usr/local/netcdf-4.7.0/gcc-
6.3.0/lib/libnetcdf_c++4.so.1
> >> >> >> > >> (0x00007fdf5d1b2000)
> >> >> >> > >>         libnetcdf.so.15 =>
> >> >> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
> >> >> >> > (0x00007fdf5ce31000)
> >> >> >> > >>
> >> >> >> > >>
> >> >> >> > >> ldd met-9.0.3/bin/regrid_data_plane | grep netcdf
> >> >> >> > >>         libnetcdf_c++4.so.1 =>
> >> >> >> > >> /usr/local/netcdf-4.7.0/gcc-
6.3.0/lib/libnetcdf_c++4.so.1
> >> >> >> > >> (0x00007f7dfe91d000)
> >> >> >> > >>         libnetcdf.so.15 =>
> >> >> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
> >> >> >> > (0x00007f7dfe59c000)
> >> >> >> > >>
> >> >> >> > >>
> >> >> >> > >> On Wed Aug 05 08:36:09 2020, mccabe wrote:
> >> >> >> > >> > Hi John,
> >> >> >> > >> >
> >> >> >> > >> > I see you have questions about compression in
regards to
> run
> >> >> time
> >> >> >> > >> > length and file sizes. I have assigned this ticket
to
> Howard
> >> >> Soh.
> >> >> >> > >> > Please allow a few business days for a response.
> >> >> >> > >> >
> >> >> >> > >> > Thanks,
> >> >> >> > >> > George
> >> >> >> > >>
> >> >> >> > >>
> >> >> >> > >>
> >> >> >> > >>
> >> >> >> > >>
> >> >> >> > >> --
> >> >> >> > >> John Wagner
> >> >> >> > >> Verification Task Lead
> >> >> >> > >> NOAA/National Weather Service
> >> >> >> > >> Meteorological Development Laboratory
> >> >> >> > >> Digital Forecast Services Division
> >> >> >> > >> SSMC2 Room 10106
> >> >> >> > >> Silver Spring, MD 20910
> >> >> >> > >> (301) 427-9471 (office)
> >> >> >> > >> (908) 902-4155 (cell/text)
> >> >> >> > >>
> >> >> >> > >
> >> >> >> > >
> >> >> >> > > --
> >> >> >> > > Julie Prestopnik (she/her/hers)
> >> >> >> > > Software Engineer
> >> >> >> > > National Center for Atmospheric Research
> >> >> >> > > Research Applications Laboratory
> >> >> >> > > Phone: 303.497.8399
> >> >> >> > > Email: jpresto at ucar.edu
> >> >> >> > >
> >> >> >> > > My working day may not be your working day.  Please do
not
> feel
> >> >> >> obliged
> >> >> >> > to
> >> >> >> > > reply to this email outside of your normal working
hours.
> >> >> >> > >
> >> >> >> >
> >> >> >> >
> >> >> >> > --
> >> >> >> > John Wagner
> >> >> >> > Verification Task Lead
> >> >> >> > NOAA/National Weather Service
> >> >> >> > Meteorological Development Laboratory
> >> >> >> > Digital Forecast Services Division
> >> >> >> > SSMC2 Room 10106
> >> >> >> > Silver Spring, MD 20910
> >> >> >> > (301) 427-9471 (office)
> >> >> >> > (908) 902-4155 (cell/text)
> >> >> >> >
> >> >> >> >
> >> >> >>
> >> >> >>
> >> >> >
> >> >> > --
> >> >> > John Wagner
> >> >> > Verification Task Lead
> >> >> > NOAA/National Weather Service
> >> >> > Meteorological Development Laboratory
> >> >> > Digital Forecast Services Division
> >> >> > SSMC2 Room 10106
> >> >> > Silver Spring, MD 20910
> >> >> > (301) 427-9471 (office)
> >> >> > (908) 902-4155 (cell/text)
> >> >> >
> >> >>
> >> >>
> >> >> --
> >> >> John Wagner
> >> >> Verification Task Lead
> >> >> NOAA/National Weather Service
> >> >> Meteorological Development Laboratory
> >> >> Digital Forecast Services Division
> >> >> SSMC2 Room 10106
> >> >> Silver Spring, MD 20910
> >> >> (301) 427-9471 (office)
> >> >> (908) 902-4155 (cell/text)
> >> >>
> >> >>
> >>
> >>
> >
> > --
> > John Wagner
> > Verification Task Lead
> > NOAA/National Weather Service
> > Meteorological Development Laboratory
> > Digital Forecast Services Division
> > SSMC2 Room 10106
> > Silver Spring, MD 20910
> > (301) 427-9471 (office)
> > (908) 902-4155 (cell/text)
> >
>
>
> --
> John Wagner
> Verification Task Lead
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Division
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>

------------------------------------------------
Subject: Compression and grid_stat run time
From: John L Wagner - NOAA Federal
Time: Fri Sep 11 16:07:27 2020

I think it would be helpful to have a time stamp with the regrid
message,
so I knew how long the regridding was taking.  Its not necessary
though.
The message is clear.  I was just hesitant to turn up the verbosity as
it
has slowed things down in the past and I was trying to speed things
up.
Thanks
John

On Fri, Sep 11, 2020 at 2:03 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:

> John,
>
> Great, thanks for the update! You know, in earlier versions of MET,
when
> the grid of the masking region did not match the verification
domain, the
> tools errored out stating that fact. At the request of EMC, we
updated the
> logic to do the automatic regridding using nearest neighbor
interpolation.
>
> But I've always been a bit uncomfortable about this "feature". In my
> opinion, it's better to define masking regions separately for each
domain
> rather than doing automated regridding. I know that there is a log
message
> stating that the regridding is occurring, but I wonder if you have
any
> other recommendations about this? What sort of functionality/log
> messages/warnings/config file options would make the behavior of the
> software more transparent?
>
> Thanks,
> John
>
> On Fri, Sep 11, 2020 at 10:16 AM John L Wagner - NOAA Federal via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126 >
> >
> > Hey John
> > I meant to mention that I figured out what was going on last week.
Our
> > mask that we were using for regrid_data_plane and the subdomain
masks
> that
> > we were using in grid_stat were not set up using the same grid
> > coordinates.  The mask in regrid_data_plane was the NBM grid
(2345x1597
> > gridpoints) and the grid_stat subdomains (WFOs, regions, RFCs)
were all
> > from the URMA grid (2145x1377 gridpoints).  This is why
regrid_data_plane
> > wasn't compressing to the level I expected.  When grid_stat ran,
it was
> > regridding for each subdomain, which led to the longer run times.
Once I
> > regridded the subdomains to the NBM grid, the run times went back
to
> normal
> > for everything except PoP12.  I'm still looking into that one.
> > Just wanted you to have the solution in case anyone else ran into
> > this issue.
> > Thanks
> > John
> >
> > On Thu, Aug 27, 2020 at 4:31 PM John L Wagner - NOAA Federal <
> > john.l.wagner at noaa.gov> wrote:
> >
> > > Thanks John.  I noted earlier that grid_stat run times were as
long as
> 25
> > > minutes.  We have more files on disk now, so I think I/O is the
real
> > > issue.  We'll know for sure on Monday, after the prod switch
when we
> have
> > > to start from scratch essentially.
> > > Switching to grib2 files will give us smaller file sizes and
fewer
> files
> > > that we need to keep. Hoping that it alleviates the I/O issues.
> > >
> > > On Thu, Aug 27, 2020 at 4:07 PM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >> John,
> > >>
> > >> I'm going to go ahead and resolve this old met-help ticket. As
we
> > >> discussed
> > >> at the NOAA METplus telecon earlier this week, you're going to
try
> > >> switching to using GRIB input files and check to see what
impact that
> > has
> > >> on runtimes.
> > >>
> > >> Thanks,
> > >> John HG
> > >>
> > >> On Thu, Aug 13, 2020 at 11:53 AM John Halley Gotway
<johnhg at ucar.edu>
> > >> wrote:
> > >>
> > >> > John,
> > >> >
> > >> > Thanks for sending this data. Unfortunately, I didn't get a
chance
> to
> > >> dig
> > >> > into the details before heading out of town. But I'll be back
from
> PTO
> > >> on
> > >> > Monday.
> > >> >
> > >> > Thanks,
> > >> > John
> > >> >
> > >> > On Wed, Aug 12, 2020 at 6:42 AM John L Wagner - NOAA Federal
via RT
> <
> > >> > met_help at ucar.edu> wrote:
> > >> >
> > >> >>
> > >> >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126 >
> > >> >>
> > >> >> Good morning John
> > >> >> I ran the tests yesterday afternoon that I mentioned in my
previous
> > >> email.
> > >> >> Running for just the CNT output and using the Cray's ptmp to
read
> and
> > >> >> write
> > >> >> data had no effect on the run times.  Running for just the
CONUS
> mask
> > >> >> initially took about 57 seconds to run with compressed
files.
> > >> Switching
> > >> >> to
> > >> >> uncompressed files, it took about 38 seconds to run for just
the
> > CONUS
> > >> >> mask
> > >> >> and 8 minutes and 40 seconds for all masks.
> > >> >> This morning, I tested running with just the CONUS mask
again and
> > used
> > >> the
> > >> >> uncompressed files.  For some reason this morning, grid_stat
runs
> in
> > >> about
> > >> >> 5-6 seconds, which is what I was seeing a few weeks ago.  I
made no
> > >> other
> > >> >> changes to my scripts or modules other than to limit the
runs to
> just
> > >> the
> > >> >> CONUS mask again.  The start and stop times from my logs are
below.
> > >> >> I'm really at a loss as to what could be causing these
fluctuations
> > in
> > >> run
> > >> >> times.
> > >> >> Thanks
> > >> >> John
> > >> >>
> > >> >>
> > >> >>
> > >>
> >
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
> > >> >>
> > >> >>
> > >>
> >
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_ndfdconc2p519110112ttf006
> > >> >>
> > >> >>
> > >>
> >
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
> > >> >> -v 0 -outdir
> > /gpfs/dell2/ptmp/John.L.Wagner/ndfd/co/dbtx/1911/01/18/tt
> > >> >>
> > >> >>
> > >>
> >
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_ndfd
> > >> >>
> > >> >> *GRID STAT STARTWed Aug 12 12:17:20 UTC 2020*
> > >> >> GSL_RNG_TYPE=mt19937
> > >> >> GSL_RNG_SEED=18446744072918741855
> > >> >>
> > >> >> *GRID STAT STOPWed Aug 12 12:17:26 UTC 2020*
> > >> >> missedTracking not set.  Running grid_stat for blend
201911017
> > >> >>
> > >> >>
> > >>
> >
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
> > >> >>
> > >> >>
> > >>
> >
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_blendconc2p519110107ttf011
> > >> >>
> > >> >>
> > >>
> >
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
> > >> >> -v 0 -outdir
> > /gpfs/dell2/ptmp/John.L.Wagner/blend/co/dbtx/1911/01/18/tt
> > >> >>
> > >> >>
> > >>
> >
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_blend
> > >> >>
> > >> >> *GRID STAT STARTWed Aug 12 12:17:26 UTC 2020*
> > >> >> GSL_RNG_TYPE=mt19937
> > >> >> GSL_RNG_SEED=18446744073019405151
> > >> >>
> > >> >> *GRID STAT STOPWed Aug 12 12:17:31 UTC 2020*
> > >> >> missedTracking not set.  Running grid_stat for blendp
201911017
> > >> >>
> > >> >>
> > >>
> >
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
> > >> >>
> > >> >>
> > >>
> >
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_blendpconc2p519110107ttf011
> > >> >>
> > >> >>
> > >>
> >
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
> > >> >> -v 0 -outdir
> > >> /gpfs/dell2/ptmp/John.L.Wagner/blendp/co/dbtx/1911/01/18/tt
> > >> >>
> > >> >>
> > >>
> >
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_blendp
> > >> >>
> > >> >> *GRID STAT STARTWed Aug 12 12:17:31 UTC 2020*
> > >> >> GSL_RNG_TYPE=mt19937
> > >> >> GSL_RNG_SEED=18446744073103291231
> > >> >>
> > >> >> *GRID STAT STOPWed Aug 12 12:17:36 UTC 2020*
> > >> >> missedTracking not set.  Running grid_stat for gmos
201911010
> > >> >>
> > >> >>
> > >>
> >
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
> > >> >>
> > >> >>
> > >>
> >
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_gmosconc2p519110100ttf018
> > >> >>
> > >> >>
> > >>
> >
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
> > >> >> -v 0 -outdir
> > /gpfs/dell2/ptmp/John.L.Wagner/gmos/co/dbtx/1911/01/18/tt
> > >> >>
> > >> >>
> > >>
> >
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_gmos
> > >> >>
> > >> >> *GRID STAT STARTWed Aug 12 12:17:37 UTC 2020*
> > >> >> GSL_RNG_TYPE=mt19937
> > >> >> GSL_RNG_SEED=18446744073203954527
> > >> >>
> > >> >> *GRID STAT STOPWed Aug 12 12:17:42 UTC 2020*
> > >> >>
> > >> >> On Tue, Aug 11, 2020 at 2:03 PM John L Wagner - NOAA Federal
<
> > >> >> john.l.wagner at noaa.gov> wrote:
> > >> >>
> > >> >> > Thanks John.  Attached are the NDFD and URMA grib2 files
that we
> > are
> > >> >> > starting with and the grid_stat config file we are using.
The
> > >> >> > regrid_data_plane command we are using is
> > >> >> >
> > >> >> > regrid_data_plane <grib2 file> <CONUS mask> <output netCDF
file>
> > >> -method
> > >> >> > NEAREST -width 1 -v 0 -name "tt" -compress 9 -field
"name=tt;
> > >> level=Z2;"
> > >> >> >
> > >> >> > I have access to the prod machine (venus) so I have been
running
> > some
> > >> >> > tests today with grid_stat.  I've run tests with V8.1 out
of
> NCO's
> > >> area
> > >> >> and
> > >> >> > with V8.1, V9.0.1, V9.0.2, and V9.0.3 in Julie's area.
The
> > shortest
> > >> and
> > >> >> > longest run times I've seen today were both with Julie's
V8.1.
> > This
> > >> >> > morning, it ran in about 5 minutes.  At lunch time, it ran
in
> about
> > >> 40
> > >> >> > minutes.  All other runs took about 10 minutes.
> > >> >> >
> > >> >> > Tests I am going to run this afternoon:
> > >> >> > 1.  We are currently running with masks for all of the
CONUS WFOs
> > and
> > >> >> > regions (about 135 masks).  I'm going to test with just
the CONUS
> > >> mask.
> > >> >> > 2.  I'll need to look at what elements we use this
> grid_stat_config
> > >> file
> > >> >> > for.  For temperature, we only need the cnt file.  I'll
turn off
> > the
> > >> >> other
> > >> >> > files.
> > >> >> > 3.  I am going to try on the Cray instead of the Dell.
We've run
> > >> into
> > >> >> > several problems where the only logical problem is the
read and
> > write
> > >> >> > speeds on the Dell (phase 3).  If your tests on the
project
> machine
> > >> at
> > >> >> NCAR
> > >> >> > don't show any issues, then this will continue to be what
I think
> > the
> > >> >> real
> > >> >> > problem is.
> > >> >> >
> > >> >> > If you need anything else from me, let me know.
> > >> >> > Thanks
> > >> >> > John
> > >> >> >
> > >> >> > On Tue, Aug 11, 2020 at 11:57 AM John Halley Gotway via RT
<
> > >> >> > met_help at ucar.edu> wrote:
> > >> >> >
> > >> >> >> John,
> > >> >> >>
> > >> >> >> I did see your email come through met-help last week.
Howard is
> > the
> > >> >> person
> > >> >> >> who's worked the most on NetCDF compression, so I was
glad he
> was
> > >> able
> > >> >> to
> > >> >> >> respond. Unfortunately, I don't have any quick and
obvious
> answers
> > >> for
> > >> >> >> you.
> > >> >> >> We just posted the met-9.1 release yesterday. It sounds
like
> > you're
> > >> >> >> weighing 2 main considerations... the output file size of
> > >> >> >> regrid_data_plane
> > >> >> >> and the runtime of grid_stat.
> > >> >> >>
> > >> >> >> Seems like we should do some comparison tests between
met-8.1
> and
> > >> >> met-9.1.
> > >> >> >> What I'd like to figure out is whether the longer
runtimes are
> due
> > >> >> solely
> > >> >> >> to the difference in NetCDF compression levels. Howard
has told
> me
> > >> that
> > >> >> >> writing highly compressed NetCDF files is much, much
slower.
> > >> >> >>
> > >> >> >> We could pick a single case and run it through
> > >> >> regrid_data_plane/grid_stat
> > >> >> >> 6 times... with no/medium/high compression and met-
8.1/9.1. Then
> > for
> > >> >> each
> > >> >> >> configuration, note the runtime and file sizes. And I
also want
> to
> > >> >> confirm
> > >> >> >> that the output generated is the same (i.e. met-9.1 isn't
> writing
> > >> any
> > >> >> >> extra
> > >> >> >> output variables). WCOSS is down this week, but I could
do this
> > >> >> testing on
> > >> >> >> a project machine at NCAR. But I'd like to approximate
the
> > datasets
> > >> >> you're
> > >> >> >> using on WCOSS.
> > >> >> >>
> > >> >> >> Can you point me to sample data files, the
regrid_data_plane
> > command
> > >> >> >> you're
> > >> >> >> running, and the Grid-Stat configuration files you're
using?
> > >> >> >>
> > >> >> >> Thanks,
> > >> >> >> John
> > >> >> >>
> > >> >> >> On Mon, Aug 10, 2020 at 10:11 AM John L Wagner - NOAA
Federal
> via
> > >> RT <
> > >> >> >> met_help at ucar.edu> wrote:
> > >> >> >>
> > >> >> >> >
> > >> >> >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126
> >
> > >> >> >> >
> > >> >> >> > Thanks Julie.  I did submit this to the help desk last
week.
> > >> That is
> > >> >> >> where
> > >> >> >> > Howard mentioned checking the netcdf library.  I'll
submit
> again
> > >> to
> > >> >> get
> > >> >> >> > John's thoughts.
> > >> >> >> > John, please see my email below.  If you need any more
> > information
> > >> >> >> please
> > >> >> >> > let me know.
> > >> >> >> > Thanks
> > >> >> >> > John
> > >> >> >> >
> > >> >> >> > On Mon, Aug 10, 2020 at 12:05 PM Julie Prestopnik <
> > >> jpresto at ucar.edu>
> > >> >> >> > wrote:
> > >> >> >> >
> > >> >> >> > > Hi John.
> > >> >> >> > >
> > >> >> >> > > I am not the best person to address these issues
about
> > >> >> compression.  I
> > >> >> >> > can
> > >> >> >> > > confirm that I started linking to v4.5.0 of the
NetCDF
> library
> > >> for
> > >> >> the
> > >> >> >> > > reason you indicated - to use the standard WCOSS
libraries,
> as
> > >> >> that is
> > >> >> >> > how
> > >> >> >> > > it will be done for the operational installation of
MET.
> > >> >> >> > >
> > >> >> >> > > Could you please submit your questions about
compression to
> > >> >> >> > > met_help at ucar.edu so that your questions can be
addressed
> by
> > >> the
> > >> >> >> right
> > >> >> >> > > person (likely John, who also happens to be on
helpdesk
> > today)?
> > >> >> >> > >
> > >> >> >> > > Thanks!
> > >> >> >> > >
> > >> >> >> > > Julie
> > >> >> >> > >
> > >> >> >> > > On Mon, Aug 10, 2020 at 8:12 AM John L Wagner - NOAA
> Federal <
> > >> >> >> > > john.l.wagner at noaa.gov> wrote:
> > >> >> >> > >
> > >> >> >> > >> Good morning Julie
> > >> >> >> > >> Over the past few weeks, we've been seeing less
compression
> > >> with
> > >> >> >> > >> regrid_data_plane and significantly longer run times
with
> > >> >> grid_stat.
> > >> >> >> > For
> > >> >> >> > >> regrid_data_plane, our CONUS files used to compress
from
> > ~44MB
> > >> to
> > >> >> >> ~23MB.
> > >> >> >> > >> Now they only compress to ~30MB.  For grid_stat, a
single
> > model
> > >> >> and
> > >> >> >> > >> analysis grid would run in a minute or two.  It now
takes
> > 8-10
> > >> >> >> minutes
> > >> >> >> > to
> > >> >> >> > >> run.  We have been testing mainly with MET V9.0.2
and
> V9.0.3
> > >> out
> > >> >> of
> > >> >> >> your
> > >> >> >> > >> directory on mars.
> > >> >> >> > >> Howard mentioned that the compression is controlled
by the
> > >> netcdf
> > >> >> >> > >> library.  Starting with V9.0.1, you started linking
to
> > V4.5.0.
> > >> >> >> Prior to
> > >> >> >> > >> that, it looks like you had your own version of the
> > library.  I
> > >> >> >> assume
> > >> >> >> > >> the change was made just to use the standard WCOSS
> libraries.
> > >> I
> > >> >> was
> > >> >> >> > just
> > >> >> >> > >> wondering if anyone else had experienced the same
issues
> that
> > >> we
> > >> >> have
> > >> >> >> > after
> > >> >> >> > >> this change?
> > >> >> >> > >> I ran some tests over the weekend reverting back to
MET
> V8.1.
> > >> I
> > >> >> >> didn't
> > >> >> >> > >> get in early enough this morning to check the file
sizes of
> > the
> > >> >> >> > >> regrid_data_plane output, but the grid_stat run
times were
> > >> still
> > >> >> >> longer
> > >> >> >> > >> than expected.  I suspect the netcdf library is not
the
> issue
> > >> >> >> there.  If
> > >> >> >> > >> you know of any issues that might be causing longer
> grid_stat
> > >> run
> > >> >> >> times
> > >> >> >> > or
> > >> >> >> > >> have any suggestions for other tests we can try,
please let
> > me
> > >> >> know.
> > >> >> >> > >> Thanks
> > >> >> >> > >> John
> > >> >> >> > >>
> > >> >> >> > >> ---------- Forwarded message ---------
> > >> >> >> > >> From: Howard Soh via RT <met_help at ucar.edu>
> > >> >> >> > >> Date: Thu, Aug 6, 2020 at 12:29 AM
> > >> >> >> > >> Subject: [rt.rap.ucar.edu #96126] Compression and
> grid_stat
> > >> run
> > >> >> time
> > >> >> >> > >> To: <john.l.wagner at noaa.gov>
> > >> >> >> > >> Cc: <dana.strom at noaa.gov>, <tamarah.curtis at noaa.gov>
> > >> >> >> > >>
> > >> >> >> > >>
> > >> >> >> > >> The compression is done by NetCDF library. The
NetCDF
> output
> > >> (size
> > >> >> >> > >> difference) is caused by the NetCDF library, not by
MET
> code.
> > >> >> Please
> > >> >> >> > check
> > >> >> >> > >> the NetCDF library version:
> > >> >> >> > >>    ldd regrid_data_plane | grep netcdf
> > >> >> >> > >>
> > >> >> >> > >> The issue 1363 (applied 9.0.3 and 9.1 beta 2)
improves the
> > >> speed
> > >> >> of
> > >> >> >> > >> reading NetCDF input, and no update on writing
NetCDF
> output.
> > >> >> >> > >>
> > >> >> >> > >> Please let us know if 9.0.3 runs slower than 9.0.2
with
> using
> > >> the
> > >> >> >> same
> > >> >> >> > >> NetCDF library.
> > >> >> >> > >>
> > >> >> >> > >> Cheers,
> > >> >> >> > >> Howard
> > >> >> >> > >>
> > >> >> >> > >> Here are examples to check NetCDF library:
> > >> >> >> > >> ldd met-8.1/bin/regrid_data_plane | grep netcdf
> > >> >> >> > >>         libnetcdf_c++4.so.1 =>
> > >> >> >> /usr/local/netcdf/lib/libnetcdf_c++4.so.1
> > >> >> >> > >> (0x00007fee87e70000)
> > >> >> >> > >>         libnetcdf.so.7 => not found
> > >> >> >> > >>         libnetcdf.so.13 =>
> > >> /usr/local/netcdf/lib/libnetcdf.so.13
> > >> >> >> > >> (0x00007fee83f05000)
> > >> >> >> > >>
> > >> >> >> > >> ldd met-9.0.2/bin/regrid_data_plane | grep netcdf
> > >> >> >> > >>         libnetcdf_c++4.so.1 =>
> > >> >> >> > >> /usr/local/netcdf-4.7.0/gcc-
6.3.0/lib/libnetcdf_c++4.so.1
> > >> >> >> > >> (0x00007fdf5d1b2000)
> > >> >> >> > >>         libnetcdf.so.15 =>
> > >> >> >> > >> /usr/local/netcdf-4.7.0/gcc-
6.3.0/lib/libnetcdf.so.15
> > >> >> >> > (0x00007fdf5ce31000)
> > >> >> >> > >>
> > >> >> >> > >>
> > >> >> >> > >> ldd met-9.0.3/bin/regrid_data_plane | grep netcdf
> > >> >> >> > >>         libnetcdf_c++4.so.1 =>
> > >> >> >> > >> /usr/local/netcdf-4.7.0/gcc-
6.3.0/lib/libnetcdf_c++4.so.1
> > >> >> >> > >> (0x00007f7dfe91d000)
> > >> >> >> > >>         libnetcdf.so.15 =>
> > >> >> >> > >> /usr/local/netcdf-4.7.0/gcc-
6.3.0/lib/libnetcdf.so.15
> > >> >> >> > (0x00007f7dfe59c000)
> > >> >> >> > >>
> > >> >> >> > >>
> > >> >> >> > >> On Wed Aug 05 08:36:09 2020, mccabe wrote:
> > >> >> >> > >> > Hi John,
> > >> >> >> > >> >
> > >> >> >> > >> > I see you have questions about compression in
regards to
> > run
> > >> >> time
> > >> >> >> > >> > length and file sizes. I have assigned this ticket
to
> > Howard
> > >> >> Soh.
> > >> >> >> > >> > Please allow a few business days for a response.
> > >> >> >> > >> >
> > >> >> >> > >> > Thanks,
> > >> >> >> > >> > George
> > >> >> >> > >>
> > >> >> >> > >>
> > >> >> >> > >>
> > >> >> >> > >>
> > >> >> >> > >>
> > >> >> >> > >> --
> > >> >> >> > >> John Wagner
> > >> >> >> > >> Verification Task Lead
> > >> >> >> > >> NOAA/National Weather Service
> > >> >> >> > >> Meteorological Development Laboratory
> > >> >> >> > >> Digital Forecast Services Division
> > >> >> >> > >> SSMC2 Room 10106
> > >> >> >> > >> Silver Spring, MD 20910
> > >> >> >> > >> (301) 427-9471 (office)
> > >> >> >> > >> (908) 902-4155 (cell/text)
> > >> >> >> > >>
> > >> >> >> > >
> > >> >> >> > >
> > >> >> >> > > --
> > >> >> >> > > Julie Prestopnik (she/her/hers)
> > >> >> >> > > Software Engineer
> > >> >> >> > > National Center for Atmospheric Research
> > >> >> >> > > Research Applications Laboratory
> > >> >> >> > > Phone: 303.497.8399
> > >> >> >> > > Email: jpresto at ucar.edu
> > >> >> >> > >
> > >> >> >> > > My working day may not be your working day.  Please
do not
> > feel
> > >> >> >> obliged
> > >> >> >> > to
> > >> >> >> > > reply to this email outside of your normal working
hours.
> > >> >> >> > >
> > >> >> >> >
> > >> >> >> >
> > >> >> >> > --
> > >> >> >> > John Wagner
> > >> >> >> > Verification Task Lead
> > >> >> >> > NOAA/National Weather Service
> > >> >> >> > Meteorological Development Laboratory
> > >> >> >> > Digital Forecast Services Division
> > >> >> >> > SSMC2 Room 10106
> > >> >> >> > Silver Spring, MD 20910
> > >> >> >> > (301) 427-9471 (office)
> > >> >> >> > (908) 902-4155 (cell/text)
> > >> >> >> >
> > >> >> >> >
> > >> >> >>
> > >> >> >>
> > >> >> >
> > >> >> > --
> > >> >> > John Wagner
> > >> >> > Verification Task Lead
> > >> >> > NOAA/National Weather Service
> > >> >> > Meteorological Development Laboratory
> > >> >> > Digital Forecast Services Division
> > >> >> > SSMC2 Room 10106
> > >> >> > Silver Spring, MD 20910
> > >> >> > (301) 427-9471 (office)
> > >> >> > (908) 902-4155 (cell/text)
> > >> >> >
> > >> >>
> > >> >>
> > >> >> --
> > >> >> John Wagner
> > >> >> Verification Task Lead
> > >> >> NOAA/National Weather Service
> > >> >> Meteorological Development Laboratory
> > >> >> Digital Forecast Services Division
> > >> >> SSMC2 Room 10106
> > >> >> Silver Spring, MD 20910
> > >> >> (301) 427-9471 (office)
> > >> >> (908) 902-4155 (cell/text)
> > >> >>
> > >> >>
> > >>
> > >>
> > >
> > > --
> > > John Wagner
> > > Verification Task Lead
> > > NOAA/National Weather Service
> > > Meteorological Development Laboratory
> > > Digital Forecast Services Division
> > > SSMC2 Room 10106
> > > Silver Spring, MD 20910
> > > (301) 427-9471 (office)
> > > (908) 902-4155 (cell/text)
> > >
> >
> >
> > --
> > John Wagner
> > Verification Task Lead
> > NOAA/National Weather Service
> > Meteorological Development Laboratory
> > Digital Forecast Services Division
> > SSMC2 Room 10106
> > Silver Spring, MD 20910
> > (301) 427-9471 (office)
> > (908) 902-4155 (cell/text)
> >
> >
>
>

--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)

------------------------------------------------