[Met_help] [rt.rap.ucar.edu #96126] History for Compression and grid_stat run time
John Halley Gotway via RT
met_help at ucar.edu
Thu Aug 27 14:04:04 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Good morning MET help desk
I'm trying to figure out what compression levels we should be using for
regrid_data_plane with the latest version on WCOSS (V9.0.3) and how to get
the fastest run times out of grid_stat. We did see this ticket
<https://github.com/NCAR/MET/issues/1363> in the latest release notes. I
wasn't sure if this change would explain the results we were seeing.
For reference, an uncompressed netCDF file that comes out of
regrid_data_plane for most models for the CONUS is ~44MB. With -compress
level 9 turned on, it compresses to ~30MB using MET V9.0.3. Using MET
V8.1, it compresses to ~23MB. With the number of models and projections
that we work with, 7MB per file adds up quickly. Is it possible to get
more compression with the newer version of MET?
As for grid_stat, it is currently taking 8-10 minutes to run grid_stat for
a single case (or ~40 minutes for all models). I don't recall if this
started when we started using V9.0.2 or V9.0.3, but it used to take only a
minute or two to run grid_stat for a single case (or ~5 minutes for all
models). I'm not sure if the compression of the netCDF files is the issue
(likely not) or if something else changed in grid_stat to increase the run
times. If you know of any ways to speed this up to the previous run time,
please let me know. If not, we will likely revert back to v8.1. I have
not tested the V9.1 betas yet.
Thanks
John
--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Compression and grid_stat run time
From: George McCabe
Time: Wed Aug 05 08:36:09 2020
Hi John,
I see you have questions about compression in regards to run time
length and file sizes. I have assigned this ticket to Howard Soh.
Please allow a few business days for a response.
Thanks,
George
------------------------------------------------
Subject: Compression and grid_stat run time
From: Howard Soh
Time: Wed Aug 05 22:29:00 2020
The compression is done by NetCDF library. The NetCDF output (size
difference) is caused by the NetCDF library, not by MET code. Please
check the NetCDF library version:
ldd regrid_data_plane | grep netcdf
The issue 1363 (applied 9.0.3 and 9.1 beta 2) improves the speed of
reading NetCDF input, and no update on writing NetCDF output.
Please let us know if 9.0.3 runs slower than 9.0.2 with using the same
NetCDF library.
Cheers,
Howard
Here are examples to check NetCDF library:
ldd met-8.1/bin/regrid_data_plane | grep netcdf
libnetcdf_c++4.so.1 =>
/usr/local/netcdf/lib/libnetcdf_c++4.so.1 (0x00007fee87e70000)
libnetcdf.so.7 => not found
libnetcdf.so.13 => /usr/local/netcdf/lib/libnetcdf.so.13
(0x00007fee83f05000)
ldd met-9.0.2/bin/regrid_data_plane | grep netcdf
libnetcdf_c++4.so.1 => /usr/local/netcdf-4.7.0/gcc-
6.3.0/lib/libnetcdf_c++4.so.1 (0x00007fdf5d1b2000)
libnetcdf.so.15 => /usr/local/netcdf-4.7.0/gcc-
6.3.0/lib/libnetcdf.so.15 (0x00007fdf5ce31000)
ldd met-9.0.3/bin/regrid_data_plane | grep netcdf
libnetcdf_c++4.so.1 => /usr/local/netcdf-4.7.0/gcc-
6.3.0/lib/libnetcdf_c++4.so.1 (0x00007f7dfe91d000)
libnetcdf.so.15 => /usr/local/netcdf-4.7.0/gcc-
6.3.0/lib/libnetcdf.so.15 (0x00007f7dfe59c000)
On Wed Aug 05 08:36:09 2020, mccabe wrote:
> Hi John,
>
> I see you have questions about compression in regards to run time
> length and file sizes. I have assigned this ticket to Howard Soh.
> Please allow a few business days for a response.
>
> Thanks,
> George
------------------------------------------------
Subject: Compression and grid_stat run time
From: John L Wagner - NOAA Federal
Time: Mon Aug 10 10:10:44 2020
Thanks Julie. I did submit this to the help desk last week. That is
where
Howard mentioned checking the netcdf library. I'll submit again to
get
John's thoughts.
John, please see my email below. If you need any more information
please
let me know.
Thanks
John
On Mon, Aug 10, 2020 at 12:05 PM Julie Prestopnik <jpresto at ucar.edu>
wrote:
> Hi John.
>
> I am not the best person to address these issues about compression.
I can
> confirm that I started linking to v4.5.0 of the NetCDF library for
the
> reason you indicated - to use the standard WCOSS libraries, as that
is how
> it will be done for the operational installation of MET.
>
> Could you please submit your questions about compression to
> met_help at ucar.edu so that your questions can be addressed by the
right
> person (likely John, who also happens to be on helpdesk today)?
>
> Thanks!
>
> Julie
>
> On Mon, Aug 10, 2020 at 8:12 AM John L Wagner - NOAA Federal <
> john.l.wagner at noaa.gov> wrote:
>
>> Good morning Julie
>> Over the past few weeks, we've been seeing less compression with
>> regrid_data_plane and significantly longer run times with
grid_stat. For
>> regrid_data_plane, our CONUS files used to compress from ~44MB to
~23MB.
>> Now they only compress to ~30MB. For grid_stat, a single model and
>> analysis grid would run in a minute or two. It now takes 8-10
minutes to
>> run. We have been testing mainly with MET V9.0.2 and V9.0.3 out of
your
>> directory on mars.
>> Howard mentioned that the compression is controlled by the netcdf
>> library. Starting with V9.0.1, you started linking to V4.5.0.
Prior to
>> that, it looks like you had your own version of the library. I
assume
>> the change was made just to use the standard WCOSS libraries. I
was just
>> wondering if anyone else had experienced the same issues that we
have after
>> this change?
>> I ran some tests over the weekend reverting back to MET V8.1. I
didn't
>> get in early enough this morning to check the file sizes of the
>> regrid_data_plane output, but the grid_stat run times were still
longer
>> than expected. I suspect the netcdf library is not the issue
there. If
>> you know of any issues that might be causing longer grid_stat run
times or
>> have any suggestions for other tests we can try, please let me
know.
>> Thanks
>> John
>>
>> ---------- Forwarded message ---------
>> From: Howard Soh via RT <met_help at ucar.edu>
>> Date: Thu, Aug 6, 2020 at 12:29 AM
>> Subject: [rt.rap.ucar.edu #96126] Compression and grid_stat run
time
>> To: <john.l.wagner at noaa.gov>
>> Cc: <dana.strom at noaa.gov>, <tamarah.curtis at noaa.gov>
>>
>>
>> The compression is done by NetCDF library. The NetCDF output (size
>> difference) is caused by the NetCDF library, not by MET code.
Please check
>> the NetCDF library version:
>> ldd regrid_data_plane | grep netcdf
>>
>> The issue 1363 (applied 9.0.3 and 9.1 beta 2) improves the speed of
>> reading NetCDF input, and no update on writing NetCDF output.
>>
>> Please let us know if 9.0.3 runs slower than 9.0.2 with using the
same
>> NetCDF library.
>>
>> Cheers,
>> Howard
>>
>> Here are examples to check NetCDF library:
>> ldd met-8.1/bin/regrid_data_plane | grep netcdf
>> libnetcdf_c++4.so.1 =>
/usr/local/netcdf/lib/libnetcdf_c++4.so.1
>> (0x00007fee87e70000)
>> libnetcdf.so.7 => not found
>> libnetcdf.so.13 => /usr/local/netcdf/lib/libnetcdf.so.13
>> (0x00007fee83f05000)
>>
>> ldd met-9.0.2/bin/regrid_data_plane | grep netcdf
>> libnetcdf_c++4.so.1 =>
>> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
>> (0x00007fdf5d1b2000)
>> libnetcdf.so.15 =>
>> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
(0x00007fdf5ce31000)
>>
>>
>> ldd met-9.0.3/bin/regrid_data_plane | grep netcdf
>> libnetcdf_c++4.so.1 =>
>> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
>> (0x00007f7dfe91d000)
>> libnetcdf.so.15 =>
>> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
(0x00007f7dfe59c000)
>>
>>
>> On Wed Aug 05 08:36:09 2020, mccabe wrote:
>> > Hi John,
>> >
>> > I see you have questions about compression in regards to run time
>> > length and file sizes. I have assigned this ticket to Howard Soh.
>> > Please allow a few business days for a response.
>> >
>> > Thanks,
>> > George
>>
>>
>>
>>
>>
>> --
>> John Wagner
>> Verification Task Lead
>> NOAA/National Weather Service
>> Meteorological Development Laboratory
>> Digital Forecast Services Division
>> SSMC2 Room 10106
>> Silver Spring, MD 20910
>> (301) 427-9471 (office)
>> (908) 902-4155 (cell/text)
>>
>
>
> --
> Julie Prestopnik (she/her/hers)
> Software Engineer
> National Center for Atmospheric Research
> Research Applications Laboratory
> Phone: 303.497.8399
> Email: jpresto at ucar.edu
>
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
------------------------------------------------
Subject: Compression and grid_stat run time
From: John Halley Gotway
Time: Tue Aug 11 09:57:04 2020
John,
I did see your email come through met-help last week. Howard is the
person
who's worked the most on NetCDF compression, so I was glad he was able
to
respond. Unfortunately, I don't have any quick and obvious answers for
you.
We just posted the met-9.1 release yesterday. It sounds like you're
weighing 2 main considerations... the output file size of
regrid_data_plane
and the runtime of grid_stat.
Seems like we should do some comparison tests between met-8.1 and met-
9.1.
What I'd like to figure out is whether the longer runtimes are due
solely
to the difference in NetCDF compression levels. Howard has told me
that
writing highly compressed NetCDF files is much, much slower.
We could pick a single case and run it through
regrid_data_plane/grid_stat
6 times... with no/medium/high compression and met-8.1/9.1. Then for
each
configuration, note the runtime and file sizes. And I also want to
confirm
that the output generated is the same (i.e. met-9.1 isn't writing any
extra
output variables). WCOSS is down this week, but I could do this
testing on
a project machine at NCAR. But I'd like to approximate the datasets
you're
using on WCOSS.
Can you point me to sample data files, the regrid_data_plane command
you're
running, and the Grid-Stat configuration files you're using?
Thanks,
John
On Mon, Aug 10, 2020 at 10:11 AM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126 >
>
> Thanks Julie. I did submit this to the help desk last week. That
is where
> Howard mentioned checking the netcdf library. I'll submit again to
get
> John's thoughts.
> John, please see my email below. If you need any more information
please
> let me know.
> Thanks
> John
>
> On Mon, Aug 10, 2020 at 12:05 PM Julie Prestopnik <jpresto at ucar.edu>
> wrote:
>
> > Hi John.
> >
> > I am not the best person to address these issues about
compression. I
> can
> > confirm that I started linking to v4.5.0 of the NetCDF library for
the
> > reason you indicated - to use the standard WCOSS libraries, as
that is
> how
> > it will be done for the operational installation of MET.
> >
> > Could you please submit your questions about compression to
> > met_help at ucar.edu so that your questions can be addressed by the
right
> > person (likely John, who also happens to be on helpdesk today)?
> >
> > Thanks!
> >
> > Julie
> >
> > On Mon, Aug 10, 2020 at 8:12 AM John L Wagner - NOAA Federal <
> > john.l.wagner at noaa.gov> wrote:
> >
> >> Good morning Julie
> >> Over the past few weeks, we've been seeing less compression with
> >> regrid_data_plane and significantly longer run times with
grid_stat.
> For
> >> regrid_data_plane, our CONUS files used to compress from ~44MB to
~23MB.
> >> Now they only compress to ~30MB. For grid_stat, a single model
and
> >> analysis grid would run in a minute or two. It now takes 8-10
minutes
> to
> >> run. We have been testing mainly with MET V9.0.2 and V9.0.3 out
of your
> >> directory on mars.
> >> Howard mentioned that the compression is controlled by the netcdf
> >> library. Starting with V9.0.1, you started linking to V4.5.0.
Prior to
> >> that, it looks like you had your own version of the library. I
assume
> >> the change was made just to use the standard WCOSS libraries. I
was
> just
> >> wondering if anyone else had experienced the same issues that we
have
> after
> >> this change?
> >> I ran some tests over the weekend reverting back to MET V8.1. I
didn't
> >> get in early enough this morning to check the file sizes of the
> >> regrid_data_plane output, but the grid_stat run times were still
longer
> >> than expected. I suspect the netcdf library is not the issue
there. If
> >> you know of any issues that might be causing longer grid_stat run
times
> or
> >> have any suggestions for other tests we can try, please let me
know.
> >> Thanks
> >> John
> >>
> >> ---------- Forwarded message ---------
> >> From: Howard Soh via RT <met_help at ucar.edu>
> >> Date: Thu, Aug 6, 2020 at 12:29 AM
> >> Subject: [rt.rap.ucar.edu #96126] Compression and grid_stat run
time
> >> To: <john.l.wagner at noaa.gov>
> >> Cc: <dana.strom at noaa.gov>, <tamarah.curtis at noaa.gov>
> >>
> >>
> >> The compression is done by NetCDF library. The NetCDF output
(size
> >> difference) is caused by the NetCDF library, not by MET code.
Please
> check
> >> the NetCDF library version:
> >> ldd regrid_data_plane | grep netcdf
> >>
> >> The issue 1363 (applied 9.0.3 and 9.1 beta 2) improves the speed
of
> >> reading NetCDF input, and no update on writing NetCDF output.
> >>
> >> Please let us know if 9.0.3 runs slower than 9.0.2 with using the
same
> >> NetCDF library.
> >>
> >> Cheers,
> >> Howard
> >>
> >> Here are examples to check NetCDF library:
> >> ldd met-8.1/bin/regrid_data_plane | grep netcdf
> >> libnetcdf_c++4.so.1 =>
/usr/local/netcdf/lib/libnetcdf_c++4.so.1
> >> (0x00007fee87e70000)
> >> libnetcdf.so.7 => not found
> >> libnetcdf.so.13 => /usr/local/netcdf/lib/libnetcdf.so.13
> >> (0x00007fee83f05000)
> >>
> >> ldd met-9.0.2/bin/regrid_data_plane | grep netcdf
> >> libnetcdf_c++4.so.1 =>
> >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
> >> (0x00007fdf5d1b2000)
> >> libnetcdf.so.15 =>
> >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
> (0x00007fdf5ce31000)
> >>
> >>
> >> ldd met-9.0.3/bin/regrid_data_plane | grep netcdf
> >> libnetcdf_c++4.so.1 =>
> >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
> >> (0x00007f7dfe91d000)
> >> libnetcdf.so.15 =>
> >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
> (0x00007f7dfe59c000)
> >>
> >>
> >> On Wed Aug 05 08:36:09 2020, mccabe wrote:
> >> > Hi John,
> >> >
> >> > I see you have questions about compression in regards to run
time
> >> > length and file sizes. I have assigned this ticket to Howard
Soh.
> >> > Please allow a few business days for a response.
> >> >
> >> > Thanks,
> >> > George
> >>
> >>
> >>
> >>
> >>
> >> --
> >> John Wagner
> >> Verification Task Lead
> >> NOAA/National Weather Service
> >> Meteorological Development Laboratory
> >> Digital Forecast Services Division
> >> SSMC2 Room 10106
> >> Silver Spring, MD 20910
> >> (301) 427-9471 (office)
> >> (908) 902-4155 (cell/text)
> >>
> >
> >
> > --
> > Julie Prestopnik (she/her/hers)
> > Software Engineer
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > Phone: 303.497.8399
> > Email: jpresto at ucar.edu
> >
> > My working day may not be your working day. Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.
> >
>
>
> --
> John Wagner
> Verification Task Lead
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Division
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>
------------------------------------------------
Subject: Compression and grid_stat run time
From: John L Wagner - NOAA Federal
Time: Tue Aug 11 12:03:36 2020
Thanks John. Attached are the NDFD and URMA grib2 files that we are
starting with and the grid_stat config file we are using. The
regrid_data_plane command we are using is
regrid_data_plane <grib2 file> <CONUS mask> <output netCDF file>
-method
NEAREST -width 1 -v 0 -name "tt" -compress 9 -field "name=tt;
level=Z2;"
I have access to the prod machine (venus) so I have been running some
tests
today with grid_stat. I've run tests with V8.1 out of NCO's area and
with
V8.1, V9.0.1, V9.0.2, and V9.0.3 in Julie's area. The shortest and
longest
run times I've seen today were both with Julie's V8.1. This morning,
it
ran in about 5 minutes. At lunch time, it ran in about 40 minutes.
All
other runs took about 10 minutes.
Tests I am going to run this afternoon:
1. We are currently running with masks for all of the CONUS WFOs and
regions (about 135 masks). I'm going to test with just the CONUS
mask.
2. I'll need to look at what elements we use this grid_stat_config
file
for. For temperature, we only need the cnt file. I'll turn off the
other
files.
3. I am going to try on the Cray instead of the Dell. We've run into
several problems where the only logical problem is the read and write
speeds on the Dell (phase 3). If your tests on the project machine at
NCAR
don't show any issues, then this will continue to be what I think the
real
problem is.
If you need anything else from me, let me know.
Thanks
John
On Tue, Aug 11, 2020 at 11:57 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> John,
>
> I did see your email come through met-help last week. Howard is the
person
> who's worked the most on NetCDF compression, so I was glad he was
able to
> respond. Unfortunately, I don't have any quick and obvious answers
for you.
> We just posted the met-9.1 release yesterday. It sounds like you're
> weighing 2 main considerations... the output file size of
regrid_data_plane
> and the runtime of grid_stat.
>
> Seems like we should do some comparison tests between met-8.1 and
met-9.1.
> What I'd like to figure out is whether the longer runtimes are due
solely
> to the difference in NetCDF compression levels. Howard has told me
that
> writing highly compressed NetCDF files is much, much slower.
>
> We could pick a single case and run it through
regrid_data_plane/grid_stat
> 6 times... with no/medium/high compression and met-8.1/9.1. Then for
each
> configuration, note the runtime and file sizes. And I also want to
confirm
> that the output generated is the same (i.e. met-9.1 isn't writing
any extra
> output variables). WCOSS is down this week, but I could do this
testing on
> a project machine at NCAR. But I'd like to approximate the datasets
you're
> using on WCOSS.
>
> Can you point me to sample data files, the regrid_data_plane command
you're
> running, and the Grid-Stat configuration files you're using?
>
> Thanks,
> John
>
> On Mon, Aug 10, 2020 at 10:11 AM John L Wagner - NOAA Federal via RT
<
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126 >
> >
> > Thanks Julie. I did submit this to the help desk last week. That
is
> where
> > Howard mentioned checking the netcdf library. I'll submit again
to get
> > John's thoughts.
> > John, please see my email below. If you need any more information
please
> > let me know.
> > Thanks
> > John
> >
> > On Mon, Aug 10, 2020 at 12:05 PM Julie Prestopnik
<jpresto at ucar.edu>
> > wrote:
> >
> > > Hi John.
> > >
> > > I am not the best person to address these issues about
compression. I
> > can
> > > confirm that I started linking to v4.5.0 of the NetCDF library
for the
> > > reason you indicated - to use the standard WCOSS libraries, as
that is
> > how
> > > it will be done for the operational installation of MET.
> > >
> > > Could you please submit your questions about compression to
> > > met_help at ucar.edu so that your questions can be addressed by the
right
> > > person (likely John, who also happens to be on helpdesk today)?
> > >
> > > Thanks!
> > >
> > > Julie
> > >
> > > On Mon, Aug 10, 2020 at 8:12 AM John L Wagner - NOAA Federal <
> > > john.l.wagner at noaa.gov> wrote:
> > >
> > >> Good morning Julie
> > >> Over the past few weeks, we've been seeing less compression
with
> > >> regrid_data_plane and significantly longer run times with
grid_stat.
> > For
> > >> regrid_data_plane, our CONUS files used to compress from ~44MB
to
> ~23MB.
> > >> Now they only compress to ~30MB. For grid_stat, a single model
and
> > >> analysis grid would run in a minute or two. It now takes 8-10
minutes
> > to
> > >> run. We have been testing mainly with MET V9.0.2 and V9.0.3
out of
> your
> > >> directory on mars.
> > >> Howard mentioned that the compression is controlled by the
netcdf
> > >> library. Starting with V9.0.1, you started linking to V4.5.0.
Prior
> to
> > >> that, it looks like you had your own version of the library. I
assume
> > >> the change was made just to use the standard WCOSS libraries.
I was
> > just
> > >> wondering if anyone else had experienced the same issues that
we have
> > after
> > >> this change?
> > >> I ran some tests over the weekend reverting back to MET V8.1.
I
> didn't
> > >> get in early enough this morning to check the file sizes of the
> > >> regrid_data_plane output, but the grid_stat run times were
still
> longer
> > >> than expected. I suspect the netcdf library is not the issue
there.
> If
> > >> you know of any issues that might be causing longer grid_stat
run
> times
> > or
> > >> have any suggestions for other tests we can try, please let me
know.
> > >> Thanks
> > >> John
> > >>
> > >> ---------- Forwarded message ---------
> > >> From: Howard Soh via RT <met_help at ucar.edu>
> > >> Date: Thu, Aug 6, 2020 at 12:29 AM
> > >> Subject: [rt.rap.ucar.edu #96126] Compression and grid_stat run
time
> > >> To: <john.l.wagner at noaa.gov>
> > >> Cc: <dana.strom at noaa.gov>, <tamarah.curtis at noaa.gov>
> > >>
> > >>
> > >> The compression is done by NetCDF library. The NetCDF output
(size
> > >> difference) is caused by the NetCDF library, not by MET code.
Please
> > check
> > >> the NetCDF library version:
> > >> ldd regrid_data_plane | grep netcdf
> > >>
> > >> The issue 1363 (applied 9.0.3 and 9.1 beta 2) improves the
speed of
> > >> reading NetCDF input, and no update on writing NetCDF output.
> > >>
> > >> Please let us know if 9.0.3 runs slower than 9.0.2 with using
the same
> > >> NetCDF library.
> > >>
> > >> Cheers,
> > >> Howard
> > >>
> > >> Here are examples to check NetCDF library:
> > >> ldd met-8.1/bin/regrid_data_plane | grep netcdf
> > >> libnetcdf_c++4.so.1 =>
> /usr/local/netcdf/lib/libnetcdf_c++4.so.1
> > >> (0x00007fee87e70000)
> > >> libnetcdf.so.7 => not found
> > >> libnetcdf.so.13 =>
/usr/local/netcdf/lib/libnetcdf.so.13
> > >> (0x00007fee83f05000)
> > >>
> > >> ldd met-9.0.2/bin/regrid_data_plane | grep netcdf
> > >> libnetcdf_c++4.so.1 =>
> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
> > >> (0x00007fdf5d1b2000)
> > >> libnetcdf.so.15 =>
> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
> > (0x00007fdf5ce31000)
> > >>
> > >>
> > >> ldd met-9.0.3/bin/regrid_data_plane | grep netcdf
> > >> libnetcdf_c++4.so.1 =>
> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
> > >> (0x00007f7dfe91d000)
> > >> libnetcdf.so.15 =>
> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
> > (0x00007f7dfe59c000)
> > >>
> > >>
> > >> On Wed Aug 05 08:36:09 2020, mccabe wrote:
> > >> > Hi John,
> > >> >
> > >> > I see you have questions about compression in regards to run
time
> > >> > length and file sizes. I have assigned this ticket to Howard
Soh.
> > >> > Please allow a few business days for a response.
> > >> >
> > >> > Thanks,
> > >> > George
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> --
> > >> John Wagner
> > >> Verification Task Lead
> > >> NOAA/National Weather Service
> > >> Meteorological Development Laboratory
> > >> Digital Forecast Services Division
> > >> SSMC2 Room 10106
> > >> Silver Spring, MD 20910
> > >> (301) 427-9471 (office)
> > >> (908) 902-4155 (cell/text)
> > >>
> > >
> > >
> > > --
> > > Julie Prestopnik (she/her/hers)
> > > Software Engineer
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > Phone: 303.497.8399
> > > Email: jpresto at ucar.edu
> > >
> > > My working day may not be your working day. Please do not feel
obliged
> > to
> > > reply to this email outside of your normal working hours.
> > >
> >
> >
> > --
> > John Wagner
> > Verification Task Lead
> > NOAA/National Weather Service
> > Meteorological Development Laboratory
> > Digital Forecast Services Division
> > SSMC2 Room 10106
> > Silver Spring, MD 20910
> > (301) 427-9471 (office)
> > (908) 902-4155 (cell/text)
> >
> >
>
>
--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
------------------------------------------------
Subject: Compression and grid_stat run time
From: John L Wagner - NOAA Federal
Time: Wed Aug 12 06:42:42 2020
Good morning John
I ran the tests yesterday afternoon that I mentioned in my previous
email.
Running for just the CNT output and using the Cray's ptmp to read and
write
data had no effect on the run times. Running for just the CONUS mask
initially took about 57 seconds to run with compressed files.
Switching to
uncompressed files, it took about 38 seconds to run for just the CONUS
mask
and 8 minutes and 40 seconds for all masks.
This morning, I tested running with just the CONUS mask again and used
the
uncompressed files. For some reason this morning, grid_stat runs in
about
5-6 seconds, which is what I was seeing a few weeks ago. I made no
other
changes to my scripts or modules other than to limit the runs to just
the
CONUS mask again. The start and stop times from my logs are below.
I'm really at a loss as to what could be causing these fluctuations in
run
times.
Thanks
John
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_ndfdconc2p519110112ttf006
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
-v 0 -outdir /gpfs/dell2/ptmp/John.L.Wagner/ndfd/co/dbtx/1911/01/18/tt
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_ndfd
*GRID STAT STARTWed Aug 12 12:17:20 UTC 2020*
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=18446744072918741855
*GRID STAT STOPWed Aug 12 12:17:26 UTC 2020*
missedTracking not set. Running grid_stat for blend 201911017
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_blendconc2p519110107ttf011
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
-v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/blend/co/dbtx/1911/01/18/tt
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_blend
*GRID STAT STARTWed Aug 12 12:17:26 UTC 2020*
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=18446744073019405151
*GRID STAT STOPWed Aug 12 12:17:31 UTC 2020*
missedTracking not set. Running grid_stat for blendp 201911017
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_blendpconc2p519110107ttf011
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
-v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/blendp/co/dbtx/1911/01/18/tt
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_blendp
*GRID STAT STARTWed Aug 12 12:17:31 UTC 2020*
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=18446744073103291231
*GRID STAT STOPWed Aug 12 12:17:36 UTC 2020*
missedTracking not set. Running grid_stat for gmos 201911010
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_gmosconc2p519110100ttf018
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
-v 0 -outdir /gpfs/dell2/ptmp/John.L.Wagner/gmos/co/dbtx/1911/01/18/tt
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_gmos
*GRID STAT STARTWed Aug 12 12:17:37 UTC 2020*
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=18446744073203954527
*GRID STAT STOPWed Aug 12 12:17:42 UTC 2020*
On Tue, Aug 11, 2020 at 2:03 PM John L Wagner - NOAA Federal <
john.l.wagner at noaa.gov> wrote:
> Thanks John. Attached are the NDFD and URMA grib2 files that we are
> starting with and the grid_stat config file we are using. The
> regrid_data_plane command we are using is
>
> regrid_data_plane <grib2 file> <CONUS mask> <output netCDF file>
-method
> NEAREST -width 1 -v 0 -name "tt" -compress 9 -field "name=tt;
level=Z2;"
>
> I have access to the prod machine (venus) so I have been running
some
> tests today with grid_stat. I've run tests with V8.1 out of NCO's
area and
> with V8.1, V9.0.1, V9.0.2, and V9.0.3 in Julie's area. The shortest
and
> longest run times I've seen today were both with Julie's V8.1. This
> morning, it ran in about 5 minutes. At lunch time, it ran in about
40
> minutes. All other runs took about 10 minutes.
>
> Tests I am going to run this afternoon:
> 1. We are currently running with masks for all of the CONUS WFOs
and
> regions (about 135 masks). I'm going to test with just the CONUS
mask.
> 2. I'll need to look at what elements we use this grid_stat_config
file
> for. For temperature, we only need the cnt file. I'll turn off the
other
> files.
> 3. I am going to try on the Cray instead of the Dell. We've run
into
> several problems where the only logical problem is the read and
write
> speeds on the Dell (phase 3). If your tests on the project machine
at NCAR
> don't show any issues, then this will continue to be what I think
the real
> problem is.
>
> If you need anything else from me, let me know.
> Thanks
> John
>
> On Tue, Aug 11, 2020 at 11:57 AM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> John,
>>
>> I did see your email come through met-help last week. Howard is the
person
>> who's worked the most on NetCDF compression, so I was glad he was
able to
>> respond. Unfortunately, I don't have any quick and obvious answers
for
>> you.
>> We just posted the met-9.1 release yesterday. It sounds like you're
>> weighing 2 main considerations... the output file size of
>> regrid_data_plane
>> and the runtime of grid_stat.
>>
>> Seems like we should do some comparison tests between met-8.1 and
met-9.1.
>> What I'd like to figure out is whether the longer runtimes are due
solely
>> to the difference in NetCDF compression levels. Howard has told me
that
>> writing highly compressed NetCDF files is much, much slower.
>>
>> We could pick a single case and run it through
regrid_data_plane/grid_stat
>> 6 times... with no/medium/high compression and met-8.1/9.1. Then
for each
>> configuration, note the runtime and file sizes. And I also want to
confirm
>> that the output generated is the same (i.e. met-9.1 isn't writing
any
>> extra
>> output variables). WCOSS is down this week, but I could do this
testing on
>> a project machine at NCAR. But I'd like to approximate the datasets
you're
>> using on WCOSS.
>>
>> Can you point me to sample data files, the regrid_data_plane
command
>> you're
>> running, and the Grid-Stat configuration files you're using?
>>
>> Thanks,
>> John
>>
>> On Mon, Aug 10, 2020 at 10:11 AM John L Wagner - NOAA Federal via
RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126 >
>> >
>> > Thanks Julie. I did submit this to the help desk last week.
That is
>> where
>> > Howard mentioned checking the netcdf library. I'll submit again
to get
>> > John's thoughts.
>> > John, please see my email below. If you need any more
information
>> please
>> > let me know.
>> > Thanks
>> > John
>> >
>> > On Mon, Aug 10, 2020 at 12:05 PM Julie Prestopnik
<jpresto at ucar.edu>
>> > wrote:
>> >
>> > > Hi John.
>> > >
>> > > I am not the best person to address these issues about
compression. I
>> > can
>> > > confirm that I started linking to v4.5.0 of the NetCDF library
for the
>> > > reason you indicated - to use the standard WCOSS libraries, as
that is
>> > how
>> > > it will be done for the operational installation of MET.
>> > >
>> > > Could you please submit your questions about compression to
>> > > met_help at ucar.edu so that your questions can be addressed by
the
>> right
>> > > person (likely John, who also happens to be on helpdesk today)?
>> > >
>> > > Thanks!
>> > >
>> > > Julie
>> > >
>> > > On Mon, Aug 10, 2020 at 8:12 AM John L Wagner - NOAA Federal <
>> > > john.l.wagner at noaa.gov> wrote:
>> > >
>> > >> Good morning Julie
>> > >> Over the past few weeks, we've been seeing less compression
with
>> > >> regrid_data_plane and significantly longer run times with
grid_stat.
>> > For
>> > >> regrid_data_plane, our CONUS files used to compress from ~44MB
to
>> ~23MB.
>> > >> Now they only compress to ~30MB. For grid_stat, a single
model and
>> > >> analysis grid would run in a minute or two. It now takes 8-10
>> minutes
>> > to
>> > >> run. We have been testing mainly with MET V9.0.2 and V9.0.3
out of
>> your
>> > >> directory on mars.
>> > >> Howard mentioned that the compression is controlled by the
netcdf
>> > >> library. Starting with V9.0.1, you started linking to V4.5.0.
>> Prior to
>> > >> that, it looks like you had your own version of the library.
I
>> assume
>> > >> the change was made just to use the standard WCOSS libraries.
I was
>> > just
>> > >> wondering if anyone else had experienced the same issues that
we have
>> > after
>> > >> this change?
>> > >> I ran some tests over the weekend reverting back to MET V8.1.
I
>> didn't
>> > >> get in early enough this morning to check the file sizes of
the
>> > >> regrid_data_plane output, but the grid_stat run times were
still
>> longer
>> > >> than expected. I suspect the netcdf library is not the issue
>> there. If
>> > >> you know of any issues that might be causing longer grid_stat
run
>> times
>> > or
>> > >> have any suggestions for other tests we can try, please let me
know.
>> > >> Thanks
>> > >> John
>> > >>
>> > >> ---------- Forwarded message ---------
>> > >> From: Howard Soh via RT <met_help at ucar.edu>
>> > >> Date: Thu, Aug 6, 2020 at 12:29 AM
>> > >> Subject: [rt.rap.ucar.edu #96126] Compression and grid_stat
run time
>> > >> To: <john.l.wagner at noaa.gov>
>> > >> Cc: <dana.strom at noaa.gov>, <tamarah.curtis at noaa.gov>
>> > >>
>> > >>
>> > >> The compression is done by NetCDF library. The NetCDF output
(size
>> > >> difference) is caused by the NetCDF library, not by MET code.
Please
>> > check
>> > >> the NetCDF library version:
>> > >> ldd regrid_data_plane | grep netcdf
>> > >>
>> > >> The issue 1363 (applied 9.0.3 and 9.1 beta 2) improves the
speed of
>> > >> reading NetCDF input, and no update on writing NetCDF output.
>> > >>
>> > >> Please let us know if 9.0.3 runs slower than 9.0.2 with using
the
>> same
>> > >> NetCDF library.
>> > >>
>> > >> Cheers,
>> > >> Howard
>> > >>
>> > >> Here are examples to check NetCDF library:
>> > >> ldd met-8.1/bin/regrid_data_plane | grep netcdf
>> > >> libnetcdf_c++4.so.1 =>
>> /usr/local/netcdf/lib/libnetcdf_c++4.so.1
>> > >> (0x00007fee87e70000)
>> > >> libnetcdf.so.7 => not found
>> > >> libnetcdf.so.13 =>
/usr/local/netcdf/lib/libnetcdf.so.13
>> > >> (0x00007fee83f05000)
>> > >>
>> > >> ldd met-9.0.2/bin/regrid_data_plane | grep netcdf
>> > >> libnetcdf_c++4.so.1 =>
>> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
>> > >> (0x00007fdf5d1b2000)
>> > >> libnetcdf.so.15 =>
>> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
>> > (0x00007fdf5ce31000)
>> > >>
>> > >>
>> > >> ldd met-9.0.3/bin/regrid_data_plane | grep netcdf
>> > >> libnetcdf_c++4.so.1 =>
>> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
>> > >> (0x00007f7dfe91d000)
>> > >> libnetcdf.so.15 =>
>> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
>> > (0x00007f7dfe59c000)
>> > >>
>> > >>
>> > >> On Wed Aug 05 08:36:09 2020, mccabe wrote:
>> > >> > Hi John,
>> > >> >
>> > >> > I see you have questions about compression in regards to run
time
>> > >> > length and file sizes. I have assigned this ticket to Howard
Soh.
>> > >> > Please allow a few business days for a response.
>> > >> >
>> > >> > Thanks,
>> > >> > George
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> John Wagner
>> > >> Verification Task Lead
>> > >> NOAA/National Weather Service
>> > >> Meteorological Development Laboratory
>> > >> Digital Forecast Services Division
>> > >> SSMC2 Room 10106
>> > >> Silver Spring, MD 20910
>> > >> (301) 427-9471 (office)
>> > >> (908) 902-4155 (cell/text)
>> > >>
>> > >
>> > >
>> > > --
>> > > Julie Prestopnik (she/her/hers)
>> > > Software Engineer
>> > > National Center for Atmospheric Research
>> > > Research Applications Laboratory
>> > > Phone: 303.497.8399
>> > > Email: jpresto at ucar.edu
>> > >
>> > > My working day may not be your working day. Please do not feel
>> obliged
>> > to
>> > > reply to this email outside of your normal working hours.
>> > >
>> >
>> >
>> > --
>> > John Wagner
>> > Verification Task Lead
>> > NOAA/National Weather Service
>> > Meteorological Development Laboratory
>> > Digital Forecast Services Division
>> > SSMC2 Room 10106
>> > Silver Spring, MD 20910
>> > (301) 427-9471 (office)
>> > (908) 902-4155 (cell/text)
>> >
>> >
>>
>>
>
> --
> John Wagner
> Verification Task Lead
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Division
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
--
John Wagner
Verification Task Lead
NOAA/National Weather Service
Meteorological Development Laboratory
Digital Forecast Services Division
SSMC2 Room 10106
Silver Spring, MD 20910
(301) 427-9471 (office)
(908) 902-4155 (cell/text)
------------------------------------------------
Subject: Compression and grid_stat run time
From: John Halley Gotway
Time: Thu Aug 13 11:53:20 2020
John,
Thanks for sending this data. Unfortunately, I didn't get a chance to
dig
into the details before heading out of town. But I'll be back from PTO
on
Monday.
Thanks,
John
On Wed, Aug 12, 2020 at 6:42 AM John L Wagner - NOAA Federal via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126 >
>
> Good morning John
> I ran the tests yesterday afternoon that I mentioned in my previous
email.
> Running for just the CNT output and using the Cray's ptmp to read
and write
> data had no effect on the run times. Running for just the CONUS
mask
> initially took about 57 seconds to run with compressed files.
Switching to
> uncompressed files, it took about 38 seconds to run for just the
CONUS mask
> and 8 minutes and 40 seconds for all masks.
> This morning, I tested running with just the CONUS mask again and
used the
> uncompressed files. For some reason this morning, grid_stat runs in
about
> 5-6 seconds, which is what I was seeing a few weeks ago. I made no
other
> changes to my scripts or modules other than to limit the runs to
just the
> CONUS mask again. The start and stop times from my logs are below.
> I'm really at a loss as to what could be causing these fluctuations
in run
> times.
> Thanks
> John
>
>
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_ndfdconc2p519110112ttf006
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/ndfd/co/dbtx/1911/01/18/tt
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_ndfd
>
> *GRID STAT STARTWed Aug 12 12:17:20 UTC 2020*
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744072918741855
>
> *GRID STAT STOPWed Aug 12 12:17:26 UTC 2020*
> missedTracking not set. Running grid_stat for blend 201911017
>
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_blendconc2p519110107ttf011
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/blend/co/dbtx/1911/01/18/tt
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_blend
>
> *GRID STAT STARTWed Aug 12 12:17:26 UTC 2020*
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744073019405151
>
> *GRID STAT STOPWed Aug 12 12:17:31 UTC 2020*
> missedTracking not set. Running grid_stat for blendp 201911017
>
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_blendpconc2p519110107ttf011
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/blendp/co/dbtx/1911/01/18/tt
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_blendp
>
> *GRID STAT STARTWed Aug 12 12:17:31 UTC 2020*
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744073103291231
>
> *GRID STAT STOPWed Aug 12 12:17:36 UTC 2020*
> missedTracking not set. Running grid_stat for gmos 201911010
>
>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_gmosconc2p519110100ttf018
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/gmos/co/dbtx/1911/01/18/tt
>
>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_gmos
>
> *GRID STAT STARTWed Aug 12 12:17:37 UTC 2020*
> GSL_RNG_TYPE=mt19937
> GSL_RNG_SEED=18446744073203954527
>
> *GRID STAT STOPWed Aug 12 12:17:42 UTC 2020*
>
> On Tue, Aug 11, 2020 at 2:03 PM John L Wagner - NOAA Federal <
> john.l.wagner at noaa.gov> wrote:
>
> > Thanks John. Attached are the NDFD and URMA grib2 files that we
are
> > starting with and the grid_stat config file we are using. The
> > regrid_data_plane command we are using is
> >
> > regrid_data_plane <grib2 file> <CONUS mask> <output netCDF file>
-method
> > NEAREST -width 1 -v 0 -name "tt" -compress 9 -field "name=tt;
level=Z2;"
> >
> > I have access to the prod machine (venus) so I have been running
some
> > tests today with grid_stat. I've run tests with V8.1 out of NCO's
area
> and
> > with V8.1, V9.0.1, V9.0.2, and V9.0.3 in Julie's area. The
shortest and
> > longest run times I've seen today were both with Julie's V8.1.
This
> > morning, it ran in about 5 minutes. At lunch time, it ran in
about 40
> > minutes. All other runs took about 10 minutes.
> >
> > Tests I am going to run this afternoon:
> > 1. We are currently running with masks for all of the CONUS WFOs
and
> > regions (about 135 masks). I'm going to test with just the CONUS
mask.
> > 2. I'll need to look at what elements we use this
grid_stat_config file
> > for. For temperature, we only need the cnt file. I'll turn off
the
> other
> > files.
> > 3. I am going to try on the Cray instead of the Dell. We've run
into
> > several problems where the only logical problem is the read and
write
> > speeds on the Dell (phase 3). If your tests on the project
machine at
> NCAR
> > don't show any issues, then this will continue to be what I think
the
> real
> > problem is.
> >
> > If you need anything else from me, let me know.
> > Thanks
> > John
> >
> > On Tue, Aug 11, 2020 at 11:57 AM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> John,
> >>
> >> I did see your email come through met-help last week. Howard is
the
> person
> >> who's worked the most on NetCDF compression, so I was glad he was
able
> to
> >> respond. Unfortunately, I don't have any quick and obvious
answers for
> >> you.
> >> We just posted the met-9.1 release yesterday. It sounds like
you're
> >> weighing 2 main considerations... the output file size of
> >> regrid_data_plane
> >> and the runtime of grid_stat.
> >>
> >> Seems like we should do some comparison tests between met-8.1 and
> met-9.1.
> >> What I'd like to figure out is whether the longer runtimes are
due
> solely
> >> to the difference in NetCDF compression levels. Howard has told
me that
> >> writing highly compressed NetCDF files is much, much slower.
> >>
> >> We could pick a single case and run it through
> regrid_data_plane/grid_stat
> >> 6 times... with no/medium/high compression and met-8.1/9.1. Then
for
> each
> >> configuration, note the runtime and file sizes. And I also want
to
> confirm
> >> that the output generated is the same (i.e. met-9.1 isn't writing
any
> >> extra
> >> output variables). WCOSS is down this week, but I could do this
testing
> on
> >> a project machine at NCAR. But I'd like to approximate the
datasets
> you're
> >> using on WCOSS.
> >>
> >> Can you point me to sample data files, the regrid_data_plane
command
> >> you're
> >> running, and the Grid-Stat configuration files you're using?
> >>
> >> Thanks,
> >> John
> >>
> >> On Mon, Aug 10, 2020 at 10:11 AM John L Wagner - NOAA Federal via
RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126 >
> >> >
> >> > Thanks Julie. I did submit this to the help desk last week.
That is
> >> where
> >> > Howard mentioned checking the netcdf library. I'll submit
again to
> get
> >> > John's thoughts.
> >> > John, please see my email below. If you need any more
information
> >> please
> >> > let me know.
> >> > Thanks
> >> > John
> >> >
> >> > On Mon, Aug 10, 2020 at 12:05 PM Julie Prestopnik
<jpresto at ucar.edu>
> >> > wrote:
> >> >
> >> > > Hi John.
> >> > >
> >> > > I am not the best person to address these issues about
> compression. I
> >> > can
> >> > > confirm that I started linking to v4.5.0 of the NetCDF
library for
> the
> >> > > reason you indicated - to use the standard WCOSS libraries,
as that
> is
> >> > how
> >> > > it will be done for the operational installation of MET.
> >> > >
> >> > > Could you please submit your questions about compression to
> >> > > met_help at ucar.edu so that your questions can be addressed by
the
> >> right
> >> > > person (likely John, who also happens to be on helpdesk
today)?
> >> > >
> >> > > Thanks!
> >> > >
> >> > > Julie
> >> > >
> >> > > On Mon, Aug 10, 2020 at 8:12 AM John L Wagner - NOAA Federal
<
> >> > > john.l.wagner at noaa.gov> wrote:
> >> > >
> >> > >> Good morning Julie
> >> > >> Over the past few weeks, we've been seeing less compression
with
> >> > >> regrid_data_plane and significantly longer run times with
> grid_stat.
> >> > For
> >> > >> regrid_data_plane, our CONUS files used to compress from
~44MB to
> >> ~23MB.
> >> > >> Now they only compress to ~30MB. For grid_stat, a single
model and
> >> > >> analysis grid would run in a minute or two. It now takes 8-
10
> >> minutes
> >> > to
> >> > >> run. We have been testing mainly with MET V9.0.2 and V9.0.3
out of
> >> your
> >> > >> directory on mars.
> >> > >> Howard mentioned that the compression is controlled by the
netcdf
> >> > >> library. Starting with V9.0.1, you started linking to
V4.5.0.
> >> Prior to
> >> > >> that, it looks like you had your own version of the library.
I
> >> assume
> >> > >> the change was made just to use the standard WCOSS
libraries. I
> was
> >> > just
> >> > >> wondering if anyone else had experienced the same issues
that we
> have
> >> > after
> >> > >> this change?
> >> > >> I ran some tests over the weekend reverting back to MET
V8.1. I
> >> didn't
> >> > >> get in early enough this morning to check the file sizes of
the
> >> > >> regrid_data_plane output, but the grid_stat run times were
still
> >> longer
> >> > >> than expected. I suspect the netcdf library is not the
issue
> >> there. If
> >> > >> you know of any issues that might be causing longer
grid_stat run
> >> times
> >> > or
> >> > >> have any suggestions for other tests we can try, please let
me
> know.
> >> > >> Thanks
> >> > >> John
> >> > >>
> >> > >> ---------- Forwarded message ---------
> >> > >> From: Howard Soh via RT <met_help at ucar.edu>
> >> > >> Date: Thu, Aug 6, 2020 at 12:29 AM
> >> > >> Subject: [rt.rap.ucar.edu #96126] Compression and grid_stat
run
> time
> >> > >> To: <john.l.wagner at noaa.gov>
> >> > >> Cc: <dana.strom at noaa.gov>, <tamarah.curtis at noaa.gov>
> >> > >>
> >> > >>
> >> > >> The compression is done by NetCDF library. The NetCDF output
(size
> >> > >> difference) is caused by the NetCDF library, not by MET
code.
> Please
> >> > check
> >> > >> the NetCDF library version:
> >> > >> ldd regrid_data_plane | grep netcdf
> >> > >>
> >> > >> The issue 1363 (applied 9.0.3 and 9.1 beta 2) improves the
speed of
> >> > >> reading NetCDF input, and no update on writing NetCDF
output.
> >> > >>
> >> > >> Please let us know if 9.0.3 runs slower than 9.0.2 with
using the
> >> same
> >> > >> NetCDF library.
> >> > >>
> >> > >> Cheers,
> >> > >> Howard
> >> > >>
> >> > >> Here are examples to check NetCDF library:
> >> > >> ldd met-8.1/bin/regrid_data_plane | grep netcdf
> >> > >> libnetcdf_c++4.so.1 =>
> >> /usr/local/netcdf/lib/libnetcdf_c++4.so.1
> >> > >> (0x00007fee87e70000)
> >> > >> libnetcdf.so.7 => not found
> >> > >> libnetcdf.so.13 =>
/usr/local/netcdf/lib/libnetcdf.so.13
> >> > >> (0x00007fee83f05000)
> >> > >>
> >> > >> ldd met-9.0.2/bin/regrid_data_plane | grep netcdf
> >> > >> libnetcdf_c++4.so.1 =>
> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
> >> > >> (0x00007fdf5d1b2000)
> >> > >> libnetcdf.so.15 =>
> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
> >> > (0x00007fdf5ce31000)
> >> > >>
> >> > >>
> >> > >> ldd met-9.0.3/bin/regrid_data_plane | grep netcdf
> >> > >> libnetcdf_c++4.so.1 =>
> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
> >> > >> (0x00007f7dfe91d000)
> >> > >> libnetcdf.so.15 =>
> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
> >> > (0x00007f7dfe59c000)
> >> > >>
> >> > >>
> >> > >> On Wed Aug 05 08:36:09 2020, mccabe wrote:
> >> > >> > Hi John,
> >> > >> >
> >> > >> > I see you have questions about compression in regards to
run time
> >> > >> > length and file sizes. I have assigned this ticket to
Howard Soh.
> >> > >> > Please allow a few business days for a response.
> >> > >> >
> >> > >> > Thanks,
> >> > >> > George
> >> > >>
> >> > >>
> >> > >>
> >> > >>
> >> > >>
> >> > >> --
> >> > >> John Wagner
> >> > >> Verification Task Lead
> >> > >> NOAA/National Weather Service
> >> > >> Meteorological Development Laboratory
> >> > >> Digital Forecast Services Division
> >> > >> SSMC2 Room 10106
> >> > >> Silver Spring, MD 20910
> >> > >> (301) 427-9471 (office)
> >> > >> (908) 902-4155 (cell/text)
> >> > >>
> >> > >
> >> > >
> >> > > --
> >> > > Julie Prestopnik (she/her/hers)
> >> > > Software Engineer
> >> > > National Center for Atmospheric Research
> >> > > Research Applications Laboratory
> >> > > Phone: 303.497.8399
> >> > > Email: jpresto at ucar.edu
> >> > >
> >> > > My working day may not be your working day. Please do not
feel
> >> obliged
> >> > to
> >> > > reply to this email outside of your normal working hours.
> >> > >
> >> >
> >> >
> >> > --
> >> > John Wagner
> >> > Verification Task Lead
> >> > NOAA/National Weather Service
> >> > Meteorological Development Laboratory
> >> > Digital Forecast Services Division
> >> > SSMC2 Room 10106
> >> > Silver Spring, MD 20910
> >> > (301) 427-9471 (office)
> >> > (908) 902-4155 (cell/text)
> >> >
> >> >
> >>
> >>
> >
> > --
> > John Wagner
> > Verification Task Lead
> > NOAA/National Weather Service
> > Meteorological Development Laboratory
> > Digital Forecast Services Division
> > SSMC2 Room 10106
> > Silver Spring, MD 20910
> > (301) 427-9471 (office)
> > (908) 902-4155 (cell/text)
> >
>
>
> --
> John Wagner
> Verification Task Lead
> NOAA/National Weather Service
> Meteorological Development Laboratory
> Digital Forecast Services Division
> SSMC2 Room 10106
> Silver Spring, MD 20910
> (301) 427-9471 (office)
> (908) 902-4155 (cell/text)
>
>
------------------------------------------------
Subject: Compression and grid_stat run time
From: John Halley Gotway
Time: Thu Aug 27 14:04:02 2020
John,
I'm going to go ahead and resolve this old met-help ticket. As we
discussed
at the NOAA METplus telecon earlier this week, you're going to try
switching to using GRIB input files and check to see what impact that
has
on runtimes.
Thanks,
John HG
On Thu, Aug 13, 2020 at 11:53 AM John Halley Gotway <johnhg at ucar.edu>
wrote:
> John,
>
> Thanks for sending this data. Unfortunately, I didn't get a chance
to dig
> into the details before heading out of town. But I'll be back from
PTO on
> Monday.
>
> Thanks,
> John
>
> On Wed, Aug 12, 2020 at 6:42 AM John L Wagner - NOAA Federal via RT
<
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126 >
>>
>> Good morning John
>> I ran the tests yesterday afternoon that I mentioned in my previous
email.
>> Running for just the CNT output and using the Cray's ptmp to read
and
>> write
>> data had no effect on the run times. Running for just the CONUS
mask
>> initially took about 57 seconds to run with compressed files.
Switching
>> to
>> uncompressed files, it took about 38 seconds to run for just the
CONUS
>> mask
>> and 8 minutes and 40 seconds for all masks.
>> This morning, I tested running with just the CONUS mask again and
used the
>> uncompressed files. For some reason this morning, grid_stat runs
in about
>> 5-6 seconds, which is what I was seeing a few weeks ago. I made no
other
>> changes to my scripts or modules other than to limit the runs to
just the
>> CONUS mask again. The start and stop times from my logs are below.
>> I'm really at a loss as to what could be causing these fluctuations
in run
>> times.
>> Thanks
>> John
>>
>>
>>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_ndfdconc2p519110112ttf006
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
>> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/ndfd/co/dbtx/1911/01/18/tt
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_ndfd
>>
>> *GRID STAT STARTWed Aug 12 12:17:20 UTC 2020*
>> GSL_RNG_TYPE=mt19937
>> GSL_RNG_SEED=18446744072918741855
>>
>> *GRID STAT STOPWed Aug 12 12:17:26 UTC 2020*
>> missedTracking not set. Running grid_stat for blend 201911017
>>
>>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_blendconc2p519110107ttf011
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
>> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/blend/co/dbtx/1911/01/18/tt
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_blend
>>
>> *GRID STAT STARTWed Aug 12 12:17:26 UTC 2020*
>> GSL_RNG_TYPE=mt19937
>> GSL_RNG_SEED=18446744073019405151
>>
>> *GRID STAT STOPWed Aug 12 12:17:31 UTC 2020*
>> missedTracking not set. Running grid_stat for blendp 201911017
>>
>>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_blendpconc2p519110107ttf011
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
>> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/blendp/co/dbtx/1911/01/18/tt
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_blendp
>>
>> *GRID STAT STARTWed Aug 12 12:17:31 UTC 2020*
>> GSL_RNG_TYPE=mt19937
>> GSL_RNG_SEED=18446744073103291231
>>
>> *GRID STAT STOPWed Aug 12 12:17:36 UTC 2020*
>> missedTracking not set. Running grid_stat for gmos 201911010
>>
>>
/gpfs/dell2/emc/verification/noscrub/Julie.Prestopnik/met/9.0.2/bin/grid_stat
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_gmosconc2p519110100ttf018
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/match_co_tt_2019110118.80422/006/matched_urmaconc2p519110118tt_006
>> -v 0 -outdir
/gpfs/dell2/ptmp/John.L.Wagner/gmos/co/dbtx/1911/01/18/tt
>>
>>
/gpfs/dell2/ptmp/John.L.Wagner/matching_urma_2019110118_co_tt.80422/grid_stat_co_006_gmos
>>
>> *GRID STAT STARTWed Aug 12 12:17:37 UTC 2020*
>> GSL_RNG_TYPE=mt19937
>> GSL_RNG_SEED=18446744073203954527
>>
>> *GRID STAT STOPWed Aug 12 12:17:42 UTC 2020*
>>
>> On Tue, Aug 11, 2020 at 2:03 PM John L Wagner - NOAA Federal <
>> john.l.wagner at noaa.gov> wrote:
>>
>> > Thanks John. Attached are the NDFD and URMA grib2 files that we
are
>> > starting with and the grid_stat config file we are using. The
>> > regrid_data_plane command we are using is
>> >
>> > regrid_data_plane <grib2 file> <CONUS mask> <output netCDF file>
-method
>> > NEAREST -width 1 -v 0 -name "tt" -compress 9 -field "name=tt;
level=Z2;"
>> >
>> > I have access to the prod machine (venus) so I have been running
some
>> > tests today with grid_stat. I've run tests with V8.1 out of
NCO's area
>> and
>> > with V8.1, V9.0.1, V9.0.2, and V9.0.3 in Julie's area. The
shortest and
>> > longest run times I've seen today were both with Julie's V8.1.
This
>> > morning, it ran in about 5 minutes. At lunch time, it ran in
about 40
>> > minutes. All other runs took about 10 minutes.
>> >
>> > Tests I am going to run this afternoon:
>> > 1. We are currently running with masks for all of the CONUS WFOs
and
>> > regions (about 135 masks). I'm going to test with just the CONUS
mask.
>> > 2. I'll need to look at what elements we use this
grid_stat_config file
>> > for. For temperature, we only need the cnt file. I'll turn off
the
>> other
>> > files.
>> > 3. I am going to try on the Cray instead of the Dell. We've run
into
>> > several problems where the only logical problem is the read and
write
>> > speeds on the Dell (phase 3). If your tests on the project
machine at
>> NCAR
>> > don't show any issues, then this will continue to be what I think
the
>> real
>> > problem is.
>> >
>> > If you need anything else from me, let me know.
>> > Thanks
>> > John
>> >
>> > On Tue, Aug 11, 2020 at 11:57 AM John Halley Gotway via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> >> John,
>> >>
>> >> I did see your email come through met-help last week. Howard is
the
>> person
>> >> who's worked the most on NetCDF compression, so I was glad he
was able
>> to
>> >> respond. Unfortunately, I don't have any quick and obvious
answers for
>> >> you.
>> >> We just posted the met-9.1 release yesterday. It sounds like
you're
>> >> weighing 2 main considerations... the output file size of
>> >> regrid_data_plane
>> >> and the runtime of grid_stat.
>> >>
>> >> Seems like we should do some comparison tests between met-8.1
and
>> met-9.1.
>> >> What I'd like to figure out is whether the longer runtimes are
due
>> solely
>> >> to the difference in NetCDF compression levels. Howard has told
me that
>> >> writing highly compressed NetCDF files is much, much slower.
>> >>
>> >> We could pick a single case and run it through
>> regrid_data_plane/grid_stat
>> >> 6 times... with no/medium/high compression and met-8.1/9.1. Then
for
>> each
>> >> configuration, note the runtime and file sizes. And I also want
to
>> confirm
>> >> that the output generated is the same (i.e. met-9.1 isn't
writing any
>> >> extra
>> >> output variables). WCOSS is down this week, but I could do this
>> testing on
>> >> a project machine at NCAR. But I'd like to approximate the
datasets
>> you're
>> >> using on WCOSS.
>> >>
>> >> Can you point me to sample data files, the regrid_data_plane
command
>> >> you're
>> >> running, and the Grid-Stat configuration files you're using?
>> >>
>> >> Thanks,
>> >> John
>> >>
>> >> On Mon, Aug 10, 2020 at 10:11 AM John L Wagner - NOAA Federal
via RT <
>> >> met_help at ucar.edu> wrote:
>> >>
>> >> >
>> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96126
>
>> >> >
>> >> > Thanks Julie. I did submit this to the help desk last week.
That is
>> >> where
>> >> > Howard mentioned checking the netcdf library. I'll submit
again to
>> get
>> >> > John's thoughts.
>> >> > John, please see my email below. If you need any more
information
>> >> please
>> >> > let me know.
>> >> > Thanks
>> >> > John
>> >> >
>> >> > On Mon, Aug 10, 2020 at 12:05 PM Julie Prestopnik
<jpresto at ucar.edu>
>> >> > wrote:
>> >> >
>> >> > > Hi John.
>> >> > >
>> >> > > I am not the best person to address these issues about
>> compression. I
>> >> > can
>> >> > > confirm that I started linking to v4.5.0 of the NetCDF
library for
>> the
>> >> > > reason you indicated - to use the standard WCOSS libraries,
as
>> that is
>> >> > how
>> >> > > it will be done for the operational installation of MET.
>> >> > >
>> >> > > Could you please submit your questions about compression to
>> >> > > met_help at ucar.edu so that your questions can be addressed by
the
>> >> right
>> >> > > person (likely John, who also happens to be on helpdesk
today)?
>> >> > >
>> >> > > Thanks!
>> >> > >
>> >> > > Julie
>> >> > >
>> >> > > On Mon, Aug 10, 2020 at 8:12 AM John L Wagner - NOAA Federal
<
>> >> > > john.l.wagner at noaa.gov> wrote:
>> >> > >
>> >> > >> Good morning Julie
>> >> > >> Over the past few weeks, we've been seeing less compression
with
>> >> > >> regrid_data_plane and significantly longer run times with
>> grid_stat.
>> >> > For
>> >> > >> regrid_data_plane, our CONUS files used to compress from
~44MB to
>> >> ~23MB.
>> >> > >> Now they only compress to ~30MB. For grid_stat, a single
model
>> and
>> >> > >> analysis grid would run in a minute or two. It now takes
8-10
>> >> minutes
>> >> > to
>> >> > >> run. We have been testing mainly with MET V9.0.2 and
V9.0.3 out
>> of
>> >> your
>> >> > >> directory on mars.
>> >> > >> Howard mentioned that the compression is controlled by the
netcdf
>> >> > >> library. Starting with V9.0.1, you started linking to
V4.5.0.
>> >> Prior to
>> >> > >> that, it looks like you had your own version of the
library. I
>> >> assume
>> >> > >> the change was made just to use the standard WCOSS
libraries. I
>> was
>> >> > just
>> >> > >> wondering if anyone else had experienced the same issues
that we
>> have
>> >> > after
>> >> > >> this change?
>> >> > >> I ran some tests over the weekend reverting back to MET
V8.1. I
>> >> didn't
>> >> > >> get in early enough this morning to check the file sizes of
the
>> >> > >> regrid_data_plane output, but the grid_stat run times were
still
>> >> longer
>> >> > >> than expected. I suspect the netcdf library is not the
issue
>> >> there. If
>> >> > >> you know of any issues that might be causing longer
grid_stat run
>> >> times
>> >> > or
>> >> > >> have any suggestions for other tests we can try, please let
me
>> know.
>> >> > >> Thanks
>> >> > >> John
>> >> > >>
>> >> > >> ---------- Forwarded message ---------
>> >> > >> From: Howard Soh via RT <met_help at ucar.edu>
>> >> > >> Date: Thu, Aug 6, 2020 at 12:29 AM
>> >> > >> Subject: [rt.rap.ucar.edu #96126] Compression and grid_stat
run
>> time
>> >> > >> To: <john.l.wagner at noaa.gov>
>> >> > >> Cc: <dana.strom at noaa.gov>, <tamarah.curtis at noaa.gov>
>> >> > >>
>> >> > >>
>> >> > >> The compression is done by NetCDF library. The NetCDF
output (size
>> >> > >> difference) is caused by the NetCDF library, not by MET
code.
>> Please
>> >> > check
>> >> > >> the NetCDF library version:
>> >> > >> ldd regrid_data_plane | grep netcdf
>> >> > >>
>> >> > >> The issue 1363 (applied 9.0.3 and 9.1 beta 2) improves the
speed
>> of
>> >> > >> reading NetCDF input, and no update on writing NetCDF
output.
>> >> > >>
>> >> > >> Please let us know if 9.0.3 runs slower than 9.0.2 with
using the
>> >> same
>> >> > >> NetCDF library.
>> >> > >>
>> >> > >> Cheers,
>> >> > >> Howard
>> >> > >>
>> >> > >> Here are examples to check NetCDF library:
>> >> > >> ldd met-8.1/bin/regrid_data_plane | grep netcdf
>> >> > >> libnetcdf_c++4.so.1 =>
>> >> /usr/local/netcdf/lib/libnetcdf_c++4.so.1
>> >> > >> (0x00007fee87e70000)
>> >> > >> libnetcdf.so.7 => not found
>> >> > >> libnetcdf.so.13 =>
/usr/local/netcdf/lib/libnetcdf.so.13
>> >> > >> (0x00007fee83f05000)
>> >> > >>
>> >> > >> ldd met-9.0.2/bin/regrid_data_plane | grep netcdf
>> >> > >> libnetcdf_c++4.so.1 =>
>> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
>> >> > >> (0x00007fdf5d1b2000)
>> >> > >> libnetcdf.so.15 =>
>> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
>> >> > (0x00007fdf5ce31000)
>> >> > >>
>> >> > >>
>> >> > >> ldd met-9.0.3/bin/regrid_data_plane | grep netcdf
>> >> > >> libnetcdf_c++4.so.1 =>
>> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf_c++4.so.1
>> >> > >> (0x00007f7dfe91d000)
>> >> > >> libnetcdf.so.15 =>
>> >> > >> /usr/local/netcdf-4.7.0/gcc-6.3.0/lib/libnetcdf.so.15
>> >> > (0x00007f7dfe59c000)
>> >> > >>
>> >> > >>
>> >> > >> On Wed Aug 05 08:36:09 2020, mccabe wrote:
>> >> > >> > Hi John,
>> >> > >> >
>> >> > >> > I see you have questions about compression in regards to
run
>> time
>> >> > >> > length and file sizes. I have assigned this ticket to
Howard
>> Soh.
>> >> > >> > Please allow a few business days for a response.
>> >> > >> >
>> >> > >> > Thanks,
>> >> > >> > George
>> >> > >>
>> >> > >>
>> >> > >>
>> >> > >>
>> >> > >>
>> >> > >> --
>> >> > >> John Wagner
>> >> > >> Verification Task Lead
>> >> > >> NOAA/National Weather Service
>> >> > >> Meteorological Development Laboratory
>> >> > >> Digital Forecast Services Division
>> >> > >> SSMC2 Room 10106
>> >> > >> Silver Spring, MD 20910
>> >> > >> (301) 427-9471 (office)
>> >> > >> (908) 902-4155 (cell/text)
>> >> > >>
>> >> > >
>> >> > >
>> >> > > --
>> >> > > Julie Prestopnik (she/her/hers)
>> >> > > Software Engineer
>> >> > > National Center for Atmospheric Research
>> >> > > Research Applications Laboratory
>> >> > > Phone: 303.497.8399
>> >> > > Email: jpresto at ucar.edu
>> >> > >
>> >> > > My working day may not be your working day. Please do not
feel
>> >> obliged
>> >> > to
>> >> > > reply to this email outside of your normal working hours.
>> >> > >
>> >> >
>> >> >
>> >> > --
>> >> > John Wagner
>> >> > Verification Task Lead
>> >> > NOAA/National Weather Service
>> >> > Meteorological Development Laboratory
>> >> > Digital Forecast Services Division
>> >> > SSMC2 Room 10106
>> >> > Silver Spring, MD 20910
>> >> > (301) 427-9471 (office)
>> >> > (908) 902-4155 (cell/text)
>> >> >
>> >> >
>> >>
>> >>
>> >
>> > --
>> > John Wagner
>> > Verification Task Lead
>> > NOAA/National Weather Service
>> > Meteorological Development Laboratory
>> > Digital Forecast Services Division
>> > SSMC2 Room 10106
>> > Silver Spring, MD 20910
>> > (301) 427-9471 (office)
>> > (908) 902-4155 (cell/text)
>> >
>>
>>
>> --
>> John Wagner
>> Verification Task Lead
>> NOAA/National Weather Service
>> Meteorological Development Laboratory
>> Digital Forecast Services Division
>> SSMC2 Room 10106
>> Silver Spring, MD 20910
>> (301) 427-9471 (office)
>> (908) 902-4155 (cell/text)
>>
>>
------------------------------------------------
More information about the Met_help
mailing list