[Met_help] [rt.rap.ucar.edu #97235] History for Upper bound on size of grids for MET?
John Halley Gotway via RT
met_help at ucar.edu
Tue Nov 17 10:45:51 MST 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi, there,
Is there an upper bound on the size of grids that MET can handle? The
regional model group is introducing a North American grid that has 3 km
grid spacing and has over 15 million gridpoints. When I use FULL for the
POINT_STAT_GRID, I see that the verification is missing many fields that
exist in the grib2 file. I don't see this problem for any other domain
(that I've noticed).
Should MET be handling these grids, or is there something else missing that
I need to do?
Thanks!
Perry
--
Perry C. Shafran
Verification/MEG Group Leader for IMSG
VPPPG Branch of EMC/NCEP/NOAA/NWS
Office: 301-683-3765
Cell: 301-717-6454
perry.shafran at noaa.gov
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Upper bound on size of grids for MET?
From: Julie Prestopnik
Time: Tue Oct 27 08:13:47 2020
Good morning, Perry. I have assigned your ticket to John. Please
allow a
few business days for a response.
Thanks!
Julie
On Tue, Oct 27, 2020 at 8:04 AM perry.shafran at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> Tue Oct 27 08:03:43 2020: Request 97235 was acted upon.
> Transaction: Ticket created by perry.shafran at noaa.gov
> Queue: met_help
> Subject: Upper bound on size of grids for MET?
> Owner: Nobody
> Requestors: perry.shafran at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97235 >
>
>
> Hi, there,
>
> Is there an upper bound on the size of grids that MET can handle?
The
> regional model group is introducing a North American grid that has 3
km
> grid spacing and has over 15 million gridpoints. When I use FULL
for the
> POINT_STAT_GRID, I see that the verification is missing many fields
that
> exist in the grib2 file. I don't see this problem for any other
domain
> (that I've noticed).
>
> Should MET be handling these grids, or is there something else
missing that
> I need to do?
>
> Thanks!
>
> Perry
>
> --
> Perry C. Shafran
> Verification/MEG Group Leader for IMSG
> VPPPG Branch of EMC/NCEP/NOAA/NWS
> Office: 301-683-3765
> Cell: 301-717-6454
> perry.shafran at noaa.gov
>
>
--
Julie Prestopnik (she/her/hers)
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Upper bound on size of grids for MET?
From: John Halley Gotway
Time: Tue Oct 27 09:10:08 2020
Perry,
Theoretically no, there's no upper limit. Practically speaking, the
software is limited by the amount of memory on the machine. However if
there were a problem with memory I would expect the entire run to fail
instead of getting partial output.
"When I use FULL for the POINT_STAT_GRID, I see that the verification
is
missing many fields that
exist in the grib2 file. I don't see this problem for any other
domain
(that I've noticed)."
My interpretation of this statement is...
- You ran Point-Stat and set: mask.grid = [ "FULL" ];
- You looked at the output from Point-Stat and compared it to Point-
Stat
output from previous model runs on other domains.
- The "new" output is missing verification results for some of the
variables/levels that are present for the "old" model output.
Is that correct? If so, it sounds like it's the usual question of "Why
am I
getting 0 matched pairs from Point-Stat?".
And the way to debug this is to rerun using the "-v 3" logging option.
That'll print reason codes for why obs were or were not used.
One potential source of difference that I'd rule out first is GRIB
variable
naming conventions. Are the old/new GRIB files using the same
parameters
tables and numbers? Changes to how the variables are named would lead
to differences in the amount of verification output that's produced by
Point-Stat. Perhaps Point-Stat can't find the requested variables in
the
GRIB file... or perhaps it can find them but the variable names no
longer
match the conventions by which the point observations are named?
Hope that helps.
John
On Tue, Oct 27, 2020 at 8:14 AM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97235 >
>
> Good morning, Perry. I have assigned your ticket to John. Please
allow a
> few business days for a response.
>
> Thanks!
>
> Julie
>
> On Tue, Oct 27, 2020 at 8:04 AM perry.shafran at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Oct 27 08:03:43 2020: Request 97235 was acted upon.
> > Transaction: Ticket created by perry.shafran at noaa.gov
> > Queue: met_help
> > Subject: Upper bound on size of grids for MET?
> > Owner: Nobody
> > Requestors: perry.shafran at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97235 >
> >
> >
> > Hi, there,
> >
> > Is there an upper bound on the size of grids that MET can handle?
The
> > regional model group is introducing a North American grid that has
3 km
> > grid spacing and has over 15 million gridpoints. When I use FULL
for the
> > POINT_STAT_GRID, I see that the verification is missing many
fields that
> > exist in the grib2 file. I don't see this problem for any other
domain
> > (that I've noticed).
> >
> > Should MET be handling these grids, or is there something else
missing
> that
> > I need to do?
> >
> > Thanks!
> >
> > Perry
> >
> > --
> > Perry C. Shafran
> > Verification/MEG Group Leader for IMSG
> > VPPPG Branch of EMC/NCEP/NOAA/NWS
> > Office: 301-683-3765
> > Cell: 301-717-6454
> > perry.shafran at noaa.gov
> >
> >
>
> --
> Julie Prestopnik (she/her/hers)
> Software Engineer
> National Center for Atmospheric Research
> Research Applications Laboratory
> Email: jpresto at ucar.edu
>
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: Upper bound on size of grids for MET?
From: perry.shafran at noaa.gov
Time: Tue Oct 27 09:20:39 2020
Hi, John,
I am not certain that it has to do with grib convention. For example,
I am
not getting upper air statistics at heights higher than 500 mb even
though
there exists fields all the way up the atmosphere. But the
interesting
thing that I think is happening is that if you do a wgrib2 on a file,
the
later the field is listed in the output, the more likely the
verification
is NOT going to be listed in the output stat file. For example, the
2-m
TMP exists later in the grib file but is not in the output stat file.
There should be no difference in the point observation files, those
are the
same used in the other model verifications.
One thing we do plan to examine is if there are differences in which
the
grib is written for the new parallel. The memory issue I think is
also a
possibility, so maybe we'd need to run this example using LSF so we
can
allocate more memory to the run? Is there a way to add memory on the
point_stat command line?
BTW - the logging verbosity is set to 5 I think, so we have all the
logging
we need, I believe.
We've not had a 3-km full North American domain so this is indeed new
territory.
Thanks for your thoughts.
Perry
On Tue, Oct 27, 2020 at 11:10 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Perry,
>
> Theoretically no, there's no upper limit. Practically speaking, the
> software is limited by the amount of memory on the machine. However
if
> there were a problem with memory I would expect the entire run to
fail
> instead of getting partial output.
>
> "When I use FULL for the POINT_STAT_GRID, I see that the
verification is
> missing many fields that
> exist in the grib2 file. I don't see this problem for any other
domain
> (that I've noticed)."
>
> My interpretation of this statement is...
> - You ran Point-Stat and set: mask.grid = [ "FULL" ];
> - You looked at the output from Point-Stat and compared it to Point-
Stat
> output from previous model runs on other domains.
> - The "new" output is missing verification results for some of the
> variables/levels that are present for the "old" model output.
>
> Is that correct? If so, it sounds like it's the usual question of
"Why am I
> getting 0 matched pairs from Point-Stat?".
> And the way to debug this is to rerun using the "-v 3" logging
option.
> That'll print reason codes for why obs were or were not used.
>
> One potential source of difference that I'd rule out first is GRIB
variable
> naming conventions. Are the old/new GRIB files using the same
parameters
> tables and numbers? Changes to how the variables are named would
lead
> to differences in the amount of verification output that's produced
by
> Point-Stat. Perhaps Point-Stat can't find the requested variables in
the
> GRIB file... or perhaps it can find them but the variable names no
longer
> match the conventions by which the point observations are named?
>
> Hope that helps.
>
> John
>
> On Tue, Oct 27, 2020 at 8:14 AM Julie Prestopnik via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97235 >
> >
> > Good morning, Perry. I have assigned your ticket to John. Please
allow
> a
> > few business days for a response.
> >
> > Thanks!
> >
> > Julie
> >
> > On Tue, Oct 27, 2020 at 8:04 AM perry.shafran at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Oct 27 08:03:43 2020: Request 97235 was acted upon.
> > > Transaction: Ticket created by perry.shafran at noaa.gov
> > > Queue: met_help
> > > Subject: Upper bound on size of grids for MET?
> > > Owner: Nobody
> > > Requestors: perry.shafran at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97235
> >
> > >
> > >
> > > Hi, there,
> > >
> > > Is there an upper bound on the size of grids that MET can
handle? The
> > > regional model group is introducing a North American grid that
has 3 km
> > > grid spacing and has over 15 million gridpoints. When I use
FULL for
> the
> > > POINT_STAT_GRID, I see that the verification is missing many
fields
> that
> > > exist in the grib2 file. I don't see this problem for any other
domain
> > > (that I've noticed).
> > >
> > > Should MET be handling these grids, or is there something else
missing
> > that
> > > I need to do?
> > >
> > > Thanks!
> > >
> > > Perry
> > >
> > > --
> > > Perry C. Shafran
> > > Verification/MEG Group Leader for IMSG
> > > VPPPG Branch of EMC/NCEP/NOAA/NWS
> > > Office: 301-683-3765
> > > Cell: 301-717-6454
> > > perry.shafran at noaa.gov
> > >
> > >
> >
> > --
> > Julie Prestopnik (she/her/hers)
> > Software Engineer
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > Email: jpresto at ucar.edu
> >
> > My working day may not be your working day. Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
--
Perry C. Shafran
Verification/MEG Group Leader for IMSG
VPPPG Branch of EMC/NCEP/NOAA/NWS
Office: 301-683-3765
Cell: 301-717-6454
perry.shafran at noaa.gov
------------------------------------------------
Subject: Upper bound on size of grids for MET?
From: John Halley Gotway
Time: Tue Oct 27 10:01:56 2020
Perry,
For one of the verification tasks that's missing output... like 2-m
TMP,
can you please go look in the log file and find the rejection reason
counts
listed there? Or perhaps we didn't even make it that far. So just grep
for
"WARNING:" in that log file.
Thanks,
John
On Tue, Oct 27, 2020 at 9:21 AM perry.shafran at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97235 >
>
> Hi, John,
>
> I am not certain that it has to do with grib convention. For
example, I am
> not getting upper air statistics at heights higher than 500 mb even
though
> there exists fields all the way up the atmosphere. But the
interesting
> thing that I think is happening is that if you do a wgrib2 on a
file, the
> later the field is listed in the output, the more likely the
verification
> is NOT going to be listed in the output stat file. For example, the
2-m
> TMP exists later in the grib file but is not in the output stat
file.
> There should be no difference in the point observation files, those
are the
> same used in the other model verifications.
>
> One thing we do plan to examine is if there are differences in which
the
> grib is written for the new parallel. The memory issue I think is
also a
> possibility, so maybe we'd need to run this example using LSF so we
can
> allocate more memory to the run? Is there a way to add memory on the
> point_stat command line?
>
> BTW - the logging verbosity is set to 5 I think, so we have all the
logging
> we need, I believe.
>
> We've not had a 3-km full North American domain so this is indeed
new
> territory.
>
> Thanks for your thoughts.
>
> Perry
>
> On Tue, Oct 27, 2020 at 11:10 AM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Perry,
> >
> > Theoretically no, there's no upper limit. Practically speaking,
the
> > software is limited by the amount of memory on the machine.
However if
> > there were a problem with memory I would expect the entire run to
fail
> > instead of getting partial output.
> >
> > "When I use FULL for the POINT_STAT_GRID, I see that the
verification is
> > missing many fields that
> > exist in the grib2 file. I don't see this problem for any other
domain
> > (that I've noticed)."
> >
> > My interpretation of this statement is...
> > - You ran Point-Stat and set: mask.grid = [ "FULL" ];
> > - You looked at the output from Point-Stat and compared it to
Point-Stat
> > output from previous model runs on other domains.
> > - The "new" output is missing verification results for some of the
> > variables/levels that are present for the "old" model output.
> >
> > Is that correct? If so, it sounds like it's the usual question of
"Why
> am I
> > getting 0 matched pairs from Point-Stat?".
> > And the way to debug this is to rerun using the "-v 3" logging
option.
> > That'll print reason codes for why obs were or were not used.
> >
> > One potential source of difference that I'd rule out first is GRIB
> variable
> > naming conventions. Are the old/new GRIB files using the same
parameters
> > tables and numbers? Changes to how the variables are named would
lead
> > to differences in the amount of verification output that's
produced by
> > Point-Stat. Perhaps Point-Stat can't find the requested variables
in the
> > GRIB file... or perhaps it can find them but the variable names no
longer
> > match the conventions by which the point observations are named?
> >
> > Hope that helps.
> >
> > John
> >
> > On Tue, Oct 27, 2020 at 8:14 AM Julie Prestopnik via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97235 >
> > >
> > > Good morning, Perry. I have assigned your ticket to John.
Please
> allow
> > a
> > > few business days for a response.
> > >
> > > Thanks!
> > >
> > > Julie
> > >
> > > On Tue, Oct 27, 2020 at 8:04 AM perry.shafran at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Tue Oct 27 08:03:43 2020: Request 97235 was acted upon.
> > > > Transaction: Ticket created by perry.shafran at noaa.gov
> > > > Queue: met_help
> > > > Subject: Upper bound on size of grids for MET?
> > > > Owner: Nobody
> > > > Requestors: perry.shafran at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97235
> > >
> > > >
> > > >
> > > > Hi, there,
> > > >
> > > > Is there an upper bound on the size of grids that MET can
handle?
> The
> > > > regional model group is introducing a North American grid that
has 3
> km
> > > > grid spacing and has over 15 million gridpoints. When I use
FULL for
> > the
> > > > POINT_STAT_GRID, I see that the verification is missing many
fields
> > that
> > > > exist in the grib2 file. I don't see this problem for any
other
> domain
> > > > (that I've noticed).
> > > >
> > > > Should MET be handling these grids, or is there something else
> missing
> > > that
> > > > I need to do?
> > > >
> > > > Thanks!
> > > >
> > > > Perry
> > > >
> > > > --
> > > > Perry C. Shafran
> > > > Verification/MEG Group Leader for IMSG
> > > > VPPPG Branch of EMC/NCEP/NOAA/NWS
> > > > Office: 301-683-3765
> > > > Cell: 301-717-6454
> > > > perry.shafran at noaa.gov
> > > >
> > > >
> > >
> > > --
> > > Julie Prestopnik (she/her/hers)
> > > Software Engineer
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > Email: jpresto at ucar.edu
> > >
> > > My working day may not be your working day. Please do not feel
obliged
> > to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
> --
> Perry C. Shafran
> Verification/MEG Group Leader for IMSG
> VPPPG Branch of EMC/NCEP/NOAA/NWS
> Office: 301-683-3765
> Cell: 301-717-6454
> perry.shafran at noaa.gov
>
>
------------------------------------------------
More information about the Met_help
mailing list