[Met_help] [rt.rap.ucar.edu #90988] History for grid-stat output file limits
John Halley Gotway via RT
met_help at ucar.edu
Fri Oct 11 16:24:19 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Good Day MET Help,
I currently have code that loops through several weather parameters, levels,
taus, and mask regions to output to one output file using grid-stat. It
seems that the output files have a limit to the amount of data it can
display. Is this true? If so, is there a part of the code where I can
increase the limit or is this something hard coded into MET?
Best Regards,
Angela LaFlash
Scientific Programmer
DeVine Consulting, Inc.
409 Washington St. #202A
Monterey, CA 93940
Cell: 217-652-2115
mailto:angela.laflash.ctr at nrlmry.navy.mil
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: grid-stat output file limits
From: John Halley Gotway
Time: Tue Jul 09 14:12:51 2019
Hello Angela,
I see you have a question about file limitations for the output of
Grid-Stat. There is no upper limit to number of verification tasks
which
can be included in a single Grid-Stat run... and no limitation on the
number of output lines it cane write. But it sounds like you're
having an
issue.
Let me make a guess as to what *might* be going on. You mentioned
that
you're looping over many variables/levels/lead times. Grid-Stat
constructs
its own output file names which look like this:
grid_stat_120000L_20050807_120000V.stat
They always start with "grid_stat" and end with ".stat" (or
"_pairs.nc" or
"_LINETYPE.txt" for each of the line types it writes). And in the
middle
is timing information. If you run Grid-Stat twice using the same
input
files (perhaps to evaluate different variables), then the output of
the
second run will clobber the output of the first run. Perhaps you're
calling Grid-Stat many times and overwritting the output files for all
but
the last run?
This is handled by specifying the "output_prefix" Grid-Stat config
file
option which is a string that gets included in all output file names.
That
can be used to prevent output files from getting clobbered.
Please let me know if that doesn't fix this issue and we can try to
figure
out what's going on. Also, it sounds like you're running Grid-Stat
many
times. Please be aware that you can configure Grid-Stat to verify
multiple
fields in a single run, as long as all the input data lives in the
same
input file.
Thanks,
John Halley Gotway
On Tue, Jul 9, 2019 at 12:55 PM LaFlash, Ms. Angela, Contractor, Code
7542
via RT <met_help at ucar.edu> wrote:
>
> Tue Jul 09 12:55:55 2019: Request 90988 was acted upon.
> Transaction: Ticket created by angela.laflash.ctr at nrlmry.navy.mil
> Queue: met_help
> Subject: grid-stat output file limits
> Owner: Nobody
> Requestors: angela.laflash.ctr at nrlmry.navy.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90988 >
>
>
> Good Day MET Help,
>
>
>
> I currently have code that loops through several weather parameters,
> levels,
> taus, and mask regions to output to one output file using grid-stat.
It
> seems that the output files have a limit to the amount of data it
can
> display. Is this true? If so, is there a part of the code where I
can
> increase the limit or is this something hard coded into MET?
>
>
>
> Best Regards,
>
> Angela LaFlash
>
> Scientific Programmer
>
> DeVine Consulting, Inc.
>
> 409 Washington St. #202A
>
> Monterey, CA 93940
>
> Cell: 217-652-2115
>
> mailto:angela.laflash.ctr at nrlmry.navy.mil
>
>
>
>
>
------------------------------------------------
Subject: grid-stat output file limits
From: LaFlash, Ms. Angela, Contractor, Code 7542
Time: Tue Jul 09 14:23:21 2019
Hello John,
Thank you for your quick reply. I'm not sure if an overwriting
problem is
what's happening. The reason I say that is that I have set up my
config file
to process over 70 different forecast and obs files, thus each loop
would add
the new fcst and obs file to the list. Also, I have about 15
different
regions listed under the "mask" dictionary. So, as of now, the grid-
stat is
only being called once to produce one file with a bunch of variables,
levels,
and lead times. Unfortunately, I can't quite pin-point which
parameter is
causing the output file to limit. It will produce data for all the
regions,
but not all levels or lead times. Let me know if you need more
information
about my config file or the looping code.
Thanks,
Angela LaFlash
(217)652-2115
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, July 9, 2019 1:13 PM
To: LaFlash, Ms. Angela, Contractor, Code 7542
Subject: Re: [rt.rap.ucar.edu #90988] grid-stat output file limits
Hello Angela,
I see you have a question about file limitations for the output of
Grid-Stat. There is no upper limit to number of verification tasks
which
can be included in a single Grid-Stat run... and no limitation on the
number of output lines it cane write. But it sounds like you're
having an
issue.
Let me make a guess as to what *might* be going on. You mentioned
that
you're looping over many variables/levels/lead times. Grid-Stat
constructs
its own output file names which look like this:
grid_stat_120000L_20050807_120000V.stat
They always start with "grid_stat" and end with ".stat" (or
"_pairs.nc" or
"_LINETYPE.txt" for each of the line types it writes). And in the
middle
is timing information. If you run Grid-Stat twice using the same
input
files (perhaps to evaluate different variables), then the output of
the
second run will clobber the output of the first run. Perhaps you're
calling Grid-Stat many times and overwritting the output files for all
but
the last run?
This is handled by specifying the "output_prefix" Grid-Stat config
file
option which is a string that gets included in all output file names.
That
can be used to prevent output files from getting clobbered.
Please let me know if that doesn't fix this issue and we can try to
figure
out what's going on. Also, it sounds like you're running Grid-Stat
many
times. Please be aware that you can configure Grid-Stat to verify
multiple
fields in a single run, as long as all the input data lives in the
same
input file.
Thanks,
John Halley Gotway
On Tue, Jul 9, 2019 at 12:55 PM LaFlash, Ms. Angela, Contractor, Code
7542
via RT <met_help at ucar.edu> wrote:
>
> Tue Jul 09 12:55:55 2019: Request 90988 was acted upon.
> Transaction: Ticket created by angela.laflash.ctr at nrlmry.navy.mil
> Queue: met_help
> Subject: grid-stat output file limits
> Owner: Nobody
> Requestors: angela.laflash.ctr at nrlmry.navy.mil
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90988 >
>
>
> Good Day MET Help,
>
>
>
> I currently have code that loops through several weather parameters,
> levels,
> taus, and mask regions to output to one output file using grid-stat.
It
> seems that the output files have a limit to the amount of data it
can
> display. Is this true? If so, is there a part of the code where I
can
> increase the limit or is this something hard coded into MET?
>
>
>
> Best Regards,
>
> Angela LaFlash
>
> Scientific Programmer
>
> DeVine Consulting, Inc.
>
> 409 Washington St. #202A
>
> Monterey, CA 93940
>
> Cell: 217-652-2115
>
> mailto:angela.laflash.ctr at nrlmry.navy.mil
>
>
>
>
>
------------------------------------------------
Subject: grid-stat output file limits
From: John Halley Gotway
Time: Tue Jul 09 14:32:42 2019
Angela,
One easy thing you could do it use the "-log run_grid_stat.log"
command
line option to write the messages to a log file. You could also
increase
the logging level from the default value of 2 to 4 using "-v 4".
Then you could grep the log file for warnings or error messages:
egrep -i "warning|error" run_grid_stat.log
If there are any warning or error messages being written, they might
point
you in the right direction.
Thanks,
John
On Tue, Jul 9, 2019 at 2:23 PM LaFlash, Ms. Angela, Contractor, Code
7542
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90988 >
>
> Hello John,
>
> Thank you for your quick reply. I'm not sure if an overwriting
problem is
> what's happening. The reason I say that is that I have set up my
config
> file
> to process over 70 different forecast and obs files, thus each loop
would
> add
> the new fcst and obs file to the list. Also, I have about 15
different
> regions listed under the "mask" dictionary. So, as of now, the
grid-stat
> is
> only being called once to produce one file with a bunch of
variables,
> levels,
> and lead times. Unfortunately, I can't quite pin-point which
parameter is
> causing the output file to limit. It will produce data for all the
> regions,
> but not all levels or lead times. Let me know if you need more
> information
> about my config file or the looping code.
>
> Thanks,
> Angela LaFlash
> (217)652-2115
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, July 9, 2019 1:13 PM
> To: LaFlash, Ms. Angela, Contractor, Code 7542
> Subject: Re: [rt.rap.ucar.edu #90988] grid-stat output file limits
>
> Hello Angela,
>
> I see you have a question about file limitations for the output of
> Grid-Stat. There is no upper limit to number of verification tasks
which
> can be included in a single Grid-Stat run... and no limitation on
the
> number of output lines it cane write. But it sounds like you're
having an
> issue.
>
> Let me make a guess as to what *might* be going on. You mentioned
that
> you're looping over many variables/levels/lead times. Grid-Stat
constructs
> its own output file names which look like this:
> grid_stat_120000L_20050807_120000V.stat
>
> They always start with "grid_stat" and end with ".stat" (or
"_pairs.nc" or
> "_LINETYPE.txt" for each of the line types it writes). And in the
middle
> is timing information. If you run Grid-Stat twice using the same
input
> files (perhaps to evaluate different variables), then the output of
the
> second run will clobber the output of the first run. Perhaps you're
> calling Grid-Stat many times and overwritting the output files for
all but
> the last run?
>
> This is handled by specifying the "output_prefix" Grid-Stat config
file
> option which is a string that gets included in all output file
names. That
> can be used to prevent output files from getting clobbered.
>
> Please let me know if that doesn't fix this issue and we can try to
figure
> out what's going on. Also, it sounds like you're running Grid-Stat
many
> times. Please be aware that you can configure Grid-Stat to verify
multiple
> fields in a single run, as long as all the input data lives in the
same
> input file.
>
> Thanks,
> John Halley Gotway
>
> On Tue, Jul 9, 2019 at 12:55 PM LaFlash, Ms. Angela, Contractor,
Code 7542
> via RT <met_help at ucar.edu> wrote:
>
> >
> > Tue Jul 09 12:55:55 2019: Request 90988 was acted upon.
> > Transaction: Ticket created by angela.laflash.ctr at nrlmry.navy.mil
> > Queue: met_help
> > Subject: grid-stat output file limits
> > Owner: Nobody
> > Requestors: angela.laflash.ctr at nrlmry.navy.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90988 >
> >
> >
> > Good Day MET Help,
> >
> >
> >
> > I currently have code that loops through several weather
parameters,
> > levels,
> > taus, and mask regions to output to one output file using grid-
stat. It
> > seems that the output files have a limit to the amount of data it
can
> > display. Is this true? If so, is there a part of the code where
I can
> > increase the limit or is this something hard coded into MET?
> >
> >
> >
> > Best Regards,
> >
> > Angela LaFlash
> >
> > Scientific Programmer
> >
> > DeVine Consulting, Inc.
> >
> > 409 Washington St. #202A
> >
> > Monterey, CA 93940
> >
> > Cell: 217-652-2115
> >
> > mailto:angela.laflash.ctr at nrlmry.navy.mil
> >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: grid-stat output file limits
From: LaFlash, Ms. Angela, Contractor, Code 7542
Time: Wed Jul 10 12:05:10 2019
Good Day John,
Thank you for your advice for the log! I am getting the following
error:
WARNING: python_dataplane() -> an error occurred importing module
"read_NRL_binary.py"
WARNING: read_data_plane() -> (nul) not found in file: PYTHON_NUMPY
WARNING: python_dataplane() -> an error occurred initializing python
WARNING: read_data_plane() -> (nul) not found in file: PYTHON_NUMPY
WARNING: python_dataplane() -> an error occurred initializing python
WARNING: read_data_plane() -> (nul) not found in file: PYTHON_NUMPY
WARNING: python_dataplane() -> an error occurred initializing python
.
.
.
.
Any advice on how to go about fixing this? It seems that I can
successfully
run 1 lead time, 2 weather variables, and 4 levels with all the output
for the
15 regions listed, but this error still shows when I have a successful
run.
I'm so lost! Thank you for any advice!
Thanks,
Angela LaFlash
(217)652-2115
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Tuesday, July 9, 2019 1:33 PM
To: LaFlash, Ms. Angela, Contractor, Code 7542
Subject: Re: [rt.rap.ucar.edu #90988] grid-stat output file limits
Angela,
One easy thing you could do it use the "-log run_grid_stat.log"
command
line option to write the messages to a log file. You could also
increase
the logging level from the default value of 2 to 4 using "-v 4".
Then you could grep the log file for warnings or error messages:
egrep -i "warning|error" run_grid_stat.log
If there are any warning or error messages being written, they might
point
you in the right direction.
Thanks,
John
On Tue, Jul 9, 2019 at 2:23 PM LaFlash, Ms. Angela, Contractor, Code
7542
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90988 >
>
> Hello John,
>
> Thank you for your quick reply. I'm not sure if an overwriting
problem is
> what's happening. The reason I say that is that I have set up my
config
> file
> to process over 70 different forecast and obs files, thus each loop
would
> add
> the new fcst and obs file to the list. Also, I have about 15
different
> regions listed under the "mask" dictionary. So, as of now, the
grid-stat
> is
> only being called once to produce one file with a bunch of
variables,
> levels,
> and lead times. Unfortunately, I can't quite pin-point which
parameter is
> causing the output file to limit. It will produce data for all the
> regions,
> but not all levels or lead times. Let me know if you need more
> information
> about my config file or the looping code.
>
> Thanks,
> Angela LaFlash
> (217)652-2115
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, July 9, 2019 1:13 PM
> To: LaFlash, Ms. Angela, Contractor, Code 7542
> Subject: Re: [rt.rap.ucar.edu #90988] grid-stat output file limits
>
> Hello Angela,
>
> I see you have a question about file limitations for the output of
> Grid-Stat. There is no upper limit to number of verification tasks
which
> can be included in a single Grid-Stat run... and no limitation on
the
> number of output lines it cane write. But it sounds like you're
having an
> issue.
>
> Let me make a guess as to what *might* be going on. You mentioned
that
> you're looping over many variables/levels/lead times. Grid-Stat
constructs
> its own output file names which look like this:
> grid_stat_120000L_20050807_120000V.stat
>
> They always start with "grid_stat" and end with ".stat" (or
"_pairs.nc" or
> "_LINETYPE.txt" for each of the line types it writes). And in the
middle
> is timing information. If you run Grid-Stat twice using the same
input
> files (perhaps to evaluate different variables), then the output of
the
> second run will clobber the output of the first run. Perhaps you're
> calling Grid-Stat many times and overwritting the output files for
all but
> the last run?
>
> This is handled by specifying the "output_prefix" Grid-Stat config
file
> option which is a string that gets included in all output file
names. That
> can be used to prevent output files from getting clobbered.
>
> Please let me know if that doesn't fix this issue and we can try to
figure
> out what's going on. Also, it sounds like you're running Grid-Stat
many
> times. Please be aware that you can configure Grid-Stat to verify
multiple
> fields in a single run, as long as all the input data lives in the
same
> input file.
>
> Thanks,
> John Halley Gotway
>
> On Tue, Jul 9, 2019 at 12:55 PM LaFlash, Ms. Angela, Contractor,
Code 7542
> via RT <met_help at ucar.edu> wrote:
>
> >
> > Tue Jul 09 12:55:55 2019: Request 90988 was acted upon.
> > Transaction: Ticket created by angela.laflash.ctr at nrlmry.navy.mil
> > Queue: met_help
> > Subject: grid-stat output file limits
> > Owner: Nobody
> > Requestors: angela.laflash.ctr at nrlmry.navy.mil
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90988 >
> >
> >
> > Good Day MET Help,
> >
> >
> >
> > I currently have code that loops through several weather
parameters,
> > levels,
> > taus, and mask regions to output to one output file using grid-
stat. It
> > seems that the output files have a limit to the amount of data it
can
> > display. Is this true? If so, is there a part of the code where
I can
> > increase the limit or is this something hard coded into MET?
> >
> >
> >
> > Best Regards,
> >
> > Angela LaFlash
> >
> > Scientific Programmer
> >
> > DeVine Consulting, Inc.
> >
> > 409 Washington St. #202A
> >
> > Monterey, CA 93940
> >
> > Cell: 217-652-2115
> >
> > mailto:angela.laflash.ctr at nrlmry.navy.mil
> >
> >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: grid-stat output file limits
From: John Halley Gotway
Time: Wed Jul 10 12:41:22 2019
Angela,
If you send me some test data to replicate this behavior, I can help
you
debug.
I'll need to see the Grid-Stat config file, the Python script(s)
you're
calling, and whatever binary files those Python scripts are reading.
Please just send me the smallest amount of data which demonstrates the
behavior. You can post it to our anonymous ftp site following these
instructions:
https://dtcenter.org/community-code/model-evaluation-tools-met/met-
help-desk
Look under "How to send us data".
Thanks,
John
On Wed, Jul 10, 2019 at 12:05 PM LaFlash, Ms. Angela, Contractor, Code
7542
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90988 >
>
> Good Day John,
>
> Thank you for your advice for the log! I am getting the following
error:
>
> WARNING: python_dataplane() -> an error occurred importing module
> "read_NRL_binary.py"
> WARNING: read_data_plane() -> (nul) not found in file: PYTHON_NUMPY
> WARNING: python_dataplane() -> an error occurred initializing python
> WARNING: read_data_plane() -> (nul) not found in file: PYTHON_NUMPY
> WARNING: python_dataplane() -> an error occurred initializing python
> WARNING: read_data_plane() -> (nul) not found in file: PYTHON_NUMPY
> WARNING: python_dataplane() -> an error occurred initializing python
> .
> .
> .
> .
>
> Any advice on how to go about fixing this? It seems that I can
> successfully
> run 1 lead time, 2 weather variables, and 4 levels with all the
output for
> the
> 15 regions listed, but this error still shows when I have a
successful
> run.
> I'm so lost! Thank you for any advice!
>
> Thanks,
> Angela LaFlash
> (217)652-2115
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, July 9, 2019 1:33 PM
> To: LaFlash, Ms. Angela, Contractor, Code 7542
> Subject: Re: [rt.rap.ucar.edu #90988] grid-stat output file limits
>
> Angela,
>
> One easy thing you could do it use the "-log run_grid_stat.log"
command
> line option to write the messages to a log file. You could also
increase
> the logging level from the default value of 2 to 4 using "-v 4".
>
> Then you could grep the log file for warnings or error messages:
> egrep -i "warning|error" run_grid_stat.log
>
> If there are any warning or error messages being written, they might
point
> you in the right direction.
>
> Thanks,
> John
>
> On Tue, Jul 9, 2019 at 2:23 PM LaFlash, Ms. Angela, Contractor, Code
7542
> via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90988 >
> >
> > Hello John,
> >
> > Thank you for your quick reply. I'm not sure if an overwriting
problem
> is
> > what's happening. The reason I say that is that I have set up my
config
> > file
> > to process over 70 different forecast and obs files, thus each
loop would
> > add
> > the new fcst and obs file to the list. Also, I have about 15
different
> > regions listed under the "mask" dictionary. So, as of now, the
grid-stat
> > is
> > only being called once to produce one file with a bunch of
variables,
> > levels,
> > and lead times. Unfortunately, I can't quite pin-point which
parameter
> is
> > causing the output file to limit. It will produce data for all
the
> > regions,
> > but not all levels or lead times. Let me know if you need more
> > information
> > about my config file or the looping code.
> >
> > Thanks,
> > Angela LaFlash
> > (217)652-2115
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Tuesday, July 9, 2019 1:13 PM
> > To: LaFlash, Ms. Angela, Contractor, Code 7542
> > Subject: Re: [rt.rap.ucar.edu #90988] grid-stat output file limits
> >
> > Hello Angela,
> >
> > I see you have a question about file limitations for the output of
> > Grid-Stat. There is no upper limit to number of verification
tasks which
> > can be included in a single Grid-Stat run... and no limitation on
the
> > number of output lines it cane write. But it sounds like you're
having
> an
> > issue.
> >
> > Let me make a guess as to what *might* be going on. You mentioned
that
> > you're looping over many variables/levels/lead times. Grid-Stat
> constructs
> > its own output file names which look like this:
> > grid_stat_120000L_20050807_120000V.stat
> >
> > They always start with "grid_stat" and end with ".stat" (or
"_pairs.nc"
> or
> > "_LINETYPE.txt" for each of the line types it writes). And in the
middle
> > is timing information. If you run Grid-Stat twice using the same
input
> > files (perhaps to evaluate different variables), then the output
of the
> > second run will clobber the output of the first run. Perhaps
you're
> > calling Grid-Stat many times and overwritting the output files for
all
> but
> > the last run?
> >
> > This is handled by specifying the "output_prefix" Grid-Stat config
file
> > option which is a string that gets included in all output file
names.
> That
> > can be used to prevent output files from getting clobbered.
> >
> > Please let me know if that doesn't fix this issue and we can try
to
> figure
> > out what's going on. Also, it sounds like you're running Grid-
Stat many
> > times. Please be aware that you can configure Grid-Stat to verify
> multiple
> > fields in a single run, as long as all the input data lives in the
same
> > input file.
> >
> > Thanks,
> > John Halley Gotway
> >
> > On Tue, Jul 9, 2019 at 12:55 PM LaFlash, Ms. Angela, Contractor,
Code
> 7542
> > via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Jul 09 12:55:55 2019: Request 90988 was acted upon.
> > > Transaction: Ticket created by
angela.laflash.ctr at nrlmry.navy.mil
> > > Queue: met_help
> > > Subject: grid-stat output file limits
> > > Owner: Nobody
> > > Requestors: angela.laflash.ctr at nrlmry.navy.mil
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90988
> >
> > >
> > >
> > > Good Day MET Help,
> > >
> > >
> > >
> > > I currently have code that loops through several weather
parameters,
> > > levels,
> > > taus, and mask regions to output to one output file using grid-
stat.
> It
> > > seems that the output files have a limit to the amount of data
it can
> > > display. Is this true? If so, is there a part of the code
where I can
> > > increase the limit or is this something hard coded into MET?
> > >
> > >
> > >
> > > Best Regards,
> > >
> > > Angela LaFlash
> > >
> > > Scientific Programmer
> > >
> > > DeVine Consulting, Inc.
> > >
> > > 409 Washington St. #202A
> > >
> > > Monterey, CA 93940
> > >
> > > Cell: 217-652-2115
> > >
> > > mailto:angela.laflash.ctr at nrlmry.navy.mil
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: grid-stat output file limits
From: LaFlash, Ms. Angela, Contractor, Code 7542
Time: Thu Jul 11 10:25:34 2019
John,
Thank you for your help! I have contacted my project lead to check
the type
of data we are allowed to send over to you. Please standby while I
wait for a
reply.
Thanks,
Angela LaFlash
(217)652-2115
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, July 10, 2019 11:41 AM
To: LaFlash, Ms. Angela, Contractor, Code 7542
Subject: Re: [rt.rap.ucar.edu #90988] grid-stat output file limits
Angela,
If you send me some test data to replicate this behavior, I can help
you
debug.
I'll need to see the Grid-Stat config file, the Python script(s)
you're
calling, and whatever binary files those Python scripts are reading.
Please just send me the smallest amount of data which demonstrates the
behavior. You can post it to our anonymous ftp site following these
instructions:
https://dtcenter.org/community-code/model-evaluation-tools-met/met-
help-desk
Look under "How to send us data".
Thanks,
John
On Wed, Jul 10, 2019 at 12:05 PM LaFlash, Ms. Angela, Contractor, Code
7542
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90988 >
>
> Good Day John,
>
> Thank you for your advice for the log! I am getting the following
error:
>
> WARNING: python_dataplane() -> an error occurred importing module
> "read_NRL_binary.py"
> WARNING: read_data_plane() -> (nul) not found in file: PYTHON_NUMPY
> WARNING: python_dataplane() -> an error occurred initializing python
> WARNING: read_data_plane() -> (nul) not found in file: PYTHON_NUMPY
> WARNING: python_dataplane() -> an error occurred initializing python
> WARNING: read_data_plane() -> (nul) not found in file: PYTHON_NUMPY
> WARNING: python_dataplane() -> an error occurred initializing python
> .
> .
> .
> .
>
> Any advice on how to go about fixing this? It seems that I can
> successfully
> run 1 lead time, 2 weather variables, and 4 levels with all the
output for
> the
> 15 regions listed, but this error still shows when I have a
successful
> run.
> I'm so lost! Thank you for any advice!
>
> Thanks,
> Angela LaFlash
> (217)652-2115
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Tuesday, July 9, 2019 1:33 PM
> To: LaFlash, Ms. Angela, Contractor, Code 7542
> Subject: Re: [rt.rap.ucar.edu #90988] grid-stat output file limits
>
> Angela,
>
> One easy thing you could do it use the "-log run_grid_stat.log"
command
> line option to write the messages to a log file. You could also
increase
> the logging level from the default value of 2 to 4 using "-v 4".
>
> Then you could grep the log file for warnings or error messages:
> egrep -i "warning|error" run_grid_stat.log
>
> If there are any warning or error messages being written, they might
point
> you in the right direction.
>
> Thanks,
> John
>
> On Tue, Jul 9, 2019 at 2:23 PM LaFlash, Ms. Angela, Contractor, Code
7542
> via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90988 >
> >
> > Hello John,
> >
> > Thank you for your quick reply. I'm not sure if an overwriting
problem
> is
> > what's happening. The reason I say that is that I have set up my
config
> > file
> > to process over 70 different forecast and obs files, thus each
loop would
> > add
> > the new fcst and obs file to the list. Also, I have about 15
different
> > regions listed under the "mask" dictionary. So, as of now, the
grid-stat
> > is
> > only being called once to produce one file with a bunch of
variables,
> > levels,
> > and lead times. Unfortunately, I can't quite pin-point which
parameter
> is
> > causing the output file to limit. It will produce data for all
the
> > regions,
> > but not all levels or lead times. Let me know if you need more
> > information
> > about my config file or the looping code.
> >
> > Thanks,
> > Angela LaFlash
> > (217)652-2115
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Tuesday, July 9, 2019 1:13 PM
> > To: LaFlash, Ms. Angela, Contractor, Code 7542
> > Subject: Re: [rt.rap.ucar.edu #90988] grid-stat output file limits
> >
> > Hello Angela,
> >
> > I see you have a question about file limitations for the output of
> > Grid-Stat. There is no upper limit to number of verification
tasks which
> > can be included in a single Grid-Stat run... and no limitation on
the
> > number of output lines it cane write. But it sounds like you're
having
> an
> > issue.
> >
> > Let me make a guess as to what *might* be going on. You mentioned
that
> > you're looping over many variables/levels/lead times. Grid-Stat
> constructs
> > its own output file names which look like this:
> > grid_stat_120000L_20050807_120000V.stat
> >
> > They always start with "grid_stat" and end with ".stat" (or
"_pairs.nc"
> or
> > "_LINETYPE.txt" for each of the line types it writes). And in the
middle
> > is timing information. If you run Grid-Stat twice using the same
input
> > files (perhaps to evaluate different variables), then the output
of the
> > second run will clobber the output of the first run. Perhaps
you're
> > calling Grid-Stat many times and overwritting the output files for
all
> but
> > the last run?
> >
> > This is handled by specifying the "output_prefix" Grid-Stat config
file
> > option which is a string that gets included in all output file
names.
> That
> > can be used to prevent output files from getting clobbered.
> >
> > Please let me know if that doesn't fix this issue and we can try
to
> figure
> > out what's going on. Also, it sounds like you're running Grid-
Stat many
> > times. Please be aware that you can configure Grid-Stat to verify
> multiple
> > fields in a single run, as long as all the input data lives in the
same
> > input file.
> >
> > Thanks,
> > John Halley Gotway
> >
> > On Tue, Jul 9, 2019 at 12:55 PM LaFlash, Ms. Angela, Contractor,
Code
> 7542
> > via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Jul 09 12:55:55 2019: Request 90988 was acted upon.
> > > Transaction: Ticket created by
angela.laflash.ctr at nrlmry.navy.mil
> > > Queue: met_help
> > > Subject: grid-stat output file limits
> > > Owner: Nobody
> > > Requestors: angela.laflash.ctr at nrlmry.navy.mil
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90988
> >
> > >
> > >
> > > Good Day MET Help,
> > >
> > >
> > >
> > > I currently have code that loops through several weather
parameters,
> > > levels,
> > > taus, and mask regions to output to one output file using grid-
stat.
> It
> > > seems that the output files have a limit to the amount of data
it can
> > > display. Is this true? If so, is there a part of the code
where I can
> > > increase the limit or is this something hard coded into MET?
> > >
> > >
> > >
> > > Best Regards,
> > >
> > > Angela LaFlash
> > >
> > > Scientific Programmer
> > >
> > > DeVine Consulting, Inc.
> > >
> > > 409 Washington St. #202A
> > >
> > > Monterey, CA 93940
> > >
> > > Cell: 217-652-2115
> > >
> > > mailto:angela.laflash.ctr at nrlmry.navy.mil
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: grid-stat output file limits
From: John Halley Gotway
Time: Tue Aug 06 11:23:41 2019
Hi Angela,
Just wanted to follow up on this ticket. Are you still experiencing
issues
with the python embedding in MET?
Thanks,
John
On Thu, Jul 11, 2019 at 10:26 AM LaFlash, Ms. Angela, Contractor, Code
7542
via RT <met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90988 >
>
> John,
>
> Thank you for your help! I have contacted my project lead to check
the
> type
> of data we are allowed to send over to you. Please standby while I
wait
> for a
> reply.
>
> Thanks,
> Angela LaFlash
> (217)652-2115
>
> -----Original Message-----
> From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> Sent: Wednesday, July 10, 2019 11:41 AM
> To: LaFlash, Ms. Angela, Contractor, Code 7542
> Subject: Re: [rt.rap.ucar.edu #90988] grid-stat output file limits
>
> Angela,
>
> If you send me some test data to replicate this behavior, I can help
you
> debug.
>
> I'll need to see the Grid-Stat config file, the Python script(s)
you're
> calling, and whatever binary files those Python scripts are reading.
> Please just send me the smallest amount of data which demonstrates
the
> behavior. You can post it to our anonymous ftp site following these
> instructions:
>
>
> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
help-desk
>
> Look under "How to send us data".
>
> Thanks,
> John
>
>
> On Wed, Jul 10, 2019 at 12:05 PM LaFlash, Ms. Angela, Contractor,
Code 7542
> via RT <met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90988 >
> >
> > Good Day John,
> >
> > Thank you for your advice for the log! I am getting the following
error:
> >
> > WARNING: python_dataplane() -> an error occurred importing module
> > "read_NRL_binary.py"
> > WARNING: read_data_plane() -> (nul) not found in file:
PYTHON_NUMPY
> > WARNING: python_dataplane() -> an error occurred initializing
python
> > WARNING: read_data_plane() -> (nul) not found in file:
PYTHON_NUMPY
> > WARNING: python_dataplane() -> an error occurred initializing
python
> > WARNING: read_data_plane() -> (nul) not found in file:
PYTHON_NUMPY
> > WARNING: python_dataplane() -> an error occurred initializing
python
> > .
> > .
> > .
> > .
> >
> > Any advice on how to go about fixing this? It seems that I can
> > successfully
> > run 1 lead time, 2 weather variables, and 4 levels with all the
output
> for
> > the
> > 15 regions listed, but this error still shows when I have a
successful
> > run.
> > I'm so lost! Thank you for any advice!
> >
> > Thanks,
> > Angela LaFlash
> > (217)652-2115
> >
> > -----Original Message-----
> > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > Sent: Tuesday, July 9, 2019 1:33 PM
> > To: LaFlash, Ms. Angela, Contractor, Code 7542
> > Subject: Re: [rt.rap.ucar.edu #90988] grid-stat output file limits
> >
> > Angela,
> >
> > One easy thing you could do it use the "-log run_grid_stat.log"
command
> > line option to write the messages to a log file. You could also
increase
> > the logging level from the default value of 2 to 4 using "-v 4".
> >
> > Then you could grep the log file for warnings or error messages:
> > egrep -i "warning|error" run_grid_stat.log
> >
> > If there are any warning or error messages being written, they
might
> point
> > you in the right direction.
> >
> > Thanks,
> > John
> >
> > On Tue, Jul 9, 2019 at 2:23 PM LaFlash, Ms. Angela, Contractor,
Code 7542
> > via RT <met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90988 >
> > >
> > > Hello John,
> > >
> > > Thank you for your quick reply. I'm not sure if an overwriting
problem
> > is
> > > what's happening. The reason I say that is that I have set up
my
> config
> > > file
> > > to process over 70 different forecast and obs files, thus each
loop
> would
> > > add
> > > the new fcst and obs file to the list. Also, I have about 15
different
> > > regions listed under the "mask" dictionary. So, as of now, the
> grid-stat
> > > is
> > > only being called once to produce one file with a bunch of
variables,
> > > levels,
> > > and lead times. Unfortunately, I can't quite pin-point which
parameter
> > is
> > > causing the output file to limit. It will produce data for all
the
> > > regions,
> > > but not all levels or lead times. Let me know if you need more
> > > information
> > > about my config file or the looping code.
> > >
> > > Thanks,
> > > Angela LaFlash
> > > (217)652-2115
> > >
> > > -----Original Message-----
> > > From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
> > > Sent: Tuesday, July 9, 2019 1:13 PM
> > > To: LaFlash, Ms. Angela, Contractor, Code 7542
> > > Subject: Re: [rt.rap.ucar.edu #90988] grid-stat output file
limits
> > >
> > > Hello Angela,
> > >
> > > I see you have a question about file limitations for the output
of
> > > Grid-Stat. There is no upper limit to number of verification
tasks
> which
> > > can be included in a single Grid-Stat run... and no limitation
on the
> > > number of output lines it cane write. But it sounds like you're
having
> > an
> > > issue.
> > >
> > > Let me make a guess as to what *might* be going on. You
mentioned that
> > > you're looping over many variables/levels/lead times. Grid-Stat
> > constructs
> > > its own output file names which look like this:
> > > grid_stat_120000L_20050807_120000V.stat
> > >
> > > They always start with "grid_stat" and end with ".stat" (or
"_pairs.nc
> "
> > or
> > > "_LINETYPE.txt" for each of the line types it writes). And in
the
> middle
> > > is timing information. If you run Grid-Stat twice using the
same input
> > > files (perhaps to evaluate different variables), then the output
of the
> > > second run will clobber the output of the first run. Perhaps
you're
> > > calling Grid-Stat many times and overwritting the output files
for all
> > but
> > > the last run?
> > >
> > > This is handled by specifying the "output_prefix" Grid-Stat
config file
> > > option which is a string that gets included in all output file
names.
> > That
> > > can be used to prevent output files from getting clobbered.
> > >
> > > Please let me know if that doesn't fix this issue and we can try
to
> > figure
> > > out what's going on. Also, it sounds like you're running Grid-
Stat
> many
> > > times. Please be aware that you can configure Grid-Stat to
verify
> > multiple
> > > fields in a single run, as long as all the input data lives in
the same
> > > input file.
> > >
> > > Thanks,
> > > John Halley Gotway
> > >
> > > On Tue, Jul 9, 2019 at 12:55 PM LaFlash, Ms. Angela, Contractor,
Code
> > 7542
> > > via RT <met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Tue Jul 09 12:55:55 2019: Request 90988 was acted upon.
> > > > Transaction: Ticket created by
angela.laflash.ctr at nrlmry.navy.mil
> > > > Queue: met_help
> > > > Subject: grid-stat output file limits
> > > > Owner: Nobody
> > > > Requestors: angela.laflash.ctr at nrlmry.navy.mil
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=90988
> > >
> > > >
> > > >
> > > > Good Day MET Help,
> > > >
> > > >
> > > >
> > > > I currently have code that loops through several weather
parameters,
> > > > levels,
> > > > taus, and mask regions to output to one output file using
grid-stat.
> > It
> > > > seems that the output files have a limit to the amount of data
it can
> > > > display. Is this true? If so, is there a part of the code
where I
> can
> > > > increase the limit or is this something hard coded into MET?
> > > >
> > > >
> > > >
> > > > Best Regards,
> > > >
> > > > Angela LaFlash
> > > >
> > > > Scientific Programmer
> > > >
> > > > DeVine Consulting, Inc.
> > > >
> > > > 409 Washington St. #202A
> > > >
> > > > Monterey, CA 93940
> > > >
> > > > Cell: 217-652-2115
> > > >
> > > > mailto:angela.laflash.ctr at nrlmry.navy.mil
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
> >
> >
>
>
>
------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #90988] Resolved: grid-stat output file limits
From: LaFlash, Ms. Angela, Contractor, Code 7542
Time: Mon Sep 16 08:45:39 2019
Good Day John,
I apologize for not getting back to you. I just returned from
maternity leave today and I'm going through my large backlog of work
emails. I will contact my project lead and ensure that this problem
has been resolved.
Thanks,
Angela LaFlash
217-652-2115
-----Original Message-----
From: John Halley Gotway via RT [mailto:met_help at ucar.edu]
Sent: Wednesday, August 14, 2019 10:08 AM
To: LaFlash, Ms. Angela, Contractor, Code 7542
Subject: [rt.rap.ucar.edu #90988] Resolved: grid-stat output file
limits
According to our records, your request has been resolved. If you have
any
further questions or concerns, please respond to this message.
------------------------------------------------
More information about the Met_help
mailing list