[Met_help] [rt.rap.ucar.edu #92225] History for Issue with multiple interp widths in ensemble_stat -- netCDF error
John Halley Gotway via RT
met_help at ucar.edu
Mon Sep 23 09:33:22 MDT 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi once again MET helpdesk,
After successfully processing my model and observation data with both ensemble_stat and grid_stat, I was looking to expand on that by introducing different smoothing neighborhoods. I’ve set my ensemble_stat config file to evaluate the forecast grids at widths of 1, 3, and 5 grid points using the square method. The problem arises when I go to run ensemble_stat where the following error shows up:
DEBUG 2: Processing gridded verification APCP/A06 versus APCP_06A06, for observation type APCP, over region WoFS_radars_20180618, for interpolation method NEAREST(1), using 21311 matched pairs.
DEBUG 2: Skipping 843 of 21311 points due to missing ensemble values.
DEBUG 2: Processing gridded verification APCP/A06 versus APCP_06A06, for observation type APCP, over region WoFS_radars_20180618, for interpolation method UW_MEAN_SQUARE(9), using 21311 matched pairs.
DEBUG 2: Skipping 972 of 21311 points due to missing ensemble values.
terminate called after throwing an instance of 'netCDF::exceptions::NcNameInUse'
what(): NetCDF: String match to name in use
file: ncGroup.cpp line:496
Abort (core dumped)
What could be the nature of this error message? Is it just that the output is being written to a file that is open so it leads to a file lock error? As always, any help is greatly appreciated.
Thanks!
-Brian
—————————————————————
Brian Matilla
Research Fellow— Warn-on-Forecast Team
Cooperative Institute for Mesoscale Meteorological Studies — The University of Oklahoma
NOAA National Severe Storms Laboratory
Phone: (405) 325-1688
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Issue with multiple interp widths in ensemble_stat -- netCDF error
From: John Halley Gotway
Time: Thu Sep 19 13:59:22 2019
Hi Brian,
I see you've encountered an error from the Ensemble-Stat tool. This
error
message from NetCDF means that MET is attempting to write the same
variable
name to an output NetCDF file. I tried to replicate this behavior
locally,
by adding additional smoothing methods but wasn't able to replicate
it.
Can you please...
(1) Tell me what version of MET you're running.
(2) Send me the EnsembleStatConfig file you're using so I can see
exactly
how you've configured it.
Thanks,
John Halley Gotway
On Thu, Sep 19, 2019 at 1:40 PM Brian Matilla - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> Thu Sep 19 13:40:08 2019: Request 92225 was acted upon.
> Transaction: Ticket created by brian.matilla at noaa.gov
> Queue: met_help
> Subject: Issue with multiple interp widths in ensemble_stat --
netCDF
> error
> Owner: Nobody
> Requestors: brian.matilla at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92225 >
>
>
> Hi once again MET helpdesk,
>
> After successfully processing my model and observation data with
both
> ensemble_stat and grid_stat, I was looking to expand on that by
introducing
> different smoothing neighborhoods. I’ve set my ensemble_stat config
file to
> evaluate the forecast grids at widths of 1, 3, and 5 grid points
using the
> square method. The problem arises when I go to run ensemble_stat
where the
> following error shows up:
>
> DEBUG 2: Processing gridded verification APCP/A06 versus APCP_06A06,
for
> observation type APCP, over region WoFS_radars_20180618, for
interpolation
> method NEAREST(1), using 21311 matched pairs.
> DEBUG 2: Skipping 843 of 21311 points due to missing ensemble
values.
> DEBUG 2: Processing gridded verification APCP/A06 versus APCP_06A06,
for
> observation type APCP, over region WoFS_radars_20180618, for
interpolation
> method UW_MEAN_SQUARE(9), using 21311 matched pairs.
> DEBUG 2: Skipping 972 of 21311 points due to missing ensemble
values.
> terminate called after throwing an instance of
> 'netCDF::exceptions::NcNameInUse'
> what(): NetCDF: String match to name in use
> file: ncGroup.cpp line:496
> Abort (core dumped)
>
> What could be the nature of this error message? Is it just that the
output
> is being written to a file that is open so it leads to a file lock
error?
> As always, any help is greatly appreciated.
>
> Thanks!
>
> -Brian
> —————————————————————
> Brian Matilla
> Research Fellow— Warn-on-Forecast Team
> Cooperative Institute for Mesoscale Meteorological Studies — The
> University of Oklahoma
> NOAA National Severe Storms Laboratory
>
> Phone: (405) 325-1688
>
>
>
>
------------------------------------------------
Subject: Issue with multiple interp widths in ensemble_stat -- netCDF error
From: Brian Matilla - NOAA Affiliate
Time: Thu Sep 19 14:14:15 2019
Hi there John,
So I am using MET version 8.1.1. I have attached the ensemble_stat
config file that I am using. Thanks for the quick reply!
-Brian
------------------------------------------------
Subject: Issue with multiple interp widths in ensemble_stat -- netCDF error
From: Brian Matilla - NOAA Affiliate
Time: Thu Sep 19 14:14:15 2019
> On Sep 19, 2019, at 2:59 PM, John Halley Gotway via RT
<met_help at ucar.edu> wrote:
>
> Hi Brian,
>
> I see you've encountered an error from the Ensemble-Stat tool. This
error
> message from NetCDF means that MET is attempting to write the same
variable
> name to an output NetCDF file. I tried to replicate this behavior
locally,
> by adding additional smoothing methods but wasn't able to replicate
it.
>
> Can you please...
> (1) Tell me what version of MET you're running.
> (2) Send me the EnsembleStatConfig file you're using so I can see
exactly
> how you've configured it.
>
> Thanks,
> John Halley Gotway
>
>
> On Thu, Sep 19, 2019 at 1:40 PM Brian Matilla - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
>>
>> Thu Sep 19 13:40:08 2019: Request 92225 was acted upon.
>> Transaction: Ticket created by brian.matilla at noaa.gov
>> Queue: met_help
>> Subject: Issue with multiple interp widths in ensemble_stat --
netCDF
>> error
>> Owner: Nobody
>> Requestors: brian.matilla at noaa.gov
>> Status: new
>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92225 >
>>
>>
>> Hi once again MET helpdesk,
>>
>> After successfully processing my model and observation data with
both
>> ensemble_stat and grid_stat, I was looking to expand on that by
introducing
>> different smoothing neighborhoods. I’ve set my ensemble_stat config
file to
>> evaluate the forecast grids at widths of 1, 3, and 5 grid points
using the
>> square method. The problem arises when I go to run ensemble_stat
where the
>> following error shows up:
>>
>> DEBUG 2: Processing gridded verification APCP/A06 versus
APCP_06A06, for
>> observation type APCP, over region WoFS_radars_20180618, for
interpolation
>> method NEAREST(1), using 21311 matched pairs.
>> DEBUG 2: Skipping 843 of 21311 points due to missing ensemble
values.
>> DEBUG 2: Processing gridded verification APCP/A06 versus
APCP_06A06, for
>> observation type APCP, over region WoFS_radars_20180618, for
interpolation
>> method UW_MEAN_SQUARE(9), using 21311 matched pairs.
>> DEBUG 2: Skipping 972 of 21311 points due to missing ensemble
values.
>> terminate called after throwing an instance of
>> 'netCDF::exceptions::NcNameInUse'
>> what(): NetCDF: String match to name in use
>> file: ncGroup.cpp line:496
>> Abort (core dumped)
>>
>> What could be the nature of this error message? Is it just that the
output
>> is being written to a file that is open so it leads to a file lock
error?
>> As always, any help is greatly appreciated.
>>
>> Thanks!
>>
>> -Brian
>> —————————————————————
>> Brian Matilla
>> Research Fellow— Warn-on-Forecast Team
>> Cooperative Institute for Mesoscale Meteorological Studies — The
>> University of Oklahoma
>> NOAA National Severe Storms Laboratory
>>
>> Phone: (405) 325-1688
>>
>>
>>
>>
>
------------------------------------------------
Subject: Issue with multiple interp widths in ensemble_stat -- netCDF error
From: John Halley Gotway
Time: Thu Sep 19 14:48:20 2019
Brian,
Thanks for sending that. I was able to replicate the behavior you
describe. If you edit your Ensemble-Stat config file and disable the
orank NetCDF output file, that should get you a successful run:
ensemble_flag = {
latlon = TRUE;
mean = TRUE;
stdev = TRUE;
minus = TRUE;
plus = TRUE;
min = TRUE;
max = TRUE;
range = TRUE;
vld_count = TRUE;
frequency = TRUE;
* rank = FALSE;*
weight = TRUE;
}
Obviously I'd like to get a better solution in the next bugfix release
of
MET, but that's a quick fix for today.
Thanks,
John
On Thu, Sep 19, 2019 at 2:14 PM Brian Matilla - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92225 >
>
> Hi there John,
>
> So I am using MET version 8.1.1. I have attached the ensemble_stat
config
> file that I am using. Thanks for the quick reply!
>
> -Brian
>
>
>
> > On Sep 19, 2019, at 2:59 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
> >
> > Hi Brian,
> >
> > I see you've encountered an error from the Ensemble-Stat tool.
This
> error
> > message from NetCDF means that MET is attempting to write the same
> variable
> > name to an output NetCDF file. I tried to replicate this behavior
> locally,
> > by adding additional smoothing methods but wasn't able to
replicate it.
> >
> > Can you please...
> > (1) Tell me what version of MET you're running.
> > (2) Send me the EnsembleStatConfig file you're using so I can see
exactly
> > how you've configured it.
> >
> > Thanks,
> > John Halley Gotway
> >
> >
> > On Thu, Sep 19, 2019 at 1:40 PM Brian Matilla - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> Thu Sep 19 13:40:08 2019: Request 92225 was acted upon.
> >> Transaction: Ticket created by brian.matilla at noaa.gov
> >> Queue: met_help
> >> Subject: Issue with multiple interp widths in ensemble_stat
--
> netCDF
> >> error
> >> Owner: Nobody
> >> Requestors: brian.matilla at noaa.gov
> >> Status: new
> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92225 >
> >>
> >>
> >> Hi once again MET helpdesk,
> >>
> >> After successfully processing my model and observation data with
both
> >> ensemble_stat and grid_stat, I was looking to expand on that by
> introducing
> >> different smoothing neighborhoods. I’ve set my ensemble_stat
config
> file to
> >> evaluate the forecast grids at widths of 1, 3, and 5 grid points
using
> the
> >> square method. The problem arises when I go to run ensemble_stat
where
> the
> >> following error shows up:
> >>
> >> DEBUG 2: Processing gridded verification APCP/A06 versus
APCP_06A06, for
> >> observation type APCP, over region WoFS_radars_20180618, for
> interpolation
> >> method NEAREST(1), using 21311 matched pairs.
> >> DEBUG 2: Skipping 843 of 21311 points due to missing ensemble
values.
> >> DEBUG 2: Processing gridded verification APCP/A06 versus
APCP_06A06, for
> >> observation type APCP, over region WoFS_radars_20180618, for
> interpolation
> >> method UW_MEAN_SQUARE(9), using 21311 matched pairs.
> >> DEBUG 2: Skipping 972 of 21311 points due to missing ensemble
values.
> >> terminate called after throwing an instance of
> >> 'netCDF::exceptions::NcNameInUse'
> >> what(): NetCDF: String match to name in use
> >> file: ncGroup.cpp line:496
> >> Abort (core dumped)
> >>
> >> What could be the nature of this error message? Is it just that
the
> output
> >> is being written to a file that is open so it leads to a file
lock
> error?
> >> As always, any help is greatly appreciated.
> >>
> >> Thanks!
> >>
> >> -Brian
> >> —————————————————————
> >> Brian Matilla
> >> Research Fellow— Warn-on-Forecast Team
> >> Cooperative Institute for Mesoscale Meteorological Studies — The
> >> University of Oklahoma
> >> NOAA National Severe Storms Laboratory
> >>
> >> Phone: (405) 325-1688
> >>
> >>
> >>
> >>
> >
>
>
------------------------------------------------
Subject: Issue with multiple interp widths in ensemble_stat -- netCDF error
From: John Halley Gotway
Time: Thu Sep 19 16:16:52 2019
Brian,
FYI, I found and fixed the underlying problem. Here's a description
of the
issue:
https://github.com/NCAR/MET/issues/1198
And that change will be included in the next bugfix release, met-
8.1.2.
Thanks for uncovering this issue.
John
On Thu, Sep 19, 2019 at 2:48 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
> Brian,
>
> Thanks for sending that. I was able to replicate the behavior you
> describe. If you edit your Ensemble-Stat config file and disable
the
> orank NetCDF output file, that should get you a successful run:
>
> ensemble_flag = {
> latlon = TRUE;
> mean = TRUE;
> stdev = TRUE;
> minus = TRUE;
> plus = TRUE;
> min = TRUE;
> max = TRUE;
> range = TRUE;
> vld_count = TRUE;
> frequency = TRUE;
> * rank = FALSE;*
> weight = TRUE;
> }
>
> Obviously I'd like to get a better solution in the next bugfix
release of
> MET, but that's a quick fix for today.
>
> Thanks,
> John
>
> On Thu, Sep 19, 2019 at 2:14 PM Brian Matilla - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92225 >
>>
>> Hi there John,
>>
>> So I am using MET version 8.1.1. I have attached the ensemble_stat
config
>> file that I am using. Thanks for the quick reply!
>>
>> -Brian
>>
>>
>>
>> > On Sep 19, 2019, at 2:59 PM, John Halley Gotway via RT <
>> met_help at ucar.edu> wrote:
>> >
>> > Hi Brian,
>> >
>> > I see you've encountered an error from the Ensemble-Stat tool.
This
>> error
>> > message from NetCDF means that MET is attempting to write the
same
>> variable
>> > name to an output NetCDF file. I tried to replicate this
behavior
>> locally,
>> > by adding additional smoothing methods but wasn't able to
replicate it.
>> >
>> > Can you please...
>> > (1) Tell me what version of MET you're running.
>> > (2) Send me the EnsembleStatConfig file you're using so I can see
>> exactly
>> > how you've configured it.
>> >
>> > Thanks,
>> > John Halley Gotway
>> >
>> >
>> > On Thu, Sep 19, 2019 at 1:40 PM Brian Matilla - NOAA Affiliate
via RT <
>> > met_help at ucar.edu> wrote:
>> >
>> >>
>> >> Thu Sep 19 13:40:08 2019: Request 92225 was acted upon.
>> >> Transaction: Ticket created by brian.matilla at noaa.gov
>> >> Queue: met_help
>> >> Subject: Issue with multiple interp widths in ensemble_stat
--
>> netCDF
>> >> error
>> >> Owner: Nobody
>> >> Requestors: brian.matilla at noaa.gov
>> >> Status: new
>> >> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92225 >
>> >>
>> >>
>> >> Hi once again MET helpdesk,
>> >>
>> >> After successfully processing my model and observation data with
both
>> >> ensemble_stat and grid_stat, I was looking to expand on that by
>> introducing
>> >> different smoothing neighborhoods. I’ve set my ensemble_stat
config
>> file to
>> >> evaluate the forecast grids at widths of 1, 3, and 5 grid points
using
>> the
>> >> square method. The problem arises when I go to run ensemble_stat
where
>> the
>> >> following error shows up:
>> >>
>> >> DEBUG 2: Processing gridded verification APCP/A06 versus
APCP_06A06,
>> for
>> >> observation type APCP, over region WoFS_radars_20180618, for
>> interpolation
>> >> method NEAREST(1), using 21311 matched pairs.
>> >> DEBUG 2: Skipping 843 of 21311 points due to missing ensemble
values.
>> >> DEBUG 2: Processing gridded verification APCP/A06 versus
APCP_06A06,
>> for
>> >> observation type APCP, over region WoFS_radars_20180618, for
>> interpolation
>> >> method UW_MEAN_SQUARE(9), using 21311 matched pairs.
>> >> DEBUG 2: Skipping 972 of 21311 points due to missing ensemble
values.
>> >> terminate called after throwing an instance of
>> >> 'netCDF::exceptions::NcNameInUse'
>> >> what(): NetCDF: String match to name in use
>> >> file: ncGroup.cpp line:496
>> >> Abort (core dumped)
>> >>
>> >> What could be the nature of this error message? Is it just that
the
>> output
>> >> is being written to a file that is open so it leads to a file
lock
>> error?
>> >> As always, any help is greatly appreciated.
>> >>
>> >> Thanks!
>> >>
>> >> -Brian
>> >> —————————————————————
>> >> Brian Matilla
>> >> Research Fellow— Warn-on-Forecast Team
>> >> Cooperative Institute for Mesoscale Meteorological Studies — The
>> >> University of Oklahoma
>> >> NOAA National Severe Storms Laboratory
>> >>
>> >> Phone: (405) 325-1688
>> >>
>> >>
>> >>
>> >>
>> >
>>
>>
------------------------------------------------
Subject: Issue with multiple interp widths in ensemble_stat -- netCDF error
From: Brian Matilla - NOAA Affiliate
Time: Thu Sep 19 17:42:20 2019
John,
I appreciate you looking into the issue and I’m happy to say that did
solve the problem on my front. Ensemble_stat ran successfully!
I did have one follow up question though I’m not sure if it warrants a
separate ticket or not — when I follow that up with running grid_stat
to verify the ensemble probabilities, would I go ahead and set the
“nbrhd” width equal to the smoothing neighborhood (e.g, if I use a 3x3
square box in ensemble_stat, set the nbrhd width to 3 in grid_stat),
or is that essentially taken care of when I run the smoother in
ensemble_stat?
-Brian
> On Sep 19, 2019, at 5:16 PM, John Halley Gotway via RT
<met_help at ucar.edu> wrote:
>
> Brian,
>
> FYI, I found and fixed the underlying problem. Here's a description
of the
> issue:
> https://github.com/NCAR/MET/issues/1198
>
> And that change will be included in the next bugfix release, met-
8.1.2.
>
> Thanks for uncovering this issue.
>
> John
>
> On Thu, Sep 19, 2019 at 2:48 PM John Halley Gotway <johnhg at ucar.edu>
wrote:
>
>> Brian,
>>
>> Thanks for sending that. I was able to replicate the behavior you
>> describe. If you edit your Ensemble-Stat config file and disable
the
>> orank NetCDF output file, that should get you a successful run:
>>
>> ensemble_flag = {
>> latlon = TRUE;
>> mean = TRUE;
>> stdev = TRUE;
>> minus = TRUE;
>> plus = TRUE;
>> min = TRUE;
>> max = TRUE;
>> range = TRUE;
>> vld_count = TRUE;
>> frequency = TRUE;
>> * rank = FALSE;*
>> weight = TRUE;
>> }
>>
>> Obviously I'd like to get a better solution in the next bugfix
release of
>> MET, but that's a quick fix for today.
>>
>> Thanks,
>> John
>>
>> On Thu, Sep 19, 2019 at 2:14 PM Brian Matilla - NOAA Affiliate via
RT <
>> met_help at ucar.edu> wrote:
>>
>>>
>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92225 >
>>>
>>> Hi there John,
>>>
>>> So I am using MET version 8.1.1. I have attached the ensemble_stat
config
>>> file that I am using. Thanks for the quick reply!
>>>
>>> -Brian
>>>
>>>
>>>
>>>> On Sep 19, 2019, at 2:59 PM, John Halley Gotway via RT <
>>> met_help at ucar.edu> wrote:
>>>>
>>>> Hi Brian,
>>>>
>>>> I see you've encountered an error from the Ensemble-Stat tool.
This
>>> error
>>>> message from NetCDF means that MET is attempting to write the
same
>>> variable
>>>> name to an output NetCDF file. I tried to replicate this
behavior
>>> locally,
>>>> by adding additional smoothing methods but wasn't able to
replicate it.
>>>>
>>>> Can you please...
>>>> (1) Tell me what version of MET you're running.
>>>> (2) Send me the EnsembleStatConfig file you're using so I can see
>>> exactly
>>>> how you've configured it.
>>>>
>>>> Thanks,
>>>> John Halley Gotway
>>>>
>>>>
>>>> On Thu, Sep 19, 2019 at 1:40 PM Brian Matilla - NOAA Affiliate
via RT <
>>>> met_help at ucar.edu> wrote:
>>>>
>>>>>
>>>>> Thu Sep 19 13:40:08 2019: Request 92225 was acted upon.
>>>>> Transaction: Ticket created by brian.matilla at noaa.gov
>>>>> Queue: met_help
>>>>> Subject: Issue with multiple interp widths in ensemble_stat
--
>>> netCDF
>>>>> error
>>>>> Owner: Nobody
>>>>> Requestors: brian.matilla at noaa.gov
>>>>> Status: new
>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92225 >
>>>>>
>>>>>
>>>>> Hi once again MET helpdesk,
>>>>>
>>>>> After successfully processing my model and observation data with
both
>>>>> ensemble_stat and grid_stat, I was looking to expand on that by
>>> introducing
>>>>> different smoothing neighborhoods. I’ve set my ensemble_stat
config
>>> file to
>>>>> evaluate the forecast grids at widths of 1, 3, and 5 grid points
using
>>> the
>>>>> square method. The problem arises when I go to run ensemble_stat
where
>>> the
>>>>> following error shows up:
>>>>>
>>>>> DEBUG 2: Processing gridded verification APCP/A06 versus
APCP_06A06,
>>> for
>>>>> observation type APCP, over region WoFS_radars_20180618, for
>>> interpolation
>>>>> method NEAREST(1), using 21311 matched pairs.
>>>>> DEBUG 2: Skipping 843 of 21311 points due to missing ensemble
values.
>>>>> DEBUG 2: Processing gridded verification APCP/A06 versus
APCP_06A06,
>>> for
>>>>> observation type APCP, over region WoFS_radars_20180618, for
>>> interpolation
>>>>> method UW_MEAN_SQUARE(9), using 21311 matched pairs.
>>>>> DEBUG 2: Skipping 972 of 21311 points due to missing ensemble
values.
>>>>> terminate called after throwing an instance of
>>>>> 'netCDF::exceptions::NcNameInUse'
>>>>> what(): NetCDF: String match to name in use
>>>>> file: ncGroup.cpp line:496
>>>>> Abort (core dumped)
>>>>>
>>>>> What could be the nature of this error message? Is it just that
the
>>> output
>>>>> is being written to a file that is open so it leads to a file
lock
>>> error?
>>>>> As always, any help is greatly appreciated.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> -Brian
>>>>> —————————————————————
>>>>> Brian Matilla
>>>>> Research Fellow— Warn-on-Forecast Team
>>>>> Cooperative Institute for Mesoscale Meteorological Studies — The
>>>>> University of Oklahoma
>>>>> NOAA National Severe Storms Laboratory
>>>>>
>>>>> Phone: (405) 325-1688
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>
------------------------------------------------
Subject: Issue with multiple interp widths in ensemble_stat -- netCDF error
From: John Halley Gotway
Time: Fri Sep 20 09:30:32 2019
Brian,
This is a detail I haven't thought about much before. But I can tell
you
what the software is currently doing.
The Ensemble-Stat tool really does two different things:
(1) Derive ensemble mean and probabilities and write them to a NetCDF
*_
ens.nc output file.
(2) Compare the ensemble members to observations, compute ensemble
statistics (like ranked histograms) and write the output to the ASCII
.stat
file and NetCDF *_orank.nc file.
The first step is controlled by the contents of the "ens" section of
the
config file. You get ensemble means and probabilities for the fields
listed there. The second step is controlled by the "fcst" and "obs"
sections of the config file. They define the comparisons for which
you
want rank histograms and ensemble statistics.
Currently, the "interp" section, which defines the smoothing
parameters, is
only applied to the fcst/obs comparisons in step 2. That smoothing
operation is NOT applied to the computation of ensemble means and
probabilities. But honestly, I can't think of a good reason to say
why
they are not. It just hasn't come up before.
Logically, you could smooth the members prior to computing means and
probabilities. Or you could smooth the mean and probabilities
afterwards
when verifying them in Grid-Stat and Point-Stat. The former is not
currently possible in Ensemble-Stat. But conceivably the tool could
be
enhanced at some point to support that.
Hope that helps clarify the logic.
Thanks,
John
On Thu, Sep 19, 2019 at 5:42 PM Brian Matilla - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92225 >
>
> John,
>
> I appreciate you looking into the issue and I’m happy to say that
did
> solve the problem on my front. Ensemble_stat ran successfully!
>
> I did have one follow up question though I’m not sure if it warrants
a
> separate ticket or not — when I follow that up with running
grid_stat to
> verify the ensemble probabilities, would I go ahead and set the
“nbrhd”
> width equal to the smoothing neighborhood (e.g, if I use a 3x3
square box
> in ensemble_stat, set the nbrhd width to 3 in grid_stat), or is that
> essentially taken care of when I run the smoother in ensemble_stat?
>
> -Brian
>
> > On Sep 19, 2019, at 5:16 PM, John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
> >
> > Brian,
> >
> > FYI, I found and fixed the underlying problem. Here's a
description of
> the
> > issue:
> > https://github.com/NCAR/MET/issues/1198
> >
> > And that change will be included in the next bugfix release, met-
8.1.2.
> >
> > Thanks for uncovering this issue.
> >
> > John
> >
> > On Thu, Sep 19, 2019 at 2:48 PM John Halley Gotway
<johnhg at ucar.edu>
> wrote:
> >
> >> Brian,
> >>
> >> Thanks for sending that. I was able to replicate the behavior
you
> >> describe. If you edit your Ensemble-Stat config file and disable
the
> >> orank NetCDF output file, that should get you a successful run:
> >>
> >> ensemble_flag = {
> >> latlon = TRUE;
> >> mean = TRUE;
> >> stdev = TRUE;
> >> minus = TRUE;
> >> plus = TRUE;
> >> min = TRUE;
> >> max = TRUE;
> >> range = TRUE;
> >> vld_count = TRUE;
> >> frequency = TRUE;
> >> * rank = FALSE;*
> >> weight = TRUE;
> >> }
> >>
> >> Obviously I'd like to get a better solution in the next bugfix
release
> of
> >> MET, but that's a quick fix for today.
> >>
> >> Thanks,
> >> John
> >>
> >> On Thu, Sep 19, 2019 at 2:14 PM Brian Matilla - NOAA Affiliate
via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >>>
> >>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92225 >
> >>>
> >>> Hi there John,
> >>>
> >>> So I am using MET version 8.1.1. I have attached the
ensemble_stat
> config
> >>> file that I am using. Thanks for the quick reply!
> >>>
> >>> -Brian
> >>>
> >>>
> >>>
> >>>> On Sep 19, 2019, at 2:59 PM, John Halley Gotway via RT <
> >>> met_help at ucar.edu> wrote:
> >>>>
> >>>> Hi Brian,
> >>>>
> >>>> I see you've encountered an error from the Ensemble-Stat tool.
This
> >>> error
> >>>> message from NetCDF means that MET is attempting to write the
same
> >>> variable
> >>>> name to an output NetCDF file. I tried to replicate this
behavior
> >>> locally,
> >>>> by adding additional smoothing methods but wasn't able to
replicate
> it.
> >>>>
> >>>> Can you please...
> >>>> (1) Tell me what version of MET you're running.
> >>>> (2) Send me the EnsembleStatConfig file you're using so I can
see
> >>> exactly
> >>>> how you've configured it.
> >>>>
> >>>> Thanks,
> >>>> John Halley Gotway
> >>>>
> >>>>
> >>>> On Thu, Sep 19, 2019 at 1:40 PM Brian Matilla - NOAA Affiliate
via RT
> <
> >>>> met_help at ucar.edu> wrote:
> >>>>
> >>>>>
> >>>>> Thu Sep 19 13:40:08 2019: Request 92225 was acted upon.
> >>>>> Transaction: Ticket created by brian.matilla at noaa.gov
> >>>>> Queue: met_help
> >>>>> Subject: Issue with multiple interp widths in ensemble_stat
--
> >>> netCDF
> >>>>> error
> >>>>> Owner: Nobody
> >>>>> Requestors: brian.matilla at noaa.gov
> >>>>> Status: new
> >>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92225
> >
> >>>>>
> >>>>>
> >>>>> Hi once again MET helpdesk,
> >>>>>
> >>>>> After successfully processing my model and observation data
with both
> >>>>> ensemble_stat and grid_stat, I was looking to expand on that
by
> >>> introducing
> >>>>> different smoothing neighborhoods. I’ve set my ensemble_stat
config
> >>> file to
> >>>>> evaluate the forecast grids at widths of 1, 3, and 5 grid
points
> using
> >>> the
> >>>>> square method. The problem arises when I go to run
ensemble_stat
> where
> >>> the
> >>>>> following error shows up:
> >>>>>
> >>>>> DEBUG 2: Processing gridded verification APCP/A06 versus
APCP_06A06,
> >>> for
> >>>>> observation type APCP, over region WoFS_radars_20180618, for
> >>> interpolation
> >>>>> method NEAREST(1), using 21311 matched pairs.
> >>>>> DEBUG 2: Skipping 843 of 21311 points due to missing ensemble
values.
> >>>>> DEBUG 2: Processing gridded verification APCP/A06 versus
APCP_06A06,
> >>> for
> >>>>> observation type APCP, over region WoFS_radars_20180618, for
> >>> interpolation
> >>>>> method UW_MEAN_SQUARE(9), using 21311 matched pairs.
> >>>>> DEBUG 2: Skipping 972 of 21311 points due to missing ensemble
values.
> >>>>> terminate called after throwing an instance of
> >>>>> 'netCDF::exceptions::NcNameInUse'
> >>>>> what(): NetCDF: String match to name in use
> >>>>> file: ncGroup.cpp line:496
> >>>>> Abort (core dumped)
> >>>>>
> >>>>> What could be the nature of this error message? Is it just
that the
> >>> output
> >>>>> is being written to a file that is open so it leads to a file
lock
> >>> error?
> >>>>> As always, any help is greatly appreciated.
> >>>>>
> >>>>> Thanks!
> >>>>>
> >>>>> -Brian
> >>>>> —————————————————————
> >>>>> Brian Matilla
> >>>>> Research Fellow— Warn-on-Forecast Team
> >>>>> Cooperative Institute for Mesoscale Meteorological Studies —
The
> >>>>> University of Oklahoma
> >>>>> NOAA National Severe Storms Laboratory
> >>>>>
> >>>>> Phone: (405) 325-1688
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >
>
>
>
------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #92225] Issue with multiple interp widths in ensemble_stat -- netCDF error
From: Brian Matilla - NOAA Affiliate
Time: Fri Sep 20 16:46:31 2019
That all makes sense. I worked with grid_stat for a bit today and I
just smoothed the mean and probabilities that way. That appears to
have accomplished what I was working towards at the present time.
Looking forward to more great things in the future and again, thanks
very much for your help!
-Brian
> On Sep 20, 2019, at 10:30 AM, John Halley Gotway via RT
<met_help at ucar.edu> wrote:
>
> Brian,
>
> This is a detail I haven't thought about much before. But I can
tell you
> what the software is currently doing.
>
> The Ensemble-Stat tool really does two different things:
> (1) Derive ensemble mean and probabilities and write them to a
NetCDF *_
> ens.nc output file.
> (2) Compare the ensemble members to observations, compute ensemble
> statistics (like ranked histograms) and write the output to the
ASCII .stat
> file and NetCDF *_orank.nc file.
>
> The first step is controlled by the contents of the "ens" section of
the
> config file. You get ensemble means and probabilities for the
fields
> listed there. The second step is controlled by the "fcst" and "obs"
> sections of the config file. They define the comparisons for which
you
> want rank histograms and ensemble statistics.
>
> Currently, the "interp" section, which defines the smoothing
parameters, is
> only applied to the fcst/obs comparisons in step 2. That smoothing
> operation is NOT applied to the computation of ensemble means and
> probabilities. But honestly, I can't think of a good reason to say
why
> they are not. It just hasn't come up before.
>
> Logically, you could smooth the members prior to computing means and
> probabilities. Or you could smooth the mean and probabilities
afterwards
> when verifying them in Grid-Stat and Point-Stat. The former is not
> currently possible in Ensemble-Stat. But conceivably the tool could
be
> enhanced at some point to support that.
>
> Hope that helps clarify the logic.
>
> Thanks,
> John
>
>
>
>
> On Thu, Sep 19, 2019 at 5:42 PM Brian Matilla - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92225 >
>>
>> John,
>>
>> I appreciate you looking into the issue and I’m happy to say that
did
>> solve the problem on my front. Ensemble_stat ran successfully!
>>
>> I did have one follow up question though I’m not sure if it
warrants a
>> separate ticket or not — when I follow that up with running
grid_stat to
>> verify the ensemble probabilities, would I go ahead and set the
“nbrhd”
>> width equal to the smoothing neighborhood (e.g, if I use a 3x3
square box
>> in ensemble_stat, set the nbrhd width to 3 in grid_stat), or is
that
>> essentially taken care of when I run the smoother in ensemble_stat?
>>
>> -Brian
>>
>>> On Sep 19, 2019, at 5:16 PM, John Halley Gotway via RT <
>> met_help at ucar.edu> wrote:
>>>
>>> Brian,
>>>
>>> FYI, I found and fixed the underlying problem. Here's a
description of
>> the
>>> issue:
>>> https://github.com/NCAR/MET/issues/1198
>>>
>>> And that change will be included in the next bugfix release, met-
8.1.2.
>>>
>>> Thanks for uncovering this issue.
>>>
>>> John
>>>
>>> On Thu, Sep 19, 2019 at 2:48 PM John Halley Gotway
<johnhg at ucar.edu>
>> wrote:
>>>
>>>> Brian,
>>>>
>>>> Thanks for sending that. I was able to replicate the behavior
you
>>>> describe. If you edit your Ensemble-Stat config file and disable
the
>>>> orank NetCDF output file, that should get you a successful run:
>>>>
>>>> ensemble_flag = {
>>>> latlon = TRUE;
>>>> mean = TRUE;
>>>> stdev = TRUE;
>>>> minus = TRUE;
>>>> plus = TRUE;
>>>> min = TRUE;
>>>> max = TRUE;
>>>> range = TRUE;
>>>> vld_count = TRUE;
>>>> frequency = TRUE;
>>>> * rank = FALSE;*
>>>> weight = TRUE;
>>>> }
>>>>
>>>> Obviously I'd like to get a better solution in the next bugfix
release
>> of
>>>> MET, but that's a quick fix for today.
>>>>
>>>> Thanks,
>>>> John
>>>>
>>>> On Thu, Sep 19, 2019 at 2:14 PM Brian Matilla - NOAA Affiliate
via RT <
>>>> met_help at ucar.edu> wrote:
>>>>
>>>>>
>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92225 >
>>>>>
>>>>> Hi there John,
>>>>>
>>>>> So I am using MET version 8.1.1. I have attached the
ensemble_stat
>> config
>>>>> file that I am using. Thanks for the quick reply!
>>>>>
>>>>> -Brian
>>>>>
>>>>>
>>>>>
>>>>>> On Sep 19, 2019, at 2:59 PM, John Halley Gotway via RT <
>>>>> met_help at ucar.edu> wrote:
>>>>>>
>>>>>> Hi Brian,
>>>>>>
>>>>>> I see you've encountered an error from the Ensemble-Stat tool.
This
>>>>> error
>>>>>> message from NetCDF means that MET is attempting to write the
same
>>>>> variable
>>>>>> name to an output NetCDF file. I tried to replicate this
behavior
>>>>> locally,
>>>>>> by adding additional smoothing methods but wasn't able to
replicate
>> it.
>>>>>>
>>>>>> Can you please...
>>>>>> (1) Tell me what version of MET you're running.
>>>>>> (2) Send me the EnsembleStatConfig file you're using so I can
see
>>>>> exactly
>>>>>> how you've configured it.
>>>>>>
>>>>>> Thanks,
>>>>>> John Halley Gotway
>>>>>>
>>>>>>
>>>>>> On Thu, Sep 19, 2019 at 1:40 PM Brian Matilla - NOAA Affiliate
via RT
>> <
>>>>>> met_help at ucar.edu> wrote:
>>>>>>
>>>>>>>
>>>>>>> Thu Sep 19 13:40:08 2019: Request 92225 was acted upon.
>>>>>>> Transaction: Ticket created by brian.matilla at noaa.gov
>>>>>>> Queue: met_help
>>>>>>> Subject: Issue with multiple interp widths in ensemble_stat
--
>>>>> netCDF
>>>>>>> error
>>>>>>> Owner: Nobody
>>>>>>> Requestors: brian.matilla at noaa.gov
>>>>>>> Status: new
>>>>>>> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=92225
>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi once again MET helpdesk,
>>>>>>>
>>>>>>> After successfully processing my model and observation data
with both
>>>>>>> ensemble_stat and grid_stat, I was looking to expand on that
by
>>>>> introducing
>>>>>>> different smoothing neighborhoods. I’ve set my ensemble_stat
config
>>>>> file to
>>>>>>> evaluate the forecast grids at widths of 1, 3, and 5 grid
points
>> using
>>>>> the
>>>>>>> square method. The problem arises when I go to run
ensemble_stat
>> where
>>>>> the
>>>>>>> following error shows up:
>>>>>>>
>>>>>>> DEBUG 2: Processing gridded verification APCP/A06 versus
APCP_06A06,
>>>>> for
>>>>>>> observation type APCP, over region WoFS_radars_20180618, for
>>>>> interpolation
>>>>>>> method NEAREST(1), using 21311 matched pairs.
>>>>>>> DEBUG 2: Skipping 843 of 21311 points due to missing ensemble
values.
>>>>>>> DEBUG 2: Processing gridded verification APCP/A06 versus
APCP_06A06,
>>>>> for
>>>>>>> observation type APCP, over region WoFS_radars_20180618, for
>>>>> interpolation
>>>>>>> method UW_MEAN_SQUARE(9), using 21311 matched pairs.
>>>>>>> DEBUG 2: Skipping 972 of 21311 points due to missing ensemble
values.
>>>>>>> terminate called after throwing an instance of
>>>>>>> 'netCDF::exceptions::NcNameInUse'
>>>>>>> what(): NetCDF: String match to name in use
>>>>>>> file: ncGroup.cpp line:496
>>>>>>> Abort (core dumped)
>>>>>>>
>>>>>>> What could be the nature of this error message? Is it just
that the
>>>>> output
>>>>>>> is being written to a file that is open so it leads to a file
lock
>>>>> error?
>>>>>>> As always, any help is greatly appreciated.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> -Brian
>>>>>>> —————————————————————
>>>>>>> Brian Matilla
>>>>>>> Research Fellow— Warn-on-Forecast Team
>>>>>>> Cooperative Institute for Mesoscale Meteorological Studies —
The
>>>>>>> University of Oklahoma
>>>>>>> NOAA National Severe Storms Laboratory
>>>>>>>
>>>>>>> Phone: (405) 325-1688
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>
>>
>>
>>
>
------------------------------------------------
More information about the Met_help
mailing list