[Met_help] [rt.rap.ucar.edu #89081] History for Ensemble-Stat CRPSS is NA.
John Halley Gotway via RT
met_help at ucar.edu
Wed Feb 27 10:09:27 MST 2019
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
MET-based HREF ensemble CRPSS plot can not be generated in MEtViewer. Tatiana checked this for me and found that the crpss in the database is “NA”. I re-run the ensemble_stat and stat output shows the crpss column is “NA”. So the problem seems in ensemble_stat tool.
I have sent the complete test case onto theia:
/scratch4/NCEPDEV/fv3-cam/noscrub/Binbin.Zhou/met_test/HREF/href_20190211
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Ensemble-Stat CRPSS is NA.
From: John Halley Gotway
Time: Mon Feb 25 15:25:09 2019
Binbin,
I used the Ensemble-Stat issue you raised to create this MET-Help
ticket. You're wondering why you got a CRPSS value of NA when you ran
Ensemble-Stat on HREF data.
Thanks for pointing me to your sample data on theia. That helped a
lot.
The answer is that the what you're using as the climatological mean
field does not fully cover your verification region. Since some of
the climo values are bad data, Ensemble-Stat doesn't compute a
climatological CRPS value and therefore CRPSS is not computed either.
In the Ensemble-Stat config file, I see:
(1) You are scoring over the FULL model domain:
mask = {
grid = [ "FULL" ];
poly = [];
sid = [];
}
(2) You are using using the namnset as the climo mean field:
climo_mean = {
file_name = "namnest.t18z.grid184.f06.grib2";
When I plot TMP/Z2 from the namnest data I see swaths of missing data
along the edge... and that's why CRPSS = NA for the FULL domain.
However, when I score over CONUS.poly which includes good data for
fcst, obs, and climo, I do get a CRPSS value:
mask = {
grid = [ "FULL" ];
poly = [ "MET_BASE/poly/CONUS.poly" ];
sid = [];
}
Make sense?
Thanks,
John
------------------------------------------------
Subject: Ensemble-Stat CRPSS is NA.
From: Binbin.Zhou at noaa.gov
Time: Mon Feb 25 16:53:51 2019
John,
I'll try this solution. Can I use some filter in mask to avoid bad
grids
in reference file?
Binbin
On Mon, Feb 25, 2019 at 5:25 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binbin,
>
> I used the Ensemble-Stat issue you raised to create this MET-Help
ticket.
> You're wondering why you got a CRPSS value of NA when you ran
Ensemble-Stat
> on HREF data.
>
> Thanks for pointing me to your sample data on theia. That helped a
lot.
>
> The answer is that the what you're using as the climatological mean
field
> does not fully cover your verification region. Since some of the
climo
> values are bad data, Ensemble-Stat doesn't compute a climatological
CRPS
> value and therefore CRPSS is not computed either.
>
> In the Ensemble-Stat config file, I see:
>
> (1) You are scoring over the FULL model domain:
>
> mask = {
> grid = [ "FULL" ];
> poly = [];
> sid = [];
> }
>
> (2) You are using using the namnset as the climo mean field:
>
> climo_mean = {
> file_name = "namnest.t18z.grid184.f06.grib2";
>
> When I plot TMP/Z2 from the namnest data I see swaths of missing
data
> along the edge... and that's why CRPSS = NA for the FULL domain.
>
> However, when I score over CONUS.poly which includes good data for
fcst,
> obs, and climo, I do get a CRPSS value:
>
> mask = {
> grid = [ "FULL" ];
> poly = [ "MET_BASE/poly/CONUS.poly" ];
> sid = [];
> }
>
> Make sense?
>
> Thanks,
> John
>
------------------------------------------------
Subject: Ensemble-Stat CRPSS is NA.
From: John Halley Gotway
Time: Mon Feb 25 17:10:27 2019
Binbin,
I'm not exactly sure how to interpret you're question. Are you
asking, can
you use the extent of valid data in the reference file to define the
region
over which you want to compute the statistics?
If that's your question, then the answer is yes. I would suggest:
(1) Run gen_vx_mask to pick out the valid climo data points:
*gen_vx_mask namnest.t18z.grid184.f06.grib2 \*
*namnest.t18z.grid184.f06.grib2 \*
*climo_mask.nc <http://climo_mask.nc> \*
*-type data \*
*-mask_field 'name="TMP"; level="Z2";' \*
*-thresh !=-9999 -name CLIMO_MASK -v 3*
(2) List that mask file in your ensemble-stat config file:
*poly = [ "climo_mask.nc <http://climo_mask.nc>" ];*
Thanks,
John
On Mon, Feb 25, 2019 at 4:53 PM Binbin.Zhou at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89081 >
>
> John,
>
> I'll try this solution. Can I use some filter in mask to avoid
bad grids
> in reference file?
>
> Binbin
>
> On Mon, Feb 25, 2019 at 5:25 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binbin,
> >
> > I used the Ensemble-Stat issue you raised to create this MET-Help
ticket.
> > You're wondering why you got a CRPSS value of NA when you ran
> Ensemble-Stat
> > on HREF data.
> >
> > Thanks for pointing me to your sample data on theia. That helped
a lot.
> >
> > The answer is that the what you're using as the climatological
mean field
> > does not fully cover your verification region. Since some of the
climo
> > values are bad data, Ensemble-Stat doesn't compute a
climatological CRPS
> > value and therefore CRPSS is not computed either.
> >
> > In the Ensemble-Stat config file, I see:
> >
> > (1) You are scoring over the FULL model domain:
> >
> > mask = {
> > grid = [ "FULL" ];
> > poly = [];
> > sid = [];
> > }
> >
> > (2) You are using using the namnset as the climo mean field:
> >
> > climo_mean = {
> > file_name = "namnest.t18z.grid184.f06.grib2";
> >
> > When I plot TMP/Z2 from the namnest data I see swaths of missing
data
> > along the edge... and that's why CRPSS = NA for the FULL domain.
> >
> > However, when I score over CONUS.poly which includes good data for
fcst,
> > obs, and climo, I do get a CRPSS value:
> >
> > mask = {
> > grid = [ "FULL" ];
> > poly = [ "MET_BASE/poly/CONUS.poly" ];
> > sid = [];
> > }
> >
> > Make sense?
> >
> > Thanks,
> > John
> >
>
>
------------------------------------------------
Subject: Ensemble-Stat CRPSS is NA.
From: Binbin.Zhou at noaa.gov
Time: Mon Feb 25 18:33:43 2019
John,
I'll test both solutions.
Thanks!
Binbin
On Mon, Feb 25, 2019 at 7:10 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binbin,
>
> I'm not exactly sure how to interpret you're question. Are you
asking, can
> you use the extent of valid data in the reference file to define the
region
> over which you want to compute the statistics?
>
> If that's your question, then the answer is yes. I would suggest:
>
> (1) Run gen_vx_mask to pick out the valid climo data points:
>
> *gen_vx_mask namnest.t18z.grid184.f06.grib2 \*
> *namnest.t18z.grid184.f06.grib2 \*
> *climo_mask.nc <http://climo_mask.nc> \*
> *-type data \*
> *-mask_field 'name="TMP"; level="Z2";' \*
> *-thresh !=-9999 -name CLIMO_MASK -v 3*
>
> (2) List that mask file in your ensemble-stat config file:
>
> *poly = [ "climo_mask.nc <http://climo_mask.nc>" ];*
>
> Thanks,
> John
>
>
>
> On Mon, Feb 25, 2019 at 4:53 PM Binbin.Zhou at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89081 >
> >
> > John,
> >
> > I'll try this solution. Can I use some filter in mask to avoid
bad
> grids
> > in reference file?
> >
> > Binbin
> >
> > On Mon, Feb 25, 2019 at 5:25 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Binbin,
> > >
> > > I used the Ensemble-Stat issue you raised to create this MET-
Help
> ticket.
> > > You're wondering why you got a CRPSS value of NA when you ran
> > Ensemble-Stat
> > > on HREF data.
> > >
> > > Thanks for pointing me to your sample data on theia. That
helped a
> lot.
> > >
> > > The answer is that the what you're using as the climatological
mean
> field
> > > does not fully cover your verification region. Since some of
the climo
> > > values are bad data, Ensemble-Stat doesn't compute a
climatological
> CRPS
> > > value and therefore CRPSS is not computed either.
> > >
> > > In the Ensemble-Stat config file, I see:
> > >
> > > (1) You are scoring over the FULL model domain:
> > >
> > > mask = {
> > > grid = [ "FULL" ];
> > > poly = [];
> > > sid = [];
> > > }
> > >
> > > (2) You are using using the namnset as the climo mean field:
> > >
> > > climo_mean = {
> > > file_name = "namnest.t18z.grid184.f06.grib2";
> > >
> > > When I plot TMP/Z2 from the namnest data I see swaths of missing
data
> > > along the edge... and that's why CRPSS = NA for the FULL domain.
> > >
> > > However, when I score over CONUS.poly which includes good data
for
> fcst,
> > > obs, and climo, I do get a CRPSS value:
> > >
> > > mask = {
> > > grid = [ "FULL" ];
> > > poly = [ "MET_BASE/poly/CONUS.poly" ];
> > > sid = [];
> > > }
> > >
> > > Make sense?
> > >
> > > Thanks,
> > > John
> > >
> >
> >
>
>
------------------------------------------------
Subject: Ensemble-Stat CRPSS is NA.
From: Binbin.Zhou at noaa.gov
Time: Wed Feb 27 09:54:00 2019
John,
I have tested both solutions, both working fine. The issue is
solved.
Thanks!
Binbin
On Mon, Feb 25, 2019 at 8:33 PM Binbin Zhou - NOAA Affiliate <
binbin.zhou at noaa.gov> wrote:
> John,
>
> I'll test both solutions.
> Thanks!
>
> Binbin
>
> On Mon, Feb 25, 2019 at 7:10 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Binbin,
>>
>> I'm not exactly sure how to interpret you're question. Are you
asking,
>> can
>> you use the extent of valid data in the reference file to define
the
>> region
>> over which you want to compute the statistics?
>>
>> If that's your question, then the answer is yes. I would suggest:
>>
>> (1) Run gen_vx_mask to pick out the valid climo data points:
>>
>> *gen_vx_mask namnest.t18z.grid184.f06.grib2 \*
>> *namnest.t18z.grid184.f06.grib2 \*
>> *climo_mask.nc <http://climo_mask.nc> \*
>> *-type data \*
>> *-mask_field 'name="TMP"; level="Z2";' \*
>> *-thresh !=-9999 -name CLIMO_MASK -v 3*
>>
>> (2) List that mask file in your ensemble-stat config file:
>>
>> *poly = [ "climo_mask.nc <http://climo_mask.nc>" ];*
>>
>> Thanks,
>> John
>>
>>
>>
>> On Mon, Feb 25, 2019 at 4:53 PM Binbin.Zhou at noaa.gov via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=89081 >
>> >
>> > John,
>> >
>> > I'll try this solution. Can I use some filter in mask to avoid
bad
>> grids
>> > in reference file?
>> >
>> > Binbin
>> >
>> > On Mon, Feb 25, 2019 at 5:25 PM John Halley Gotway via RT <
>> > met_help at ucar.edu>
>> > wrote:
>> >
>> > > Binbin,
>> > >
>> > > I used the Ensemble-Stat issue you raised to create this MET-
Help
>> ticket.
>> > > You're wondering why you got a CRPSS value of NA when you ran
>> > Ensemble-Stat
>> > > on HREF data.
>> > >
>> > > Thanks for pointing me to your sample data on theia. That
helped a
>> lot.
>> > >
>> > > The answer is that the what you're using as the climatological
mean
>> field
>> > > does not fully cover your verification region. Since some of
the
>> climo
>> > > values are bad data, Ensemble-Stat doesn't compute a
climatological
>> CRPS
>> > > value and therefore CRPSS is not computed either.
>> > >
>> > > In the Ensemble-Stat config file, I see:
>> > >
>> > > (1) You are scoring over the FULL model domain:
>> > >
>> > > mask = {
>> > > grid = [ "FULL" ];
>> > > poly = [];
>> > > sid = [];
>> > > }
>> > >
>> > > (2) You are using using the namnset as the climo mean field:
>> > >
>> > > climo_mean = {
>> > > file_name = "namnest.t18z.grid184.f06.grib2";
>> > >
>> > > When I plot TMP/Z2 from the namnest data I see swaths of
missing data
>> > > along the edge... and that's why CRPSS = NA for the FULL
domain.
>> > >
>> > > However, when I score over CONUS.poly which includes good data
for
>> fcst,
>> > > obs, and climo, I do get a CRPSS value:
>> > >
>> > > mask = {
>> > > grid = [ "FULL" ];
>> > > poly = [ "MET_BASE/poly/CONUS.poly" ];
>> > > sid = [];
>> > > }
>> > >
>> > > Make sense?
>> > >
>> > > Thanks,
>> > > John
>> > >
>> >
>> >
>>
>>
------------------------------------------------
More information about the Met_help
mailing list