[Met_help] [rt.rap.ucar.edu #96557] History for TOTAL- Count of observations,
John Halley Gotway via RT
met_help at ucar.edu
Thu Oct 1 14:33:29 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello,
I have a quick question for the definition of Total on MET guide. If you
look at the page 152 on the MET guide:
https://dtcenter.org/sites/default/files/community-code/met/docs/user-guide/MET_Users_Guide_v9.0.3.pdf
On Table 9.7 TOTAL- Count of observations, is it the same as "count of
grids for matched pairs"? Eg. for grid to grid ensemble, count how many
grids have both non-zero obs. and each non-zero ensemble member (after
throwing out gridpoint with either zero individual members OR zero obs. ),
and then get the average among all the members?
Thank you.
Binyu
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: TOTAL- Count of observations,
From: Minna Win
Time: Thu Sep 03 13:35:29 2020
Hello Binyu,
It appears that you have questions about the definition of TOTAL in
ensemble-stat, as stated in table 9.7 page 252 of the user's guide.
I'm
not familiar with ensemble stat, I am assigning this ticket to John
Halley
Gotway.
Regards,
Minna
---------------
Minna Win
National Center for Atmospheric Research
Developmental Testbed Center
Phone: 303-497-8423
Fax: 303-497-8401
On Thu, Sep 3, 2020 at 12:57 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> Thu Sep 03 12:46:39 2020: Request 96557 was acted upon.
> Transaction: Ticket created by binyu.wang at noaa.gov
> Queue: met_help
> Subject: TOTAL- Count of observations,
> Owner: Nobody
> Requestors: binyu.wang at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96557 >
>
>
> Hello,
> I have a quick question for the definition of Total on MET guide.
If you
> look at the page 152 on the MET guide:
>
>
> https://dtcenter.org/sites/default/files/community-
code/met/docs/user-guide/MET_Users_Guide_v9.0.3.pdf
>
> On Table 9.7 TOTAL- Count of observations, is it the same as "count
of
> grids for matched pairs"? Eg. for grid to grid ensemble, count how
many
> grids have both non-zero obs. and each non-zero ensemble member
(after
> throwing out gridpoint with either zero individual members OR zero
obs. ),
> and then get the average among all the members?
>
> Thank you.
>
> Binyu
>
>
------------------------------------------------
Subject: TOTAL- Count of observations,
From: John Halley Gotway
Time: Thu Sep 03 13:56:15 2020
Binyu,
I see you have a question about the output of ensemble_stat.
FYI, since the met-9.1 release, we migrated the MET documentation from
PDF
file to a website. Here's a direct link to the table you are
referencing:
https://dtcenter.github.io/MET/Users_Guide/ensemble-stat.html#id2
Through the STAT output lines from the MET tools, the "TOTAL" column
is
used consistently to indicate the number of pairs that were used for
each
verification task. When verifying against point observations, its the
number of point observations within the verification masking region
that
were used. When verifying against gridded observations, its the number
of
grid points observations that were used.
For the ORANK line type, let's say you've used 100 observations.
You'll see
100 ORANK lines where TOTAL=100 on all of the lines. The next column,
INDEX, will increase from 1 up to 100.
Thanks,
John
On Thu, Sep 3, 2020 at 1:38 PM Minna Win via RT <met_help at ucar.edu>
wrote:
>
> Thu Sep 03 13:37:29 2020: Request 96557 was acted upon.
> Transaction: Given to johnhg (John Halley Gotway) by minnawin
> Queue: met_help
> Subject: TOTAL- Count of observations,
> Owner: johnhg
> Requestors: binyu.wang at noaa.gov
> Status: open
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96557 >
>
>
> This transaction appears to have no content
>
------------------------------------------------
Subject: TOTAL- Count of observations,
From: binyu.wang at noaa.gov
Time: Thu Sep 03 14:32:42 2020
So for ensemble_stat, it is NOT the mean of the matched-pair count for
each
ensemble member vs. obs, instead, it is the total count of how many
valid
(non-zero or above threshold) grids that have been used for all the
members.
eg:
we have 3 pairs for ensemble1 and obs at obs sites: grid1 grid2 grid3
4 pairs for ensemble1 and obs at obs sites: grid1 grid2
grid4 grid5
5 pairs for ensemble1 and obs at obs site : grid1 grid3
grid6 grid7 grid8
So TOTAL is 8.
I want to confirm this because this reminds me of another question I
asked
you before about ensemble HIST plot ( id=96453
<https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453>): in order
to
create a HIST plot, all the ensemble members must cover the obs. site.
So
there is no requirement for other STAT (like SSVAR)? All the LINE
types are
independent and have different rules?
Thank you.
Binyu
On Thu, Sep 3, 2020 at 3:56 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> I see you have a question about the output of ensemble_stat.
> FYI, since the met-9.1 release, we migrated the MET documentation
from PDF
> file to a website. Here's a direct link to the table you are
referencing:
> https://dtcenter.github.io/MET/Users_Guide/ensemble-stat.html#id2
>
> Through the STAT output lines from the MET tools, the "TOTAL" column
is
> used consistently to indicate the number of pairs that were used for
each
> verification task. When verifying against point observations, its
the
> number of point observations within the verification masking region
that
> were used. When verifying against gridded observations, its the
number of
> grid points observations that were used.
>
> For the ORANK line type, let's say you've used 100 observations.
You'll see
> 100 ORANK lines where TOTAL=100 on all of the lines. The next
column,
> INDEX, will increase from 1 up to 100.
>
> Thanks,
> John
>
> On Thu, Sep 3, 2020 at 1:38 PM Minna Win via RT <met_help at ucar.edu>
wrote:
>
> >
> > Thu Sep 03 13:37:29 2020: Request 96557 was acted upon.
> > Transaction: Given to johnhg (John Halley Gotway) by minnawin
> > Queue: met_help
> > Subject: TOTAL- Count of observations,
> > Owner: johnhg
> > Requestors: binyu.wang at noaa.gov
> > Status: open
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96557 >
> >
> >
> > This transaction appears to have no content
> >
>
>
------------------------------------------------
Subject: TOTAL- Count of observations,
From: John Halley Gotway
Time: Thu Sep 03 16:33:31 2020
Binyu,
No, that is incorrect.
TOTAL = the number of observations used in the current verification
task.
To illustrate, I've extracted columns 24, 25, 26, and then skipped to
35
and 36 from some sample ORANK output data:
LINE_TYPE TOTAL INDEX N_ENS_VLD N_ENS
ORANK 1125 1 6 6
ORANK 1125 2 6 6
ORANK 1125 3 6 6
ORANK 1125 4 6 6
ORANK 1125 5 6 6
ORANK 1125 6 6 6
ORANK 1125 7 6 6
ORANK 1125 8 6 6
ORANK 1125 9 6 6
...
This verification task, for whatever that data may have been, used
1125
observations. Notice that the TOTAL = 1125 remains constant across all
lines.
The INDEX column just counts up from 1 to 1125. The N_ENS column lists
the
number of ensemble members. And the N_ENS_VLD lists the number of
ensemble
members which contained a valid data value for this point. In this
sample
data, those are both constant at 6. The N_ENS column will always
remain
constant across these lines because the ensemble size does not change.
However, if you have some missing data value, the N_ENS_VLD counts
will
change.
Hope that helps clarify.
Thanks,
John
On Thu, Sep 3, 2020 at 2:33 PM binyu.wang at noaa.gov via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96557 >
>
> So for ensemble_stat, it is NOT the mean of the matched-pair count
for each
> ensemble member vs. obs, instead, it is the total count of how many
valid
> (non-zero or above threshold) grids that have been used for all the
> members.
> eg:
> we have 3 pairs for ensemble1 and obs at obs sites: grid1 grid2
grid3
> 4 pairs for ensemble1 and obs at obs sites: grid1
grid2
> grid4 grid5
> 5 pairs for ensemble1 and obs at obs site : grid1
grid3
> grid6 grid7 grid8
>
> So TOTAL is 8.
>
> I want to confirm this because this reminds me of another question
I asked
> you before about ensemble HIST plot ( id=96453
> <https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453>): in order
to
> create a HIST plot, all the ensemble members must cover the obs.
site. So
> there is no requirement for other STAT (like SSVAR)? All the LINE
types are
> independent and have different rules?
>
> Thank you.
> Binyu
>
> On Thu, Sep 3, 2020 at 3:56 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > I see you have a question about the output of ensemble_stat.
> > FYI, since the met-9.1 release, we migrated the MET documentation
from
> PDF
> > file to a website. Here's a direct link to the table you are
referencing:
> > https://dtcenter.github.io/MET/Users_Guide/ensemble-stat.html#id2
> >
> > Through the STAT output lines from the MET tools, the "TOTAL"
column is
> > used consistently to indicate the number of pairs that were used
for each
> > verification task. When verifying against point observations, its
the
> > number of point observations within the verification masking
region that
> > were used. When verifying against gridded observations, its the
number of
> > grid points observations that were used.
> >
> > For the ORANK line type, let's say you've used 100 observations.
You'll
> see
> > 100 ORANK lines where TOTAL=100 on all of the lines. The next
column,
> > INDEX, will increase from 1 up to 100.
> >
> > Thanks,
> > John
> >
> > On Thu, Sep 3, 2020 at 1:38 PM Minna Win via RT
<met_help at ucar.edu>
> wrote:
> >
> > >
> > > Thu Sep 03 13:37:29 2020: Request 96557 was acted upon.
> > > Transaction: Given to johnhg (John Halley Gotway) by minnawin
> > > Queue: met_help
> > > Subject: TOTAL- Count of observations,
> > > Owner: johnhg
> > > Requestors: binyu.wang at noaa.gov
> > > Status: open
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96557
> >
> > >
> > >
> > > This transaction appears to have no content
> > >
> >
> >
>
>
------------------------------------------------
Subject: TOTAL- Count of observations,
From: binyu.wang at noaa.gov
Time: Thu Sep 03 21:08:55 2020
Sorry I think I didn't make myself clear in the last email:
eg: there are 3 ensemble members, and overall there are only 10 obs.
sites(
or grids): we call it site1, site2, site3.....site10
member1 only matches site1, site2, site3
member2 only matches site1, site2, site4 and site5
member3 only matches site1, site2, site7 and site8
So TOTAL=10 or TOTAL=7 (site1+site2+site3+site4+site5+site7+site8)?
Or we can ONLY use "TOTAL" when ALL ensemble members cover the same
obs.
sites?
So TOTAL =2 (site1, site2)
Thank you.
On Thu, Sep 3, 2020 at 6:33 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> No, that is incorrect.
>
> TOTAL = the number of observations used in the current verification
task.
>
> To illustrate, I've extracted columns 24, 25, 26, and then skipped
to 35
> and 36 from some sample ORANK output data:
>
> LINE_TYPE TOTAL INDEX N_ENS_VLD N_ENS
>
> ORANK 1125 1 6 6
>
> ORANK 1125 2 6 6
>
> ORANK 1125 3 6 6
>
> ORANK 1125 4 6 6
>
> ORANK 1125 5 6 6
>
> ORANK 1125 6 6 6
>
> ORANK 1125 7 6 6
>
> ORANK 1125 8 6 6
>
> ORANK 1125 9 6 6
>
> ...
>
>
> This verification task, for whatever that data may have been, used
1125
> observations. Notice that the TOTAL = 1125 remains constant across
all
> lines.
>
> The INDEX column just counts up from 1 to 1125. The N_ENS column
lists the
> number of ensemble members. And the N_ENS_VLD lists the number of
ensemble
> members which contained a valid data value for this point. In this
sample
> data, those are both constant at 6. The N_ENS column will always
remain
> constant across these lines because the ensemble size does not
change.
> However, if you have some missing data value, the N_ENS_VLD counts
will
> change.
>
>
> Hope that helps clarify.
>
>
> Thanks,
> John
>
> On Thu, Sep 3, 2020 at 2:33 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96557 >
> >
> > So for ensemble_stat, it is NOT the mean of the matched-pair count
for
> each
> > ensemble member vs. obs, instead, it is the total count of how
many valid
> > (non-zero or above threshold) grids that have been used for all
the
> > members.
> > eg:
> > we have 3 pairs for ensemble1 and obs at obs sites: grid1 grid2
grid3
> > 4 pairs for ensemble1 and obs at obs sites: grid1
grid2
> > grid4 grid5
> > 5 pairs for ensemble1 and obs at obs site : grid1
grid3
> > grid6 grid7 grid8
> >
> > So TOTAL is 8.
> >
> > I want to confirm this because this reminds me of another
question I
> asked
> > you before about ensemble HIST plot ( id=96453
> > <https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453>): in
order to
> > create a HIST plot, all the ensemble members must cover the obs.
site. So
> > there is no requirement for other STAT (like SSVAR)? All the LINE
types
> are
> > independent and have different rules?
> >
> > Thank you.
> > Binyu
> >
> > On Thu, Sep 3, 2020 at 3:56 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Binyu,
> > >
> > > I see you have a question about the output of ensemble_stat.
> > > FYI, since the met-9.1 release, we migrated the MET
documentation from
> > PDF
> > > file to a website. Here's a direct link to the table you are
> referencing:
> > > https://dtcenter.github.io/MET/Users_Guide/ensemble-
stat.html#id2
> > >
> > > Through the STAT output lines from the MET tools, the "TOTAL"
column is
> > > used consistently to indicate the number of pairs that were used
for
> each
> > > verification task. When verifying against point observations,
its the
> > > number of point observations within the verification masking
region
> that
> > > were used. When verifying against gridded observations, its the
number
> of
> > > grid points observations that were used.
> > >
> > > For the ORANK line type, let's say you've used 100 observations.
You'll
> > see
> > > 100 ORANK lines where TOTAL=100 on all of the lines. The next
column,
> > > INDEX, will increase from 1 up to 100.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Sep 3, 2020 at 1:38 PM Minna Win via RT
<met_help at ucar.edu>
> > wrote:
> > >
> > > >
> > > > Thu Sep 03 13:37:29 2020: Request 96557 was acted upon.
> > > > Transaction: Given to johnhg (John Halley Gotway) by minnawin
> > > > Queue: met_help
> > > > Subject: TOTAL- Count of observations,
> > > > Owner: johnhg
> > > > Requestors: binyu.wang at noaa.gov
> > > > Status: open
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96557
> > >
> > > >
> > > >
> > > > This transaction appears to have no content
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: TOTAL- Count of observations,
From: John Halley Gotway
Time: Fri Sep 04 10:05:26 2020
Binyu,
Shouldn't you just be looking at the ORANK output lines that you're
getting
from ensemble_stat to see what these values are?
It sounds to me like...
- You are passing in 10 point observations as input.
- 7 of those 10 obs produce a match for at least one ensemble member.
- So assuming that you have the ensemble valid threshold set very low,
I
would expect that to produce 7 ORANK lines.
- Where TOTAL = 7, 7, 7, 7, 7, 7, 7 (constant value)
- INDEX = 1, 2, 3, 4, 5, 6, 7
- N_ENS = 3, 3, 3, 3, 3, 3, 3
- N_VLD_ENS = 3, 3, 1, 1, 1, 1, 1 (based on the matching you
described)
But I'd encourage you to look closely at the ORANK output lines.
John
On Thu, Sep 3, 2020 at 9:09 PM binyu.wang at noaa.gov via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96557 >
>
> Sorry I think I didn't make myself clear in the last email:
> eg: there are 3 ensemble members, and overall there are only 10 obs.
sites(
> or grids): we call it site1, site2, site3.....site10
> member1 only matches site1, site2, site3
> member2 only matches site1, site2, site4 and site5
> member3 only matches site1, site2, site7 and site8
>
> So TOTAL=10 or TOTAL=7 (site1+site2+site3+site4+site5+site7+site8)?
>
> Or we can ONLY use "TOTAL" when ALL ensemble members cover the same
obs.
> sites?
> So TOTAL =2 (site1, site2)
>
> Thank you.
>
>
>
> On Thu, Sep 3, 2020 at 6:33 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > No, that is incorrect.
> >
> > TOTAL = the number of observations used in the current
verification task.
> >
> > To illustrate, I've extracted columns 24, 25, 26, and then skipped
to 35
> > and 36 from some sample ORANK output data:
> >
> > LINE_TYPE TOTAL INDEX N_ENS_VLD N_ENS
> >
> > ORANK 1125 1 6 6
> >
> > ORANK 1125 2 6 6
> >
> > ORANK 1125 3 6 6
> >
> > ORANK 1125 4 6 6
> >
> > ORANK 1125 5 6 6
> >
> > ORANK 1125 6 6 6
> >
> > ORANK 1125 7 6 6
> >
> > ORANK 1125 8 6 6
> >
> > ORANK 1125 9 6 6
> >
> > ...
> >
> >
> > This verification task, for whatever that data may have been, used
1125
> > observations. Notice that the TOTAL = 1125 remains constant across
all
> > lines.
> >
> > The INDEX column just counts up from 1 to 1125. The N_ENS column
lists
> the
> > number of ensemble members. And the N_ENS_VLD lists the number of
> ensemble
> > members which contained a valid data value for this point. In this
sample
> > data, those are both constant at 6. The N_ENS column will always
remain
> > constant across these lines because the ensemble size does not
change.
> > However, if you have some missing data value, the N_ENS_VLD counts
will
> > change.
> >
> >
> > Hope that helps clarify.
> >
> >
> > Thanks,
> > John
> >
> > On Thu, Sep 3, 2020 at 2:33 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96557 >
> > >
> > > So for ensemble_stat, it is NOT the mean of the matched-pair
count for
> > each
> > > ensemble member vs. obs, instead, it is the total count of how
many
> valid
> > > (non-zero or above threshold) grids that have been used for all
the
> > > members.
> > > eg:
> > > we have 3 pairs for ensemble1 and obs at obs sites: grid1 grid2
grid3
> > > 4 pairs for ensemble1 and obs at obs sites: grid1
grid2
> > > grid4 grid5
> > > 5 pairs for ensemble1 and obs at obs site : grid1
grid3
> > > grid6 grid7 grid8
> > >
> > > So TOTAL is 8.
> > >
> > > I want to confirm this because this reminds me of another
question I
> > asked
> > > you before about ensemble HIST plot ( id=96453
> > > <https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453>): in
order
> to
> > > create a HIST plot, all the ensemble members must cover the obs.
site.
> So
> > > there is no requirement for other STAT (like SSVAR)? All the
LINE types
> > are
> > > independent and have different rules?
> > >
> > > Thank you.
> > > Binyu
> > >
> > > On Thu, Sep 3, 2020 at 3:56 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Binyu,
> > > >
> > > > I see you have a question about the output of ensemble_stat.
> > > > FYI, since the met-9.1 release, we migrated the MET
documentation
> from
> > > PDF
> > > > file to a website. Here's a direct link to the table you are
> > referencing:
> > > > https://dtcenter.github.io/MET/Users_Guide/ensemble-
stat.html#id2
> > > >
> > > > Through the STAT output lines from the MET tools, the "TOTAL"
column
> is
> > > > used consistently to indicate the number of pairs that were
used for
> > each
> > > > verification task. When verifying against point observations,
its the
> > > > number of point observations within the verification masking
region
> > that
> > > > were used. When verifying against gridded observations, its
the
> number
> > of
> > > > grid points observations that were used.
> > > >
> > > > For the ORANK line type, let's say you've used 100
observations.
> You'll
> > > see
> > > > 100 ORANK lines where TOTAL=100 on all of the lines. The next
column,
> > > > INDEX, will increase from 1 up to 100.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Thu, Sep 3, 2020 at 1:38 PM Minna Win via RT
<met_help at ucar.edu>
> > > wrote:
> > > >
> > > > >
> > > > > Thu Sep 03 13:37:29 2020: Request 96557 was acted upon.
> > > > > Transaction: Given to johnhg (John Halley Gotway) by
minnawin
> > > > > Queue: met_help
> > > > > Subject: TOTAL- Count of observations,
> > > > > Owner: johnhg
> > > > > Requestors: binyu.wang at noaa.gov
> > > > > Status: open
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96557
> > > >
> > > > >
> > > > >
> > > > > This transaction appears to have no content
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: TOTAL- Count of observations,
From: binyu.wang at noaa.gov
Time: Fri Sep 04 12:33:01 2020
Thank you, John.
Actually, I never get *rank.txt" file (I can get all other stat txt
file,
but not *rank.txt.) even I did set "orank = BOTH; " in the config
file.
This is why I can not test it using my own script.
Here is the config file I use
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/verf_g2g_ens_stat_regn_config_Raikoke
The script:
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/verf_g2g_Raikoke.sh
Output dir:
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Test
Any hint on this? Thank you.
Binyu
On Fri, Sep 4, 2020 at 12:05 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> Shouldn't you just be looking at the ORANK output lines that you're
getting
> from ensemble_stat to see what these values are?
>
> It sounds to me like...
> - You are passing in 10 point observations as input.
> - 7 of those 10 obs produce a match for at least one ensemble
member.
> - So assuming that you have the ensemble valid threshold set very
low, I
> would expect that to produce 7 ORANK lines.
> - Where TOTAL = 7, 7, 7, 7, 7, 7, 7 (constant value)
> - INDEX = 1, 2, 3, 4, 5, 6, 7
> - N_ENS = 3, 3, 3, 3, 3, 3, 3
> - N_VLD_ENS = 3, 3, 1, 1, 1, 1, 1 (based on the matching you
described)
>
> But I'd encourage you to look closely at the ORANK output lines.
>
> John
>
> On Thu, Sep 3, 2020 at 9:09 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu>
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96557 >
> >
> > Sorry I think I didn't make myself clear in the last email:
> > eg: there are 3 ensemble members, and overall there are only 10
obs.
> sites(
> > or grids): we call it site1, site2, site3.....site10
> > member1 only matches site1, site2, site3
> > member2 only matches site1, site2, site4 and site5
> > member3 only matches site1, site2, site7 and site8
> >
> > So TOTAL=10 or TOTAL=7
(site1+site2+site3+site4+site5+site7+site8)?
> >
> > Or we can ONLY use "TOTAL" when ALL ensemble members cover the
same obs.
> > sites?
> > So TOTAL =2 (site1, site2)
> >
> > Thank you.
> >
> >
> >
> > On Thu, Sep 3, 2020 at 6:33 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Binyu,
> > >
> > > No, that is incorrect.
> > >
> > > TOTAL = the number of observations used in the current
verification
> task.
> > >
> > > To illustrate, I've extracted columns 24, 25, 26, and then
skipped to
> 35
> > > and 36 from some sample ORANK output data:
> > >
> > > LINE_TYPE TOTAL INDEX N_ENS_VLD N_ENS
> > >
> > > ORANK 1125 1 6 6
> > >
> > > ORANK 1125 2 6 6
> > >
> > > ORANK 1125 3 6 6
> > >
> > > ORANK 1125 4 6 6
> > >
> > > ORANK 1125 5 6 6
> > >
> > > ORANK 1125 6 6 6
> > >
> > > ORANK 1125 7 6 6
> > >
> > > ORANK 1125 8 6 6
> > >
> > > ORANK 1125 9 6 6
> > >
> > > ...
> > >
> > >
> > > This verification task, for whatever that data may have been,
used 1125
> > > observations. Notice that the TOTAL = 1125 remains constant
across all
> > > lines.
> > >
> > > The INDEX column just counts up from 1 to 1125. The N_ENS column
lists
> > the
> > > number of ensemble members. And the N_ENS_VLD lists the number
of
> > ensemble
> > > members which contained a valid data value for this point. In
this
> sample
> > > data, those are both constant at 6. The N_ENS column will always
remain
> > > constant across these lines because the ensemble size does not
change.
> > > However, if you have some missing data value, the N_ENS_VLD
counts will
> > > change.
> > >
> > >
> > > Hope that helps clarify.
> > >
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Sep 3, 2020 at 2:33 PM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96557
>
> > > >
> > > > So for ensemble_stat, it is NOT the mean of the matched-pair
count
> for
> > > each
> > > > ensemble member vs. obs, instead, it is the total count of how
many
> > valid
> > > > (non-zero or above threshold) grids that have been used for
all the
> > > > members.
> > > > eg:
> > > > we have 3 pairs for ensemble1 and obs at obs sites: grid1
grid2 grid3
> > > > 4 pairs for ensemble1 and obs at obs sites:
grid1
> grid2
> > > > grid4 grid5
> > > > 5 pairs for ensemble1 and obs at obs site :
grid1
> grid3
> > > > grid6 grid7 grid8
> > > >
> > > > So TOTAL is 8.
> > > >
> > > > I want to confirm this because this reminds me of another
question I
> > > asked
> > > > you before about ensemble HIST plot ( id=96453
> > > > <https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453>): in
order
> > to
> > > > create a HIST plot, all the ensemble members must cover the
obs.
> site.
> > So
> > > > there is no requirement for other STAT (like SSVAR)? All the
LINE
> types
> > > are
> > > > independent and have different rules?
> > > >
> > > > Thank you.
> > > > Binyu
> > > >
> > > > On Thu, Sep 3, 2020 at 3:56 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Binyu,
> > > > >
> > > > > I see you have a question about the output of ensemble_stat.
> > > > > FYI, since the met-9.1 release, we migrated the MET
documentation
> > from
> > > > PDF
> > > > > file to a website. Here's a direct link to the table you are
> > > referencing:
> > > > > https://dtcenter.github.io/MET/Users_Guide/ensemble-
stat.html#id2
> > > > >
> > > > > Through the STAT output lines from the MET tools, the
"TOTAL"
> column
> > is
> > > > > used consistently to indicate the number of pairs that were
used
> for
> > > each
> > > > > verification task. When verifying against point
observations, its
> the
> > > > > number of point observations within the verification masking
region
> > > that
> > > > > were used. When verifying against gridded observations, its
the
> > number
> > > of
> > > > > grid points observations that were used.
> > > > >
> > > > > For the ORANK line type, let's say you've used 100
observations.
> > You'll
> > > > see
> > > > > 100 ORANK lines where TOTAL=100 on all of the lines. The
next
> column,
> > > > > INDEX, will increase from 1 up to 100.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Thu, Sep 3, 2020 at 1:38 PM Minna Win via RT
<met_help at ucar.edu
> >
> > > > wrote:
> > > > >
> > > > > >
> > > > > > Thu Sep 03 13:37:29 2020: Request 96557 was acted upon.
> > > > > > Transaction: Given to johnhg (John Halley Gotway) by
minnawin
> > > > > > Queue: met_help
> > > > > > Subject: TOTAL- Count of observations,
> > > > > > Owner: johnhg
> > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > Status: open
> > > > > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96557
> > > > >
> > > > > >
> > > > > >
> > > > > > This transaction appears to have no content
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: TOTAL- Count of observations,
From: John Halley Gotway
Time: Tue Sep 08 14:35:50 2020
Binyu,
OK, I understand. I took a look in your script:
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/
verf_met_ens/ush/verf_g2g_Raikoke.sh
And see that you're passing gridded observations using the "-grid_obs"
flag. Ensemble-Stat does NOT write the ORANK line type when gridded
observations are used... only for point observations. Typically there
are
so many observation grid points that it isn't realistic to write an
ASCII
ORANK line type to the output for each one. And that's why you're are
not
seeing them in the output.
Thanks,
John
On Fri, Sep 4, 2020 at 12:33 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96557 >
>
> Thank you, John.
> Actually, I never get *rank.txt" file (I can get all other stat txt
file,
> but not *rank.txt.) even I did set "orank = BOTH; " in the config
file.
> This is why I can not test it using my own script.
>
> Here is the config file I use
>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/parm/verf_g2g_ens_stat_regn_config_Raikoke
> The script:
>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/verf_g2g_Raikoke.sh
> Output dir:
>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/grid2grid/verf_met_ens/ush/out/ensemble_stat/Test
>
> Any hint on this? Thank you.
>
> Binyu
>
> On Fri, Sep 4, 2020 at 12:05 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > Shouldn't you just be looking at the ORANK output lines that
you're
> getting
> > from ensemble_stat to see what these values are?
> >
> > It sounds to me like...
> > - You are passing in 10 point observations as input.
> > - 7 of those 10 obs produce a match for at least one ensemble
member.
> > - So assuming that you have the ensemble valid threshold set very
low, I
> > would expect that to produce 7 ORANK lines.
> > - Where TOTAL = 7, 7, 7, 7, 7, 7, 7 (constant value)
> > - INDEX = 1, 2, 3, 4, 5, 6, 7
> > - N_ENS = 3, 3, 3, 3, 3, 3, 3
> > - N_VLD_ENS = 3, 3, 1, 1, 1, 1, 1 (based on the matching you
described)
> >
> > But I'd encourage you to look closely at the ORANK output lines.
> >
> > John
> >
> > On Thu, Sep 3, 2020 at 9:09 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96557 >
> > >
> > > Sorry I think I didn't make myself clear in the last email:
> > > eg: there are 3 ensemble members, and overall there are only 10
obs.
> > sites(
> > > or grids): we call it site1, site2, site3.....site10
> > > member1 only matches site1, site2, site3
> > > member2 only matches site1, site2, site4 and site5
> > > member3 only matches site1, site2, site7 and site8
> > >
> > > So TOTAL=10 or TOTAL=7
(site1+site2+site3+site4+site5+site7+site8)?
> > >
> > > Or we can ONLY use "TOTAL" when ALL ensemble members cover the
same
> obs.
> > > sites?
> > > So TOTAL =2 (site1, site2)
> > >
> > > Thank you.
> > >
> > >
> > >
> > > On Thu, Sep 3, 2020 at 6:33 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Binyu,
> > > >
> > > > No, that is incorrect.
> > > >
> > > > TOTAL = the number of observations used in the current
verification
> > task.
> > > >
> > > > To illustrate, I've extracted columns 24, 25, 26, and then
skipped to
> > 35
> > > > and 36 from some sample ORANK output data:
> > > >
> > > > LINE_TYPE TOTAL INDEX N_ENS_VLD N_ENS
> > > >
> > > > ORANK 1125 1 6 6
> > > >
> > > > ORANK 1125 2 6 6
> > > >
> > > > ORANK 1125 3 6 6
> > > >
> > > > ORANK 1125 4 6 6
> > > >
> > > > ORANK 1125 5 6 6
> > > >
> > > > ORANK 1125 6 6 6
> > > >
> > > > ORANK 1125 7 6 6
> > > >
> > > > ORANK 1125 8 6 6
> > > >
> > > > ORANK 1125 9 6 6
> > > >
> > > > ...
> > > >
> > > >
> > > > This verification task, for whatever that data may have been,
used
> 1125
> > > > observations. Notice that the TOTAL = 1125 remains constant
across
> all
> > > > lines.
> > > >
> > > > The INDEX column just counts up from 1 to 1125. The N_ENS
column
> lists
> > > the
> > > > number of ensemble members. And the N_ENS_VLD lists the number
of
> > > ensemble
> > > > members which contained a valid data value for this point. In
this
> > sample
> > > > data, those are both constant at 6. The N_ENS column will
always
> remain
> > > > constant across these lines because the ensemble size does not
> change.
> > > > However, if you have some missing data value, the N_ENS_VLD
counts
> will
> > > > change.
> > > >
> > > >
> > > > Hope that helps clarify.
> > > >
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Thu, Sep 3, 2020 at 2:33 PM binyu.wang at noaa.gov via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96557 >
> > > > >
> > > > > So for ensemble_stat, it is NOT the mean of the matched-pair
count
> > for
> > > > each
> > > > > ensemble member vs. obs, instead, it is the total count of
how many
> > > valid
> > > > > (non-zero or above threshold) grids that have been used for
all the
> > > > > members.
> > > > > eg:
> > > > > we have 3 pairs for ensemble1 and obs at obs sites: grid1
grid2
> grid3
> > > > > 4 pairs for ensemble1 and obs at obs sites:
grid1
> > grid2
> > > > > grid4 grid5
> > > > > 5 pairs for ensemble1 and obs at obs site :
grid1
> > grid3
> > > > > grid6 grid7 grid8
> > > > >
> > > > > So TOTAL is 8.
> > > > >
> > > > > I want to confirm this because this reminds me of another
> question I
> > > > asked
> > > > > you before about ensemble HIST plot ( id=96453
> > > > > <https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453>):
in
> order
> > > to
> > > > > create a HIST plot, all the ensemble members must cover the
obs.
> > site.
> > > So
> > > > > there is no requirement for other STAT (like SSVAR)? All the
LINE
> > types
> > > > are
> > > > > independent and have different rules?
> > > > >
> > > > > Thank you.
> > > > > Binyu
> > > > >
> > > > > On Thu, Sep 3, 2020 at 3:56 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Binyu,
> > > > > >
> > > > > > I see you have a question about the output of
ensemble_stat.
> > > > > > FYI, since the met-9.1 release, we migrated the MET
documentation
> > > from
> > > > > PDF
> > > > > > file to a website. Here's a direct link to the table you
are
> > > > referencing:
> > > > > >
> https://dtcenter.github.io/MET/Users_Guide/ensemble-stat.html#id2
> > > > > >
> > > > > > Through the STAT output lines from the MET tools, the
"TOTAL"
> > column
> > > is
> > > > > > used consistently to indicate the number of pairs that
were used
> > for
> > > > each
> > > > > > verification task. When verifying against point
observations, its
> > the
> > > > > > number of point observations within the verification
masking
> region
> > > > that
> > > > > > were used. When verifying against gridded observations,
its the
> > > number
> > > > of
> > > > > > grid points observations that were used.
> > > > > >
> > > > > > For the ORANK line type, let's say you've used 100
observations.
> > > You'll
> > > > > see
> > > > > > 100 ORANK lines where TOTAL=100 on all of the lines. The
next
> > column,
> > > > > > INDEX, will increase from 1 up to 100.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Thu, Sep 3, 2020 at 1:38 PM Minna Win via RT <
> met_help at ucar.edu
> > >
> > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > Thu Sep 03 13:37:29 2020: Request 96557 was acted upon.
> > > > > > > Transaction: Given to johnhg (John Halley Gotway) by
minnawin
> > > > > > > Queue: met_help
> > > > > > > Subject: TOTAL- Count of observations,
> > > > > > > Owner: johnhg
> > > > > > > Requestors: binyu.wang at noaa.gov
> > > > > > > Status: open
> > > > > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96557
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > This transaction appears to have no content
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list