[Met_help] [rt.rap.ucar.edu #98261] History for May you please help me to set up a config file to generate ROC curves?
Tatiana Burek via RT
met_help at ucar.edu
Mon Feb 8 07:23:00 MST 2021
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hi,
I come back to ask for help again. All related files are transferred to
your ftp server, ftp.rap.ucar.edu, under the folder:
/incoming/irap/met_help/mao
Script: test4help.sh (Start with it generate a stat file, but it may need a
bit modifications)
config file: wafs_verf_g2g_config.icesev (Here I need your help)
obs file: obsv.icesev.grib2
fcst file: fcst.icesev.grib2
Grib2 configuration file: grib2_icesev.txt (icesev is a very new field
name, grib2 lib hasn't been updated with it yet, so MET_GRIB_TABLES
environment variable is defined in the test script)
Spatial Masking files: *nc
I was able to generate a stat file by running the test script,
test4help.sh. But the plotting looks weird, at least not like a ROC curve.
Please check more details of the plotting from the history 20210115_200405
on https://metviewer.nws.noaa.gov/metviewer1.jsp
So I must have maken mistakes when setting up wafs_verf_g2g_config.icesev.
Both observation and forecast data are grib2 data, both values are integers
at 0,1,2,3,4. For observations, I can define a 'yes' event when value>=1,
and I want to plot ROC curve based on the forecast values at >=1, >=2, >=3,
=4.
Thank you.
Yali
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: May you please help me to set up a config file to generate ROC curves?
From: David Fillmore
Time: Fri Jan 15 14:24:04 2021
Hi Yali -
I'm watching met_help today and have retrieved your data and config
files
for grid_stat testing.
Are you able to attach any of the bad ROC curves to this email chain?
I will likely try to reproduce with the METplotpy roc_diagram.py
script.
I have cc'ed John on this ticket so he can also have a look at your
config
file.
thanks,
David
On Fri, Jan 15, 2021 at 1:48 PM Yali Mao - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> Fri Jan 15 13:47:43 2021: Request 98261 was acted upon.
> Transaction: Ticket created by yali.mao at noaa.gov
> Queue: met_help
> Subject: May you please help me to set up a config file to
generate
> ROC curves?
> Owner: Nobody
> Requestors: yali.mao at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261 >
>
>
> Hi,
> I come back to ask for help again. All related files are transferred
to
> your ftp server, ftp.rap.ucar.edu, under the folder:
> /incoming/irap/met_help/mao
> Script: test4help.sh (Start with it generate a stat file, but it may
need a
> bit modifications)
> config file: wafs_verf_g2g_config.icesev (Here I need your help)
> obs file: obsv.icesev.grib2
> fcst file: fcst.icesev.grib2
> Grib2 configuration file: grib2_icesev.txt (icesev is a very new
field
> name, grib2 lib hasn't been updated with it yet, so MET_GRIB_TABLES
> environment variable is defined in the test script)
> Spatial Masking files: *nc
>
> I was able to generate a stat file by running the test script,
> test4help.sh. But the plotting looks weird, at least not like a ROC
curve.
> Please check more details of the plotting from the history
20210115_200405
> on https://metviewer.nws.noaa.gov/metviewer1.jsp
>
> So I must have maken mistakes when setting up
wafs_verf_g2g_config.icesev.
> Both observation and forecast data are grib2 data, both values are
integers
> at 0,1,2,3,4. For observations, I can define a 'yes' event when
value>=1,
> and I want to plot ROC curve based on the forecast values at >=1,
>=2, >=3,
> =4.
>
> Thank you.
> Yali
>
>
------------------------------------------------
Subject: May you please help me to set up a config file to generate ROC curves?
From: David Fillmore
Time: Fri Jan 15 14:32:30 2021
On Fri Jan 15 13:47:43 2021, yali.mao at noaa.gov wrote:
> Hi,
> I come back to ask for help again. All related files are transferred
to
> your ftp server, ftp.rap.ucar.edu, under the folder:
> /incoming/irap/met_help/mao
> Script: test4help.sh (Start with it generate a stat file, but it may
need a
> bit modifications)
> config file: wafs_verf_g2g_config.icesev (Here I need your help)
> obs file: obsv.icesev.grib2
> fcst file: fcst.icesev.grib2
> Grib2 configuration file: grib2_icesev.txt (icesev is a very new
field
> name, grib2 lib hasn't been updated with it yet, so MET_GRIB_TABLES
> environment variable is defined in the test script)
> Spatial Masking files: *nc
User provided data and config file on
kiowa at /d1/projects/MET/MET_Help/mao_data_20210115
> I was able to generate a stat file by running the test script,
> test4help.sh. But the plotting looks weird, at least not like a ROC
curve.
> Please check more details of the plotting from the history
20210115_200405
> on https://metviewer.nws.noaa.gov/metviewer1.jsp
>
> So I must have maken mistakes when setting up
wafs_verf_g2g_config.icesev.
> Both observation and forecast data are grib2 data, both values are
integers
> at 0,1,2,3,4. For observations, I can define a 'yes' event when
value>=1,
> and I want to plot ROC curve based on the forecast values at >=1,
>=2, >=3,
> =4.
>
> Thank you.
> Yali
------------------------------------------------
Subject: May you please help me to set up a config file to generate ROC curves?
From: Yali Mao - NOAA Affiliate
Time: Fri Jan 15 20:43:38 2021
Hi David,
Sorry I thought the plot could be seen on
https://metviewer.nws.noaa.gov/metviewer1.jsp by checking the history
#20210115_200405.
Attached please find the plot.
Thank you.
Yali
On Fri, Jan 15, 2021 at 4:24 PM David Fillmore via RT
<met_help at ucar.edu>
wrote:
> Hi Yali -
> I'm watching met_help today and have retrieved your data and config
files
> for grid_stat testing.
> Are you able to attach any of the bad ROC curves to this email
chain?
> I will likely try to reproduce with the METplotpy roc_diagram.py
script.
> I have cc'ed John on this ticket so he can also have a look at your
config
> file.
> thanks,
> David
>
>
> On Fri, Jan 15, 2021 at 1:48 PM Yali Mao - NOAA Affiliate via RT <
> met_help at ucar.edu> wrote:
>
> >
> > Fri Jan 15 13:47:43 2021: Request 98261 was acted upon.
> > Transaction: Ticket created by yali.mao at noaa.gov
> > Queue: met_help
> > Subject: May you please help me to set up a config file to
generate
> > ROC curves?
> > Owner: Nobody
> > Requestors: yali.mao at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261 >
> >
> >
> > Hi,
> > I come back to ask for help again. All related files are
transferred to
> > your ftp server, ftp.rap.ucar.edu, under the folder:
> > /incoming/irap/met_help/mao
> > Script: test4help.sh (Start with it generate a stat file, but it
may
> need a
> > bit modifications)
> > config file: wafs_verf_g2g_config.icesev (Here I need your help)
> > obs file: obsv.icesev.grib2
> > fcst file: fcst.icesev.grib2
> > Grib2 configuration file: grib2_icesev.txt (icesev is a very new
field
> > name, grib2 lib hasn't been updated with it yet, so
MET_GRIB_TABLES
> > environment variable is defined in the test script)
> > Spatial Masking files: *nc
> >
> > I was able to generate a stat file by running the test script,
> > test4help.sh. But the plotting looks weird, at least not like a
ROC
> curve.
> > Please check more details of the plotting from the history
> 20210115_200405
> > on https://metviewer.nws.noaa.gov/metviewer1.jsp
> >
> > So I must have maken mistakes when setting up
> wafs_verf_g2g_config.icesev.
> > Both observation and forecast data are grib2 data, both values are
> integers
> > at 0,1,2,3,4. For observations, I can define a 'yes' event when
value>=1,
> > and I want to plot ROC curve based on the forecast values at >=1,
>=2,
> >=3,
> > =4.
> >
> > Thank you.
> > Yali
> >
> >
>
>
------------------------------------------------
Subject: May you please help me to set up a config file to generate ROC curves?
From: John Halley Gotway
Time: Thu Jan 21 17:12:48 2021
Yali,
I apologize for the long delay in answering this question! I just got
a
chance to take a look at your example. I'm definitely able to
replicate the
behavior you described. I suspect the problem is that the dots on that
plot
are being connected in the wrong order.
I'm going to reassign this ticket to Tatiana (cc'ed here) so she can
comment on my theory.
Tatiana, in the attached plot/xml, Yali is creating a ROC curve from
2x2
contingency tables.
https://metviewer.nws.noaa.gov/servlet
The forecast thresholds are defined as: cat_thresh = [ >=1, >=2, >=3,
>=4 ];
And the observation thresholds are defined as: cat_thresh = [
>=1, >=1, >=1, >=1 ];
This setup matches the instructions from the METviewer user's guide:
https://dtcenter.github.io/METviewer/develop/Users_Guide/rocplot.html
I think the order of connecting the dots would depend on the threshold
*type*, greater-than in one order and less-than in another order. But
I'm
not totally positive on that. I wonder if we should just add a
user-settable checkbox to reverse the connection order or something?
Can you please take a look when you get a chance?
Thanks,
John
On Fri, Jan 15, 2021 at 8:44 PM Yali Mao - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261 >
>
> Hi David,
> Sorry I thought the plot could be seen on
> https://metviewer.nws.noaa.gov/metviewer1.jsp by checking the
history
> #20210115_200405.
> Attached please find the plot.
> Thank you.
> Yali
>
> On Fri, Jan 15, 2021 at 4:24 PM David Fillmore via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Yali -
> > I'm watching met_help today and have retrieved your data and
config files
> > for grid_stat testing.
> > Are you able to attach any of the bad ROC curves to this email
chain?
> > I will likely try to reproduce with the METplotpy roc_diagram.py
script.
> > I have cc'ed John on this ticket so he can also have a look at
your
> config
> > file.
> > thanks,
> > David
> >
> >
> > On Fri, Jan 15, 2021 at 1:48 PM Yali Mao - NOAA Affiliate via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Fri Jan 15 13:47:43 2021: Request 98261 was acted upon.
> > > Transaction: Ticket created by yali.mao at noaa.gov
> > > Queue: met_help
> > > Subject: May you please help me to set up a config file to
> generate
> > > ROC curves?
> > > Owner: Nobody
> > > Requestors: yali.mao at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261
> >
> > >
> > >
> > > Hi,
> > > I come back to ask for help again. All related files are
transferred to
> > > your ftp server, ftp.rap.ucar.edu, under the folder:
> > > /incoming/irap/met_help/mao
> > > Script: test4help.sh (Start with it generate a stat file, but it
may
> > need a
> > > bit modifications)
> > > config file: wafs_verf_g2g_config.icesev (Here I need your help)
> > > obs file: obsv.icesev.grib2
> > > fcst file: fcst.icesev.grib2
> > > Grib2 configuration file: grib2_icesev.txt (icesev is a very new
field
> > > name, grib2 lib hasn't been updated with it yet, so
MET_GRIB_TABLES
> > > environment variable is defined in the test script)
> > > Spatial Masking files: *nc
> > >
> > > I was able to generate a stat file by running the test script,
> > > test4help.sh. But the plotting looks weird, at least not like a
ROC
> > curve.
> > > Please check more details of the plotting from the history
> > 20210115_200405
> > > on https://metviewer.nws.noaa.gov/metviewer1.jsp
> > >
> > > So I must have maken mistakes when setting up
> > wafs_verf_g2g_config.icesev.
> > > Both observation and forecast data are grib2 data, both values
are
> > integers
> > > at 0,1,2,3,4. For observations, I can define a 'yes' event when
> value>=1,
> > > and I want to plot ROC curve based on the forecast values at
>=1, >=2,
> > >=3,
> > > =4.
> > >
> > > Thank you.
> > > Yali
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: May you please help me to set up a config file to generate ROC curves?
From: Yali Mao - NOAA Affiliate
Time: Thu Jan 21 19:29:07 2021
John,
I set cat_thresh by myself and am not sure whether I set it up
correctly. I
am more concerned whether I create stat data correctly.
Should I change to cat_thresh = [ <=1, <=2, <=3, <=4 ] ?
The variable of the verification is icing severity, the larger number
the
more severe.
Thank you.
Yali
On Thu, Jan 21, 2021 at 7:14 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Yali,
>
> I apologize for the long delay in answering this question! I just
got a
> chance to take a look at your example. I'm definitely able to
replicate the
> behavior you described. I suspect the problem is that the dots on
that plot
> are being connected in the wrong order.
>
> I'm going to reassign this ticket to Tatiana (cc'ed here) so she can
> comment on my theory.
>
> Tatiana, in the attached plot/xml, Yali is creating a ROC curve from
2x2
> contingency tables.
> https://metviewer.nws.noaa.gov/servlet
>
> The forecast thresholds are defined as: cat_thresh = [ >=1, >=2,
>=3, >=4
> ];
> And the observation thresholds are defined as: cat_thresh = [
> >=1, >=1, >=1, >=1 ];
>
> This setup matches the instructions from the METviewer user's guide:
>
https://dtcenter.github.io/METviewer/develop/Users_Guide/rocplot.html
>
> I think the order of connecting the dots would depend on the
threshold
> *type*, greater-than in one order and less-than in another order.
But I'm
> not totally positive on that. I wonder if we should just add a
> user-settable checkbox to reverse the connection order or something?
>
> Can you please take a look when you get a chance?
>
> Thanks,
> John
>
>
>
> On Fri, Jan 15, 2021 at 8:44 PM Yali Mao - NOAA Affiliate via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261 >
> >
> > Hi David,
> > Sorry I thought the plot could be seen on
> > https://metviewer.nws.noaa.gov/metviewer1.jsp by checking the
history
> > #20210115_200405.
> > Attached please find the plot.
> > Thank you.
> > Yali
> >
> > On Fri, Jan 15, 2021 at 4:24 PM David Fillmore via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> > > Hi Yali -
> > > I'm watching met_help today and have retrieved your data and
config
> files
> > > for grid_stat testing.
> > > Are you able to attach any of the bad ROC curves to this email
chain?
> > > I will likely try to reproduce with the METplotpy roc_diagram.py
> script.
> > > I have cc'ed John on this ticket so he can also have a look at
your
> > config
> > > file.
> > > thanks,
> > > David
> > >
> > >
> > > On Fri, Jan 15, 2021 at 1:48 PM Yali Mao - NOAA Affiliate via RT
<
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Fri Jan 15 13:47:43 2021: Request 98261 was acted upon.
> > > > Transaction: Ticket created by yali.mao at noaa.gov
> > > > Queue: met_help
> > > > Subject: May you please help me to set up a config file
to
> > generate
> > > > ROC curves?
> > > > Owner: Nobody
> > > > Requestors: yali.mao at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261
> > >
> > > >
> > > >
> > > > Hi,
> > > > I come back to ask for help again. All related files are
transferred
> to
> > > > your ftp server, ftp.rap.ucar.edu, under the folder:
> > > > /incoming/irap/met_help/mao
> > > > Script: test4help.sh (Start with it generate a stat file, but
it may
> > > need a
> > > > bit modifications)
> > > > config file: wafs_verf_g2g_config.icesev (Here I need your
help)
> > > > obs file: obsv.icesev.grib2
> > > > fcst file: fcst.icesev.grib2
> > > > Grib2 configuration file: grib2_icesev.txt (icesev is a very
new
> field
> > > > name, grib2 lib hasn't been updated with it yet, so
MET_GRIB_TABLES
> > > > environment variable is defined in the test script)
> > > > Spatial Masking files: *nc
> > > >
> > > > I was able to generate a stat file by running the test script,
> > > > test4help.sh. But the plotting looks weird, at least not like
a ROC
> > > curve.
> > > > Please check more details of the plotting from the history
> > > 20210115_200405
> > > > on https://metviewer.nws.noaa.gov/metviewer1.jsp
> > > >
> > > > So I must have maken mistakes when setting up
> > > wafs_verf_g2g_config.icesev.
> > > > Both observation and forecast data are grib2 data, both values
are
> > > integers
> > > > at 0,1,2,3,4. For observations, I can define a 'yes' event
when
> > value>=1,
> > > > and I want to plot ROC curve based on the forecast values at
>=1,
> >=2,
> > > >=3,
> > > > =4.
> > > >
> > > > Thank you.
> > > > Yali
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: May you please help me to set up a config file to generate ROC curves?
From: John Halley Gotway
Time: Fri Jan 22 09:43:53 2021
Yali,
You could certainly try reversing the sign of the thresholds to see if
that
produces a better result in METviewer for this specific example. In
fact,
for testing, you could do it both ways in the same config file. Just
specify the same field twice. And since the "name" and "level" setting
remain constant, I just factored them out to a higher level of context
for
simplicity:
*name = "ICESEV";*
*level = [ "P800", "P700", "P600", "P500", "P400" ];*
*fcst = {*
* field = [ { cat_thresh = [ >=1, >=2, >=3, >=4 ]; }, { cat_thresh =
[
<=1, <=2, <=3, <=4 ]; } ];*
*}*
*obs = {*
* field = [ { cat_thresh = [ >=1, >=1, >=1, >=1 ]; }, { cat_thresh =
[
<=1, <=1, <=1, <=1 ]; } ];*
*}*
When testing the result in METviewer, you'll need to explicitly
specify the
"fcst_thresh" and "obs_thresh" values to be used in the "plot_fix"
section.
But regardless, I suspect we'll need some tweaks within METviewer to
make
the plotting of ROC curves from 2x2 contingency tables more robust.
Your
originally settings *should* have worked and produced a sensible ROC
curve.
I'm hoping Tatiana can find a nice solution here.
Thanks,
John
On Thu, Jan 21, 2021 at 7:29 PM Yali Mao - NOAA Affiliate via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261 >
>
> John,
> I set cat_thresh by myself and am not sure whether I set it up
correctly. I
> am more concerned whether I create stat data correctly.
> Should I change to cat_thresh = [ <=1, <=2, <=3, <=4 ] ?
>
> The variable of the verification is icing severity, the larger
number the
> more severe.
> Thank you.
> Yali
>
> On Thu, Jan 21, 2021 at 7:14 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Yali,
> >
> > I apologize for the long delay in answering this question! I just
got a
> > chance to take a look at your example. I'm definitely able to
replicate
> the
> > behavior you described. I suspect the problem is that the dots on
that
> plot
> > are being connected in the wrong order.
> >
> > I'm going to reassign this ticket to Tatiana (cc'ed here) so she
can
> > comment on my theory.
> >
> > Tatiana, in the attached plot/xml, Yali is creating a ROC curve
from 2x2
> > contingency tables.
> > https://metviewer.nws.noaa.gov/servlet
> >
> > The forecast thresholds are defined as: cat_thresh = [ >=1, >=2,
>=3, >=4
> > ];
> > And the observation thresholds are defined as: cat_thresh = [
> > >=1, >=1, >=1, >=1 ];
> >
> > This setup matches the instructions from the METviewer user's
guide:
> >
https://dtcenter.github.io/METviewer/develop/Users_Guide/rocplot.html
> >
> > I think the order of connecting the dots would depend on the
threshold
> > *type*, greater-than in one order and less-than in another order.
But I'm
> > not totally positive on that. I wonder if we should just add a
> > user-settable checkbox to reverse the connection order or
something?
> >
> > Can you please take a look when you get a chance?
> >
> > Thanks,
> > John
> >
> >
> >
> > On Fri, Jan 15, 2021 at 8:44 PM Yali Mao - NOAA Affiliate via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261 >
> > >
> > > Hi David,
> > > Sorry I thought the plot could be seen on
> > > https://metviewer.nws.noaa.gov/metviewer1.jsp by checking the
history
> > > #20210115_200405.
> > > Attached please find the plot.
> > > Thank you.
> > > Yali
> > >
> > > On Fri, Jan 15, 2021 at 4:24 PM David Fillmore via RT <
> met_help at ucar.edu
> > >
> > > wrote:
> > >
> > > > Hi Yali -
> > > > I'm watching met_help today and have retrieved your data and
config
> > files
> > > > for grid_stat testing.
> > > > Are you able to attach any of the bad ROC curves to this email
chain?
> > > > I will likely try to reproduce with the METplotpy
roc_diagram.py
> > script.
> > > > I have cc'ed John on this ticket so he can also have a look at
your
> > > config
> > > > file.
> > > > thanks,
> > > > David
> > > >
> > > >
> > > > On Fri, Jan 15, 2021 at 1:48 PM Yali Mao - NOAA Affiliate via
RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Fri Jan 15 13:47:43 2021: Request 98261 was acted upon.
> > > > > Transaction: Ticket created by yali.mao at noaa.gov
> > > > > Queue: met_help
> > > > > Subject: May you please help me to set up a config file
to
> > > generate
> > > > > ROC curves?
> > > > > Owner: Nobody
> > > > > Requestors: yali.mao at noaa.gov
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261
> > > >
> > > > >
> > > > >
> > > > > Hi,
> > > > > I come back to ask for help again. All related files are
> transferred
> > to
> > > > > your ftp server, ftp.rap.ucar.edu, under the folder:
> > > > > /incoming/irap/met_help/mao
> > > > > Script: test4help.sh (Start with it generate a stat file,
but it
> may
> > > > need a
> > > > > bit modifications)
> > > > > config file: wafs_verf_g2g_config.icesev (Here I need your
help)
> > > > > obs file: obsv.icesev.grib2
> > > > > fcst file: fcst.icesev.grib2
> > > > > Grib2 configuration file: grib2_icesev.txt (icesev is a very
new
> > field
> > > > > name, grib2 lib hasn't been updated with it yet, so
MET_GRIB_TABLES
> > > > > environment variable is defined in the test script)
> > > > > Spatial Masking files: *nc
> > > > >
> > > > > I was able to generate a stat file by running the test
script,
> > > > > test4help.sh. But the plotting looks weird, at least not
like a ROC
> > > > curve.
> > > > > Please check more details of the plotting from the history
> > > > 20210115_200405
> > > > > on https://metviewer.nws.noaa.gov/metviewer1.jsp
> > > > >
> > > > > So I must have maken mistakes when setting up
> > > > wafs_verf_g2g_config.icesev.
> > > > > Both observation and forecast data are grib2 data, both
values are
> > > > integers
> > > > > at 0,1,2,3,4. For observations, I can define a 'yes' event
when
> > > value>=1,
> > > > > and I want to plot ROC curve based on the forecast values at
>=1,
> > >=2,
> > > > >=3,
> > > > > =4.
> > > > >
> > > > > Thank you.
> > > > > Yali
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: May you please help me to set up a config file to generate ROC curves?
From: Yali Mao - NOAA Affiliate
Time: Fri Jan 22 13:56:32 2021
Hi John,
I will try more and let you know my next sample result.
Thank you.
Yali
On Fri, Jan 22, 2021 at 11:44 AM John Halley Gotway via RT <
met_help at ucar.edu> wrote:
> Yali,
>
> You could certainly try reversing the sign of the thresholds to see
if that
> produces a better result in METviewer for this specific example. In
fact,
> for testing, you could do it both ways in the same config file. Just
> specify the same field twice. And since the "name" and "level"
setting
> remain constant, I just factored them out to a higher level of
context for
> simplicity:
>
>
> *name = "ICESEV";*
>
> *level = [ "P800", "P700", "P600", "P500", "P400" ];*
> *fcst = {*
> * field = [ { cat_thresh = [ >=1, >=2, >=3, >=4 ]; }, { cat_thresh
= [
> <=1, <=2, <=3, <=4 ]; } ];*
> *}*
>
> *obs = {*
> * field = [ { cat_thresh = [ >=1, >=1, >=1, >=1 ]; }, { cat_thresh
= [
> <=1, <=1, <=1, <=1 ]; } ];*
> *}*
>
> When testing the result in METviewer, you'll need to explicitly
specify the
> "fcst_thresh" and "obs_thresh" values to be used in the "plot_fix"
section.
>
> But regardless, I suspect we'll need some tweaks within METviewer to
make
> the plotting of ROC curves from 2x2 contingency tables more robust.
Your
> originally settings *should* have worked and produced a sensible ROC
curve.
> I'm hoping Tatiana can find a nice solution here.
>
> Thanks,
> John
>
> On Thu, Jan 21, 2021 at 7:29 PM Yali Mao - NOAA Affiliate via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261 >
> >
> > John,
> > I set cat_thresh by myself and am not sure whether I set it up
> correctly. I
> > am more concerned whether I create stat data correctly.
> > Should I change to cat_thresh = [ <=1, <=2, <=3, <=4 ] ?
> >
> > The variable of the verification is icing severity, the larger
number the
> > more severe.
> > Thank you.
> > Yali
> >
> > On Thu, Jan 21, 2021 at 7:14 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Yali,
> > >
> > > I apologize for the long delay in answering this question! I
just got a
> > > chance to take a look at your example. I'm definitely able to
replicate
> > the
> > > behavior you described. I suspect the problem is that the dots
on that
> > plot
> > > are being connected in the wrong order.
> > >
> > > I'm going to reassign this ticket to Tatiana (cc'ed here) so she
can
> > > comment on my theory.
> > >
> > > Tatiana, in the attached plot/xml, Yali is creating a ROC curve
from
> 2x2
> > > contingency tables.
> > > https://metviewer.nws.noaa.gov/servlet
> > >
> > > The forecast thresholds are defined as: cat_thresh = [ >=1, >=2,
>=3,
> >=4
> > > ];
> > > And the observation thresholds are defined as: cat_thresh = [
> > > >=1, >=1, >=1, >=1 ];
> > >
> > > This setup matches the instructions from the METviewer user's
guide:
> > >
https://dtcenter.github.io/METviewer/develop/Users_Guide/rocplot.html
> > >
> > > I think the order of connecting the dots would depend on the
threshold
> > > *type*, greater-than in one order and less-than in another
order. But
> I'm
> > > not totally positive on that. I wonder if we should just add a
> > > user-settable checkbox to reverse the connection order or
something?
> > >
> > > Can you please take a look when you get a chance?
> > >
> > > Thanks,
> > > John
> > >
> > >
> > >
> > > On Fri, Jan 15, 2021 at 8:44 PM Yali Mao - NOAA Affiliate via RT
<
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261
>
> > > >
> > > > Hi David,
> > > > Sorry I thought the plot could be seen on
> > > > https://metviewer.nws.noaa.gov/metviewer1.jsp by checking the
> history
> > > > #20210115_200405.
> > > > Attached please find the plot.
> > > > Thank you.
> > > > Yali
> > > >
> > > > On Fri, Jan 15, 2021 at 4:24 PM David Fillmore via RT <
> > met_help at ucar.edu
> > > >
> > > > wrote:
> > > >
> > > > > Hi Yali -
> > > > > I'm watching met_help today and have retrieved your data and
config
> > > files
> > > > > for grid_stat testing.
> > > > > Are you able to attach any of the bad ROC curves to this
email
> chain?
> > > > > I will likely try to reproduce with the METplotpy
roc_diagram.py
> > > script.
> > > > > I have cc'ed John on this ticket so he can also have a look
at your
> > > > config
> > > > > file.
> > > > > thanks,
> > > > > David
> > > > >
> > > > >
> > > > > On Fri, Jan 15, 2021 at 1:48 PM Yali Mao - NOAA Affiliate
via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Fri Jan 15 13:47:43 2021: Request 98261 was acted upon.
> > > > > > Transaction: Ticket created by yali.mao at noaa.gov
> > > > > > Queue: met_help
> > > > > > Subject: May you please help me to set up a config
file to
> > > > generate
> > > > > > ROC curves?
> > > > > > Owner: Nobody
> > > > > > Requestors: yali.mao at noaa.gov
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261
> > > > >
> > > > > >
> > > > > >
> > > > > > Hi,
> > > > > > I come back to ask for help again. All related files are
> > transferred
> > > to
> > > > > > your ftp server, ftp.rap.ucar.edu, under the folder:
> > > > > > /incoming/irap/met_help/mao
> > > > > > Script: test4help.sh (Start with it generate a stat file,
but it
> > may
> > > > > need a
> > > > > > bit modifications)
> > > > > > config file: wafs_verf_g2g_config.icesev (Here I need your
help)
> > > > > > obs file: obsv.icesev.grib2
> > > > > > fcst file: fcst.icesev.grib2
> > > > > > Grib2 configuration file: grib2_icesev.txt (icesev is a
very new
> > > field
> > > > > > name, grib2 lib hasn't been updated with it yet, so
> MET_GRIB_TABLES
> > > > > > environment variable is defined in the test script)
> > > > > > Spatial Masking files: *nc
> > > > > >
> > > > > > I was able to generate a stat file by running the test
script,
> > > > > > test4help.sh. But the plotting looks weird, at least not
like a
> ROC
> > > > > curve.
> > > > > > Please check more details of the plotting from the history
> > > > > 20210115_200405
> > > > > > on https://metviewer.nws.noaa.gov/metviewer1.jsp
> > > > > >
> > > > > > So I must have maken mistakes when setting up
> > > > > wafs_verf_g2g_config.icesev.
> > > > > > Both observation and forecast data are grib2 data, both
values
> are
> > > > > integers
> > > > > > at 0,1,2,3,4. For observations, I can define a 'yes' event
when
> > > > value>=1,
> > > > > > and I want to plot ROC curve based on the forecast values
at >=1,
> > > >=2,
> > > > > >=3,
> > > > > > =4.
> > > > > >
> > > > > > Thank you.
> > > > > > Yali
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: May you please help me to set up a config file to generate ROC curves?
From: Tatiana Burek
Time: Mon Jan 25 09:14:18 2021
Yali,
I created a GitHub issue and will will let you now when I make some
progress:
https://github.com/dtcenter/METviewer/issues/246
Tatiana
On Fri Jan 22 13:56:32 2021, yali.mao at noaa.gov wrote:
> Hi John,
> I will try more and let you know my next sample result.
> Thank you.
> Yali
>
> On Fri, Jan 22, 2021 at 11:44 AM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
> > Yali,
> >
> > You could certainly try reversing the sign of the thresholds to
see
> > if that
> > produces a better result in METviewer for this specific example.
In
> > fact,
> > for testing, you could do it both ways in the same config file.
Just
> > specify the same field twice. And since the "name" and "level"
> > setting
> > remain constant, I just factored them out to a higher level of
> > context for
> > simplicity:
> >
> >
> > *name = "ICESEV";*
> >
> > *level = [ "P800", "P700", "P600", "P500", "P400" ];*
> > *fcst = {*
> > * field = [ { cat_thresh = [ >=1, >=2, >=3, >=4 ]; }, {
cat_thresh
> > = [
> > <=1, <=2, <=3, <=4 ]; } ];*
> > *}*
> >
> > *obs = {*
> > * field = [ { cat_thresh = [ >=1, >=1, >=1, >=1 ]; }, {
cat_thresh
> > = [
> > <=1, <=1, <=1, <=1 ]; } ];*
> > *}*
> >
> > When testing the result in METviewer, you'll need to explicitly
> > specify the
> > "fcst_thresh" and "obs_thresh" values to be used in the "plot_fix"
> > section.
> >
> > But regardless, I suspect we'll need some tweaks within METviewer
to
> > make
> > the plotting of ROC curves from 2x2 contingency tables more
robust.
> > Your
> > originally settings *should* have worked and produced a sensible
ROC
> > curve.
> > I'm hoping Tatiana can find a nice solution here.
> >
> > Thanks,
> > John
> >
> > On Thu, Jan 21, 2021 at 7:29 PM Yali Mao - NOAA Affiliate via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261 >
> > >
> > > John,
> > > I set cat_thresh by myself and am not sure whether I set it up
> > correctly. I
> > > am more concerned whether I create stat data correctly.
> > > Should I change to cat_thresh = [ <=1, <=2, <=3, <=4 ] ?
> > >
> > > The variable of the verification is icing severity, the larger
> > > number the
> > > more severe.
> > > Thank you.
> > > Yali
> > >
> > > On Thu, Jan 21, 2021 at 7:14 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Yali,
> > > >
> > > > I apologize for the long delay in answering this question! I
just
> > > > got a
> > > > chance to take a look at your example. I'm definitely able to
> > > > replicate
> > > the
> > > > behavior you described. I suspect the problem is that the dots
on
> > > > that
> > > plot
> > > > are being connected in the wrong order.
> > > >
> > > > I'm going to reassign this ticket to Tatiana (cc'ed here) so
she
> > > > can
> > > > comment on my theory.
> > > >
> > > > Tatiana, in the attached plot/xml, Yali is creating a ROC
curve
> > > > from
> > 2x2
> > > > contingency tables.
> > > > https://metviewer.nws.noaa.gov/servlet
> > > >
> > > > The forecast thresholds are defined as: cat_thresh = [ >=1,
>=2,
> > > > >=3,
> > > =4
> > > > ];
> > > > And the observation thresholds are defined as: cat_thresh = [
> > > > > =1, >=1, >=1, >=1 ];
> > > >
> > > > This setup matches the instructions from the METviewer user's
> > > > guide:
> > > >
https://dtcenter.github.io/METviewer/develop/Users_Guide/rocplot.html
> > > >
> > > > I think the order of connecting the dots would depend on the
> > > > threshold
> > > > *type*, greater-than in one order and less-than in another
order.
> > > > But
> > I'm
> > > > not totally positive on that. I wonder if we should just add a
> > > > user-settable checkbox to reverse the connection order or
> > > > something?
> > > >
> > > > Can you please take a look when you get a chance?
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > >
> > > >
> > > > On Fri, Jan 15, 2021 at 8:44 PM Yali Mao - NOAA Affiliate via
RT
> > > > <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261 >
> > > > >
> > > > > Hi David,
> > > > > Sorry I thought the plot could be seen on
> > > > > https://metviewer.nws.noaa.gov/metviewer1.jsp by checking
the
> > history
> > > > > #20210115_200405.
> > > > > Attached please find the plot.
> > > > > Thank you.
> > > > > Yali
> > > > >
> > > > > On Fri, Jan 15, 2021 at 4:24 PM David Fillmore via RT <
> > > met_help at ucar.edu
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Yali -
> > > > > > I'm watching met_help today and have retrieved your data
and
> > > > > > config
> > > > files
> > > > > > for grid_stat testing.
> > > > > > Are you able to attach any of the bad ROC curves to this
> > > > > > email
> > chain?
> > > > > > I will likely try to reproduce with the METplotpy
> > > > > > roc_diagram.py
> > > > script.
> > > > > > I have cc'ed John on this ticket so he can also have a
look
> > > > > > at your
> > > > > config
> > > > > > file.
> > > > > > thanks,
> > > > > > David
> > > > > >
> > > > > >
> > > > > > On Fri, Jan 15, 2021 at 1:48 PM Yali Mao - NOAA Affiliate
via
> > > > > > RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > Fri Jan 15 13:47:43 2021: Request 98261 was acted upon.
> > > > > > > Transaction: Ticket created by yali.mao at noaa.gov
> > > > > > > Queue: met_help
> > > > > > > Subject: May you please help me to set up a config
> > > > > > > file to
> > > > > generate
> > > > > > > ROC curves?
> > > > > > > Owner: Nobody
> > > > > > > Requestors: yali.mao at noaa.gov
> > > > > > > Status: new
> > > > > > > Ticket <URL:
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261
> > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hi,
> > > > > > > I come back to ask for help again. All related files are
> > > transferred
> > > > to
> > > > > > > your ftp server, ftp.rap.ucar.edu, under the folder:
> > > > > > > /incoming/irap/met_help/mao
> > > > > > > Script: test4help.sh (Start with it generate a stat
file,
> > > > > > > but it
> > > may
> > > > > > need a
> > > > > > > bit modifications)
> > > > > > > config file: wafs_verf_g2g_config.icesev (Here I need
your
> > > > > > > help)
> > > > > > > obs file: obsv.icesev.grib2
> > > > > > > fcst file: fcst.icesev.grib2
> > > > > > > Grib2 configuration file: grib2_icesev.txt (icesev is a
> > > > > > > very new
> > > > field
> > > > > > > name, grib2 lib hasn't been updated with it yet, so
> > MET_GRIB_TABLES
> > > > > > > environment variable is defined in the test script)
> > > > > > > Spatial Masking files: *nc
> > > > > > >
> > > > > > > I was able to generate a stat file by running the test
> > > > > > > script,
> > > > > > > test4help.sh. But the plotting looks weird, at least not
> > > > > > > like a
> > ROC
> > > > > > curve.
> > > > > > > Please check more details of the plotting from the
history
> > > > > > 20210115_200405
> > > > > > > on https://metviewer.nws.noaa.gov/metviewer1.jsp
> > > > > > >
> > > > > > > So I must have maken mistakes when setting up
> > > > > > wafs_verf_g2g_config.icesev.
> > > > > > > Both observation and forecast data are grib2 data, both
> > > > > > > values
> > are
> > > > > > integers
> > > > > > > at 0,1,2,3,4. For observations, I can define a 'yes'
event
> > > > > > > when
> > > > > value> =1,
> > > > > > > and I want to plot ROC curve based on the forecast
values
> > > > > > > at >=1,
> > > > > =2,
> > > > > > > =3,
> > > > > > > =4.
> > > > > > >
> > > > > > > Thank you.
> > > > > > > Yali
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
------------------------------------------------
Subject: May you please help me to set up a config file to generate ROC curves?
From: Yali Mao - NOAA Affiliate
Time: Mon Jan 25 11:50:21 2021
Taiana,
Thank you. Wish it could be solved soon.
Yali
On Mon, Jan 25, 2021 at 11:14 AM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> Yali,
>
> I created a GitHub issue and will will let you now when I make some
> progress:
> https://github.com/dtcenter/METviewer/issues/246
>
> Tatiana
>
>
> On Fri Jan 22 13:56:32 2021, yali.mao at noaa.gov wrote:
> > Hi John,
> > I will try more and let you know my next sample result.
> > Thank you.
> > Yali
> >
> > On Fri, Jan 22, 2021 at 11:44 AM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> > > Yali,
> > >
> > > You could certainly try reversing the sign of the thresholds to
see
> > > if that
> > > produces a better result in METviewer for this specific example.
In
> > > fact,
> > > for testing, you could do it both ways in the same config file.
Just
> > > specify the same field twice. And since the "name" and "level"
> > > setting
> > > remain constant, I just factored them out to a higher level of
> > > context for
> > > simplicity:
> > >
> > >
> > > *name = "ICESEV";*
> > >
> > > *level = [ "P800", "P700", "P600", "P500", "P400" ];*
> > > *fcst = {*
> > > * field = [ { cat_thresh = [ >=1, >=2, >=3, >=4 ]; }, {
cat_thresh
> > > = [
> > > <=1, <=2, <=3, <=4 ]; } ];*
> > > *}*
> > >
> > > *obs = {*
> > > * field = [ { cat_thresh = [ >=1, >=1, >=1, >=1 ]; }, {
cat_thresh
> > > = [
> > > <=1, <=1, <=1, <=1 ]; } ];*
> > > *}*
> > >
> > > When testing the result in METviewer, you'll need to explicitly
> > > specify the
> > > "fcst_thresh" and "obs_thresh" values to be used in the
"plot_fix"
> > > section.
> > >
> > > But regardless, I suspect we'll need some tweaks within
METviewer to
> > > make
> > > the plotting of ROC curves from 2x2 contingency tables more
robust.
> > > Your
> > > originally settings *should* have worked and produced a sensible
ROC
> > > curve.
> > > I'm hoping Tatiana can find a nice solution here.
> > >
> > > Thanks,
> > > John
> > >
> > > On Thu, Jan 21, 2021 at 7:29 PM Yali Mao - NOAA Affiliate via RT
<
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261
>
> > > >
> > > > John,
> > > > I set cat_thresh by myself and am not sure whether I set it up
> > > correctly. I
> > > > am more concerned whether I create stat data correctly.
> > > > Should I change to cat_thresh = [ <=1, <=2, <=3, <=4 ] ?
> > > >
> > > > The variable of the verification is icing severity, the larger
> > > > number the
> > > > more severe.
> > > > Thank you.
> > > > Yali
> > > >
> > > > On Thu, Jan 21, 2021 at 7:14 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Yali,
> > > > >
> > > > > I apologize for the long delay in answering this question! I
just
> > > > > got a
> > > > > chance to take a look at your example. I'm definitely able
to
> > > > > replicate
> > > > the
> > > > > behavior you described. I suspect the problem is that the
dots on
> > > > > that
> > > > plot
> > > > > are being connected in the wrong order.
> > > > >
> > > > > I'm going to reassign this ticket to Tatiana (cc'ed here) so
she
> > > > > can
> > > > > comment on my theory.
> > > > >
> > > > > Tatiana, in the attached plot/xml, Yali is creating a ROC
curve
> > > > > from
> > > 2x2
> > > > > contingency tables.
> > > > > https://metviewer.nws.noaa.gov/servlet
> > > > >
> > > > > The forecast thresholds are defined as: cat_thresh = [ >=1,
>=2,
> > > > > >=3,
> > > > =4
> > > > > ];
> > > > > And the observation thresholds are defined as: cat_thresh =
[
> > > > > > =1, >=1, >=1, >=1 ];
> > > > >
> > > > > This setup matches the instructions from the METviewer
user's
> > > > > guide:
> > > > >
>
https://dtcenter.github.io/METviewer/develop/Users_Guide/rocplot.html
> > > > >
> > > > > I think the order of connecting the dots would depend on the
> > > > > threshold
> > > > > *type*, greater-than in one order and less-than in another
order.
> > > > > But
> > > I'm
> > > > > not totally positive on that. I wonder if we should just add
a
> > > > > user-settable checkbox to reverse the connection order or
> > > > > something?
> > > > >
> > > > > Can you please take a look when you get a chance?
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Jan 15, 2021 at 8:44 PM Yali Mao - NOAA Affiliate
via RT
> > > > > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261 >
> > > > > >
> > > > > > Hi David,
> > > > > > Sorry I thought the plot could be seen on
> > > > > > https://metviewer.nws.noaa.gov/metviewer1.jsp by checking
the
> > > history
> > > > > > #20210115_200405.
> > > > > > Attached please find the plot.
> > > > > > Thank you.
> > > > > > Yali
> > > > > >
> > > > > > On Fri, Jan 15, 2021 at 4:24 PM David Fillmore via RT <
> > > > met_help at ucar.edu
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Yali -
> > > > > > > I'm watching met_help today and have retrieved your data
and
> > > > > > > config
> > > > > files
> > > > > > > for grid_stat testing.
> > > > > > > Are you able to attach any of the bad ROC curves to this
> > > > > > > email
> > > chain?
> > > > > > > I will likely try to reproduce with the METplotpy
> > > > > > > roc_diagram.py
> > > > > script.
> > > > > > > I have cc'ed John on this ticket so he can also have a
look
> > > > > > > at your
> > > > > > config
> > > > > > > file.
> > > > > > > thanks,
> > > > > > > David
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jan 15, 2021 at 1:48 PM Yali Mao - NOAA
Affiliate via
> > > > > > > RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > Fri Jan 15 13:47:43 2021: Request 98261 was acted
upon.
> > > > > > > > Transaction: Ticket created by yali.mao at noaa.gov
> > > > > > > > Queue: met_help
> > > > > > > > Subject: May you please help me to set up a
config
> > > > > > > > file to
> > > > > > generate
> > > > > > > > ROC curves?
> > > > > > > > Owner: Nobody
> > > > > > > > Requestors: yali.mao at noaa.gov
> > > > > > > > Status: new
> > > > > > > > Ticket <URL:
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > > I come back to ask for help again. All related files
are
> > > > transferred
> > > > > to
> > > > > > > > your ftp server, ftp.rap.ucar.edu, under the folder:
> > > > > > > > /incoming/irap/met_help/mao
> > > > > > > > Script: test4help.sh (Start with it generate a stat
file,
> > > > > > > > but it
> > > > may
> > > > > > > need a
> > > > > > > > bit modifications)
> > > > > > > > config file: wafs_verf_g2g_config.icesev (Here I need
your
> > > > > > > > help)
> > > > > > > > obs file: obsv.icesev.grib2
> > > > > > > > fcst file: fcst.icesev.grib2
> > > > > > > > Grib2 configuration file: grib2_icesev.txt (icesev is
a
> > > > > > > > very new
> > > > > field
> > > > > > > > name, grib2 lib hasn't been updated with it yet, so
> > > MET_GRIB_TABLES
> > > > > > > > environment variable is defined in the test script)
> > > > > > > > Spatial Masking files: *nc
> > > > > > > >
> > > > > > > > I was able to generate a stat file by running the test
> > > > > > > > script,
> > > > > > > > test4help.sh. But the plotting looks weird, at least
not
> > > > > > > > like a
> > > ROC
> > > > > > > curve.
> > > > > > > > Please check more details of the plotting from the
history
> > > > > > > 20210115_200405
> > > > > > > > on https://metviewer.nws.noaa.gov/metviewer1.jsp
> > > > > > > >
> > > > > > > > So I must have maken mistakes when setting up
> > > > > > > wafs_verf_g2g_config.icesev.
> > > > > > > > Both observation and forecast data are grib2 data,
both
> > > > > > > > values
> > > are
> > > > > > > integers
> > > > > > > > at 0,1,2,3,4. For observations, I can define a 'yes'
event
> > > > > > > > when
> > > > > > value> =1,
> > > > > > > > and I want to plot ROC curve based on the forecast
values
> > > > > > > > at >=1,
> > > > > > =2,
> > > > > > > > =3,
> > > > > > > > =4.
> > > > > > > >
> > > > > > > > Thank you.
> > > > > > > > Yali
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
>
>
>
>
------------------------------------------------
Subject: May you please help me to set up a config file to generate ROC curves?
From: Tatiana Burek
Time: Tue Jan 26 11:53:33 2021
Yali
I added a checkbox and a parameter to XML file that would control the
connection order.
This functionality will come with METviewer4.0-beta3 and will be
available on the dev instance of METviewer. We are planning to release
it in a few days.
There is the link to the GitHub issue:
https://github.com/dtcenter/METviewer/issues/246
Tatiana
On Mon Jan 25 11:50:21 2021, yali.mao at noaa.gov wrote:
> Taiana,
> Thank you. Wish it could be solved soon.
> Yali
>
> On Mon, Jan 25, 2021 at 11:14 AM Tatiana Burek via RT
<met_help at ucar.edu>
> wrote:
>
> > Yali,
> >
> > I created a GitHub issue and will will let you now when I make
some
> > progress:
> > https://github.com/dtcenter/METviewer/issues/246
> >
> > Tatiana
> >
> >
> > On Fri Jan 22 13:56:32 2021, yali.mao at noaa.gov wrote:
> > > Hi John,
> > > I will try more and let you know my next sample result.
> > > Thank you.
> > > Yali
> > >
> > > On Fri, Jan 22, 2021 at 11:44 AM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > > Yali,
> > > >
> > > > You could certainly try reversing the sign of the thresholds
to see
> > > > if that
> > > > produces a better result in METviewer for this specific
example. In
> > > > fact,
> > > > for testing, you could do it both ways in the same config
file. Just
> > > > specify the same field twice. And since the "name" and "level"
> > > > setting
> > > > remain constant, I just factored them out to a higher level of
> > > > context for
> > > > simplicity:
> > > >
> > > >
> > > > *name = "ICESEV";*
> > > >
> > > > *level = [ "P800", "P700", "P600", "P500", "P400" ];*
> > > > *fcst = {*
> > > > * field = [ { cat_thresh = [ >=1, >=2, >=3, >=4 ]; }, {
cat_thresh
> > > > = [
> > > > <=1, <=2, <=3, <=4 ]; } ];*
> > > > *}*
> > > >
> > > > *obs = {*
> > > > * field = [ { cat_thresh = [ >=1, >=1, >=1, >=1 ]; }, {
cat_thresh
> > > > = [
> > > > <=1, <=1, <=1, <=1 ]; } ];*
> > > > *}*
> > > >
> > > > When testing the result in METviewer, you'll need to
explicitly
> > > > specify the
> > > > "fcst_thresh" and "obs_thresh" values to be used in the
"plot_fix"
> > > > section.
> > > >
> > > > But regardless, I suspect we'll need some tweaks within
METviewer to
> > > > make
> > > > the plotting of ROC curves from 2x2 contingency tables more
robust.
> > > > Your
> > > > originally settings *should* have worked and produced a
sensible ROC
> > > > curve.
> > > > I'm hoping Tatiana can find a nice solution here.
> > > >
> > > > Thanks,
> > > > John
> > > >
> > > > On Thu, Jan 21, 2021 at 7:29 PM Yali Mao - NOAA Affiliate via
RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261 >
> > > > >
> > > > > John,
> > > > > I set cat_thresh by myself and am not sure whether I set it
up
> > > > correctly. I
> > > > > am more concerned whether I create stat data correctly.
> > > > > Should I change to cat_thresh = [ <=1, <=2, <=3, <=4 ] ?
> > > > >
> > > > > The variable of the verification is icing severity, the
larger
> > > > > number the
> > > > > more severe.
> > > > > Thank you.
> > > > > Yali
> > > > >
> > > > > On Thu, Jan 21, 2021 at 7:14 PM John Halley Gotway via RT <
> > > > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > Yali,
> > > > > >
> > > > > > I apologize for the long delay in answering this question!
I just
> > > > > > got a
> > > > > > chance to take a look at your example. I'm definitely able
to
> > > > > > replicate
> > > > > the
> > > > > > behavior you described. I suspect the problem is that the
dots on
> > > > > > that
> > > > > plot
> > > > > > are being connected in the wrong order.
> > > > > >
> > > > > > I'm going to reassign this ticket to Tatiana (cc'ed here)
so she
> > > > > > can
> > > > > > comment on my theory.
> > > > > >
> > > > > > Tatiana, in the attached plot/xml, Yali is creating a ROC
curve
> > > > > > from
> > > > 2x2
> > > > > > contingency tables.
> > > > > > https://metviewer.nws.noaa.gov/servlet
> > > > > >
> > > > > > The forecast thresholds are defined as: cat_thresh = [
>=1, >=2,
> > > > > > >=3,
> > > > > =4
> > > > > > ];
> > > > > > And the observation thresholds are defined as: cat_thresh
= [
> > > > > > > =1, >=1, >=1, >=1 ];
> > > > > >
> > > > > > This setup matches the instructions from the METviewer
user's
> > > > > > guide:
> > > > > >
> >
https://dtcenter.github.io/METviewer/develop/Users_Guide/rocplot.html
> > > > > >
> > > > > > I think the order of connecting the dots would depend on
the
> > > > > > threshold
> > > > > > *type*, greater-than in one order and less-than in another
order.
> > > > > > But
> > > > I'm
> > > > > > not totally positive on that. I wonder if we should just
add a
> > > > > > user-settable checkbox to reverse the connection order or
> > > > > > something?
> > > > > >
> > > > > > Can you please take a look when you get a chance?
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Jan 15, 2021 at 8:44 PM Yali Mao - NOAA Affiliate
via RT
> > > > > > <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261 >
> > > > > > >
> > > > > > > Hi David,
> > > > > > > Sorry I thought the plot could be seen on
> > > > > > > https://metviewer.nws.noaa.gov/metviewer1.jsp by
checking the
> > > > history
> > > > > > > #20210115_200405.
> > > > > > > Attached please find the plot.
> > > > > > > Thank you.
> > > > > > > Yali
> > > > > > >
> > > > > > > On Fri, Jan 15, 2021 at 4:24 PM David Fillmore via RT <
> > > > > met_help at ucar.edu
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Yali -
> > > > > > > > I'm watching met_help today and have retrieved your
data and
> > > > > > > > config
> > > > > > files
> > > > > > > > for grid_stat testing.
> > > > > > > > Are you able to attach any of the bad ROC curves to
this
> > > > > > > > email
> > > > chain?
> > > > > > > > I will likely try to reproduce with the METplotpy
> > > > > > > > roc_diagram.py
> > > > > > script.
> > > > > > > > I have cc'ed John on this ticket so he can also have a
look
> > > > > > > > at your
> > > > > > > config
> > > > > > > > file.
> > > > > > > > thanks,
> > > > > > > > David
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jan 15, 2021 at 1:48 PM Yali Mao - NOAA
Affiliate via
> > > > > > > > RT <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Fri Jan 15 13:47:43 2021: Request 98261 was acted
upon.
> > > > > > > > > Transaction: Ticket created by yali.mao at noaa.gov
> > > > > > > > > Queue: met_help
> > > > > > > > > Subject: May you please help me to set up a
config
> > > > > > > > > file to
> > > > > > > generate
> > > > > > > > > ROC curves?
> > > > > > > > > Owner: Nobody
> > > > > > > > > Requestors: yali.mao at noaa.gov
> > > > > > > > > Status: new
> > > > > > > > > Ticket <URL:
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261
> > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > > I come back to ask for help again. All related files
are
> > > > > transferred
> > > > > > to
> > > > > > > > > your ftp server, ftp.rap.ucar.edu, under the folder:
> > > > > > > > > /incoming/irap/met_help/mao
> > > > > > > > > Script: test4help.sh (Start with it generate a stat
file,
> > > > > > > > > but it
> > > > > may
> > > > > > > > need a
> > > > > > > > > bit modifications)
> > > > > > > > > config file: wafs_verf_g2g_config.icesev (Here I
need your
> > > > > > > > > help)
> > > > > > > > > obs file: obsv.icesev.grib2
> > > > > > > > > fcst file: fcst.icesev.grib2
> > > > > > > > > Grib2 configuration file: grib2_icesev.txt (icesev
is a
> > > > > > > > > very new
> > > > > > field
> > > > > > > > > name, grib2 lib hasn't been updated with it yet, so
> > > > MET_GRIB_TABLES
> > > > > > > > > environment variable is defined in the test script)
> > > > > > > > > Spatial Masking files: *nc
> > > > > > > > >
> > > > > > > > > I was able to generate a stat file by running the
test
> > > > > > > > > script,
> > > > > > > > > test4help.sh. But the plotting looks weird, at least
not
> > > > > > > > > like a
> > > > ROC
> > > > > > > > curve.
> > > > > > > > > Please check more details of the plotting from the
history
> > > > > > > > 20210115_200405
> > > > > > > > > on https://metviewer.nws.noaa.gov/metviewer1.jsp
> > > > > > > > >
> > > > > > > > > So I must have maken mistakes when setting up
> > > > > > > > wafs_verf_g2g_config.icesev.
> > > > > > > > > Both observation and forecast data are grib2 data,
both
> > > > > > > > > values
> > > > are
> > > > > > > > integers
> > > > > > > > > at 0,1,2,3,4. For observations, I can define a 'yes'
event
> > > > > > > > > when
> > > > > > > value> =1,
> > > > > > > > > and I want to plot ROC curve based on the forecast
values
> > > > > > > > > at >=1,
> > > > > > > =2,
> > > > > > > > > =3,
> > > > > > > > > =4.
> > > > > > > > >
> > > > > > > > > Thank you.
> > > > > > > > > Yali
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> >
> >
> >
> >
------------------------------------------------
Subject: May you please help me to set up a config file to generate ROC curves?
From: Yali Mao - NOAA Affiliate
Time: Tue Jan 26 12:45:36 2021
Tatiana,
This is great. I am looking forward to seeing the new feature.
Thank you.
Yali
On Tue, Jan 26, 2021 at 1:53 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> Yali
> I added a checkbox and a parameter to XML file that would control
the
> connection order.
> This functionality will come with METviewer4.0-beta3 and will be
available
> on the dev instance of METviewer. We are planning to release it in a
few
> days.
> There is the link to the GitHub issue:
> https://github.com/dtcenter/METviewer/issues/246
>
> Tatiana
>
> On Mon Jan 25 11:50:21 2021, yali.mao at noaa.gov wrote:
> > Taiana,
> > Thank you. Wish it could be solved soon.
> > Yali
> >
> > On Mon, Jan 25, 2021 at 11:14 AM Tatiana Burek via RT
<met_help at ucar.edu
> >
> > wrote:
> >
> > > Yali,
> > >
> > > I created a GitHub issue and will will let you now when I make
some
> > > progress:
> > > https://github.com/dtcenter/METviewer/issues/246
> > >
> > > Tatiana
> > >
> > >
> > > On Fri Jan 22 13:56:32 2021, yali.mao at noaa.gov wrote:
> > > > Hi John,
> > > > I will try more and let you know my next sample result.
> > > > Thank you.
> > > > Yali
> > > >
> > > > On Fri, Jan 22, 2021 at 11:44 AM John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > > Yali,
> > > > >
> > > > > You could certainly try reversing the sign of the thresholds
to see
> > > > > if that
> > > > > produces a better result in METviewer for this specific
example. In
> > > > > fact,
> > > > > for testing, you could do it both ways in the same config
file.
> Just
> > > > > specify the same field twice. And since the "name" and
"level"
> > > > > setting
> > > > > remain constant, I just factored them out to a higher level
of
> > > > > context for
> > > > > simplicity:
> > > > >
> > > > >
> > > > > *name = "ICESEV";*
> > > > >
> > > > > *level = [ "P800", "P700", "P600", "P500", "P400" ];*
> > > > > *fcst = {*
> > > > > * field = [ { cat_thresh = [ >=1, >=2, >=3, >=4 ]; }, {
> cat_thresh
> > > > > = [
> > > > > <=1, <=2, <=3, <=4 ]; } ];*
> > > > > *}*
> > > > >
> > > > > *obs = {*
> > > > > * field = [ { cat_thresh = [ >=1, >=1, >=1, >=1 ]; }, {
> cat_thresh
> > > > > = [
> > > > > <=1, <=1, <=1, <=1 ]; } ];*
> > > > > *}*
> > > > >
> > > > > When testing the result in METviewer, you'll need to
explicitly
> > > > > specify the
> > > > > "fcst_thresh" and "obs_thresh" values to be used in the
"plot_fix"
> > > > > section.
> > > > >
> > > > > But regardless, I suspect we'll need some tweaks within
METviewer
> to
> > > > > make
> > > > > the plotting of ROC curves from 2x2 contingency tables more
robust.
> > > > > Your
> > > > > originally settings *should* have worked and produced a
sensible
> ROC
> > > > > curve.
> > > > > I'm hoping Tatiana can find a nice solution here.
> > > > >
> > > > > Thanks,
> > > > > John
> > > > >
> > > > > On Thu, Jan 21, 2021 at 7:29 PM Yali Mao - NOAA Affiliate
via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261 >
> > > > > >
> > > > > > John,
> > > > > > I set cat_thresh by myself and am not sure whether I set
it up
> > > > > correctly. I
> > > > > > am more concerned whether I create stat data correctly.
> > > > > > Should I change to cat_thresh = [ <=1, <=2, <=3, <=4 ] ?
> > > > > >
> > > > > > The variable of the verification is icing severity, the
larger
> > > > > > number the
> > > > > > more severe.
> > > > > > Thank you.
> > > > > > Yali
> > > > > >
> > > > > > On Thu, Jan 21, 2021 at 7:14 PM John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > > > Yali,
> > > > > > >
> > > > > > > I apologize for the long delay in answering this
question! I
> just
> > > > > > > got a
> > > > > > > chance to take a look at your example. I'm definitely
able to
> > > > > > > replicate
> > > > > > the
> > > > > > > behavior you described. I suspect the problem is that
the dots
> on
> > > > > > > that
> > > > > > plot
> > > > > > > are being connected in the wrong order.
> > > > > > >
> > > > > > > I'm going to reassign this ticket to Tatiana (cc'ed
here) so
> she
> > > > > > > can
> > > > > > > comment on my theory.
> > > > > > >
> > > > > > > Tatiana, in the attached plot/xml, Yali is creating a
ROC curve
> > > > > > > from
> > > > > 2x2
> > > > > > > contingency tables.
> > > > > > > https://metviewer.nws.noaa.gov/servlet
> > > > > > >
> > > > > > > The forecast thresholds are defined as: cat_thresh = [
>=1,
> >=2,
> > > > > > > >=3,
> > > > > > =4
> > > > > > > ];
> > > > > > > And the observation thresholds are defined as:
cat_thresh = [
> > > > > > > > =1, >=1, >=1, >=1 ];
> > > > > > >
> > > > > > > This setup matches the instructions from the METviewer
user's
> > > > > > > guide:
> > > > > > >
> > >
https://dtcenter.github.io/METviewer/develop/Users_Guide/rocplot.html
> > > > > > >
> > > > > > > I think the order of connecting the dots would depend on
the
> > > > > > > threshold
> > > > > > > *type*, greater-than in one order and less-than in
another
> order.
> > > > > > > But
> > > > > I'm
> > > > > > > not totally positive on that. I wonder if we should just
add a
> > > > > > > user-settable checkbox to reverse the connection order
or
> > > > > > > something?
> > > > > > >
> > > > > > > Can you please take a look when you get a chance?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jan 15, 2021 at 8:44 PM Yali Mao - NOAA
Affiliate via
> RT
> > > > > > > <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261 >
> > > > > > > >
> > > > > > > > Hi David,
> > > > > > > > Sorry I thought the plot could be seen on
> > > > > > > > https://metviewer.nws.noaa.gov/metviewer1.jsp by
checking
> the
> > > > > history
> > > > > > > > #20210115_200405.
> > > > > > > > Attached please find the plot.
> > > > > > > > Thank you.
> > > > > > > > Yali
> > > > > > > >
> > > > > > > > On Fri, Jan 15, 2021 at 4:24 PM David Fillmore via RT
<
> > > > > > met_help at ucar.edu
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Yali -
> > > > > > > > > I'm watching met_help today and have retrieved your
data
> and
> > > > > > > > > config
> > > > > > > files
> > > > > > > > > for grid_stat testing.
> > > > > > > > > Are you able to attach any of the bad ROC curves to
this
> > > > > > > > > email
> > > > > chain?
> > > > > > > > > I will likely try to reproduce with the METplotpy
> > > > > > > > > roc_diagram.py
> > > > > > > script.
> > > > > > > > > I have cc'ed John on this ticket so he can also have
a look
> > > > > > > > > at your
> > > > > > > > config
> > > > > > > > > file.
> > > > > > > > > thanks,
> > > > > > > > > David
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Jan 15, 2021 at 1:48 PM Yali Mao - NOAA
Affiliate
> via
> > > > > > > > > RT <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Fri Jan 15 13:47:43 2021: Request 98261 was acted
upon.
> > > > > > > > > > Transaction: Ticket created by yali.mao at noaa.gov
> > > > > > > > > > Queue: met_help
> > > > > > > > > > Subject: May you please help me to set up a
config
> > > > > > > > > > file to
> > > > > > > > generate
> > > > > > > > > > ROC curves?
> > > > > > > > > > Owner: Nobody
> > > > > > > > > > Requestors: yali.mao at noaa.gov
> > > > > > > > > > Status: new
> > > > > > > > > > Ticket <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > > I come back to ask for help again. All related
files are
> > > > > > transferred
> > > > > > > to
> > > > > > > > > > your ftp server, ftp.rap.ucar.edu, under the
folder:
> > > > > > > > > > /incoming/irap/met_help/mao
> > > > > > > > > > Script: test4help.sh (Start with it generate a
stat file,
> > > > > > > > > > but it
> > > > > > may
> > > > > > > > > need a
> > > > > > > > > > bit modifications)
> > > > > > > > > > config file: wafs_verf_g2g_config.icesev (Here I
need
> your
> > > > > > > > > > help)
> > > > > > > > > > obs file: obsv.icesev.grib2
> > > > > > > > > > fcst file: fcst.icesev.grib2
> > > > > > > > > > Grib2 configuration file: grib2_icesev.txt (icesev
is a
> > > > > > > > > > very new
> > > > > > > field
> > > > > > > > > > name, grib2 lib hasn't been updated with it yet,
so
> > > > > MET_GRIB_TABLES
> > > > > > > > > > environment variable is defined in the test
script)
> > > > > > > > > > Spatial Masking files: *nc
> > > > > > > > > >
> > > > > > > > > > I was able to generate a stat file by running the
test
> > > > > > > > > > script,
> > > > > > > > > > test4help.sh. But the plotting looks weird, at
least not
> > > > > > > > > > like a
> > > > > ROC
> > > > > > > > > curve.
> > > > > > > > > > Please check more details of the plotting from the
> history
> > > > > > > > > 20210115_200405
> > > > > > > > > > on https://metviewer.nws.noaa.gov/metviewer1.jsp
> > > > > > > > > >
> > > > > > > > > > So I must have maken mistakes when setting up
> > > > > > > > > wafs_verf_g2g_config.icesev.
> > > > > > > > > > Both observation and forecast data are grib2 data,
both
> > > > > > > > > > values
> > > > > are
> > > > > > > > > integers
> > > > > > > > > > at 0,1,2,3,4. For observations, I can define a
'yes'
> event
> > > > > > > > > > when
> > > > > > > > value> =1,
> > > > > > > > > > and I want to plot ROC curve based on the forecast
values
> > > > > > > > > > at >=1,
> > > > > > > > =2,
> > > > > > > > > > =3,
> > > > > > > > > > =4.
> > > > > > > > > >
> > > > > > > > > > Thank you.
> > > > > > > > > > Yali
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > >
> > >
> > >
> > >
>
>
>
>
------------------------------------------------
Subject: May you please help me to set up a config file to generate ROC curves?
From: Tatiana Burek
Time: Thu Feb 04 07:30:49 2021
Yali
I deployed METviewer 4.0-beta3 to AWS with the fix for this issue.
if you use https://metviewer.nws.noaa.gov/metviewer_dev/ instance you
should see a new checkbox "Reverse connection order" on the Roc tab.
It would reverse the connection of the points. This fix works for only
in Rscript. We plan to update Python in the next release.
Let me know if this works for you.
Tatiana
On Tue Jan 26 12:45:36 2021, yali.mao at noaa.gov wrote:
> Tatiana,
> This is great. I am looking forward to seeing the new feature.
> Thank you.
> Yali
>
> On Tue, Jan 26, 2021 at 1:53 PM Tatiana Burek via RT
> <met_help at ucar.edu>
> wrote:
>
> > Yali
> > I added a checkbox and a parameter to XML file that would control
the
> > connection order.
> > This functionality will come with METviewer4.0-beta3 and will be
> > available
> > on the dev instance of METviewer. We are planning to release it in
a
> > few
> > days.
> > There is the link to the GitHub issue:
> > https://github.com/dtcenter/METviewer/issues/246
> >
> > Tatiana
> >
> > On Mon Jan 25 11:50:21 2021, yali.mao at noaa.gov wrote:
> > > Taiana,
> > > Thank you. Wish it could be solved soon.
> > > Yali
> > >
> > > On Mon, Jan 25, 2021 at 11:14 AM Tatiana Burek via RT
> > > <met_help at ucar.edu
> > >
> > > wrote:
> > >
> > > > Yali,
> > > >
> > > > I created a GitHub issue and will will let you now when I make
> > > > some
> > > > progress:
> > > > https://github.com/dtcenter/METviewer/issues/246
> > > >
> > > > Tatiana
> > > >
> > > >
> > > > On Fri Jan 22 13:56:32 2021, yali.mao at noaa.gov wrote:
> > > > > Hi John,
> > > > > I will try more and let you know my next sample result.
> > > > > Thank you.
> > > > > Yali
> > > > >
> > > > > On Fri, Jan 22, 2021 at 11:44 AM John Halley Gotway via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > > Yali,
> > > > > >
> > > > > > You could certainly try reversing the sign of the
thresholds
> > > > > > to see
> > > > > > if that
> > > > > > produces a better result in METviewer for this specific
> > > > > > example. In
> > > > > > fact,
> > > > > > for testing, you could do it both ways in the same config
> > > > > > file.
> > Just
> > > > > > specify the same field twice. And since the "name" and
> > > > > > "level"
> > > > > > setting
> > > > > > remain constant, I just factored them out to a higher
level
> > > > > > of
> > > > > > context for
> > > > > > simplicity:
> > > > > >
> > > > > >
> > > > > > *name = "ICESEV";*
> > > > > >
> > > > > > *level = [ "P800", "P700", "P600", "P500", "P400" ];*
> > > > > > *fcst = {*
> > > > > > * field = [ { cat_thresh = [ >=1, >=2, >=3, >=4 ]; }, {
> > cat_thresh
> > > > > > = [
> > > > > > <=1, <=2, <=3, <=4 ]; } ];*
> > > > > > *}*
> > > > > >
> > > > > > *obs = {*
> > > > > > * field = [ { cat_thresh = [ >=1, >=1, >=1, >=1 ]; }, {
> > cat_thresh
> > > > > > = [
> > > > > > <=1, <=1, <=1, <=1 ]; } ];*
> > > > > > *}*
> > > > > >
> > > > > > When testing the result in METviewer, you'll need to
> > > > > > explicitly
> > > > > > specify the
> > > > > > "fcst_thresh" and "obs_thresh" values to be used in the
> > > > > > "plot_fix"
> > > > > > section.
> > > > > >
> > > > > > But regardless, I suspect we'll need some tweaks within
> > > > > > METviewer
> > to
> > > > > > make
> > > > > > the plotting of ROC curves from 2x2 contingency tables
more
> > > > > > robust.
> > > > > > Your
> > > > > > originally settings *should* have worked and produced a
> > > > > > sensible
> > ROC
> > > > > > curve.
> > > > > > I'm hoping Tatiana can find a nice solution here.
> > > > > >
> > > > > > Thanks,
> > > > > > John
> > > > > >
> > > > > > On Thu, Jan 21, 2021 at 7:29 PM Yali Mao - NOAA Affiliate
via
> > > > > > RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261
>
> > > > > > >
> > > > > > > John,
> > > > > > > I set cat_thresh by myself and am not sure whether I set
it
> > > > > > > up
> > > > > > correctly. I
> > > > > > > am more concerned whether I create stat data correctly.
> > > > > > > Should I change to cat_thresh = [ <=1, <=2, <=3, <=4 ] ?
> > > > > > >
> > > > > > > The variable of the verification is icing severity, the
> > > > > > > larger
> > > > > > > number the
> > > > > > > more severe.
> > > > > > > Thank you.
> > > > > > > Yali
> > > > > > >
> > > > > > > On Thu, Jan 21, 2021 at 7:14 PM John Halley Gotway via
RT <
> > > > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Yali,
> > > > > > > >
> > > > > > > > I apologize for the long delay in answering this
> > > > > > > > question! I
> > just
> > > > > > > > got a
> > > > > > > > chance to take a look at your example. I'm definitely
> > > > > > > > able to
> > > > > > > > replicate
> > > > > > > the
> > > > > > > > behavior you described. I suspect the problem is that
the
> > > > > > > > dots
> > on
> > > > > > > > that
> > > > > > > plot
> > > > > > > > are being connected in the wrong order.
> > > > > > > >
> > > > > > > > I'm going to reassign this ticket to Tatiana (cc'ed
here)
> > > > > > > > so
> > she
> > > > > > > > can
> > > > > > > > comment on my theory.
> > > > > > > >
> > > > > > > > Tatiana, in the attached plot/xml, Yali is creating a
ROC
> > > > > > > > curve
> > > > > > > > from
> > > > > > 2x2
> > > > > > > > contingency tables.
> > > > > > > > https://metviewer.nws.noaa.gov/servlet
> > > > > > > >
> > > > > > > > The forecast thresholds are defined as: cat_thresh = [
> > > > > > > > >=1,
> > > =2,
> > > > > > > > > =3,
> > > > > > > =4
> > > > > > > > ];
> > > > > > > > And the observation thresholds are defined as:
cat_thresh
> > > > > > > > = [
> > > > > > > > > =1, >=1, >=1, >=1 ];
> > > > > > > >
> > > > > > > > This setup matches the instructions from the METviewer
> > > > > > > > user's
> > > > > > > > guide:
> > > > > > > >
> > > >
https://dtcenter.github.io/METviewer/develop/Users_Guide/rocplot.html
> > > > > > > >
> > > > > > > > I think the order of connecting the dots would depend
on
> > > > > > > > the
> > > > > > > > threshold
> > > > > > > > *type*, greater-than in one order and less-than in
> > > > > > > > another
> > order.
> > > > > > > > But
> > > > > > I'm
> > > > > > > > not totally positive on that. I wonder if we should
just
> > > > > > > > add a
> > > > > > > > user-settable checkbox to reverse the connection order
or
> > > > > > > > something?
> > > > > > > >
> > > > > > > > Can you please take a look when you get a chance?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > John
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Jan 15, 2021 at 8:44 PM Yali Mao - NOAA
Affiliate
> > > > > > > > via
> > RT
> > > > > > > > <
> > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261 >
> > > > > > > > >
> > > > > > > > > Hi David,
> > > > > > > > > Sorry I thought the plot could be seen on
> > > > > > > > > https://metviewer.nws.noaa.gov/metviewer1.jsp by
> > > > > > > > > checking
> > the
> > > > > > history
> > > > > > > > > #20210115_200405.
> > > > > > > > > Attached please find the plot.
> > > > > > > > > Thank you.
> > > > > > > > > Yali
> > > > > > > > >
> > > > > > > > > On Fri, Jan 15, 2021 at 4:24 PM David Fillmore via
RT <
> > > > > > > met_help at ucar.edu
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Yali -
> > > > > > > > > > I'm watching met_help today and have retrieved
your
> > > > > > > > > > data
> > and
> > > > > > > > > > config
> > > > > > > > files
> > > > > > > > > > for grid_stat testing.
> > > > > > > > > > Are you able to attach any of the bad ROC curves
to
> > > > > > > > > > this
> > > > > > > > > > email
> > > > > > chain?
> > > > > > > > > > I will likely try to reproduce with the METplotpy
> > > > > > > > > > roc_diagram.py
> > > > > > > > script.
> > > > > > > > > > I have cc'ed John on this ticket so he can also
have
> > > > > > > > > > a look
> > > > > > > > > > at your
> > > > > > > > > config
> > > > > > > > > > file.
> > > > > > > > > > thanks,
> > > > > > > > > > David
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Fri, Jan 15, 2021 at 1:48 PM Yali Mao - NOAA
> > > > > > > > > > Affiliate
> > via
> > > > > > > > > > RT <
> > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Fri Jan 15 13:47:43 2021: Request 98261 was
acted
> > > > > > > > > > > upon.
> > > > > > > > > > > Transaction: Ticket created by yali.mao at noaa.gov
> > > > > > > > > > > Queue: met_help
> > > > > > > > > > > Subject: May you please help me to set up a
> > > > > > > > > > > config
> > > > > > > > > > > file to
> > > > > > > > > generate
> > > > > > > > > > > ROC curves?
> > > > > > > > > > > Owner: Nobody
> > > > > > > > > > > Requestors: yali.mao at noaa.gov
> > > > > > > > > > > Status: new
> > > > > > > > > > > Ticket <URL:
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > > I come back to ask for help again. All related
> > > > > > > > > > > files are
> > > > > > > transferred
> > > > > > > > to
> > > > > > > > > > > your ftp server, ftp.rap.ucar.edu, under the
> > > > > > > > > > > folder:
> > > > > > > > > > > /incoming/irap/met_help/mao
> > > > > > > > > > > Script: test4help.sh (Start with it generate a
stat
> > > > > > > > > > > file,
> > > > > > > > > > > but it
> > > > > > > may
> > > > > > > > > > need a
> > > > > > > > > > > bit modifications)
> > > > > > > > > > > config file: wafs_verf_g2g_config.icesev (Here I
> > > > > > > > > > > need
> > your
> > > > > > > > > > > help)
> > > > > > > > > > > obs file: obsv.icesev.grib2
> > > > > > > > > > > fcst file: fcst.icesev.grib2
> > > > > > > > > > > Grib2 configuration file: grib2_icesev.txt
(icesev
> > > > > > > > > > > is a
> > > > > > > > > > > very new
> > > > > > > > field
> > > > > > > > > > > name, grib2 lib hasn't been updated with it yet,
so
> > > > > > MET_GRIB_TABLES
> > > > > > > > > > > environment variable is defined in the test
script)
> > > > > > > > > > > Spatial Masking files: *nc
> > > > > > > > > > >
> > > > > > > > > > > I was able to generate a stat file by running
the
> > > > > > > > > > > test
> > > > > > > > > > > script,
> > > > > > > > > > > test4help.sh. But the plotting looks weird, at
> > > > > > > > > > > least not
> > > > > > > > > > > like a
> > > > > > ROC
> > > > > > > > > > curve.
> > > > > > > > > > > Please check more details of the plotting from
the
> > history
> > > > > > > > > > 20210115_200405
> > > > > > > > > > > on https://metviewer.nws.noaa.gov/metviewer1.jsp
> > > > > > > > > > >
> > > > > > > > > > > So I must have maken mistakes when setting up
> > > > > > > > > > wafs_verf_g2g_config.icesev.
> > > > > > > > > > > Both observation and forecast data are grib2
data,
> > > > > > > > > > > both
> > > > > > > > > > > values
> > > > > > are
> > > > > > > > > > integers
> > > > > > > > > > > at 0,1,2,3,4. For observations, I can define a
> > > > > > > > > > > 'yes'
> > event
> > > > > > > > > > > when
> > > > > > > > > value> =1,
> > > > > > > > > > > and I want to plot ROC curve based on the
forecast
> > > > > > > > > > > values
> > > > > > > > > > > at >=1,
> > > > > > > > > =2,
> > > > > > > > > > > =3,
> > > > > > > > > > > =4.
> > > > > > > > > > >
> > > > > > > > > > > Thank you.
> > > > > > > > > > > Yali
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > >
> > > >
> > > >
> > > >
> >
> >
> >
> >
------------------------------------------------
Subject: May you please help me to set up a config file to generate ROC curves?
From: Yali Mao - NOAA Affiliate
Time: Sun Feb 07 19:59:48 2021
Tatiana,
This is really great. Now it looks good to me. Thank you for your fast
fix!
Yali
On Thu, Feb 4, 2021 at 9:30 AM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> Yali
> I deployed METviewer 4.0-beta3 to AWS with the fix for this issue.
> if you use https://metviewer.nws.noaa.gov/metviewer_dev/ instance
you
> should see a new checkbox "Reverse connection order" on the Roc tab.
It
> would reverse the connection of the points. This fix works for only
in
> Rscript. We plan to update Python in the next release.
> Let me know if this works for you.
>
> Tatiana
>
> On Tue Jan 26 12:45:36 2021, yali.mao at noaa.gov wrote:
> > Tatiana,
> > This is great. I am looking forward to seeing the new feature.
> > Thank you.
> > Yali
> >
> > On Tue, Jan 26, 2021 at 1:53 PM Tatiana Burek via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > Yali
> > > I added a checkbox and a parameter to XML file that would
control the
> > > connection order.
> > > This functionality will come with METviewer4.0-beta3 and will be
> > > available
> > > on the dev instance of METviewer. We are planning to release it
in a
> > > few
> > > days.
> > > There is the link to the GitHub issue:
> > > https://github.com/dtcenter/METviewer/issues/246
> > >
> > > Tatiana
> > >
> > > On Mon Jan 25 11:50:21 2021, yali.mao at noaa.gov wrote:
> > > > Taiana,
> > > > Thank you. Wish it could be solved soon.
> > > > Yali
> > > >
> > > > On Mon, Jan 25, 2021 at 11:14 AM Tatiana Burek via RT
> > > > <met_help at ucar.edu
> > > >
> > > > wrote:
> > > >
> > > > > Yali,
> > > > >
> > > > > I created a GitHub issue and will will let you now when I
make
> > > > > some
> > > > > progress:
> > > > > https://github.com/dtcenter/METviewer/issues/246
> > > > >
> > > > > Tatiana
> > > > >
> > > > >
> > > > > On Fri Jan 22 13:56:32 2021, yali.mao at noaa.gov wrote:
> > > > > > Hi John,
> > > > > > I will try more and let you know my next sample result.
> > > > > > Thank you.
> > > > > > Yali
> > > > > >
> > > > > > On Fri, Jan 22, 2021 at 11:44 AM John Halley Gotway via RT
<
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > > Yali,
> > > > > > >
> > > > > > > You could certainly try reversing the sign of the
thresholds
> > > > > > > to see
> > > > > > > if that
> > > > > > > produces a better result in METviewer for this specific
> > > > > > > example. In
> > > > > > > fact,
> > > > > > > for testing, you could do it both ways in the same
config
> > > > > > > file.
> > > Just
> > > > > > > specify the same field twice. And since the "name" and
> > > > > > > "level"
> > > > > > > setting
> > > > > > > remain constant, I just factored them out to a higher
level
> > > > > > > of
> > > > > > > context for
> > > > > > > simplicity:
> > > > > > >
> > > > > > >
> > > > > > > *name = "ICESEV";*
> > > > > > >
> > > > > > > *level = [ "P800", "P700", "P600", "P500", "P400" ];*
> > > > > > > *fcst = {*
> > > > > > > * field = [ { cat_thresh = [ >=1, >=2, >=3, >=4 ]; },
{
> > > cat_thresh
> > > > > > > = [
> > > > > > > <=1, <=2, <=3, <=4 ]; } ];*
> > > > > > > *}*
> > > > > > >
> > > > > > > *obs = {*
> > > > > > > * field = [ { cat_thresh = [ >=1, >=1, >=1, >=1 ]; },
{
> > > cat_thresh
> > > > > > > = [
> > > > > > > <=1, <=1, <=1, <=1 ]; } ];*
> > > > > > > *}*
> > > > > > >
> > > > > > > When testing the result in METviewer, you'll need to
> > > > > > > explicitly
> > > > > > > specify the
> > > > > > > "fcst_thresh" and "obs_thresh" values to be used in the
> > > > > > > "plot_fix"
> > > > > > > section.
> > > > > > >
> > > > > > > But regardless, I suspect we'll need some tweaks within
> > > > > > > METviewer
> > > to
> > > > > > > make
> > > > > > > the plotting of ROC curves from 2x2 contingency tables
more
> > > > > > > robust.
> > > > > > > Your
> > > > > > > originally settings *should* have worked and produced a
> > > > > > > sensible
> > > ROC
> > > > > > > curve.
> > > > > > > I'm hoping Tatiana can find a nice solution here.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > John
> > > > > > >
> > > > > > > On Thu, Jan 21, 2021 at 7:29 PM Yali Mao - NOAA
Affiliate via
> > > > > > > RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261 >
> > > > > > > >
> > > > > > > > John,
> > > > > > > > I set cat_thresh by myself and am not sure whether I
set it
> > > > > > > > up
> > > > > > > correctly. I
> > > > > > > > am more concerned whether I create stat data
correctly.
> > > > > > > > Should I change to cat_thresh = [ <=1, <=2, <=3, <=4 ]
?
> > > > > > > >
> > > > > > > > The variable of the verification is icing severity,
the
> > > > > > > > larger
> > > > > > > > number the
> > > > > > > > more severe.
> > > > > > > > Thank you.
> > > > > > > > Yali
> > > > > > > >
> > > > > > > > On Thu, Jan 21, 2021 at 7:14 PM John Halley Gotway via
RT <
> > > > > > > > met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Yali,
> > > > > > > > >
> > > > > > > > > I apologize for the long delay in answering this
> > > > > > > > > question! I
> > > just
> > > > > > > > > got a
> > > > > > > > > chance to take a look at your example. I'm
definitely
> > > > > > > > > able to
> > > > > > > > > replicate
> > > > > > > > the
> > > > > > > > > behavior you described. I suspect the problem is
that the
> > > > > > > > > dots
> > > on
> > > > > > > > > that
> > > > > > > > plot
> > > > > > > > > are being connected in the wrong order.
> > > > > > > > >
> > > > > > > > > I'm going to reassign this ticket to Tatiana (cc'ed
here)
> > > > > > > > > so
> > > she
> > > > > > > > > can
> > > > > > > > > comment on my theory.
> > > > > > > > >
> > > > > > > > > Tatiana, in the attached plot/xml, Yali is creating
a ROC
> > > > > > > > > curve
> > > > > > > > > from
> > > > > > > 2x2
> > > > > > > > > contingency tables.
> > > > > > > > > https://metviewer.nws.noaa.gov/servlet
> > > > > > > > >
> > > > > > > > > The forecast thresholds are defined as: cat_thresh =
[
> > > > > > > > > >=1,
> > > > =2,
> > > > > > > > > > =3,
> > > > > > > > =4
> > > > > > > > > ];
> > > > > > > > > And the observation thresholds are defined as:
cat_thresh
> > > > > > > > > = [
> > > > > > > > > > =1, >=1, >=1, >=1 ];
> > > > > > > > >
> > > > > > > > > This setup matches the instructions from the
METviewer
> > > > > > > > > user's
> > > > > > > > > guide:
> > > > > > > > >
> > > > >
>
https://dtcenter.github.io/METviewer/develop/Users_Guide/rocplot.html
> > > > > > > > >
> > > > > > > > > I think the order of connecting the dots would
depend on
> > > > > > > > > the
> > > > > > > > > threshold
> > > > > > > > > *type*, greater-than in one order and less-than in
> > > > > > > > > another
> > > order.
> > > > > > > > > But
> > > > > > > I'm
> > > > > > > > > not totally positive on that. I wonder if we should
just
> > > > > > > > > add a
> > > > > > > > > user-settable checkbox to reverse the connection
order or
> > > > > > > > > something?
> > > > > > > > >
> > > > > > > > > Can you please take a look when you get a chance?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > John
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Jan 15, 2021 at 8:44 PM Yali Mao - NOAA
Affiliate
> > > > > > > > > via
> > > RT
> > > > > > > > > <
> > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261 >
> > > > > > > > > >
> > > > > > > > > > Hi David,
> > > > > > > > > > Sorry I thought the plot could be seen on
> > > > > > > > > > https://metviewer.nws.noaa.gov/metviewer1.jsp by
> > > > > > > > > > checking
> > > the
> > > > > > > history
> > > > > > > > > > #20210115_200405.
> > > > > > > > > > Attached please find the plot.
> > > > > > > > > > Thank you.
> > > > > > > > > > Yali
> > > > > > > > > >
> > > > > > > > > > On Fri, Jan 15, 2021 at 4:24 PM David Fillmore via
RT <
> > > > > > > > met_help at ucar.edu
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Yali -
> > > > > > > > > > > I'm watching met_help today and have retrieved
your
> > > > > > > > > > > data
> > > and
> > > > > > > > > > > config
> > > > > > > > > files
> > > > > > > > > > > for grid_stat testing.
> > > > > > > > > > > Are you able to attach any of the bad ROC curves
to
> > > > > > > > > > > this
> > > > > > > > > > > email
> > > > > > > chain?
> > > > > > > > > > > I will likely try to reproduce with the
METplotpy
> > > > > > > > > > > roc_diagram.py
> > > > > > > > > script.
> > > > > > > > > > > I have cc'ed John on this ticket so he can also
have
> > > > > > > > > > > a look
> > > > > > > > > > > at your
> > > > > > > > > > config
> > > > > > > > > > > file.
> > > > > > > > > > > thanks,
> > > > > > > > > > > David
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Jan 15, 2021 at 1:48 PM Yali Mao - NOAA
> > > > > > > > > > > Affiliate
> > > via
> > > > > > > > > > > RT <
> > > > > > > > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Fri Jan 15 13:47:43 2021: Request 98261 was
acted
> > > > > > > > > > > > upon.
> > > > > > > > > > > > Transaction: Ticket created by
yali.mao at noaa.gov
> > > > > > > > > > > > Queue: met_help
> > > > > > > > > > > > Subject: May you please help me to set up
a
> > > > > > > > > > > > config
> > > > > > > > > > > > file to
> > > > > > > > > > generate
> > > > > > > > > > > > ROC curves?
> > > > > > > > > > > > Owner: Nobody
> > > > > > > > > > > > Requestors: yali.mao at noaa.gov
> > > > > > > > > > > > Status: new
> > > > > > > > > > > > Ticket <URL:
> > > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=98261
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Hi,
> > > > > > > > > > > > I come back to ask for help again. All related
> > > > > > > > > > > > files are
> > > > > > > > transferred
> > > > > > > > > to
> > > > > > > > > > > > your ftp server, ftp.rap.ucar.edu, under the
> > > > > > > > > > > > folder:
> > > > > > > > > > > > /incoming/irap/met_help/mao
> > > > > > > > > > > > Script: test4help.sh (Start with it generate a
stat
> > > > > > > > > > > > file,
> > > > > > > > > > > > but it
> > > > > > > > may
> > > > > > > > > > > need a
> > > > > > > > > > > > bit modifications)
> > > > > > > > > > > > config file: wafs_verf_g2g_config.icesev (Here
I
> > > > > > > > > > > > need
> > > your
> > > > > > > > > > > > help)
> > > > > > > > > > > > obs file: obsv.icesev.grib2
> > > > > > > > > > > > fcst file: fcst.icesev.grib2
> > > > > > > > > > > > Grib2 configuration file: grib2_icesev.txt
(icesev
> > > > > > > > > > > > is a
> > > > > > > > > > > > very new
> > > > > > > > > field
> > > > > > > > > > > > name, grib2 lib hasn't been updated with it
yet, so
> > > > > > > MET_GRIB_TABLES
> > > > > > > > > > > > environment variable is defined in the test
script)
> > > > > > > > > > > > Spatial Masking files: *nc
> > > > > > > > > > > >
> > > > > > > > > > > > I was able to generate a stat file by running
the
> > > > > > > > > > > > test
> > > > > > > > > > > > script,
> > > > > > > > > > > > test4help.sh. But the plotting looks weird, at
> > > > > > > > > > > > least not
> > > > > > > > > > > > like a
> > > > > > > ROC
> > > > > > > > > > > curve.
> > > > > > > > > > > > Please check more details of the plotting from
the
> > > history
> > > > > > > > > > > 20210115_200405
> > > > > > > > > > > > on
https://metviewer.nws.noaa.gov/metviewer1.jsp
> > > > > > > > > > > >
> > > > > > > > > > > > So I must have maken mistakes when setting up
> > > > > > > > > > > wafs_verf_g2g_config.icesev.
> > > > > > > > > > > > Both observation and forecast data are grib2
data,
> > > > > > > > > > > > both
> > > > > > > > > > > > values
> > > > > > > are
> > > > > > > > > > > integers
> > > > > > > > > > > > at 0,1,2,3,4. For observations, I can define a
> > > > > > > > > > > > 'yes'
> > > event
> > > > > > > > > > > > when
> > > > > > > > > > value> =1,
> > > > > > > > > > > > and I want to plot ROC curve based on the
forecast
> > > > > > > > > > > > values
> > > > > > > > > > > > at >=1,
> > > > > > > > > > =2,
> > > > > > > > > > > > =3,
> > > > > > > > > > > > =4.
> > > > > > > > > > > >
> > > > > > > > > > > > Thank you.
> > > > > > > > > > > > Yali
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > >
> > >
> > >
> > >
>
>
>
>
------------------------------------------------
More information about the Met_help
mailing list