[Met_help] [rt.rap.ucar.edu #96971] History for Implementing Climatology Data to Point Stat
John Halley Gotway via RT
met_help at ucar.edu
Fri Oct 23 10:44:21 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
To whom this may concern,
I am wanting to extend the statistics I currently compute for my point stat
processing with no progress. Currently, I am able to produce mean
forecast/obs and RMSE but I am also wanting to include the following:
-
CNT
-
Forecast/Observation Standard Deviation
-
Pearson Correlation Coefficient
-
Standard deviation of the error
-
Anomaly Correlation Coefficient
-
CST
-
Base Rate
-
Accuracy
-
Frequency Bias
-
Heidke Skill Score
At the moment, those statistics listed are all reporting either NA, 0, or 1
(depending on the stat). I understand I need to include thresholds which I
have tried both within my point stat config and outside (but including when
running stat analysis) with no luck.
I thought perhaps my problem was my climo_mean attributes were empty,
however when I try to list txt files it does not accept them (which I
assumed would happen). I then tried to convert these files using
ascii2nc with no luck. I am assuming the error is due to the fact each
climo file uses julian day rather than an individual date (to avoid 35
files having the same information, as these are applicable to 35 years
worth of data). My txt files have the following format:
*Message_Type Station_ID Valid_Time Lat Lon Elevation Variable_Name Level
Height QC_String Observation_Value\n*
If I do not need to provide climatology values to receive those stats
listed, then my issue must be with how I am formatting my point stat and
stat analysis calls which is as follows:
*for proj in {6..384..6}*
*do PR=`echo $(printf %03d ${proj})`*
* ${metDIR}/point_stat ${fcstdir}/${WSFILE}_pr${PR}.nc ${VALIDFILE}
PointStatConfig_ws -v 4 -outdir ${writeDIR} -obs_valid_beg
${VALIDDATE}_000000 -obs_valid_end ${VALIDDATE}_235959 *
*done*
*fileLIST=($(ls ${outdir}/point_stat_*_${currDAY}_*0000V_mpr.txt))*
*${metDIR}/stat_analysis -lookin ${fileLIST} -config
"STATAnalysisConfig_ws" -v 0 -vx_mask FULL -job aggregate_stat -line_type
MPR -out_line_type CNT -out_stat
"${outdir}/${currDAY}cy${CY}_mpr_to_cnt.stat" -by FCST_VAR,OBS_SID -set_hdr
VX_MASK OBS_SID -set_hdr DESC CASE -out
"${outdir}/${YYYY}${MM}${DD}_v${currDAY}_ws_cnt.stat"*
* ${metDIR}/stat_analysis -lookin ${fileLIST} -config
"STATAnalysisConfig_ws" -v 0 -vx_mask FULL -job aggregate_stat -line_type
MPR -out_line_type MCTS -out_stat
"${outdir}/${currDAY}cy${CY}_mpr_to_mcts.stat" -by FCST_VAR,OBS_SID
-set_hdr VX_MASK OBS_SID -set_hdr DESC CASE -out_thresh
ge4.12,ge10.29,ge18.01,ge25.72 -out
"${outdir}/${YYYY}${MM}${DD}_v${currDAY}_ws_mcts.stat" *
If it is more beneficial, I have provided the following attachments:
1. Current working point stat config file (PointStatConfig_ws)
2. Testing point stat config file for climo (PointStatConfig_ws.test)
3. Stat analysis config (STATAnalysisConfig_ws)
4. 4 fcst data files (ws_YYYYMMDDCY_c00_prPPP)
5. 1 obs data file (YYYYMMDD_CY:00:00) which used ascii2nc to be created
6. 1 climo file (jdDDD_climatology)
Hopefully I did not provide too much information on what I have been trying
to accomplish. But basically, I am trying to utilize point_stat to compute
those stats listed for 36 core airports across CONUS, and at this point I
am at a loss as to why some work and others do not. Any and all guidance is
greatly appreciated.
Thanks!
Erica Burrows
Atmospheric Scientist
INNOVIM/Climate Prediction Center
NOAA Center for Weather and Climate Prediction
5830 University Research Court
College Park, MD 20740
Work:
Cell: 818-925-5795
Email: erica.burrows at noaa.gov
*Innovim LLC*
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: Implementing Climatology Data to Point Stat
From: Julie Prestopnik
Time: Tue Oct 06 10:56:01 2020
Hi Erica.
I see that you are having trouble getting the statistics that you
would
like from Point-Stat. I am assigning this ticket to John Halley
Gotway.
Please allow a few business days for a response.
Julie
On Tue, Oct 6, 2020 at 10:37 AM Erica Burrows - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> Tue Oct 06 10:37:13 2020: Request 96971 was acted upon.
> Transaction: Ticket created by erica.burrows at noaa.gov
> Queue: met_help
> Subject: Implementing Climatology Data to Point Stat
> Owner: Nobody
> Requestors: erica.burrows at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96971 >
>
>
> To whom this may concern,
>
> I am wanting to extend the statistics I currently compute for my
point stat
> processing with no progress. Currently, I am able to produce mean
> forecast/obs and RMSE but I am also wanting to include the
following:
>
> -
>
> CNT
> -
>
> Forecast/Observation Standard Deviation
> -
>
> Pearson Correlation Coefficient
> -
>
> Standard deviation of the error
> -
>
> Anomaly Correlation Coefficient
> -
>
> CST
> -
>
> Base Rate
> -
>
> Accuracy
> -
>
> Frequency Bias
> -
>
> Heidke Skill Score
>
>
> At the moment, those statistics listed are all reporting either NA,
0, or 1
> (depending on the stat). I understand I need to include thresholds
which I
> have tried both within my point stat config and outside (but
including when
> running stat analysis) with no luck.
>
> I thought perhaps my problem was my climo_mean attributes were
empty,
> however when I try to list txt files it does not accept them (which
I
> assumed would happen). I then tried to convert these files using
> ascii2nc with no luck. I am assuming the error is due to the fact
each
> climo file uses julian day rather than an individual date (to avoid
35
> files having the same information, as these are applicable to 35
years
> worth of data). My txt files have the following format:
> *Message_Type Station_ID Valid_Time Lat Lon Elevation Variable_Name
Level
> Height QC_String Observation_Value\n*
>
> If I do not need to provide climatology values to receive those
stats
> listed, then my issue must be with how I am formatting my point stat
and
> stat analysis calls which is as follows:
>
> *for proj in {6..384..6}*
>
>
> *do PR=`echo $(printf %03d ${proj})`*
>
> * ${metDIR}/point_stat ${fcstdir}/${WSFILE}_pr${PR}.nc
${VALIDFILE}
> PointStatConfig_ws -v 4 -outdir ${writeDIR} -obs_valid_beg
> ${VALIDDATE}_000000 -obs_valid_end ${VALIDDATE}_235959 *
> *done*
>
> *fileLIST=($(ls ${outdir}/point_stat_*_${currDAY}_*0000V_mpr.txt))*
>
>
> *${metDIR}/stat_analysis -lookin ${fileLIST} -config
> "STATAnalysisConfig_ws" -v 0 -vx_mask FULL -job aggregate_stat
-line_type
> MPR -out_line_type CNT -out_stat
> "${outdir}/${currDAY}cy${CY}_mpr_to_cnt.stat" -by FCST_VAR,OBS_SID
-set_hdr
> VX_MASK OBS_SID -set_hdr DESC CASE -out
> "${outdir}/${YYYY}${MM}${DD}_v${currDAY}_ws_cnt.stat"*
>
> * ${metDIR}/stat_analysis -lookin ${fileLIST} -config
> "STATAnalysisConfig_ws" -v 0 -vx_mask FULL -job aggregate_stat
-line_type
> MPR -out_line_type MCTS -out_stat
> "${outdir}/${currDAY}cy${CY}_mpr_to_mcts.stat" -by FCST_VAR,OBS_SID
> -set_hdr VX_MASK OBS_SID -set_hdr DESC CASE -out_thresh
> ge4.12,ge10.29,ge18.01,ge25.72 -out
> "${outdir}/${YYYY}${MM}${DD}_v${currDAY}_ws_mcts.stat" *
>
> If it is more beneficial, I have provided the following attachments:
> 1. Current working point stat config file (PointStatConfig_ws)
> 2. Testing point stat config file for climo
(PointStatConfig_ws.test)
> 3. Stat analysis config (STATAnalysisConfig_ws)
> 4. 4 fcst data files (ws_YYYYMMDDCY_c00_prPPP)
> 5. 1 obs data file (YYYYMMDD_CY:00:00) which used ascii2nc to be
created
> 6. 1 climo file (jdDDD_climatology)
>
> Hopefully I did not provide too much information on what I have been
trying
> to accomplish. But basically, I am trying to utilize point_stat to
compute
> those stats listed for 36 core airports across CONUS, and at this
point I
> am at a loss as to why some work and others do not. Any and all
guidance is
> greatly appreciated.
>
> Thanks!
>
> Erica Burrows
> Atmospheric Scientist
> INNOVIM/Climate Prediction Center
> NOAA Center for Weather and Climate Prediction
> 5830 University Research Court
> College Park, MD 20740
>
> Work:
> Cell: 818-925-5795
> Email: erica.burrows at noaa.gov
> *Innovim LLC*
>
>
--
Julie Prestopnik (she/her/hers)
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: Implementing Climatology Data to Point Stat
From: John Halley Gotway
Time: Thu Oct 08 09:57:35 2020
Hello Erica,
My apologies for the delay in responding to your question.
I see that you're running Point-Stat and would like to increase the
number
of statistics in your evaluation. First, let me point out that the
output of Point-Stat is controlled by turning on/off line types. You
don't
control output statistic by statistic. Instead, you configure the
"output_flag" settings in the config file to request groups of
statistics
by their line type.
Looking in the Point-Stat config files you sent, I see that you're
only
requesting the MPR, i.e. matched pair, output line type.
And from your description, I see why. You'd like to compute stats
separately for each of 36 airports stations. So rather than computing
a
spatial aggregation with Point-Stat, you'd like to run Stat-Analysis
to
compute stats through time for each station.
So I suspect the real question is, what jobs should you be running in
STAT-Analysis?
I see that you sent some config files but no actual output from Point-
Stat.
If you send that I could take a look and recommend some STAT-Analysis
jobs.
You could either tar them up and send them or point me to them on a
NOAA
machine, like WCOSS, jet, or theia (can’t remember it successor right
now).
Here’s instructions for posting data to our anonymous ftp site:
https://dtcenter.org/community-code/model-evaluation-tools-met/met-
help-desk#ftp
Thanks
John
On Tue, Oct 6, 2020 at 10:56 AM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96971 >
>
> Hi Erica.
>
> I see that you are having trouble getting the statistics that you
would
> like from Point-Stat. I am assigning this ticket to John Halley
Gotway.
> Please allow a few business days for a response.
>
> Julie
>
> On Tue, Oct 6, 2020 at 10:37 AM Erica Burrows - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Oct 06 10:37:13 2020: Request 96971 was acted upon.
> > Transaction: Ticket created by erica.burrows at noaa.gov
> > Queue: met_help
> > Subject: Implementing Climatology Data to Point Stat
> > Owner: Nobody
> > Requestors: erica.burrows at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96971 >
> >
> >
> > To whom this may concern,
> >
> > I am wanting to extend the statistics I currently compute for my
point
> stat
> > processing with no progress. Currently, I am able to produce mean
> > forecast/obs and RMSE but I am also wanting to include the
following:
> >
> > -
> >
> > CNT
> > -
> >
> > Forecast/Observation Standard Deviation
> > -
> >
> > Pearson Correlation Coefficient
> > -
> >
> > Standard deviation of the error
> > -
> >
> > Anomaly Correlation Coefficient
> > -
> >
> > CST
> > -
> >
> > Base Rate
> > -
> >
> > Accuracy
> > -
> >
> > Frequency Bias
> > -
> >
> > Heidke Skill Score
> >
> >
> > At the moment, those statistics listed are all reporting either
NA, 0,
> or 1
> > (depending on the stat). I understand I need to include thresholds
which
> I
> > have tried both within my point stat config and outside (but
including
> when
> > running stat analysis) with no luck.
> >
> > I thought perhaps my problem was my climo_mean attributes were
empty,
> > however when I try to list txt files it does not accept them
(which I
> > assumed would happen). I then tried to convert these files using
> > ascii2nc with no luck. I am assuming the error is due to the fact
each
> > climo file uses julian day rather than an individual date (to
avoid 35
> > files having the same information, as these are applicable to 35
years
> > worth of data). My txt files have the following format:
> > *Message_Type Station_ID Valid_Time Lat Lon Elevation
Variable_Name Level
> > Height QC_String Observation_Value\n*
> >
> > If I do not need to provide climatology values to receive those
stats
> > listed, then my issue must be with how I am formatting my point
stat and
> > stat analysis calls which is as follows:
> >
> > *for proj in {6..384..6}*
> >
> >
> > *do PR=`echo $(printf %03d ${proj})`*
> >
> > * ${metDIR}/point_stat ${fcstdir}/${WSFILE}_pr${PR}.nc
${VALIDFILE}
> > PointStatConfig_ws -v 4 -outdir ${writeDIR} -obs_valid_beg
> > ${VALIDDATE}_000000 -obs_valid_end ${VALIDDATE}_235959 *
> > *done*
> >
> > *fileLIST=($(ls
${outdir}/point_stat_*_${currDAY}_*0000V_mpr.txt))*
> >
> >
> > *${metDIR}/stat_analysis -lookin ${fileLIST} -config
> > "STATAnalysisConfig_ws" -v 0 -vx_mask FULL -job aggregate_stat
-line_type
> > MPR -out_line_type CNT -out_stat
> > "${outdir}/${currDAY}cy${CY}_mpr_to_cnt.stat" -by FCST_VAR,OBS_SID
> -set_hdr
> > VX_MASK OBS_SID -set_hdr DESC CASE -out
> > "${outdir}/${YYYY}${MM}${DD}_v${currDAY}_ws_cnt.stat"*
> >
> > * ${metDIR}/stat_analysis -lookin ${fileLIST} -config
> > "STATAnalysisConfig_ws" -v 0 -vx_mask FULL -job aggregate_stat
-line_type
> > MPR -out_line_type MCTS -out_stat
> > "${outdir}/${currDAY}cy${CY}_mpr_to_mcts.stat" -by
FCST_VAR,OBS_SID
> > -set_hdr VX_MASK OBS_SID -set_hdr DESC CASE -out_thresh
> > ge4.12,ge10.29,ge18.01,ge25.72 -out
> > "${outdir}/${YYYY}${MM}${DD}_v${currDAY}_ws_mcts.stat" *
> >
> > If it is more beneficial, I have provided the following
attachments:
> > 1. Current working point stat config file (PointStatConfig_ws)
> > 2. Testing point stat config file for climo
(PointStatConfig_ws.test)
> > 3. Stat analysis config (STATAnalysisConfig_ws)
> > 4. 4 fcst data files (ws_YYYYMMDDCY_c00_prPPP)
> > 5. 1 obs data file (YYYYMMDD_CY:00:00) which used ascii2nc to be
created
> > 6. 1 climo file (jdDDD_climatology)
> >
> > Hopefully I did not provide too much information on what I have
been
> trying
> > to accomplish. But basically, I am trying to utilize point_stat to
> compute
> > those stats listed for 36 core airports across CONUS, and at this
point I
> > am at a loss as to why some work and others do not. Any and all
guidance
> is
> > greatly appreciated.
> >
> > Thanks!
> >
> > Erica Burrows
> > Atmospheric Scientist
> > INNOVIM/Climate Prediction Center
> > NOAA Center for Weather and Climate Prediction
> > 5830 University Research Court
> > College Park, MD 20740
> >
> > Work:
> > Cell: 818-925-5795
> > Email: erica.burrows at noaa.gov
> > *Innovim LLC*
> >
> >
>
> --
> Julie Prestopnik (she/her/hers)
> Software Engineer
> National Center for Atmospheric Research
> Research Applications Laboratory
> Email: jpresto at ucar.edu
>
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
------------------------------------------------
Subject: Implementing Climatology Data to Point Stat
From: Erica Burrows - NOAA Affiliate
Time: Thu Oct 08 10:50:45 2020
Hello John,
Not a problem! Sorry for the lengthy email, I was trying to be
thorough in
explaining what I was trying to do. Unfortunately, at the moment I am
using
one of the CPC work stations as I am using gefs v10. Once I have all
my
stats working I plan to switch over to WCOSS and utilize gefs v12.
I have tarred 3 days each going out to 384 hours. Let me know if you
want
more!
In case I did not mention it before, I am computing daily stats so
point
stat files with proj 00, 06, 12, 18 will go to day 1 and proj 24, 30,
36,
42 will go to day 2, and so on until proj 384.
Erica Burrows
Atmospheric Scientist
INNOVIM/Climate Prediction Center
NOAA Center for Weather and Climate Prediction
5830 University Research Court
College Park, MD 20740
Work:
Cell: 818-925-5795
Email: erica.burrows at noaa.gov
*Innovim LLC*
On Thu, Oct 8, 2020 at 11:57 AM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Hello Erica,
>
> My apologies for the delay in responding to your question.
>
> I see that you're running Point-Stat and would like to increase the
number
> of statistics in your evaluation. First, let me point out that the
> output of Point-Stat is controlled by turning on/off line types. You
don't
> control output statistic by statistic. Instead, you configure the
> "output_flag" settings in the config file to request groups of
statistics
> by their line type.
>
> Looking in the Point-Stat config files you sent, I see that you're
only
> requesting the MPR, i.e. matched pair, output line type.
>
> And from your description, I see why. You'd like to compute stats
> separately for each of 36 airports stations. So rather than
computing a
> spatial aggregation with Point-Stat, you'd like to run Stat-Analysis
to
> compute stats through time for each station.
>
> So I suspect the real question is, what jobs should you be running
in
> STAT-Analysis?
>
> I see that you sent some config files but no actual output from
Point-Stat.
> If you send that I could take a look and recommend some STAT-
Analysis jobs.
> You could either tar them up and send them or point me to them on a
NOAA
> machine, like WCOSS, jet, or theia (can’t remember it successor
right now).
>
> Here’s instructions for posting data to our anonymous ftp site:
>
>
> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
help-desk#ftp
>
> Thanks
> John
>
>
>
>
>
> On Tue, Oct 6, 2020 at 10:56 AM Julie Prestopnik via RT
<met_help at ucar.edu
> >
> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96971 >
> >
> > Hi Erica.
> >
> > I see that you are having trouble getting the statistics that you
would
> > like from Point-Stat. I am assigning this ticket to John Halley
Gotway.
> > Please allow a few business days for a response.
> >
> > Julie
> >
> > On Tue, Oct 6, 2020 at 10:37 AM Erica Burrows - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Oct 06 10:37:13 2020: Request 96971 was acted upon.
> > > Transaction: Ticket created by erica.burrows at noaa.gov
> > > Queue: met_help
> > > Subject: Implementing Climatology Data to Point Stat
> > > Owner: Nobody
> > > Requestors: erica.burrows at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96971
> >
> > >
> > >
> > > To whom this may concern,
> > >
> > > I am wanting to extend the statistics I currently compute for my
point
> > stat
> > > processing with no progress. Currently, I am able to produce
mean
> > > forecast/obs and RMSE but I am also wanting to include the
following:
> > >
> > > -
> > >
> > > CNT
> > > -
> > >
> > > Forecast/Observation Standard Deviation
> > > -
> > >
> > > Pearson Correlation Coefficient
> > > -
> > >
> > > Standard deviation of the error
> > > -
> > >
> > > Anomaly Correlation Coefficient
> > > -
> > >
> > > CST
> > > -
> > >
> > > Base Rate
> > > -
> > >
> > > Accuracy
> > > -
> > >
> > > Frequency Bias
> > > -
> > >
> > > Heidke Skill Score
> > >
> > >
> > > At the moment, those statistics listed are all reporting either
NA, 0,
> > or 1
> > > (depending on the stat). I understand I need to include
thresholds
> which
> > I
> > > have tried both within my point stat config and outside (but
including
> > when
> > > running stat analysis) with no luck.
> > >
> > > I thought perhaps my problem was my climo_mean attributes were
empty,
> > > however when I try to list txt files it does not accept them
(which I
> > > assumed would happen). I then tried to convert these files using
> > > ascii2nc with no luck. I am assuming the error is due to the
fact each
> > > climo file uses julian day rather than an individual date (to
avoid 35
> > > files having the same information, as these are applicable to 35
years
> > > worth of data). My txt files have the following format:
> > > *Message_Type Station_ID Valid_Time Lat Lon Elevation
Variable_Name
> Level
> > > Height QC_String Observation_Value\n*
> > >
> > > If I do not need to provide climatology values to receive those
stats
> > > listed, then my issue must be with how I am formatting my point
stat
> and
> > > stat analysis calls which is as follows:
> > >
> > > *for proj in {6..384..6}*
> > >
> > >
> > > *do PR=`echo $(printf %03d ${proj})`*
> > >
> > > * ${metDIR}/point_stat ${fcstdir}/${WSFILE}_pr${PR}.nc
${VALIDFILE}
> > > PointStatConfig_ws -v 4 -outdir ${writeDIR} -obs_valid_beg
> > > ${VALIDDATE}_000000 -obs_valid_end ${VALIDDATE}_235959 *
> > > *done*
> > >
> > > *fileLIST=($(ls
${outdir}/point_stat_*_${currDAY}_*0000V_mpr.txt))*
> > >
> > >
> > > *${metDIR}/stat_analysis -lookin ${fileLIST} -config
> > > "STATAnalysisConfig_ws" -v 0 -vx_mask FULL -job aggregate_stat
> -line_type
> > > MPR -out_line_type CNT -out_stat
> > > "${outdir}/${currDAY}cy${CY}_mpr_to_cnt.stat" -by
FCST_VAR,OBS_SID
> > -set_hdr
> > > VX_MASK OBS_SID -set_hdr DESC CASE -out
> > > "${outdir}/${YYYY}${MM}${DD}_v${currDAY}_ws_cnt.stat"*
> > >
> > > * ${metDIR}/stat_analysis -lookin ${fileLIST} -config
> > > "STATAnalysisConfig_ws" -v 0 -vx_mask FULL -job aggregate_stat
> -line_type
> > > MPR -out_line_type MCTS -out_stat
> > > "${outdir}/${currDAY}cy${CY}_mpr_to_mcts.stat" -by
FCST_VAR,OBS_SID
> > > -set_hdr VX_MASK OBS_SID -set_hdr DESC CASE -out_thresh
> > > ge4.12,ge10.29,ge18.01,ge25.72 -out
> > > "${outdir}/${YYYY}${MM}${DD}_v${currDAY}_ws_mcts.stat" *
> > >
> > > If it is more beneficial, I have provided the following
attachments:
> > > 1. Current working point stat config file (PointStatConfig_ws)
> > > 2. Testing point stat config file for climo
(PointStatConfig_ws.test)
> > > 3. Stat analysis config (STATAnalysisConfig_ws)
> > > 4. 4 fcst data files (ws_YYYYMMDDCY_c00_prPPP)
> > > 5. 1 obs data file (YYYYMMDD_CY:00:00) which used ascii2nc to be
> created
> > > 6. 1 climo file (jdDDD_climatology)
> > >
> > > Hopefully I did not provide too much information on what I have
been
> > trying
> > > to accomplish. But basically, I am trying to utilize point_stat
to
> > compute
> > > those stats listed for 36 core airports across CONUS, and at
this
> point I
> > > am at a loss as to why some work and others do not. Any and all
> guidance
> > is
> > > greatly appreciated.
> > >
> > > Thanks!
> > >
> > > Erica Burrows
> > > Atmospheric Scientist
> > > INNOVIM/Climate Prediction Center
> > > NOAA Center for Weather and Climate Prediction
> > > 5830 University Research Court
> > > College Park, MD 20740
> > >
> > > Work:
> > > Cell: 818-925-5795
> > > Email: erica.burrows at noaa.gov
> > > *Innovim LLC*
> > >
> > >
> >
> > --
> > Julie Prestopnik (she/her/hers)
> > Software Engineer
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > Email: jpresto at ucar.edu
> >
> > My working day may not be your working day. Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.
> >
> >
>
>
------------------------------------------------
Subject: Implementing Climatology Data to Point Stat
From: John Halley Gotway
Time: Thu Oct 08 21:32:54 2020
Erica,
Thanks for sending the sample files. Some recommendations are listed
below:
(1) Reconfigure Point-Stat by changing "mpr = BOTH;" to "mpr = STAT;".
Currently you're writing the same output to .stat and _mpr.txt output
files. So that's double the amount of output. Stat-Analysis reads the
.stat
files, not the _mpr.txt files. So I'd recommend only writing those.
I think your first stat_analysis jobs looks great, converting MPR to
CNT.
I ran essentially the same one:
/Volumes/d1/projects/MET/MET_development/MET-
main_v9.1/met/bin/stat_analysis
-lookin data -job aggregate_stat -line_type MPR -out_line_type CNT -by
OBS_SID,FCST_VAR -out_stat mpr_to_cnt.stat -set_hdr VX_MASK OBS_SID
And that produced output for 7020 unique cases (OBS_SID and FCST_VAR
combos).
(2) I see that your second job requests MCTS output which is multi-
category
contingency table output. I suspect that you really want CTS instead.
It
looks like you're interested in the performance for 4 wind speed
thresholds: ge4.12,ge10.29,ge18.01,ge25.72
Unfortunately, you'll need to run 4 separate Stat-Analysis jobs for
that...
one for each threshold:
For example...
/Volumes/d1/projects/MET/MET_development/MET-
main_v9.1/met/bin/stat_analysis
-lookin data -job aggregate_stat -line_type MPR -out_line_type CTS -by
OBS_SID,FCST_VAR -out_stat mpr_to_cts_ge4.12.stat -set_hdr VX_MASK
OBS_SID
-out_thresh ge4.12
You could either run 4 separate jobs on the command line... or you
could
group them together into a single Stat-Analysis config file, and then
run
Stat-Analysis once. Something like this...
jobs = [
"-job aggregate_stat -line_type MPR -out_line_type CTS -by
OBS_SID,FCST_VAR -out_stat mpr_to_cts_ge4.12.stat -set_hdr VX_MASK
OBS_SID
-out_thresh ge4.12",
"-job aggregate_stat -line_type MPR -out_line_type CTS -by
OBS_SID,FCST_VAR -out_stat mpr_to_cts_ge10.29.stat -set_hdr VX_MASK
OBS_SID
-out_thresh ge10.29",
"-job aggregate_stat -line_type MPR -out_line_type CTS -by
OBS_SID,FCST_VAR -out_stat mpr_to_cts_ge18.01.stat -set_hdr VX_MASK
OBS_SID
-out_thresh ge18.01",
"-job aggregate_stat -line_type MPR -out_line_type CTS -by
OBS_SID,FCST_VAR -out_stat mpr_to_cts_ge25.72.stat -set_hdr VX_MASK
OBS_SID
-out_thresh ge25.72"
];
So I think you have this setup very well. Just change BOTH to STAT...
and
then switch from requesting the MCTS line type with 4 thresholds to
requesting the CTS line type 4 times.
Are there other specific issues or questions you want me to take a
closer
look at?
Thanks,
John
On Thu, Oct 8, 2020 at 10:51 AM Erica Burrows - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96971 >
>
> Hello John,
>
> Not a problem! Sorry for the lengthy email, I was trying to be
thorough in
> explaining what I was trying to do. Unfortunately, at the moment I
am using
> one of the CPC work stations as I am using gefs v10. Once I have all
my
> stats working I plan to switch over to WCOSS and utilize gefs v12.
>
> I have tarred 3 days each going out to 384 hours. Let me know if you
want
> more!
>
> In case I did not mention it before, I am computing daily stats so
point
> stat files with proj 00, 06, 12, 18 will go to day 1 and proj 24,
30, 36,
> 42 will go to day 2, and so on until proj 384.
>
> Erica Burrows
> Atmospheric Scientist
> INNOVIM/Climate Prediction Center
> NOAA Center for Weather and Climate Prediction
> 5830 University Research Court
> College Park, MD 20740
>
> Work:
> Cell: 818-925-5795
> Email: erica.burrows at noaa.gov
> *Innovim LLC*
>
>
> On Thu, Oct 8, 2020 at 11:57 AM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Hello Erica,
> >
> > My apologies for the delay in responding to your question.
> >
> > I see that you're running Point-Stat and would like to increase
the
> number
> > of statistics in your evaluation. First, let me point out that the
> > output of Point-Stat is controlled by turning on/off line types.
You
> don't
> > control output statistic by statistic. Instead, you configure the
> > "output_flag" settings in the config file to request groups of
statistics
> > by their line type.
> >
> > Looking in the Point-Stat config files you sent, I see that you're
only
> > requesting the MPR, i.e. matched pair, output line type.
> >
> > And from your description, I see why. You'd like to compute stats
> > separately for each of 36 airports stations. So rather than
computing a
> > spatial aggregation with Point-Stat, you'd like to run Stat-
Analysis to
> > compute stats through time for each station.
> >
> > So I suspect the real question is, what jobs should you be running
in
> > STAT-Analysis?
> >
> > I see that you sent some config files but no actual output from
> Point-Stat.
> > If you send that I could take a look and recommend some STAT-
Analysis
> jobs.
> > You could either tar them up and send them or point me to them on
a NOAA
> > machine, like WCOSS, jet, or theia (can’t remember it successor
right
> now).
> >
> > Here’s instructions for posting data to our anonymous ftp site:
> >
> >
> >
> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
help-desk#ftp
> >
> > Thanks
> > John
> >
> >
> >
> >
> >
> > On Tue, Oct 6, 2020 at 10:56 AM Julie Prestopnik via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96971 >
> > >
> > > Hi Erica.
> > >
> > > I see that you are having trouble getting the statistics that
you would
> > > like from Point-Stat. I am assigning this ticket to John Halley
> Gotway.
> > > Please allow a few business days for a response.
> > >
> > > Julie
> > >
> > > On Tue, Oct 6, 2020 at 10:37 AM Erica Burrows - NOAA Affiliate
via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Tue Oct 06 10:37:13 2020: Request 96971 was acted upon.
> > > > Transaction: Ticket created by erica.burrows at noaa.gov
> > > > Queue: met_help
> > > > Subject: Implementing Climatology Data to Point Stat
> > > > Owner: Nobody
> > > > Requestors: erica.burrows at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96971
> > >
> > > >
> > > >
> > > > To whom this may concern,
> > > >
> > > > I am wanting to extend the statistics I currently compute for
my
> point
> > > stat
> > > > processing with no progress. Currently, I am able to produce
mean
> > > > forecast/obs and RMSE but I am also wanting to include the
following:
> > > >
> > > > -
> > > >
> > > > CNT
> > > > -
> > > >
> > > > Forecast/Observation Standard Deviation
> > > > -
> > > >
> > > > Pearson Correlation Coefficient
> > > > -
> > > >
> > > > Standard deviation of the error
> > > > -
> > > >
> > > > Anomaly Correlation Coefficient
> > > > -
> > > >
> > > > CST
> > > > -
> > > >
> > > > Base Rate
> > > > -
> > > >
> > > > Accuracy
> > > > -
> > > >
> > > > Frequency Bias
> > > > -
> > > >
> > > > Heidke Skill Score
> > > >
> > > >
> > > > At the moment, those statistics listed are all reporting
either NA,
> 0,
> > > or 1
> > > > (depending on the stat). I understand I need to include
thresholds
> > which
> > > I
> > > > have tried both within my point stat config and outside (but
> including
> > > when
> > > > running stat analysis) with no luck.
> > > >
> > > > I thought perhaps my problem was my climo_mean attributes were
empty,
> > > > however when I try to list txt files it does not accept them
(which I
> > > > assumed would happen). I then tried to convert these files
using
> > > > ascii2nc with no luck. I am assuming the error is due to the
fact
> each
> > > > climo file uses julian day rather than an individual date (to
avoid
> 35
> > > > files having the same information, as these are applicable to
35
> years
> > > > worth of data). My txt files have the following format:
> > > > *Message_Type Station_ID Valid_Time Lat Lon Elevation
Variable_Name
> > Level
> > > > Height QC_String Observation_Value\n*
> > > >
> > > > If I do not need to provide climatology values to receive
those stats
> > > > listed, then my issue must be with how I am formatting my
point stat
> > and
> > > > stat analysis calls which is as follows:
> > > >
> > > > *for proj in {6..384..6}*
> > > >
> > > >
> > > > *do PR=`echo $(printf %03d ${proj})`*
> > > >
> > > > * ${metDIR}/point_stat ${fcstdir}/${WSFILE}_pr${PR}.nc
> ${VALIDFILE}
> > > > PointStatConfig_ws -v 4 -outdir ${writeDIR} -obs_valid_beg
> > > > ${VALIDDATE}_000000 -obs_valid_end ${VALIDDATE}_235959 *
> > > > *done*
> > > >
> > > > *fileLIST=($(ls
${outdir}/point_stat_*_${currDAY}_*0000V_mpr.txt))*
> > > >
> > > >
> > > > *${metDIR}/stat_analysis -lookin ${fileLIST} -config
> > > > "STATAnalysisConfig_ws" -v 0 -vx_mask FULL -job aggregate_stat
> > -line_type
> > > > MPR -out_line_type CNT -out_stat
> > > > "${outdir}/${currDAY}cy${CY}_mpr_to_cnt.stat" -by
FCST_VAR,OBS_SID
> > > -set_hdr
> > > > VX_MASK OBS_SID -set_hdr DESC CASE -out
> > > > "${outdir}/${YYYY}${MM}${DD}_v${currDAY}_ws_cnt.stat"*
> > > >
> > > > * ${metDIR}/stat_analysis -lookin ${fileLIST} -config
> > > > "STATAnalysisConfig_ws" -v 0 -vx_mask FULL -job aggregate_stat
> > -line_type
> > > > MPR -out_line_type MCTS -out_stat
> > > > "${outdir}/${currDAY}cy${CY}_mpr_to_mcts.stat" -by
FCST_VAR,OBS_SID
> > > > -set_hdr VX_MASK OBS_SID -set_hdr DESC CASE -out_thresh
> > > > ge4.12,ge10.29,ge18.01,ge25.72 -out
> > > > "${outdir}/${YYYY}${MM}${DD}_v${currDAY}_ws_mcts.stat" *
> > > >
> > > > If it is more beneficial, I have provided the following
attachments:
> > > > 1. Current working point stat config file (PointStatConfig_ws)
> > > > 2. Testing point stat config file for climo
(PointStatConfig_ws.test)
> > > > 3. Stat analysis config (STATAnalysisConfig_ws)
> > > > 4. 4 fcst data files (ws_YYYYMMDDCY_c00_prPPP)
> > > > 5. 1 obs data file (YYYYMMDD_CY:00:00) which used ascii2nc to
be
> > created
> > > > 6. 1 climo file (jdDDD_climatology)
> > > >
> > > > Hopefully I did not provide too much information on what I
have been
> > > trying
> > > > to accomplish. But basically, I am trying to utilize
point_stat to
> > > compute
> > > > those stats listed for 36 core airports across CONUS, and at
this
> > point I
> > > > am at a loss as to why some work and others do not. Any and
all
> > guidance
> > > is
> > > > greatly appreciated.
> > > >
> > > > Thanks!
> > > >
> > > > Erica Burrows
> > > > Atmospheric Scientist
> > > > INNOVIM/Climate Prediction Center
> > > > NOAA Center for Weather and Climate Prediction
> > > > 5830 University Research Court
> > > > College Park, MD 20740
> > > >
> > > > Work:
> > > > Cell: 818-925-5795
> > > > Email: erica.burrows at noaa.gov
> > > > *Innovim LLC*
> > > >
> > > >
> > >
> > > --
> > > Julie Prestopnik (she/her/hers)
> > > Software Engineer
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > Email: jpresto at ucar.edu
> > >
> > > My working day may not be your working day. Please do not feel
obliged
> > to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Implementing Climatology Data to Point Stat
From: Erica Burrows - NOAA Affiliate
Time: Sat Oct 10 13:44:58 2020
John,
Thank you for help with this thus far. Your changes helped, however my
stats of interest for the cts type (BASER, ACC, FBIAS, and HSS) are
still
always returning as NA. Any idea why this may be occurring? Am I
utilizing
too few files?
Erica Burrows
Atmospheric Scientist
INNOVIM/Climate Prediction Center
NOAA Center for Weather and Climate Prediction
5830 University Research Court
College Park, MD 20740
Work:
Cell: 818-925-5795
Email: erica.burrows at noaa.gov
*Innovim LLC*
On Thu, Oct 8, 2020 at 11:33 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Erica,
>
> Thanks for sending the sample files. Some recommendations are listed
below:
>
> (1) Reconfigure Point-Stat by changing "mpr = BOTH;" to "mpr =
STAT;".
> Currently you're writing the same output to .stat and _mpr.txt
output
> files. So that's double the amount of output. Stat-Analysis reads
the .stat
> files, not the _mpr.txt files. So I'd recommend only writing those.
>
> I think your first stat_analysis jobs looks great, converting MPR to
CNT.
>
> I ran essentially the same one:
>
> /Volumes/d1/projects/MET/MET_development/MET-
main_v9.1/met/bin/stat_analysis
> -lookin data -job aggregate_stat -line_type MPR -out_line_type CNT
-by
> OBS_SID,FCST_VAR -out_stat mpr_to_cnt.stat -set_hdr VX_MASK OBS_SID
>
> And that produced output for 7020 unique cases (OBS_SID and FCST_VAR
> combos).
>
> (2) I see that your second job requests MCTS output which is multi-
category
> contingency table output. I suspect that you really want CTS
instead. It
> looks like you're interested in the performance for 4 wind speed
> thresholds: ge4.12,ge10.29,ge18.01,ge25.72
> Unfortunately, you'll need to run 4 separate Stat-Analysis jobs for
that...
> one for each threshold:
>
> For example...
>
>
> /Volumes/d1/projects/MET/MET_development/MET-
main_v9.1/met/bin/stat_analysis
> -lookin data -job aggregate_stat -line_type MPR -out_line_type CTS
-by
> OBS_SID,FCST_VAR -out_stat mpr_to_cts_ge4.12.stat -set_hdr VX_MASK
OBS_SID
> -out_thresh ge4.12
>
> You could either run 4 separate jobs on the command line... or you
could
> group them together into a single Stat-Analysis config file, and
then run
> Stat-Analysis once. Something like this...
>
> jobs = [
> "-job aggregate_stat -line_type MPR -out_line_type CTS -by
> OBS_SID,FCST_VAR -out_stat mpr_to_cts_ge4.12.stat -set_hdr VX_MASK
OBS_SID
> -out_thresh ge4.12",
> "-job aggregate_stat -line_type MPR -out_line_type CTS -by
> OBS_SID,FCST_VAR -out_stat mpr_to_cts_ge10.29.stat -set_hdr VX_MASK
OBS_SID
> -out_thresh ge10.29",
> "-job aggregate_stat -line_type MPR -out_line_type CTS -by
> OBS_SID,FCST_VAR -out_stat mpr_to_cts_ge18.01.stat -set_hdr VX_MASK
OBS_SID
> -out_thresh ge18.01",
> "-job aggregate_stat -line_type MPR -out_line_type CTS -by
> OBS_SID,FCST_VAR -out_stat mpr_to_cts_ge25.72.stat -set_hdr VX_MASK
OBS_SID
> -out_thresh ge25.72"
> ];
>
> So I think you have this setup very well. Just change BOTH to
STAT... and
> then switch from requesting the MCTS line type with 4 thresholds to
> requesting the CTS line type 4 times.
>
> Are there other specific issues or questions you want me to take a
closer
> look at?
>
> Thanks,
> John
>
>
> On Thu, Oct 8, 2020 at 10:51 AM Erica Burrows - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96971 >
> >
> > Hello John,
> >
> > Not a problem! Sorry for the lengthy email, I was trying to be
thorough
> in
> > explaining what I was trying to do. Unfortunately, at the moment I
am
> using
> > one of the CPC work stations as I am using gefs v10. Once I have
all my
> > stats working I plan to switch over to WCOSS and utilize gefs
v12.
> >
> > I have tarred 3 days each going out to 384 hours. Let me know if
you want
> > more!
> >
> > In case I did not mention it before, I am computing daily stats so
point
> > stat files with proj 00, 06, 12, 18 will go to day 1 and proj 24,
30, 36,
> > 42 will go to day 2, and so on until proj 384.
> >
> > Erica Burrows
> > Atmospheric Scientist
> > INNOVIM/Climate Prediction Center
> > NOAA Center for Weather and Climate Prediction
> > 5830 University Research Court
> > College Park, MD 20740
> >
> > Work:
> > Cell: 818-925-5795
> > Email: erica.burrows at noaa.gov
> > *Innovim LLC*
> >
> >
> > On Thu, Oct 8, 2020 at 11:57 AM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Hello Erica,
> > >
> > > My apologies for the delay in responding to your question.
> > >
> > > I see that you're running Point-Stat and would like to increase
the
> > number
> > > of statistics in your evaluation. First, let me point out that
the
> > > output of Point-Stat is controlled by turning on/off line types.
You
> > don't
> > > control output statistic by statistic. Instead, you configure
the
> > > "output_flag" settings in the config file to request groups of
> statistics
> > > by their line type.
> > >
> > > Looking in the Point-Stat config files you sent, I see that
you're only
> > > requesting the MPR, i.e. matched pair, output line type.
> > >
> > > And from your description, I see why. You'd like to compute
stats
> > > separately for each of 36 airports stations. So rather than
computing a
> > > spatial aggregation with Point-Stat, you'd like to run Stat-
Analysis to
> > > compute stats through time for each station.
> > >
> > > So I suspect the real question is, what jobs should you be
running in
> > > STAT-Analysis?
> > >
> > > I see that you sent some config files but no actual output from
> > Point-Stat.
> > > If you send that I could take a look and recommend some STAT-
Analysis
> > jobs.
> > > You could either tar them up and send them or point me to them
on a
> NOAA
> > > machine, like WCOSS, jet, or theia (can’t remember it successor
right
> > now).
> > >
> > > Here’s instructions for posting data to our anonymous ftp site:
> > >
> > >
> > >
> >
> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
help-desk#ftp
> > >
> > > Thanks
> > > John
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Oct 6, 2020 at 10:56 AM Julie Prestopnik via RT <
> > met_help at ucar.edu
> > > >
> > > wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96971
>
> > > >
> > > > Hi Erica.
> > > >
> > > > I see that you are having trouble getting the statistics that
you
> would
> > > > like from Point-Stat. I am assigning this ticket to John
Halley
> > Gotway.
> > > > Please allow a few business days for a response.
> > > >
> > > > Julie
> > > >
> > > > On Tue, Oct 6, 2020 at 10:37 AM Erica Burrows - NOAA Affiliate
via
> RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Tue Oct 06 10:37:13 2020: Request 96971 was acted upon.
> > > > > Transaction: Ticket created by erica.burrows at noaa.gov
> > > > > Queue: met_help
> > > > > Subject: Implementing Climatology Data to Point Stat
> > > > > Owner: Nobody
> > > > > Requestors: erica.burrows at noaa.gov
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96971
> > > >
> > > > >
> > > > >
> > > > > To whom this may concern,
> > > > >
> > > > > I am wanting to extend the statistics I currently compute
for my
> > point
> > > > stat
> > > > > processing with no progress. Currently, I am able to produce
mean
> > > > > forecast/obs and RMSE but I am also wanting to include the
> following:
> > > > >
> > > > > -
> > > > >
> > > > > CNT
> > > > > -
> > > > >
> > > > > Forecast/Observation Standard Deviation
> > > > > -
> > > > >
> > > > > Pearson Correlation Coefficient
> > > > > -
> > > > >
> > > > > Standard deviation of the error
> > > > > -
> > > > >
> > > > > Anomaly Correlation Coefficient
> > > > > -
> > > > >
> > > > > CST
> > > > > -
> > > > >
> > > > > Base Rate
> > > > > -
> > > > >
> > > > > Accuracy
> > > > > -
> > > > >
> > > > > Frequency Bias
> > > > > -
> > > > >
> > > > > Heidke Skill Score
> > > > >
> > > > >
> > > > > At the moment, those statistics listed are all reporting
either NA,
> > 0,
> > > > or 1
> > > > > (depending on the stat). I understand I need to include
thresholds
> > > which
> > > > I
> > > > > have tried both within my point stat config and outside (but
> > including
> > > > when
> > > > > running stat analysis) with no luck.
> > > > >
> > > > > I thought perhaps my problem was my climo_mean attributes
were
> empty,
> > > > > however when I try to list txt files it does not accept them
> (which I
> > > > > assumed would happen). I then tried to convert these files
using
> > > > > ascii2nc with no luck. I am assuming the error is due to the
fact
> > each
> > > > > climo file uses julian day rather than an individual date
(to avoid
> > 35
> > > > > files having the same information, as these are applicable
to 35
> > years
> > > > > worth of data). My txt files have the following format:
> > > > > *Message_Type Station_ID Valid_Time Lat Lon Elevation
Variable_Name
> > > Level
> > > > > Height QC_String Observation_Value\n*
> > > > >
> > > > > If I do not need to provide climatology values to receive
those
> stats
> > > > > listed, then my issue must be with how I am formatting my
point
> stat
> > > and
> > > > > stat analysis calls which is as follows:
> > > > >
> > > > > *for proj in {6..384..6}*
> > > > >
> > > > >
> > > > > *do PR=`echo $(printf %03d ${proj})`*
> > > > >
> > > > > * ${metDIR}/point_stat ${fcstdir}/${WSFILE}_pr${PR}.nc
> > ${VALIDFILE}
> > > > > PointStatConfig_ws -v 4 -outdir ${writeDIR} -obs_valid_beg
> > > > > ${VALIDDATE}_000000 -obs_valid_end ${VALIDDATE}_235959 *
> > > > > *done*
> > > > >
> > > > > *fileLIST=($(ls
${outdir}/point_stat_*_${currDAY}_*0000V_mpr.txt))*
> > > > >
> > > > >
> > > > > *${metDIR}/stat_analysis -lookin ${fileLIST} -config
> > > > > "STATAnalysisConfig_ws" -v 0 -vx_mask FULL -job
aggregate_stat
> > > -line_type
> > > > > MPR -out_line_type CNT -out_stat
> > > > > "${outdir}/${currDAY}cy${CY}_mpr_to_cnt.stat" -by
FCST_VAR,OBS_SID
> > > > -set_hdr
> > > > > VX_MASK OBS_SID -set_hdr DESC CASE -out
> > > > > "${outdir}/${YYYY}${MM}${DD}_v${currDAY}_ws_cnt.stat"*
> > > > >
> > > > > * ${metDIR}/stat_analysis -lookin ${fileLIST} -config
> > > > > "STATAnalysisConfig_ws" -v 0 -vx_mask FULL -job
aggregate_stat
> > > -line_type
> > > > > MPR -out_line_type MCTS -out_stat
> > > > > "${outdir}/${currDAY}cy${CY}_mpr_to_mcts.stat" -by
FCST_VAR,OBS_SID
> > > > > -set_hdr VX_MASK OBS_SID -set_hdr DESC CASE -out_thresh
> > > > > ge4.12,ge10.29,ge18.01,ge25.72 -out
> > > > > "${outdir}/${YYYY}${MM}${DD}_v${currDAY}_ws_mcts.stat" *
> > > > >
> > > > > If it is more beneficial, I have provided the following
> attachments:
> > > > > 1. Current working point stat config file
(PointStatConfig_ws)
> > > > > 2. Testing point stat config file for climo
> (PointStatConfig_ws.test)
> > > > > 3. Stat analysis config (STATAnalysisConfig_ws)
> > > > > 4. 4 fcst data files (ws_YYYYMMDDCY_c00_prPPP)
> > > > > 5. 1 obs data file (YYYYMMDD_CY:00:00) which used ascii2nc
to be
> > > created
> > > > > 6. 1 climo file (jdDDD_climatology)
> > > > >
> > > > > Hopefully I did not provide too much information on what I
have
> been
> > > > trying
> > > > > to accomplish. But basically, I am trying to utilize
point_stat to
> > > > compute
> > > > > those stats listed for 36 core airports across CONUS, and at
this
> > > point I
> > > > > am at a loss as to why some work and others do not. Any and
all
> > > guidance
> > > > is
> > > > > greatly appreciated.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Erica Burrows
> > > > > Atmospheric Scientist
> > > > > INNOVIM/Climate Prediction Center
> > > > > NOAA Center for Weather and Climate Prediction
> > > > > 5830 University Research Court
> > > > > College Park, MD 20740
> > > > >
> > > > > Work:
> > > > > Cell: 818-925-5795
> > > > > Email: erica.burrows at noaa.gov
> > > > > *Innovim LLC*
> > > > >
> > > > >
> > > >
> > > > --
> > > > Julie Prestopnik (she/her/hers)
> > > > Software Engineer
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > Email: jpresto at ucar.edu
> > > >
> > > > My working day may not be your working day. Please do not
feel
> obliged
> > > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Implementing Climatology Data to Point Stat
From: John Halley Gotway
Time: Mon Oct 12 12:24:29 2020
Erica,
There is something else going on here that we need to figure out.
Let's focus on the base rate (BASER) for example. The base rate is the
proportion of the matched pairs at which the event is occurring in the
observation data. And "the event" is defined by the observation
threshold.
I do not think it's possible for the base rate to have a value of NA.
The
only way it could be NA is if there are 0 matched pairs, which would
result
in BASER = 0/0 = NA. However, if there are 0 matched pairs, then the
MET
tools should not even write an output line in that case.
If possible, can you please send me data to replicate the computed
BASER =
NA value?
Thanks,
John
On Sat, Oct 10, 2020 at 1:45 PM Erica Burrows - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96971 >
>
> John,
>
> Thank you for help with this thus far. Your changes helped, however
my
> stats of interest for the cts type (BASER, ACC, FBIAS, and HSS) are
still
> always returning as NA. Any idea why this may be occurring? Am I
utilizing
> too few files?
>
> Erica Burrows
> Atmospheric Scientist
> INNOVIM/Climate Prediction Center
> NOAA Center for Weather and Climate Prediction
> 5830 University Research Court
> College Park, MD 20740
>
> Work:
> Cell: 818-925-5795
> Email: erica.burrows at noaa.gov
> *Innovim LLC*
>
>
> On Thu, Oct 8, 2020 at 11:33 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Erica,
> >
> > Thanks for sending the sample files. Some recommendations are
listed
> below:
> >
> > (1) Reconfigure Point-Stat by changing "mpr = BOTH;" to "mpr =
STAT;".
> > Currently you're writing the same output to .stat and _mpr.txt
output
> > files. So that's double the amount of output. Stat-Analysis reads
the
> .stat
> > files, not the _mpr.txt files. So I'd recommend only writing
those.
> >
> > I think your first stat_analysis jobs looks great, converting MPR
to CNT.
> >
> > I ran essentially the same one:
> >
> >
> /Volumes/d1/projects/MET/MET_development/MET-
main_v9.1/met/bin/stat_analysis
> > -lookin data -job aggregate_stat -line_type MPR -out_line_type CNT
-by
> > OBS_SID,FCST_VAR -out_stat mpr_to_cnt.stat -set_hdr VX_MASK
OBS_SID
> >
> > And that produced output for 7020 unique cases (OBS_SID and
FCST_VAR
> > combos).
> >
> > (2) I see that your second job requests MCTS output which is
> multi-category
> > contingency table output. I suspect that you really want CTS
instead. It
> > looks like you're interested in the performance for 4 wind speed
> > thresholds: ge4.12,ge10.29,ge18.01,ge25.72
> > Unfortunately, you'll need to run 4 separate Stat-Analysis jobs
for
> that...
> > one for each threshold:
> >
> > For example...
> >
> >
> >
> /Volumes/d1/projects/MET/MET_development/MET-
main_v9.1/met/bin/stat_analysis
> > -lookin data -job aggregate_stat -line_type MPR -out_line_type CTS
-by
> > OBS_SID,FCST_VAR -out_stat mpr_to_cts_ge4.12.stat -set_hdr VX_MASK
> OBS_SID
> > -out_thresh ge4.12
> >
> > You could either run 4 separate jobs on the command line... or you
could
> > group them together into a single Stat-Analysis config file, and
then run
> > Stat-Analysis once. Something like this...
> >
> > jobs = [
> > "-job aggregate_stat -line_type MPR -out_line_type CTS -by
> > OBS_SID,FCST_VAR -out_stat mpr_to_cts_ge4.12.stat -set_hdr VX_MASK
> OBS_SID
> > -out_thresh ge4.12",
> > "-job aggregate_stat -line_type MPR -out_line_type CTS -by
> > OBS_SID,FCST_VAR -out_stat mpr_to_cts_ge10.29.stat -set_hdr
VX_MASK
> OBS_SID
> > -out_thresh ge10.29",
> > "-job aggregate_stat -line_type MPR -out_line_type CTS -by
> > OBS_SID,FCST_VAR -out_stat mpr_to_cts_ge18.01.stat -set_hdr
VX_MASK
> OBS_SID
> > -out_thresh ge18.01",
> > "-job aggregate_stat -line_type MPR -out_line_type CTS -by
> > OBS_SID,FCST_VAR -out_stat mpr_to_cts_ge25.72.stat -set_hdr
VX_MASK
> OBS_SID
> > -out_thresh ge25.72"
> > ];
> >
> > So I think you have this setup very well. Just change BOTH to
STAT... and
> > then switch from requesting the MCTS line type with 4 thresholds
to
> > requesting the CTS line type 4 times.
> >
> > Are there other specific issues or questions you want me to take a
closer
> > look at?
> >
> > Thanks,
> > John
> >
> >
> > On Thu, Oct 8, 2020 at 10:51 AM Erica Burrows - NOAA Affiliate via
RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96971 >
> > >
> > > Hello John,
> > >
> > > Not a problem! Sorry for the lengthy email, I was trying to be
thorough
> > in
> > > explaining what I was trying to do. Unfortunately, at the moment
I am
> > using
> > > one of the CPC work stations as I am using gefs v10. Once I have
all my
> > > stats working I plan to switch over to WCOSS and utilize gefs
v12.
> > >
> > > I have tarred 3 days each going out to 384 hours. Let me know if
you
> want
> > > more!
> > >
> > > In case I did not mention it before, I am computing daily stats
so
> point
> > > stat files with proj 00, 06, 12, 18 will go to day 1 and proj
24, 30,
> 36,
> > > 42 will go to day 2, and so on until proj 384.
> > >
> > > Erica Burrows
> > > Atmospheric Scientist
> > > INNOVIM/Climate Prediction Center
> > > NOAA Center for Weather and Climate Prediction
> > > 5830 University Research Court
> > > College Park, MD 20740
> > >
> > > Work:
> > > Cell: 818-925-5795
> > > Email: erica.burrows at noaa.gov
> > > *Innovim LLC*
> > >
> > >
> > > On Thu, Oct 8, 2020 at 11:57 AM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hello Erica,
> > > >
> > > > My apologies for the delay in responding to your question.
> > > >
> > > > I see that you're running Point-Stat and would like to
increase the
> > > number
> > > > of statistics in your evaluation. First, let me point out that
the
> > > > output of Point-Stat is controlled by turning on/off line
types. You
> > > don't
> > > > control output statistic by statistic. Instead, you configure
the
> > > > "output_flag" settings in the config file to request groups of
> > statistics
> > > > by their line type.
> > > >
> > > > Looking in the Point-Stat config files you sent, I see that
you're
> only
> > > > requesting the MPR, i.e. matched pair, output line type.
> > > >
> > > > And from your description, I see why. You'd like to compute
stats
> > > > separately for each of 36 airports stations. So rather than
> computing a
> > > > spatial aggregation with Point-Stat, you'd like to run Stat-
Analysis
> to
> > > > compute stats through time for each station.
> > > >
> > > > So I suspect the real question is, what jobs should you be
running in
> > > > STAT-Analysis?
> > > >
> > > > I see that you sent some config files but no actual output
from
> > > Point-Stat.
> > > > If you send that I could take a look and recommend some STAT-
Analysis
> > > jobs.
> > > > You could either tar them up and send them or point me to them
on a
> > NOAA
> > > > machine, like WCOSS, jet, or theia (can’t remember it
successor right
> > > now).
> > > >
> > > > Here’s instructions for posting data to our anonymous ftp
site:
> > > >
> > > >
> > > >
> > >
> >
> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
help-desk#ftp
> > > >
> > > > Thanks
> > > > John
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Oct 6, 2020 at 10:56 AM Julie Prestopnik via RT <
> > > met_help at ucar.edu
> > > > >
> > > > wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96971 >
> > > > >
> > > > > Hi Erica.
> > > > >
> > > > > I see that you are having trouble getting the statistics
that you
> > would
> > > > > like from Point-Stat. I am assigning this ticket to John
Halley
> > > Gotway.
> > > > > Please allow a few business days for a response.
> > > > >
> > > > > Julie
> > > > >
> > > > > On Tue, Oct 6, 2020 at 10:37 AM Erica Burrows - NOAA
Affiliate via
> > RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > Tue Oct 06 10:37:13 2020: Request 96971 was acted upon.
> > > > > > Transaction: Ticket created by erica.burrows at noaa.gov
> > > > > > Queue: met_help
> > > > > > Subject: Implementing Climatology Data to Point Stat
> > > > > > Owner: Nobody
> > > > > > Requestors: erica.burrows at noaa.gov
> > > > > > Status: new
> > > > > > Ticket <URL:
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96971
> > > > >
> > > > > >
> > > > > >
> > > > > > To whom this may concern,
> > > > > >
> > > > > > I am wanting to extend the statistics I currently compute
for my
> > > point
> > > > > stat
> > > > > > processing with no progress. Currently, I am able to
produce mean
> > > > > > forecast/obs and RMSE but I am also wanting to include the
> > following:
> > > > > >
> > > > > > -
> > > > > >
> > > > > > CNT
> > > > > > -
> > > > > >
> > > > > > Forecast/Observation Standard Deviation
> > > > > > -
> > > > > >
> > > > > > Pearson Correlation Coefficient
> > > > > > -
> > > > > >
> > > > > > Standard deviation of the error
> > > > > > -
> > > > > >
> > > > > > Anomaly Correlation Coefficient
> > > > > > -
> > > > > >
> > > > > > CST
> > > > > > -
> > > > > >
> > > > > > Base Rate
> > > > > > -
> > > > > >
> > > > > > Accuracy
> > > > > > -
> > > > > >
> > > > > > Frequency Bias
> > > > > > -
> > > > > >
> > > > > > Heidke Skill Score
> > > > > >
> > > > > >
> > > > > > At the moment, those statistics listed are all reporting
either
> NA,
> > > 0,
> > > > > or 1
> > > > > > (depending on the stat). I understand I need to include
> thresholds
> > > > which
> > > > > I
> > > > > > have tried both within my point stat config and outside
(but
> > > including
> > > > > when
> > > > > > running stat analysis) with no luck.
> > > > > >
> > > > > > I thought perhaps my problem was my climo_mean attributes
were
> > empty,
> > > > > > however when I try to list txt files it does not accept
them
> > (which I
> > > > > > assumed would happen). I then tried to convert these files
using
> > > > > > ascii2nc with no luck. I am assuming the error is due to
the fact
> > > each
> > > > > > climo file uses julian day rather than an individual date
(to
> avoid
> > > 35
> > > > > > files having the same information, as these are applicable
to 35
> > > years
> > > > > > worth of data). My txt files have the following format:
> > > > > > *Message_Type Station_ID Valid_Time Lat Lon Elevation
> Variable_Name
> > > > Level
> > > > > > Height QC_String Observation_Value\n*
> > > > > >
> > > > > > If I do not need to provide climatology values to receive
those
> > stats
> > > > > > listed, then my issue must be with how I am formatting my
point
> > stat
> > > > and
> > > > > > stat analysis calls which is as follows:
> > > > > >
> > > > > > *for proj in {6..384..6}*
> > > > > >
> > > > > >
> > > > > > *do PR=`echo $(printf %03d ${proj})`*
> > > > > >
> > > > > > * ${metDIR}/point_stat ${fcstdir}/${WSFILE}_pr${PR}.nc
> > > ${VALIDFILE}
> > > > > > PointStatConfig_ws -v 4 -outdir ${writeDIR} -obs_valid_beg
> > > > > > ${VALIDDATE}_000000 -obs_valid_end ${VALIDDATE}_235959 *
> > > > > > *done*
> > > > > >
> > > > > > *fileLIST=($(ls
> ${outdir}/point_stat_*_${currDAY}_*0000V_mpr.txt))*
> > > > > >
> > > > > >
> > > > > > *${metDIR}/stat_analysis -lookin ${fileLIST} -config
> > > > > > "STATAnalysisConfig_ws" -v 0 -vx_mask FULL -job
aggregate_stat
> > > > -line_type
> > > > > > MPR -out_line_type CNT -out_stat
> > > > > > "${outdir}/${currDAY}cy${CY}_mpr_to_cnt.stat" -by
> FCST_VAR,OBS_SID
> > > > > -set_hdr
> > > > > > VX_MASK OBS_SID -set_hdr DESC CASE -out
> > > > > > "${outdir}/${YYYY}${MM}${DD}_v${currDAY}_ws_cnt.stat"*
> > > > > >
> > > > > > * ${metDIR}/stat_analysis -lookin ${fileLIST} -config
> > > > > > "STATAnalysisConfig_ws" -v 0 -vx_mask FULL -job
aggregate_stat
> > > > -line_type
> > > > > > MPR -out_line_type MCTS -out_stat
> > > > > > "${outdir}/${currDAY}cy${CY}_mpr_to_mcts.stat" -by
> FCST_VAR,OBS_SID
> > > > > > -set_hdr VX_MASK OBS_SID -set_hdr DESC CASE -out_thresh
> > > > > > ge4.12,ge10.29,ge18.01,ge25.72 -out
> > > > > > "${outdir}/${YYYY}${MM}${DD}_v${currDAY}_ws_mcts.stat" *
> > > > > >
> > > > > > If it is more beneficial, I have provided the following
> > attachments:
> > > > > > 1. Current working point stat config file
(PointStatConfig_ws)
> > > > > > 2. Testing point stat config file for climo
> > (PointStatConfig_ws.test)
> > > > > > 3. Stat analysis config (STATAnalysisConfig_ws)
> > > > > > 4. 4 fcst data files (ws_YYYYMMDDCY_c00_prPPP)
> > > > > > 5. 1 obs data file (YYYYMMDD_CY:00:00) which used ascii2nc
to be
> > > > created
> > > > > > 6. 1 climo file (jdDDD_climatology)
> > > > > >
> > > > > > Hopefully I did not provide too much information on what I
have
> > been
> > > > > trying
> > > > > > to accomplish. But basically, I am trying to utilize
point_stat
> to
> > > > > compute
> > > > > > those stats listed for 36 core airports across CONUS, and
at this
> > > > point I
> > > > > > am at a loss as to why some work and others do not. Any
and all
> > > > guidance
> > > > > is
> > > > > > greatly appreciated.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Erica Burrows
> > > > > > Atmospheric Scientist
> > > > > > INNOVIM/Climate Prediction Center
> > > > > > NOAA Center for Weather and Climate Prediction
> > > > > > 5830 University Research Court
> > > > > > College Park, MD 20740
> > > > > >
> > > > > > Work:
> > > > > > Cell: 818-925-5795
> > > > > > Email: erica.burrows at noaa.gov
> > > > > > *Innovim LLC*
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Julie Prestopnik (she/her/hers)
> > > > > Software Engineer
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > Email: jpresto at ucar.edu
> > > > >
> > > > > My working day may not be your working day. Please do not
feel
> > obliged
> > > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: Implementing Climatology Data to Point Stat
From: Erica Burrows - NOAA Affiliate
Time: Mon Oct 12 14:34:03 2020
John,
It is occurring within all my files, the ones I previously sent should
return the same result. Or are you requesting the new point stat files
after your previous suggestion?
Thanks,
Erica Burrows
Atmospheric Scientist
INNOVIM/Climate Prediction Center
NOAA Center for Weather and Climate Prediction
5830 University Research Court
College Park, MD 20740
Work:
Cell: 818-925-5795
Email: erica.burrows at noaa.gov
Innovim LLC
> On Oct 12, 2020, at 2:24 PM, John Halley Gotway via RT
<met_help at ucar.edu> wrote:
>
> Erica,
>
> There is something else going on here that we need to figure out.
>
> Let's focus on the base rate (BASER) for example. The base rate is
the
> proportion of the matched pairs at which the event is occurring in
the
> observation data. And "the event" is defined by the observation
threshold.
> I do not think it's possible for the base rate to have a value of
NA. The
> only way it could be NA is if there are 0 matched pairs, which would
result
> in BASER = 0/0 = NA. However, if there are 0 matched pairs, then the
MET
> tools should not even write an output line in that case.
>
> If possible, can you please send me data to replicate the computed
BASER =
> NA value?
>
> Thanks,
> John
>
>> On Sat, Oct 10, 2020 at 1:45 PM Erica Burrows - NOAA Affiliate via
RT <
>> met_help at ucar.edu> wrote:
>>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96971 >
>>
>> John,
>>
>> Thank you for help with this thus far. Your changes helped, however
my
>> stats of interest for the cts type (BASER, ACC, FBIAS, and HSS) are
still
>> always returning as NA. Any idea why this may be occurring? Am I
utilizing
>> too few files?
>>
>> Erica Burrows
>> Atmospheric Scientist
>> INNOVIM/Climate Prediction Center
>> NOAA Center for Weather and Climate Prediction
>> 5830 University Research Court
>> College Park, MD 20740
>>
>> Work:
>> Cell: 818-925-5795
>> Email: erica.burrows at noaa.gov
>> *Innovim LLC*
>>
>>
>> On Thu, Oct 8, 2020 at 11:33 PM John Halley Gotway via RT <
>> met_help at ucar.edu>
>> wrote:
>>
>>> Erica,
>>>
>>> Thanks for sending the sample files. Some recommendations are
listed
>> below:
>>>
>>> (1) Reconfigure Point-Stat by changing "mpr = BOTH;" to "mpr =
STAT;".
>>> Currently you're writing the same output to .stat and _mpr.txt
output
>>> files. So that's double the amount of output. Stat-Analysis reads
the
>> .stat
>>> files, not the _mpr.txt files. So I'd recommend only writing
those.
>>>
>>> I think your first stat_analysis jobs looks great, converting MPR
to CNT.
>>>
>>> I ran essentially the same one:
>>>
>>>
>> /Volumes/d1/projects/MET/MET_development/MET-
main_v9.1/met/bin/stat_analysis
>>> -lookin data -job aggregate_stat -line_type MPR -out_line_type CNT
-by
>>> OBS_SID,FCST_VAR -out_stat mpr_to_cnt.stat -set_hdr VX_MASK
OBS_SID
>>>
>>> And that produced output for 7020 unique cases (OBS_SID and
FCST_VAR
>>> combos).
>>>
>>> (2) I see that your second job requests MCTS output which is
>> multi-category
>>> contingency table output. I suspect that you really want CTS
instead. It
>>> looks like you're interested in the performance for 4 wind speed
>>> thresholds: ge4.12,ge10.29,ge18.01,ge25.72
>>> Unfortunately, you'll need to run 4 separate Stat-Analysis jobs
for
>> that...
>>> one for each threshold:
>>>
>>> For example...
>>>
>>>
>>>
>> /Volumes/d1/projects/MET/MET_development/MET-
main_v9.1/met/bin/stat_analysis
>>> -lookin data -job aggregate_stat -line_type MPR -out_line_type CTS
-by
>>> OBS_SID,FCST_VAR -out_stat mpr_to_cts_ge4.12.stat -set_hdr VX_MASK
>> OBS_SID
>>> -out_thresh ge4.12
>>>
>>> You could either run 4 separate jobs on the command line... or you
could
>>> group them together into a single Stat-Analysis config file, and
then run
>>> Stat-Analysis once. Something like this...
>>>
>>> jobs = [
>>> "-job aggregate_stat -line_type MPR -out_line_type CTS -by
>>> OBS_SID,FCST_VAR -out_stat mpr_to_cts_ge4.12.stat -set_hdr VX_MASK
>> OBS_SID
>>> -out_thresh ge4.12",
>>> "-job aggregate_stat -line_type MPR -out_line_type CTS -by
>>> OBS_SID,FCST_VAR -out_stat mpr_to_cts_ge10.29.stat -set_hdr
VX_MASK
>> OBS_SID
>>> -out_thresh ge10.29",
>>> "-job aggregate_stat -line_type MPR -out_line_type CTS -by
>>> OBS_SID,FCST_VAR -out_stat mpr_to_cts_ge18.01.stat -set_hdr
VX_MASK
>> OBS_SID
>>> -out_thresh ge18.01",
>>> "-job aggregate_stat -line_type MPR -out_line_type CTS -by
>>> OBS_SID,FCST_VAR -out_stat mpr_to_cts_ge25.72.stat -set_hdr
VX_MASK
>> OBS_SID
>>> -out_thresh ge25.72"
>>> ];
>>>
>>> So I think you have this setup very well. Just change BOTH to
STAT... and
>>> then switch from requesting the MCTS line type with 4 thresholds
to
>>> requesting the CTS line type 4 times.
>>>
>>> Are there other specific issues or questions you want me to take a
closer
>>> look at?
>>>
>>> Thanks,
>>> John
>>>
>>>
>>> On Thu, Oct 8, 2020 at 10:51 AM Erica Burrows - NOAA Affiliate via
RT <
>>> met_help at ucar.edu> wrote:
>>>
>>>>
>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96971 >
>>>>
>>>> Hello John,
>>>>
>>>> Not a problem! Sorry for the lengthy email, I was trying to be
thorough
>>> in
>>>> explaining what I was trying to do. Unfortunately, at the moment
I am
>>> using
>>>> one of the CPC work stations as I am using gefs v10. Once I have
all my
>>>> stats working I plan to switch over to WCOSS and utilize gefs
v12.
>>>>
>>>> I have tarred 3 days each going out to 384 hours. Let me know if
you
>> want
>>>> more!
>>>>
>>>> In case I did not mention it before, I am computing daily stats
so
>> point
>>>> stat files with proj 00, 06, 12, 18 will go to day 1 and proj 24,
30,
>> 36,
>>>> 42 will go to day 2, and so on until proj 384.
>>>>
>>>> Erica Burrows
>>>> Atmospheric Scientist
>>>> INNOVIM/Climate Prediction Center
>>>> NOAA Center for Weather and Climate Prediction
>>>> 5830 University Research Court
>>>> College Park, MD 20740
>>>>
>>>> Work:
>>>> Cell: 818-925-5795
>>>> Email: erica.burrows at noaa.gov
>>>> *Innovim LLC*
>>>>
>>>>
>>>> On Thu, Oct 8, 2020 at 11:57 AM John Halley Gotway via RT <
>>>> met_help at ucar.edu>
>>>> wrote:
>>>>
>>>>> Hello Erica,
>>>>>
>>>>> My apologies for the delay in responding to your question.
>>>>>
>>>>> I see that you're running Point-Stat and would like to increase
the
>>>> number
>>>>> of statistics in your evaluation. First, let me point out that
the
>>>>> output of Point-Stat is controlled by turning on/off line types.
You
>>>> don't
>>>>> control output statistic by statistic. Instead, you configure
the
>>>>> "output_flag" settings in the config file to request groups of
>>> statistics
>>>>> by their line type.
>>>>>
>>>>> Looking in the Point-Stat config files you sent, I see that
you're
>> only
>>>>> requesting the MPR, i.e. matched pair, output line type.
>>>>>
>>>>> And from your description, I see why. You'd like to compute
stats
>>>>> separately for each of 36 airports stations. So rather than
>> computing a
>>>>> spatial aggregation with Point-Stat, you'd like to run Stat-
Analysis
>> to
>>>>> compute stats through time for each station.
>>>>>
>>>>> So I suspect the real question is, what jobs should you be
running in
>>>>> STAT-Analysis?
>>>>>
>>>>> I see that you sent some config files but no actual output from
>>>> Point-Stat.
>>>>> If you send that I could take a look and recommend some STAT-
Analysis
>>>> jobs.
>>>>> You could either tar them up and send them or point me to them
on a
>>> NOAA
>>>>> machine, like WCOSS, jet, or theia (can’t remember it successor
right
>>>> now).
>>>>>
>>>>> Here’s instructions for posting data to our anonymous ftp site:
>>>>>
>>>>>
>>>>>
>>>>
>>>
>> https://dtcenter.org/community-code/model-evaluation-tools-met/met-
help-desk#ftp
>>>>>
>>>>> Thanks
>>>>> John
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Oct 6, 2020 at 10:56 AM Julie Prestopnik via RT <
>>>> met_help at ucar.edu
>>>>>>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96971 >
>>>>>>
>>>>>> Hi Erica.
>>>>>>
>>>>>> I see that you are having trouble getting the statistics that
you
>>> would
>>>>>> like from Point-Stat. I am assigning this ticket to John
Halley
>>>> Gotway.
>>>>>> Please allow a few business days for a response.
>>>>>>
>>>>>> Julie
>>>>>>
>>>>>> On Tue, Oct 6, 2020 at 10:37 AM Erica Burrows - NOAA Affiliate
via
>>> RT <
>>>>>> met_help at ucar.edu> wrote:
>>>>>>
>>>>>>>
>>>>>>> Tue Oct 06 10:37:13 2020: Request 96971 was acted upon.
>>>>>>> Transaction: Ticket created by erica.burrows at noaa.gov
>>>>>>> Queue: met_help
>>>>>>> Subject: Implementing Climatology Data to Point Stat
>>>>>>> Owner: Nobody
>>>>>>> Requestors: erica.burrows at noaa.gov
>>>>>>> Status: new
>>>>>>> Ticket <URL:
>>>> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96971
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> To whom this may concern,
>>>>>>>
>>>>>>> I am wanting to extend the statistics I currently compute for
my
>>>> point
>>>>>> stat
>>>>>>> processing with no progress. Currently, I am able to produce
mean
>>>>>>> forecast/obs and RMSE but I am also wanting to include the
>>> following:
>>>>>>>
>>>>>>> -
>>>>>>>
>>>>>>> CNT
>>>>>>> -
>>>>>>>
>>>>>>> Forecast/Observation Standard Deviation
>>>>>>> -
>>>>>>>
>>>>>>> Pearson Correlation Coefficient
>>>>>>> -
>>>>>>>
>>>>>>> Standard deviation of the error
>>>>>>> -
>>>>>>>
>>>>>>> Anomaly Correlation Coefficient
>>>>>>> -
>>>>>>>
>>>>>>> CST
>>>>>>> -
>>>>>>>
>>>>>>> Base Rate
>>>>>>> -
>>>>>>>
>>>>>>> Accuracy
>>>>>>> -
>>>>>>>
>>>>>>> Frequency Bias
>>>>>>> -
>>>>>>>
>>>>>>> Heidke Skill Score
>>>>>>>
>>>>>>>
>>>>>>> At the moment, those statistics listed are all reporting
either
>> NA,
>>>> 0,
>>>>>> or 1
>>>>>>> (depending on the stat). I understand I need to include
>> thresholds
>>>>> which
>>>>>> I
>>>>>>> have tried both within my point stat config and outside (but
>>>> including
>>>>>> when
>>>>>>> running stat analysis) with no luck.
>>>>>>>
>>>>>>> I thought perhaps my problem was my climo_mean attributes were
>>> empty,
>>>>>>> however when I try to list txt files it does not accept them
>>> (which I
>>>>>>> assumed would happen). I then tried to convert these files
using
>>>>>>> ascii2nc with no luck. I am assuming the error is due to the
fact
>>>> each
>>>>>>> climo file uses julian day rather than an individual date (to
>> avoid
>>>> 35
>>>>>>> files having the same information, as these are applicable to
35
>>>> years
>>>>>>> worth of data). My txt files have the following format:
>>>>>>> *Message_Type Station_ID Valid_Time Lat Lon Elevation
>> Variable_Name
>>>>> Level
>>>>>>> Height QC_String Observation_Value\n*
>>>>>>>
>>>>>>> If I do not need to provide climatology values to receive
those
>>> stats
>>>>>>> listed, then my issue must be with how I am formatting my
point
>>> stat
>>>>> and
>>>>>>> stat analysis calls which is as follows:
>>>>>>>
>>>>>>> *for proj in {6..384..6}*
>>>>>>>
>>>>>>>
>>>>>>> *do PR=`echo $(printf %03d ${proj})`*
>>>>>>>
>>>>>>> * ${metDIR}/point_stat ${fcstdir}/${WSFILE}_pr${PR}.nc
>>>> ${VALIDFILE}
>>>>>>> PointStatConfig_ws -v 4 -outdir ${writeDIR} -obs_valid_beg
>>>>>>> ${VALIDDATE}_000000 -obs_valid_end ${VALIDDATE}_235959 *
>>>>>>> *done*
>>>>>>>
>>>>>>> *fileLIST=($(ls
>> ${outdir}/point_stat_*_${currDAY}_*0000V_mpr.txt))*
>>>>>>>
>>>>>>>
>>>>>>> *${metDIR}/stat_analysis -lookin ${fileLIST} -config
>>>>>>> "STATAnalysisConfig_ws" -v 0 -vx_mask FULL -job aggregate_stat
>>>>> -line_type
>>>>>>> MPR -out_line_type CNT -out_stat
>>>>>>> "${outdir}/${currDAY}cy${CY}_mpr_to_cnt.stat" -by
>> FCST_VAR,OBS_SID
>>>>>> -set_hdr
>>>>>>> VX_MASK OBS_SID -set_hdr DESC CASE -out
>>>>>>> "${outdir}/${YYYY}${MM}${DD}_v${currDAY}_ws_cnt.stat"*
>>>>>>>
>>>>>>> * ${metDIR}/stat_analysis -lookin ${fileLIST} -config
>>>>>>> "STATAnalysisConfig_ws" -v 0 -vx_mask FULL -job aggregate_stat
>>>>> -line_type
>>>>>>> MPR -out_line_type MCTS -out_stat
>>>>>>> "${outdir}/${currDAY}cy${CY}_mpr_to_mcts.stat" -by
>> FCST_VAR,OBS_SID
>>>>>>> -set_hdr VX_MASK OBS_SID -set_hdr DESC CASE -out_thresh
>>>>>>> ge4.12,ge10.29,ge18.01,ge25.72 -out
>>>>>>> "${outdir}/${YYYY}${MM}${DD}_v${currDAY}_ws_mcts.stat" *
>>>>>>>
>>>>>>> If it is more beneficial, I have provided the following
>>> attachments:
>>>>>>> 1. Current working point stat config file (PointStatConfig_ws)
>>>>>>> 2. Testing point stat config file for climo
>>> (PointStatConfig_ws.test)
>>>>>>> 3. Stat analysis config (STATAnalysisConfig_ws)
>>>>>>> 4. 4 fcst data files (ws_YYYYMMDDCY_c00_prPPP)
>>>>>>> 5. 1 obs data file (YYYYMMDD_CY:00:00) which used ascii2nc to
be
>>>>> created
>>>>>>> 6. 1 climo file (jdDDD_climatology)
>>>>>>>
>>>>>>> Hopefully I did not provide too much information on what I
have
>>> been
>>>>>> trying
>>>>>>> to accomplish. But basically, I am trying to utilize
point_stat
>> to
>>>>>> compute
>>>>>>> those stats listed for 36 core airports across CONUS, and at
this
>>>>> point I
>>>>>>> am at a loss as to why some work and others do not. Any and
all
>>>>> guidance
>>>>>> is
>>>>>>> greatly appreciated.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Erica Burrows
>>>>>>> Atmospheric Scientist
>>>>>>> INNOVIM/Climate Prediction Center
>>>>>>> NOAA Center for Weather and Climate Prediction
>>>>>>> 5830 University Research Court
>>>>>>> College Park, MD 20740
>>>>>>>
>>>>>>> Work:
>>>>>>> Cell: 818-925-5795
>>>>>>> Email: erica.burrows at noaa.gov
>>>>>>> *Innovim LLC*
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Julie Prestopnik (she/her/hers)
>>>>>> Software Engineer
>>>>>> National Center for Atmospheric Research
>>>>>> Research Applications Laboratory
>>>>>> Email: jpresto at ucar.edu
>>>>>>
>>>>>> My working day may not be your working day. Please do not feel
>>> obliged
>>>>> to
>>>>>> reply to this email outside of your normal working hours.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
------------------------------------------------
Subject: Implementing Climatology Data to Point Stat
From: John Halley Gotway
Time: Tue Oct 13 13:25:26 2020
Eric,
OK, testing with the data you already sent me, I ran the following
stat_analysis jobs and have attached the output files:
stat_analysis -lookin data -job aggregate_stat -line_type MPR
-out_line_type CTC -by FCST_VAR,FCST_LEV,OBS_SID -out_thresh ge5
-set_hdr
VX_MASK OBS_SID -out_stat mpr_to_ctc.stat
stat_analysis -lookin data -job aggregate_stat -line_type MPR
-out_line_type CTS -by FCST_VAR,FCST_LEV,OBS_SID -out_thresh ge5
-set_hdr
VX_MASK OBS_SID -out_stat mpr_to_cts.stat
So this reads the MPR data and applies the threshold of >=5 to define
contingency table counts (mpr_to_ctc.stat) and the corresponding
contingency table statistics (mpr_to_cts.stat).
The 26th and 31st columns of the output file contain BASER and FMEAN,
respectively. Those are the proportion of the event occurrence in the
observation and forecast values, respectively:
> cat mpr_to_cts.stat | awk '{print $26, $31}'
Here's the first 5 lines of output:
BASER FMEAN
0.45641 0.11795
0.061538 0.010256
0.48205 0.30769
0.12308 0.46667
None of those values are NA. They are all values between 0 and 1. Can
you
please run the same jobs on your version of MET? Do you see value
numbers
between 0 and 1 or do you still see NA? If the latter, perhaps there's
some
problem in the compilation of MET you're using.
Thanks,
John
------------------------------------------------
Subject: Implementing Climatology Data to Point Stat
From: Erica Burrows - NOAA Affiliate
Time: Tue Oct 13 14:15:35 2020
John,
Thank you! Unfortunately I am still not getting the same return, my
BASER
is always either 1 or 0 and ACC is always 1 or NA (which even though
ACC
of would be ideal I know it is not right...). Any idea how I can
check if
MET was compiled correctly? It was compiled by someone else so I am
not
sure what to look for.
Since I am still in the development phase I was waiting before
switching to
WCOSS. However, if need be, is there a version of MET on there I
should be
pointing to or would I need to compile on there as well?
Erica Burrows
Atmospheric Scientist
INNOVIM/Climate Prediction Center
NOAA Center for Weather and Climate Prediction
5830 University Research Court
College Park, MD 20740
Work:
Cell: 818-925-5795
Email: erica.burrows at noaa.gov
*Innovim LLC*
On Tue, Oct 13, 2020 at 3:25 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Eric,
>
> OK, testing with the data you already sent me, I ran the following
> stat_analysis jobs and have attached the output files:
>
> stat_analysis -lookin data -job aggregate_stat -line_type MPR
> -out_line_type CTC -by FCST_VAR,FCST_LEV,OBS_SID -out_thresh ge5
-set_hdr
> VX_MASK OBS_SID -out_stat mpr_to_ctc.stat
>
> stat_analysis -lookin data -job aggregate_stat -line_type MPR
> -out_line_type CTS -by FCST_VAR,FCST_LEV,OBS_SID -out_thresh ge5
-set_hdr
> VX_MASK OBS_SID -out_stat mpr_to_cts.stat
>
> So this reads the MPR data and applies the threshold of >=5 to
define
> contingency table counts (mpr_to_ctc.stat) and the corresponding
> contingency table statistics (mpr_to_cts.stat).
>
> The 26th and 31st columns of the output file contain BASER and
FMEAN,
> respectively. Those are the proportion of the event occurrence in
the
> observation and forecast values, respectively:
>
> > cat mpr_to_cts.stat | awk '{print $26, $31}'
>
> Here's the first 5 lines of output:
>
> BASER FMEAN
> 0.45641 0.11795
> 0.061538 0.010256
> 0.48205 0.30769
> 0.12308 0.46667
>
> None of those values are NA. They are all values between 0 and 1.
Can you
> please run the same jobs on your version of MET? Do you see value
numbers
> between 0 and 1 or do you still see NA? If the latter, perhaps
there's some
> problem in the compilation of MET you're using.
>
> Thanks,
> John
>
>
------------------------------------------------
Subject: Implementing Climatology Data to Point Stat
From: John Halley Gotway
Time: Tue Oct 13 15:06:45 2020
Erica,
The nice thing about running on WCOSS is that the compilations there
are
widely tested and supported.
If nothing else, I'd recommend copying a few of the .stat files with
MPR
output that you sent to me up to WCOSS and run the stat_analysis
commands
up there. Then check to see if you get different results there vs your
local machine.
I'm having a difficult time guessing what underlying problem would
explain
this behavior... that point_stat is writing output fine, but
stat_analysis
is not parsing that data well... or not parsing the thresholds you're
defining well.
Please see this page from the METplus website about accessing existing
builds of MET/METplus:
http://dtcenter.org/community-code/metplus/metplus-3-1-existing-
builds
Just click on the "NOAA machines" box to see the instructions for each
machine. For example, on mars you'd run:
module use
/gpfs/dell2/emc/verification/noscrub/emc.metplus/modulefiles
module load met/9.1
This nice organization is thanks to Julie Prestopnik, a member of the
METplus team. She installs new versions of MET and METplus on several
common machines and keeps this list of existing builds information
up-to-date.
Thanks,
John
On Tue, Oct 13, 2020 at 2:16 PM Erica Burrows - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96971 >
>
> John,
>
> Thank you! Unfortunately I am still not getting the same return, my
BASER
> is always either 1 or 0 and ACC is always 1 or NA (which even though
ACC
> of would be ideal I know it is not right...). Any idea how I can
check if
> MET was compiled correctly? It was compiled by someone else so I am
not
> sure what to look for.
>
> Since I am still in the development phase I was waiting before
switching to
> WCOSS. However, if need be, is there a version of MET on there I
should be
> pointing to or would I need to compile on there as well?
>
> Erica Burrows
> Atmospheric Scientist
> INNOVIM/Climate Prediction Center
> NOAA Center for Weather and Climate Prediction
> 5830 University Research Court
> College Park, MD 20740
>
> Work:
> Cell: 818-925-5795
> Email: erica.burrows at noaa.gov
> *Innovim LLC*
>
>
> On Tue, Oct 13, 2020 at 3:25 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Eric,
> >
> > OK, testing with the data you already sent me, I ran the following
> > stat_analysis jobs and have attached the output files:
> >
> > stat_analysis -lookin data -job aggregate_stat -line_type MPR
> > -out_line_type CTC -by FCST_VAR,FCST_LEV,OBS_SID -out_thresh ge5
-set_hdr
> > VX_MASK OBS_SID -out_stat mpr_to_ctc.stat
> >
> > stat_analysis -lookin data -job aggregate_stat -line_type MPR
> > -out_line_type CTS -by FCST_VAR,FCST_LEV,OBS_SID -out_thresh ge5
-set_hdr
> > VX_MASK OBS_SID -out_stat mpr_to_cts.stat
> >
> > So this reads the MPR data and applies the threshold of >=5 to
define
> > contingency table counts (mpr_to_ctc.stat) and the corresponding
> > contingency table statistics (mpr_to_cts.stat).
> >
> > The 26th and 31st columns of the output file contain BASER and
FMEAN,
> > respectively. Those are the proportion of the event occurrence in
the
> > observation and forecast values, respectively:
> >
> > > cat mpr_to_cts.stat | awk '{print $26, $31}'
> >
> > Here's the first 5 lines of output:
> >
> > BASER FMEAN
> > 0.45641 0.11795
> > 0.061538 0.010256
> > 0.48205 0.30769
> > 0.12308 0.46667
> >
> > None of those values are NA. They are all values between 0 and 1.
Can you
> > please run the same jobs on your version of MET? Do you see value
numbers
> > between 0 and 1 or do you still see NA? If the latter, perhaps
there's
> some
> > problem in the compilation of MET you're using.
> >
> > Thanks,
> > John
> >
> >
>
>
------------------------------------------------
More information about the Met_help
mailing list