[Met_help] [rt.rap.ucar.edu #96453] History for can not upload files to MET_Viewer
Tatiana Burek via RT
met_help at ucar.edu
Tue Oct 13 12:19:39 MDT 2020
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
Hello,
I created 5 ensemble_stat* files (from hr18- hr22) under the directory
below, but only the first two (hr18 and hr19) could be read from
METVIewer. Those files are under database "mv_g2g_met_verf_Reventador"
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
What is the problem? Thank you.
Binyu
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: can not upload files to MET_Viewer
From: Tatiana Burek
Time: Wed Aug 26 07:30:30 2020
Hi Binyu,
I assume that your files are on WCOSS. I don't have an access to the
server and can't see the files.
Do you have logging messages from the load? Any errors?
I see that only 5 ensemble_stat files were loaded to this database:
| ensemble_stat_Reventador_hr02_20190225_180000V.stat |
| ensemble_stat_Reventador_hr03_20190225_190000V.stat |
| ensemble_stat_Reventador_hr04_20190225_200000V.stat |
| ensemble_stat_Reventador_hr05_20190225_210000V.stat |
| ensemble_stat_Reventador_hr06_20190225_220000V.stat
Tatiana
On Tue Aug 25 20:02:54 2020, binyu.wang at noaa.gov wrote:
> Hello,
>
> I created 5 ensemble_stat* files (from hr18- hr22) under the
directory
> below, but only the first two (hr18 and hr19) could be read from
> METVIewer. Those files are under database
"mv_g2g_met_verf_Reventador"
>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
>
> What is the problem? Thank you.
> Binyu
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: binyu.wang at noaa.gov
Time: Wed Aug 26 08:32:17 2020
Tatiana,
I didn't see any error from the log file. I attached the log file and
two
of the stat files:
ensemble_stat_Reventador_hr03_20190225_190000V.stat is
readable from METViewer, but
"ensemble_stat_Reventador_hr04_20190225_200000V.stat" not.
Thank you.
On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> Hi Binyu,
> I assume that your files are on WCOSS. I don't have an access to the
> server and can't see the files.
> Do you have logging messages from the load? Any errors?
>
> I see that only 5 ensemble_stat files were loaded to this database:
> | ensemble_stat_Reventador_hr02_20190225_180000V.stat |
> | ensemble_stat_Reventador_hr03_20190225_190000V.stat |
> | ensemble_stat_Reventador_hr04_20190225_200000V.stat |
> | ensemble_stat_Reventador_hr05_20190225_210000V.stat |
> | ensemble_stat_Reventador_hr06_20190225_220000V.stat
>
> Tatiana
>
> On Tue Aug 25 20:02:54 2020, binyu.wang at noaa.gov wrote:
> > Hello,
> >
> > I created 5 ensemble_stat* files (from hr18- hr22) under the
directory
> > below, but only the first two (hr18 and hr19) could be read from
> > METVIewer. Those files are under database
"mv_g2g_met_verf_Reventador"
> >
> >
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> >
> > What is the problem? Thank you.
> > Binyu
>
>
>
>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: Tatiana Burek
Time: Wed Aug 26 09:23:58 2020
Binyu,
In your log file there is a record that
'ensemble_stat_Reventador_hr04_20190225_200000V.stat' was successfully
loaded - all 53 lines.
I also can create a plot using the data from this file - ORANK
statistics with 40000 forecast lead: 20200826_151853.
Tatiana
On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov wrote:
> Tatiana,
>
> I didn't see any error from the log file. I attached the log file
and two
> of the stat files:
ensemble_stat_Reventador_hr03_20190225_190000V.stat is
> readable from METViewer, but
> "ensemble_stat_Reventador_hr04_20190225_200000V.stat" not.
>
> Thank you.
>
> On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek via RT
<met_help at ucar.edu>
> wrote:
>
> > Hi Binyu,
> > I assume that your files are on WCOSS. I don't have an access to
the
> > server and can't see the files.
> > Do you have logging messages from the load? Any errors?
> >
> > I see that only 5 ensemble_stat files were loaded to this
database:
> > | ensemble_stat_Reventador_hr02_20190225_180000V.stat |
> > | ensemble_stat_Reventador_hr03_20190225_190000V.stat |
> > | ensemble_stat_Reventador_hr04_20190225_200000V.stat |
> > | ensemble_stat_Reventador_hr05_20190225_210000V.stat |
> > | ensemble_stat_Reventador_hr06_20190225_220000V.stat
> >
> > Tatiana
> >
> > On Tue Aug 25 20:02:54 2020, binyu.wang at noaa.gov wrote:
> > > Hello,
> > >
> > > I created 5 ensemble_stat* files (from hr18- hr22) under the
directory
> > > below, but only the first two (hr18 and hr19) could be read
from
> > > METVIewer. Those files are under database
"mv_g2g_met_verf_Reventador"
> > >
> > >
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > >
> > > What is the problem? Thank you.
> > > Binyu
> >
> >
> >
> >
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: binyu.wang at noaa.gov
Time: Wed Aug 26 10:29:05 2020
Tatiana,
Yes that is weird. If you see the attached file, you can see
there only two files were uploaded. I expect to have all 5.
Binyu
On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> In your log file there is a record that
> 'ensemble_stat_Reventador_hr04_20190225_200000V.stat' was
successfully
> loaded - all 53 lines.
> I also can create a plot using the data from this file - ORANK
statistics
> with 40000 forecast lead: 20200826_151853.
>
> Tatiana
>
>
> On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov wrote:
> > Tatiana,
> >
> > I didn't see any error from the log file. I attached the log file
and two
> > of the stat files:
ensemble_stat_Reventador_hr03_20190225_190000V.stat
> is
> > readable from METViewer, but
> > "ensemble_stat_Reventador_hr04_20190225_200000V.stat" not.
> >
> > Thank you.
> >
> > On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > Hi Binyu,
> > > I assume that your files are on WCOSS. I don't have an access to
the
> > > server and can't see the files.
> > > Do you have logging messages from the load? Any errors?
> > >
> > > I see that only 5 ensemble_stat files were loaded to this
database:
> > > | ensemble_stat_Reventador_hr02_20190225_180000V.stat |
> > > | ensemble_stat_Reventador_hr03_20190225_190000V.stat |
> > > | ensemble_stat_Reventador_hr04_20190225_200000V.stat |
> > > | ensemble_stat_Reventador_hr05_20190225_210000V.stat |
> > > | ensemble_stat_Reventador_hr06_20190225_220000V.stat
> > >
> > > Tatiana
> > >
> > > On Tue Aug 25 20:02:54 2020, binyu.wang at noaa.gov wrote:
> > > > Hello,
> > > >
> > > > I created 5 ensemble_stat* files (from hr18- hr22) under the
> directory
> > > > below, but only the first two (hr18 and hr19) could be read
from
> > > > METVIewer. Those files are under database
> "mv_g2g_met_verf_Reventador"
> > > >
> > > >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > > >
> > > > What is the problem? Thank you.
> > > > Binyu
> > >
> > >
> > >
> > >
>
>
>
>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: Tatiana Burek
Time: Wed Aug 26 11:20:47 2020
Binyu,
All 5 ensemble_stat files were successfully loaded to the DB and the
data is available for potting. Here is the example: 20200826_164848.
Your screenshot displays that you try to create a histogram plot. The
data for this plot comes from line_data_rhist table. This table gets
filled with 'RHIST' line type.
'ensemble_stat_Reventador_hr04_20190225_200000V.stat' file contains
only 'ORANK' line type so non of it's records were saved into
line_data_rhist table.
Currently line_data_rhist table contains data for fcst_valid_beg:
'2019-02-25 18:00:00' and '2019-02-25 19:00:00' ( fcst_lead: 20000 and
30000). The data came from
ensemble_stat_Reventador_hr02_20190225_180000V.stat and
ensemble_stat_Reventador_hr03_20190225_190000V.stat
Could you check if other files have records with RHIST line type.
Tatiana
n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov wrote:
> Tatiana,
>
> Yes that is weird. If you see the attached file, you can see
> there only two files were uploaded. I expect to have all 5.
> Binyu
>
> On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek via RT
> <met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > In your log file there is a record that
> > 'ensemble_stat_Reventador_hr04_20190225_200000V.stat' was
> > successfully
> > loaded - all 53 lines.
> > I also can create a plot using the data from this file - ORANK
> > statistics
> > with 40000 forecast lead: 20200826_151853.
> >
> > Tatiana
> >
> >
> > On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov wrote:
> > > Tatiana,
> > >
> > > I didn't see any error from the log file. I attached the log
file
> > > and two
> > > of the stat files:
> > > ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > is
> > > readable from METViewer, but
> > > "ensemble_stat_Reventador_hr04_20190225_200000V.stat" not.
> > >
> > > Thank you.
> > >
> > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hi Binyu,
> > > > I assume that your files are on WCOSS. I don't have an access
to
> > > > the
> > > > server and can't see the files.
> > > > Do you have logging messages from the load? Any errors?
> > > >
> > > > I see that only 5 ensemble_stat files were loaded to this
> > > > database:
> > > > | ensemble_stat_Reventador_hr02_20190225_180000V.stat |
> > > > | ensemble_stat_Reventador_hr03_20190225_190000V.stat |
> > > > | ensemble_stat_Reventador_hr04_20190225_200000V.stat |
> > > > | ensemble_stat_Reventador_hr05_20190225_210000V.stat |
> > > > | ensemble_stat_Reventador_hr06_20190225_220000V.stat
> > > >
> > > > Tatiana
> > > >
> > > > On Tue Aug 25 20:02:54 2020, binyu.wang at noaa.gov wrote:
> > > > > Hello,
> > > > >
> > > > > I created 5 ensemble_stat* files (from hr18- hr22) under
the
> > directory
> > > > > below, but only the first two (hr18 and hr19) could be read
> > > > > from
> > > > > METVIewer. Those files are under database
> > "mv_g2g_met_verf_Reventador"
> > > > >
> > > > >
> >
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > > > >
> > > > > What is the problem? Thank you.
> > > > > Binyu
> > > >
> > > >
> > > >
> > > >
> >
> >
> >
> >
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: binyu.wang at noaa.gov
Time: Wed Aug 26 13:36:13 2020
You are right. I checked the other stat files and the column for RHIST
is
empty. Why do the first two hours have RHIST, but the following three
hours
don't? They have the same config file. Not enough matching
pairs between obs. and model?
Thank you.
Binyu
On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> Binyu,
> All 5 ensemble_stat files were successfully loaded to the DB and the
data
> is available for potting. Here is the example: 20200826_164848.
> Your screenshot displays that you try to create a histogram plot.
The data
> for this plot comes from line_data_rhist table. This table gets
filled with
> 'RHIST' line type.
'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
> file contains only 'ORANK' line type so non of it's records were
saved into
> line_data_rhist table.
> Currently line_data_rhist table contains data for fcst_valid_beg:
> '2019-02-25 18:00:00' and '2019-02-25 19:00:00' ( fcst_lead: 20000
and
> 30000). The data came from
> ensemble_stat_Reventador_hr02_20190225_180000V.stat and
> ensemble_stat_Reventador_hr03_20190225_190000V.stat
>
> Could you check if other files have records with RHIST line type.
>
> Tatiana
>
> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov wrote:
> > Tatiana,
> >
> > Yes that is weird. If you see the attached file, you can see
> > there only two files were uploaded. I expect to have all 5.
> > Binyu
> >
> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > Binyu,
> > >
> > > In your log file there is a record that
> > > 'ensemble_stat_Reventador_hr04_20190225_200000V.stat' was
> > > successfully
> > > loaded - all 53 lines.
> > > I also can create a plot using the data from this file - ORANK
> > > statistics
> > > with 40000 forecast lead: 20200826_151853.
> > >
> > > Tatiana
> > >
> > >
> > > On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov wrote:
> > > > Tatiana,
> > > >
> > > > I didn't see any error from the log file. I attached the log
file
> > > > and two
> > > > of the stat files:
> > > > ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > is
> > > > readable from METViewer, but
> > > > "ensemble_stat_Reventador_hr04_20190225_200000V.stat" not.
> > > >
> > > > Thank you.
> > > >
> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek via RT
> > > > <met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Binyu,
> > > > > I assume that your files are on WCOSS. I don't have an
access to
> > > > > the
> > > > > server and can't see the files.
> > > > > Do you have logging messages from the load? Any errors?
> > > > >
> > > > > I see that only 5 ensemble_stat files were loaded to this
> > > > > database:
> > > > > | ensemble_stat_Reventador_hr02_20190225_180000V.stat |
> > > > > | ensemble_stat_Reventador_hr03_20190225_190000V.stat |
> > > > > | ensemble_stat_Reventador_hr04_20190225_200000V.stat |
> > > > > | ensemble_stat_Reventador_hr05_20190225_210000V.stat |
> > > > > | ensemble_stat_Reventador_hr06_20190225_220000V.stat
> > > > >
> > > > > Tatiana
> > > > >
> > > > > On Tue Aug 25 20:02:54 2020, binyu.wang at noaa.gov wrote:
> > > > > > Hello,
> > > > > >
> > > > > > I created 5 ensemble_stat* files (from hr18- hr22) under
the
> > > directory
> > > > > > below, but only the first two (hr18 and hr19) could be
read
> > > > > > from
> > > > > > METVIewer. Those files are under database
> > > "mv_g2g_met_verf_Reventador"
> > > > > >
> > > > > >
> > >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > > > > >
> > > > > > What is the problem? Thank you.
> > > > > > Binyu
> > > > >
> > > > >
> > > > >
> > > > >
> > >
> > >
> > >
> > >
>
>
>
>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: binyu.wang at noaa.gov
Time: Wed Aug 26 20:47:54 2020
Hello,
I think I know the answer now, but just to confirm with you: if there
is no
RHIST line type in the stat file, which indicates there is no matching
pair
at all between obs. and forecast, is that correct?
Thank you.
On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> Binyu,
> All 5 ensemble_stat files were successfully loaded to the DB and the
data
> is available for potting. Here is the example: 20200826_164848.
> Your screenshot displays that you try to create a histogram plot.
The data
> for this plot comes from line_data_rhist table. This table gets
filled with
> 'RHIST' line type.
'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
> file contains only 'ORANK' line type so non of it's records were
saved into
> line_data_rhist table.
> Currently line_data_rhist table contains data for fcst_valid_beg:
> '2019-02-25 18:00:00' and '2019-02-25 19:00:00' ( fcst_lead: 20000
and
> 30000). The data came from
> ensemble_stat_Reventador_hr02_20190225_180000V.stat and
> ensemble_stat_Reventador_hr03_20190225_190000V.stat
>
> Could you check if other files have records with RHIST line type.
>
> Tatiana
>
> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov wrote:
> > Tatiana,
> >
> > Yes that is weird. If you see the attached file, you can see
> > there only two files were uploaded. I expect to have all 5.
> > Binyu
> >
> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > Binyu,
> > >
> > > In your log file there is a record that
> > > 'ensemble_stat_Reventador_hr04_20190225_200000V.stat' was
> > > successfully
> > > loaded - all 53 lines.
> > > I also can create a plot using the data from this file - ORANK
> > > statistics
> > > with 40000 forecast lead: 20200826_151853.
> > >
> > > Tatiana
> > >
> > >
> > > On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov wrote:
> > > > Tatiana,
> > > >
> > > > I didn't see any error from the log file. I attached the log
file
> > > > and two
> > > > of the stat files:
> > > > ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > is
> > > > readable from METViewer, but
> > > > "ensemble_stat_Reventador_hr04_20190225_200000V.stat" not.
> > > >
> > > > Thank you.
> > > >
> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek via RT
> > > > <met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Hi Binyu,
> > > > > I assume that your files are on WCOSS. I don't have an
access to
> > > > > the
> > > > > server and can't see the files.
> > > > > Do you have logging messages from the load? Any errors?
> > > > >
> > > > > I see that only 5 ensemble_stat files were loaded to this
> > > > > database:
> > > > > | ensemble_stat_Reventador_hr02_20190225_180000V.stat |
> > > > > | ensemble_stat_Reventador_hr03_20190225_190000V.stat |
> > > > > | ensemble_stat_Reventador_hr04_20190225_200000V.stat |
> > > > > | ensemble_stat_Reventador_hr05_20190225_210000V.stat |
> > > > > | ensemble_stat_Reventador_hr06_20190225_220000V.stat
> > > > >
> > > > > Tatiana
> > > > >
> > > > > On Tue Aug 25 20:02:54 2020, binyu.wang at noaa.gov wrote:
> > > > > > Hello,
> > > > > >
> > > > > > I created 5 ensemble_stat* files (from hr18- hr22) under
the
> > > directory
> > > > > > below, but only the first two (hr18 and hr19) could be
read
> > > > > > from
> > > > > > METVIewer. Those files are under database
> > > "mv_g2g_met_verf_Reventador"
> > > > > >
> > > > > >
> > >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > > > > >
> > > > > > What is the problem? Thank you.
> > > > > > Binyu
> > > > >
> > > > >
> > > > >
> > > > >
> > >
> > >
> > >
> > >
>
>
>
>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: binyu.wang at noaa.gov
Time: Thu Aug 27 08:48:35 2020
Hello Tatiana,
Sorry to ask more: as I said in previous mail I can not find the other
3
files from MET_Viewer because there is no RHIST inside of those files,
if
this is because there is no matched pairs between obs. and forecast,
why I
can see them in the RELY plot? (see the attachment please). Are RELY
plots
created based on matching pairs?
Thank you.
Binyu
On Wed, Aug 26, 2020 at 10:47 PM Binyu Wang - NOAA Affiliate <
binyu.wang at noaa.gov> wrote:
> Hello,
>
> I think I know the answer now, but just to confirm with you: if
there is
> no RHIST line type in the stat file, which indicates there is no
matching
> pair at all between obs. and forecast, is that correct?
>
> Thank you.
>
> On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via RT
<met_help at ucar.edu>
> wrote:
>
>> Binyu,
>> All 5 ensemble_stat files were successfully loaded to the DB and
the data
>> is available for potting. Here is the example: 20200826_164848.
>> Your screenshot displays that you try to create a histogram plot.
The
>> data for this plot comes from line_data_rhist table. This table
gets filled
>> with 'RHIST' line type.
>> 'ensemble_stat_Reventador_hr04_20190225_200000V.stat' file contains
only
>> 'ORANK' line type so non of it's records were saved into
line_data_rhist
>> table.
>> Currently line_data_rhist table contains data for fcst_valid_beg:
>> '2019-02-25 18:00:00' and '2019-02-25 19:00:00' ( fcst_lead: 20000
and
>> 30000). The data came from
>> ensemble_stat_Reventador_hr02_20190225_180000V.stat and
>> ensemble_stat_Reventador_hr03_20190225_190000V.stat
>>
>> Could you check if other files have records with RHIST line type.
>>
>> Tatiana
>>
>> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov wrote:
>> > Tatiana,
>> >
>> > Yes that is weird. If you see the attached file, you can see
>> > there only two files were uploaded. I expect to have all 5.
>> > Binyu
>> >
>> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek via RT
>> > <met_help at ucar.edu>
>> > wrote:
>> >
>> > > Binyu,
>> > >
>> > > In your log file there is a record that
>> > > 'ensemble_stat_Reventador_hr04_20190225_200000V.stat' was
>> > > successfully
>> > > loaded - all 53 lines.
>> > > I also can create a plot using the data from this file - ORANK
>> > > statistics
>> > > with 40000 forecast lead: 20200826_151853.
>> > >
>> > > Tatiana
>> > >
>> > >
>> > > On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov wrote:
>> > > > Tatiana,
>> > > >
>> > > > I didn't see any error from the log file. I attached the log
file
>> > > > and two
>> > > > of the stat files:
>> > > > ensemble_stat_Reventador_hr03_20190225_190000V.stat
>> > > is
>> > > > readable from METViewer, but
>> > > > "ensemble_stat_Reventador_hr04_20190225_200000V.stat" not.
>> > > >
>> > > > Thank you.
>> > > >
>> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek via RT
>> > > > <met_help at ucar.edu>
>> > > > wrote:
>> > > >
>> > > > > Hi Binyu,
>> > > > > I assume that your files are on WCOSS. I don't have an
access to
>> > > > > the
>> > > > > server and can't see the files.
>> > > > > Do you have logging messages from the load? Any errors?
>> > > > >
>> > > > > I see that only 5 ensemble_stat files were loaded to this
>> > > > > database:
>> > > > > | ensemble_stat_Reventador_hr02_20190225_180000V.stat |
>> > > > > | ensemble_stat_Reventador_hr03_20190225_190000V.stat |
>> > > > > | ensemble_stat_Reventador_hr04_20190225_200000V.stat |
>> > > > > | ensemble_stat_Reventador_hr05_20190225_210000V.stat |
>> > > > > | ensemble_stat_Reventador_hr06_20190225_220000V.stat
>> > > > >
>> > > > > Tatiana
>> > > > >
>> > > > > On Tue Aug 25 20:02:54 2020, binyu.wang at noaa.gov wrote:
>> > > > > > Hello,
>> > > > > >
>> > > > > > I created 5 ensemble_stat* files (from hr18- hr22) under
the
>> > > directory
>> > > > > > below, but only the first two (hr18 and hr19) could be
read
>> > > > > > from
>> > > > > > METVIewer. Those files are under database
>> > > "mv_g2g_met_verf_Reventador"
>> > > > > >
>> > > > > >
>> > >
>>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
>> > > > > >
>> > > > > > What is the problem? Thank you.
>> > > > > > Binyu
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > >
>> > >
>> > >
>> > >
>>
>>
>>
>>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: Tatiana Burek
Time: Thu Aug 27 11:43:12 2020
Binyu
Reliability plot uses data from line_data_pct table ( "PCT" line type
).
Both stat files that you sent me don't contain PCT line type and
variable 'VAFTD_ENS_FREQ_ge0.1'. Did you have data in this database
before the latest load?
Tatiana
On Thu Aug 27 08:48:35 2020, binyu.wang at noaa.gov wrote:
> Hello Tatiana,
>
> Sorry to ask more: as I said in previous mail I can not find the
other
> 3
> files from MET_Viewer because there is no RHIST inside of those
files,
> if
> this is because there is no matched pairs between obs. and forecast,
> why I
> can see them in the RELY plot? (see the attachment please). Are RELY
> plots
> created based on matching pairs?
>
> Thank you.
> Binyu
>
>
> On Wed, Aug 26, 2020 at 10:47 PM Binyu Wang - NOAA Affiliate <
> binyu.wang at noaa.gov> wrote:
>
> > Hello,
> >
> > I think I know the answer now, but just to confirm with you: if
there
> > is
> > no RHIST line type in the stat file, which indicates there is no
> > matching
> > pair at all between obs. and forecast, is that correct?
> >
> > Thank you.
> >
> > On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> >> Binyu,
> >> All 5 ensemble_stat files were successfully loaded to the DB and
the
> >> data
> >> is available for potting. Here is the example: 20200826_164848.
> >> Your screenshot displays that you try to create a histogram plot.
> >> The
> >> data for this plot comes from line_data_rhist table. This table
gets
> >> filled
> >> with 'RHIST' line type.
> >> 'ensemble_stat_Reventador_hr04_20190225_200000V.stat' file
contains
> >> only
> >> 'ORANK' line type so non of it's records were saved into
> >> line_data_rhist
> >> table.
> >> Currently line_data_rhist table contains data for fcst_valid_beg:
> >> '2019-02-25 18:00:00' and '2019-02-25 19:00:00' ( fcst_lead:
20000
> >> and
> >> 30000). The data came from
> >> ensemble_stat_Reventador_hr02_20190225_180000V.stat and
> >> ensemble_stat_Reventador_hr03_20190225_190000V.stat
> >>
> >> Could you check if other files have records with RHIST line type.
> >>
> >> Tatiana
> >>
> >> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov wrote:
> >> > Tatiana,
> >> >
> >> > Yes that is weird. If you see the attached file, you can see
> >> > there only two files were uploaded. I expect to have all 5.
> >> > Binyu
> >> >
> >> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek via RT
> >> > <met_help at ucar.edu>
> >> > wrote:
> >> >
> >> > > Binyu,
> >> > >
> >> > > In your log file there is a record that
> >> > > 'ensemble_stat_Reventador_hr04_20190225_200000V.stat' was
> >> > > successfully
> >> > > loaded - all 53 lines.
> >> > > I also can create a plot using the data from this file -
ORANK
> >> > > statistics
> >> > > with 40000 forecast lead: 20200826_151853.
> >> > >
> >> > > Tatiana
> >> > >
> >> > >
> >> > > On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov wrote:
> >> > > > Tatiana,
> >> > > >
> >> > > > I didn't see any error from the log file. I attached the
log
> >> > > > file
> >> > > > and two
> >> > > > of the stat files:
> >> > > > ensemble_stat_Reventador_hr03_20190225_190000V.stat
> >> > > is
> >> > > > readable from METViewer, but
> >> > > > "ensemble_stat_Reventador_hr04_20190225_200000V.stat" not.
> >> > > >
> >> > > > Thank you.
> >> > > >
> >> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek via RT
> >> > > > <met_help at ucar.edu>
> >> > > > wrote:
> >> > > >
> >> > > > > Hi Binyu,
> >> > > > > I assume that your files are on WCOSS. I don't have an
> >> > > > > access to
> >> > > > > the
> >> > > > > server and can't see the files.
> >> > > > > Do you have logging messages from the load? Any errors?
> >> > > > >
> >> > > > > I see that only 5 ensemble_stat files were loaded to this
> >> > > > > database:
> >> > > > > | ensemble_stat_Reventador_hr02_20190225_180000V.stat |
> >> > > > > | ensemble_stat_Reventador_hr03_20190225_190000V.stat |
> >> > > > > | ensemble_stat_Reventador_hr04_20190225_200000V.stat |
> >> > > > > | ensemble_stat_Reventador_hr05_20190225_210000V.stat |
> >> > > > > | ensemble_stat_Reventador_hr06_20190225_220000V.stat
> >> > > > >
> >> > > > > Tatiana
> >> > > > >
> >> > > > > On Tue Aug 25 20:02:54 2020, binyu.wang at noaa.gov wrote:
> >> > > > > > Hello,
> >> > > > > >
> >> > > > > > I created 5 ensemble_stat* files (from hr18- hr22)
under
> >> > > > > > the
> >> > > directory
> >> > > > > > below, but only the first two (hr18 and hr19) could be
> >> > > > > > read
> >> > > > > > from
> >> > > > > > METVIewer. Those files are under database
> >> > > "mv_g2g_met_verf_Reventador"
> >> > > > > >
> >> > > > > >
> >> > >
> >>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> >> > > > > >
> >> > > > > > What is the problem? Thank you.
> >> > > > > > Binyu
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > >
> >> > >
> >> > >
> >> > >
> >>
> >>
> >>
> >>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: John Halley Gotway
Time: Thu Aug 27 12:00:55 2020
Binyu,
Tatiana asked me to take a look at the RHIST line type being
unexpectedly
absent from some of your runs of Ensemble-Stat. I logged onto WCOSS
and
took a look in:
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
In there, I see 5 ensemble_stat output files. I have listed each below
followed by the line types which appear in each file. So that's column
#24
from each:
ensemble_stat_Reventador_hr02_20190225_180000V.stat ECNT ORANK PHIST
RELP
RHIST SSVAR
ensemble_stat_Reventador_hr03_20190225_190000V.stat ECNT ORANK PHIST
RELP
RHIST SSVAR
ensemble_stat_Reventador_hr04_20190225_200000V.stat ORANK
ensemble_stat_Reventador_hr05_20190225_210000V.stat ORANK
ensemble_stat_Reventador_hr06_20190225_220000V.stat ORANK
The ORANK line type is basically the "matched pairs" for ensemble
verification. Notice that the first 2 hours have the ORANK lines along
with
other statistic line types, such as RHIST. However, the last 3 hours
only
have the ORANK line type.
Looking at the output for the first 2 hours, I see that you've applied
the
OBS_THRESH filtering threshold as ">=0.1".
Here's a link to the table which defines the contents of the ORANK
line
type:
https://dtcenter.github.io/MET/Users_Guide/ensemble-
stat.html?highlight=orank%20line%20type#id7
Note that the actual observation rank value is in column #34. Checking
the
data in those last 3 files, you'll notice that the observation ranks
are
always listed as bad data:
cat ensemble_stat_Reventador_hr04_20190225_200000V.stat
ensemble_stat_Reventador_hr05_20190225_210000V.stat
ensemble_stat_Reventador_hr06_20190225_220000V.stat | grep ORANK |
awk
'{print $34}' | sort -u
NA
So now the question is why? You're using the "censor_thresh" and
"censor_val" options to reset the forecast values that are "<0.1" to
bad
data. So that introduces a lot of bad data values. And that results in
the
rank of the observation being computed as bad data. This is likely
controlled by how you have the "vld_thresh" or "ens_vld_thresh" set in
your
Ensemble-Stat configuration file.
John
On Thu, Aug 27, 2020 at 8:48 AM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453 >
>
> Hello Tatiana,
>
> Sorry to ask more: as I said in previous mail I can not find the
other 3
> files from MET_Viewer because there is no RHIST inside of those
files, if
> this is because there is no matched pairs between obs. and forecast,
why I
> can see them in the RELY plot? (see the attachment please). Are RELY
plots
> created based on matching pairs?
>
> Thank you.
> Binyu
>
>
> On Wed, Aug 26, 2020 at 10:47 PM Binyu Wang - NOAA Affiliate <
> binyu.wang at noaa.gov> wrote:
>
> > Hello,
> >
> > I think I know the answer now, but just to confirm with you: if
there is
> > no RHIST line type in the stat file, which indicates there is no
matching
> > pair at all between obs. and forecast, is that correct?
> >
> > Thank you.
> >
> > On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via RT
<met_help at ucar.edu>
> > wrote:
> >
> >> Binyu,
> >> All 5 ensemble_stat files were successfully loaded to the DB and
the
> data
> >> is available for potting. Here is the example: 20200826_164848.
> >> Your screenshot displays that you try to create a histogram plot.
The
> >> data for this plot comes from line_data_rhist table. This table
gets
> filled
> >> with 'RHIST' line type.
> >> 'ensemble_stat_Reventador_hr04_20190225_200000V.stat' file
contains only
> >> 'ORANK' line type so non of it's records were saved into
line_data_rhist
> >> table.
> >> Currently line_data_rhist table contains data for fcst_valid_beg:
> >> '2019-02-25 18:00:00' and '2019-02-25 19:00:00' ( fcst_lead:
20000 and
> >> 30000). The data came from
> >> ensemble_stat_Reventador_hr02_20190225_180000V.stat and
> >> ensemble_stat_Reventador_hr03_20190225_190000V.stat
> >>
> >> Could you check if other files have records with RHIST line type.
> >>
> >> Tatiana
> >>
> >> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov wrote:
> >> > Tatiana,
> >> >
> >> > Yes that is weird. If you see the attached file, you can see
> >> > there only two files were uploaded. I expect to have all 5.
> >> > Binyu
> >> >
> >> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek via RT
> >> > <met_help at ucar.edu>
> >> > wrote:
> >> >
> >> > > Binyu,
> >> > >
> >> > > In your log file there is a record that
> >> > > 'ensemble_stat_Reventador_hr04_20190225_200000V.stat' was
> >> > > successfully
> >> > > loaded - all 53 lines.
> >> > > I also can create a plot using the data from this file -
ORANK
> >> > > statistics
> >> > > with 40000 forecast lead: 20200826_151853.
> >> > >
> >> > > Tatiana
> >> > >
> >> > >
> >> > > On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov wrote:
> >> > > > Tatiana,
> >> > > >
> >> > > > I didn't see any error from the log file. I attached the
log file
> >> > > > and two
> >> > > > of the stat files:
> >> > > > ensemble_stat_Reventador_hr03_20190225_190000V.stat
> >> > > is
> >> > > > readable from METViewer, but
> >> > > > "ensemble_stat_Reventador_hr04_20190225_200000V.stat" not.
> >> > > >
> >> > > > Thank you.
> >> > > >
> >> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek via RT
> >> > > > <met_help at ucar.edu>
> >> > > > wrote:
> >> > > >
> >> > > > > Hi Binyu,
> >> > > > > I assume that your files are on WCOSS. I don't have an
access to
> >> > > > > the
> >> > > > > server and can't see the files.
> >> > > > > Do you have logging messages from the load? Any errors?
> >> > > > >
> >> > > > > I see that only 5 ensemble_stat files were loaded to this
> >> > > > > database:
> >> > > > > | ensemble_stat_Reventador_hr02_20190225_180000V.stat |
> >> > > > > | ensemble_stat_Reventador_hr03_20190225_190000V.stat |
> >> > > > > | ensemble_stat_Reventador_hr04_20190225_200000V.stat |
> >> > > > > | ensemble_stat_Reventador_hr05_20190225_210000V.stat |
> >> > > > > | ensemble_stat_Reventador_hr06_20190225_220000V.stat
> >> > > > >
> >> > > > > Tatiana
> >> > > > >
> >> > > > > On Tue Aug 25 20:02:54 2020, binyu.wang at noaa.gov wrote:
> >> > > > > > Hello,
> >> > > > > >
> >> > > > > > I created 5 ensemble_stat* files (from hr18- hr22)
under the
> >> > > directory
> >> > > > > > below, but only the first two (hr18 and hr19) could be
read
> >> > > > > > from
> >> > > > > > METVIewer. Those files are under database
> >> > > "mv_g2g_met_verf_Reventador"
> >> > > > > >
> >> > > > > >
> >> > >
> >>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> >> > > > > >
> >> > > > > > What is the problem? Thank you.
> >> > > > > > Binyu
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > >
> >> > >
> >> > >
> >> > >
> >>
> >>
> >>
> >>
>
>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: binyu.wang at noaa.gov
Time: Thu Aug 27 12:03:27 2020
Tatiana,
"PCT" is generated later using the ensemble*stat files which I sent to
you
earlier. So you didn't see it in those two files. I didn't understand:
if
the 3 files didn't have "RHIST" because there was no matching pair
during that time period, why could they be used for RELY plot?
Binyu
On Thu, Aug 27, 2020 at 1:43 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> Binyu
>
> Reliability plot uses data from line_data_pct table ( "PCT" line
type ).
> Both stat files that you sent me don't contain PCT line type and
variable
> 'VAFTD_ENS_FREQ_ge0.1'. Did you have data in this database before
the
> latest load?
>
> Tatiana
>
> On Thu Aug 27 08:48:35 2020, binyu.wang at noaa.gov wrote:
> > Hello Tatiana,
> >
> > Sorry to ask more: as I said in previous mail I can not find the
other
> > 3
> > files from MET_Viewer because there is no RHIST inside of those
files,
> > if
> > this is because there is no matched pairs between obs. and
forecast,
> > why I
> > can see them in the RELY plot? (see the attachment please). Are
RELY
> > plots
> > created based on matching pairs?
> >
> > Thank you.
> > Binyu
> >
> >
> > On Wed, Aug 26, 2020 at 10:47 PM Binyu Wang - NOAA Affiliate <
> > binyu.wang at noaa.gov> wrote:
> >
> > > Hello,
> > >
> > > I think I know the answer now, but just to confirm with you: if
there
> > > is
> > > no RHIST line type in the stat file, which indicates there is no
> > > matching
> > > pair at all between obs. and forecast, is that correct?
> > >
> > > Thank you.
> > >
> > > On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > >> Binyu,
> > >> All 5 ensemble_stat files were successfully loaded to the DB
and the
> > >> data
> > >> is available for potting. Here is the example: 20200826_164848.
> > >> Your screenshot displays that you try to create a histogram
plot.
> > >> The
> > >> data for this plot comes from line_data_rhist table. This table
gets
> > >> filled
> > >> with 'RHIST' line type.
> > >> 'ensemble_stat_Reventador_hr04_20190225_200000V.stat' file
contains
> > >> only
> > >> 'ORANK' line type so non of it's records were saved into
> > >> line_data_rhist
> > >> table.
> > >> Currently line_data_rhist table contains data for
fcst_valid_beg:
> > >> '2019-02-25 18:00:00' and '2019-02-25 19:00:00' ( fcst_lead:
20000
> > >> and
> > >> 30000). The data came from
> > >> ensemble_stat_Reventador_hr02_20190225_180000V.stat and
> > >> ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > >>
> > >> Could you check if other files have records with RHIST line
type.
> > >>
> > >> Tatiana
> > >>
> > >> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov wrote:
> > >> > Tatiana,
> > >> >
> > >> > Yes that is weird. If you see the attached file, you can see
> > >> > there only two files were uploaded. I expect to have all 5.
> > >> > Binyu
> > >> >
> > >> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek via RT
> > >> > <met_help at ucar.edu>
> > >> > wrote:
> > >> >
> > >> > > Binyu,
> > >> > >
> > >> > > In your log file there is a record that
> > >> > > 'ensemble_stat_Reventador_hr04_20190225_200000V.stat' was
> > >> > > successfully
> > >> > > loaded - all 53 lines.
> > >> > > I also can create a plot using the data from this file -
ORANK
> > >> > > statistics
> > >> > > with 40000 forecast lead: 20200826_151853.
> > >> > >
> > >> > > Tatiana
> > >> > >
> > >> > >
> > >> > > On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov wrote:
> > >> > > > Tatiana,
> > >> > > >
> > >> > > > I didn't see any error from the log file. I attached the
log
> > >> > > > file
> > >> > > > and two
> > >> > > > of the stat files:
> > >> > > > ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > >> > > is
> > >> > > > readable from METViewer, but
> > >> > > > "ensemble_stat_Reventador_hr04_20190225_200000V.stat"
not.
> > >> > > >
> > >> > > > Thank you.
> > >> > > >
> > >> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek via RT
> > >> > > > <met_help at ucar.edu>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Hi Binyu,
> > >> > > > > I assume that your files are on WCOSS. I don't have an
> > >> > > > > access to
> > >> > > > > the
> > >> > > > > server and can't see the files.
> > >> > > > > Do you have logging messages from the load? Any errors?
> > >> > > > >
> > >> > > > > I see that only 5 ensemble_stat files were loaded to
this
> > >> > > > > database:
> > >> > > > > | ensemble_stat_Reventador_hr02_20190225_180000V.stat |
> > >> > > > > | ensemble_stat_Reventador_hr03_20190225_190000V.stat |
> > >> > > > > | ensemble_stat_Reventador_hr04_20190225_200000V.stat |
> > >> > > > > | ensemble_stat_Reventador_hr05_20190225_210000V.stat |
> > >> > > > > | ensemble_stat_Reventador_hr06_20190225_220000V.stat
> > >> > > > >
> > >> > > > > Tatiana
> > >> > > > >
> > >> > > > > On Tue Aug 25 20:02:54 2020, binyu.wang at noaa.gov wrote:
> > >> > > > > > Hello,
> > >> > > > > >
> > >> > > > > > I created 5 ensemble_stat* files (from hr18- hr22)
under
> > >> > > > > > the
> > >> > > directory
> > >> > > > > > below, but only the first two (hr18 and hr19) could
be
> > >> > > > > > read
> > >> > > > > > from
> > >> > > > > > METVIewer. Those files are under database
> > >> > > "mv_g2g_met_verf_Reventador"
> > >> > > > > >
> > >> > > > > >
> > >> > >
> > >>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > >> > > > > >
> > >> > > > > > What is the problem? Thank you.
> > >> > > > > > Binyu
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >>
> > >>
> > >>
> > >>
>
>
>
>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: binyu.wang at noaa.gov
Time: Thu Aug 27 12:04:01 2020
Another quick question: Can we change the database name without having
to
drop the database and reload all of the data again?
Binyu
On Thu, Aug 27, 2020 at 2:03 PM Binyu Wang - NOAA Affiliate <
binyu.wang at noaa.gov> wrote:
> Tatiana,
>
> "PCT" is generated later using the ensemble*stat files which I sent
to you
> earlier. So you didn't see it in those two files. I didn't
understand: if
> the 3 files didn't have "RHIST" because there was no matching pair
> during that time period, why could they be used for RELY plot?
>
> Binyu
>
> On Thu, Aug 27, 2020 at 1:43 PM Tatiana Burek via RT
<met_help at ucar.edu>
> wrote:
>
>> Binyu
>>
>> Reliability plot uses data from line_data_pct table ( "PCT" line
type ).
>> Both stat files that you sent me don't contain PCT line type and
variable
>> 'VAFTD_ENS_FREQ_ge0.1'. Did you have data in this database before
the
>> latest load?
>>
>> Tatiana
>>
>> On Thu Aug 27 08:48:35 2020, binyu.wang at noaa.gov wrote:
>> > Hello Tatiana,
>> >
>> > Sorry to ask more: as I said in previous mail I can not find the
other
>> > 3
>> > files from MET_Viewer because there is no RHIST inside of those
files,
>> > if
>> > this is because there is no matched pairs between obs. and
forecast,
>> > why I
>> > can see them in the RELY plot? (see the attachment please). Are
RELY
>> > plots
>> > created based on matching pairs?
>> >
>> > Thank you.
>> > Binyu
>> >
>> >
>> > On Wed, Aug 26, 2020 at 10:47 PM Binyu Wang - NOAA Affiliate <
>> > binyu.wang at noaa.gov> wrote:
>> >
>> > > Hello,
>> > >
>> > > I think I know the answer now, but just to confirm with you: if
there
>> > > is
>> > > no RHIST line type in the stat file, which indicates there is
no
>> > > matching
>> > > pair at all between obs. and forecast, is that correct?
>> > >
>> > > Thank you.
>> > >
>> > > On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via RT
>> > > <met_help at ucar.edu>
>> > > wrote:
>> > >
>> > >> Binyu,
>> > >> All 5 ensemble_stat files were successfully loaded to the DB
and the
>> > >> data
>> > >> is available for potting. Here is the example:
20200826_164848.
>> > >> Your screenshot displays that you try to create a histogram
plot.
>> > >> The
>> > >> data for this plot comes from line_data_rhist table. This
table gets
>> > >> filled
>> > >> with 'RHIST' line type.
>> > >> 'ensemble_stat_Reventador_hr04_20190225_200000V.stat' file
contains
>> > >> only
>> > >> 'ORANK' line type so non of it's records were saved into
>> > >> line_data_rhist
>> > >> table.
>> > >> Currently line_data_rhist table contains data for
fcst_valid_beg:
>> > >> '2019-02-25 18:00:00' and '2019-02-25 19:00:00' ( fcst_lead:
20000
>> > >> and
>> > >> 30000). The data came from
>> > >> ensemble_stat_Reventador_hr02_20190225_180000V.stat and
>> > >> ensemble_stat_Reventador_hr03_20190225_190000V.stat
>> > >>
>> > >> Could you check if other files have records with RHIST line
type.
>> > >>
>> > >> Tatiana
>> > >>
>> > >> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov wrote:
>> > >> > Tatiana,
>> > >> >
>> > >> > Yes that is weird. If you see the attached file, you can see
>> > >> > there only two files were uploaded. I expect to have all 5.
>> > >> > Binyu
>> > >> >
>> > >> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek via RT
>> > >> > <met_help at ucar.edu>
>> > >> > wrote:
>> > >> >
>> > >> > > Binyu,
>> > >> > >
>> > >> > > In your log file there is a record that
>> > >> > > 'ensemble_stat_Reventador_hr04_20190225_200000V.stat' was
>> > >> > > successfully
>> > >> > > loaded - all 53 lines.
>> > >> > > I also can create a plot using the data from this file -
ORANK
>> > >> > > statistics
>> > >> > > with 40000 forecast lead: 20200826_151853.
>> > >> > >
>> > >> > > Tatiana
>> > >> > >
>> > >> > >
>> > >> > > On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov wrote:
>> > >> > > > Tatiana,
>> > >> > > >
>> > >> > > > I didn't see any error from the log file. I attached the
log
>> > >> > > > file
>> > >> > > > and two
>> > >> > > > of the stat files:
>> > >> > > > ensemble_stat_Reventador_hr03_20190225_190000V.stat
>> > >> > > is
>> > >> > > > readable from METViewer, but
>> > >> > > > "ensemble_stat_Reventador_hr04_20190225_200000V.stat"
not.
>> > >> > > >
>> > >> > > > Thank you.
>> > >> > > >
>> > >> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek via RT
>> > >> > > > <met_help at ucar.edu>
>> > >> > > > wrote:
>> > >> > > >
>> > >> > > > > Hi Binyu,
>> > >> > > > > I assume that your files are on WCOSS. I don't have an
>> > >> > > > > access to
>> > >> > > > > the
>> > >> > > > > server and can't see the files.
>> > >> > > > > Do you have logging messages from the load? Any
errors?
>> > >> > > > >
>> > >> > > > > I see that only 5 ensemble_stat files were loaded to
this
>> > >> > > > > database:
>> > >> > > > > | ensemble_stat_Reventador_hr02_20190225_180000V.stat
|
>> > >> > > > > | ensemble_stat_Reventador_hr03_20190225_190000V.stat
|
>> > >> > > > > | ensemble_stat_Reventador_hr04_20190225_200000V.stat
|
>> > >> > > > > | ensemble_stat_Reventador_hr05_20190225_210000V.stat
|
>> > >> > > > > | ensemble_stat_Reventador_hr06_20190225_220000V.stat
>> > >> > > > >
>> > >> > > > > Tatiana
>> > >> > > > >
>> > >> > > > > On Tue Aug 25 20:02:54 2020, binyu.wang at noaa.gov
wrote:
>> > >> > > > > > Hello,
>> > >> > > > > >
>> > >> > > > > > I created 5 ensemble_stat* files (from hr18- hr22)
under
>> > >> > > > > > the
>> > >> > > directory
>> > >> > > > > > below, but only the first two (hr18 and hr19) could
be
>> > >> > > > > > read
>> > >> > > > > > from
>> > >> > > > > > METVIewer. Those files are under database
>> > >> > > "mv_g2g_met_verf_Reventador"
>> > >> > > > > >
>> > >> > > > > >
>> > >> > >
>> > >>
>>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
>> > >> > > > > >
>> > >> > > > > > What is the problem? Thank you.
>> > >> > > > > > Binyu
>> > >> > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > >
>> > >>
>> > >>
>> > >>
>> > >>
>>
>>
>>
>>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: Tatiana Burek
Time: Thu Aug 27 12:15:54 2020
There is no easy way to rename the database.
I can create a new database with a new name, dump the data from the
old database, load it to the new one and finally remove the old
database.
Tatiana
On Thu Aug 27 12:04:00 2020, binyu.wang at noaa.gov wrote:
> Another quick question: Can we change the database name without
having to
> drop the database and reload all of the data again?
>
> Binyu
>
> On Thu, Aug 27, 2020 at 2:03 PM Binyu Wang - NOAA Affiliate <
> binyu.wang at noaa.gov> wrote:
>
> > Tatiana,
> >
> > "PCT" is generated later using the ensemble*stat files which I
sent to you
> > earlier. So you didn't see it in those two files. I didn't
understand: if
> > the 3 files didn't have "RHIST" because there was no matching
pair
> > during that time period, why could they be used for RELY plot?
> >
> > Binyu
> >
> > On Thu, Aug 27, 2020 at 1:43 PM Tatiana Burek via RT
<met_help at ucar.edu>
> > wrote:
> >
> >> Binyu
> >>
> >> Reliability plot uses data from line_data_pct table ( "PCT" line
type ).
> >> Both stat files that you sent me don't contain PCT line type and
variable
> >> 'VAFTD_ENS_FREQ_ge0.1'. Did you have data in this database before
the
> >> latest load?
> >>
> >> Tatiana
> >>
> >> On Thu Aug 27 08:48:35 2020, binyu.wang at noaa.gov wrote:
> >> > Hello Tatiana,
> >> >
> >> > Sorry to ask more: as I said in previous mail I can not find
the other
> >> > 3
> >> > files from MET_Viewer because there is no RHIST inside of those
files,
> >> > if
> >> > this is because there is no matched pairs between obs. and
forecast,
> >> > why I
> >> > can see them in the RELY plot? (see the attachment please). Are
RELY
> >> > plots
> >> > created based on matching pairs?
> >> >
> >> > Thank you.
> >> > Binyu
> >> >
> >> >
> >> > On Wed, Aug 26, 2020 at 10:47 PM Binyu Wang - NOAA Affiliate <
> >> > binyu.wang at noaa.gov> wrote:
> >> >
> >> > > Hello,
> >> > >
> >> > > I think I know the answer now, but just to confirm with you:
if there
> >> > > is
> >> > > no RHIST line type in the stat file, which indicates there is
no
> >> > > matching
> >> > > pair at all between obs. and forecast, is that correct?
> >> > >
> >> > > Thank you.
> >> > >
> >> > > On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via RT
> >> > > <met_help at ucar.edu>
> >> > > wrote:
> >> > >
> >> > >> Binyu,
> >> > >> All 5 ensemble_stat files were successfully loaded to the DB
and the
> >> > >> data
> >> > >> is available for potting. Here is the example:
20200826_164848.
> >> > >> Your screenshot displays that you try to create a histogram
plot.
> >> > >> The
> >> > >> data for this plot comes from line_data_rhist table. This
table gets
> >> > >> filled
> >> > >> with 'RHIST' line type.
> >> > >> 'ensemble_stat_Reventador_hr04_20190225_200000V.stat' file
contains
> >> > >> only
> >> > >> 'ORANK' line type so non of it's records were saved into
> >> > >> line_data_rhist
> >> > >> table.
> >> > >> Currently line_data_rhist table contains data for
fcst_valid_beg:
> >> > >> '2019-02-25 18:00:00' and '2019-02-25 19:00:00' ( fcst_lead:
20000
> >> > >> and
> >> > >> 30000). The data came from
> >> > >> ensemble_stat_Reventador_hr02_20190225_180000V.stat and
> >> > >> ensemble_stat_Reventador_hr03_20190225_190000V.stat
> >> > >>
> >> > >> Could you check if other files have records with RHIST line
type.
> >> > >>
> >> > >> Tatiana
> >> > >>
> >> > >> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov wrote:
> >> > >> > Tatiana,
> >> > >> >
> >> > >> > Yes that is weird. If you see the attached file, you can
see
> >> > >> > there only two files were uploaded. I expect to have all
5.
> >> > >> > Binyu
> >> > >> >
> >> > >> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek via RT
> >> > >> > <met_help at ucar.edu>
> >> > >> > wrote:
> >> > >> >
> >> > >> > > Binyu,
> >> > >> > >
> >> > >> > > In your log file there is a record that
> >> > >> > > 'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
was
> >> > >> > > successfully
> >> > >> > > loaded - all 53 lines.
> >> > >> > > I also can create a plot using the data from this file -
ORANK
> >> > >> > > statistics
> >> > >> > > with 40000 forecast lead: 20200826_151853.
> >> > >> > >
> >> > >> > > Tatiana
> >> > >> > >
> >> > >> > >
> >> > >> > > On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov wrote:
> >> > >> > > > Tatiana,
> >> > >> > > >
> >> > >> > > > I didn't see any error from the log file. I attached
the log
> >> > >> > > > file
> >> > >> > > > and two
> >> > >> > > > of the stat files:
> >> > >> > > > ensemble_stat_Reventador_hr03_20190225_190000V.stat
> >> > >> > > is
> >> > >> > > > readable from METViewer, but
> >> > >> > > > "ensemble_stat_Reventador_hr04_20190225_200000V.stat"
not.
> >> > >> > > >
> >> > >> > > > Thank you.
> >> > >> > > >
> >> > >> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek via RT
> >> > >> > > > <met_help at ucar.edu>
> >> > >> > > > wrote:
> >> > >> > > >
> >> > >> > > > > Hi Binyu,
> >> > >> > > > > I assume that your files are on WCOSS. I don't have
an
> >> > >> > > > > access to
> >> > >> > > > > the
> >> > >> > > > > server and can't see the files.
> >> > >> > > > > Do you have logging messages from the load? Any
errors?
> >> > >> > > > >
> >> > >> > > > > I see that only 5 ensemble_stat files were loaded to
this
> >> > >> > > > > database:
> >> > >> > > > > |
ensemble_stat_Reventador_hr02_20190225_180000V.stat |
> >> > >> > > > > |
ensemble_stat_Reventador_hr03_20190225_190000V.stat |
> >> > >> > > > > |
ensemble_stat_Reventador_hr04_20190225_200000V.stat |
> >> > >> > > > > |
ensemble_stat_Reventador_hr05_20190225_210000V.stat |
> >> > >> > > > > |
ensemble_stat_Reventador_hr06_20190225_220000V.stat
> >> > >> > > > >
> >> > >> > > > > Tatiana
> >> > >> > > > >
> >> > >> > > > > On Tue Aug 25 20:02:54 2020, binyu.wang at noaa.gov
wrote:
> >> > >> > > > > > Hello,
> >> > >> > > > > >
> >> > >> > > > > > I created 5 ensemble_stat* files (from hr18- hr22)
under
> >> > >> > > > > > the
> >> > >> > > directory
> >> > >> > > > > > below, but only the first two (hr18 and hr19)
could be
> >> > >> > > > > > read
> >> > >> > > > > > from
> >> > >> > > > > > METVIewer. Those files are under database
> >> > >> > > "mv_g2g_met_verf_Reventador"
> >> > >> > > > > >
> >> > >> > > > > >
> >> > >> > >
> >> > >>
> >>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> >> > >> > > > > >
> >> > >> > > > > > What is the problem? Thank you.
> >> > >> > > > > > Binyu
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >>
> >> > >>
> >> > >>
> >> > >>
> >>
> >>
> >>
> >>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: binyu.wang at noaa.gov
Time: Thu Aug 27 13:41:41 2020
Thank you very much for the detailed explanation, John.
So basically, that means there is no any matched case (or site b/c I
used
point_obs) between obs. and forecasting that meet all the
threshold/censors for the last 3 files. If you look at part of the log
file
below (this is for the 3rd file):
DEBUG 2: Processing point verification VAFTD/L14000-0 versus
ash/Z10000-0,
for observation type ADPUPA, over region FULL, for interpolation
method
NEAREST(1), using 53 matched pairs.
DEBUG 2: Skipping 53 of 53 points due to missing ensemble values.
DEBUG 3: Using 0 of 53 ensemble pairs for observation filtering
threshold
>=0.1.
So what is the difference between "matched pairs" and "ensemble
pairs"? It
seems "ensemble pairs is 0" is why I got no "PHIST", how about the
matched
pairs? Does that mean if one site was catched by all 20 ensemble
members,
then I have 20 matched pairs?
Thank you.
On Thu, Aug 27, 2020 at 2:01 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> Tatiana asked me to take a look at the RHIST line type being
unexpectedly
> absent from some of your runs of Ensemble-Stat. I logged onto WCOSS
and
> took a look in:
>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
>
> In there, I see 5 ensemble_stat output files. I have listed each
below
> followed by the line types which appear in each file. So that's
column #24
> from each:
>
> ensemble_stat_Reventador_hr02_20190225_180000V.stat ECNT ORANK PHIST
RELP
> RHIST SSVAR
>
> ensemble_stat_Reventador_hr03_20190225_190000V.stat ECNT ORANK PHIST
RELP
> RHIST SSVAR
>
> ensemble_stat_Reventador_hr04_20190225_200000V.stat ORANK
>
> ensemble_stat_Reventador_hr05_20190225_210000V.stat ORANK
>
> ensemble_stat_Reventador_hr06_20190225_220000V.stat ORANK
>
>
> The ORANK line type is basically the "matched pairs" for ensemble
> verification. Notice that the first 2 hours have the ORANK lines
along with
> other statistic line types, such as RHIST. However, the last 3 hours
only
> have the ORANK line type.
>
>
> Looking at the output for the first 2 hours, I see that you've
applied the
> OBS_THRESH filtering threshold as ">=0.1".
>
>
> Here's a link to the table which defines the contents of the ORANK
line
> type:
>
>
> https://dtcenter.github.io/MET/Users_Guide/ensemble-
stat.html?highlight=orank%20line%20type#id7
>
>
> Note that the actual observation rank value is in column #34.
Checking the
> data in those last 3 files, you'll notice that the observation ranks
are
> always listed as bad data:
>
>
> cat ensemble_stat_Reventador_hr04_20190225_200000V.stat
> ensemble_stat_Reventador_hr05_20190225_210000V.stat
> ensemble_stat_Reventador_hr06_20190225_220000V.stat | grep ORANK |
awk
> '{print $34}' | sort -u
>
>
> NA
>
>
> So now the question is why? You're using the "censor_thresh" and
> "censor_val" options to reset the forecast values that are "<0.1" to
bad
> data. So that introduces a lot of bad data values. And that results
in the
> rank of the observation being computed as bad data. This is likely
> controlled by how you have the "vld_thresh" or "ens_vld_thresh" set
in your
> Ensemble-Stat configuration file.
>
>
> John
>
> On Thu, Aug 27, 2020 at 8:48 AM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453 >
> >
> > Hello Tatiana,
> >
> > Sorry to ask more: as I said in previous mail I can not find the
other 3
> > files from MET_Viewer because there is no RHIST inside of those
files, if
> > this is because there is no matched pairs between obs. and
forecast, why
> I
> > can see them in the RELY plot? (see the attachment please). Are
RELY
> plots
> > created based on matching pairs?
> >
> > Thank you.
> > Binyu
> >
> >
> > On Wed, Aug 26, 2020 at 10:47 PM Binyu Wang - NOAA Affiliate <
> > binyu.wang at noaa.gov> wrote:
> >
> > > Hello,
> > >
> > > I think I know the answer now, but just to confirm with you: if
there
> is
> > > no RHIST line type in the stat file, which indicates there is no
> matching
> > > pair at all between obs. and forecast, is that correct?
> > >
> > > Thank you.
> > >
> > > On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > >> Binyu,
> > >> All 5 ensemble_stat files were successfully loaded to the DB
and the
> > data
> > >> is available for potting. Here is the example: 20200826_164848.
> > >> Your screenshot displays that you try to create a histogram
plot. The
> > >> data for this plot comes from line_data_rhist table. This table
gets
> > filled
> > >> with 'RHIST' line type.
> > >> 'ensemble_stat_Reventador_hr04_20190225_200000V.stat' file
contains
> only
> > >> 'ORANK' line type so non of it's records were saved into
> line_data_rhist
> > >> table.
> > >> Currently line_data_rhist table contains data for
fcst_valid_beg:
> > >> '2019-02-25 18:00:00' and '2019-02-25 19:00:00' ( fcst_lead:
20000 and
> > >> 30000). The data came from
> > >> ensemble_stat_Reventador_hr02_20190225_180000V.stat and
> > >> ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > >>
> > >> Could you check if other files have records with RHIST line
type.
> > >>
> > >> Tatiana
> > >>
> > >> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov wrote:
> > >> > Tatiana,
> > >> >
> > >> > Yes that is weird. If you see the attached file, you can see
> > >> > there only two files were uploaded. I expect to have all 5.
> > >> > Binyu
> > >> >
> > >> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek via RT
> > >> > <met_help at ucar.edu>
> > >> > wrote:
> > >> >
> > >> > > Binyu,
> > >> > >
> > >> > > In your log file there is a record that
> > >> > > 'ensemble_stat_Reventador_hr04_20190225_200000V.stat' was
> > >> > > successfully
> > >> > > loaded - all 53 lines.
> > >> > > I also can create a plot using the data from this file -
ORANK
> > >> > > statistics
> > >> > > with 40000 forecast lead: 20200826_151853.
> > >> > >
> > >> > > Tatiana
> > >> > >
> > >> > >
> > >> > > On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov wrote:
> > >> > > > Tatiana,
> > >> > > >
> > >> > > > I didn't see any error from the log file. I attached the
log
> file
> > >> > > > and two
> > >> > > > of the stat files:
> > >> > > > ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > >> > > is
> > >> > > > readable from METViewer, but
> > >> > > > "ensemble_stat_Reventador_hr04_20190225_200000V.stat"
not.
> > >> > > >
> > >> > > > Thank you.
> > >> > > >
> > >> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek via RT
> > >> > > > <met_help at ucar.edu>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Hi Binyu,
> > >> > > > > I assume that your files are on WCOSS. I don't have an
access
> to
> > >> > > > > the
> > >> > > > > server and can't see the files.
> > >> > > > > Do you have logging messages from the load? Any errors?
> > >> > > > >
> > >> > > > > I see that only 5 ensemble_stat files were loaded to
this
> > >> > > > > database:
> > >> > > > > | ensemble_stat_Reventador_hr02_20190225_180000V.stat |
> > >> > > > > | ensemble_stat_Reventador_hr03_20190225_190000V.stat |
> > >> > > > > | ensemble_stat_Reventador_hr04_20190225_200000V.stat |
> > >> > > > > | ensemble_stat_Reventador_hr05_20190225_210000V.stat |
> > >> > > > > | ensemble_stat_Reventador_hr06_20190225_220000V.stat
> > >> > > > >
> > >> > > > > Tatiana
> > >> > > > >
> > >> > > > > On Tue Aug 25 20:02:54 2020, binyu.wang at noaa.gov wrote:
> > >> > > > > > Hello,
> > >> > > > > >
> > >> > > > > > I created 5 ensemble_stat* files (from hr18- hr22)
under
> the
> > >> > > directory
> > >> > > > > > below, but only the first two (hr18 and hr19) could
be read
> > >> > > > > > from
> > >> > > > > > METVIewer. Those files are under database
> > >> > > "mv_g2g_met_verf_Reventador"
> > >> > > > > >
> > >> > > > > >
> > >> > >
> > >>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > >> > > > > >
> > >> > > > > > What is the problem? Thank you.
> > >> > > > > > Binyu
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >>
> > >>
> > >>
> > >>
> >
> >
>
>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: John Halley Gotway
Time: Thu Aug 27 14:00:42 2020
Binyu,
In ensemble-stat, there is no difference between "matched pairs" and
"ensemble pairs". It would be better if we made those log messages
use
consistent language.
So regarding this log message:
*DEBUG 2: Skipping 53 of 53 points due to missing ensemble values.*
Here's the line of code where that's being written:
https://github.com/dtcenter/MET/blob/2c9ae440a84024fbf62caa64c1b747f9a912236f/met/src/libcode/vx_statistics/pair_data_ensemble.cc#L432
So the rule here is that if any of the ensemble members contain bad
data
for a particular observation, then that point is not used in the
verification. And why not? We ultimately compute the rank of the
observation value, relative to the ensemble member values. If the
ensemble
member values contain bad data, there's no good way of computing the
rank
from them.
So when you use the "censor_thresh" and "censor_val" logic on the
input
ensemble fields, you're introducing lots of bad data values into your
forecast fields. For hours 2 and 3, there are 17 and 11 points,
respectively, for which none of the ensemble members contain bad data.
But
for the rest of the hours, there are 0 points for which none of the
ensemble members contain bad data.
So I would recommend that you stop using the "censor_thresh" and
"censor_val" logic. It think it's confusing your results. Requiring
that
ALL ensemble members and the observation be >=0.1 results in no pairs
left
with which to compute statistics.
John
On Thu, Aug 27, 2020 at 1:41 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453 >
>
> Thank you very much for the detailed explanation, John.
>
> So basically, that means there is no any matched case (or site b/c I
used
> point_obs) between obs. and forecasting that meet all the
> threshold/censors for the last 3 files. If you look at part of the
log file
> below (this is for the 3rd file):
>
> DEBUG 2: Processing point verification VAFTD/L14000-0 versus
ash/Z10000-0,
> for observation type ADPUPA, over region FULL, for interpolation
method
> NEAREST(1), using 53 matched pairs.
> DEBUG 2: Skipping 53 of 53 points due to missing ensemble values.
> DEBUG 3: Using 0 of 53 ensemble pairs for observation filtering
threshold
> >=0.1.
>
> So what is the difference between "matched pairs" and "ensemble
pairs"? It
> seems "ensemble pairs is 0" is why I got no "PHIST", how about the
matched
> pairs? Does that mean if one site was catched by all 20 ensemble
members,
> then I have 20 matched pairs?
>
> Thank you.
>
> On Thu, Aug 27, 2020 at 2:01 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > Tatiana asked me to take a look at the RHIST line type being
unexpectedly
> > absent from some of your runs of Ensemble-Stat. I logged onto
WCOSS and
> > took a look in:
> >
> >
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> >
> > In there, I see 5 ensemble_stat output files. I have listed each
below
> > followed by the line types which appear in each file. So that's
column
> #24
> > from each:
> >
> > ensemble_stat_Reventador_hr02_20190225_180000V.stat ECNT ORANK
PHIST RELP
> > RHIST SSVAR
> >
> > ensemble_stat_Reventador_hr03_20190225_190000V.stat ECNT ORANK
PHIST RELP
> > RHIST SSVAR
> >
> > ensemble_stat_Reventador_hr04_20190225_200000V.stat ORANK
> >
> > ensemble_stat_Reventador_hr05_20190225_210000V.stat ORANK
> >
> > ensemble_stat_Reventador_hr06_20190225_220000V.stat ORANK
> >
> >
> > The ORANK line type is basically the "matched pairs" for ensemble
> > verification. Notice that the first 2 hours have the ORANK lines
along
> with
> > other statistic line types, such as RHIST. However, the last 3
hours only
> > have the ORANK line type.
> >
> >
> > Looking at the output for the first 2 hours, I see that you've
applied
> the
> > OBS_THRESH filtering threshold as ">=0.1".
> >
> >
> > Here's a link to the table which defines the contents of the ORANK
line
> > type:
> >
> >
> >
> https://dtcenter.github.io/MET/Users_Guide/ensemble-
stat.html?highlight=orank%20line%20type#id7
> >
> >
> > Note that the actual observation rank value is in column #34.
Checking
> the
> > data in those last 3 files, you'll notice that the observation
ranks are
> > always listed as bad data:
> >
> >
> > cat ensemble_stat_Reventador_hr04_20190225_200000V.stat
> > ensemble_stat_Reventador_hr05_20190225_210000V.stat
> > ensemble_stat_Reventador_hr06_20190225_220000V.stat | grep ORANK
| awk
> > '{print $34}' | sort -u
> >
> >
> > NA
> >
> >
> > So now the question is why? You're using the "censor_thresh" and
> > "censor_val" options to reset the forecast values that are "<0.1"
to bad
> > data. So that introduces a lot of bad data values. And that
results in
> the
> > rank of the observation being computed as bad data. This is likely
> > controlled by how you have the "vld_thresh" or "ens_vld_thresh"
set in
> your
> > Ensemble-Stat configuration file.
> >
> >
> > John
> >
> > On Thu, Aug 27, 2020 at 8:48 AM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453 >
> > >
> > > Hello Tatiana,
> > >
> > > Sorry to ask more: as I said in previous mail I can not find the
other
> 3
> > > files from MET_Viewer because there is no RHIST inside of those
files,
> if
> > > this is because there is no matched pairs between obs. and
forecast,
> why
> > I
> > > can see them in the RELY plot? (see the attachment please). Are
RELY
> > plots
> > > created based on matching pairs?
> > >
> > > Thank you.
> > > Binyu
> > >
> > >
> > > On Wed, Aug 26, 2020 at 10:47 PM Binyu Wang - NOAA Affiliate <
> > > binyu.wang at noaa.gov> wrote:
> > >
> > > > Hello,
> > > >
> > > > I think I know the answer now, but just to confirm with you:
if there
> > is
> > > > no RHIST line type in the stat file, which indicates there is
no
> > matching
> > > > pair at all between obs. and forecast, is that correct?
> > > >
> > > > Thank you.
> > > >
> > > > On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via RT <
> > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > >> Binyu,
> > > >> All 5 ensemble_stat files were successfully loaded to the DB
and the
> > > data
> > > >> is available for potting. Here is the example:
20200826_164848.
> > > >> Your screenshot displays that you try to create a histogram
plot.
> The
> > > >> data for this plot comes from line_data_rhist table. This
table gets
> > > filled
> > > >> with 'RHIST' line type.
> > > >> 'ensemble_stat_Reventador_hr04_20190225_200000V.stat' file
contains
> > only
> > > >> 'ORANK' line type so non of it's records were saved into
> > line_data_rhist
> > > >> table.
> > > >> Currently line_data_rhist table contains data for
fcst_valid_beg:
> > > >> '2019-02-25 18:00:00' and '2019-02-25 19:00:00' ( fcst_lead:
20000
> and
> > > >> 30000). The data came from
> > > >> ensemble_stat_Reventador_hr02_20190225_180000V.stat and
> > > >> ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > >>
> > > >> Could you check if other files have records with RHIST line
type.
> > > >>
> > > >> Tatiana
> > > >>
> > > >> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov wrote:
> > > >> > Tatiana,
> > > >> >
> > > >> > Yes that is weird. If you see the attached file, you can
see
> > > >> > there only two files were uploaded. I expect to have all
5.
> > > >> > Binyu
> > > >> >
> > > >> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek via RT
> > > >> > <met_help at ucar.edu>
> > > >> > wrote:
> > > >> >
> > > >> > > Binyu,
> > > >> > >
> > > >> > > In your log file there is a record that
> > > >> > > 'ensemble_stat_Reventador_hr04_20190225_200000V.stat' was
> > > >> > > successfully
> > > >> > > loaded - all 53 lines.
> > > >> > > I also can create a plot using the data from this file -
ORANK
> > > >> > > statistics
> > > >> > > with 40000 forecast lead: 20200826_151853.
> > > >> > >
> > > >> > > Tatiana
> > > >> > >
> > > >> > >
> > > >> > > On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov wrote:
> > > >> > > > Tatiana,
> > > >> > > >
> > > >> > > > I didn't see any error from the log file. I attached
the log
> > file
> > > >> > > > and two
> > > >> > > > of the stat files:
> > > >> > > > ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > >> > > is
> > > >> > > > readable from METViewer, but
> > > >> > > > "ensemble_stat_Reventador_hr04_20190225_200000V.stat"
not.
> > > >> > > >
> > > >> > > > Thank you.
> > > >> > > >
> > > >> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek via RT
> > > >> > > > <met_help at ucar.edu>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Hi Binyu,
> > > >> > > > > I assume that your files are on WCOSS. I don't have
an
> access
> > to
> > > >> > > > > the
> > > >> > > > > server and can't see the files.
> > > >> > > > > Do you have logging messages from the load? Any
errors?
> > > >> > > > >
> > > >> > > > > I see that only 5 ensemble_stat files were loaded to
this
> > > >> > > > > database:
> > > >> > > > > | ensemble_stat_Reventador_hr02_20190225_180000V.stat
|
> > > >> > > > > | ensemble_stat_Reventador_hr03_20190225_190000V.stat
|
> > > >> > > > > | ensemble_stat_Reventador_hr04_20190225_200000V.stat
|
> > > >> > > > > | ensemble_stat_Reventador_hr05_20190225_210000V.stat
|
> > > >> > > > > | ensemble_stat_Reventador_hr06_20190225_220000V.stat
> > > >> > > > >
> > > >> > > > > Tatiana
> > > >> > > > >
> > > >> > > > > On Tue Aug 25 20:02:54 2020, binyu.wang at noaa.gov
wrote:
> > > >> > > > > > Hello,
> > > >> > > > > >
> > > >> > > > > > I created 5 ensemble_stat* files (from hr18- hr22)
under
> > the
> > > >> > > directory
> > > >> > > > > > below, but only the first two (hr18 and hr19)
could be
> read
> > > >> > > > > > from
> > > >> > > > > > METVIewer. Those files are under database
> > > >> > > "mv_g2g_met_verf_Reventador"
> > > >> > > > > >
> > > >> > > > > >
> > > >> > >
> > > >>
> >
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > > >> > > > > >
> > > >> > > > > > What is the problem? Thank you.
> > > >> > > > > > Binyu
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > >
> > > >>
> > > >>
> > > >>
> > > >>
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: binyu.wang at noaa.gov
Time: Thu Aug 27 14:32:09 2020
Thank you very much for the clarification. I will discuss with my
advisor.
I think we will need to keep the " censor_thresh ", but we can change
the
"vld_thresh" or "ens_vld_thresh" .
Binyu
On Thu, Aug 27, 2020 at 4:00 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> In ensemble-stat, there is no difference between "matched pairs" and
> "ensemble pairs". It would be better if we made those log messages
use
> consistent language.
>
> So regarding this log message:
> *DEBUG 2: Skipping 53 of 53 points due to missing ensemble values.*
>
> Here's the line of code where that's being written:
>
>
https://github.com/dtcenter/MET/blob/2c9ae440a84024fbf62caa64c1b747f9a912236f/met/src/libcode/vx_statistics/pair_data_ensemble.cc#L432
>
> So the rule here is that if any of the ensemble members contain bad
data
> for a particular observation, then that point is not used in the
> verification. And why not? We ultimately compute the rank of the
> observation value, relative to the ensemble member values. If the
ensemble
> member values contain bad data, there's no good way of computing the
rank
> from them.
>
> So when you use the "censor_thresh" and "censor_val" logic on the
input
> ensemble fields, you're introducing lots of bad data values into
your
> forecast fields. For hours 2 and 3, there are 17 and 11 points,
> respectively, for which none of the ensemble members contain bad
data. But
> for the rest of the hours, there are 0 points for which none of the
> ensemble members contain bad data.
>
> So I would recommend that you stop using the "censor_thresh" and
> "censor_val" logic. It think it's confusing your results. Requiring
that
> ALL ensemble members and the observation be >=0.1 results in no
pairs left
> with which to compute statistics.
>
> John
>
> On Thu, Aug 27, 2020 at 1:41 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453 >
> >
> > Thank you very much for the detailed explanation, John.
> >
> > So basically, that means there is no any matched case (or site b/c
I used
> > point_obs) between obs. and forecasting that meet all the
> > threshold/censors for the last 3 files. If you look at part of the
log
> file
> > below (this is for the 3rd file):
> >
> > DEBUG 2: Processing point verification VAFTD/L14000-0 versus
> ash/Z10000-0,
> > for observation type ADPUPA, over region FULL, for interpolation
method
> > NEAREST(1), using 53 matched pairs.
> > DEBUG 2: Skipping 53 of 53 points due to missing ensemble values.
> > DEBUG 3: Using 0 of 53 ensemble pairs for observation filtering
threshold
> > >=0.1.
> >
> > So what is the difference between "matched pairs" and "ensemble
pairs"?
> It
> > seems "ensemble pairs is 0" is why I got no "PHIST", how about the
> matched
> > pairs? Does that mean if one site was catched by all 20 ensemble
> members,
> > then I have 20 matched pairs?
> >
> > Thank you.
> >
> > On Thu, Aug 27, 2020 at 2:01 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Binyu,
> > >
> > > Tatiana asked me to take a look at the RHIST line type being
> unexpectedly
> > > absent from some of your runs of Ensemble-Stat. I logged onto
WCOSS and
> > > took a look in:
> > >
> > >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > >
> > > In there, I see 5 ensemble_stat output files. I have listed each
below
> > > followed by the line types which appear in each file. So that's
column
> > #24
> > > from each:
> > >
> > > ensemble_stat_Reventador_hr02_20190225_180000V.stat ECNT ORANK
PHIST
> RELP
> > > RHIST SSVAR
> > >
> > > ensemble_stat_Reventador_hr03_20190225_190000V.stat ECNT ORANK
PHIST
> RELP
> > > RHIST SSVAR
> > >
> > > ensemble_stat_Reventador_hr04_20190225_200000V.stat ORANK
> > >
> > > ensemble_stat_Reventador_hr05_20190225_210000V.stat ORANK
> > >
> > > ensemble_stat_Reventador_hr06_20190225_220000V.stat ORANK
> > >
> > >
> > > The ORANK line type is basically the "matched pairs" for
ensemble
> > > verification. Notice that the first 2 hours have the ORANK lines
along
> > with
> > > other statistic line types, such as RHIST. However, the last 3
hours
> only
> > > have the ORANK line type.
> > >
> > >
> > > Looking at the output for the first 2 hours, I see that you've
applied
> > the
> > > OBS_THRESH filtering threshold as ">=0.1".
> > >
> > >
> > > Here's a link to the table which defines the contents of the
ORANK line
> > > type:
> > >
> > >
> > >
> >
> https://dtcenter.github.io/MET/Users_Guide/ensemble-
stat.html?highlight=orank%20line%20type#id7
> > >
> > >
> > > Note that the actual observation rank value is in column #34.
Checking
> > the
> > > data in those last 3 files, you'll notice that the observation
ranks
> are
> > > always listed as bad data:
> > >
> > >
> > > cat ensemble_stat_Reventador_hr04_20190225_200000V.stat
> > > ensemble_stat_Reventador_hr05_20190225_210000V.stat
> > > ensemble_stat_Reventador_hr06_20190225_220000V.stat | grep
ORANK | awk
> > > '{print $34}' | sort -u
> > >
> > >
> > > NA
> > >
> > >
> > > So now the question is why? You're using the "censor_thresh" and
> > > "censor_val" options to reset the forecast values that are
"<0.1" to
> bad
> > > data. So that introduces a lot of bad data values. And that
results in
> > the
> > > rank of the observation being computed as bad data. This is
likely
> > > controlled by how you have the "vld_thresh" or "ens_vld_thresh"
set in
> > your
> > > Ensemble-Stat configuration file.
> > >
> > >
> > > John
> > >
> > > On Thu, Aug 27, 2020 at 8:48 AM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453
>
> > > >
> > > > Hello Tatiana,
> > > >
> > > > Sorry to ask more: as I said in previous mail I can not find
the
> other
> > 3
> > > > files from MET_Viewer because there is no RHIST inside of
those
> files,
> > if
> > > > this is because there is no matched pairs between obs. and
forecast,
> > why
> > > I
> > > > can see them in the RELY plot? (see the attachment please).
Are RELY
> > > plots
> > > > created based on matching pairs?
> > > >
> > > > Thank you.
> > > > Binyu
> > > >
> > > >
> > > > On Wed, Aug 26, 2020 at 10:47 PM Binyu Wang - NOAA Affiliate <
> > > > binyu.wang at noaa.gov> wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > I think I know the answer now, but just to confirm with you:
if
> there
> > > is
> > > > > no RHIST line type in the stat file, which indicates there
is no
> > > matching
> > > > > pair at all between obs. and forecast, is that correct?
> > > > >
> > > > > Thank you.
> > > > >
> > > > > On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via RT <
> > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > >> Binyu,
> > > > >> All 5 ensemble_stat files were successfully loaded to the
DB and
> the
> > > > data
> > > > >> is available for potting. Here is the example:
20200826_164848.
> > > > >> Your screenshot displays that you try to create a histogram
plot.
> > The
> > > > >> data for this plot comes from line_data_rhist table. This
table
> gets
> > > > filled
> > > > >> with 'RHIST' line type.
> > > > >> 'ensemble_stat_Reventador_hr04_20190225_200000V.stat' file
> contains
> > > only
> > > > >> 'ORANK' line type so non of it's records were saved into
> > > line_data_rhist
> > > > >> table.
> > > > >> Currently line_data_rhist table contains data for
fcst_valid_beg:
> > > > >> '2019-02-25 18:00:00' and '2019-02-25 19:00:00' (
fcst_lead: 20000
> > and
> > > > >> 30000). The data came from
> > > > >> ensemble_stat_Reventador_hr02_20190225_180000V.stat and
> > > > >> ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > > >>
> > > > >> Could you check if other files have records with RHIST line
type.
> > > > >>
> > > > >> Tatiana
> > > > >>
> > > > >> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov wrote:
> > > > >> > Tatiana,
> > > > >> >
> > > > >> > Yes that is weird. If you see the attached file, you can
see
> > > > >> > there only two files were uploaded. I expect to have all
5.
> > > > >> > Binyu
> > > > >> >
> > > > >> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek via RT
> > > > >> > <met_help at ucar.edu>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Binyu,
> > > > >> > >
> > > > >> > > In your log file there is a record that
> > > > >> > > 'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
was
> > > > >> > > successfully
> > > > >> > > loaded - all 53 lines.
> > > > >> > > I also can create a plot using the data from this file
- ORANK
> > > > >> > > statistics
> > > > >> > > with 40000 forecast lead: 20200826_151853.
> > > > >> > >
> > > > >> > > Tatiana
> > > > >> > >
> > > > >> > >
> > > > >> > > On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov wrote:
> > > > >> > > > Tatiana,
> > > > >> > > >
> > > > >> > > > I didn't see any error from the log file. I attached
the log
> > > file
> > > > >> > > > and two
> > > > >> > > > of the stat files:
> > > > >> > > > ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > > >> > > is
> > > > >> > > > readable from METViewer, but
> > > > >> > > > "ensemble_stat_Reventador_hr04_20190225_200000V.stat"
not.
> > > > >> > > >
> > > > >> > > > Thank you.
> > > > >> > > >
> > > > >> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek via RT
> > > > >> > > > <met_help at ucar.edu>
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Hi Binyu,
> > > > >> > > > > I assume that your files are on WCOSS. I don't have
an
> > access
> > > to
> > > > >> > > > > the
> > > > >> > > > > server and can't see the files.
> > > > >> > > > > Do you have logging messages from the load? Any
errors?
> > > > >> > > > >
> > > > >> > > > > I see that only 5 ensemble_stat files were loaded
to this
> > > > >> > > > > database:
> > > > >> > > > > |
ensemble_stat_Reventador_hr02_20190225_180000V.stat |
> > > > >> > > > > |
ensemble_stat_Reventador_hr03_20190225_190000V.stat |
> > > > >> > > > > |
ensemble_stat_Reventador_hr04_20190225_200000V.stat |
> > > > >> > > > > |
ensemble_stat_Reventador_hr05_20190225_210000V.stat |
> > > > >> > > > > |
ensemble_stat_Reventador_hr06_20190225_220000V.stat
> > > > >> > > > >
> > > > >> > > > > Tatiana
> > > > >> > > > >
> > > > >> > > > > On Tue Aug 25 20:02:54 2020, binyu.wang at noaa.gov
wrote:
> > > > >> > > > > > Hello,
> > > > >> > > > > >
> > > > >> > > > > > I created 5 ensemble_stat* files (from hr18-
hr22)
> under
> > > the
> > > > >> > > directory
> > > > >> > > > > > below, but only the first two (hr18 and hr19)
could be
> > read
> > > > >> > > > > > from
> > > > >> > > > > > METVIewer. Those files are under database
> > > > >> > > "mv_g2g_met_verf_Reventador"
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > >
> > > > >>
> > >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > > > >> > > > > >
> > > > >> > > > > > What is the problem? Thank you.
> > > > >> > > > > > Binyu
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: John Halley Gotway
Time: Thu Aug 27 15:04:39 2020
Binyu,
I do not think that will work as you expect. For each observation
point, if
any of the ensemble members contain a bad data value at that point,
then it
will not be included in the statistics. As we've seen in your data,
that is
the case for ALL observations other than some in the first 2 hours.
I would recommend using the "obs_thresh = >=0.1;" filtering threshold
but
NOT apply the censoring logic to the forecast data.
But as long as you understand the options, it's up to you to decide
how to
configure it.
John
On Thu, Aug 27, 2020 at 2:37 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453 >
>
> Thank you very much for the clarification. I will discuss with my
advisor.
> I think we will need to keep the " censor_thresh ", but we can
change the
> "vld_thresh" or "ens_vld_thresh" .
>
> Binyu
>
> On Thu, Aug 27, 2020 at 4:00 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > Binyu,
> >
> > In ensemble-stat, there is no difference between "matched pairs"
and
> > "ensemble pairs". It would be better if we made those log
messages use
> > consistent language.
> >
> > So regarding this log message:
> > *DEBUG 2: Skipping 53 of 53 points due to missing ensemble
values.*
> >
> > Here's the line of code where that's being written:
> >
> >
>
https://github.com/dtcenter/MET/blob/2c9ae440a84024fbf62caa64c1b747f9a912236f/met/src/libcode/vx_statistics/pair_data_ensemble.cc#L432
> >
> > So the rule here is that if any of the ensemble members contain
bad data
> > for a particular observation, then that point is not used in the
> > verification. And why not? We ultimately compute the rank of the
> > observation value, relative to the ensemble member values. If the
> ensemble
> > member values contain bad data, there's no good way of computing
the rank
> > from them.
> >
> > So when you use the "censor_thresh" and "censor_val" logic on the
input
> > ensemble fields, you're introducing lots of bad data values into
your
> > forecast fields. For hours 2 and 3, there are 17 and 11 points,
> > respectively, for which none of the ensemble members contain bad
data.
> But
> > for the rest of the hours, there are 0 points for which none of
the
> > ensemble members contain bad data.
> >
> > So I would recommend that you stop using the "censor_thresh" and
> > "censor_val" logic. It think it's confusing your results.
Requiring that
> > ALL ensemble members and the observation be >=0.1 results in no
pairs
> left
> > with which to compute statistics.
> >
> > John
> >
> > On Thu, Aug 27, 2020 at 1:41 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453 >
> > >
> > > Thank you very much for the detailed explanation, John.
> > >
> > > So basically, that means there is no any matched case (or site
b/c I
> used
> > > point_obs) between obs. and forecasting that meet all the
> > > threshold/censors for the last 3 files. If you look at part of
the log
> > file
> > > below (this is for the 3rd file):
> > >
> > > DEBUG 2: Processing point verification VAFTD/L14000-0 versus
> > ash/Z10000-0,
> > > for observation type ADPUPA, over region FULL, for interpolation
method
> > > NEAREST(1), using 53 matched pairs.
> > > DEBUG 2: Skipping 53 of 53 points due to missing ensemble
values.
> > > DEBUG 3: Using 0 of 53 ensemble pairs for observation filtering
> threshold
> > > >=0.1.
> > >
> > > So what is the difference between "matched pairs" and "ensemble
pairs"?
> > It
> > > seems "ensemble pairs is 0" is why I got no "PHIST", how about
the
> > matched
> > > pairs? Does that mean if one site was catched by all 20
ensemble
> > members,
> > > then I have 20 matched pairs?
> > >
> > > Thank you.
> > >
> > > On Thu, Aug 27, 2020 at 2:01 PM John Halley Gotway via RT <
> > > met_help at ucar.edu>
> > > wrote:
> > >
> > > > Binyu,
> > > >
> > > > Tatiana asked me to take a look at the RHIST line type being
> > unexpectedly
> > > > absent from some of your runs of Ensemble-Stat. I logged onto
WCOSS
> and
> > > > took a look in:
> > > >
> > > >
> >
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > > >
> > > > In there, I see 5 ensemble_stat output files. I have listed
each
> below
> > > > followed by the line types which appear in each file. So
that's
> column
> > > #24
> > > > from each:
> > > >
> > > > ensemble_stat_Reventador_hr02_20190225_180000V.stat ECNT ORANK
PHIST
> > RELP
> > > > RHIST SSVAR
> > > >
> > > > ensemble_stat_Reventador_hr03_20190225_190000V.stat ECNT ORANK
PHIST
> > RELP
> > > > RHIST SSVAR
> > > >
> > > > ensemble_stat_Reventador_hr04_20190225_200000V.stat ORANK
> > > >
> > > > ensemble_stat_Reventador_hr05_20190225_210000V.stat ORANK
> > > >
> > > > ensemble_stat_Reventador_hr06_20190225_220000V.stat ORANK
> > > >
> > > >
> > > > The ORANK line type is basically the "matched pairs" for
ensemble
> > > > verification. Notice that the first 2 hours have the ORANK
lines
> along
> > > with
> > > > other statistic line types, such as RHIST. However, the last 3
hours
> > only
> > > > have the ORANK line type.
> > > >
> > > >
> > > > Looking at the output for the first 2 hours, I see that you've
> applied
> > > the
> > > > OBS_THRESH filtering threshold as ">=0.1".
> > > >
> > > >
> > > > Here's a link to the table which defines the contents of the
ORANK
> line
> > > > type:
> > > >
> > > >
> > > >
> > >
> >
> https://dtcenter.github.io/MET/Users_Guide/ensemble-
stat.html?highlight=orank%20line%20type#id7
> > > >
> > > >
> > > > Note that the actual observation rank value is in column #34.
> Checking
> > > the
> > > > data in those last 3 files, you'll notice that the observation
ranks
> > are
> > > > always listed as bad data:
> > > >
> > > >
> > > > cat ensemble_stat_Reventador_hr04_20190225_200000V.stat
> > > > ensemble_stat_Reventador_hr05_20190225_210000V.stat
> > > > ensemble_stat_Reventador_hr06_20190225_220000V.stat | grep
ORANK |
> awk
> > > > '{print $34}' | sort -u
> > > >
> > > >
> > > > NA
> > > >
> > > >
> > > > So now the question is why? You're using the "censor_thresh"
and
> > > > "censor_val" options to reset the forecast values that are
"<0.1" to
> > bad
> > > > data. So that introduces a lot of bad data values. And that
results
> in
> > > the
> > > > rank of the observation being computed as bad data. This is
likely
> > > > controlled by how you have the "vld_thresh" or
"ens_vld_thresh" set
> in
> > > your
> > > > Ensemble-Stat configuration file.
> > > >
> > > >
> > > > John
> > > >
> > > > On Thu, Aug 27, 2020 at 8:48 AM binyu.wang at noaa.gov via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453 >
> > > > >
> > > > > Hello Tatiana,
> > > > >
> > > > > Sorry to ask more: as I said in previous mail I can not find
the
> > other
> > > 3
> > > > > files from MET_Viewer because there is no RHIST inside of
those
> > files,
> > > if
> > > > > this is because there is no matched pairs between obs. and
> forecast,
> > > why
> > > > I
> > > > > can see them in the RELY plot? (see the attachment please).
Are
> RELY
> > > > plots
> > > > > created based on matching pairs?
> > > > >
> > > > > Thank you.
> > > > > Binyu
> > > > >
> > > > >
> > > > > On Wed, Aug 26, 2020 at 10:47 PM Binyu Wang - NOAA Affiliate
<
> > > > > binyu.wang at noaa.gov> wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > I think I know the answer now, but just to confirm with
you: if
> > there
> > > > is
> > > > > > no RHIST line type in the stat file, which indicates there
is no
> > > > matching
> > > > > > pair at all between obs. and forecast, is that correct?
> > > > > >
> > > > > > Thank you.
> > > > > >
> > > > > > On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via RT <
> > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > >> Binyu,
> > > > > >> All 5 ensemble_stat files were successfully loaded to the
DB and
> > the
> > > > > data
> > > > > >> is available for potting. Here is the example:
20200826_164848.
> > > > > >> Your screenshot displays that you try to create a
histogram
> plot.
> > > The
> > > > > >> data for this plot comes from line_data_rhist table. This
table
> > gets
> > > > > filled
> > > > > >> with 'RHIST' line type.
> > > > > >> 'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
file
> > contains
> > > > only
> > > > > >> 'ORANK' line type so non of it's records were saved into
> > > > line_data_rhist
> > > > > >> table.
> > > > > >> Currently line_data_rhist table contains data for
> fcst_valid_beg:
> > > > > >> '2019-02-25 18:00:00' and '2019-02-25 19:00:00' (
fcst_lead:
> 20000
> > > and
> > > > > >> 30000). The data came from
> > > > > >> ensemble_stat_Reventador_hr02_20190225_180000V.stat and
> > > > > >> ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > > > >>
> > > > > >> Could you check if other files have records with RHIST
line
> type.
> > > > > >>
> > > > > >> Tatiana
> > > > > >>
> > > > > >> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov wrote:
> > > > > >> > Tatiana,
> > > > > >> >
> > > > > >> > Yes that is weird. If you see the attached file, you
can see
> > > > > >> > there only two files were uploaded. I expect to have
all 5.
> > > > > >> > Binyu
> > > > > >> >
> > > > > >> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek via RT
> > > > > >> > <met_help at ucar.edu>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Binyu,
> > > > > >> > >
> > > > > >> > > In your log file there is a record that
> > > > > >> > > 'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
was
> > > > > >> > > successfully
> > > > > >> > > loaded - all 53 lines.
> > > > > >> > > I also can create a plot using the data from this
file -
> ORANK
> > > > > >> > > statistics
> > > > > >> > > with 40000 forecast lead: 20200826_151853.
> > > > > >> > >
> > > > > >> > > Tatiana
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov
wrote:
> > > > > >> > > > Tatiana,
> > > > > >> > > >
> > > > > >> > > > I didn't see any error from the log file. I
attached the
> log
> > > > file
> > > > > >> > > > and two
> > > > > >> > > > of the stat files:
> > > > > >> > > > ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > > > >> > > is
> > > > > >> > > > readable from METViewer, but
> > > > > >> > > >
"ensemble_stat_Reventador_hr04_20190225_200000V.stat" not.
> > > > > >> > > >
> > > > > >> > > > Thank you.
> > > > > >> > > >
> > > > > >> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek via
RT
> > > > > >> > > > <met_help at ucar.edu>
> > > > > >> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > > Hi Binyu,
> > > > > >> > > > > I assume that your files are on WCOSS. I don't
have an
> > > access
> > > > to
> > > > > >> > > > > the
> > > > > >> > > > > server and can't see the files.
> > > > > >> > > > > Do you have logging messages from the load? Any
errors?
> > > > > >> > > > >
> > > > > >> > > > > I see that only 5 ensemble_stat files were loaded
to
> this
> > > > > >> > > > > database:
> > > > > >> > > > > |
ensemble_stat_Reventador_hr02_20190225_180000V.stat |
> > > > > >> > > > > |
ensemble_stat_Reventador_hr03_20190225_190000V.stat |
> > > > > >> > > > > |
ensemble_stat_Reventador_hr04_20190225_200000V.stat |
> > > > > >> > > > > |
ensemble_stat_Reventador_hr05_20190225_210000V.stat |
> > > > > >> > > > > |
ensemble_stat_Reventador_hr06_20190225_220000V.stat
> > > > > >> > > > >
> > > > > >> > > > > Tatiana
> > > > > >> > > > >
> > > > > >> > > > > On Tue Aug 25 20:02:54 2020, binyu.wang at noaa.gov
wrote:
> > > > > >> > > > > > Hello,
> > > > > >> > > > > >
> > > > > >> > > > > > I created 5 ensemble_stat* files (from hr18-
hr22)
> > under
> > > > the
> > > > > >> > > directory
> > > > > >> > > > > > below, but only the first two (hr18 and hr19)
could
> be
> > > read
> > > > > >> > > > > > from
> > > > > >> > > > > > METVIewer. Those files are under database
> > > > > >> > > "mv_g2g_met_verf_Reventador"
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > >
> > > > > >>
> > > >
> >
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > > > > >> > > > > >
> > > > > >> > > > > > What is the problem? Thank you.
> > > > > >> > > > > > Binyu
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: binyu.wang at noaa.gov
Time: Thu Aug 27 15:21:44 2020
I got more questions for the ORANK table at "
https://dtcenter.github.io/MET/Users_Guide/ensemble-
stat.html?highlight=orank%20line%20type#id7
",
1. For column 35 (Number of valid ensemble values): look at file
"ensemble_stat_Reventador_hr04_20190225_200000V.stat", If I have 0
pairs
that meet the requirement, why I have "10, 20, 18...." under that
column?
Does that mean I have 10 (or 20 or 18) out of 21 ensemble members?
(totally I have 21 ensemble members) . So if I reduce "ens_vld_thresh"
from
1 to 0.5, I will get "RHIST"?
2. What does that mean "Last-6"?
Thank you very much for your help.
Binyu
On Thu, Aug 27, 2020 at 5:04 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> Binyu,
>
> I do not think that will work as you expect. For each observation
point, if
> any of the ensemble members contain a bad data value at that point,
then it
> will not be included in the statistics. As we've seen in your data,
that is
> the case for ALL observations other than some in the first 2 hours.
>
> I would recommend using the "obs_thresh = >=0.1;" filtering
threshold but
> NOT apply the censoring logic to the forecast data.
>
> But as long as you understand the options, it's up to you to decide
how to
> configure it.
>
> John
>
> On Thu, Aug 27, 2020 at 2:37 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453 >
> >
> > Thank you very much for the clarification. I will discuss with my
> advisor.
> > I think we will need to keep the " censor_thresh ", but we can
change the
> > "vld_thresh" or "ens_vld_thresh" .
> >
> > Binyu
> >
> > On Thu, Aug 27, 2020 at 4:00 PM John Halley Gotway via RT <
> > met_help at ucar.edu>
> > wrote:
> >
> > > Binyu,
> > >
> > > In ensemble-stat, there is no difference between "matched pairs"
and
> > > "ensemble pairs". It would be better if we made those log
messages use
> > > consistent language.
> > >
> > > So regarding this log message:
> > > *DEBUG 2: Skipping 53 of 53 points due to missing ensemble
values.*
> > >
> > > Here's the line of code where that's being written:
> > >
> > >
> >
>
https://github.com/dtcenter/MET/blob/2c9ae440a84024fbf62caa64c1b747f9a912236f/met/src/libcode/vx_statistics/pair_data_ensemble.cc#L432
> > >
> > > So the rule here is that if any of the ensemble members contain
bad
> data
> > > for a particular observation, then that point is not used in the
> > > verification. And why not? We ultimately compute the rank of the
> > > observation value, relative to the ensemble member values. If
the
> > ensemble
> > > member values contain bad data, there's no good way of computing
the
> rank
> > > from them.
> > >
> > > So when you use the "censor_thresh" and "censor_val" logic on
the input
> > > ensemble fields, you're introducing lots of bad data values into
your
> > > forecast fields. For hours 2 and 3, there are 17 and 11 points,
> > > respectively, for which none of the ensemble members contain bad
data.
> > But
> > > for the rest of the hours, there are 0 points for which none of
the
> > > ensemble members contain bad data.
> > >
> > > So I would recommend that you stop using the "censor_thresh" and
> > > "censor_val" logic. It think it's confusing your results.
Requiring
> that
> > > ALL ensemble members and the observation be >=0.1 results in no
pairs
> > left
> > > with which to compute statistics.
> > >
> > > John
> > >
> > > On Thu, Aug 27, 2020 at 1:41 PM binyu.wang at noaa.gov via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453
>
> > > >
> > > > Thank you very much for the detailed explanation, John.
> > > >
> > > > So basically, that means there is no any matched case (or site
b/c I
> > used
> > > > point_obs) between obs. and forecasting that meet all the
> > > > threshold/censors for the last 3 files. If you look at part of
the
> log
> > > file
> > > > below (this is for the 3rd file):
> > > >
> > > > DEBUG 2: Processing point verification VAFTD/L14000-0 versus
> > > ash/Z10000-0,
> > > > for observation type ADPUPA, over region FULL, for
interpolation
> method
> > > > NEAREST(1), using 53 matched pairs.
> > > > DEBUG 2: Skipping 53 of 53 points due to missing ensemble
values.
> > > > DEBUG 3: Using 0 of 53 ensemble pairs for observation
filtering
> > threshold
> > > > >=0.1.
> > > >
> > > > So what is the difference between "matched pairs" and
"ensemble
> pairs"?
> > > It
> > > > seems "ensemble pairs is 0" is why I got no "PHIST", how about
the
> > > matched
> > > > pairs? Does that mean if one site was catched by all 20
ensemble
> > > members,
> > > > then I have 20 matched pairs?
> > > >
> > > > Thank you.
> > > >
> > > > On Thu, Aug 27, 2020 at 2:01 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Binyu,
> > > > >
> > > > > Tatiana asked me to take a look at the RHIST line type being
> > > unexpectedly
> > > > > absent from some of your runs of Ensemble-Stat. I logged
onto WCOSS
> > and
> > > > > took a look in:
> > > > >
> > > > >
> > >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > > > >
> > > > > In there, I see 5 ensemble_stat output files. I have listed
each
> > below
> > > > > followed by the line types which appear in each file. So
that's
> > column
> > > > #24
> > > > > from each:
> > > > >
> > > > > ensemble_stat_Reventador_hr02_20190225_180000V.stat ECNT
ORANK
> PHIST
> > > RELP
> > > > > RHIST SSVAR
> > > > >
> > > > > ensemble_stat_Reventador_hr03_20190225_190000V.stat ECNT
ORANK
> PHIST
> > > RELP
> > > > > RHIST SSVAR
> > > > >
> > > > > ensemble_stat_Reventador_hr04_20190225_200000V.stat ORANK
> > > > >
> > > > > ensemble_stat_Reventador_hr05_20190225_210000V.stat ORANK
> > > > >
> > > > > ensemble_stat_Reventador_hr06_20190225_220000V.stat ORANK
> > > > >
> > > > >
> > > > > The ORANK line type is basically the "matched pairs" for
ensemble
> > > > > verification. Notice that the first 2 hours have the ORANK
lines
> > along
> > > > with
> > > > > other statistic line types, such as RHIST. However, the last
3
> hours
> > > only
> > > > > have the ORANK line type.
> > > > >
> > > > >
> > > > > Looking at the output for the first 2 hours, I see that
you've
> > applied
> > > > the
> > > > > OBS_THRESH filtering threshold as ">=0.1".
> > > > >
> > > > >
> > > > > Here's a link to the table which defines the contents of the
ORANK
> > line
> > > > > type:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://dtcenter.github.io/MET/Users_Guide/ensemble-
stat.html?highlight=orank%20line%20type#id7
> > > > >
> > > > >
> > > > > Note that the actual observation rank value is in column
#34.
> > Checking
> > > > the
> > > > > data in those last 3 files, you'll notice that the
observation
> ranks
> > > are
> > > > > always listed as bad data:
> > > > >
> > > > >
> > > > > cat ensemble_stat_Reventador_hr04_20190225_200000V.stat
> > > > > ensemble_stat_Reventador_hr05_20190225_210000V.stat
> > > > > ensemble_stat_Reventador_hr06_20190225_220000V.stat | grep
ORANK |
> > awk
> > > > > '{print $34}' | sort -u
> > > > >
> > > > >
> > > > > NA
> > > > >
> > > > >
> > > > > So now the question is why? You're using the "censor_thresh"
and
> > > > > "censor_val" options to reset the forecast values that are
"<0.1"
> to
> > > bad
> > > > > data. So that introduces a lot of bad data values. And that
results
> > in
> > > > the
> > > > > rank of the observation being computed as bad data. This is
likely
> > > > > controlled by how you have the "vld_thresh" or
"ens_vld_thresh" set
> > in
> > > > your
> > > > > Ensemble-Stat configuration file.
> > > > >
> > > > >
> > > > > John
> > > > >
> > > > > On Thu, Aug 27, 2020 at 8:48 AM binyu.wang at noaa.gov via RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453 >
> > > > > >
> > > > > > Hello Tatiana,
> > > > > >
> > > > > > Sorry to ask more: as I said in previous mail I can not
find the
> > > other
> > > > 3
> > > > > > files from MET_Viewer because there is no RHIST inside of
those
> > > files,
> > > > if
> > > > > > this is because there is no matched pairs between obs. and
> > forecast,
> > > > why
> > > > > I
> > > > > > can see them in the RELY plot? (see the attachment
please). Are
> > RELY
> > > > > plots
> > > > > > created based on matching pairs?
> > > > > >
> > > > > > Thank you.
> > > > > > Binyu
> > > > > >
> > > > > >
> > > > > > On Wed, Aug 26, 2020 at 10:47 PM Binyu Wang - NOAA
Affiliate <
> > > > > > binyu.wang at noaa.gov> wrote:
> > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > I think I know the answer now, but just to confirm with
you: if
> > > there
> > > > > is
> > > > > > > no RHIST line type in the stat file, which indicates
there is
> no
> > > > > matching
> > > > > > > pair at all between obs. and forecast, is that correct?
> > > > > > >
> > > > > > > Thank you.
> > > > > > >
> > > > > > > On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via RT <
> > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Binyu,
> > > > > > >> All 5 ensemble_stat files were successfully loaded to
the DB
> and
> > > the
> > > > > > data
> > > > > > >> is available for potting. Here is the example:
> 20200826_164848.
> > > > > > >> Your screenshot displays that you try to create a
histogram
> > plot.
> > > > The
> > > > > > >> data for this plot comes from line_data_rhist table.
This
> table
> > > gets
> > > > > > filled
> > > > > > >> with 'RHIST' line type.
> > > > > > >> 'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
file
> > > contains
> > > > > only
> > > > > > >> 'ORANK' line type so non of it's records were saved
into
> > > > > line_data_rhist
> > > > > > >> table.
> > > > > > >> Currently line_data_rhist table contains data for
> > fcst_valid_beg:
> > > > > > >> '2019-02-25 18:00:00' and '2019-02-25 19:00:00' (
fcst_lead:
> > 20000
> > > > and
> > > > > > >> 30000). The data came from
> > > > > > >> ensemble_stat_Reventador_hr02_20190225_180000V.stat and
> > > > > > >> ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > > > > >>
> > > > > > >> Could you check if other files have records with RHIST
line
> > type.
> > > > > > >>
> > > > > > >> Tatiana
> > > > > > >>
> > > > > > >> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov wrote:
> > > > > > >> > Tatiana,
> > > > > > >> >
> > > > > > >> > Yes that is weird. If you see the attached file, you
can see
> > > > > > >> > there only two files were uploaded. I expect to have
all 5.
> > > > > > >> > Binyu
> > > > > > >> >
> > > > > > >> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek via RT
> > > > > > >> > <met_help at ucar.edu>
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > Binyu,
> > > > > > >> > >
> > > > > > >> > > In your log file there is a record that
> > > > > > >> > >
'ensemble_stat_Reventador_hr04_20190225_200000V.stat' was
> > > > > > >> > > successfully
> > > > > > >> > > loaded - all 53 lines.
> > > > > > >> > > I also can create a plot using the data from this
file -
> > ORANK
> > > > > > >> > > statistics
> > > > > > >> > > with 40000 forecast lead: 20200826_151853.
> > > > > > >> > >
> > > > > > >> > > Tatiana
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov
wrote:
> > > > > > >> > > > Tatiana,
> > > > > > >> > > >
> > > > > > >> > > > I didn't see any error from the log file. I
attached the
> > log
> > > > > file
> > > > > > >> > > > and two
> > > > > > >> > > > of the stat files:
> > > > > > >> > > >
ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > > > > >> > > is
> > > > > > >> > > > readable from METViewer, but
> > > > > > >> > > >
"ensemble_stat_Reventador_hr04_20190225_200000V.stat"
> not.
> > > > > > >> > > >
> > > > > > >> > > > Thank you.
> > > > > > >> > > >
> > > > > > >> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek via
RT
> > > > > > >> > > > <met_help at ucar.edu>
> > > > > > >> > > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Hi Binyu,
> > > > > > >> > > > > I assume that your files are on WCOSS. I don't
have an
> > > > access
> > > > > to
> > > > > > >> > > > > the
> > > > > > >> > > > > server and can't see the files.
> > > > > > >> > > > > Do you have logging messages from the load? Any
> errors?
> > > > > > >> > > > >
> > > > > > >> > > > > I see that only 5 ensemble_stat files were
loaded to
> > this
> > > > > > >> > > > > database:
> > > > > > >> > > > > |
ensemble_stat_Reventador_hr02_20190225_180000V.stat
> |
> > > > > > >> > > > > |
ensemble_stat_Reventador_hr03_20190225_190000V.stat
> |
> > > > > > >> > > > > |
ensemble_stat_Reventador_hr04_20190225_200000V.stat
> |
> > > > > > >> > > > > |
ensemble_stat_Reventador_hr05_20190225_210000V.stat
> |
> > > > > > >> > > > > |
ensemble_stat_Reventador_hr06_20190225_220000V.stat
> > > > > > >> > > > >
> > > > > > >> > > > > Tatiana
> > > > > > >> > > > >
> > > > > > >> > > > > On Tue Aug 25 20:02:54 2020,
binyu.wang at noaa.gov
> wrote:
> > > > > > >> > > > > > Hello,
> > > > > > >> > > > > >
> > > > > > >> > > > > > I created 5 ensemble_stat* files (from hr18-
hr22)
> > > under
> > > > > the
> > > > > > >> > > directory
> > > > > > >> > > > > > below, but only the first two (hr18 and
hr19) could
> > be
> > > > read
> > > > > > >> > > > > > from
> > > > > > >> > > > > > METVIewer. Those files are under database
> > > > > > >> > > "mv_g2g_met_verf_Reventador"
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > >
> > > > > > >>
> > > > >
> > >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > > > > > >> > > > > >
> > > > > > >> > > > > > What is the problem? Thank you.
> > > > > > >> > > > > > Binyu
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > > > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: binyu.wang at noaa.gov
Time: Thu Aug 27 16:09:12 2020
I figured out the answer for question 2. But for the first one: based
on
column 35, it seems the majority of them are bigger than 10 (totally
I
have 21 members), does that mean if I set "ens_vld_thresh" = 0.5,
then I
could get "RHIST"? for the last 3 members?
Binyu
On Thu, Aug 27, 2020 at 5:21 PM Binyu Wang - NOAA Affiliate <
binyu.wang at noaa.gov> wrote:
> I got more questions for the ORANK table at "
> https://dtcenter.github.io/MET/Users_Guide/ensemble-
stat.html?highlight=orank%20line%20type#id7
> ",
>
> 1. For column 35 (Number of valid ensemble values): look at file
> "ensemble_stat_Reventador_hr04_20190225_200000V.stat", If I have 0
pairs
> that meet the requirement, why I have "10, 20, 18...." under that
column?
> Does that mean I have 10 (or 20 or 18) out of 21 ensemble members?
> (totally I have 21 ensemble members) . So if I reduce
"ens_vld_thresh" from
> 1 to 0.5, I will get "RHIST"?
>
>
> 2. What does that mean "Last-6"?
>
> Thank you very much for your help.
> Binyu
>
>
> On Thu, Aug 27, 2020 at 5:04 PM John Halley Gotway via RT <
> met_help at ucar.edu> wrote:
>
>> Binyu,
>>
>> I do not think that will work as you expect. For each observation
point,
>> if
>> any of the ensemble members contain a bad data value at that point,
then
>> it
>> will not be included in the statistics. As we've seen in your data,
that
>> is
>> the case for ALL observations other than some in the first 2 hours.
>>
>> I would recommend using the "obs_thresh = >=0.1;" filtering
threshold but
>> NOT apply the censoring logic to the forecast data.
>>
>> But as long as you understand the options, it's up to you to decide
how to
>> configure it.
>>
>> John
>>
>> On Thu, Aug 27, 2020 at 2:37 PM binyu.wang at noaa.gov via RT <
>> met_help at ucar.edu> wrote:
>>
>> >
>> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453 >
>> >
>> > Thank you very much for the clarification. I will discuss with my
>> advisor.
>> > I think we will need to keep the " censor_thresh ", but we can
change
>> the
>> > "vld_thresh" or "ens_vld_thresh" .
>> >
>> > Binyu
>> >
>> > On Thu, Aug 27, 2020 at 4:00 PM John Halley Gotway via RT <
>> > met_help at ucar.edu>
>> > wrote:
>> >
>> > > Binyu,
>> > >
>> > > In ensemble-stat, there is no difference between "matched
pairs" and
>> > > "ensemble pairs". It would be better if we made those log
messages
>> use
>> > > consistent language.
>> > >
>> > > So regarding this log message:
>> > > *DEBUG 2: Skipping 53 of 53 points due to missing ensemble
values.*
>> > >
>> > > Here's the line of code where that's being written:
>> > >
>> > >
>> >
>>
https://github.com/dtcenter/MET/blob/2c9ae440a84024fbf62caa64c1b747f9a912236f/met/src/libcode/vx_statistics/pair_data_ensemble.cc#L432
>> > >
>> > > So the rule here is that if any of the ensemble members contain
bad
>> data
>> > > for a particular observation, then that point is not used in
the
>> > > verification. And why not? We ultimately compute the rank of
the
>> > > observation value, relative to the ensemble member values. If
the
>> > ensemble
>> > > member values contain bad data, there's no good way of
computing the
>> rank
>> > > from them.
>> > >
>> > > So when you use the "censor_thresh" and "censor_val" logic on
the
>> input
>> > > ensemble fields, you're introducing lots of bad data values
into your
>> > > forecast fields. For hours 2 and 3, there are 17 and 11 points,
>> > > respectively, for which none of the ensemble members contain
bad data.
>> > But
>> > > for the rest of the hours, there are 0 points for which none of
the
>> > > ensemble members contain bad data.
>> > >
>> > > So I would recommend that you stop using the "censor_thresh"
and
>> > > "censor_val" logic. It think it's confusing your results.
Requiring
>> that
>> > > ALL ensemble members and the observation be >=0.1 results in no
pairs
>> > left
>> > > with which to compute statistics.
>> > >
>> > > John
>> > >
>> > > On Thu, Aug 27, 2020 at 1:41 PM binyu.wang at noaa.gov via RT <
>> > > met_help at ucar.edu> wrote:
>> > >
>> > > >
>> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453
>
>> > > >
>> > > > Thank you very much for the detailed explanation, John.
>> > > >
>> > > > So basically, that means there is no any matched case (or
site b/c I
>> > used
>> > > > point_obs) between obs. and forecasting that meet all the
>> > > > threshold/censors for the last 3 files. If you look at part
of the
>> log
>> > > file
>> > > > below (this is for the 3rd file):
>> > > >
>> > > > DEBUG 2: Processing point verification VAFTD/L14000-0 versus
>> > > ash/Z10000-0,
>> > > > for observation type ADPUPA, over region FULL, for
interpolation
>> method
>> > > > NEAREST(1), using 53 matched pairs.
>> > > > DEBUG 2: Skipping 53 of 53 points due to missing ensemble
values.
>> > > > DEBUG 3: Using 0 of 53 ensemble pairs for observation
filtering
>> > threshold
>> > > > >=0.1.
>> > > >
>> > > > So what is the difference between "matched pairs" and
"ensemble
>> pairs"?
>> > > It
>> > > > seems "ensemble pairs is 0" is why I got no "PHIST", how
about the
>> > > matched
>> > > > pairs? Does that mean if one site was catched by all 20
ensemble
>> > > members,
>> > > > then I have 20 matched pairs?
>> > > >
>> > > > Thank you.
>> > > >
>> > > > On Thu, Aug 27, 2020 at 2:01 PM John Halley Gotway via RT <
>> > > > met_help at ucar.edu>
>> > > > wrote:
>> > > >
>> > > > > Binyu,
>> > > > >
>> > > > > Tatiana asked me to take a look at the RHIST line type
being
>> > > unexpectedly
>> > > > > absent from some of your runs of Ensemble-Stat. I logged
onto
>> WCOSS
>> > and
>> > > > > took a look in:
>> > > > >
>> > > > >
>> > >
>>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
>> > > > >
>> > > > > In there, I see 5 ensemble_stat output files. I have listed
each
>> > below
>> > > > > followed by the line types which appear in each file. So
that's
>> > column
>> > > > #24
>> > > > > from each:
>> > > > >
>> > > > > ensemble_stat_Reventador_hr02_20190225_180000V.stat ECNT
ORANK
>> PHIST
>> > > RELP
>> > > > > RHIST SSVAR
>> > > > >
>> > > > > ensemble_stat_Reventador_hr03_20190225_190000V.stat ECNT
ORANK
>> PHIST
>> > > RELP
>> > > > > RHIST SSVAR
>> > > > >
>> > > > > ensemble_stat_Reventador_hr04_20190225_200000V.stat ORANK
>> > > > >
>> > > > > ensemble_stat_Reventador_hr05_20190225_210000V.stat ORANK
>> > > > >
>> > > > > ensemble_stat_Reventador_hr06_20190225_220000V.stat ORANK
>> > > > >
>> > > > >
>> > > > > The ORANK line type is basically the "matched pairs" for
ensemble
>> > > > > verification. Notice that the first 2 hours have the ORANK
lines
>> > along
>> > > > with
>> > > > > other statistic line types, such as RHIST. However, the
last 3
>> hours
>> > > only
>> > > > > have the ORANK line type.
>> > > > >
>> > > > >
>> > > > > Looking at the output for the first 2 hours, I see that
you've
>> > applied
>> > > > the
>> > > > > OBS_THRESH filtering threshold as ">=0.1".
>> > > > >
>> > > > >
>> > > > > Here's a link to the table which defines the contents of
the ORANK
>> > line
>> > > > > type:
>> > > > >
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://dtcenter.github.io/MET/Users_Guide/ensemble-
stat.html?highlight=orank%20line%20type#id7
>> > > > >
>> > > > >
>> > > > > Note that the actual observation rank value is in column
#34.
>> > Checking
>> > > > the
>> > > > > data in those last 3 files, you'll notice that the
observation
>> ranks
>> > > are
>> > > > > always listed as bad data:
>> > > > >
>> > > > >
>> > > > > cat ensemble_stat_Reventador_hr04_20190225_200000V.stat
>> > > > > ensemble_stat_Reventador_hr05_20190225_210000V.stat
>> > > > > ensemble_stat_Reventador_hr06_20190225_220000V.stat | grep
ORANK
>> |
>> > awk
>> > > > > '{print $34}' | sort -u
>> > > > >
>> > > > >
>> > > > > NA
>> > > > >
>> > > > >
>> > > > > So now the question is why? You're using the
"censor_thresh" and
>> > > > > "censor_val" options to reset the forecast values that are
"<0.1"
>> to
>> > > bad
>> > > > > data. So that introduces a lot of bad data values. And that
>> results
>> > in
>> > > > the
>> > > > > rank of the observation being computed as bad data. This is
likely
>> > > > > controlled by how you have the "vld_thresh" or
"ens_vld_thresh"
>> set
>> > in
>> > > > your
>> > > > > Ensemble-Stat configuration file.
>> > > > >
>> > > > >
>> > > > > John
>> > > > >
>> > > > > On Thu, Aug 27, 2020 at 8:48 AM binyu.wang at noaa.gov via RT
<
>> > > > > met_help at ucar.edu> wrote:
>> > > > >
>> > > > > >
>> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453 >
>> > > > > >
>> > > > > > Hello Tatiana,
>> > > > > >
>> > > > > > Sorry to ask more: as I said in previous mail I can not
find the
>> > > other
>> > > > 3
>> > > > > > files from MET_Viewer because there is no RHIST inside of
those
>> > > files,
>> > > > if
>> > > > > > this is because there is no matched pairs between obs.
and
>> > forecast,
>> > > > why
>> > > > > I
>> > > > > > can see them in the RELY plot? (see the attachment
please). Are
>> > RELY
>> > > > > plots
>> > > > > > created based on matching pairs?
>> > > > > >
>> > > > > > Thank you.
>> > > > > > Binyu
>> > > > > >
>> > > > > >
>> > > > > > On Wed, Aug 26, 2020 at 10:47 PM Binyu Wang - NOAA
Affiliate <
>> > > > > > binyu.wang at noaa.gov> wrote:
>> > > > > >
>> > > > > > > Hello,
>> > > > > > >
>> > > > > > > I think I know the answer now, but just to confirm with
you:
>> if
>> > > there
>> > > > > is
>> > > > > > > no RHIST line type in the stat file, which indicates
there is
>> no
>> > > > > matching
>> > > > > > > pair at all between obs. and forecast, is that correct?
>> > > > > > >
>> > > > > > > Thank you.
>> > > > > > >
>> > > > > > > On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via RT <
>> > > > > met_help at ucar.edu>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > >> Binyu,
>> > > > > > >> All 5 ensemble_stat files were successfully loaded to
the DB
>> and
>> > > the
>> > > > > > data
>> > > > > > >> is available for potting. Here is the example:
>> 20200826_164848.
>> > > > > > >> Your screenshot displays that you try to create a
histogram
>> > plot.
>> > > > The
>> > > > > > >> data for this plot comes from line_data_rhist table.
This
>> table
>> > > gets
>> > > > > > filled
>> > > > > > >> with 'RHIST' line type.
>> > > > > > >> 'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
file
>> > > contains
>> > > > > only
>> > > > > > >> 'ORANK' line type so non of it's records were saved
into
>> > > > > line_data_rhist
>> > > > > > >> table.
>> > > > > > >> Currently line_data_rhist table contains data for
>> > fcst_valid_beg:
>> > > > > > >> '2019-02-25 18:00:00' and '2019-02-25 19:00:00' (
fcst_lead:
>> > 20000
>> > > > and
>> > > > > > >> 30000). The data came from
>> > > > > > >> ensemble_stat_Reventador_hr02_20190225_180000V.stat
and
>> > > > > > >> ensemble_stat_Reventador_hr03_20190225_190000V.stat
>> > > > > > >>
>> > > > > > >> Could you check if other files have records with RHIST
line
>> > type.
>> > > > > > >>
>> > > > > > >> Tatiana
>> > > > > > >>
>> > > > > > >> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov wrote:
>> > > > > > >> > Tatiana,
>> > > > > > >> >
>> > > > > > >> > Yes that is weird. If you see the attached file, you
can
>> see
>> > > > > > >> > there only two files were uploaded. I expect to
have all
>> 5.
>> > > > > > >> > Binyu
>> > > > > > >> >
>> > > > > > >> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek via
RT
>> > > > > > >> > <met_help at ucar.edu>
>> > > > > > >> > wrote:
>> > > > > > >> >
>> > > > > > >> > > Binyu,
>> > > > > > >> > >
>> > > > > > >> > > In your log file there is a record that
>> > > > > > >> > >
'ensemble_stat_Reventador_hr04_20190225_200000V.stat' was
>> > > > > > >> > > successfully
>> > > > > > >> > > loaded - all 53 lines.
>> > > > > > >> > > I also can create a plot using the data from this
file -
>> > ORANK
>> > > > > > >> > > statistics
>> > > > > > >> > > with 40000 forecast lead: 20200826_151853.
>> > > > > > >> > >
>> > > > > > >> > > Tatiana
>> > > > > > >> > >
>> > > > > > >> > >
>> > > > > > >> > > On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov
wrote:
>> > > > > > >> > > > Tatiana,
>> > > > > > >> > > >
>> > > > > > >> > > > I didn't see any error from the log file. I
attached
>> the
>> > log
>> > > > > file
>> > > > > > >> > > > and two
>> > > > > > >> > > > of the stat files:
>> > > > > > >> > > >
ensemble_stat_Reventador_hr03_20190225_190000V.stat
>> > > > > > >> > > is
>> > > > > > >> > > > readable from METViewer, but
>> > > > > > >> > > >
"ensemble_stat_Reventador_hr04_20190225_200000V.stat"
>> not.
>> > > > > > >> > > >
>> > > > > > >> > > > Thank you.
>> > > > > > >> > > >
>> > > > > > >> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek
via RT
>> > > > > > >> > > > <met_help at ucar.edu>
>> > > > > > >> > > > wrote:
>> > > > > > >> > > >
>> > > > > > >> > > > > Hi Binyu,
>> > > > > > >> > > > > I assume that your files are on WCOSS. I don't
have
>> an
>> > > > access
>> > > > > to
>> > > > > > >> > > > > the
>> > > > > > >> > > > > server and can't see the files.
>> > > > > > >> > > > > Do you have logging messages from the load?
Any
>> errors?
>> > > > > > >> > > > >
>> > > > > > >> > > > > I see that only 5 ensemble_stat files were
loaded to
>> > this
>> > > > > > >> > > > > database:
>> > > > > > >> > > > > |
>> ensemble_stat_Reventador_hr02_20190225_180000V.stat |
>> > > > > > >> > > > > |
>> ensemble_stat_Reventador_hr03_20190225_190000V.stat |
>> > > > > > >> > > > > |
>> ensemble_stat_Reventador_hr04_20190225_200000V.stat |
>> > > > > > >> > > > > |
>> ensemble_stat_Reventador_hr05_20190225_210000V.stat |
>> > > > > > >> > > > > |
ensemble_stat_Reventador_hr06_20190225_220000V.stat
>> > > > > > >> > > > >
>> > > > > > >> > > > > Tatiana
>> > > > > > >> > > > >
>> > > > > > >> > > > > On Tue Aug 25 20:02:54 2020,
binyu.wang at noaa.gov
>> wrote:
>> > > > > > >> > > > > > Hello,
>> > > > > > >> > > > > >
>> > > > > > >> > > > > > I created 5 ensemble_stat* files (from hr18-
hr22)
>> > > under
>> > > > > the
>> > > > > > >> > > directory
>> > > > > > >> > > > > > below, but only the first two (hr18 and
hr19)
>> could
>> > be
>> > > > read
>> > > > > > >> > > > > > from
>> > > > > > >> > > > > > METVIewer. Those files are under database
>> > > > > > >> > > "mv_g2g_met_verf_Reventador"
>> > > > > > >> > > > > >
>> > > > > > >> > > > > >
>> > > > > > >> > >
>> > > > > > >>
>> > > > >
>> > >
>>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
>> > > > > > >> > > > > >
>> > > > > > >> > > > > > What is the problem? Thank you.
>> > > > > > >> > > > > > Binyu
>> > > > > > >> > > > >
>> > > > > > >> > > > >
>> > > > > > >> > > > >
>> > > > > > >> > > > >
>> > > > > > >> > >
>> > > > > > >> > >
>> > > > > > >> > >
>> > > > > > >> > >
>> > > > > > >>
>> > > > > > >>
>> > > > > > >>
>> > > > > > >>
>> > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> > >
>> >
>> >
>>
>>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: John Halley Gotway
Time: Thu Aug 27 16:21:20 2020
No, that will not work as you'd hope. In fact, this same issue came up
recently on a GitHub issue, and I wrote up a comment about it.
Please read through that comment here:
https://github.com/dtcenter/MET/issues/1475#issuecomment-682202522
The point here is that if any of the ensemble member values have bad
data,
then that point is excluded from the ensemble statistics. There is no
configuration option to override that behavior. And the reason one is
not
provided is that we have not defined a clear algorithm for how to
compute
statistics when missing data values are present.
For example, let's say you have a 21 member ensemble and just 3
observations:
- For point 1, all 21 members contain valid data.
- For point 2, 15 of 21 contain valid data.
- For point 3, 10 of 21 contain valid data.
Assume that you just ignore the bad data values and compute the
observation
rank against the valid values. Maybe the obs 1 has rank 5 of 21, obs 2
has
rank 10 of 15, and obs 3 has rank 1 of 10. These 3 ranks cannot be
combined
into a rank histogram. The number of members must remain constant in
order
for them to be combined. You could conceivably still compute a PHIST
line,
but most users won't understand the nuance of what is occurring. And
dealing with missing data values like this gets very messy. To change
this
behavior, we'd need an algorithm with a clear set of rules to enforce
based
on advice from statisticians on proper methods.
John
On Thu, Aug 27, 2020 at 4:09 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453 >
>
> I figured out the answer for question 2. But for the first one:
based on
> column 35, it seems the majority of them are bigger than 10
(totally I
> have 21 members), does that mean if I set "ens_vld_thresh" = 0.5,
then I
> could get "RHIST"? for the last 3 members?
>
> Binyu
>
> On Thu, Aug 27, 2020 at 5:21 PM Binyu Wang - NOAA Affiliate <
> binyu.wang at noaa.gov> wrote:
>
> > I got more questions for the ORANK table at "
> >
> https://dtcenter.github.io/MET/Users_Guide/ensemble-
stat.html?highlight=orank%20line%20type#id7
> > ",
> >
> > 1. For column 35 (Number of valid ensemble values): look at file
> > "ensemble_stat_Reventador_hr04_20190225_200000V.stat", If I have 0
pairs
> > that meet the requirement, why I have "10, 20, 18...." under that
column?
> > Does that mean I have 10 (or 20 or 18) out of 21 ensemble
members?
> > (totally I have 21 ensemble members) . So if I reduce
"ens_vld_thresh"
> from
> > 1 to 0.5, I will get "RHIST"?
> >
> >
> > 2. What does that mean "Last-6"?
> >
> > Thank you very much for your help.
> > Binyu
> >
> >
> > On Thu, Aug 27, 2020 at 5:04 PM John Halley Gotway via RT <
> > met_help at ucar.edu> wrote:
> >
> >> Binyu,
> >>
> >> I do not think that will work as you expect. For each observation
point,
> >> if
> >> any of the ensemble members contain a bad data value at that
point, then
> >> it
> >> will not be included in the statistics. As we've seen in your
data, that
> >> is
> >> the case for ALL observations other than some in the first 2
hours.
> >>
> >> I would recommend using the "obs_thresh = >=0.1;" filtering
threshold
> but
> >> NOT apply the censoring logic to the forecast data.
> >>
> >> But as long as you understand the options, it's up to you to
decide how
> to
> >> configure it.
> >>
> >> John
> >>
> >> On Thu, Aug 27, 2020 at 2:37 PM binyu.wang at noaa.gov via RT <
> >> met_help at ucar.edu> wrote:
> >>
> >> >
> >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453 >
> >> >
> >> > Thank you very much for the clarification. I will discuss with
my
> >> advisor.
> >> > I think we will need to keep the " censor_thresh ", but we can
change
> >> the
> >> > "vld_thresh" or "ens_vld_thresh" .
> >> >
> >> > Binyu
> >> >
> >> > On Thu, Aug 27, 2020 at 4:00 PM John Halley Gotway via RT <
> >> > met_help at ucar.edu>
> >> > wrote:
> >> >
> >> > > Binyu,
> >> > >
> >> > > In ensemble-stat, there is no difference between "matched
pairs" and
> >> > > "ensemble pairs". It would be better if we made those log
messages
> >> use
> >> > > consistent language.
> >> > >
> >> > > So regarding this log message:
> >> > > *DEBUG 2: Skipping 53 of 53 points due to missing ensemble
values.*
> >> > >
> >> > > Here's the line of code where that's being written:
> >> > >
> >> > >
> >> >
> >>
>
https://github.com/dtcenter/MET/blob/2c9ae440a84024fbf62caa64c1b747f9a912236f/met/src/libcode/vx_statistics/pair_data_ensemble.cc#L432
> >> > >
> >> > > So the rule here is that if any of the ensemble members
contain bad
> >> data
> >> > > for a particular observation, then that point is not used in
the
> >> > > verification. And why not? We ultimately compute the rank of
the
> >> > > observation value, relative to the ensemble member values. If
the
> >> > ensemble
> >> > > member values contain bad data, there's no good way of
computing the
> >> rank
> >> > > from them.
> >> > >
> >> > > So when you use the "censor_thresh" and "censor_val" logic on
the
> >> input
> >> > > ensemble fields, you're introducing lots of bad data values
into
> your
> >> > > forecast fields. For hours 2 and 3, there are 17 and 11
points,
> >> > > respectively, for which none of the ensemble members contain
bad
> data.
> >> > But
> >> > > for the rest of the hours, there are 0 points for which none
of the
> >> > > ensemble members contain bad data.
> >> > >
> >> > > So I would recommend that you stop using the "censor_thresh"
and
> >> > > "censor_val" logic. It think it's confusing your results.
Requiring
> >> that
> >> > > ALL ensemble members and the observation be >=0.1 results in
no
> pairs
> >> > left
> >> > > with which to compute statistics.
> >> > >
> >> > > John
> >> > >
> >> > > On Thu, Aug 27, 2020 at 1:41 PM binyu.wang at noaa.gov via RT <
> >> > > met_help at ucar.edu> wrote:
> >> > >
> >> > > >
> >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453 >
> >> > > >
> >> > > > Thank you very much for the detailed explanation, John.
> >> > > >
> >> > > > So basically, that means there is no any matched case (or
site
> b/c I
> >> > used
> >> > > > point_obs) between obs. and forecasting that meet all the
> >> > > > threshold/censors for the last 3 files. If you look at part
of the
> >> log
> >> > > file
> >> > > > below (this is for the 3rd file):
> >> > > >
> >> > > > DEBUG 2: Processing point verification VAFTD/L14000-0
versus
> >> > > ash/Z10000-0,
> >> > > > for observation type ADPUPA, over region FULL, for
interpolation
> >> method
> >> > > > NEAREST(1), using 53 matched pairs.
> >> > > > DEBUG 2: Skipping 53 of 53 points due to missing ensemble
values.
> >> > > > DEBUG 3: Using 0 of 53 ensemble pairs for observation
filtering
> >> > threshold
> >> > > > >=0.1.
> >> > > >
> >> > > > So what is the difference between "matched pairs" and
"ensemble
> >> pairs"?
> >> > > It
> >> > > > seems "ensemble pairs is 0" is why I got no "PHIST", how
about the
> >> > > matched
> >> > > > pairs? Does that mean if one site was catched by all 20
ensemble
> >> > > members,
> >> > > > then I have 20 matched pairs?
> >> > > >
> >> > > > Thank you.
> >> > > >
> >> > > > On Thu, Aug 27, 2020 at 2:01 PM John Halley Gotway via RT <
> >> > > > met_help at ucar.edu>
> >> > > > wrote:
> >> > > >
> >> > > > > Binyu,
> >> > > > >
> >> > > > > Tatiana asked me to take a look at the RHIST line type
being
> >> > > unexpectedly
> >> > > > > absent from some of your runs of Ensemble-Stat. I logged
onto
> >> WCOSS
> >> > and
> >> > > > > took a look in:
> >> > > > >
> >> > > > >
> >> > >
> >>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> >> > > > >
> >> > > > > In there, I see 5 ensemble_stat output files. I have
listed each
> >> > below
> >> > > > > followed by the line types which appear in each file. So
that's
> >> > column
> >> > > > #24
> >> > > > > from each:
> >> > > > >
> >> > > > > ensemble_stat_Reventador_hr02_20190225_180000V.stat ECNT
ORANK
> >> PHIST
> >> > > RELP
> >> > > > > RHIST SSVAR
> >> > > > >
> >> > > > > ensemble_stat_Reventador_hr03_20190225_190000V.stat ECNT
ORANK
> >> PHIST
> >> > > RELP
> >> > > > > RHIST SSVAR
> >> > > > >
> >> > > > > ensemble_stat_Reventador_hr04_20190225_200000V.stat ORANK
> >> > > > >
> >> > > > > ensemble_stat_Reventador_hr05_20190225_210000V.stat ORANK
> >> > > > >
> >> > > > > ensemble_stat_Reventador_hr06_20190225_220000V.stat ORANK
> >> > > > >
> >> > > > >
> >> > > > > The ORANK line type is basically the "matched pairs" for
> ensemble
> >> > > > > verification. Notice that the first 2 hours have the
ORANK lines
> >> > along
> >> > > > with
> >> > > > > other statistic line types, such as RHIST. However, the
last 3
> >> hours
> >> > > only
> >> > > > > have the ORANK line type.
> >> > > > >
> >> > > > >
> >> > > > > Looking at the output for the first 2 hours, I see that
you've
> >> > applied
> >> > > > the
> >> > > > > OBS_THRESH filtering threshold as ">=0.1".
> >> > > > >
> >> > > > >
> >> > > > > Here's a link to the table which defines the contents of
the
> ORANK
> >> > line
> >> > > > > type:
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://dtcenter.github.io/MET/Users_Guide/ensemble-
stat.html?highlight=orank%20line%20type#id7
> >> > > > >
> >> > > > >
> >> > > > > Note that the actual observation rank value is in column
#34.
> >> > Checking
> >> > > > the
> >> > > > > data in those last 3 files, you'll notice that the
observation
> >> ranks
> >> > > are
> >> > > > > always listed as bad data:
> >> > > > >
> >> > > > >
> >> > > > > cat ensemble_stat_Reventador_hr04_20190225_200000V.stat
> >> > > > > ensemble_stat_Reventador_hr05_20190225_210000V.stat
> >> > > > > ensemble_stat_Reventador_hr06_20190225_220000V.stat |
grep
> ORANK
> >> |
> >> > awk
> >> > > > > '{print $34}' | sort -u
> >> > > > >
> >> > > > >
> >> > > > > NA
> >> > > > >
> >> > > > >
> >> > > > > So now the question is why? You're using the
"censor_thresh" and
> >> > > > > "censor_val" options to reset the forecast values that
are
> "<0.1"
> >> to
> >> > > bad
> >> > > > > data. So that introduces a lot of bad data values. And
that
> >> results
> >> > in
> >> > > > the
> >> > > > > rank of the observation being computed as bad data. This
is
> likely
> >> > > > > controlled by how you have the "vld_thresh" or
"ens_vld_thresh"
> >> set
> >> > in
> >> > > > your
> >> > > > > Ensemble-Stat configuration file.
> >> > > > >
> >> > > > >
> >> > > > > John
> >> > > > >
> >> > > > > On Thu, Aug 27, 2020 at 8:48 AM binyu.wang at noaa.gov via
RT <
> >> > > > > met_help at ucar.edu> wrote:
> >> > > > >
> >> > > > > >
> >> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453
> >
> >> > > > > >
> >> > > > > > Hello Tatiana,
> >> > > > > >
> >> > > > > > Sorry to ask more: as I said in previous mail I can not
find
> the
> >> > > other
> >> > > > 3
> >> > > > > > files from MET_Viewer because there is no RHIST inside
of
> those
> >> > > files,
> >> > > > if
> >> > > > > > this is because there is no matched pairs between obs.
and
> >> > forecast,
> >> > > > why
> >> > > > > I
> >> > > > > > can see them in the RELY plot? (see the attachment
please).
> Are
> >> > RELY
> >> > > > > plots
> >> > > > > > created based on matching pairs?
> >> > > > > >
> >> > > > > > Thank you.
> >> > > > > > Binyu
> >> > > > > >
> >> > > > > >
> >> > > > > > On Wed, Aug 26, 2020 at 10:47 PM Binyu Wang - NOAA
Affiliate <
> >> > > > > > binyu.wang at noaa.gov> wrote:
> >> > > > > >
> >> > > > > > > Hello,
> >> > > > > > >
> >> > > > > > > I think I know the answer now, but just to confirm
with you:
> >> if
> >> > > there
> >> > > > > is
> >> > > > > > > no RHIST line type in the stat file, which indicates
there
> is
> >> no
> >> > > > > matching
> >> > > > > > > pair at all between obs. and forecast, is that
correct?
> >> > > > > > >
> >> > > > > > > Thank you.
> >> > > > > > >
> >> > > > > > > On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via RT
<
> >> > > > > met_help at ucar.edu>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > >> Binyu,
> >> > > > > > >> All 5 ensemble_stat files were successfully loaded
to the
> DB
> >> and
> >> > > the
> >> > > > > > data
> >> > > > > > >> is available for potting. Here is the example:
> >> 20200826_164848.
> >> > > > > > >> Your screenshot displays that you try to create a
histogram
> >> > plot.
> >> > > > The
> >> > > > > > >> data for this plot comes from line_data_rhist table.
This
> >> table
> >> > > gets
> >> > > > > > filled
> >> > > > > > >> with 'RHIST' line type.
> >> > > > > > >>
'ensemble_stat_Reventador_hr04_20190225_200000V.stat' file
> >> > > contains
> >> > > > > only
> >> > > > > > >> 'ORANK' line type so non of it's records were saved
into
> >> > > > > line_data_rhist
> >> > > > > > >> table.
> >> > > > > > >> Currently line_data_rhist table contains data for
> >> > fcst_valid_beg:
> >> > > > > > >> '2019-02-25 18:00:00' and '2019-02-25 19:00:00' (
> fcst_lead:
> >> > 20000
> >> > > > and
> >> > > > > > >> 30000). The data came from
> >> > > > > > >> ensemble_stat_Reventador_hr02_20190225_180000V.stat
and
> >> > > > > > >> ensemble_stat_Reventador_hr03_20190225_190000V.stat
> >> > > > > > >>
> >> > > > > > >> Could you check if other files have records with
RHIST line
> >> > type.
> >> > > > > > >>
> >> > > > > > >> Tatiana
> >> > > > > > >>
> >> > > > > > >> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov
wrote:
> >> > > > > > >> > Tatiana,
> >> > > > > > >> >
> >> > > > > > >> > Yes that is weird. If you see the attached file,
you can
> >> see
> >> > > > > > >> > there only two files were uploaded. I expect to
have all
> >> 5.
> >> > > > > > >> > Binyu
> >> > > > > > >> >
> >> > > > > > >> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek via
RT
> >> > > > > > >> > <met_help at ucar.edu>
> >> > > > > > >> > wrote:
> >> > > > > > >> >
> >> > > > > > >> > > Binyu,
> >> > > > > > >> > >
> >> > > > > > >> > > In your log file there is a record that
> >> > > > > > >> > >
'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
> was
> >> > > > > > >> > > successfully
> >> > > > > > >> > > loaded - all 53 lines.
> >> > > > > > >> > > I also can create a plot using the data from
this file
> -
> >> > ORANK
> >> > > > > > >> > > statistics
> >> > > > > > >> > > with 40000 forecast lead: 20200826_151853.
> >> > > > > > >> > >
> >> > > > > > >> > > Tatiana
> >> > > > > > >> > >
> >> > > > > > >> > >
> >> > > > > > >> > > On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov
> wrote:
> >> > > > > > >> > > > Tatiana,
> >> > > > > > >> > > >
> >> > > > > > >> > > > I didn't see any error from the log file. I
attached
> >> the
> >> > log
> >> > > > > file
> >> > > > > > >> > > > and two
> >> > > > > > >> > > > of the stat files:
> >> > > > > > >> > > >
ensemble_stat_Reventador_hr03_20190225_190000V.stat
> >> > > > > > >> > > is
> >> > > > > > >> > > > readable from METViewer, but
> >> > > > > > >> > > >
"ensemble_stat_Reventador_hr04_20190225_200000V.stat"
> >> not.
> >> > > > > > >> > > >
> >> > > > > > >> > > > Thank you.
> >> > > > > > >> > > >
> >> > > > > > >> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek
via RT
> >> > > > > > >> > > > <met_help at ucar.edu>
> >> > > > > > >> > > > wrote:
> >> > > > > > >> > > >
> >> > > > > > >> > > > > Hi Binyu,
> >> > > > > > >> > > > > I assume that your files are on WCOSS. I
don't have
> >> an
> >> > > > access
> >> > > > > to
> >> > > > > > >> > > > > the
> >> > > > > > >> > > > > server and can't see the files.
> >> > > > > > >> > > > > Do you have logging messages from the load?
Any
> >> errors?
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > I see that only 5 ensemble_stat files were
loaded
> to
> >> > this
> >> > > > > > >> > > > > database:
> >> > > > > > >> > > > > |
> >> ensemble_stat_Reventador_hr02_20190225_180000V.stat |
> >> > > > > > >> > > > > |
> >> ensemble_stat_Reventador_hr03_20190225_190000V.stat |
> >> > > > > > >> > > > > |
> >> ensemble_stat_Reventador_hr04_20190225_200000V.stat |
> >> > > > > > >> > > > > |
> >> ensemble_stat_Reventador_hr05_20190225_210000V.stat |
> >> > > > > > >> > > > > |
> ensemble_stat_Reventador_hr06_20190225_220000V.stat
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > Tatiana
> >> > > > > > >> > > > >
> >> > > > > > >> > > > > On Tue Aug 25 20:02:54 2020,
binyu.wang at noaa.gov
> >> wrote:
> >> > > > > > >> > > > > > Hello,
> >> > > > > > >> > > > > >
> >> > > > > > >> > > > > > I created 5 ensemble_stat* files (from
hr18-
> hr22)
> >> > > under
> >> > > > > the
> >> > > > > > >> > > directory
> >> > > > > > >> > > > > > below, but only the first two (hr18 and
hr19)
> >> could
> >> > be
> >> > > > read
> >> > > > > > >> > > > > > from
> >> > > > > > >> > > > > > METVIewer. Those files are under database
> >> > > > > > >> > > "mv_g2g_met_verf_Reventador"
> >> > > > > > >> > > > > >
> >> > > > > > >> > > > > >
> >> > > > > > >> > >
> >> > > > > > >>
> >> > > > >
> >> > >
> >>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> >> > > > > > >> > > > > >
> >> > > > > > >> > > > > > What is the problem? Thank you.
> >> > > > > > >> > > > > > Binyu
> >> > > > > > >> > > > >
> >> > > > > > >> > > > >
> >> > > > > > >> > > > >
> >> > > > > > >> > > > >
> >> > > > > > >> > >
> >> > > > > > >> > >
> >> > > > > > >> > >
> >> > > > > > >> > >
> >> > > > > > >>
> >> > > > > > >>
> >> > > > > > >>
> >> > > > > > >>
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> >
> >> >
> >>
> >>
>
>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: binyu.wang at noaa.gov
Time: Thu Aug 27 16:23:18 2020
Hello Tatiana,
Would you please help to rename the database "mv_g2g_met_verf_Raikoke"
to
"mv_g2g_met_verf_Raikoke.all.data"?
Thank you.
On Thu, Aug 27, 2020 at 2:16 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> There is no easy way to rename the database.
> I can create a new database with a new name, dump the data from the
old
> database, load it to the new one and finally remove the old
database.
>
>
> Tatiana
>
> On Thu Aug 27 12:04:00 2020, binyu.wang at noaa.gov wrote:
> > Another quick question: Can we change the database name without
having to
> > drop the database and reload all of the data again?
> >
> > Binyu
> >
> > On Thu, Aug 27, 2020 at 2:03 PM Binyu Wang - NOAA Affiliate <
> > binyu.wang at noaa.gov> wrote:
> >
> > > Tatiana,
> > >
> > > "PCT" is generated later using the ensemble*stat files which I
sent to
> you
> > > earlier. So you didn't see it in those two files. I didn't
understand:
> if
> > > the 3 files didn't have "RHIST" because there was no matching
pair
> > > during that time period, why could they be used for RELY plot?
> > >
> > > Binyu
> > >
> > > On Thu, Aug 27, 2020 at 1:43 PM Tatiana Burek via RT <
> met_help at ucar.edu>
> > > wrote:
> > >
> > >> Binyu
> > >>
> > >> Reliability plot uses data from line_data_pct table ( "PCT"
line type
> ).
> > >> Both stat files that you sent me don't contain PCT line type
and
> variable
> > >> 'VAFTD_ENS_FREQ_ge0.1'. Did you have data in this database
before the
> > >> latest load?
> > >>
> > >> Tatiana
> > >>
> > >> On Thu Aug 27 08:48:35 2020, binyu.wang at noaa.gov wrote:
> > >> > Hello Tatiana,
> > >> >
> > >> > Sorry to ask more: as I said in previous mail I can not find
the
> other
> > >> > 3
> > >> > files from MET_Viewer because there is no RHIST inside of
those
> files,
> > >> > if
> > >> > this is because there is no matched pairs between obs. and
forecast,
> > >> > why I
> > >> > can see them in the RELY plot? (see the attachment please).
Are RELY
> > >> > plots
> > >> > created based on matching pairs?
> > >> >
> > >> > Thank you.
> > >> > Binyu
> > >> >
> > >> >
> > >> > On Wed, Aug 26, 2020 at 10:47 PM Binyu Wang - NOAA Affiliate
<
> > >> > binyu.wang at noaa.gov> wrote:
> > >> >
> > >> > > Hello,
> > >> > >
> > >> > > I think I know the answer now, but just to confirm with
you: if
> there
> > >> > > is
> > >> > > no RHIST line type in the stat file, which indicates there
is no
> > >> > > matching
> > >> > > pair at all between obs. and forecast, is that correct?
> > >> > >
> > >> > > Thank you.
> > >> > >
> > >> > > On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via RT
> > >> > > <met_help at ucar.edu>
> > >> > > wrote:
> > >> > >
> > >> > >> Binyu,
> > >> > >> All 5 ensemble_stat files were successfully loaded to the
DB and
> the
> > >> > >> data
> > >> > >> is available for potting. Here is the example:
20200826_164848.
> > >> > >> Your screenshot displays that you try to create a
histogram plot.
> > >> > >> The
> > >> > >> data for this plot comes from line_data_rhist table. This
table
> gets
> > >> > >> filled
> > >> > >> with 'RHIST' line type.
> > >> > >> 'ensemble_stat_Reventador_hr04_20190225_200000V.stat' file
> contains
> > >> > >> only
> > >> > >> 'ORANK' line type so non of it's records were saved into
> > >> > >> line_data_rhist
> > >> > >> table.
> > >> > >> Currently line_data_rhist table contains data for
fcst_valid_beg:
> > >> > >> '2019-02-25 18:00:00' and '2019-02-25 19:00:00' (
fcst_lead:
> 20000
> > >> > >> and
> > >> > >> 30000). The data came from
> > >> > >> ensemble_stat_Reventador_hr02_20190225_180000V.stat and
> > >> > >> ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > >> > >>
> > >> > >> Could you check if other files have records with RHIST
line type.
> > >> > >>
> > >> > >> Tatiana
> > >> > >>
> > >> > >> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov wrote:
> > >> > >> > Tatiana,
> > >> > >> >
> > >> > >> > Yes that is weird. If you see the attached file, you can
see
> > >> > >> > there only two files were uploaded. I expect to have
all 5.
> > >> > >> > Binyu
> > >> > >> >
> > >> > >> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek via RT
> > >> > >> > <met_help at ucar.edu>
> > >> > >> > wrote:
> > >> > >> >
> > >> > >> > > Binyu,
> > >> > >> > >
> > >> > >> > > In your log file there is a record that
> > >> > >> > > 'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
was
> > >> > >> > > successfully
> > >> > >> > > loaded - all 53 lines.
> > >> > >> > > I also can create a plot using the data from this file
-
> ORANK
> > >> > >> > > statistics
> > >> > >> > > with 40000 forecast lead: 20200826_151853.
> > >> > >> > >
> > >> > >> > > Tatiana
> > >> > >> > >
> > >> > >> > >
> > >> > >> > > On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov
wrote:
> > >> > >> > > > Tatiana,
> > >> > >> > > >
> > >> > >> > > > I didn't see any error from the log file. I attached
the
> log
> > >> > >> > > > file
> > >> > >> > > > and two
> > >> > >> > > > of the stat files:
> > >> > >> > > > ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > >> > >> > > is
> > >> > >> > > > readable from METViewer, but
> > >> > >> > > >
"ensemble_stat_Reventador_hr04_20190225_200000V.stat" not.
> > >> > >> > > >
> > >> > >> > > > Thank you.
> > >> > >> > > >
> > >> > >> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek via RT
> > >> > >> > > > <met_help at ucar.edu>
> > >> > >> > > > wrote:
> > >> > >> > > >
> > >> > >> > > > > Hi Binyu,
> > >> > >> > > > > I assume that your files are on WCOSS. I don't
have an
> > >> > >> > > > > access to
> > >> > >> > > > > the
> > >> > >> > > > > server and can't see the files.
> > >> > >> > > > > Do you have logging messages from the load? Any
errors?
> > >> > >> > > > >
> > >> > >> > > > > I see that only 5 ensemble_stat files were loaded
to this
> > >> > >> > > > > database:
> > >> > >> > > > > |
ensemble_stat_Reventador_hr02_20190225_180000V.stat |
> > >> > >> > > > > |
ensemble_stat_Reventador_hr03_20190225_190000V.stat |
> > >> > >> > > > > |
ensemble_stat_Reventador_hr04_20190225_200000V.stat |
> > >> > >> > > > > |
ensemble_stat_Reventador_hr05_20190225_210000V.stat |
> > >> > >> > > > > |
ensemble_stat_Reventador_hr06_20190225_220000V.stat
> > >> > >> > > > >
> > >> > >> > > > > Tatiana
> > >> > >> > > > >
> > >> > >> > > > > On Tue Aug 25 20:02:54 2020, binyu.wang at noaa.gov
wrote:
> > >> > >> > > > > > Hello,
> > >> > >> > > > > >
> > >> > >> > > > > > I created 5 ensemble_stat* files (from hr18-
hr22)
> under
> > >> > >> > > > > > the
> > >> > >> > > directory
> > >> > >> > > > > > below, but only the first two (hr18 and hr19)
could be
> > >> > >> > > > > > read
> > >> > >> > > > > > from
> > >> > >> > > > > > METVIewer. Those files are under database
> > >> > >> > > "mv_g2g_met_verf_Reventador"
> > >> > >> > > > > >
> > >> > >> > > > > >
> > >> > >> > >
> > >> > >>
> > >>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > >> > >> > > > > >
> > >> > >> > > > > > What is the problem? Thank you.
> > >> > >> > > > > > Binyu
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > >
> > >> > >> > >
> > >> > >> > >
> > >> > >> > >
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >>
> > >>
> > >>
> > >>
> > >>
>
>
>
>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: binyu.wang at noaa.gov
Time: Thu Aug 27 16:31:11 2020
That makes sense. Then what is the point of the option "ens_thresh" in
the
ensemble_stat config file?It should always be 1, is that correct?
On Thu, Aug 27, 2020 at 6:21 PM John Halley Gotway via RT
<met_help at ucar.edu>
wrote:
> No, that will not work as you'd hope. In fact, this same issue came
up
> recently on a GitHub issue, and I wrote up a comment about it.
>
> Please read through that comment here:
> https://github.com/dtcenter/MET/issues/1475#issuecomment-682202522
>
> The point here is that if any of the ensemble member values have bad
data,
> then that point is excluded from the ensemble statistics. There is
no
> configuration option to override that behavior. And the reason one
is not
> provided is that we have not defined a clear algorithm for how to
compute
> statistics when missing data values are present.
>
> For example, let's say you have a 21 member ensemble and just 3
> observations:
> - For point 1, all 21 members contain valid data.
> - For point 2, 15 of 21 contain valid data.
> - For point 3, 10 of 21 contain valid data.
>
> Assume that you just ignore the bad data values and compute the
observation
> rank against the valid values. Maybe the obs 1 has rank 5 of 21, obs
2 has
> rank 10 of 15, and obs 3 has rank 1 of 10. These 3 ranks cannot be
combined
> into a rank histogram. The number of members must remain constant in
order
> for them to be combined. You could conceivably still compute a PHIST
line,
> but most users won't understand the nuance of what is occurring. And
> dealing with missing data values like this gets very messy. To
change this
> behavior, we'd need an algorithm with a clear set of rules to
enforce based
> on advice from statisticians on proper methods.
>
> John
>
> On Thu, Aug 27, 2020 at 4:09 PM binyu.wang at noaa.gov via RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453 >
> >
> > I figured out the answer for question 2. But for the first one:
based on
> > column 35, it seems the majority of them are bigger than 10
(totally I
> > have 21 members), does that mean if I set "ens_vld_thresh" = 0.5,
then I
> > could get "RHIST"? for the last 3 members?
> >
> > Binyu
> >
> > On Thu, Aug 27, 2020 at 5:21 PM Binyu Wang - NOAA Affiliate <
> > binyu.wang at noaa.gov> wrote:
> >
> > > I got more questions for the ORANK table at "
> > >
> >
> https://dtcenter.github.io/MET/Users_Guide/ensemble-
stat.html?highlight=orank%20line%20type#id7
> > > ",
> > >
> > > 1. For column 35 (Number of valid ensemble values): look at file
> > > "ensemble_stat_Reventador_hr04_20190225_200000V.stat", If I have
0
> pairs
> > > that meet the requirement, why I have "10, 20, 18...." under
that
> column?
> > > Does that mean I have 10 (or 20 or 18) out of 21 ensemble
members?
> > > (totally I have 21 ensemble members) . So if I reduce
"ens_vld_thresh"
> > from
> > > 1 to 0.5, I will get "RHIST"?
> > >
> > >
> > > 2. What does that mean "Last-6"?
> > >
> > > Thank you very much for your help.
> > > Binyu
> > >
> > >
> > > On Thu, Aug 27, 2020 at 5:04 PM John Halley Gotway via RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >> Binyu,
> > >>
> > >> I do not think that will work as you expect. For each
observation
> point,
> > >> if
> > >> any of the ensemble members contain a bad data value at that
point,
> then
> > >> it
> > >> will not be included in the statistics. As we've seen in your
data,
> that
> > >> is
> > >> the case for ALL observations other than some in the first 2
hours.
> > >>
> > >> I would recommend using the "obs_thresh = >=0.1;" filtering
threshold
> > but
> > >> NOT apply the censoring logic to the forecast data.
> > >>
> > >> But as long as you understand the options, it's up to you to
decide
> how
> > to
> > >> configure it.
> > >>
> > >> John
> > >>
> > >> On Thu, Aug 27, 2020 at 2:37 PM binyu.wang at noaa.gov via RT <
> > >> met_help at ucar.edu> wrote:
> > >>
> > >> >
> > >> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453
>
> > >> >
> > >> > Thank you very much for the clarification. I will discuss
with my
> > >> advisor.
> > >> > I think we will need to keep the " censor_thresh ", but we
can
> change
> > >> the
> > >> > "vld_thresh" or "ens_vld_thresh" .
> > >> >
> > >> > Binyu
> > >> >
> > >> > On Thu, Aug 27, 2020 at 4:00 PM John Halley Gotway via RT <
> > >> > met_help at ucar.edu>
> > >> > wrote:
> > >> >
> > >> > > Binyu,
> > >> > >
> > >> > > In ensemble-stat, there is no difference between "matched
pairs"
> and
> > >> > > "ensemble pairs". It would be better if we made those log
> messages
> > >> use
> > >> > > consistent language.
> > >> > >
> > >> > > So regarding this log message:
> > >> > > *DEBUG 2: Skipping 53 of 53 points due to missing ensemble
> values.*
> > >> > >
> > >> > > Here's the line of code where that's being written:
> > >> > >
> > >> > >
> > >> >
> > >>
> >
>
https://github.com/dtcenter/MET/blob/2c9ae440a84024fbf62caa64c1b747f9a912236f/met/src/libcode/vx_statistics/pair_data_ensemble.cc#L432
> > >> > >
> > >> > > So the rule here is that if any of the ensemble members
contain
> bad
> > >> data
> > >> > > for a particular observation, then that point is not used
in the
> > >> > > verification. And why not? We ultimately compute the rank
of the
> > >> > > observation value, relative to the ensemble member values.
If the
> > >> > ensemble
> > >> > > member values contain bad data, there's no good way of
computing
> the
> > >> rank
> > >> > > from them.
> > >> > >
> > >> > > So when you use the "censor_thresh" and "censor_val" logic
on the
> > >> input
> > >> > > ensemble fields, you're introducing lots of bad data values
into
> > your
> > >> > > forecast fields. For hours 2 and 3, there are 17 and 11
points,
> > >> > > respectively, for which none of the ensemble members
contain bad
> > data.
> > >> > But
> > >> > > for the rest of the hours, there are 0 points for which
none of
> the
> > >> > > ensemble members contain bad data.
> > >> > >
> > >> > > So I would recommend that you stop using the
"censor_thresh" and
> > >> > > "censor_val" logic. It think it's confusing your results.
> Requiring
> > >> that
> > >> > > ALL ensemble members and the observation be >=0.1 results
in no
> > pairs
> > >> > left
> > >> > > with which to compute statistics.
> > >> > >
> > >> > > John
> > >> > >
> > >> > > On Thu, Aug 27, 2020 at 1:41 PM binyu.wang at noaa.gov via RT
<
> > >> > > met_help at ucar.edu> wrote:
> > >> > >
> > >> > > >
> > >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453 >
> > >> > > >
> > >> > > > Thank you very much for the detailed explanation, John.
> > >> > > >
> > >> > > > So basically, that means there is no any matched case (or
site
> > b/c I
> > >> > used
> > >> > > > point_obs) between obs. and forecasting that meet all
the
> > >> > > > threshold/censors for the last 3 files. If you look at
part of
> the
> > >> log
> > >> > > file
> > >> > > > below (this is for the 3rd file):
> > >> > > >
> > >> > > > DEBUG 2: Processing point verification VAFTD/L14000-0
versus
> > >> > > ash/Z10000-0,
> > >> > > > for observation type ADPUPA, over region FULL, for
interpolation
> > >> method
> > >> > > > NEAREST(1), using 53 matched pairs.
> > >> > > > DEBUG 2: Skipping 53 of 53 points due to missing ensemble
> values.
> > >> > > > DEBUG 3: Using 0 of 53 ensemble pairs for observation
filtering
> > >> > threshold
> > >> > > > >=0.1.
> > >> > > >
> > >> > > > So what is the difference between "matched pairs" and
"ensemble
> > >> pairs"?
> > >> > > It
> > >> > > > seems "ensemble pairs is 0" is why I got no "PHIST", how
about
> the
> > >> > > matched
> > >> > > > pairs? Does that mean if one site was catched by all 20
> ensemble
> > >> > > members,
> > >> > > > then I have 20 matched pairs?
> > >> > > >
> > >> > > > Thank you.
> > >> > > >
> > >> > > > On Thu, Aug 27, 2020 at 2:01 PM John Halley Gotway via RT
<
> > >> > > > met_help at ucar.edu>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Binyu,
> > >> > > > >
> > >> > > > > Tatiana asked me to take a look at the RHIST line type
being
> > >> > > unexpectedly
> > >> > > > > absent from some of your runs of Ensemble-Stat. I
logged onto
> > >> WCOSS
> > >> > and
> > >> > > > > took a look in:
> > >> > > > >
> > >> > > > >
> > >> > >
> > >>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > >> > > > >
> > >> > > > > In there, I see 5 ensemble_stat output files. I have
listed
> each
> > >> > below
> > >> > > > > followed by the line types which appear in each file.
So
> that's
> > >> > column
> > >> > > > #24
> > >> > > > > from each:
> > >> > > > >
> > >> > > > > ensemble_stat_Reventador_hr02_20190225_180000V.stat
ECNT ORANK
> > >> PHIST
> > >> > > RELP
> > >> > > > > RHIST SSVAR
> > >> > > > >
> > >> > > > > ensemble_stat_Reventador_hr03_20190225_190000V.stat
ECNT ORANK
> > >> PHIST
> > >> > > RELP
> > >> > > > > RHIST SSVAR
> > >> > > > >
> > >> > > > > ensemble_stat_Reventador_hr04_20190225_200000V.stat
ORANK
> > >> > > > >
> > >> > > > > ensemble_stat_Reventador_hr05_20190225_210000V.stat
ORANK
> > >> > > > >
> > >> > > > > ensemble_stat_Reventador_hr06_20190225_220000V.stat
ORANK
> > >> > > > >
> > >> > > > >
> > >> > > > > The ORANK line type is basically the "matched pairs"
for
> > ensemble
> > >> > > > > verification. Notice that the first 2 hours have the
ORANK
> lines
> > >> > along
> > >> > > > with
> > >> > > > > other statistic line types, such as RHIST. However, the
last 3
> > >> hours
> > >> > > only
> > >> > > > > have the ORANK line type.
> > >> > > > >
> > >> > > > >
> > >> > > > > Looking at the output for the first 2 hours, I see that
you've
> > >> > applied
> > >> > > > the
> > >> > > > > OBS_THRESH filtering threshold as ">=0.1".
> > >> > > > >
> > >> > > > >
> > >> > > > > Here's a link to the table which defines the contents
of the
> > ORANK
> > >> > line
> > >> > > > > type:
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://dtcenter.github.io/MET/Users_Guide/ensemble-
stat.html?highlight=orank%20line%20type#id7
> > >> > > > >
> > >> > > > >
> > >> > > > > Note that the actual observation rank value is in
column #34.
> > >> > Checking
> > >> > > > the
> > >> > > > > data in those last 3 files, you'll notice that the
observation
> > >> ranks
> > >> > > are
> > >> > > > > always listed as bad data:
> > >> > > > >
> > >> > > > >
> > >> > > > > cat ensemble_stat_Reventador_hr04_20190225_200000V.stat
> > >> > > > > ensemble_stat_Reventador_hr05_20190225_210000V.stat
> > >> > > > > ensemble_stat_Reventador_hr06_20190225_220000V.stat |
grep
> > ORANK
> > >> |
> > >> > awk
> > >> > > > > '{print $34}' | sort -u
> > >> > > > >
> > >> > > > >
> > >> > > > > NA
> > >> > > > >
> > >> > > > >
> > >> > > > > So now the question is why? You're using the
"censor_thresh"
> and
> > >> > > > > "censor_val" options to reset the forecast values that
are
> > "<0.1"
> > >> to
> > >> > > bad
> > >> > > > > data. So that introduces a lot of bad data values. And
that
> > >> results
> > >> > in
> > >> > > > the
> > >> > > > > rank of the observation being computed as bad data.
This is
> > likely
> > >> > > > > controlled by how you have the "vld_thresh" or
> "ens_vld_thresh"
> > >> set
> > >> > in
> > >> > > > your
> > >> > > > > Ensemble-Stat configuration file.
> > >> > > > >
> > >> > > > >
> > >> > > > > John
> > >> > > > >
> > >> > > > > On Thu, Aug 27, 2020 at 8:48 AM binyu.wang at noaa.gov via
RT <
> > >> > > > > met_help at ucar.edu> wrote:
> > >> > > > >
> > >> > > > > >
> > >> > > > > > <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453
> > >
> > >> > > > > >
> > >> > > > > > Hello Tatiana,
> > >> > > > > >
> > >> > > > > > Sorry to ask more: as I said in previous mail I can
not find
> > the
> > >> > > other
> > >> > > > 3
> > >> > > > > > files from MET_Viewer because there is no RHIST
inside of
> > those
> > >> > > files,
> > >> > > > if
> > >> > > > > > this is because there is no matched pairs between
obs. and
> > >> > forecast,
> > >> > > > why
> > >> > > > > I
> > >> > > > > > can see them in the RELY plot? (see the attachment
please).
> > Are
> > >> > RELY
> > >> > > > > plots
> > >> > > > > > created based on matching pairs?
> > >> > > > > >
> > >> > > > > > Thank you.
> > >> > > > > > Binyu
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Wed, Aug 26, 2020 at 10:47 PM Binyu Wang - NOAA
> Affiliate <
> > >> > > > > > binyu.wang at noaa.gov> wrote:
> > >> > > > > >
> > >> > > > > > > Hello,
> > >> > > > > > >
> > >> > > > > > > I think I know the answer now, but just to confirm
with
> you:
> > >> if
> > >> > > there
> > >> > > > > is
> > >> > > > > > > no RHIST line type in the stat file, which
indicates there
> > is
> > >> no
> > >> > > > > matching
> > >> > > > > > > pair at all between obs. and forecast, is that
correct?
> > >> > > > > > >
> > >> > > > > > > Thank you.
> > >> > > > > > >
> > >> > > > > > > On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via
RT <
> > >> > > > > met_help at ucar.edu>
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > >> Binyu,
> > >> > > > > > >> All 5 ensemble_stat files were successfully loaded
to the
> > DB
> > >> and
> > >> > > the
> > >> > > > > > data
> > >> > > > > > >> is available for potting. Here is the example:
> > >> 20200826_164848.
> > >> > > > > > >> Your screenshot displays that you try to create a
> histogram
> > >> > plot.
> > >> > > > The
> > >> > > > > > >> data for this plot comes from line_data_rhist
table. This
> > >> table
> > >> > > gets
> > >> > > > > > filled
> > >> > > > > > >> with 'RHIST' line type.
> > >> > > > > > >>
'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
> file
> > >> > > contains
> > >> > > > > only
> > >> > > > > > >> 'ORANK' line type so non of it's records were
saved into
> > >> > > > > line_data_rhist
> > >> > > > > > >> table.
> > >> > > > > > >> Currently line_data_rhist table contains data for
> > >> > fcst_valid_beg:
> > >> > > > > > >> '2019-02-25 18:00:00' and '2019-02-25 19:00:00' (
> > fcst_lead:
> > >> > 20000
> > >> > > > and
> > >> > > > > > >> 30000). The data came from
> > >> > > > > > >>
ensemble_stat_Reventador_hr02_20190225_180000V.stat and
> > >> > > > > > >>
ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > >> > > > > > >>
> > >> > > > > > >> Could you check if other files have records with
RHIST
> line
> > >> > type.
> > >> > > > > > >>
> > >> > > > > > >> Tatiana
> > >> > > > > > >>
> > >> > > > > > >> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov
wrote:
> > >> > > > > > >> > Tatiana,
> > >> > > > > > >> >
> > >> > > > > > >> > Yes that is weird. If you see the attached file,
you
> can
> > >> see
> > >> > > > > > >> > there only two files were uploaded. I expect to
have
> all
> > >> 5.
> > >> > > > > > >> > Binyu
> > >> > > > > > >> >
> > >> > > > > > >> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek
via RT
> > >> > > > > > >> > <met_help at ucar.edu>
> > >> > > > > > >> > wrote:
> > >> > > > > > >> >
> > >> > > > > > >> > > Binyu,
> > >> > > > > > >> > >
> > >> > > > > > >> > > In your log file there is a record that
> > >> > > > > > >> > >
'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
> > was
> > >> > > > > > >> > > successfully
> > >> > > > > > >> > > loaded - all 53 lines.
> > >> > > > > > >> > > I also can create a plot using the data from
this
> file
> > -
> > >> > ORANK
> > >> > > > > > >> > > statistics
> > >> > > > > > >> > > with 40000 forecast lead: 20200826_151853.
> > >> > > > > > >> > >
> > >> > > > > > >> > > Tatiana
> > >> > > > > > >> > >
> > >> > > > > > >> > >
> > >> > > > > > >> > > On Wed Aug 26 08:32:17 2020,
binyu.wang at noaa.gov
> > wrote:
> > >> > > > > > >> > > > Tatiana,
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > I didn't see any error from the log file. I
> attached
> > >> the
> > >> > log
> > >> > > > > file
> > >> > > > > > >> > > > and two
> > >> > > > > > >> > > > of the stat files:
> > >> > > > > > >> > > >
ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > >> > > > > > >> > > is
> > >> > > > > > >> > > > readable from METViewer, but
> > >> > > > > > >> > > >
> "ensemble_stat_Reventador_hr04_20190225_200000V.stat"
> > >> not.
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > Thank you.
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana
Burek via
> RT
> > >> > > > > > >> > > > <met_help at ucar.edu>
> > >> > > > > > >> > > > wrote:
> > >> > > > > > >> > > >
> > >> > > > > > >> > > > > Hi Binyu,
> > >> > > > > > >> > > > > I assume that your files are on WCOSS. I
don't
> have
> > >> an
> > >> > > > access
> > >> > > > > to
> > >> > > > > > >> > > > > the
> > >> > > > > > >> > > > > server and can't see the files.
> > >> > > > > > >> > > > > Do you have logging messages from the
load? Any
> > >> errors?
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > I see that only 5 ensemble_stat files were
loaded
> > to
> > >> > this
> > >> > > > > > >> > > > > database:
> > >> > > > > > >> > > > > |
> > >> ensemble_stat_Reventador_hr02_20190225_180000V.stat |
> > >> > > > > > >> > > > > |
> > >> ensemble_stat_Reventador_hr03_20190225_190000V.stat |
> > >> > > > > > >> > > > > |
> > >> ensemble_stat_Reventador_hr04_20190225_200000V.stat |
> > >> > > > > > >> > > > > |
> > >> ensemble_stat_Reventador_hr05_20190225_210000V.stat |
> > >> > > > > > >> > > > > |
> > ensemble_stat_Reventador_hr06_20190225_220000V.stat
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > Tatiana
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > > On Tue Aug 25 20:02:54 2020,
binyu.wang at noaa.gov
> > >> wrote:
> > >> > > > > > >> > > > > > Hello,
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > > > I created 5 ensemble_stat* files (from
hr18-
> > hr22)
> > >> > > under
> > >> > > > > the
> > >> > > > > > >> > > directory
> > >> > > > > > >> > > > > > below, but only the first two (hr18 and
hr19)
> > >> could
> > >> > be
> > >> > > > read
> > >> > > > > > >> > > > > > from
> > >> > > > > > >> > > > > > METVIewer. Those files are under
database
> > >> > > > > > >> > > "mv_g2g_met_verf_Reventador"
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > >
> > >> > > > > > >>
> > >> > > > >
> > >> > >
> > >>
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > >> > > > > > >> > > > > >
> > >> > > > > > >> > > > > > What is the problem? Thank you.
> > >> > > > > > >> > > > > > Binyu
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > >
> > >> > > > > > >> > > > >
> > >> > > > > > >> > >
> > >> > > > > > >> > >
> > >> > > > > > >> > >
> > >> > > > > > >> > >
> > >> > > > > > >>
> > >> > > > > > >>
> > >> > > > > > >>
> > >> > > > > > >>
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > > >
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >>
> > >>
> >
> >
>
>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: John Halley Gotway
Time: Thu Aug 27 16:43:10 2020
Ensemble-Stat does 2 main things:
(1) Derive fields, such as ensemble mean, spread, and probabilities.
(2) Compute verification statistics from the full ensemble.
The "ens_thresh" option applies to step (1). It allows the user to
decide
how much data is required for ensemble mean and probability fields to
be
defined. Perhaps 1 missing out of 21 members is fine. But perhaps 20
missing out of 21 members is not fine. If the required "ens_thresh"
criteria is not met, Ensemble-Stat will ERROR OUT and return bad
status.
And that's a good thing. Because otherwise, you'd be mixing results
for
when you had "enough" data with results for when you did not have
"enough"
data.
Handling missing data for step (1) is much more obvious than it is for
step
(2).
But I will say that these 2 steps of Ensemble-Stat have caused
confusion
over the years. And we have considered splitting this functionality
out
into 2 separate tools: one for the ensemble post-processing of step
(1) and
one for the ensemble verification of step (2). But we're not currently
working on that.
John
On Thu, Aug 27, 2020 at 4:31 PM binyu.wang at noaa.gov via RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453 >
>
> That makes sense. Then what is the point of the option "ens_thresh"
in the
> ensemble_stat config file?It should always be 1, is that correct?
>
> On Thu, Aug 27, 2020 at 6:21 PM John Halley Gotway via RT <
> met_help at ucar.edu>
> wrote:
>
> > No, that will not work as you'd hope. In fact, this same issue
came up
> > recently on a GitHub issue, and I wrote up a comment about it.
> >
> > Please read through that comment here:
> > https://github.com/dtcenter/MET/issues/1475#issuecomment-682202522
> >
> > The point here is that if any of the ensemble member values have
bad
> data,
> > then that point is excluded from the ensemble statistics. There is
no
> > configuration option to override that behavior. And the reason one
is not
> > provided is that we have not defined a clear algorithm for how to
compute
> > statistics when missing data values are present.
> >
> > For example, let's say you have a 21 member ensemble and just 3
> > observations:
> > - For point 1, all 21 members contain valid data.
> > - For point 2, 15 of 21 contain valid data.
> > - For point 3, 10 of 21 contain valid data.
> >
> > Assume that you just ignore the bad data values and compute the
> observation
> > rank against the valid values. Maybe the obs 1 has rank 5 of 21,
obs 2
> has
> > rank 10 of 15, and obs 3 has rank 1 of 10. These 3 ranks cannot be
> combined
> > into a rank histogram. The number of members must remain constant
in
> order
> > for them to be combined. You could conceivably still compute a
PHIST
> line,
> > but most users won't understand the nuance of what is occurring.
And
> > dealing with missing data values like this gets very messy. To
change
> this
> > behavior, we'd need an algorithm with a clear set of rules to
enforce
> based
> > on advice from statisticians on proper methods.
> >
> > John
> >
> > On Thu, Aug 27, 2020 at 4:09 PM binyu.wang at noaa.gov via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453 >
> > >
> > > I figured out the answer for question 2. But for the first one:
based
> on
> > > column 35, it seems the majority of them are bigger than 10
(totally I
> > > have 21 members), does that mean if I set "ens_vld_thresh" =
0.5,
> then I
> > > could get "RHIST"? for the last 3 members?
> > >
> > > Binyu
> > >
> > > On Thu, Aug 27, 2020 at 5:21 PM Binyu Wang - NOAA Affiliate <
> > > binyu.wang at noaa.gov> wrote:
> > >
> > > > I got more questions for the ORANK table at "
> > > >
> > >
> >
> https://dtcenter.github.io/MET/Users_Guide/ensemble-
stat.html?highlight=orank%20line%20type#id7
> > > > ",
> > > >
> > > > 1. For column 35 (Number of valid ensemble values): look at
file
> > > > "ensemble_stat_Reventador_hr04_20190225_200000V.stat", If I
have 0
> > pairs
> > > > that meet the requirement, why I have "10, 20, 18...." under
that
> > column?
> > > > Does that mean I have 10 (or 20 or 18) out of 21 ensemble
members?
> > > > (totally I have 21 ensemble members) . So if I reduce
> "ens_vld_thresh"
> > > from
> > > > 1 to 0.5, I will get "RHIST"?
> > > >
> > > >
> > > > 2. What does that mean "Last-6"?
> > > >
> > > > Thank you very much for your help.
> > > > Binyu
> > > >
> > > >
> > > > On Thu, Aug 27, 2020 at 5:04 PM John Halley Gotway via RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >> Binyu,
> > > >>
> > > >> I do not think that will work as you expect. For each
observation
> > point,
> > > >> if
> > > >> any of the ensemble members contain a bad data value at that
point,
> > then
> > > >> it
> > > >> will not be included in the statistics. As we've seen in your
data,
> > that
> > > >> is
> > > >> the case for ALL observations other than some in the first 2
hours.
> > > >>
> > > >> I would recommend using the "obs_thresh = >=0.1;" filtering
> threshold
> > > but
> > > >> NOT apply the censoring logic to the forecast data.
> > > >>
> > > >> But as long as you understand the options, it's up to you to
decide
> > how
> > > to
> > > >> configure it.
> > > >>
> > > >> John
> > > >>
> > > >> On Thu, Aug 27, 2020 at 2:37 PM binyu.wang at noaa.gov via RT <
> > > >> met_help at ucar.edu> wrote:
> > > >>
> > > >> >
> > > >> > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453 >
> > > >> >
> > > >> > Thank you very much for the clarification. I will discuss
with my
> > > >> advisor.
> > > >> > I think we will need to keep the " censor_thresh ", but we
can
> > change
> > > >> the
> > > >> > "vld_thresh" or "ens_vld_thresh" .
> > > >> >
> > > >> > Binyu
> > > >> >
> > > >> > On Thu, Aug 27, 2020 at 4:00 PM John Halley Gotway via RT <
> > > >> > met_help at ucar.edu>
> > > >> > wrote:
> > > >> >
> > > >> > > Binyu,
> > > >> > >
> > > >> > > In ensemble-stat, there is no difference between "matched
pairs"
> > and
> > > >> > > "ensemble pairs". It would be better if we made those
log
> > messages
> > > >> use
> > > >> > > consistent language.
> > > >> > >
> > > >> > > So regarding this log message:
> > > >> > > *DEBUG 2: Skipping 53 of 53 points due to missing
ensemble
> > values.*
> > > >> > >
> > > >> > > Here's the line of code where that's being written:
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
https://github.com/dtcenter/MET/blob/2c9ae440a84024fbf62caa64c1b747f9a912236f/met/src/libcode/vx_statistics/pair_data_ensemble.cc#L432
> > > >> > >
> > > >> > > So the rule here is that if any of the ensemble members
contain
> > bad
> > > >> data
> > > >> > > for a particular observation, then that point is not used
in the
> > > >> > > verification. And why not? We ultimately compute the rank
of the
> > > >> > > observation value, relative to the ensemble member
values. If
> the
> > > >> > ensemble
> > > >> > > member values contain bad data, there's no good way of
computing
> > the
> > > >> rank
> > > >> > > from them.
> > > >> > >
> > > >> > > So when you use the "censor_thresh" and "censor_val"
logic on
> the
> > > >> input
> > > >> > > ensemble fields, you're introducing lots of bad data
values into
> > > your
> > > >> > > forecast fields. For hours 2 and 3, there are 17 and 11
points,
> > > >> > > respectively, for which none of the ensemble members
contain bad
> > > data.
> > > >> > But
> > > >> > > for the rest of the hours, there are 0 points for which
none of
> > the
> > > >> > > ensemble members contain bad data.
> > > >> > >
> > > >> > > So I would recommend that you stop using the
"censor_thresh" and
> > > >> > > "censor_val" logic. It think it's confusing your results.
> > Requiring
> > > >> that
> > > >> > > ALL ensemble members and the observation be >=0.1 results
in no
> > > pairs
> > > >> > left
> > > >> > > with which to compute statistics.
> > > >> > >
> > > >> > > John
> > > >> > >
> > > >> > > On Thu, Aug 27, 2020 at 1:41 PM binyu.wang at noaa.gov via
RT <
> > > >> > > met_help at ucar.edu> wrote:
> > > >> > >
> > > >> > > >
> > > >> > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453
> >
> > > >> > > >
> > > >> > > > Thank you very much for the detailed explanation, John.
> > > >> > > >
> > > >> > > > So basically, that means there is no any matched case
(or site
> > > b/c I
> > > >> > used
> > > >> > > > point_obs) between obs. and forecasting that meet all
the
> > > >> > > > threshold/censors for the last 3 files. If you look at
part of
> > the
> > > >> log
> > > >> > > file
> > > >> > > > below (this is for the 3rd file):
> > > >> > > >
> > > >> > > > DEBUG 2: Processing point verification VAFTD/L14000-0
versus
> > > >> > > ash/Z10000-0,
> > > >> > > > for observation type ADPUPA, over region FULL, for
> interpolation
> > > >> method
> > > >> > > > NEAREST(1), using 53 matched pairs.
> > > >> > > > DEBUG 2: Skipping 53 of 53 points due to missing
ensemble
> > values.
> > > >> > > > DEBUG 3: Using 0 of 53 ensemble pairs for observation
> filtering
> > > >> > threshold
> > > >> > > > >=0.1.
> > > >> > > >
> > > >> > > > So what is the difference between "matched pairs" and
> "ensemble
> > > >> pairs"?
> > > >> > > It
> > > >> > > > seems "ensemble pairs is 0" is why I got no "PHIST",
how about
> > the
> > > >> > > matched
> > > >> > > > pairs? Does that mean if one site was catched by all
20
> > ensemble
> > > >> > > members,
> > > >> > > > then I have 20 matched pairs?
> > > >> > > >
> > > >> > > > Thank you.
> > > >> > > >
> > > >> > > > On Thu, Aug 27, 2020 at 2:01 PM John Halley Gotway via
RT <
> > > >> > > > met_help at ucar.edu>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Binyu,
> > > >> > > > >
> > > >> > > > > Tatiana asked me to take a look at the RHIST line
type being
> > > >> > > unexpectedly
> > > >> > > > > absent from some of your runs of Ensemble-Stat. I
logged
> onto
> > > >> WCOSS
> > > >> > and
> > > >> > > > > took a look in:
> > > >> > > > >
> > > >> > > > >
> > > >> > >
> > > >>
> >
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > > >> > > > >
> > > >> > > > > In there, I see 5 ensemble_stat output files. I have
listed
> > each
> > > >> > below
> > > >> > > > > followed by the line types which appear in each file.
So
> > that's
> > > >> > column
> > > >> > > > #24
> > > >> > > > > from each:
> > > >> > > > >
> > > >> > > > > ensemble_stat_Reventador_hr02_20190225_180000V.stat
ECNT
> ORANK
> > > >> PHIST
> > > >> > > RELP
> > > >> > > > > RHIST SSVAR
> > > >> > > > >
> > > >> > > > > ensemble_stat_Reventador_hr03_20190225_190000V.stat
ECNT
> ORANK
> > > >> PHIST
> > > >> > > RELP
> > > >> > > > > RHIST SSVAR
> > > >> > > > >
> > > >> > > > > ensemble_stat_Reventador_hr04_20190225_200000V.stat
ORANK
> > > >> > > > >
> > > >> > > > > ensemble_stat_Reventador_hr05_20190225_210000V.stat
ORANK
> > > >> > > > >
> > > >> > > > > ensemble_stat_Reventador_hr06_20190225_220000V.stat
ORANK
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > The ORANK line type is basically the "matched pairs"
for
> > > ensemble
> > > >> > > > > verification. Notice that the first 2 hours have the
ORANK
> > lines
> > > >> > along
> > > >> > > > with
> > > >> > > > > other statistic line types, such as RHIST. However,
the
> last 3
> > > >> hours
> > > >> > > only
> > > >> > > > > have the ORANK line type.
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > Looking at the output for the first 2 hours, I see
that
> you've
> > > >> > applied
> > > >> > > > the
> > > >> > > > > OBS_THRESH filtering threshold as ">=0.1".
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > Here's a link to the table which defines the contents
of the
> > > ORANK
> > > >> > line
> > > >> > > > > type:
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://dtcenter.github.io/MET/Users_Guide/ensemble-
stat.html?highlight=orank%20line%20type#id7
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > Note that the actual observation rank value is in
column
> #34.
> > > >> > Checking
> > > >> > > > the
> > > >> > > > > data in those last 3 files, you'll notice that the
> observation
> > > >> ranks
> > > >> > > are
> > > >> > > > > always listed as bad data:
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > cat
ensemble_stat_Reventador_hr04_20190225_200000V.stat
> > > >> > > > > ensemble_stat_Reventador_hr05_20190225_210000V.stat
> > > >> > > > > ensemble_stat_Reventador_hr06_20190225_220000V.stat
| grep
> > > ORANK
> > > >> |
> > > >> > awk
> > > >> > > > > '{print $34}' | sort -u
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > NA
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > So now the question is why? You're using the
"censor_thresh"
> > and
> > > >> > > > > "censor_val" options to reset the forecast values
that are
> > > "<0.1"
> > > >> to
> > > >> > > bad
> > > >> > > > > data. So that introduces a lot of bad data values.
And that
> > > >> results
> > > >> > in
> > > >> > > > the
> > > >> > > > > rank of the observation being computed as bad data.
This is
> > > likely
> > > >> > > > > controlled by how you have the "vld_thresh" or
> > "ens_vld_thresh"
> > > >> set
> > > >> > in
> > > >> > > > your
> > > >> > > > > Ensemble-Stat configuration file.
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > John
> > > >> > > > >
> > > >> > > > > On Thu, Aug 27, 2020 at 8:48 AM binyu.wang at noaa.gov
via RT
> <
> > > >> > > > > met_help at ucar.edu> wrote:
> > > >> > > > >
> > > >> > > > > >
> > > >> > > > > > <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=96453
> > > >
> > > >> > > > > >
> > > >> > > > > > Hello Tatiana,
> > > >> > > > > >
> > > >> > > > > > Sorry to ask more: as I said in previous mail I can
not
> find
> > > the
> > > >> > > other
> > > >> > > > 3
> > > >> > > > > > files from MET_Viewer because there is no RHIST
inside of
> > > those
> > > >> > > files,
> > > >> > > > if
> > > >> > > > > > this is because there is no matched pairs between
obs. and
> > > >> > forecast,
> > > >> > > > why
> > > >> > > > > I
> > > >> > > > > > can see them in the RELY plot? (see the attachment
> please).
> > > Are
> > > >> > RELY
> > > >> > > > > plots
> > > >> > > > > > created based on matching pairs?
> > > >> > > > > >
> > > >> > > > > > Thank you.
> > > >> > > > > > Binyu
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On Wed, Aug 26, 2020 at 10:47 PM Binyu Wang - NOAA
> > Affiliate <
> > > >> > > > > > binyu.wang at noaa.gov> wrote:
> > > >> > > > > >
> > > >> > > > > > > Hello,
> > > >> > > > > > >
> > > >> > > > > > > I think I know the answer now, but just to
confirm with
> > you:
> > > >> if
> > > >> > > there
> > > >> > > > > is
> > > >> > > > > > > no RHIST line type in the stat file, which
indicates
> there
> > > is
> > > >> no
> > > >> > > > > matching
> > > >> > > > > > > pair at all between obs. and forecast, is that
correct?
> > > >> > > > > > >
> > > >> > > > > > > Thank you.
> > > >> > > > > > >
> > > >> > > > > > > On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via
RT <
> > > >> > > > > met_help at ucar.edu>
> > > >> > > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > >> Binyu,
> > > >> > > > > > >> All 5 ensemble_stat files were successfully
loaded to
> the
> > > DB
> > > >> and
> > > >> > > the
> > > >> > > > > > data
> > > >> > > > > > >> is available for potting. Here is the example:
> > > >> 20200826_164848.
> > > >> > > > > > >> Your screenshot displays that you try to create
a
> > histogram
> > > >> > plot.
> > > >> > > > The
> > > >> > > > > > >> data for this plot comes from line_data_rhist
table.
> This
> > > >> table
> > > >> > > gets
> > > >> > > > > > filled
> > > >> > > > > > >> with 'RHIST' line type.
> > > >> > > > > > >>
'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
> > file
> > > >> > > contains
> > > >> > > > > only
> > > >> > > > > > >> 'ORANK' line type so non of it's records were
saved
> into
> > > >> > > > > line_data_rhist
> > > >> > > > > > >> table.
> > > >> > > > > > >> Currently line_data_rhist table contains data
for
> > > >> > fcst_valid_beg:
> > > >> > > > > > >> '2019-02-25 18:00:00' and '2019-02-25 19:00:00'
(
> > > fcst_lead:
> > > >> > 20000
> > > >> > > > and
> > > >> > > > > > >> 30000). The data came from
> > > >> > > > > > >>
ensemble_stat_Reventador_hr02_20190225_180000V.stat and
> > > >> > > > > > >>
ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > >> > > > > > >>
> > > >> > > > > > >> Could you check if other files have records with
RHIST
> > line
> > > >> > type.
> > > >> > > > > > >>
> > > >> > > > > > >> Tatiana
> > > >> > > > > > >>
> > > >> > > > > > >> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov
wrote:
> > > >> > > > > > >> > Tatiana,
> > > >> > > > > > >> >
> > > >> > > > > > >> > Yes that is weird. If you see the attached
file, you
> > can
> > > >> see
> > > >> > > > > > >> > there only two files were uploaded. I expect
to have
> > all
> > > >> 5.
> > > >> > > > > > >> > Binyu
> > > >> > > > > > >> >
> > > >> > > > > > >> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek
via RT
> > > >> > > > > > >> > <met_help at ucar.edu>
> > > >> > > > > > >> > wrote:
> > > >> > > > > > >> >
> > > >> > > > > > >> > > Binyu,
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > In your log file there is a record that
> > > >> > > > > > >> > >
> 'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
> > > was
> > > >> > > > > > >> > > successfully
> > > >> > > > > > >> > > loaded - all 53 lines.
> > > >> > > > > > >> > > I also can create a plot using the data from
this
> > file
> > > -
> > > >> > ORANK
> > > >> > > > > > >> > > statistics
> > > >> > > > > > >> > > with 40000 forecast lead: 20200826_151853.
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > Tatiana
> > > >> > > > > > >> > >
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > On Wed Aug 26 08:32:17 2020,
binyu.wang at noaa.gov
> > > wrote:
> > > >> > > > > > >> > > > Tatiana,
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > > > I didn't see any error from the log file.
I
> > attached
> > > >> the
> > > >> > log
> > > >> > > > > file
> > > >> > > > > > >> > > > and two
> > > >> > > > > > >> > > > of the stat files:
> > > >> > > > > > >> > > >
> ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > >> > > > > > >> > > is
> > > >> > > > > > >> > > > readable from METViewer, but
> > > >> > > > > > >> > > >
> > "ensemble_stat_Reventador_hr04_20190225_200000V.stat"
> > > >> not.
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > > > Thank you.
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana
Burek via
> > RT
> > > >> > > > > > >> > > > <met_help at ucar.edu>
> > > >> > > > > > >> > > > wrote:
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > > > > Hi Binyu,
> > > >> > > > > > >> > > > > I assume that your files are on WCOSS. I
don't
> > have
> > > >> an
> > > >> > > > access
> > > >> > > > > to
> > > >> > > > > > >> > > > > the
> > > >> > > > > > >> > > > > server and can't see the files.
> > > >> > > > > > >> > > > > Do you have logging messages from the
load? Any
> > > >> errors?
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > I see that only 5 ensemble_stat files
were
> loaded
> > > to
> > > >> > this
> > > >> > > > > > >> > > > > database:
> > > >> > > > > > >> > > > > |
> > > >> ensemble_stat_Reventador_hr02_20190225_180000V.stat |
> > > >> > > > > > >> > > > > |
> > > >> ensemble_stat_Reventador_hr03_20190225_190000V.stat |
> > > >> > > > > > >> > > > > |
> > > >> ensemble_stat_Reventador_hr04_20190225_200000V.stat |
> > > >> > > > > > >> > > > > |
> > > >> ensemble_stat_Reventador_hr05_20190225_210000V.stat |
> > > >> > > > > > >> > > > > |
> > > ensemble_stat_Reventador_hr06_20190225_220000V.stat
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > Tatiana
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > On Tue Aug 25 20:02:54 2020,
> binyu.wang at noaa.gov
> > > >> wrote:
> > > >> > > > > > >> > > > > > Hello,
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > > > I created 5 ensemble_stat* files (from
hr18-
> > > hr22)
> > > >> > > under
> > > >> > > > > the
> > > >> > > > > > >> > > directory
> > > >> > > > > > >> > > > > > below, but only the first two (hr18
and
> hr19)
> > > >> could
> > > >> > be
> > > >> > > > read
> > > >> > > > > > >> > > > > > from
> > > >> > > > > > >> > > > > > METVIewer. Those files are under
database
> > > >> > > > > > >> > > "mv_g2g_met_verf_Reventador"
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > >
> > > >> > > > > > >>
> > > >> > > > >
> > > >> > >
> > > >>
> >
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > > > What is the problem? Thank you.
> > > >> > > > > > >> > > > > > Binyu
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > >
> > > >> > > > > > >> > >
> > > >> > > > > > >> > >
> > > >> > > > > > >> > >
> > > >> > > > > > >>
> > > >> > > > > > >>
> > > >> > > > > > >>
> > > >> > > > > > >>
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >>
> > > >>
> > >
> > >
> >
> >
>
>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: Tatiana Burek
Time: Fri Aug 28 12:03:11 2020
Hello Binyu
I created a new database 'mv_g2g_met_verf_Raikoke_all_data' and copied
all data from 'mv_g2g_met_verf_Raikoke' to it. Please verify that all
data is correct and let me know. I will delete
'mv_g2g_met_verf_Raikoke' after that.
Tatiana
On Thu Aug 27 16:23:18 2020, binyu.wang at noaa.gov wrote:
> Hello Tatiana,
>
> Would you please help to rename the database
"mv_g2g_met_verf_Raikoke"
> to
> "mv_g2g_met_verf_Raikoke.all.data"?
> Thank you.
>
>
> On Thu, Aug 27, 2020 at 2:16 PM Tatiana Burek via RT
> <met_help at ucar.edu>
> wrote:
>
> > There is no easy way to rename the database.
> > I can create a new database with a new name, dump the data from
the
> > old
> > database, load it to the new one and finally remove the old
database.
> >
> >
> > Tatiana
> >
> > On Thu Aug 27 12:04:00 2020, binyu.wang at noaa.gov wrote:
> > > Another quick question: Can we change the database name without
> > > having to
> > > drop the database and reload all of the data again?
> > >
> > > Binyu
> > >
> > > On Thu, Aug 27, 2020 at 2:03 PM Binyu Wang - NOAA Affiliate <
> > > binyu.wang at noaa.gov> wrote:
> > >
> > > > Tatiana,
> > > >
> > > > "PCT" is generated later using the ensemble*stat files which I
> > > > sent to
> > you
> > > > earlier. So you didn't see it in those two files. I didn't
> > > > understand:
> > if
> > > > the 3 files didn't have "RHIST" because there was no matching
> > > > pair
> > > > during that time period, why could they be used for RELY plot?
> > > >
> > > > Binyu
> > > >
> > > > On Thu, Aug 27, 2020 at 1:43 PM Tatiana Burek via RT <
> > met_help at ucar.edu>
> > > > wrote:
> > > >
> > > >> Binyu
> > > >>
> > > >> Reliability plot uses data from line_data_pct table ( "PCT"
line
> > > >> type
> > ).
> > > >> Both stat files that you sent me don't contain PCT line type
and
> > variable
> > > >> 'VAFTD_ENS_FREQ_ge0.1'. Did you have data in this database
> > > >> before the
> > > >> latest load?
> > > >>
> > > >> Tatiana
> > > >>
> > > >> On Thu Aug 27 08:48:35 2020, binyu.wang at noaa.gov wrote:
> > > >> > Hello Tatiana,
> > > >> >
> > > >> > Sorry to ask more: as I said in previous mail I can not
find
> > > >> > the
> > other
> > > >> > 3
> > > >> > files from MET_Viewer because there is no RHIST inside of
> > > >> > those
> > files,
> > > >> > if
> > > >> > this is because there is no matched pairs between obs. and
> > > >> > forecast,
> > > >> > why I
> > > >> > can see them in the RELY plot? (see the attachment please).
> > > >> > Are RELY
> > > >> > plots
> > > >> > created based on matching pairs?
> > > >> >
> > > >> > Thank you.
> > > >> > Binyu
> > > >> >
> > > >> >
> > > >> > On Wed, Aug 26, 2020 at 10:47 PM Binyu Wang - NOAA
Affiliate <
> > > >> > binyu.wang at noaa.gov> wrote:
> > > >> >
> > > >> > > Hello,
> > > >> > >
> > > >> > > I think I know the answer now, but just to confirm with
you:
> > > >> > > if
> > there
> > > >> > > is
> > > >> > > no RHIST line type in the stat file, which indicates
there
> > > >> > > is no
> > > >> > > matching
> > > >> > > pair at all between obs. and forecast, is that correct?
> > > >> > >
> > > >> > > Thank you.
> > > >> > >
> > > >> > > On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via RT
> > > >> > > <met_help at ucar.edu>
> > > >> > > wrote:
> > > >> > >
> > > >> > >> Binyu,
> > > >> > >> All 5 ensemble_stat files were successfully loaded to
the
> > > >> > >> DB and
> > the
> > > >> > >> data
> > > >> > >> is available for potting. Here is the example:
> > > >> > >> 20200826_164848.
> > > >> > >> Your screenshot displays that you try to create a
histogram
> > > >> > >> plot.
> > > >> > >> The
> > > >> > >> data for this plot comes from line_data_rhist table.
This
> > > >> > >> table
> > gets
> > > >> > >> filled
> > > >> > >> with 'RHIST' line type.
> > > >> > >> 'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
file
> > contains
> > > >> > >> only
> > > >> > >> 'ORANK' line type so non of it's records were saved into
> > > >> > >> line_data_rhist
> > > >> > >> table.
> > > >> > >> Currently line_data_rhist table contains data for
> > > >> > >> fcst_valid_beg:
> > > >> > >> '2019-02-25 18:00:00' and '2019-02-25 19:00:00' (
> > > >> > >> fcst_lead:
> > 20000
> > > >> > >> and
> > > >> > >> 30000). The data came from
> > > >> > >> ensemble_stat_Reventador_hr02_20190225_180000V.stat and
> > > >> > >> ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > >> > >>
> > > >> > >> Could you check if other files have records with RHIST
line
> > > >> > >> type.
> > > >> > >>
> > > >> > >> Tatiana
> > > >> > >>
> > > >> > >> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov wrote:
> > > >> > >> > Tatiana,
> > > >> > >> >
> > > >> > >> > Yes that is weird. If you see the attached file, you
can
> > > >> > >> > see
> > > >> > >> > there only two files were uploaded. I expect to have
all
> > > >> > >> > 5.
> > > >> > >> > Binyu
> > > >> > >> >
> > > >> > >> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek via RT
> > > >> > >> > <met_help at ucar.edu>
> > > >> > >> > wrote:
> > > >> > >> >
> > > >> > >> > > Binyu,
> > > >> > >> > >
> > > >> > >> > > In your log file there is a record that
> > > >> > >> > >
'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
> > > >> > >> > > was
> > > >> > >> > > successfully
> > > >> > >> > > loaded - all 53 lines.
> > > >> > >> > > I also can create a plot using the data from this
file
> > > >> > >> > > -
> > ORANK
> > > >> > >> > > statistics
> > > >> > >> > > with 40000 forecast lead: 20200826_151853.
> > > >> > >> > >
> > > >> > >> > > Tatiana
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> > > On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov
wrote:
> > > >> > >> > > > Tatiana,
> > > >> > >> > > >
> > > >> > >> > > > I didn't see any error from the log file. I
attached
> > > >> > >> > > > the
> > log
> > > >> > >> > > > file
> > > >> > >> > > > and two
> > > >> > >> > > > of the stat files:
> > > >> > >> > > >
ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > >> > >> > > is
> > > >> > >> > > > readable from METViewer, but
> > > >> > >> > > >
"ensemble_stat_Reventador_hr04_20190225_200000V.stat"
> > > >> > >> > > > not.
> > > >> > >> > > >
> > > >> > >> > > > Thank you.
> > > >> > >> > > >
> > > >> > >> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek via
RT
> > > >> > >> > > > <met_help at ucar.edu>
> > > >> > >> > > > wrote:
> > > >> > >> > > >
> > > >> > >> > > > > Hi Binyu,
> > > >> > >> > > > > I assume that your files are on WCOSS. I don't
have
> > > >> > >> > > > > an
> > > >> > >> > > > > access to
> > > >> > >> > > > > the
> > > >> > >> > > > > server and can't see the files.
> > > >> > >> > > > > Do you have logging messages from the load? Any
> > > >> > >> > > > > errors?
> > > >> > >> > > > >
> > > >> > >> > > > > I see that only 5 ensemble_stat files were
loaded
> > > >> > >> > > > > to this
> > > >> > >> > > > > database:
> > > >> > >> > > > > |
> > > >> > >> > > > >
ensemble_stat_Reventador_hr02_20190225_180000V.stat
> > > >> > >> > > > > |
> > > >> > >> > > > > |
> > > >> > >> > > > >
ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > >> > >> > > > > |
> > > >> > >> > > > > |
> > > >> > >> > > > >
ensemble_stat_Reventador_hr04_20190225_200000V.stat
> > > >> > >> > > > > |
> > > >> > >> > > > > |
> > > >> > >> > > > >
ensemble_stat_Reventador_hr05_20190225_210000V.stat
> > > >> > >> > > > > |
> > > >> > >> > > > > |
> > > >> > >> > > > >
ensemble_stat_Reventador_hr06_20190225_220000V.stat
> > > >> > >> > > > >
> > > >> > >> > > > > Tatiana
> > > >> > >> > > > >
> > > >> > >> > > > > On Tue Aug 25 20:02:54 2020, binyu.wang at noaa.gov
> > > >> > >> > > > > wrote:
> > > >> > >> > > > > > Hello,
> > > >> > >> > > > > >
> > > >> > >> > > > > > I created 5 ensemble_stat* files (from hr18-
> > > >> > >> > > > > > hr22)
> > under
> > > >> > >> > > > > > the
> > > >> > >> > > directory
> > > >> > >> > > > > > below, but only the first two (hr18 and hr19)
> > > >> > >> > > > > > could be
> > > >> > >> > > > > > read
> > > >> > >> > > > > > from
> > > >> > >> > > > > > METVIewer. Those files are under database
> > > >> > >> > > "mv_g2g_met_verf_Reventador"
> > > >> > >> > > > > >
> > > >> > >> > > > > >
> > > >> > >> > >
> > > >> > >>
> > > >>
> >
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > > >> > >> > > > > >
> > > >> > >> > > > > > What is the problem? Thank you.
> > > >> > >> > > > > > Binyu
> > > >> > >> > > > >
> > > >> > >> > > > >
> > > >> > >> > > > >
> > > >> > >> > > > >
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >>
> > > >>
> > > >>
> > > >>
> >
> >
> >
> >
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: binyu.wang at noaa.gov
Time: Fri Aug 28 13:05:57 2020
Yes, it looks correct. Thank you.
Binyu
On Fri, Aug 28, 2020 at 2:03 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> Hello Binyu
> I created a new database 'mv_g2g_met_verf_Raikoke_all_data' and
copied all
> data from 'mv_g2g_met_verf_Raikoke' to it. Please verify that all
data is
> correct and let me know. I will delete 'mv_g2g_met_verf_Raikoke'
after that.
>
> Tatiana
>
> On Thu Aug 27 16:23:18 2020, binyu.wang at noaa.gov wrote:
> > Hello Tatiana,
> >
> > Would you please help to rename the database
"mv_g2g_met_verf_Raikoke"
> > to
> > "mv_g2g_met_verf_Raikoke.all.data"?
> > Thank you.
> >
> >
> > On Thu, Aug 27, 2020 at 2:16 PM Tatiana Burek via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > There is no easy way to rename the database.
> > > I can create a new database with a new name, dump the data from
the
> > > old
> > > database, load it to the new one and finally remove the old
database.
> > >
> > >
> > > Tatiana
> > >
> > > On Thu Aug 27 12:04:00 2020, binyu.wang at noaa.gov wrote:
> > > > Another quick question: Can we change the database name
without
> > > > having to
> > > > drop the database and reload all of the data again?
> > > >
> > > > Binyu
> > > >
> > > > On Thu, Aug 27, 2020 at 2:03 PM Binyu Wang - NOAA Affiliate <
> > > > binyu.wang at noaa.gov> wrote:
> > > >
> > > > > Tatiana,
> > > > >
> > > > > "PCT" is generated later using the ensemble*stat files which
I
> > > > > sent to
> > > you
> > > > > earlier. So you didn't see it in those two files. I didn't
> > > > > understand:
> > > if
> > > > > the 3 files didn't have "RHIST" because there was no
matching
> > > > > pair
> > > > > during that time period, why could they be used for RELY
plot?
> > > > >
> > > > > Binyu
> > > > >
> > > > > On Thu, Aug 27, 2020 at 1:43 PM Tatiana Burek via RT <
> > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > >> Binyu
> > > > >>
> > > > >> Reliability plot uses data from line_data_pct table ( "PCT"
line
> > > > >> type
> > > ).
> > > > >> Both stat files that you sent me don't contain PCT line
type and
> > > variable
> > > > >> 'VAFTD_ENS_FREQ_ge0.1'. Did you have data in this database
> > > > >> before the
> > > > >> latest load?
> > > > >>
> > > > >> Tatiana
> > > > >>
> > > > >> On Thu Aug 27 08:48:35 2020, binyu.wang at noaa.gov wrote:
> > > > >> > Hello Tatiana,
> > > > >> >
> > > > >> > Sorry to ask more: as I said in previous mail I can not
find
> > > > >> > the
> > > other
> > > > >> > 3
> > > > >> > files from MET_Viewer because there is no RHIST inside of
> > > > >> > those
> > > files,
> > > > >> > if
> > > > >> > this is because there is no matched pairs between obs.
and
> > > > >> > forecast,
> > > > >> > why I
> > > > >> > can see them in the RELY plot? (see the attachment
please).
> > > > >> > Are RELY
> > > > >> > plots
> > > > >> > created based on matching pairs?
> > > > >> >
> > > > >> > Thank you.
> > > > >> > Binyu
> > > > >> >
> > > > >> >
> > > > >> > On Wed, Aug 26, 2020 at 10:47 PM Binyu Wang - NOAA
Affiliate <
> > > > >> > binyu.wang at noaa.gov> wrote:
> > > > >> >
> > > > >> > > Hello,
> > > > >> > >
> > > > >> > > I think I know the answer now, but just to confirm with
you:
> > > > >> > > if
> > > there
> > > > >> > > is
> > > > >> > > no RHIST line type in the stat file, which indicates
there
> > > > >> > > is no
> > > > >> > > matching
> > > > >> > > pair at all between obs. and forecast, is that correct?
> > > > >> > >
> > > > >> > > Thank you.
> > > > >> > >
> > > > >> > > On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via RT
> > > > >> > > <met_help at ucar.edu>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > >> Binyu,
> > > > >> > >> All 5 ensemble_stat files were successfully loaded to
the
> > > > >> > >> DB and
> > > the
> > > > >> > >> data
> > > > >> > >> is available for potting. Here is the example:
> > > > >> > >> 20200826_164848.
> > > > >> > >> Your screenshot displays that you try to create a
histogram
> > > > >> > >> plot.
> > > > >> > >> The
> > > > >> > >> data for this plot comes from line_data_rhist table.
This
> > > > >> > >> table
> > > gets
> > > > >> > >> filled
> > > > >> > >> with 'RHIST' line type.
> > > > >> > >> 'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
file
> > > contains
> > > > >> > >> only
> > > > >> > >> 'ORANK' line type so non of it's records were saved
into
> > > > >> > >> line_data_rhist
> > > > >> > >> table.
> > > > >> > >> Currently line_data_rhist table contains data for
> > > > >> > >> fcst_valid_beg:
> > > > >> > >> '2019-02-25 18:00:00' and '2019-02-25 19:00:00' (
> > > > >> > >> fcst_lead:
> > > 20000
> > > > >> > >> and
> > > > >> > >> 30000). The data came from
> > > > >> > >> ensemble_stat_Reventador_hr02_20190225_180000V.stat
and
> > > > >> > >> ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > > >> > >>
> > > > >> > >> Could you check if other files have records with RHIST
line
> > > > >> > >> type.
> > > > >> > >>
> > > > >> > >> Tatiana
> > > > >> > >>
> > > > >> > >> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov wrote:
> > > > >> > >> > Tatiana,
> > > > >> > >> >
> > > > >> > >> > Yes that is weird. If you see the attached file, you
can
> > > > >> > >> > see
> > > > >> > >> > there only two files were uploaded. I expect to
have all
> > > > >> > >> > 5.
> > > > >> > >> > Binyu
> > > > >> > >> >
> > > > >> > >> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek via
RT
> > > > >> > >> > <met_help at ucar.edu>
> > > > >> > >> > wrote:
> > > > >> > >> >
> > > > >> > >> > > Binyu,
> > > > >> > >> > >
> > > > >> > >> > > In your log file there is a record that
> > > > >> > >> > >
'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
> > > > >> > >> > > was
> > > > >> > >> > > successfully
> > > > >> > >> > > loaded - all 53 lines.
> > > > >> > >> > > I also can create a plot using the data from this
file
> > > > >> > >> > > -
> > > ORANK
> > > > >> > >> > > statistics
> > > > >> > >> > > with 40000 forecast lead: 20200826_151853.
> > > > >> > >> > >
> > > > >> > >> > > Tatiana
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> > > On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov
wrote:
> > > > >> > >> > > > Tatiana,
> > > > >> > >> > > >
> > > > >> > >> > > > I didn't see any error from the log file. I
attached
> > > > >> > >> > > > the
> > > log
> > > > >> > >> > > > file
> > > > >> > >> > > > and two
> > > > >> > >> > > > of the stat files:
> > > > >> > >> > > >
ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > > >> > >> > > is
> > > > >> > >> > > > readable from METViewer, but
> > > > >> > >> > > >
"ensemble_stat_Reventador_hr04_20190225_200000V.stat"
> > > > >> > >> > > > not.
> > > > >> > >> > > >
> > > > >> > >> > > > Thank you.
> > > > >> > >> > > >
> > > > >> > >> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek
via RT
> > > > >> > >> > > > <met_help at ucar.edu>
> > > > >> > >> > > > wrote:
> > > > >> > >> > > >
> > > > >> > >> > > > > Hi Binyu,
> > > > >> > >> > > > > I assume that your files are on WCOSS. I don't
have
> > > > >> > >> > > > > an
> > > > >> > >> > > > > access to
> > > > >> > >> > > > > the
> > > > >> > >> > > > > server and can't see the files.
> > > > >> > >> > > > > Do you have logging messages from the load?
Any
> > > > >> > >> > > > > errors?
> > > > >> > >> > > > >
> > > > >> > >> > > > > I see that only 5 ensemble_stat files were
loaded
> > > > >> > >> > > > > to this
> > > > >> > >> > > > > database:
> > > > >> > >> > > > > |
> > > > >> > >> > > > >
ensemble_stat_Reventador_hr02_20190225_180000V.stat
> > > > >> > >> > > > > |
> > > > >> > >> > > > > |
> > > > >> > >> > > > >
ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > > >> > >> > > > > |
> > > > >> > >> > > > > |
> > > > >> > >> > > > >
ensemble_stat_Reventador_hr04_20190225_200000V.stat
> > > > >> > >> > > > > |
> > > > >> > >> > > > > |
> > > > >> > >> > > > >
ensemble_stat_Reventador_hr05_20190225_210000V.stat
> > > > >> > >> > > > > |
> > > > >> > >> > > > > |
> > > > >> > >> > > > >
ensemble_stat_Reventador_hr06_20190225_220000V.stat
> > > > >> > >> > > > >
> > > > >> > >> > > > > Tatiana
> > > > >> > >> > > > >
> > > > >> > >> > > > > On Tue Aug 25 20:02:54 2020,
binyu.wang at noaa.gov
> > > > >> > >> > > > > wrote:
> > > > >> > >> > > > > > Hello,
> > > > >> > >> > > > > >
> > > > >> > >> > > > > > I created 5 ensemble_stat* files (from hr18-
> > > > >> > >> > > > > > hr22)
> > > under
> > > > >> > >> > > > > > the
> > > > >> > >> > > directory
> > > > >> > >> > > > > > below, but only the first two (hr18 and
hr19)
> > > > >> > >> > > > > > could be
> > > > >> > >> > > > > > read
> > > > >> > >> > > > > > from
> > > > >> > >> > > > > > METVIewer. Those files are under database
> > > > >> > >> > > "mv_g2g_met_verf_Reventador"
> > > > >> > >> > > > > >
> > > > >> > >> > > > > >
> > > > >> > >> > >
> > > > >> > >>
> > > > >>
> > >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > > > >> > >> > > > > >
> > > > >> > >> > > > > > What is the problem? Thank you.
> > > > >> > >> > > > > > Binyu
> > > > >> > >> > > > >
> > > > >> > >> > > > >
> > > > >> > >> > > > >
> > > > >> > >> > > > >
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > >
> > >
> > >
> > >
>
>
>
>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: binyu.wang at noaa.gov
Time: Sun Aug 30 20:10:14 2020
Hello,
Tatiana, if you hadn't deleted 'mv_g2g_met_verf_Raikoke', please just
keep
it. Please confirm. Thank you.
Binyu
On Fri, Aug 28, 2020 at 2:03 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> Hello Binyu
> I created a new database 'mv_g2g_met_verf_Raikoke_all_data' and
copied all
> data from 'mv_g2g_met_verf_Raikoke' to it. Please verify that all
data is
> correct and let me know. I will delete 'mv_g2g_met_verf_Raikoke'
after that.
>
> Tatiana
>
> On Thu Aug 27 16:23:18 2020, binyu.wang at noaa.gov wrote:
> > Hello Tatiana,
> >
> > Would you please help to rename the database
"mv_g2g_met_verf_Raikoke"
> > to
> > "mv_g2g_met_verf_Raikoke.all.data"?
> > Thank you.
> >
> >
> > On Thu, Aug 27, 2020 at 2:16 PM Tatiana Burek via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > There is no easy way to rename the database.
> > > I can create a new database with a new name, dump the data from
the
> > > old
> > > database, load it to the new one and finally remove the old
database.
> > >
> > >
> > > Tatiana
> > >
> > > On Thu Aug 27 12:04:00 2020, binyu.wang at noaa.gov wrote:
> > > > Another quick question: Can we change the database name
without
> > > > having to
> > > > drop the database and reload all of the data again?
> > > >
> > > > Binyu
> > > >
> > > > On Thu, Aug 27, 2020 at 2:03 PM Binyu Wang - NOAA Affiliate <
> > > > binyu.wang at noaa.gov> wrote:
> > > >
> > > > > Tatiana,
> > > > >
> > > > > "PCT" is generated later using the ensemble*stat files which
I
> > > > > sent to
> > > you
> > > > > earlier. So you didn't see it in those two files. I didn't
> > > > > understand:
> > > if
> > > > > the 3 files didn't have "RHIST" because there was no
matching
> > > > > pair
> > > > > during that time period, why could they be used for RELY
plot?
> > > > >
> > > > > Binyu
> > > > >
> > > > > On Thu, Aug 27, 2020 at 1:43 PM Tatiana Burek via RT <
> > > met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > >> Binyu
> > > > >>
> > > > >> Reliability plot uses data from line_data_pct table ( "PCT"
line
> > > > >> type
> > > ).
> > > > >> Both stat files that you sent me don't contain PCT line
type and
> > > variable
> > > > >> 'VAFTD_ENS_FREQ_ge0.1'. Did you have data in this database
> > > > >> before the
> > > > >> latest load?
> > > > >>
> > > > >> Tatiana
> > > > >>
> > > > >> On Thu Aug 27 08:48:35 2020, binyu.wang at noaa.gov wrote:
> > > > >> > Hello Tatiana,
> > > > >> >
> > > > >> > Sorry to ask more: as I said in previous mail I can not
find
> > > > >> > the
> > > other
> > > > >> > 3
> > > > >> > files from MET_Viewer because there is no RHIST inside of
> > > > >> > those
> > > files,
> > > > >> > if
> > > > >> > this is because there is no matched pairs between obs.
and
> > > > >> > forecast,
> > > > >> > why I
> > > > >> > can see them in the RELY plot? (see the attachment
please).
> > > > >> > Are RELY
> > > > >> > plots
> > > > >> > created based on matching pairs?
> > > > >> >
> > > > >> > Thank you.
> > > > >> > Binyu
> > > > >> >
> > > > >> >
> > > > >> > On Wed, Aug 26, 2020 at 10:47 PM Binyu Wang - NOAA
Affiliate <
> > > > >> > binyu.wang at noaa.gov> wrote:
> > > > >> >
> > > > >> > > Hello,
> > > > >> > >
> > > > >> > > I think I know the answer now, but just to confirm with
you:
> > > > >> > > if
> > > there
> > > > >> > > is
> > > > >> > > no RHIST line type in the stat file, which indicates
there
> > > > >> > > is no
> > > > >> > > matching
> > > > >> > > pair at all between obs. and forecast, is that correct?
> > > > >> > >
> > > > >> > > Thank you.
> > > > >> > >
> > > > >> > > On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via RT
> > > > >> > > <met_help at ucar.edu>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > >> Binyu,
> > > > >> > >> All 5 ensemble_stat files were successfully loaded to
the
> > > > >> > >> DB and
> > > the
> > > > >> > >> data
> > > > >> > >> is available for potting. Here is the example:
> > > > >> > >> 20200826_164848.
> > > > >> > >> Your screenshot displays that you try to create a
histogram
> > > > >> > >> plot.
> > > > >> > >> The
> > > > >> > >> data for this plot comes from line_data_rhist table.
This
> > > > >> > >> table
> > > gets
> > > > >> > >> filled
> > > > >> > >> with 'RHIST' line type.
> > > > >> > >> 'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
file
> > > contains
> > > > >> > >> only
> > > > >> > >> 'ORANK' line type so non of it's records were saved
into
> > > > >> > >> line_data_rhist
> > > > >> > >> table.
> > > > >> > >> Currently line_data_rhist table contains data for
> > > > >> > >> fcst_valid_beg:
> > > > >> > >> '2019-02-25 18:00:00' and '2019-02-25 19:00:00' (
> > > > >> > >> fcst_lead:
> > > 20000
> > > > >> > >> and
> > > > >> > >> 30000). The data came from
> > > > >> > >> ensemble_stat_Reventador_hr02_20190225_180000V.stat
and
> > > > >> > >> ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > > >> > >>
> > > > >> > >> Could you check if other files have records with RHIST
line
> > > > >> > >> type.
> > > > >> > >>
> > > > >> > >> Tatiana
> > > > >> > >>
> > > > >> > >> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov wrote:
> > > > >> > >> > Tatiana,
> > > > >> > >> >
> > > > >> > >> > Yes that is weird. If you see the attached file, you
can
> > > > >> > >> > see
> > > > >> > >> > there only two files were uploaded. I expect to
have all
> > > > >> > >> > 5.
> > > > >> > >> > Binyu
> > > > >> > >> >
> > > > >> > >> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek via
RT
> > > > >> > >> > <met_help at ucar.edu>
> > > > >> > >> > wrote:
> > > > >> > >> >
> > > > >> > >> > > Binyu,
> > > > >> > >> > >
> > > > >> > >> > > In your log file there is a record that
> > > > >> > >> > >
'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
> > > > >> > >> > > was
> > > > >> > >> > > successfully
> > > > >> > >> > > loaded - all 53 lines.
> > > > >> > >> > > I also can create a plot using the data from this
file
> > > > >> > >> > > -
> > > ORANK
> > > > >> > >> > > statistics
> > > > >> > >> > > with 40000 forecast lead: 20200826_151853.
> > > > >> > >> > >
> > > > >> > >> > > Tatiana
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> > > On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov
wrote:
> > > > >> > >> > > > Tatiana,
> > > > >> > >> > > >
> > > > >> > >> > > > I didn't see any error from the log file. I
attached
> > > > >> > >> > > > the
> > > log
> > > > >> > >> > > > file
> > > > >> > >> > > > and two
> > > > >> > >> > > > of the stat files:
> > > > >> > >> > > >
ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > > >> > >> > > is
> > > > >> > >> > > > readable from METViewer, but
> > > > >> > >> > > >
"ensemble_stat_Reventador_hr04_20190225_200000V.stat"
> > > > >> > >> > > > not.
> > > > >> > >> > > >
> > > > >> > >> > > > Thank you.
> > > > >> > >> > > >
> > > > >> > >> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek
via RT
> > > > >> > >> > > > <met_help at ucar.edu>
> > > > >> > >> > > > wrote:
> > > > >> > >> > > >
> > > > >> > >> > > > > Hi Binyu,
> > > > >> > >> > > > > I assume that your files are on WCOSS. I don't
have
> > > > >> > >> > > > > an
> > > > >> > >> > > > > access to
> > > > >> > >> > > > > the
> > > > >> > >> > > > > server and can't see the files.
> > > > >> > >> > > > > Do you have logging messages from the load?
Any
> > > > >> > >> > > > > errors?
> > > > >> > >> > > > >
> > > > >> > >> > > > > I see that only 5 ensemble_stat files were
loaded
> > > > >> > >> > > > > to this
> > > > >> > >> > > > > database:
> > > > >> > >> > > > > |
> > > > >> > >> > > > >
ensemble_stat_Reventador_hr02_20190225_180000V.stat
> > > > >> > >> > > > > |
> > > > >> > >> > > > > |
> > > > >> > >> > > > >
ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > > >> > >> > > > > |
> > > > >> > >> > > > > |
> > > > >> > >> > > > >
ensemble_stat_Reventador_hr04_20190225_200000V.stat
> > > > >> > >> > > > > |
> > > > >> > >> > > > > |
> > > > >> > >> > > > >
ensemble_stat_Reventador_hr05_20190225_210000V.stat
> > > > >> > >> > > > > |
> > > > >> > >> > > > > |
> > > > >> > >> > > > >
ensemble_stat_Reventador_hr06_20190225_220000V.stat
> > > > >> > >> > > > >
> > > > >> > >> > > > > Tatiana
> > > > >> > >> > > > >
> > > > >> > >> > > > > On Tue Aug 25 20:02:54 2020,
binyu.wang at noaa.gov
> > > > >> > >> > > > > wrote:
> > > > >> > >> > > > > > Hello,
> > > > >> > >> > > > > >
> > > > >> > >> > > > > > I created 5 ensemble_stat* files (from hr18-
> > > > >> > >> > > > > > hr22)
> > > under
> > > > >> > >> > > > > > the
> > > > >> > >> > > directory
> > > > >> > >> > > > > > below, but only the first two (hr18 and
hr19)
> > > > >> > >> > > > > > could be
> > > > >> > >> > > > > > read
> > > > >> > >> > > > > > from
> > > > >> > >> > > > > > METVIewer. Those files are under database
> > > > >> > >> > > "mv_g2g_met_verf_Reventador"
> > > > >> > >> > > > > >
> > > > >> > >> > > > > >
> > > > >> > >> > >
> > > > >> > >>
> > > > >>
> > >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > > > >> > >> > > > > >
> > > > >> > >> > > > > > What is the problem? Thank you.
> > > > >> > >> > > > > > Binyu
> > > > >> > >> > > > >
> > > > >> > >> > > > >
> > > > >> > >> > > > >
> > > > >> > >> > > > >
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > >
> > >
> > >
> > >
>
>
>
>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: Tatiana Burek
Time: Mon Aug 31 07:44:27 2020
Binyu
mv_g2g_met_verf_Raikoke database was deleted on Friday.
Tatiana
On Sun Aug 30 20:10:14 2020, binyu.wang at noaa.gov wrote:
> Hello,
> Tatiana, if you hadn't deleted 'mv_g2g_met_verf_Raikoke', please
just
> keep
> it. Please confirm. Thank you.
> Binyu
>
> On Fri, Aug 28, 2020 at 2:03 PM Tatiana Burek via RT
> <met_help at ucar.edu>
> wrote:
>
> > Hello Binyu
> > I created a new database 'mv_g2g_met_verf_Raikoke_all_data' and
> > copied all
> > data from 'mv_g2g_met_verf_Raikoke' to it. Please verify that all
> > data is
> > correct and let me know. I will delete 'mv_g2g_met_verf_Raikoke'
> > after that.
> >
> > Tatiana
> >
> > On Thu Aug 27 16:23:18 2020, binyu.wang at noaa.gov wrote:
> > > Hello Tatiana,
> > >
> > > Would you please help to rename the database
> > > "mv_g2g_met_verf_Raikoke"
> > > to
> > > "mv_g2g_met_verf_Raikoke.all.data"?
> > > Thank you.
> > >
> > >
> > > On Thu, Aug 27, 2020 at 2:16 PM Tatiana Burek via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > > > There is no easy way to rename the database.
> > > > I can create a new database with a new name, dump the data
from
> > > > the
> > > > old
> > > > database, load it to the new one and finally remove the old
> > > > database.
> > > >
> > > >
> > > > Tatiana
> > > >
> > > > On Thu Aug 27 12:04:00 2020, binyu.wang at noaa.gov wrote:
> > > > > Another quick question: Can we change the database name
without
> > > > > having to
> > > > > drop the database and reload all of the data again?
> > > > >
> > > > > Binyu
> > > > >
> > > > > On Thu, Aug 27, 2020 at 2:03 PM Binyu Wang - NOAA Affiliate
<
> > > > > binyu.wang at noaa.gov> wrote:
> > > > >
> > > > > > Tatiana,
> > > > > >
> > > > > > "PCT" is generated later using the ensemble*stat files
which
> > > > > > I
> > > > > > sent to
> > > > you
> > > > > > earlier. So you didn't see it in those two files. I didn't
> > > > > > understand:
> > > > if
> > > > > > the 3 files didn't have "RHIST" because there was no
> > > > > > matching
> > > > > > pair
> > > > > > during that time period, why could they be used for RELY
> > > > > > plot?
> > > > > >
> > > > > > Binyu
> > > > > >
> > > > > > On Thu, Aug 27, 2020 at 1:43 PM Tatiana Burek via RT <
> > > > met_help at ucar.edu>
> > > > > > wrote:
> > > > > >
> > > > > >> Binyu
> > > > > >>
> > > > > >> Reliability plot uses data from line_data_pct table (
"PCT"
> > > > > >> line
> > > > > >> type
> > > > ).
> > > > > >> Both stat files that you sent me don't contain PCT line
type
> > > > > >> and
> > > > variable
> > > > > >> 'VAFTD_ENS_FREQ_ge0.1'. Did you have data in this
database
> > > > > >> before the
> > > > > >> latest load?
> > > > > >>
> > > > > >> Tatiana
> > > > > >>
> > > > > >> On Thu Aug 27 08:48:35 2020, binyu.wang at noaa.gov wrote:
> > > > > >> > Hello Tatiana,
> > > > > >> >
> > > > > >> > Sorry to ask more: as I said in previous mail I can not
> > > > > >> > find
> > > > > >> > the
> > > > other
> > > > > >> > 3
> > > > > >> > files from MET_Viewer because there is no RHIST inside
of
> > > > > >> > those
> > > > files,
> > > > > >> > if
> > > > > >> > this is because there is no matched pairs between obs.
and
> > > > > >> > forecast,
> > > > > >> > why I
> > > > > >> > can see them in the RELY plot? (see the attachment
> > > > > >> > please).
> > > > > >> > Are RELY
> > > > > >> > plots
> > > > > >> > created based on matching pairs?
> > > > > >> >
> > > > > >> > Thank you.
> > > > > >> > Binyu
> > > > > >> >
> > > > > >> >
> > > > > >> > On Wed, Aug 26, 2020 at 10:47 PM Binyu Wang - NOAA
> > > > > >> > Affiliate <
> > > > > >> > binyu.wang at noaa.gov> wrote:
> > > > > >> >
> > > > > >> > > Hello,
> > > > > >> > >
> > > > > >> > > I think I know the answer now, but just to confirm
with
> > > > > >> > > you:
> > > > > >> > > if
> > > > there
> > > > > >> > > is
> > > > > >> > > no RHIST line type in the stat file, which indicates
> > > > > >> > > there
> > > > > >> > > is no
> > > > > >> > > matching
> > > > > >> > > pair at all between obs. and forecast, is that
correct?
> > > > > >> > >
> > > > > >> > > Thank you.
> > > > > >> > >
> > > > > >> > > On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via RT
> > > > > >> > > <met_help at ucar.edu>
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > >> Binyu,
> > > > > >> > >> All 5 ensemble_stat files were successfully loaded
to
> > > > > >> > >> the
> > > > > >> > >> DB and
> > > > the
> > > > > >> > >> data
> > > > > >> > >> is available for potting. Here is the example:
> > > > > >> > >> 20200826_164848.
> > > > > >> > >> Your screenshot displays that you try to create a
> > > > > >> > >> histogram
> > > > > >> > >> plot.
> > > > > >> > >> The
> > > > > >> > >> data for this plot comes from line_data_rhist table.
> > > > > >> > >> This
> > > > > >> > >> table
> > > > gets
> > > > > >> > >> filled
> > > > > >> > >> with 'RHIST' line type.
> > > > > >> > >>
'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
> > > > > >> > >> file
> > > > contains
> > > > > >> > >> only
> > > > > >> > >> 'ORANK' line type so non of it's records were saved
> > > > > >> > >> into
> > > > > >> > >> line_data_rhist
> > > > > >> > >> table.
> > > > > >> > >> Currently line_data_rhist table contains data for
> > > > > >> > >> fcst_valid_beg:
> > > > > >> > >> '2019-02-25 18:00:00' and '2019-02-25 19:00:00' (
> > > > > >> > >> fcst_lead:
> > > > 20000
> > > > > >> > >> and
> > > > > >> > >> 30000). The data came from
> > > > > >> > >> ensemble_stat_Reventador_hr02_20190225_180000V.stat
and
> > > > > >> > >> ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > > > >> > >>
> > > > > >> > >> Could you check if other files have records with
RHIST
> > > > > >> > >> line
> > > > > >> > >> type.
> > > > > >> > >>
> > > > > >> > >> Tatiana
> > > > > >> > >>
> > > > > >> > >> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov
wrote:
> > > > > >> > >> > Tatiana,
> > > > > >> > >> >
> > > > > >> > >> > Yes that is weird. If you see the attached file,
you
> > > > > >> > >> > can
> > > > > >> > >> > see
> > > > > >> > >> > there only two files were uploaded. I expect to
have
> > > > > >> > >> > all
> > > > > >> > >> > 5.
> > > > > >> > >> > Binyu
> > > > > >> > >> >
> > > > > >> > >> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek via
RT
> > > > > >> > >> > <met_help at ucar.edu>
> > > > > >> > >> > wrote:
> > > > > >> > >> >
> > > > > >> > >> > > Binyu,
> > > > > >> > >> > >
> > > > > >> > >> > > In your log file there is a record that
> > > > > >> > >> > >
'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
> > > > > >> > >> > > was
> > > > > >> > >> > > successfully
> > > > > >> > >> > > loaded - all 53 lines.
> > > > > >> > >> > > I also can create a plot using the data from
this
> > > > > >> > >> > > file
> > > > > >> > >> > > -
> > > > ORANK
> > > > > >> > >> > > statistics
> > > > > >> > >> > > with 40000 forecast lead: 20200826_151853.
> > > > > >> > >> > >
> > > > > >> > >> > > Tatiana
> > > > > >> > >> > >
> > > > > >> > >> > >
> > > > > >> > >> > > On Wed Aug 26 08:32:17 2020, binyu.wang at noaa.gov
> > > > > >> > >> > > wrote:
> > > > > >> > >> > > > Tatiana,
> > > > > >> > >> > > >
> > > > > >> > >> > > > I didn't see any error from the log file. I
> > > > > >> > >> > > > attached
> > > > > >> > >> > > > the
> > > > log
> > > > > >> > >> > > > file
> > > > > >> > >> > > > and two
> > > > > >> > >> > > > of the stat files:
> > > > > >> > >> > > >
ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > > > >> > >> > > is
> > > > > >> > >> > > > readable from METViewer, but
> > > > > >> > >> > > >
"ensemble_stat_Reventador_hr04_20190225_200000V.stat"
> > > > > >> > >> > > > not.
> > > > > >> > >> > > >
> > > > > >> > >> > > > Thank you.
> > > > > >> > >> > > >
> > > > > >> > >> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana Burek
via
> > > > > >> > >> > > > RT
> > > > > >> > >> > > > <met_help at ucar.edu>
> > > > > >> > >> > > > wrote:
> > > > > >> > >> > > >
> > > > > >> > >> > > > > Hi Binyu,
> > > > > >> > >> > > > > I assume that your files are on WCOSS. I
don't
> > > > > >> > >> > > > > have
> > > > > >> > >> > > > > an
> > > > > >> > >> > > > > access to
> > > > > >> > >> > > > > the
> > > > > >> > >> > > > > server and can't see the files.
> > > > > >> > >> > > > > Do you have logging messages from the load?
Any
> > > > > >> > >> > > > > errors?
> > > > > >> > >> > > > >
> > > > > >> > >> > > > > I see that only 5 ensemble_stat files were
> > > > > >> > >> > > > > loaded
> > > > > >> > >> > > > > to this
> > > > > >> > >> > > > > database:
> > > > > >> > >> > > > > |
> > > > > >> > >> > > > >
ensemble_stat_Reventador_hr02_20190225_180000V.stat
> > > > > >> > >> > > > > |
> > > > > >> > >> > > > > |
> > > > > >> > >> > > > >
ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > > > >> > >> > > > > |
> > > > > >> > >> > > > > |
> > > > > >> > >> > > > >
ensemble_stat_Reventador_hr04_20190225_200000V.stat
> > > > > >> > >> > > > > |
> > > > > >> > >> > > > > |
> > > > > >> > >> > > > >
ensemble_stat_Reventador_hr05_20190225_210000V.stat
> > > > > >> > >> > > > > |
> > > > > >> > >> > > > > |
> > > > > >> > >> > > > >
ensemble_stat_Reventador_hr06_20190225_220000V.stat
> > > > > >> > >> > > > >
> > > > > >> > >> > > > > Tatiana
> > > > > >> > >> > > > >
> > > > > >> > >> > > > > On Tue Aug 25 20:02:54 2020,
> > > > > >> > >> > > > > binyu.wang at noaa.gov
> > > > > >> > >> > > > > wrote:
> > > > > >> > >> > > > > > Hello,
> > > > > >> > >> > > > > >
> > > > > >> > >> > > > > > I created 5 ensemble_stat* files (from
hr18-
> > > > > >> > >> > > > > > hr22)
> > > > under
> > > > > >> > >> > > > > > the
> > > > > >> > >> > > directory
> > > > > >> > >> > > > > > below, but only the first two (hr18 and
> > > > > >> > >> > > > > > hr19)
> > > > > >> > >> > > > > > could be
> > > > > >> > >> > > > > > read
> > > > > >> > >> > > > > > from
> > > > > >> > >> > > > > > METVIewer. Those files are under database
> > > > > >> > >> > > "mv_g2g_met_verf_Reventador"
> > > > > >> > >> > > > > >
> > > > > >> > >> > > > > >
> > > > > >> > >> > >
> > > > > >> > >>
> > > > > >>
> > > >
> >
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > > > > >> > >> > > > > >
> > > > > >> > >> > > > > > What is the problem? Thank you.
> > > > > >> > >> > > > > > Binyu
> > > > > >> > >> > > > >
> > > > > >> > >> > > > >
> > > > > >> > >> > > > >
> > > > > >> > >> > > > >
> > > > > >> > >> > >
> > > > > >> > >> > >
> > > > > >> > >> > >
> > > > > >> > >> > >
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >>
> > > > > >> > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > >
> > > >
> > > >
> > > >
> >
> >
> >
> >
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: binyu.wang at noaa.gov
Time: Mon Aug 31 07:58:22 2020
Thank you for letting me know, but why I still can see it on
METViewer?
Binyu
On Mon, Aug 31, 2020 at 9:44 AM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> Binyu
>
> mv_g2g_met_verf_Raikoke database was deleted on Friday.
>
> Tatiana
>
> On Sun Aug 30 20:10:14 2020, binyu.wang at noaa.gov wrote:
> > Hello,
> > Tatiana, if you hadn't deleted 'mv_g2g_met_verf_Raikoke', please
just
> > keep
> > it. Please confirm. Thank you.
> > Binyu
> >
> > On Fri, Aug 28, 2020 at 2:03 PM Tatiana Burek via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > Hello Binyu
> > > I created a new database 'mv_g2g_met_verf_Raikoke_all_data' and
> > > copied all
> > > data from 'mv_g2g_met_verf_Raikoke' to it. Please verify that
all
> > > data is
> > > correct and let me know. I will delete 'mv_g2g_met_verf_Raikoke'
> > > after that.
> > >
> > > Tatiana
> > >
> > > On Thu Aug 27 16:23:18 2020, binyu.wang at noaa.gov wrote:
> > > > Hello Tatiana,
> > > >
> > > > Would you please help to rename the database
> > > > "mv_g2g_met_verf_Raikoke"
> > > > to
> > > > "mv_g2g_met_verf_Raikoke.all.data"?
> > > > Thank you.
> > > >
> > > >
> > > > On Thu, Aug 27, 2020 at 2:16 PM Tatiana Burek via RT
> > > > <met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > There is no easy way to rename the database.
> > > > > I can create a new database with a new name, dump the data
from
> > > > > the
> > > > > old
> > > > > database, load it to the new one and finally remove the old
> > > > > database.
> > > > >
> > > > >
> > > > > Tatiana
> > > > >
> > > > > On Thu Aug 27 12:04:00 2020, binyu.wang at noaa.gov wrote:
> > > > > > Another quick question: Can we change the database name
without
> > > > > > having to
> > > > > > drop the database and reload all of the data again?
> > > > > >
> > > > > > Binyu
> > > > > >
> > > > > > On Thu, Aug 27, 2020 at 2:03 PM Binyu Wang - NOAA
Affiliate <
> > > > > > binyu.wang at noaa.gov> wrote:
> > > > > >
> > > > > > > Tatiana,
> > > > > > >
> > > > > > > "PCT" is generated later using the ensemble*stat files
which
> > > > > > > I
> > > > > > > sent to
> > > > > you
> > > > > > > earlier. So you didn't see it in those two files. I
didn't
> > > > > > > understand:
> > > > > if
> > > > > > > the 3 files didn't have "RHIST" because there was no
> > > > > > > matching
> > > > > > > pair
> > > > > > > during that time period, why could they be used for RELY
> > > > > > > plot?
> > > > > > >
> > > > > > > Binyu
> > > > > > >
> > > > > > > On Thu, Aug 27, 2020 at 1:43 PM Tatiana Burek via RT <
> > > > > met_help at ucar.edu>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Binyu
> > > > > > >>
> > > > > > >> Reliability plot uses data from line_data_pct table (
"PCT"
> > > > > > >> line
> > > > > > >> type
> > > > > ).
> > > > > > >> Both stat files that you sent me don't contain PCT line
type
> > > > > > >> and
> > > > > variable
> > > > > > >> 'VAFTD_ENS_FREQ_ge0.1'. Did you have data in this
database
> > > > > > >> before the
> > > > > > >> latest load?
> > > > > > >>
> > > > > > >> Tatiana
> > > > > > >>
> > > > > > >> On Thu Aug 27 08:48:35 2020, binyu.wang at noaa.gov wrote:
> > > > > > >> > Hello Tatiana,
> > > > > > >> >
> > > > > > >> > Sorry to ask more: as I said in previous mail I can
not
> > > > > > >> > find
> > > > > > >> > the
> > > > > other
> > > > > > >> > 3
> > > > > > >> > files from MET_Viewer because there is no RHIST
inside of
> > > > > > >> > those
> > > > > files,
> > > > > > >> > if
> > > > > > >> > this is because there is no matched pairs between
obs. and
> > > > > > >> > forecast,
> > > > > > >> > why I
> > > > > > >> > can see them in the RELY plot? (see the attachment
> > > > > > >> > please).
> > > > > > >> > Are RELY
> > > > > > >> > plots
> > > > > > >> > created based on matching pairs?
> > > > > > >> >
> > > > > > >> > Thank you.
> > > > > > >> > Binyu
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Wed, Aug 26, 2020 at 10:47 PM Binyu Wang - NOAA
> > > > > > >> > Affiliate <
> > > > > > >> > binyu.wang at noaa.gov> wrote:
> > > > > > >> >
> > > > > > >> > > Hello,
> > > > > > >> > >
> > > > > > >> > > I think I know the answer now, but just to confirm
with
> > > > > > >> > > you:
> > > > > > >> > > if
> > > > > there
> > > > > > >> > > is
> > > > > > >> > > no RHIST line type in the stat file, which
indicates
> > > > > > >> > > there
> > > > > > >> > > is no
> > > > > > >> > > matching
> > > > > > >> > > pair at all between obs. and forecast, is that
correct?
> > > > > > >> > >
> > > > > > >> > > Thank you.
> > > > > > >> > >
> > > > > > >> > > On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via
RT
> > > > > > >> > > <met_help at ucar.edu>
> > > > > > >> > > wrote:
> > > > > > >> > >
> > > > > > >> > >> Binyu,
> > > > > > >> > >> All 5 ensemble_stat files were successfully loaded
to
> > > > > > >> > >> the
> > > > > > >> > >> DB and
> > > > > the
> > > > > > >> > >> data
> > > > > > >> > >> is available for potting. Here is the example:
> > > > > > >> > >> 20200826_164848.
> > > > > > >> > >> Your screenshot displays that you try to create a
> > > > > > >> > >> histogram
> > > > > > >> > >> plot.
> > > > > > >> > >> The
> > > > > > >> > >> data for this plot comes from line_data_rhist
table.
> > > > > > >> > >> This
> > > > > > >> > >> table
> > > > > gets
> > > > > > >> > >> filled
> > > > > > >> > >> with 'RHIST' line type.
> > > > > > >> > >>
'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
> > > > > > >> > >> file
> > > > > contains
> > > > > > >> > >> only
> > > > > > >> > >> 'ORANK' line type so non of it's records were
saved
> > > > > > >> > >> into
> > > > > > >> > >> line_data_rhist
> > > > > > >> > >> table.
> > > > > > >> > >> Currently line_data_rhist table contains data for
> > > > > > >> > >> fcst_valid_beg:
> > > > > > >> > >> '2019-02-25 18:00:00' and '2019-02-25 19:00:00' (
> > > > > > >> > >> fcst_lead:
> > > > > 20000
> > > > > > >> > >> and
> > > > > > >> > >> 30000). The data came from
> > > > > > >> > >>
ensemble_stat_Reventador_hr02_20190225_180000V.stat and
> > > > > > >> > >>
ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > > > > >> > >>
> > > > > > >> > >> Could you check if other files have records with
RHIST
> > > > > > >> > >> line
> > > > > > >> > >> type.
> > > > > > >> > >>
> > > > > > >> > >> Tatiana
> > > > > > >> > >>
> > > > > > >> > >> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov
wrote:
> > > > > > >> > >> > Tatiana,
> > > > > > >> > >> >
> > > > > > >> > >> > Yes that is weird. If you see the attached file,
you
> > > > > > >> > >> > can
> > > > > > >> > >> > see
> > > > > > >> > >> > there only two files were uploaded. I expect to
have
> > > > > > >> > >> > all
> > > > > > >> > >> > 5.
> > > > > > >> > >> > Binyu
> > > > > > >> > >> >
> > > > > > >> > >> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek
via RT
> > > > > > >> > >> > <met_help at ucar.edu>
> > > > > > >> > >> > wrote:
> > > > > > >> > >> >
> > > > > > >> > >> > > Binyu,
> > > > > > >> > >> > >
> > > > > > >> > >> > > In your log file there is a record that
> > > > > > >> > >> > >
'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
> > > > > > >> > >> > > was
> > > > > > >> > >> > > successfully
> > > > > > >> > >> > > loaded - all 53 lines.
> > > > > > >> > >> > > I also can create a plot using the data from
this
> > > > > > >> > >> > > file
> > > > > > >> > >> > > -
> > > > > ORANK
> > > > > > >> > >> > > statistics
> > > > > > >> > >> > > with 40000 forecast lead: 20200826_151853.
> > > > > > >> > >> > >
> > > > > > >> > >> > > Tatiana
> > > > > > >> > >> > >
> > > > > > >> > >> > >
> > > > > > >> > >> > > On Wed Aug 26 08:32:17 2020,
binyu.wang at noaa.gov
> > > > > > >> > >> > > wrote:
> > > > > > >> > >> > > > Tatiana,
> > > > > > >> > >> > > >
> > > > > > >> > >> > > > I didn't see any error from the log file. I
> > > > > > >> > >> > > > attached
> > > > > > >> > >> > > > the
> > > > > log
> > > > > > >> > >> > > > file
> > > > > > >> > >> > > > and two
> > > > > > >> > >> > > > of the stat files:
> > > > > > >> > >> > > >
ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > > > > >> > >> > > is
> > > > > > >> > >> > > > readable from METViewer, but
> > > > > > >> > >> > > >
> "ensemble_stat_Reventador_hr04_20190225_200000V.stat"
> > > > > > >> > >> > > > not.
> > > > > > >> > >> > > >
> > > > > > >> > >> > > > Thank you.
> > > > > > >> > >> > > >
> > > > > > >> > >> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana
Burek via
> > > > > > >> > >> > > > RT
> > > > > > >> > >> > > > <met_help at ucar.edu>
> > > > > > >> > >> > > > wrote:
> > > > > > >> > >> > > >
> > > > > > >> > >> > > > > Hi Binyu,
> > > > > > >> > >> > > > > I assume that your files are on WCOSS. I
don't
> > > > > > >> > >> > > > > have
> > > > > > >> > >> > > > > an
> > > > > > >> > >> > > > > access to
> > > > > > >> > >> > > > > the
> > > > > > >> > >> > > > > server and can't see the files.
> > > > > > >> > >> > > > > Do you have logging messages from the
load? Any
> > > > > > >> > >> > > > > errors?
> > > > > > >> > >> > > > >
> > > > > > >> > >> > > > > I see that only 5 ensemble_stat files were
> > > > > > >> > >> > > > > loaded
> > > > > > >> > >> > > > > to this
> > > > > > >> > >> > > > > database:
> > > > > > >> > >> > > > > |
> > > > > > >> > >> > > > >
> ensemble_stat_Reventador_hr02_20190225_180000V.stat
> > > > > > >> > >> > > > > |
> > > > > > >> > >> > > > > |
> > > > > > >> > >> > > > >
> ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > > > > >> > >> > > > > |
> > > > > > >> > >> > > > > |
> > > > > > >> > >> > > > >
> ensemble_stat_Reventador_hr04_20190225_200000V.stat
> > > > > > >> > >> > > > > |
> > > > > > >> > >> > > > > |
> > > > > > >> > >> > > > >
> ensemble_stat_Reventador_hr05_20190225_210000V.stat
> > > > > > >> > >> > > > > |
> > > > > > >> > >> > > > > |
> > > > > > >> > >> > > > >
> ensemble_stat_Reventador_hr06_20190225_220000V.stat
> > > > > > >> > >> > > > >
> > > > > > >> > >> > > > > Tatiana
> > > > > > >> > >> > > > >
> > > > > > >> > >> > > > > On Tue Aug 25 20:02:54 2020,
> > > > > > >> > >> > > > > binyu.wang at noaa.gov
> > > > > > >> > >> > > > > wrote:
> > > > > > >> > >> > > > > > Hello,
> > > > > > >> > >> > > > > >
> > > > > > >> > >> > > > > > I created 5 ensemble_stat* files (from
hr18-
> > > > > > >> > >> > > > > > hr22)
> > > > > under
> > > > > > >> > >> > > > > > the
> > > > > > >> > >> > > directory
> > > > > > >> > >> > > > > > below, but only the first two (hr18 and
> > > > > > >> > >> > > > > > hr19)
> > > > > > >> > >> > > > > > could be
> > > > > > >> > >> > > > > > read
> > > > > > >> > >> > > > > > from
> > > > > > >> > >> > > > > > METVIewer. Those files are under
database
> > > > > > >> > >> > > "mv_g2g_met_verf_Reventador"
> > > > > > >> > >> > > > > >
> > > > > > >> > >> > > > > >
> > > > > > >> > >> > >
> > > > > > >> > >>
> > > > > > >>
> > > > >
> > >
>
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > > > > > >> > >> > > > > >
> > > > > > >> > >> > > > > > What is the problem? Thank you.
> > > > > > >> > >> > > > > > Binyu
> > > > > > >> > >> > > > >
> > > > > > >> > >> > > > >
> > > > > > >> > >> > > > >
> > > > > > >> > >> > > > >
> > > > > > >> > >> > >
> > > > > > >> > >> > >
> > > > > > >> > >> > >
> > > > > > >> > >> > >
> > > > > > >> > >>
> > > > > > >> > >>
> > > > > > >> > >>
> > > > > > >> > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > >
> > > > >
> > > > >
> > > > >
> > >
> > >
> > >
> > >
>
>
>
>
------------------------------------------------
Subject: can not upload files to MET_Viewer
From: Tatiana Burek
Time: Mon Aug 31 09:29:53 2020
METviewer was caching the database name.
It is not available anymore.
Tatiana
On Mon Aug 31 07:58:22 2020, binyu.wang at noaa.gov wrote:
> Thank you for letting me know, but why I still can see it on
> METViewer?
>
> Binyu
>
> On Mon, Aug 31, 2020 at 9:44 AM Tatiana Burek via RT
> <met_help at ucar.edu>
> wrote:
>
> > Binyu
> >
> > mv_g2g_met_verf_Raikoke database was deleted on Friday.
> >
> > Tatiana
> >
> > On Sun Aug 30 20:10:14 2020, binyu.wang at noaa.gov wrote:
> > > Hello,
> > > Tatiana, if you hadn't deleted 'mv_g2g_met_verf_Raikoke',
please
> > > just
> > > keep
> > > it. Please confirm. Thank you.
> > > Binyu
> > >
> > > On Fri, Aug 28, 2020 at 2:03 PM Tatiana Burek via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > > > Hello Binyu
> > > > I created a new database 'mv_g2g_met_verf_Raikoke_all_data'
and
> > > > copied all
> > > > data from 'mv_g2g_met_verf_Raikoke' to it. Please verify that
all
> > > > data is
> > > > correct and let me know. I will delete
'mv_g2g_met_verf_Raikoke'
> > > > after that.
> > > >
> > > > Tatiana
> > > >
> > > > On Thu Aug 27 16:23:18 2020, binyu.wang at noaa.gov wrote:
> > > > > Hello Tatiana,
> > > > >
> > > > > Would you please help to rename the database
> > > > > "mv_g2g_met_verf_Raikoke"
> > > > > to
> > > > > "mv_g2g_met_verf_Raikoke.all.data"?
> > > > > Thank you.
> > > > >
> > > > >
> > > > > On Thu, Aug 27, 2020 at 2:16 PM Tatiana Burek via RT
> > > > > <met_help at ucar.edu>
> > > > > wrote:
> > > > >
> > > > > > There is no easy way to rename the database.
> > > > > > I can create a new database with a new name, dump the data
> > > > > > from
> > > > > > the
> > > > > > old
> > > > > > database, load it to the new one and finally remove the
old
> > > > > > database.
> > > > > >
> > > > > >
> > > > > > Tatiana
> > > > > >
> > > > > > On Thu Aug 27 12:04:00 2020, binyu.wang at noaa.gov wrote:
> > > > > > > Another quick question: Can we change the database name
> > > > > > > without
> > > > > > > having to
> > > > > > > drop the database and reload all of the data again?
> > > > > > >
> > > > > > > Binyu
> > > > > > >
> > > > > > > On Thu, Aug 27, 2020 at 2:03 PM Binyu Wang - NOAA
Affiliate
> > > > > > > <
> > > > > > > binyu.wang at noaa.gov> wrote:
> > > > > > >
> > > > > > > > Tatiana,
> > > > > > > >
> > > > > > > > "PCT" is generated later using the ensemble*stat files
> > > > > > > > which
> > > > > > > > I
> > > > > > > > sent to
> > > > > > you
> > > > > > > > earlier. So you didn't see it in those two files. I
> > > > > > > > didn't
> > > > > > > > understand:
> > > > > > if
> > > > > > > > the 3 files didn't have "RHIST" because there was no
> > > > > > > > matching
> > > > > > > > pair
> > > > > > > > during that time period, why could they be used for
RELY
> > > > > > > > plot?
> > > > > > > >
> > > > > > > > Binyu
> > > > > > > >
> > > > > > > > On Thu, Aug 27, 2020 at 1:43 PM Tatiana Burek via RT <
> > > > > > met_help at ucar.edu>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Binyu
> > > > > > > >>
> > > > > > > >> Reliability plot uses data from line_data_pct table (
> > > > > > > >> "PCT"
> > > > > > > >> line
> > > > > > > >> type
> > > > > > ).
> > > > > > > >> Both stat files that you sent me don't contain PCT
line
> > > > > > > >> type
> > > > > > > >> and
> > > > > > variable
> > > > > > > >> 'VAFTD_ENS_FREQ_ge0.1'. Did you have data in this
> > > > > > > >> database
> > > > > > > >> before the
> > > > > > > >> latest load?
> > > > > > > >>
> > > > > > > >> Tatiana
> > > > > > > >>
> > > > > > > >> On Thu Aug 27 08:48:35 2020, binyu.wang at noaa.gov
wrote:
> > > > > > > >> > Hello Tatiana,
> > > > > > > >> >
> > > > > > > >> > Sorry to ask more: as I said in previous mail I can
> > > > > > > >> > not
> > > > > > > >> > find
> > > > > > > >> > the
> > > > > > other
> > > > > > > >> > 3
> > > > > > > >> > files from MET_Viewer because there is no RHIST
inside
> > > > > > > >> > of
> > > > > > > >> > those
> > > > > > files,
> > > > > > > >> > if
> > > > > > > >> > this is because there is no matched pairs between
obs.
> > > > > > > >> > and
> > > > > > > >> > forecast,
> > > > > > > >> > why I
> > > > > > > >> > can see them in the RELY plot? (see the attachment
> > > > > > > >> > please).
> > > > > > > >> > Are RELY
> > > > > > > >> > plots
> > > > > > > >> > created based on matching pairs?
> > > > > > > >> >
> > > > > > > >> > Thank you.
> > > > > > > >> > Binyu
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Wed, Aug 26, 2020 at 10:47 PM Binyu Wang - NOAA
> > > > > > > >> > Affiliate <
> > > > > > > >> > binyu.wang at noaa.gov> wrote:
> > > > > > > >> >
> > > > > > > >> > > Hello,
> > > > > > > >> > >
> > > > > > > >> > > I think I know the answer now, but just to
confirm
> > > > > > > >> > > with
> > > > > > > >> > > you:
> > > > > > > >> > > if
> > > > > > there
> > > > > > > >> > > is
> > > > > > > >> > > no RHIST line type in the stat file, which
indicates
> > > > > > > >> > > there
> > > > > > > >> > > is no
> > > > > > > >> > > matching
> > > > > > > >> > > pair at all between obs. and forecast, is that
> > > > > > > >> > > correct?
> > > > > > > >> > >
> > > > > > > >> > > Thank you.
> > > > > > > >> > >
> > > > > > > >> > > On Wed, Aug 26, 2020 at 1:45 PM Tatiana Burek via
RT
> > > > > > > >> > > <met_help at ucar.edu>
> > > > > > > >> > > wrote:
> > > > > > > >> > >
> > > > > > > >> > >> Binyu,
> > > > > > > >> > >> All 5 ensemble_stat files were successfully
loaded
> > > > > > > >> > >> to
> > > > > > > >> > >> the
> > > > > > > >> > >> DB and
> > > > > > the
> > > > > > > >> > >> data
> > > > > > > >> > >> is available for potting. Here is the example:
> > > > > > > >> > >> 20200826_164848.
> > > > > > > >> > >> Your screenshot displays that you try to create
a
> > > > > > > >> > >> histogram
> > > > > > > >> > >> plot.
> > > > > > > >> > >> The
> > > > > > > >> > >> data for this plot comes from line_data_rhist
> > > > > > > >> > >> table.
> > > > > > > >> > >> This
> > > > > > > >> > >> table
> > > > > > gets
> > > > > > > >> > >> filled
> > > > > > > >> > >> with 'RHIST' line type.
> > > > > > > >> > >>
'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
> > > > > > > >> > >> file
> > > > > > contains
> > > > > > > >> > >> only
> > > > > > > >> > >> 'ORANK' line type so non of it's records were
saved
> > > > > > > >> > >> into
> > > > > > > >> > >> line_data_rhist
> > > > > > > >> > >> table.
> > > > > > > >> > >> Currently line_data_rhist table contains data
for
> > > > > > > >> > >> fcst_valid_beg:
> > > > > > > >> > >> '2019-02-25 18:00:00' and '2019-02-25 19:00:00'
(
> > > > > > > >> > >> fcst_lead:
> > > > > > 20000
> > > > > > > >> > >> and
> > > > > > > >> > >> 30000). The data came from
> > > > > > > >> > >>
ensemble_stat_Reventador_hr02_20190225_180000V.stat
> > > > > > > >> > >> and
> > > > > > > >> > >>
ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > > > > > >> > >>
> > > > > > > >> > >> Could you check if other files have records with
> > > > > > > >> > >> RHIST
> > > > > > > >> > >> line
> > > > > > > >> > >> type.
> > > > > > > >> > >>
> > > > > > > >> > >> Tatiana
> > > > > > > >> > >>
> > > > > > > >> > >> n Wed Aug 26 10:29:05 2020, binyu.wang at noaa.gov
> > > > > > > >> > >> wrote:
> > > > > > > >> > >> > Tatiana,
> > > > > > > >> > >> >
> > > > > > > >> > >> > Yes that is weird. If you see the attached
file,
> > > > > > > >> > >> > you
> > > > > > > >> > >> > can
> > > > > > > >> > >> > see
> > > > > > > >> > >> > there only two files were uploaded. I expect
to
> > > > > > > >> > >> > have
> > > > > > > >> > >> > all
> > > > > > > >> > >> > 5.
> > > > > > > >> > >> > Binyu
> > > > > > > >> > >> >
> > > > > > > >> > >> > On Wed, Aug 26, 2020 at 11:24 AM Tatiana Burek
> > > > > > > >> > >> > via RT
> > > > > > > >> > >> > <met_help at ucar.edu>
> > > > > > > >> > >> > wrote:
> > > > > > > >> > >> >
> > > > > > > >> > >> > > Binyu,
> > > > > > > >> > >> > >
> > > > > > > >> > >> > > In your log file there is a record that
> > > > > > > >> > >> > >
'ensemble_stat_Reventador_hr04_20190225_200000V.stat'
> > > > > > > >> > >> > > was
> > > > > > > >> > >> > > successfully
> > > > > > > >> > >> > > loaded - all 53 lines.
> > > > > > > >> > >> > > I also can create a plot using the data from
> > > > > > > >> > >> > > this
> > > > > > > >> > >> > > file
> > > > > > > >> > >> > > -
> > > > > > ORANK
> > > > > > > >> > >> > > statistics
> > > > > > > >> > >> > > with 40000 forecast lead: 20200826_151853.
> > > > > > > >> > >> > >
> > > > > > > >> > >> > > Tatiana
> > > > > > > >> > >> > >
> > > > > > > >> > >> > >
> > > > > > > >> > >> > > On Wed Aug 26 08:32:17 2020,
> > > > > > > >> > >> > > binyu.wang at noaa.gov
> > > > > > > >> > >> > > wrote:
> > > > > > > >> > >> > > > Tatiana,
> > > > > > > >> > >> > > >
> > > > > > > >> > >> > > > I didn't see any error from the log file.
I
> > > > > > > >> > >> > > > attached
> > > > > > > >> > >> > > > the
> > > > > > log
> > > > > > > >> > >> > > > file
> > > > > > > >> > >> > > > and two
> > > > > > > >> > >> > > > of the stat files:
> > > > > > > >> > >> > > >
ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > > > > > >> > >> > > is
> > > > > > > >> > >> > > > readable from METViewer, but
> > > > > > > >> > >> > > >
> > "ensemble_stat_Reventador_hr04_20190225_200000V.stat"
> > > > > > > >> > >> > > > not.
> > > > > > > >> > >> > > >
> > > > > > > >> > >> > > > Thank you.
> > > > > > > >> > >> > > >
> > > > > > > >> > >> > > > On Wed, Aug 26, 2020 at 9:30 AM Tatiana
Burek
> > > > > > > >> > >> > > > via
> > > > > > > >> > >> > > > RT
> > > > > > > >> > >> > > > <met_help at ucar.edu>
> > > > > > > >> > >> > > > wrote:
> > > > > > > >> > >> > > >
> > > > > > > >> > >> > > > > Hi Binyu,
> > > > > > > >> > >> > > > > I assume that your files are on WCOSS. I
> > > > > > > >> > >> > > > > don't
> > > > > > > >> > >> > > > > have
> > > > > > > >> > >> > > > > an
> > > > > > > >> > >> > > > > access to
> > > > > > > >> > >> > > > > the
> > > > > > > >> > >> > > > > server and can't see the files.
> > > > > > > >> > >> > > > > Do you have logging messages from the
load?
> > > > > > > >> > >> > > > > Any
> > > > > > > >> > >> > > > > errors?
> > > > > > > >> > >> > > > >
> > > > > > > >> > >> > > > > I see that only 5 ensemble_stat files
were
> > > > > > > >> > >> > > > > loaded
> > > > > > > >> > >> > > > > to this
> > > > > > > >> > >> > > > > database:
> > > > > > > >> > >> > > > > |
> > > > > > > >> > >> > > > >
> > ensemble_stat_Reventador_hr02_20190225_180000V.stat
> > > > > > > >> > >> > > > > |
> > > > > > > >> > >> > > > > |
> > > > > > > >> > >> > > > >
> > ensemble_stat_Reventador_hr03_20190225_190000V.stat
> > > > > > > >> > >> > > > > |
> > > > > > > >> > >> > > > > |
> > > > > > > >> > >> > > > >
> > ensemble_stat_Reventador_hr04_20190225_200000V.stat
> > > > > > > >> > >> > > > > |
> > > > > > > >> > >> > > > > |
> > > > > > > >> > >> > > > >
> > ensemble_stat_Reventador_hr05_20190225_210000V.stat
> > > > > > > >> > >> > > > > |
> > > > > > > >> > >> > > > > |
> > > > > > > >> > >> > > > >
> > ensemble_stat_Reventador_hr06_20190225_220000V.stat
> > > > > > > >> > >> > > > >
> > > > > > > >> > >> > > > > Tatiana
> > > > > > > >> > >> > > > >
> > > > > > > >> > >> > > > > On Tue Aug 25 20:02:54 2020,
> > > > > > > >> > >> > > > > binyu.wang at noaa.gov
> > > > > > > >> > >> > > > > wrote:
> > > > > > > >> > >> > > > > > Hello,
> > > > > > > >> > >> > > > > >
> > > > > > > >> > >> > > > > > I created 5 ensemble_stat* files (from
> > > > > > > >> > >> > > > > > hr18-
> > > > > > > >> > >> > > > > > hr22)
> > > > > > under
> > > > > > > >> > >> > > > > > the
> > > > > > > >> > >> > > directory
> > > > > > > >> > >> > > > > > below, but only the first two (hr18
and
> > > > > > > >> > >> > > > > > hr19)
> > > > > > > >> > >> > > > > > could be
> > > > > > > >> > >> > > > > > read
> > > > > > > >> > >> > > > > > from
> > > > > > > >> > >> > > > > > METVIewer. Those files are under
database
> > > > > > > >> > >> > > "mv_g2g_met_verf_Reventador"
> > > > > > > >> > >> > > > > >
> > > > > > > >> > >> > > > > >
> > > > > > > >> > >> > >
> > > > > > > >> > >>
> > > > > > > >>
> > > > > >
> > > >
> >
/gpfs/dell2/emc/modeling/noscrub/Binyu.Wang/MET/stat/met_Reventador_stat
> > > > > > > >> > >> > > > > >
> > > > > > > >> > >> > > > > > What is the problem? Thank you.
> > > > > > > >> > >> > > > > > Binyu
> > > > > > > >> > >> > > > >
> > > > > > > >> > >> > > > >
> > > > > > > >> > >> > > > >
> > > > > > > >> > >> > > > >
> > > > > > > >> > >> > >
> > > > > > > >> > >> > >
> > > > > > > >> > >> > >
> > > > > > > >> > >> > >
> > > > > > > >> > >>
> > > > > > > >> > >>
> > > > > > > >> > >>
> > > > > > > >> > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > >
> > > >
> > > >
> > > >
> >
> >
> >
> >
------------------------------------------------
More information about the Met_help
mailing list