[Met_help] [rt.rap.ucar.edu #90858] History for troubles after updating to METv8.1

[Tatiana Burek via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hi,

I'm trying to sort out some issues I'm having after upgrading to MET v8.1
(after the v6 build I'd been using disappeared from WCOSS).  I got my MET
jobs running okay after pointing at v8.1, and the .stat files look okay to
my untrained eye, but the data uploaded to the metviewer server behaves
oddly.  Namely for probability output, only the lowest threshold seems to
plot normally.  Looking at thresholds above the lowest level and just using
output from v8.1 gives this error in the log:

Failed to create a plot. ROC/Reliability plots must contain data from at
least one obs_thresh

A couple of questions floating in my head that you might be able to answer
for me:

- Could my data not be getting fully ingested to the metviewer server?
- Is a database with output from different MET versions problematic?

I can keep digging here, but if you can point me in the right direction I'd
appreciate it!

Thanks,

Matt


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: troubles after updating to METv8.1
From: Tatiana Burek
Time: Mon Jul 01 14:34:22 2019

Hi Mat,

METviewer should handle data from all MET versions. There are some
cases when the data could not fully be loaded into DB. It could happen
when the size of the data files is larger then the limit or the length
of some string is bigger than expected. Did you get any error message
during the loading?

I looked at your data in mv_hrefv3 database. The latest fcst_valid_beg
value in line_data_pct table is '2019-05-30 12:00:00'. You don't have
data for June. I also could not find .stat files to verify that they
are valid. I was looking in /metviewer/staging/mpyle/hrefv3 directory
but it is empty.

Tatiana


On Mon Jul 01 12:15:23 2019, Matthew.Pyle at noaa.gov wrote:
> Hi,
>
> I'm trying to sort out some issues I'm having after upgrading to MET
v8.1
> (after the v6 build I'd been using disappeared from WCOSS).  I got
my MET
> jobs running okay after pointing at v8.1, and the .stat files look
okay to
> my untrained eye, but the data uploaded to the metviewer server
behaves
> oddly.  Namely for probability output, only the lowest threshold
seems to
> plot normally.  Looking at thresholds above the lowest level and
just using
> output from v8.1 gives this error in the log:
>
> Failed to create a plot. ROC/Reliability plots must contain data
from at
> least one obs_thresh
>
> A couple of questions floating in my head that you might be able to
answer
> for me:
>
> - Could my data not be getting fully ingested to the metviewer
server?
> - Is a database with output from different MET versions problematic?
>
> I can keep digging here, but if you can point me in the right
direction I'd
> appreciate it!
>
> Thanks,
>
> Matt



------------------------------------------------
Subject: troubles after updating to METv8.1
From: "matthew.pyle at noaa.gov"
Time: Tue Jul 02 07:16:20 2019

Hi Tatiana,

I scrub out the files from the staging directory after loading them
into
the database.  But I did just upload another day that still is in my
hrefv3
directory.  The log file /usr1/metviewer/capture_8.1_load.log from
loading
this new data shows some errors with PSTD records.

Thanks,

Matt

On Mon, Jul 1, 2019 at 4:34 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:

> Hi Mat,
>
> METviewer should handle data from all MET versions. There are some
cases
> when the data could not fully be loaded into DB. It could happen
when the
> size of the data files is larger then the limit or the length of
some
> string is bigger than expected. Did you get any error message during
the
> loading?
>
> I looked at your data in mv_hrefv3 database. The latest
fcst_valid_beg
> value in line_data_pct table is '2019-05-30 12:00:00'. You don't
have data
> for June. I also could not find .stat files to verify that they are
valid.
> I was looking in /metviewer/staging/mpyle/hrefv3 directory but it is
empty.
>
> Tatiana
>
>
> On Mon Jul 01 12:15:23 2019, Matthew.Pyle at noaa.gov wrote:
> > Hi,
> >
> > I'm trying to sort out some issues I'm having after upgrading to
MET v8.1
> > (after the v6 build I'd been using disappeared from WCOSS).  I got
my MET
> > jobs running okay after pointing at v8.1, and the .stat files look
okay
> to
> > my untrained eye, but the data uploaded to the metviewer server
behaves
> > oddly.  Namely for probability output, only the lowest threshold
seems to
> > plot normally.  Looking at thresholds above the lowest level and
just
> using
> > output from v8.1 gives this error in the log:
> >
> > Failed to create a plot. ROC/Reliability plots must contain data
from at
> > least one obs_thresh
> >
> > A couple of questions floating in my head that you might be able
to
> answer
> > for me:
> >
> > - Could my data not be getting fully ingested to the metviewer
server?
> > - Is a database with output from different MET versions
problematic?
> >
> > I can keep digging here, but if you can point me in the right
direction
> I'd
> > appreciate it!
> >
> > Thanks,
> >
> > Matt
>
>
>
>

------------------------------------------------
Subject: troubles after updating to METv8.1
From: Tatiana Burek
Time: Tue Jul 02 11:33:08 2019

Matt,
It was a bug in loading PSTD data which we were going to fix in v2.11.
I patched the current EMC METviewer instance with the fix and you
should be able to load PSTD data now.
My concern is creating duplicates for other data types when you reload
data. We are working on 'removing duplicates' issue but it is not
ready yet. Would it be possible if you drop the existing database and
load all data to the new one.

Tatiana

On Tue Jul 02 07:16:20 2019, Matthew.Pyle at noaa.gov wrote:
> Hi Tatiana,
>
> I scrub out the files from the staging directory after loading them
> into
> the database.  But I did just upload another day that still is in my
> hrefv3
> directory.  The log file /usr1/metviewer/capture_8.1_load.log from
> loading
> this new data shows some errors with PSTD records.
>
> Thanks,
>
> Matt
>
> On Mon, Jul 1, 2019 at 4:34 PM Tatiana Burek via RT
> <met_help at ucar.edu>
> wrote:
>
> > Hi Mat,
> >
> > METviewer should handle data from all MET versions. There are some
> > cases
> > when the data could not fully be loaded into DB. It could happen
when
> > the
> > size of the data files is larger then the limit or the length of
some
> > string is bigger than expected. Did you get any error message
during
> > the
> > loading?
> >
> > I looked at your data in mv_hrefv3 database. The latest
> > fcst_valid_beg
> > value in line_data_pct table is '2019-05-30 12:00:00'. You don't
have
> > data
> > for June. I also could not find .stat files to verify that they
are
> > valid.
> > I was looking in /metviewer/staging/mpyle/hrefv3 directory but it
is
> > empty.
> >
> > Tatiana
> >
> >
> > On Mon Jul 01 12:15:23 2019, Matthew.Pyle at noaa.gov wrote:
> > > Hi,
> > >
> > > I'm trying to sort out some issues I'm having after upgrading to
> > > MET v8.1
> > > (after the v6 build I'd been using disappeared from WCOSS).  I
got
> > > my MET
> > > jobs running okay after pointing at v8.1, and the .stat files
look
> > > okay
> > to
> > > my untrained eye, but the data uploaded to the metviewer server
> > > behaves
> > > oddly.  Namely for probability output, only the lowest threshold
> > > seems to
> > > plot normally.  Looking at thresholds above the lowest level and
> > > just
> > using
> > > output from v8.1 gives this error in the log:
> > >
> > > Failed to create a plot. ROC/Reliability plots must contain data
> > > from at
> > > least one obs_thresh
> > >
> > > A couple of questions floating in my head that you might be able
to
> > answer
> > > for me:
> > >
> > > - Could my data not be getting fully ingested to the metviewer
> > > server?
> > > - Is a database with output from different MET versions
> > > problematic?
> > >
> > > I can keep digging here, but if you can point me in the right
> > > direction
> > I'd
> > > appreciate it!
> > >
> > > Thanks,
> > >
> > > Matt
> >
> >
> >
> >



------------------------------------------------
Subject: troubles after updating to METv8.1
From: "matthew.pyle at noaa.gov"
Time: Tue Jul 02 11:40:39 2019

Having it working going forward is fine with me - unfortunately I
religiously scrub my .stat files (mainly to avoid repeated uploading
of the
same data, but also to control disk usage) so can't prune and reload.

Thanks,

-Matt

On Tue, Jul 2, 2019 at 1:33 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:

> Matt,
> It was a bug in loading PSTD data which we were going to fix in
v2.11.
> I patched the current EMC METviewer instance with the fix and you
should
> be able to load PSTD data now.
> My concern is creating duplicates for other data types when you
reload
> data. We are working on 'removing duplicates' issue but it is not
ready
> yet. Would it be possible if you drop the existing database and load
all
> data to the new one.
>
> Tatiana
>
> On Tue Jul 02 07:16:20 2019, Matthew.Pyle at noaa.gov wrote:
> > Hi Tatiana,
> >
> > I scrub out the files from the staging directory after loading
them
> > into
> > the database.  But I did just upload another day that still is in
my
> > hrefv3
> > directory.  The log file /usr1/metviewer/capture_8.1_load.log from
> > loading
> > this new data shows some errors with PSTD records.
> >
> > Thanks,
> >
> > Matt
> >
> > On Mon, Jul 1, 2019 at 4:34 PM Tatiana Burek via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > Hi Mat,
> > >
> > > METviewer should handle data from all MET versions. There are
some
> > > cases
> > > when the data could not fully be loaded into DB. It could happen
when
> > > the
> > > size of the data files is larger then the limit or the length of
some
> > > string is bigger than expected. Did you get any error message
during
> > > the
> > > loading?
> > >
> > > I looked at your data in mv_hrefv3 database. The latest
> > > fcst_valid_beg
> > > value in line_data_pct table is '2019-05-30 12:00:00'. You don't
have
> > > data
> > > for June. I also could not find .stat files to verify that they
are
> > > valid.
> > > I was looking in /metviewer/staging/mpyle/hrefv3 directory but
it is
> > > empty.
> > >
> > > Tatiana
> > >
> > >
> > > On Mon Jul 01 12:15:23 2019, Matthew.Pyle at noaa.gov wrote:
> > > > Hi,
> > > >
> > > > I'm trying to sort out some issues I'm having after upgrading
to
> > > > MET v8.1
> > > > (after the v6 build I'd been using disappeared from WCOSS).  I
got
> > > > my MET
> > > > jobs running okay after pointing at v8.1, and the .stat files
look
> > > > okay
> > > to
> > > > my untrained eye, but the data uploaded to the metviewer
server
> > > > behaves
> > > > oddly.  Namely for probability output, only the lowest
threshold
> > > > seems to
> > > > plot normally.  Looking at thresholds above the lowest level
and
> > > > just
> > > using
> > > > output from v8.1 gives this error in the log:
> > > >
> > > > Failed to create a plot. ROC/Reliability plots must contain
data
> > > > from at
> > > > least one obs_thresh
> > > >
> > > > A couple of questions floating in my head that you might be
able to
> > > answer
> > > > for me:
> > > >
> > > > - Could my data not be getting fully ingested to the metviewer
> > > > server?
> > > > - Is a database with output from different MET versions
> > > > problematic?
> > > >
> > > > I can keep digging here, but if you can point me in the right
> > > > direction
> > > I'd
> > > > appreciate it!
> > > >
> > > > Thanks,
> > > >
> > > > Matt
> > >
> > >
> > >
> > >
>
>
>
>

------------------------------------------------