[Met_help] [rt.rap.ucar.edu #76857] History for MET Viewer: Scrubbing bad records from a database

[Tatiana Burek via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hi MET Help,

A few days of bad verification records for a particular modeling system
were loaded into a database and I'd like to try and purge those records.  I
would like to do this without removing any records for other models in the
database for the same time range.  Is this possible?

Here is the command I was going to try:

/usr1/metviewer/metviewer/bin/scrubbing.sh /metviewer/bin/scrubbing.sh
/usr1/metviewer/metviewer/sql/scrubbing.sql mv_emc_g2o fcst_valid_beg
'"2016-05-03 19:00:00"' '"2016-05-07 11:00:00"'

But without an option to specify what model this applies to I am concerned
it would wipe out all records over that time window, and we have multiple
models loaded into the database.

Any suggestions/advice that you might be able to offer would be appreciated.

Thanks!
Jacob


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: MET Viewer: Scrubbing bad records from a database
From: Tatiana Burek
Time: Mon Jun 27 08:30:15 2016

Jacob,

You are correct - the scrubbing script excepts and works with only one
field.
It will delete all data with fcst_valid_beg between chosen dates.
Currently, we don't have a script to accommodate you needs.
I can create a custom sql script for you to remove data if needed.
Unfortunately, I still do not have an access to NCEP servers and will
not be able to help you with execution.

Tatiana

On Fri Jun 24 13:35:34 2016, jacob.carley at noaa.gov wrote:
> Hi MET Help,
>
> A few days of bad verification records for a particular modeling
system
> were loaded into a database and I'd like to try and purge those
records.  I
> would like to do this without removing any records for other models
in the
> database for the same time range.  Is this possible?
>
> Here is the command I was going to try:
>
> /usr1/metviewer/metviewer/bin/scrubbing.sh
/metviewer/bin/scrubbing.sh
> /usr1/metviewer/metviewer/sql/scrubbing.sql mv_emc_g2o
fcst_valid_beg
> '"2016-05-03 19:00:00"' '"2016-05-07 11:00:00"'
>
> But without an option to specify what model this applies to I am
concerned
> it would wipe out all records over that time window, and we have
multiple
> models loaded into the database.
>
> Any suggestions/advice that you might be able to offer would be
appreciated.
>
> Thanks!
> Jacob



------------------------------------------------
Subject: MET Viewer: Scrubbing bad records from a database
From: Jacob Carley - NOAA Affiliate
Time: Tue Jun 28 07:54:39 2016

Hi Tatiana,

Thanks for the response.  If it's not too much trouble, a custom
script
would be quite helpful.  We can take care of putting it on the system
and
its subsequent execution.  Hopefully this also helps us (me) learn
more
about sql in the process.

Thanks for your help,
Jacob

On Mon, Jun 27, 2016 at 10:30 AM, Tatiana Burek via RT
<met_help at ucar.edu>
wrote:

> Jacob,
>
> You are correct - the scrubbing script excepts and works with only
one
> field.
> It will delete all data with fcst_valid_beg between chosen dates.
> Currently, we don't have a script to accommodate you needs.
> I can create a custom sql script for you to remove data if needed.
> Unfortunately, I still do not have an access to NCEP servers and
will not
> be able to help you with execution.
>
> Tatiana
>
> On Fri Jun 24 13:35:34 2016, jacob.carley at noaa.gov wrote:
> > Hi MET Help,
> >
> > A few days of bad verification records for a particular modeling
system
> > were loaded into a database and I'd like to try and purge those
> records.  I
> > would like to do this without removing any records for other
models in
> the
> > database for the same time range.  Is this possible?
> >
> > Here is the command I was going to try:
> >
> > /usr1/metviewer/metviewer/bin/scrubbing.sh
/metviewer/bin/scrubbing.sh
> > /usr1/metviewer/metviewer/sql/scrubbing.sql mv_emc_g2o
fcst_valid_beg
> > '"2016-05-03 19:00:00"' '"2016-05-07 11:00:00"'
> >
> > But without an option to specify what model this applies to I am
> concerned
> > it would wipe out all records over that time window, and we have
multiple
> > models loaded into the database.
> >
> > Any suggestions/advice that you might be able to offer would be
> appreciated.
> >
> > Thanks!
> > Jacob
>
>
>
>

------------------------------------------------
Subject: MET Viewer: Scrubbing bad records from a database
From: Tatiana Burek
Time: Tue Jun 28 08:12:33 2016

As I understand from your previous email you need to delete
fcst_valid_beg from "2016-05-03 19:00:00" to "2016-05-07 11:00:00".
Which model do you need to trim?

Tatiana

On Tue Jun 28 07:54:39 2016, jacob.carley at noaa.gov wrote:
> Hi Tatiana,
>
> Thanks for the response.  If it's not too much trouble, a custom
> script
> would be quite helpful.  We can take care of putting it on the
system
> and
> its subsequent execution.  Hopefully this also helps us (me) learn
> more
> about sql in the process.
>
> Thanks for your help,
> Jacob
>
> On Mon, Jun 27, 2016 at 10:30 AM, Tatiana Burek via RT
> <met_help at ucar.edu>
> wrote:
>
> > Jacob,
> >
> > You are correct - the scrubbing script excepts and works with only
> > one
> > field.
> > It will delete all data with fcst_valid_beg between chosen dates.
> > Currently, we don't have a script to accommodate you needs.
> > I can create a custom sql script for you to remove data if needed.
> > Unfortunately, I still do not have an access to NCEP servers and
will
> > not
> > be able to help you with execution.
> >
> > Tatiana
> >
> > On Fri Jun 24 13:35:34 2016, jacob.carley at noaa.gov wrote:
> > > Hi MET Help,
> > >
> > > A few days of bad verification records for a particular modeling
> > > system
> > > were loaded into a database and I'd like to try and purge those
> > records.  I
> > > would like to do this without removing any records for other
models
> > > in
> > the
> > > database for the same time range.  Is this possible?
> > >
> > > Here is the command I was going to try:
> > >
> > > /usr1/metviewer/metviewer/bin/scrubbing.sh
> > > /metviewer/bin/scrubbing.sh
> > > /usr1/metviewer/metviewer/sql/scrubbing.sql mv_emc_g2o
> > > fcst_valid_beg
> > > '"2016-05-03 19:00:00"' '"2016-05-07 11:00:00"'
> > >
> > > But without an option to specify what model this applies to I am
> > concerned
> > > it would wipe out all records over that time window, and we have
> > > multiple
> > > models loaded into the database.
> > >
> > > Any suggestions/advice that you might be able to offer would be
> > appreciated.
> > >
> > > Thanks!
> > > Jacob
> >
> >
> >
> >



------------------------------------------------
Subject: MET Viewer: Scrubbing bad records from a database
From: Jacob Carley - NOAA Affiliate
Time: Tue Jun 28 08:40:31 2016

Hi Tatiana,

It would be output from both 'namrr' and 'conusnestrr'.

Thanks!
Jacob

On Tue, Jun 28, 2016 at 10:12 AM, Tatiana Burek via RT
<met_help at ucar.edu>
wrote:

> As I understand from your previous email you need to delete
fcst_valid_beg
> from "2016-05-03 19:00:00" to "2016-05-07 11:00:00". Which model do
you
> need to trim?
>
> Tatiana
>
> On Tue Jun 28 07:54:39 2016, jacob.carley at noaa.gov wrote:
> > Hi Tatiana,
> >
> > Thanks for the response.  If it's not too much trouble, a custom
> > script
> > would be quite helpful.  We can take care of putting it on the
system
> > and
> > its subsequent execution.  Hopefully this also helps us (me) learn
> > more
> > about sql in the process.
> >
> > Thanks for your help,
> > Jacob
> >
> > On Mon, Jun 27, 2016 at 10:30 AM, Tatiana Burek via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > Jacob,
> > >
> > > You are correct - the scrubbing script excepts and works with
only
> > > one
> > > field.
> > > It will delete all data with fcst_valid_beg between chosen
dates.
> > > Currently, we don't have a script to accommodate you needs.
> > > I can create a custom sql script for you to remove data if
needed.
> > > Unfortunately, I still do not have an access to NCEP servers and
will
> > > not
> > > be able to help you with execution.
> > >
> > > Tatiana
> > >
> > > On Fri Jun 24 13:35:34 2016, jacob.carley at noaa.gov wrote:
> > > > Hi MET Help,
> > > >
> > > > A few days of bad verification records for a particular
modeling
> > > > system
> > > > were loaded into a database and I'd like to try and purge
those
> > > records.  I
> > > > would like to do this without removing any records for other
models
> > > > in
> > > the
> > > > database for the same time range.  Is this possible?
> > > >
> > > > Here is the command I was going to try:
> > > >
> > > > /usr1/metviewer/metviewer/bin/scrubbing.sh
> > > > /metviewer/bin/scrubbing.sh
> > > > /usr1/metviewer/metviewer/sql/scrubbing.sql mv_emc_g2o
> > > > fcst_valid_beg
> > > > '"2016-05-03 19:00:00"' '"2016-05-07 11:00:00"'
> > > >
> > > > But without an option to specify what model this applies to I
am
> > > concerned
> > > > it would wipe out all records over that time window, and we
have
> > > > multiple
> > > > models loaded into the database.
> > > >
> > > > Any suggestions/advice that you might be able to offer would
be
> > > appreciated.
> > > >
> > > > Thanks!
> > > > Jacob
> > >
> > >
> > >
> > >
>
>
>
>

------------------------------------------------
Subject: MET Viewer: Scrubbing bad records from a database
From: Tatiana Burek
Time: Wed Jun 29 14:19:15 2016

Hi Jacob,

Attached SQL script would delete all stat records that have
fcst_valid_beg date between '2016-05-03 19:00:00' and '2016-05-07
11:00:00' inclusively and models 'namrr' and 'conusnestrr'.
Since we do not have you data I could not fully test this script so
please if possible back up your data before running it.

Tatiana

On Tue Jun 28 08:40:31 2016, jacob.carley at noaa.gov wrote:
> Hi Tatiana,
>
> It would be output from both 'namrr' and 'conusnestrr'.
>
> Thanks!
> Jacob
>
> On Tue, Jun 28, 2016 at 10:12 AM, Tatiana Burek via RT
<met_help at ucar.edu>
> wrote:
>
> > As I understand from your previous email you need to delete
fcst_valid_beg
> > from "2016-05-03 19:00:00" to "2016-05-07 11:00:00". Which model
do you
> > need to trim?
> >
> > Tatiana
> >
> > On Tue Jun 28 07:54:39 2016, jacob.carley at noaa.gov wrote:
> > > Hi Tatiana,
> > >
> > > Thanks for the response.  If it's not too much trouble, a custom
> > > script
> > > would be quite helpful.  We can take care of putting it on the
system
> > > and
> > > its subsequent execution.  Hopefully this also helps us (me)
learn
> > > more
> > > about sql in the process.
> > >
> > > Thanks for your help,
> > > Jacob
> > >
> > > On Mon, Jun 27, 2016 at 10:30 AM, Tatiana Burek via RT
> > > <met_help at ucar.edu>
> > > wrote:
> > >
> > > > Jacob,
> > > >
> > > > You are correct - the scrubbing script excepts and works with
only
> > > > one
> > > > field.
> > > > It will delete all data with fcst_valid_beg between chosen
dates.
> > > > Currently, we don't have a script to accommodate you needs.
> > > > I can create a custom sql script for you to remove data if
needed.
> > > > Unfortunately, I still do not have an access to NCEP servers
and will
> > > > not
> > > > be able to help you with execution.
> > > >
> > > > Tatiana
> > > >
> > > > On Fri Jun 24 13:35:34 2016, jacob.carley at noaa.gov wrote:
> > > > > Hi MET Help,
> > > > >
> > > > > A few days of bad verification records for a particular
modeling
> > > > > system
> > > > > were loaded into a database and I'd like to try and purge
those
> > > > records.  I
> > > > > would like to do this without removing any records for other
models
> > > > > in
> > > > the
> > > > > database for the same time range.  Is this possible?
> > > > >
> > > > > Here is the command I was going to try:
> > > > >
> > > > > /usr1/metviewer/metviewer/bin/scrubbing.sh
> > > > > /metviewer/bin/scrubbing.sh
> > > > > /usr1/metviewer/metviewer/sql/scrubbing.sql mv_emc_g2o
> > > > > fcst_valid_beg
> > > > > '"2016-05-03 19:00:00"' '"2016-05-07 11:00:00"'
> > > > >
> > > > > But without an option to specify what model this applies to
I am
> > > > concerned
> > > > > it would wipe out all records over that time window, and we
have
> > > > > multiple
> > > > > models loaded into the database.
> > > > >
> > > > > Any suggestions/advice that you might be able to offer would
be
> > > > appreciated.
> > > > >
> > > > > Thanks!
> > > > > Jacob
> > > >
> > > >
> > > >
> > > >
> >
> >
> >
> >



------------------------------------------------
Subject: MET Viewer: Scrubbing bad records from a database
From: Jacob Carley - NOAA Affiliate
Time: Tue Jul 05 13:19:59 2016

Hi Tatiana,

Great - thanks so much for sharing.

I'll most definitely follow your advice and will back things up before
giving this a try.

Thanks again,
Jacob

On Wed, Jun 29, 2016 at 4:19 PM, Tatiana Burek via RT
<met_help at ucar.edu>
wrote:

> Hi Jacob,
>
> Attached SQL script would delete all stat records that have
fcst_valid_beg
> date between '2016-05-03 19:00:00' and '2016-05-07 11:00:00'
inclusively
> and models 'namrr' and 'conusnestrr'.
> Since we do not have you data I could not fully test this script so
please
> if possible back up your data before running it.
>
> Tatiana
>
> On Tue Jun 28 08:40:31 2016, jacob.carley at noaa.gov wrote:
> > Hi Tatiana,
> >
> > It would be output from both 'namrr' and 'conusnestrr'.
> >
> > Thanks!
> > Jacob
> >
> > On Tue, Jun 28, 2016 at 10:12 AM, Tatiana Burek via RT <
> met_help at ucar.edu>
> > wrote:
> >
> > > As I understand from your previous email you need to delete
> fcst_valid_beg
> > > from "2016-05-03 19:00:00" to "2016-05-07 11:00:00". Which model
do you
> > > need to trim?
> > >
> > > Tatiana
> > >
> > > On Tue Jun 28 07:54:39 2016, jacob.carley at noaa.gov wrote:
> > > > Hi Tatiana,
> > > >
> > > > Thanks for the response.  If it's not too much trouble, a
custom
> > > > script
> > > > would be quite helpful.  We can take care of putting it on the
system
> > > > and
> > > > its subsequent execution.  Hopefully this also helps us (me)
learn
> > > > more
> > > > about sql in the process.
> > > >
> > > > Thanks for your help,
> > > > Jacob
> > > >
> > > > On Mon, Jun 27, 2016 at 10:30 AM, Tatiana Burek via RT
> > > > <met_help at ucar.edu>
> > > > wrote:
> > > >
> > > > > Jacob,
> > > > >
> > > > > You are correct - the scrubbing script excepts and works
with only
> > > > > one
> > > > > field.
> > > > > It will delete all data with fcst_valid_beg between chosen
dates.
> > > > > Currently, we don't have a script to accommodate you needs.
> > > > > I can create a custom sql script for you to remove data if
needed.
> > > > > Unfortunately, I still do not have an access to NCEP servers
and
> will
> > > > > not
> > > > > be able to help you with execution.
> > > > >
> > > > > Tatiana
> > > > >
> > > > > On Fri Jun 24 13:35:34 2016, jacob.carley at noaa.gov wrote:
> > > > > > Hi MET Help,
> > > > > >
> > > > > > A few days of bad verification records for a particular
modeling
> > > > > > system
> > > > > > were loaded into a database and I'd like to try and purge
those
> > > > > records.  I
> > > > > > would like to do this without removing any records for
other
> models
> > > > > > in
> > > > > the
> > > > > > database for the same time range.  Is this possible?
> > > > > >
> > > > > > Here is the command I was going to try:
> > > > > >
> > > > > > /usr1/metviewer/metviewer/bin/scrubbing.sh
> > > > > > /metviewer/bin/scrubbing.sh
> > > > > > /usr1/metviewer/metviewer/sql/scrubbing.sql mv_emc_g2o
> > > > > > fcst_valid_beg
> > > > > > '"2016-05-03 19:00:00"' '"2016-05-07 11:00:00"'
> > > > > >
> > > > > > But without an option to specify what model this applies
to I am
> > > > > concerned
> > > > > > it would wipe out all records over that time window, and
we have
> > > > > > multiple
> > > > > > models loaded into the database.
> > > > > >
> > > > > > Any suggestions/advice that you might be able to offer
would be
> > > > > appreciated.
> > > > > >
> > > > > > Thanks!
> > > > > > Jacob
> > > > >
> > > > >
> > > > >
> > > > >
> > >
> > >
> > >
> > >
>
>
>
>

------------------------------------------------