[Met_help] [rt.rap.ucar.edu #76741] History for MET Viewer Loading: Duplicate entries in table data_file?

[Tatiana Burek via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hi MET help:

I was recently trying to update a database and forgot to purge the data I
had already loaded from my staging directory.  So, in this case the staging
directory had some older data that had already been loaded and some new
data that I did wish to load.  When I ran mv_load.sh I received several
print outs of the following warning:

**  WARNING: file already present in table data_file

Which was not surprising given that I had old data in the staging
directory, but will these data be re-loaded into the database?  Or are the
data skipped?  The system's course of action is not clear from the warning
print.  My concern is having duplicate entries present in the database and
potentially becoming problematic when I use MET Viewer to calculate
statistics.

Thanks!
Jacob


----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: MET Viewer Loading: Duplicate entries in table data_file?
From: Tatiana Burek
Time: Tue Jun 14 09:07:56 2016

Jacob,
METViewer users have two choices how to load duplicated files : ignore
or append. This behavior is controlled by <force_dup_file> element in
the load XML file.

<force_dup_file>false</force_dup_file>
loader checks if the file with the specified name and path already
exists in database and if "yes" it skips it

<force_dup_file>true</force_dup_file>
loader checks if the file with the specified name and path already
exists in database and if "yes" it gets file's unique id and uses it
to load file's content. If the file's content was not changed since
the last loading then you will end up with the duplicated data.

The warning you received indicates that you are loading data with
force_dup_file=true and possibly have duplicates in your database.

Tatiana
On Mon Jun 13 15:22:10 2016, jacob.carley at noaa.gov wrote:
> Hi MET help:
>
> I was recently trying to update a database and forgot to purge the
data I
> had already loaded from my staging directory.  So, in this case the
staging
> directory had some older data that had already been loaded and some
new
> data that I did wish to load.  When I ran mv_load.sh I received
several
> print outs of the following warning:
>
> **  WARNING: file already present in table data_file
>
> Which was not surprising given that I had old data in the staging
> directory, but will these data be re-loaded into the database?  Or
are the
> data skipped?  The system's course of action is not clear from the
warning
> print.  My concern is having duplicate entries present in the
database and
> potentially becoming problematic when I use MET Viewer to calculate
> statistics.
>
> Thanks!
> Jacob



------------------------------------------------
Subject: MET Viewer Loading: Duplicate entries in table data_file?
From: Jacob Carley - NOAA Affiliate
Time: Tue Jun 14 14:18:06 2016

Hi Tatiana,

Thanks for the explanation.

Is there a simple way to scrub duplicates or would it be better to
simply
re-make the database?

Thanks,
Jacob

On Tue, Jun 14, 2016 at 11:07 AM, Tatiana Burek via RT
<met_help at ucar.edu>
wrote:

> Jacob,
> METViewer users have two choices how to load duplicated files :
ignore or
> append. This behavior is controlled by <force_dup_file> element in
the load
> XML file.
>
> <force_dup_file>false</force_dup_file>
> loader checks if the file with the specified name and path already
exists
> in database and if "yes" it skips it
>
> <force_dup_file>true</force_dup_file>
> loader checks if the file with the specified name and path already
exists
> in database and if "yes" it gets file's unique id and uses it to
load
> file's content. If the file's content was not changed since the last
> loading then you will end up with the duplicated data.
>
> The warning you received indicates that you are loading data with
> force_dup_file=true and possibly have duplicates in your database.
>
> Tatiana
> On Mon Jun 13 15:22:10 2016, jacob.carley at noaa.gov wrote:
> > Hi MET help:
> >
> > I was recently trying to update a database and forgot to purge the
data I
> > had already loaded from my staging directory.  So, in this case
the
> staging
> > directory had some older data that had already been loaded and
some new
> > data that I did wish to load.  When I ran mv_load.sh I received
several
> > print outs of the following warning:
> >
> > **  WARNING: file already present in table data_file
> >
> > Which was not surprising given that I had old data in the staging
> > directory, but will these data be re-loaded into the database?  Or
are
> the
> > data skipped?  The system's course of action is not clear from the
> warning
> > print.  My concern is having duplicate entries present in the
database
> and
> > potentially becoming problematic when I use MET Viewer to
calculate
> > statistics.
> >
> > Thanks!
> > Jacob
>
>
>
>

------------------------------------------------
Subject: MET Viewer Loading: Duplicate entries in table data_file?
From: Tatiana Burek
Time: Tue Jun 14 14:28:20 2016

Re-making database (drop, create, load) is the simplest.

Tatiana

On Tue Jun 14 14:18:06 2016, jacob.carley at noaa.gov wrote:
> Hi Tatiana,
>
> Thanks for the explanation.
>
> Is there a simple way to scrub duplicates or would it be better to
> simply
> re-make the database?
>
> Thanks,
> Jacob
>
> On Tue, Jun 14, 2016 at 11:07 AM, Tatiana Burek via RT
> <met_help at ucar.edu>
> wrote:
>
> > Jacob,
> > METViewer users have two choices how to load duplicated files :
> > ignore or
> > append. This behavior is controlled by <force_dup_file> element in
> > the load
> > XML file.
> >
> > <force_dup_file>false</force_dup_file>
> > loader checks if the file with the specified name and path already
> > exists
> > in database and if "yes" it skips it
> >
> > <force_dup_file>true</force_dup_file>
> > loader checks if the file with the specified name and path already
> > exists
> > in database and if "yes" it gets file's unique id and uses it to
load
> > file's content. If the file's content was not changed since the
last
> > loading then you will end up with the duplicated data.
> >
> > The warning you received indicates that you are loading data with
> > force_dup_file=true and possibly have duplicates in your database.
> >
> > Tatiana
> > On Mon Jun 13 15:22:10 2016, jacob.carley at noaa.gov wrote:
> > > Hi MET help:
> > >
> > > I was recently trying to update a database and forgot to purge
the
> > > data I
> > > had already loaded from my staging directory.  So, in this case
the
> > staging
> > > directory had some older data that had already been loaded and
some
> > > new
> > > data that I did wish to load.  When I ran mv_load.sh I received
> > > several
> > > print outs of the following warning:
> > >
> > > **  WARNING: file already present in table data_file
> > >
> > > Which was not surprising given that I had old data in the
staging
> > > directory, but will these data be re-loaded into the database?
Or
> > > are
> > the
> > > data skipped?  The system's course of action is not clear from
the
> > warning
> > > print.  My concern is having duplicate entries present in the
> > > database
> > and
> > > potentially becoming problematic when I use MET Viewer to
calculate
> > > statistics.
> > >
> > > Thanks!
> > > Jacob
> >
> >
> >
> >



------------------------------------------------
Subject: MET Viewer Loading: Duplicate entries in table data_file?
From: Jacob Carley - NOAA Affiliate
Time: Tue Jun 14 14:31:24 2016

Thanks! I am doing that now and it's working fine.

-Jacob

On Tue, Jun 14, 2016 at 4:28 PM, Tatiana Burek via RT
<met_help at ucar.edu>
wrote:

> Re-making database (drop, create, load) is the simplest.
>
> Tatiana
>
> On Tue Jun 14 14:18:06 2016, jacob.carley at noaa.gov wrote:
> > Hi Tatiana,
> >
> > Thanks for the explanation.
> >
> > Is there a simple way to scrub duplicates or would it be better to
> > simply
> > re-make the database?
> >
> > Thanks,
> > Jacob
> >
> > On Tue, Jun 14, 2016 at 11:07 AM, Tatiana Burek via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > Jacob,
> > > METViewer users have two choices how to load duplicated files :
> > > ignore or
> > > append. This behavior is controlled by <force_dup_file> element
in
> > > the load
> > > XML file.
> > >
> > > <force_dup_file>false</force_dup_file>
> > > loader checks if the file with the specified name and path
already
> > > exists
> > > in database and if "yes" it skips it
> > >
> > > <force_dup_file>true</force_dup_file>
> > > loader checks if the file with the specified name and path
already
> > > exists
> > > in database and if "yes" it gets file's unique id and uses it to
load
> > > file's content. If the file's content was not changed since the
last
> > > loading then you will end up with the duplicated data.
> > >
> > > The warning you received indicates that you are loading data
with
> > > force_dup_file=true and possibly have duplicates in your
database.
> > >
> > > Tatiana
> > > On Mon Jun 13 15:22:10 2016, jacob.carley at noaa.gov wrote:
> > > > Hi MET help:
> > > >
> > > > I was recently trying to update a database and forgot to purge
the
> > > > data I
> > > > had already loaded from my staging directory.  So, in this
case the
> > > staging
> > > > directory had some older data that had already been loaded and
some
> > > > new
> > > > data that I did wish to load.  When I ran mv_load.sh I
received
> > > > several
> > > > print outs of the following warning:
> > > >
> > > > **  WARNING: file already present in table data_file
> > > >
> > > > Which was not surprising given that I had old data in the
staging
> > > > directory, but will these data be re-loaded into the database?
Or
> > > > are
> > > the
> > > > data skipped?  The system's course of action is not clear from
the
> > > warning
> > > > print.  My concern is having duplicate entries present in the
> > > > database
> > > and
> > > > potentially becoming problematic when I use MET Viewer to
calculate
> > > > statistics.
> > > >
> > > > Thanks!
> > > > Jacob
> > >
> > >
> > >
> > >
>
>
>
>

------------------------------------------------