[Met_help] [rt.rap.ucar.edu #97687] History for can't copy plots over
Julie Prestopnik via RT
met_help at ucar.edu
Mon Jan 11 09:01:57 MST 2021
----------------------------------------------------------------
Initial Request
----------------------------------------------------------------
I'm trying to copy plots over to Dell from the MET server. I would like to
store my plots in the usual place but there is an scp error which I'm
attaching below.
AND BINARY h.fcst_var = 'PMTF'
AND ld.stat_header_id = h.stat_header_id;
Connection to 205.156.8.85 closed by remote host.
CALLING: scp edward.strobach at 205.156.8.85://data/mv_data//edward.strobach/plots/*
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
You are accessing a U.S. Government information
system, which includes: 1) this computer, 2) this
computer network, 3) all computers connected to this
network, and 4) all devices and storage media attached
to this network or to a computer on this network. You
understand and consent to the following: you may
access this information system for authorized use only;
you have no reasonable expectation of privacy regarding
any communication of data transiting or stored on this
information system; at any time and for any lawful
Government purpose, the Government may monitor,
intercept, and search and seize any communication or
data transiting or stored on this information system; and
any communications or data transiting or stored on this
information system may be disclosed or used for any
lawful Government purpose.
scp: //data/mv_data//edward.strobach/plots/*: No such file or directory
ERROR: Command returned with non-zero status (1): scp edward.strobach@
205.156.8.85://data/mv_data//edward.strobach/plots/*
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
++ printf '\033]0;%s@%s:%s\007' Edward.Strobach v71a1
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
NOTE: nothing about this set-up has changed other than the directory path
onto dell. The path has been created so it should support the transfer of
plots generated on the MET server. The preceding information indicates
that the plot was processed successfully. It's not clear why I can't
transfer my plots over.
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
----------------------------------------------------------------
Complete Ticket History
----------------------------------------------------------------
Subject: can't copy plots over
From: Julie Prestopnik
Time: Tue Dec 01 10:13:13 2020
Hi Edward.
Please consult with Mallory Row and Perry Shafran for assistance with
this
issue. They are most familiar with these capabilities and will be
best
able to assist you.
Thanks!
Julie
On Tue, Dec 1, 2020 at 9:25 AM Edward Strobach - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> Tue Dec 01 09:25:26 2020: Request 97687 was acted upon.
> Transaction: Ticket created by edward.strobach at noaa.gov
> Queue: met_help
> Subject: can't copy plots over
> Owner: Nobody
> Requestors: edward.strobach at noaa.gov
> Status: new
> Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
>
>
> I'm trying to copy plots over to Dell from the MET server. I would
like to
> store my plots in the usual place but there is an scp error which
I'm
> attaching below.
>
>
> AND BINARY h.fcst_var = 'PMTF'
> AND ld.stat_header_id = h.stat_header_id;
>
> Connection to 205.156.8.85 closed by remote host.
> CALLING: scp edward.strobach at 205.156.8.85
> ://data/mv_data//edward.strobach/plots/*
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
>
> You are accessing a U.S. Government information
> system, which includes: 1) this computer, 2) this
> computer network, 3) all computers connected to this
> network, and 4) all devices and storage media attached
> to this network or to a computer on this network. You
> understand and consent to the following: you may
> access this information system for authorized use only;
> you have no reasonable expectation of privacy regarding
> any communication of data transiting or stored on this
> information system; at any time and for any lawful
> Government purpose, the Government may monitor,
> intercept, and search and seize any communication or
> data transiting or stored on this information system; and
> any communications or data transiting or stored on this
> information system may be disclosed or used for any
> lawful Government purpose.
>
> scp: //data/mv_data//edward.strobach/plots/*: No such file or
directory
> ERROR: Command returned with non-zero status (1): scp
edward.strobach@
> 205.156.8.85://data/mv_data//edward.strobach/plots/*
>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> ++ printf '\033]0;%s@%s:%s\007' Edward.Strobach v71a1
> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>
>
> NOTE: nothing about this set-up has changed other than the
directory path
> onto dell. The path has been created so it should support the
transfer of
> plots generated on the MET server. The preceding information
indicates
> that the plot was processed successfully. It's not clear why I
can't
> transfer my plots over.
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
--
Julie Prestopnik (she/her)
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: can't copy plots over
From: Tatiana Burek
Time: Tue Dec 01 10:19:46 2020
Hi Edward.
There were some security changes on the AWS server and what you see
might be the result of this. I will contact AWS administrator about
this.
Tatiana
On Tue Dec 01 10:13:13 2020, jpresto wrote:
> Hi Edward.
>
> Please consult with Mallory Row and Perry Shafran for assistance
with
> this
> issue. They are most familiar with these capabilities and will be
> best
> able to assist you.
>
> Thanks!
>
> Julie
>
> On Tue, Dec 1, 2020 at 9:25 AM Edward Strobach - NOAA Affiliate via
RT
> <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Dec 01 09:25:26 2020: Request 97687 was acted upon.
> > Transaction: Ticket created by edward.strobach at noaa.gov
> > Queue: met_help
> > Subject: can't copy plots over
> > Owner: Nobody
> > Requestors: edward.strobach at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > >
> >
> >
> > I'm trying to copy plots over to Dell from the MET server. I
would
> > like to
> > store my plots in the usual place but there is an scp error which
I'm
> > attaching below.
> >
> >
> > AND BINARY h.fcst_var = 'PMTF'
> > AND ld.stat_header_id = h.stat_header_id;
> >
> > Connection to 205.156.8.85 closed by remote host.
> > CALLING: scp edward.strobach at 205.156.8.85
> > ://data/mv_data//edward.strobach/plots/*
> >
> >
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> >
> > You are accessing a U.S. Government information
> > system, which includes: 1) this computer, 2) this
> > computer network, 3) all computers connected to this
> > network, and 4) all devices and storage media attached
> > to this network or to a computer on this network. You
> > understand and consent to the following: you may
> > access this information system for authorized use only;
> > you have no reasonable expectation of privacy regarding
> > any communication of data transiting or stored on this
> > information system; at any time and for any lawful
> > Government purpose, the Government may monitor,
> > intercept, and search and seize any communication or
> > data transiting or stored on this information system; and
> > any communications or data transiting or stored on this
> > information system may be disclosed or used for any
> > lawful Government purpose.
> >
> > scp: //data/mv_data//edward.strobach/plots/*: No such file or
> > directory
> > ERROR: Command returned with non-zero status (1): scp
> > edward.strobach@
> > 205.156.8.85://data/mv_data//edward.strobach/plots/*
> >
> >
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > ++ printf '\033]0;%s@%s:%s\007' Edward.Strobach v71a1
> > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> >
> >
> > NOTE: nothing about this set-up has changed other than the
directory
> > path
> > onto dell. The path has been created so it should support the
> > transfer of
> > plots generated on the MET server. The preceding information
> > indicates
> > that the plot was processed successfully. It's not clear why I
can't
> > transfer my plots over.
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
------------------------------------------------
Subject: can't copy plots over
From: Edward Strobach - NOAA Affiliate
Time: Tue Dec 01 10:49:24 2020
Perry does not do batch plotting with MET. I was under the impression
that
the MET helpdesk helped with MET related issues.
On Tue, Dec 1, 2020 at 12:13 PM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
> Hi Edward.
>
> Please consult with Mallory Row and Perry Shafran for assistance
with this
> issue. They are most familiar with these capabilities and will be
best
> able to assist you.
>
> Thanks!
>
> Julie
>
> On Tue, Dec 1, 2020 at 9:25 AM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > Tue Dec 01 09:25:26 2020: Request 97687 was acted upon.
> > Transaction: Ticket created by edward.strobach at noaa.gov
> > Queue: met_help
> > Subject: can't copy plots over
> > Owner: Nobody
> > Requestors: edward.strobach at noaa.gov
> > Status: new
> > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
> >
> >
> > I'm trying to copy plots over to Dell from the MET server. I
would like
> to
> > store my plots in the usual place but there is an scp error which
I'm
> > attaching below.
> >
> >
> > AND BINARY h.fcst_var = 'PMTF'
> > AND ld.stat_header_id = h.stat_header_id;
> >
> > Connection to 205.156.8.85 closed by remote host.
> > CALLING: scp edward.strobach at 205.156.8.85
> > ://data/mv_data//edward.strobach/plots/*
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> >
> > You are accessing a U.S. Government information
> > system, which includes: 1) this computer, 2) this
> > computer network, 3) all computers connected to this
> > network, and 4) all devices and storage media attached
> > to this network or to a computer on this network. You
> > understand and consent to the following: you may
> > access this information system for authorized use only;
> > you have no reasonable expectation of privacy regarding
> > any communication of data transiting or stored on this
> > information system; at any time and for any lawful
> > Government purpose, the Government may monitor,
> > intercept, and search and seize any communication or
> > data transiting or stored on this information system; and
> > any communications or data transiting or stored on this
> > information system may be disclosed or used for any
> > lawful Government purpose.
> >
> > scp: //data/mv_data//edward.strobach/plots/*: No such file or
directory
> > ERROR: Command returned with non-zero status (1): scp
edward.strobach@
> > 205.156.8.85://data/mv_data//edward.strobach/plots/*
> >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > ++ printf '\033]0;%s@%s:%s\007' Edward.Strobach v71a1
> > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> >
> >
> > NOTE: nothing about this set-up has changed other than the
directory
> path
> > onto dell. The path has been created so it should support the
transfer
> of
> > plots generated on the MET server. The preceding information
indicates
> > that the plot was processed successfully. It's not clear why I
can't
> > transfer my plots over.
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
> --
> Julie Prestopnik (she/her)
> Software Engineer
> National Center for Atmospheric Research
> Research Applications Laboratory
> Email: jpresto at ucar.edu
>
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: can't copy plots over
From: Julie Prestopnik
Time: Tue Dec 01 10:59:38 2020
Hi Edward.
Are you using the scripts that are located in
/gpfs/hps3/emc/global/noscrub/Mallory.Row/VRFY/METviewer_AWS and
/gpfs/dell2/emc/verification/noscrub/emc.metplus/METviewer_AWS
for interaction between AWS and WCOSS?
Thanks!
Julie
On Tue, Dec 1, 2020 at 10:49 AM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
>
> Perry does not do batch plotting with MET. I was under the
impression that
> the MET helpdesk helped with MET related issues.
>
> On Tue, Dec 1, 2020 at 12:13 PM Julie Prestopnik via RT
<met_help at ucar.edu
> >
> wrote:
>
> > Hi Edward.
> >
> > Please consult with Mallory Row and Perry Shafran for assistance
with
> this
> > issue. They are most familiar with these capabilities and will be
best
> > able to assist you.
> >
> > Thanks!
> >
> > Julie
> >
> > On Tue, Dec 1, 2020 at 9:25 AM Edward Strobach - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > Tue Dec 01 09:25:26 2020: Request 97687 was acted upon.
> > > Transaction: Ticket created by edward.strobach at noaa.gov
> > > Queue: met_help
> > > Subject: can't copy plots over
> > > Owner: Nobody
> > > Requestors: edward.strobach at noaa.gov
> > > Status: new
> > > Ticket <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> >
> > >
> > >
> > > I'm trying to copy plots over to Dell from the MET server. I
would
> like
> > to
> > > store my plots in the usual place but there is an scp error
which I'm
> > > attaching below.
> > >
> > >
> > > AND BINARY h.fcst_var = 'PMTF'
> > > AND ld.stat_header_id = h.stat_header_id;
> > >
> > > Connection to 205.156.8.85 closed by remote host.
> > > CALLING: scp edward.strobach at 205.156.8.85
> > > ://data/mv_data//edward.strobach/plots/*
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > >
> > > You are accessing a U.S. Government information
> > > system, which includes: 1) this computer, 2) this
> > > computer network, 3) all computers connected to this
> > > network, and 4) all devices and storage media attached
> > > to this network or to a computer on this network. You
> > > understand and consent to the following: you may
> > > access this information system for authorized use only;
> > > you have no reasonable expectation of privacy regarding
> > > any communication of data transiting or stored on this
> > > information system; at any time and for any lawful
> > > Government purpose, the Government may monitor,
> > > intercept, and search and seize any communication or
> > > data transiting or stored on this information system; and
> > > any communications or data transiting or stored on this
> > > information system may be disclosed or used for any
> > > lawful Government purpose.
> > >
> > > scp: //data/mv_data//edward.strobach/plots/*: No such file or
directory
> > > ERROR: Command returned with non-zero status (1): scp
edward.strobach@
> > > 205.156.8.85://data/mv_data//edward.strobach/plots/*
> > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > ++ printf '\033]0;%s@%s:%s\007' Edward.Strobach v71a1
> > > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > >
> > >
> > > NOTE: nothing about this set-up has changed other than the
directory
> > path
> > > onto dell. The path has been created so it should support the
transfer
> > of
> > > plots generated on the MET server. The preceding information
indicates
> > > that the plot was processed successfully. It's not clear why I
can't
> > > transfer my plots over.
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> > >
> >
> > --
> > Julie Prestopnik (she/her)
> > Software Engineer
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > Email: jpresto at ucar.edu
> >
> > My working day may not be your working day. Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
--
Julie Prestopnik (she/her)
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: can't copy plots over
From: Edward Strobach - NOAA Affiliate
Time: Tue Dec 01 11:15:08 2020
Hi Julie,
I'm actually using the scripts located here:
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts.
These
were scripts that I either got from Mallory or Perry a while back -
around
May.
On Tue, Dec 1, 2020 at 12:59 PM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
> Hi Edward.
>
> Are you using the scripts that are located in
> /gpfs/hps3/emc/global/noscrub/Mallory.Row/VRFY/METviewer_AWS and
> /gpfs/dell2/emc/verification/noscrub/emc.metplus/METviewer_AWS
> for interaction between AWS and WCOSS?
>
> Thanks!
>
> Julie
>
> On Tue, Dec 1, 2020 at 10:49 AM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
> >
> > Perry does not do batch plotting with MET. I was under the
impression
> that
> > the MET helpdesk helped with MET related issues.
> >
> > On Tue, Dec 1, 2020 at 12:13 PM Julie Prestopnik via RT <
> met_help at ucar.edu
> > >
> > wrote:
> >
> > > Hi Edward.
> > >
> > > Please consult with Mallory Row and Perry Shafran for assistance
with
> > this
> > > issue. They are most familiar with these capabilities and will
be best
> > > able to assist you.
> > >
> > > Thanks!
> > >
> > > Julie
> > >
> > > On Tue, Dec 1, 2020 at 9:25 AM Edward Strobach - NOAA Affiliate
via RT
> <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > Tue Dec 01 09:25:26 2020: Request 97687 was acted upon.
> > > > Transaction: Ticket created by edward.strobach at noaa.gov
> > > > Queue: met_help
> > > > Subject: can't copy plots over
> > > > Owner: Nobody
> > > > Requestors: edward.strobach at noaa.gov
> > > > Status: new
> > > > Ticket <URL:
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > >
> > > >
> > > >
> > > > I'm trying to copy plots over to Dell from the MET server. I
would
> > like
> > > to
> > > > store my plots in the usual place but there is an scp error
which I'm
> > > > attaching below.
> > > >
> > > >
> > > > AND BINARY h.fcst_var = 'PMTF'
> > > > AND ld.stat_header_id = h.stat_header_id;
> > > >
> > > > Connection to 205.156.8.85 closed by remote host.
> > > > CALLING: scp edward.strobach at 205.156.8.85
> > > > ://data/mv_data//edward.strobach/plots/*
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > >
> > > > You are accessing a U.S. Government information
> > > > system, which includes: 1) this computer, 2) this
> > > > computer network, 3) all computers connected to this
> > > > network, and 4) all devices and storage media attached
> > > > to this network or to a computer on this network. You
> > > > understand and consent to the following: you may
> > > > access this information system for authorized use only;
> > > > you have no reasonable expectation of privacy regarding
> > > > any communication of data transiting or stored on this
> > > > information system; at any time and for any lawful
> > > > Government purpose, the Government may monitor,
> > > > intercept, and search and seize any communication or
> > > > data transiting or stored on this information system; and
> > > > any communications or data transiting or stored on this
> > > > information system may be disclosed or used for any
> > > > lawful Government purpose.
> > > >
> > > > scp: //data/mv_data//edward.strobach/plots/*: No such file or
> directory
> > > > ERROR: Command returned with non-zero status (1): scp
> edward.strobach@
> > > > 205.156.8.85://data/mv_data//edward.strobach/plots/*
> > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > > ++ printf '\033]0;%s@%s:%s\007' Edward.Strobach v71a1
> > > > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > > >
> > > >
> > > > NOTE: nothing about this set-up has changed other than the
directory
> > > path
> > > > onto dell. The path has been created so it should support the
> transfer
> > > of
> > > > plots generated on the MET server. The preceding information
> indicates
> > > > that the plot was processed successfully. It's not clear why
I can't
> > > > transfer my plots over.
> > > > --
> > > > Edward Strobach
> > > > EMC/NCEP/NWS/
> > > > IMSG Contractor
> > > > Cubicle#: 2029
> > > > 301-683-3717
> > > >
> > > >
> > >
> > > --
> > > Julie Prestopnik (she/her)
> > > Software Engineer
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > Email: jpresto at ucar.edu
> > >
> > > My working day may not be your working day. Please do not feel
obliged
> > to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
> --
> Julie Prestopnik (she/her)
> Software Engineer
> National Center for Atmospheric Research
> Research Applications Laboratory
> Email: jpresto at ucar.edu
>
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: can't copy plots over
From: Julie Prestopnik
Time: Tue Dec 01 11:22:42 2020
Thanks, Edward. I directed you to Mallory and Perry because I thought
you
might be using scripts from them. We did not develop and have not
used
those scripts. We have no experience interacting with AWS and WCOSS,
so
they seemed best suited to answer your questions related to that
process.
However, as Tatiana pointed out, perhaps the security changes on the
AWS
server have something to do with what you are experiencing. If that
turns
out not to be the case, Mallory and Perry would be best able to help
you.
If they are unable to help, please let us know, and we can see what
they
tried and how we can help you get this problem figured out.
Julie
On Tue, Dec 1, 2020 at 11:15 AM Edward Strobach - NOAA Affiliate via
RT <
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
>
> Hi Julie,
>
> I'm actually using the scripts located here:
> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts.
These
> were scripts that I either got from Mallory or Perry a while back -
around
> May.
>
> On Tue, Dec 1, 2020 at 12:59 PM Julie Prestopnik via RT
<met_help at ucar.edu
> >
> wrote:
>
> > Hi Edward.
> >
> > Are you using the scripts that are located in
> > /gpfs/hps3/emc/global/noscrub/Mallory.Row/VRFY/METviewer_AWS and
> > /gpfs/dell2/emc/verification/noscrub/emc.metplus/METviewer_AWS
> > for interaction between AWS and WCOSS?
> >
> > Thanks!
> >
> > Julie
> >
> > On Tue, Dec 1, 2020 at 10:49 AM Edward Strobach - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
> > >
> > > Perry does not do batch plotting with MET. I was under the
impression
> > that
> > > the MET helpdesk helped with MET related issues.
> > >
> > > On Tue, Dec 1, 2020 at 12:13 PM Julie Prestopnik via RT <
> > met_help at ucar.edu
> > > >
> > > wrote:
> > >
> > > > Hi Edward.
> > > >
> > > > Please consult with Mallory Row and Perry Shafran for
assistance with
> > > this
> > > > issue. They are most familiar with these capabilities and
will be
> best
> > > > able to assist you.
> > > >
> > > > Thanks!
> > > >
> > > > Julie
> > > >
> > > > On Tue, Dec 1, 2020 at 9:25 AM Edward Strobach - NOAA
Affiliate via
> RT
> > <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > Tue Dec 01 09:25:26 2020: Request 97687 was acted upon.
> > > > > Transaction: Ticket created by edward.strobach at noaa.gov
> > > > > Queue: met_help
> > > > > Subject: can't copy plots over
> > > > > Owner: Nobody
> > > > > Requestors: edward.strobach at noaa.gov
> > > > > Status: new
> > > > > Ticket <URL:
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > >
> > > > >
> > > > >
> > > > > I'm trying to copy plots over to Dell from the MET server.
I would
> > > like
> > > > to
> > > > > store my plots in the usual place but there is an scp error
which
> I'm
> > > > > attaching below.
> > > > >
> > > > >
> > > > > AND BINARY h.fcst_var = 'PMTF'
> > > > > AND ld.stat_header_id = h.stat_header_id;
> > > > >
> > > > > Connection to 205.156.8.85 closed by remote host.
> > > > > CALLING: scp edward.strobach at 205.156.8.85
> > > > > ://data/mv_data//edward.strobach/plots/*
> > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > > >
> > > > > You are accessing a U.S. Government information
> > > > > system, which includes: 1) this computer, 2) this
> > > > > computer network, 3) all computers connected to this
> > > > > network, and 4) all devices and storage media attached
> > > > > to this network or to a computer on this network. You
> > > > > understand and consent to the following: you may
> > > > > access this information system for authorized use only;
> > > > > you have no reasonable expectation of privacy regarding
> > > > > any communication of data transiting or stored on this
> > > > > information system; at any time and for any lawful
> > > > > Government purpose, the Government may monitor,
> > > > > intercept, and search and seize any communication or
> > > > > data transiting or stored on this information system; and
> > > > > any communications or data transiting or stored on this
> > > > > information system may be disclosed or used for any
> > > > > lawful Government purpose.
> > > > >
> > > > > scp: //data/mv_data//edward.strobach/plots/*: No such file
or
> > directory
> > > > > ERROR: Command returned with non-zero status (1): scp
> > edward.strobach@
> > > > > 205.156.8.85://data/mv_data//edward.strobach/plots/*
> > > > >
> > > > >
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > > > ++ printf '\033]0;%s@%s:%s\007' Edward.Strobach v71a1
> > > > > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > > > >
> > > > >
> > > > > NOTE: nothing about this set-up has changed other than the
> directory
> > > > path
> > > > > onto dell. The path has been created so it should support
the
> > transfer
> > > > of
> > > > > plots generated on the MET server. The preceding
information
> > indicates
> > > > > that the plot was processed successfully. It's not clear
why I
> can't
> > > > > transfer my plots over.
> > > > > --
> > > > > Edward Strobach
> > > > > EMC/NCEP/NWS/
> > > > > IMSG Contractor
> > > > > Cubicle#: 2029
> > > > > 301-683-3717
> > > > >
> > > > >
> > > >
> > > > --
> > > > Julie Prestopnik (she/her)
> > > > Software Engineer
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > Email: jpresto at ucar.edu
> > > >
> > > > My working day may not be your working day. Please do not
feel
> obliged
> > > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> > >
> >
> > --
> > Julie Prestopnik (she/her)
> > Software Engineer
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > Email: jpresto at ucar.edu
> >
> > My working day may not be your working day. Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
--
Julie Prestopnik (she/her)
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: can't copy plots over
From: Julie Prestopnik
Time: Tue Dec 01 11:35:03 2020
Hi Edward.
After talking with Tatiana, it sounds like I am misinformed and the
scripts
you are using were, in fact, developed by our METviewer team. My
apologies. METplus has a lot of components, and while I thought I had
a
good understanding of who did what, it turns out, I did not. Tatiana
is
working on this issue and will follow up once she knows more. Thank
you
for your patience and understanding.
Julie
On Tue, Dec 1, 2020 at 11:22 AM Julie Prestopnik <jpresto at ucar.edu>
wrote:
> Thanks, Edward. I directed you to Mallory and Perry because I
thought you
> might be using scripts from them. We did not develop and have not
used
> those scripts. We have no experience interacting with AWS and
WCOSS, so
> they seemed best suited to answer your questions related to that
process.
> However, as Tatiana pointed out, perhaps the security changes on the
AWS
> server have something to do with what you are experiencing. If that
turns
> out not to be the case, Mallory and Perry would be best able to help
you.
> If they are unable to help, please let us know, and we can see what
they
> tried and how we can help you get this problem figured out.
>
> Julie
>
> On Tue, Dec 1, 2020 at 11:15 AM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
>>
>> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
>>
>> Hi Julie,
>>
>> I'm actually using the scripts located here:
>> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts.
These
>> were scripts that I either got from Mallory or Perry a while back -
around
>> May.
>>
>> On Tue, Dec 1, 2020 at 12:59 PM Julie Prestopnik via RT <
>> met_help at ucar.edu>
>> wrote:
>>
>> > Hi Edward.
>> >
>> > Are you using the scripts that are located in
>> > /gpfs/hps3/emc/global/noscrub/Mallory.Row/VRFY/METviewer_AWS and
>> > /gpfs/dell2/emc/verification/noscrub/emc.metplus/METviewer_AWS
>> > for interaction between AWS and WCOSS?
>> >
>> > Thanks!
>> >
>> > Julie
>> >
>> > On Tue, Dec 1, 2020 at 10:49 AM Edward Strobach - NOAA Affiliate
via RT
>> <
>> > met_help at ucar.edu> wrote:
>> >
>> > >
>> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
>> > >
>> > > Perry does not do batch plotting with MET. I was under the
impression
>> > that
>> > > the MET helpdesk helped with MET related issues.
>> > >
>> > > On Tue, Dec 1, 2020 at 12:13 PM Julie Prestopnik via RT <
>> > met_help at ucar.edu
>> > > >
>> > > wrote:
>> > >
>> > > > Hi Edward.
>> > > >
>> > > > Please consult with Mallory Row and Perry Shafran for
assistance
>> with
>> > > this
>> > > > issue. They are most familiar with these capabilities and
will be
>> best
>> > > > able to assist you.
>> > > >
>> > > > Thanks!
>> > > >
>> > > > Julie
>> > > >
>> > > > On Tue, Dec 1, 2020 at 9:25 AM Edward Strobach - NOAA
Affiliate via
>> RT
>> > <
>> > > > met_help at ucar.edu> wrote:
>> > > >
>> > > > >
>> > > > > Tue Dec 01 09:25:26 2020: Request 97687 was acted upon.
>> > > > > Transaction: Ticket created by edward.strobach at noaa.gov
>> > > > > Queue: met_help
>> > > > > Subject: can't copy plots over
>> > > > > Owner: Nobody
>> > > > > Requestors: edward.strobach at noaa.gov
>> > > > > Status: new
>> > > > > Ticket <URL:
>> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
>> > > >
>> > > > >
>> > > > >
>> > > > > I'm trying to copy plots over to Dell from the MET server.
I
>> would
>> > > like
>> > > > to
>> > > > > store my plots in the usual place but there is an scp error
which
>> I'm
>> > > > > attaching below.
>> > > > >
>> > > > >
>> > > > > AND BINARY h.fcst_var = 'PMTF'
>> > > > > AND ld.stat_header_id = h.stat_header_id;
>> > > > >
>> > > > > Connection to 205.156.8.85 closed by remote host.
>> > > > > CALLING: scp edward.strobach at 205.156.8.85
>> > > > > ://data/mv_data//edward.strobach/plots/*
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
>> > > > >
>> > > > > You are accessing a U.S. Government information
>> > > > > system, which includes: 1) this computer, 2) this
>> > > > > computer network, 3) all computers connected to this
>> > > > > network, and 4) all devices and storage media attached
>> > > > > to this network or to a computer on this network. You
>> > > > > understand and consent to the following: you may
>> > > > > access this information system for authorized use only;
>> > > > > you have no reasonable expectation of privacy regarding
>> > > > > any communication of data transiting or stored on this
>> > > > > information system; at any time and for any lawful
>> > > > > Government purpose, the Government may monitor,
>> > > > > intercept, and search and seize any communication or
>> > > > > data transiting or stored on this information system; and
>> > > > > any communications or data transiting or stored on this
>> > > > > information system may be disclosed or used for any
>> > > > > lawful Government purpose.
>> > > > >
>> > > > > scp: //data/mv_data//edward.strobach/plots/*: No such file
or
>> > directory
>> > > > > ERROR: Command returned with non-zero status (1): scp
>> > edward.strobach@
>> > > > > 205.156.8.85://data/mv_data//edward.strobach/plots/*
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
>> > > > > ++ printf '\033]0;%s@%s:%s\007' Edward.Strobach v71a1
>> > > > > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>> > > > >
>> > > > >
>> > > > > NOTE: nothing about this set-up has changed other than the
>> directory
>> > > > path
>> > > > > onto dell. The path has been created so it should support
the
>> > transfer
>> > > > of
>> > > > > plots generated on the MET server. The preceding
information
>> > indicates
>> > > > > that the plot was processed successfully. It's not clear
why I
>> can't
>> > > > > transfer my plots over.
>> > > > > --
>> > > > > Edward Strobach
>> > > > > EMC/NCEP/NWS/
>> > > > > IMSG Contractor
>> > > > > Cubicle#: 2029
>> > > > > 301-683-3717
>> > > > >
>> > > > >
>> > > >
>> > > > --
>> > > > Julie Prestopnik (she/her)
>> > > > Software Engineer
>> > > > National Center for Atmospheric Research
>> > > > Research Applications Laboratory
>> > > > Email: jpresto at ucar.edu
>> > > >
>> > > > My working day may not be your working day. Please do not
feel
>> obliged
>> > > to
>> > > > reply to this email outside of your normal working hours.
>> > > >
>> > > >
>> > >
>> > > --
>> > > Edward Strobach
>> > > EMC/NCEP/NWS/
>> > > IMSG Contractor
>> > > Cubicle#: 2029
>> > > 301-683-3717
>> > >
>> > >
>> >
>> > --
>> > Julie Prestopnik (she/her)
>> > Software Engineer
>> > National Center for Atmospheric Research
>> > Research Applications Laboratory
>> > Email: jpresto at ucar.edu
>> >
>> > My working day may not be your working day. Please do not feel
obliged
>> to
>> > reply to this email outside of your normal working hours.
>> >
>> >
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
>>
>
> --
> Julie Prestopnik (she/her)
> Software Engineer
> National Center for Atmospheric Research
> Research Applications Laboratory
> Email: jpresto at ucar.edu
>
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
--
Julie Prestopnik (she/her)
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: can't copy plots over
From: Tatiana Burek
Time: Tue Dec 01 12:44:15 2020
Edward
I run a few butch jobs from my laptop and made sure that METviewer
code and scripts working as expected: I was able to produce and copy
plots.
I also got an error like the one that you reported but only when
METviewer did not generate any plots and the script couldn't copy
empty directory.
During my testing I noticed a few batch jobs that where started by
you and were handing. Each of these jobs started a database query and
they also were piling up and handing. They all were waiting for the
indexing that takes place after the data loading.
I had to kill all database and batch processes in order to move
forward with the testing.
To avoid this gridlock you could:
- in your loading xml use <drop_indexes>false</drop_indexes>
<apply_indexes>false</apply_indexes>
- do not start a few parallel batch jobs
I will add a info message to the script that would explain why the
error occurs.
For now, could you test batch with some simple XML that produce at
least one image.
Tatiana
On Tue Dec 01 11:35:03 2020, jpresto wrote:
> Hi Edward.
>
> After talking with Tatiana, it sounds like I am misinformed and the
> scripts
> you are using were, in fact, developed by our METviewer team. My
> apologies. METplus has a lot of components, and while I thought I
had
> a
> good understanding of who did what, it turns out, I did not.
Tatiana
> is
> working on this issue and will follow up once she knows more. Thank
> you
> for your patience and understanding.
>
> Julie
>
> On Tue, Dec 1, 2020 at 11:22 AM Julie Prestopnik <jpresto at ucar.edu>
> wrote:
>
> > Thanks, Edward. I directed you to Mallory and Perry because I
> > thought you
> > might be using scripts from them. We did not develop and have not
> > used
> > those scripts. We have no experience interacting with AWS and
WCOSS,
> > so
> > they seemed best suited to answer your questions related to that
> > process.
> > However, as Tatiana pointed out, perhaps the security changes on
the
> > AWS
> > server have something to do with what you are experiencing. If
that
> > turns
> > out not to be the case, Mallory and Perry would be best able to
help
> > you.
> > If they are unable to help, please let us know, and we can see
what
> > they
> > tried and how we can help you get this problem figured out.
> >
> > Julie
> >
> > On Tue, Dec 1, 2020 at 11:15 AM Edward Strobach - NOAA Affiliate
via
> > RT <
> > met_help at ucar.edu> wrote:
> >
> >>
> >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
> >>
> >> Hi Julie,
> >>
> >> I'm actually using the scripts located here:
> >> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts.
> >> These
> >> were scripts that I either got from Mallory or Perry a while back
-
> >> around
> >> May.
> >>
> >> On Tue, Dec 1, 2020 at 12:59 PM Julie Prestopnik via RT <
> >> met_help at ucar.edu>
> >> wrote:
> >>
> >> > Hi Edward.
> >> >
> >> > Are you using the scripts that are located in
> >> > /gpfs/hps3/emc/global/noscrub/Mallory.Row/VRFY/METviewer_AWS
and
> >> > /gpfs/dell2/emc/verification/noscrub/emc.metplus/METviewer_AWS
> >> > for interaction between AWS and WCOSS?
> >> >
> >> > Thanks!
> >> >
> >> > Julie
> >> >
> >> > On Tue, Dec 1, 2020 at 10:49 AM Edward Strobach - NOAA
Affiliate
> >> > via RT
> >> <
> >> > met_help at ucar.edu> wrote:
> >> >
> >> > >
> >> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
>
> >> > >
> >> > > Perry does not do batch plotting with MET. I was under the
> >> > > impression
> >> > that
> >> > > the MET helpdesk helped with MET related issues.
> >> > >
> >> > > On Tue, Dec 1, 2020 at 12:13 PM Julie Prestopnik via RT <
> >> > met_help at ucar.edu
> >> > > >
> >> > > wrote:
> >> > >
> >> > > > Hi Edward.
> >> > > >
> >> > > > Please consult with Mallory Row and Perry Shafran for
> >> > > > assistance
> >> with
> >> > > this
> >> > > > issue. They are most familiar with these capabilities and
> >> > > > will be
> >> best
> >> > > > able to assist you.
> >> > > >
> >> > > > Thanks!
> >> > > >
> >> > > > Julie
> >> > > >
> >> > > > On Tue, Dec 1, 2020 at 9:25 AM Edward Strobach - NOAA
> >> > > > Affiliate via
> >> RT
> >> > <
> >> > > > met_help at ucar.edu> wrote:
> >> > > >
> >> > > > >
> >> > > > > Tue Dec 01 09:25:26 2020: Request 97687 was acted upon.
> >> > > > > Transaction: Ticket created by edward.strobach at noaa.gov
> >> > > > > Queue: met_help
> >> > > > > Subject: can't copy plots over
> >> > > > > Owner: Nobody
> >> > > > > Requestors: edward.strobach at noaa.gov
> >> > > > > Status: new
> >> > > > > Ticket <URL:
> >> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> >> > > >
> >> > > > >
> >> > > > >
> >> > > > > I'm trying to copy plots over to Dell from the MET
server.
> >> > > > > I
> >> would
> >> > > like
> >> > > > to
> >> > > > > store my plots in the usual place but there is an scp
error
> >> > > > > which
> >> I'm
> >> > > > > attaching below.
> >> > > > >
> >> > > > >
> >> > > > > AND BINARY h.fcst_var = 'PMTF'
> >> > > > > AND ld.stat_header_id = h.stat_header_id;
> >> > > > >
> >> > > > > Connection to 205.156.8.85 closed by remote host.
> >> > > > > CALLING: scp edward.strobach at 205.156.8.85
> >> > > > > ://data/mv_data//edward.strobach/plots/*
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> >> > > > >
> >> > > > > You are accessing a U.S. Government information
> >> > > > > system, which includes: 1) this computer, 2) this
> >> > > > > computer network, 3) all computers connected to this
> >> > > > > network, and 4) all devices and storage media attached
> >> > > > > to this network or to a computer on this network. You
> >> > > > > understand and consent to the following: you may
> >> > > > > access this information system for authorized use only;
> >> > > > > you have no reasonable expectation of privacy regarding
> >> > > > > any communication of data transiting or stored on this
> >> > > > > information system; at any time and for any lawful
> >> > > > > Government purpose, the Government may monitor,
> >> > > > > intercept, and search and seize any communication or
> >> > > > > data transiting or stored on this information system; and
> >> > > > > any communications or data transiting or stored on this
> >> > > > > information system may be disclosed or used for any
> >> > > > > lawful Government purpose.
> >> > > > >
> >> > > > > scp: //data/mv_data//edward.strobach/plots/*: No such
file
> >> > > > > or
> >> > directory
> >> > > > > ERROR: Command returned with non-zero status (1): scp
> >> > edward.strobach@
> >> > > > > 205.156.8.85://data/mv_data//edward.strobach/plots/*
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> >> > > > > ++ printf '\033]0;%s@%s:%s\007' Edward.Strobach v71a1
> >> > > > > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> >> > > > >
> >> > > > >
> >> > > > > NOTE: nothing about this set-up has changed other than
the
> >> directory
> >> > > > path
> >> > > > > onto dell. The path has been created so it should
support
> >> > > > > the
> >> > transfer
> >> > > > of
> >> > > > > plots generated on the MET server. The preceding
> >> > > > > information
> >> > indicates
> >> > > > > that the plot was processed successfully. It's not clear
> >> > > > > why I
> >> can't
> >> > > > > transfer my plots over.
> >> > > > > --
> >> > > > > Edward Strobach
> >> > > > > EMC/NCEP/NWS/
> >> > > > > IMSG Contractor
> >> > > > > Cubicle#: 2029
> >> > > > > 301-683-3717
> >> > > > >
> >> > > > >
> >> > > >
> >> > > > --
> >> > > > Julie Prestopnik (she/her)
> >> > > > Software Engineer
> >> > > > National Center for Atmospheric Research
> >> > > > Research Applications Laboratory
> >> > > > Email: jpresto at ucar.edu
> >> > > >
> >> > > > My working day may not be your working day. Please do not
> >> > > > feel
> >> obliged
> >> > > to
> >> > > > reply to this email outside of your normal working hours.
> >> > > >
> >> > > >
> >> > >
> >> > > --
> >> > > Edward Strobach
> >> > > EMC/NCEP/NWS/
> >> > > IMSG Contractor
> >> > > Cubicle#: 2029
> >> > > 301-683-3717
> >> > >
> >> > >
> >> >
> >> > --
> >> > Julie Prestopnik (she/her)
> >> > Software Engineer
> >> > National Center for Atmospheric Research
> >> > Research Applications Laboratory
> >> > Email: jpresto at ucar.edu
> >> >
> >> > My working day may not be your working day. Please do not feel
> >> > obliged
> >> to
> >> > reply to this email outside of your normal working hours.
> >> >
> >> >
> >>
> >> --
> >> Edward Strobach
> >> EMC/NCEP/NWS/
> >> IMSG Contractor
> >> Cubicle#: 2029
> >> 301-683-3717
> >>
> >>
> >
> > --
> > Julie Prestopnik (she/her)
> > Software Engineer
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > Email: jpresto at ucar.edu
> >
> > My working day may not be your working day. Please do not feel
> > obliged to
> > reply to this email outside of your normal working hours.
> >
------------------------------------------------
Subject: can't copy plots over
From: Edward Strobach - NOAA Affiliate
Time: Tue Dec 01 13:00:15 2020
Thanks. I just made those changes in the xml file, so I 'guess' I'm
ready
to go.
I should note that I have several steps that I run each day. I
convert
files into netcdf files that can be read by MET, I generate the stat
files,
and I use the stat files in the MET batch plotting system. I have
actually
5 total steps, but the plot batch system is really the only one that
gives
me trouble on a regular basis. I wish I knew how to reach a point
where I
have a plotting package that would produce the plots that I need each
day.
I'm expected to hand the final product off in the near future, but
this
part has delayed me extensively. I know that I'm generating a lot of
plots
which could potentially overwhelm the system, but I'm unaware how
others
work to circumvent that. I hope to move past this soon so I can
continue
development work.
There are several scripts that I don't use that might be helpful. Do
any
of these work to help me manage the system better?
check_my_db.sh mv_batch_on_aws.sh mv_create_db_on_aws.sh
mv_db_size_on_aws.sh mv_delete_db_on_aws.sh mv_load_to_aws.sh
On Tue, Dec 1, 2020 at 2:44 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> Edward
>
> I run a few butch jobs from my laptop and made sure that METviewer
code
> and scripts working as expected: I was able to produce and copy
plots.
> I also got an error like the one that you reported but only when
METviewer
> did not generate any plots and the script couldn't copy empty
directory.
>
> During my testing I noticed a few batch jobs that where started by
you
> and were handing. Each of these jobs started a database query and
they also
> were piling up and handing. They all were waiting for the indexing
that
> takes place after the data loading.
> I had to kill all database and batch processes in order to move
forward
> with the testing.
> To avoid this gridlock you could:
> - in your loading xml use <drop_indexes>false</drop_indexes>
> <apply_indexes>false</apply_indexes>
>
>
> - do not start a few parallel batch jobs
>
> I will add a info message to the script that would explain why the
error
> occurs.
> For now, could you test batch with some simple XML that produce at
least
> one image.
>
> Tatiana
>
>
>
>
>
>
>
> On Tue Dec 01 11:35:03 2020, jpresto wrote:
> > Hi Edward.
> >
> > After talking with Tatiana, it sounds like I am misinformed and
the
> > scripts
> > you are using were, in fact, developed by our METviewer team. My
> > apologies. METplus has a lot of components, and while I thought I
had
> > a
> > good understanding of who did what, it turns out, I did not.
Tatiana
> > is
> > working on this issue and will follow up once she knows more.
Thank
> > you
> > for your patience and understanding.
> >
> > Julie
> >
> > On Tue, Dec 1, 2020 at 11:22 AM Julie Prestopnik
<jpresto at ucar.edu>
> > wrote:
> >
> > > Thanks, Edward. I directed you to Mallory and Perry because I
> > > thought you
> > > might be using scripts from them. We did not develop and have
not
> > > used
> > > those scripts. We have no experience interacting with AWS and
WCOSS,
> > > so
> > > they seemed best suited to answer your questions related to that
> > > process.
> > > However, as Tatiana pointed out, perhaps the security changes on
the
> > > AWS
> > > server have something to do with what you are experiencing. If
that
> > > turns
> > > out not to be the case, Mallory and Perry would be best able to
help
> > > you.
> > > If they are unable to help, please let us know, and we can see
what
> > > they
> > > tried and how we can help you get this problem figured out.
> > >
> > > Julie
> > >
> > > On Tue, Dec 1, 2020 at 11:15 AM Edward Strobach - NOAA Affiliate
via
> > > RT <
> > > met_help at ucar.edu> wrote:
> > >
> > >>
> > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
> > >>
> > >> Hi Julie,
> > >>
> > >> I'm actually using the scripts located here:
> > >>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts.
> > >> These
> > >> were scripts that I either got from Mallory or Perry a while
back -
> > >> around
> > >> May.
> > >>
> > >> On Tue, Dec 1, 2020 at 12:59 PM Julie Prestopnik via RT <
> > >> met_help at ucar.edu>
> > >> wrote:
> > >>
> > >> > Hi Edward.
> > >> >
> > >> > Are you using the scripts that are located in
> > >> > /gpfs/hps3/emc/global/noscrub/Mallory.Row/VRFY/METviewer_AWS
and
> > >> >
/gpfs/dell2/emc/verification/noscrub/emc.metplus/METviewer_AWS
> > >> > for interaction between AWS and WCOSS?
> > >> >
> > >> > Thanks!
> > >> >
> > >> > Julie
> > >> >
> > >> > On Tue, Dec 1, 2020 at 10:49 AM Edward Strobach - NOAA
Affiliate
> > >> > via RT
> > >> <
> > >> > met_help at ucar.edu> wrote:
> > >> >
> > >> > >
> > >> > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
> > >> > >
> > >> > > Perry does not do batch plotting with MET. I was under the
> > >> > > impression
> > >> > that
> > >> > > the MET helpdesk helped with MET related issues.
> > >> > >
> > >> > > On Tue, Dec 1, 2020 at 12:13 PM Julie Prestopnik via RT <
> > >> > met_help at ucar.edu
> > >> > > >
> > >> > > wrote:
> > >> > >
> > >> > > > Hi Edward.
> > >> > > >
> > >> > > > Please consult with Mallory Row and Perry Shafran for
> > >> > > > assistance
> > >> with
> > >> > > this
> > >> > > > issue. They are most familiar with these capabilities
and
> > >> > > > will be
> > >> best
> > >> > > > able to assist you.
> > >> > > >
> > >> > > > Thanks!
> > >> > > >
> > >> > > > Julie
> > >> > > >
> > >> > > > On Tue, Dec 1, 2020 at 9:25 AM Edward Strobach - NOAA
> > >> > > > Affiliate via
> > >> RT
> > >> > <
> > >> > > > met_help at ucar.edu> wrote:
> > >> > > >
> > >> > > > >
> > >> > > > > Tue Dec 01 09:25:26 2020: Request 97687 was acted upon.
> > >> > > > > Transaction: Ticket created by edward.strobach at noaa.gov
> > >> > > > > Queue: met_help
> > >> > > > > Subject: can't copy plots over
> > >> > > > > Owner: Nobody
> > >> > > > > Requestors: edward.strobach at noaa.gov
> > >> > > > > Status: new
> > >> > > > > Ticket <URL:
> > >> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > >> > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > I'm trying to copy plots over to Dell from the MET
server.
> > >> > > > > I
> > >> would
> > >> > > like
> > >> > > > to
> > >> > > > > store my plots in the usual place but there is an scp
error
> > >> > > > > which
> > >> I'm
> > >> > > > > attaching below.
> > >> > > > >
> > >> > > > >
> > >> > > > > AND BINARY h.fcst_var = 'PMTF'
> > >> > > > > AND ld.stat_header_id = h.stat_header_id;
> > >> > > > >
> > >> > > > > Connection to 205.156.8.85 closed by remote host.
> > >> > > > > CALLING: scp edward.strobach at 205.156.8.85
> > >> > > > > ://data/mv_data//edward.strobach/plots/*
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > >> > > > >
> > >> > > > > You are accessing a U.S. Government information
> > >> > > > > system, which includes: 1) this computer, 2) this
> > >> > > > > computer network, 3) all computers connected to this
> > >> > > > > network, and 4) all devices and storage media attached
> > >> > > > > to this network or to a computer on this network. You
> > >> > > > > understand and consent to the following: you may
> > >> > > > > access this information system for authorized use only;
> > >> > > > > you have no reasonable expectation of privacy regarding
> > >> > > > > any communication of data transiting or stored on this
> > >> > > > > information system; at any time and for any lawful
> > >> > > > > Government purpose, the Government may monitor,
> > >> > > > > intercept, and search and seize any communication or
> > >> > > > > data transiting or stored on this information system;
and
> > >> > > > > any communications or data transiting or stored on this
> > >> > > > > information system may be disclosed or used for any
> > >> > > > > lawful Government purpose.
> > >> > > > >
> > >> > > > > scp: //data/mv_data//edward.strobach/plots/*: No such
file
> > >> > > > > or
> > >> > directory
> > >> > > > > ERROR: Command returned with non-zero status (1): scp
> > >> > edward.strobach@
> > >> > > > > 205.156.8.85://data/mv_data//edward.strobach/plots/*
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > >> > > > > ++ printf '\033]0;%s@%s:%s\007' Edward.Strobach v71a1
> > >> > > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > >> > > > >
> > >> > > > >
> > >> > > > > NOTE: nothing about this set-up has changed other than
the
> > >> directory
> > >> > > > path
> > >> > > > > onto dell. The path has been created so it should
support
> > >> > > > > the
> > >> > transfer
> > >> > > > of
> > >> > > > > plots generated on the MET server. The preceding
> > >> > > > > information
> > >> > indicates
> > >> > > > > that the plot was processed successfully. It's not
clear
> > >> > > > > why I
> > >> can't
> > >> > > > > transfer my plots over.
> > >> > > > > --
> > >> > > > > Edward Strobach
> > >> > > > > EMC/NCEP/NWS/
> > >> > > > > IMSG Contractor
> > >> > > > > Cubicle#: 2029
> > >> > > > > 301-683-3717
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > > > --
> > >> > > > Julie Prestopnik (she/her)
> > >> > > > Software Engineer
> > >> > > > National Center for Atmospheric Research
> > >> > > > Research Applications Laboratory
> > >> > > > Email: jpresto at ucar.edu
> > >> > > >
> > >> > > > My working day may not be your working day. Please do
not
> > >> > > > feel
> > >> obliged
> > >> > > to
> > >> > > > reply to this email outside of your normal working hours.
> > >> > > >
> > >> > > >
> > >> > >
> > >> > > --
> > >> > > Edward Strobach
> > >> > > EMC/NCEP/NWS/
> > >> > > IMSG Contractor
> > >> > > Cubicle#: 2029
> > >> > > 301-683-3717
> > >> > >
> > >> > >
> > >> >
> > >> > --
> > >> > Julie Prestopnik (she/her)
> > >> > Software Engineer
> > >> > National Center for Atmospheric Research
> > >> > Research Applications Laboratory
> > >> > Email: jpresto at ucar.edu
> > >> >
> > >> > My working day may not be your working day. Please do not
feel
> > >> > obliged
> > >> to
> > >> > reply to this email outside of your normal working hours.
> > >> >
> > >> >
> > >>
> > >> --
> > >> Edward Strobach
> > >> EMC/NCEP/NWS/
> > >> IMSG Contractor
> > >> Cubicle#: 2029
> > >> 301-683-3717
> > >>
> > >>
> > >
> > > --
> > > Julie Prestopnik (she/her)
> > > Software Engineer
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > Email: jpresto at ucar.edu
> > >
> > > My working day may not be your working day. Please do not feel
> > > obliged to
> > > reply to this email outside of your normal working hours.
> > >
>
>
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: can't copy plots over
From: Tatiana Burek
Time: Tue Dec 01 14:03:06 2020
For your process you need only two scripts:
mv_load_to_aws.sh
mv_batch_on_aws.sh
I don't think, that others would help you.
The main work is in creating efficient XMLs for the each step.
Like not recalculating indexes or performing stat_header_db_check
during data loading.
Tatiana
On Tue Dec 01 13:00:15 2020, edward.strobach at noaa.gov wrote:
> Thanks. I just made those changes in the xml file, so I 'guess' I'm
> ready
> to go.
>
> I should note that I have several steps that I run each day. I
> convert
> files into netcdf files that can be read by MET, I generate the stat
> files,
> and I use the stat files in the MET batch plotting system. I have
> actually
> 5 total steps, but the plot batch system is really the only one that
> gives
> me trouble on a regular basis. I wish I knew how to reach a point
> where I
> have a plotting package that would produce the plots that I need
each
> day.
> I'm expected to hand the final product off in the near future, but
> this
> part has delayed me extensively. I know that I'm generating a lot of
> plots
> which could potentially overwhelm the system, but I'm unaware how
> others
> work to circumvent that. I hope to move past this soon so I can
> continue
> development work.
>
> There are several scripts that I don't use that might be helpful.
Do
> any
> of these work to help me manage the system better?
> check_my_db.sh mv_batch_on_aws.sh mv_create_db_on_aws.sh
> mv_db_size_on_aws.sh mv_delete_db_on_aws.sh mv_load_to_aws.sh
>
>
> On Tue, Dec 1, 2020 at 2:44 PM Tatiana Burek via RT
> <met_help at ucar.edu>
> wrote:
>
> > Edward
> >
> > I run a few butch jobs from my laptop and made sure that METviewer
> > code
> > and scripts working as expected: I was able to produce and copy
> > plots.
> > I also got an error like the one that you reported but only when
> > METviewer
> > did not generate any plots and the script couldn't copy empty
> > directory.
> >
> > During my testing I noticed a few batch jobs that where started
by
> > you
> > and were handing. Each of these jobs started a database query and
> > they also
> > were piling up and handing. They all were waiting for the indexing
> > that
> > takes place after the data loading.
> > I had to kill all database and batch processes in order to move
> > forward
> > with the testing.
> > To avoid this gridlock you could:
> > - in your loading xml use <drop_indexes>false</drop_indexes>
> > <apply_indexes>false</apply_indexes>
> >
> >
> > - do not start a few parallel batch jobs
> >
> > I will add a info message to the script that would explain why the
> > error
> > occurs.
> > For now, could you test batch with some simple XML that produce at
> > least
> > one image.
> >
> > Tatiana
> >
> >
> >
> >
> >
> >
> >
> > On Tue Dec 01 11:35:03 2020, jpresto wrote:
> > > Hi Edward.
> > >
> > > After talking with Tatiana, it sounds like I am misinformed and
the
> > > scripts
> > > you are using were, in fact, developed by our METviewer team.
My
> > > apologies. METplus has a lot of components, and while I thought
I
> > > had
> > > a
> > > good understanding of who did what, it turns out, I did not.
> > > Tatiana
> > > is
> > > working on this issue and will follow up once she knows more.
> > > Thank
> > > you
> > > for your patience and understanding.
> > >
> > > Julie
> > >
> > > On Tue, Dec 1, 2020 at 11:22 AM Julie Prestopnik
<jpresto at ucar.edu>
> > > wrote:
> > >
> > > > Thanks, Edward. I directed you to Mallory and Perry because I
> > > > thought you
> > > > might be using scripts from them. We did not develop and have
> > > > not
> > > > used
> > > > those scripts. We have no experience interacting with AWS and
> > > > WCOSS,
> > > > so
> > > > they seemed best suited to answer your questions related to
that
> > > > process.
> > > > However, as Tatiana pointed out, perhaps the security changes
on
> > > > the
> > > > AWS
> > > > server have something to do with what you are experiencing.
If
> > > > that
> > > > turns
> > > > out not to be the case, Mallory and Perry would be best able
to
> > > > help
> > > > you.
> > > > If they are unable to help, please let us know, and we can see
> > > > what
> > > > they
> > > > tried and how we can help you get this problem figured out.
> > > >
> > > > Julie
> > > >
> > > > On Tue, Dec 1, 2020 at 11:15 AM Edward Strobach - NOAA
Affiliate
> > > > via
> > > > RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > >>
> > > >> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
>
> > > >>
> > > >> Hi Julie,
> > > >>
> > > >> I'm actually using the scripts located here:
> > > >>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts.
> > > >> These
> > > >> were scripts that I either got from Mallory or Perry a while
> > > >> back -
> > > >> around
> > > >> May.
> > > >>
> > > >> On Tue, Dec 1, 2020 at 12:59 PM Julie Prestopnik via RT <
> > > >> met_help at ucar.edu>
> > > >> wrote:
> > > >>
> > > >> > Hi Edward.
> > > >> >
> > > >> > Are you using the scripts that are located in
> > > >> >
/gpfs/hps3/emc/global/noscrub/Mallory.Row/VRFY/METviewer_AWS
> > > >> > and
> > > >> >
/gpfs/dell2/emc/verification/noscrub/emc.metplus/METviewer_AWS
> > > >> > for interaction between AWS and WCOSS?
> > > >> >
> > > >> > Thanks!
> > > >> >
> > > >> > Julie
> > > >> >
> > > >> > On Tue, Dec 1, 2020 at 10:49 AM Edward Strobach - NOAA
> > > >> > Affiliate
> > > >> > via RT
> > > >> <
> > > >> > met_help at ucar.edu> wrote:
> > > >> >
> > > >> > >
> > > >> > > <URL:
> > > >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
> > > >> > >
> > > >> > > Perry does not do batch plotting with MET. I was under
the
> > > >> > > impression
> > > >> > that
> > > >> > > the MET helpdesk helped with MET related issues.
> > > >> > >
> > > >> > > On Tue, Dec 1, 2020 at 12:13 PM Julie Prestopnik via RT <
> > > >> > met_help at ucar.edu
> > > >> > > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Hi Edward.
> > > >> > > >
> > > >> > > > Please consult with Mallory Row and Perry Shafran for
> > > >> > > > assistance
> > > >> with
> > > >> > > this
> > > >> > > > issue. They are most familiar with these capabilities
and
> > > >> > > > will be
> > > >> best
> > > >> > > > able to assist you.
> > > >> > > >
> > > >> > > > Thanks!
> > > >> > > >
> > > >> > > > Julie
> > > >> > > >
> > > >> > > > On Tue, Dec 1, 2020 at 9:25 AM Edward Strobach - NOAA
> > > >> > > > Affiliate via
> > > >> RT
> > > >> > <
> > > >> > > > met_help at ucar.edu> wrote:
> > > >> > > >
> > > >> > > > >
> > > >> > > > > Tue Dec 01 09:25:26 2020: Request 97687 was acted
upon.
> > > >> > > > > Transaction: Ticket created by
edward.strobach at noaa.gov
> > > >> > > > > Queue: met_help
> > > >> > > > > Subject: can't copy plots over
> > > >> > > > > Owner: Nobody
> > > >> > > > > Requestors: edward.strobach at noaa.gov
> > > >> > > > > Status: new
> > > >> > > > > Ticket <URL:
> > > >> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > >> > > >
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > I'm trying to copy plots over to Dell from the MET
> > > >> > > > > server.
> > > >> > > > > I
> > > >> would
> > > >> > > like
> > > >> > > > to
> > > >> > > > > store my plots in the usual place but there is an scp
> > > >> > > > > error
> > > >> > > > > which
> > > >> I'm
> > > >> > > > > attaching below.
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > AND BINARY h.fcst_var = 'PMTF'
> > > >> > > > > AND ld.stat_header_id = h.stat_header_id;
> > > >> > > > >
> > > >> > > > > Connection to 205.156.8.85 closed by remote host.
> > > >> > > > > CALLING: scp edward.strobach at 205.156.8.85
> > > >> > > > > ://data/mv_data//edward.strobach/plots/*
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> >
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > >> > > > >
> > > >> > > > > You are accessing a U.S. Government information
> > > >> > > > > system, which includes: 1) this computer, 2) this
> > > >> > > > > computer network, 3) all computers connected to this
> > > >> > > > > network, and 4) all devices and storage media
attached
> > > >> > > > > to this network or to a computer on this network. You
> > > >> > > > > understand and consent to the following: you may
> > > >> > > > > access this information system for authorized use
only;
> > > >> > > > > you have no reasonable expectation of privacy
regarding
> > > >> > > > > any communication of data transiting or stored on
this
> > > >> > > > > information system; at any time and for any lawful
> > > >> > > > > Government purpose, the Government may monitor,
> > > >> > > > > intercept, and search and seize any communication or
> > > >> > > > > data transiting or stored on this information system;
> > > >> > > > > and
> > > >> > > > > any communications or data transiting or stored on
this
> > > >> > > > > information system may be disclosed or used for any
> > > >> > > > > lawful Government purpose.
> > > >> > > > >
> > > >> > > > > scp: //data/mv_data//edward.strobach/plots/*: No such
> > > >> > > > > file
> > > >> > > > > or
> > > >> > directory
> > > >> > > > > ERROR: Command returned with non-zero status (1): scp
> > > >> > edward.strobach@
> > > >> > > > > 205.156.8.85://data/mv_data//edward.strobach/plots/*
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> >
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > >> > > > > ++ printf '\033]0;%s@%s:%s\007' Edward.Strobach v71a1
> > > >> > > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > NOTE: nothing about this set-up has changed other
than
> > > >> > > > > the
> > > >> directory
> > > >> > > > path
> > > >> > > > > onto dell. The path has been created so it should
> > > >> > > > > support
> > > >> > > > > the
> > > >> > transfer
> > > >> > > > of
> > > >> > > > > plots generated on the MET server. The preceding
> > > >> > > > > information
> > > >> > indicates
> > > >> > > > > that the plot was processed successfully. It's not
> > > >> > > > > clear
> > > >> > > > > why I
> > > >> can't
> > > >> > > > > transfer my plots over.
> > > >> > > > > --
> > > >> > > > > Edward Strobach
> > > >> > > > > EMC/NCEP/NWS/
> > > >> > > > > IMSG Contractor
> > > >> > > > > Cubicle#: 2029
> > > >> > > > > 301-683-3717
> > > >> > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > > > --
> > > >> > > > Julie Prestopnik (she/her)
> > > >> > > > Software Engineer
> > > >> > > > National Center for Atmospheric Research
> > > >> > > > Research Applications Laboratory
> > > >> > > > Email: jpresto at ucar.edu
> > > >> > > >
> > > >> > > > My working day may not be your working day. Please do
not
> > > >> > > > feel
> > > >> obliged
> > > >> > > to
> > > >> > > > reply to this email outside of your normal working
hours.
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> > > --
> > > >> > > Edward Strobach
> > > >> > > EMC/NCEP/NWS/
> > > >> > > IMSG Contractor
> > > >> > > Cubicle#: 2029
> > > >> > > 301-683-3717
> > > >> > >
> > > >> > >
> > > >> >
> > > >> > --
> > > >> > Julie Prestopnik (she/her)
> > > >> > Software Engineer
> > > >> > National Center for Atmospheric Research
> > > >> > Research Applications Laboratory
> > > >> > Email: jpresto at ucar.edu
> > > >> >
> > > >> > My working day may not be your working day. Please do not
> > > >> > feel
> > > >> > obliged
> > > >> to
> > > >> > reply to this email outside of your normal working hours.
> > > >> >
> > > >> >
> > > >>
> > > >> --
> > > >> Edward Strobach
> > > >> EMC/NCEP/NWS/
> > > >> IMSG Contractor
> > > >> Cubicle#: 2029
> > > >> 301-683-3717
> > > >>
> > > >>
> > > >
> > > > --
> > > > Julie Prestopnik (she/her)
> > > > Software Engineer
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > Email: jpresto at ucar.edu
> > > >
> > > > My working day may not be your working day. Please do not
feel
> > > > obliged to
> > > > reply to this email outside of your normal working hours.
> > > >
> >
> >
> >
> >
------------------------------------------------
Subject: can't copy plots over
From: Edward Strobach - NOAA Affiliate
Time: Tue Dec 01 14:28:30 2020
are there youtube examples or a section in a technical manual that
goes
over examples illustrating best practices related to this process?
On Tue, Dec 1, 2020 at 4:03 PM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> For your process you need only two scripts:
> mv_load_to_aws.sh
> mv_batch_on_aws.sh
> I don't think, that others would help you.
>
> The main work is in creating efficient XMLs for the each step.
> Like not recalculating indexes or performing stat_header_db_check
during
> data loading.
>
> Tatiana
>
>
> On Tue Dec 01 13:00:15 2020, edward.strobach at noaa.gov wrote:
> > Thanks. I just made those changes in the xml file, so I 'guess'
I'm
> > ready
> > to go.
> >
> > I should note that I have several steps that I run each day. I
> > convert
> > files into netcdf files that can be read by MET, I generate the
stat
> > files,
> > and I use the stat files in the MET batch plotting system. I have
> > actually
> > 5 total steps, but the plot batch system is really the only one
that
> > gives
> > me trouble on a regular basis. I wish I knew how to reach a point
> > where I
> > have a plotting package that would produce the plots that I need
each
> > day.
> > I'm expected to hand the final product off in the near future, but
> > this
> > part has delayed me extensively. I know that I'm generating a lot
of
> > plots
> > which could potentially overwhelm the system, but I'm unaware how
> > others
> > work to circumvent that. I hope to move past this soon so I can
> > continue
> > development work.
> >
> > There are several scripts that I don't use that might be helpful.
Do
> > any
> > of these work to help me manage the system better?
> > check_my_db.sh mv_batch_on_aws.sh mv_create_db_on_aws.sh
> > mv_db_size_on_aws.sh mv_delete_db_on_aws.sh mv_load_to_aws.sh
> >
> >
> > On Tue, Dec 1, 2020 at 2:44 PM Tatiana Burek via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > Edward
> > >
> > > I run a few butch jobs from my laptop and made sure that
METviewer
> > > code
> > > and scripts working as expected: I was able to produce and copy
> > > plots.
> > > I also got an error like the one that you reported but only when
> > > METviewer
> > > did not generate any plots and the script couldn't copy empty
> > > directory.
> > >
> > > During my testing I noticed a few batch jobs that where started
by
> > > you
> > > and were handing. Each of these jobs started a database query
and
> > > they also
> > > were piling up and handing. They all were waiting for the
indexing
> > > that
> > > takes place after the data loading.
> > > I had to kill all database and batch processes in order to move
> > > forward
> > > with the testing.
> > > To avoid this gridlock you could:
> > > - in your loading xml use <drop_indexes>false</drop_indexes>
> > > <apply_indexes>false</apply_indexes>
> > >
> > >
> > > - do not start a few parallel batch jobs
> > >
> > > I will add a info message to the script that would explain why
the
> > > error
> > > occurs.
> > > For now, could you test batch with some simple XML that produce
at
> > > least
> > > one image.
> > >
> > > Tatiana
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Tue Dec 01 11:35:03 2020, jpresto wrote:
> > > > Hi Edward.
> > > >
> > > > After talking with Tatiana, it sounds like I am misinformed
and the
> > > > scripts
> > > > you are using were, in fact, developed by our METviewer team.
My
> > > > apologies. METplus has a lot of components, and while I
thought I
> > > > had
> > > > a
> > > > good understanding of who did what, it turns out, I did not.
> > > > Tatiana
> > > > is
> > > > working on this issue and will follow up once she knows more.
> > > > Thank
> > > > you
> > > > for your patience and understanding.
> > > >
> > > > Julie
> > > >
> > > > On Tue, Dec 1, 2020 at 11:22 AM Julie Prestopnik
<jpresto at ucar.edu>
> > > > wrote:
> > > >
> > > > > Thanks, Edward. I directed you to Mallory and Perry because
I
> > > > > thought you
> > > > > might be using scripts from them. We did not develop and
have
> > > > > not
> > > > > used
> > > > > those scripts. We have no experience interacting with AWS
and
> > > > > WCOSS,
> > > > > so
> > > > > they seemed best suited to answer your questions related to
that
> > > > > process.
> > > > > However, as Tatiana pointed out, perhaps the security
changes on
> > > > > the
> > > > > AWS
> > > > > server have something to do with what you are experiencing.
If
> > > > > that
> > > > > turns
> > > > > out not to be the case, Mallory and Perry would be best able
to
> > > > > help
> > > > > you.
> > > > > If they are unable to help, please let us know, and we can
see
> > > > > what
> > > > > they
> > > > > tried and how we can help you get this problem figured out.
> > > > >
> > > > > Julie
> > > > >
> > > > > On Tue, Dec 1, 2020 at 11:15 AM Edward Strobach - NOAA
Affiliate
> > > > > via
> > > > > RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > >>
> > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
> > > > >>
> > > > >> Hi Julie,
> > > > >>
> > > > >> I'm actually using the scripts located here:
> > > > >>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts.
> > > > >> These
> > > > >> were scripts that I either got from Mallory or Perry a
while
> > > > >> back -
> > > > >> around
> > > > >> May.
> > > > >>
> > > > >> On Tue, Dec 1, 2020 at 12:59 PM Julie Prestopnik via RT <
> > > > >> met_help at ucar.edu>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Edward.
> > > > >> >
> > > > >> > Are you using the scripts that are located in
> > > > >> >
/gpfs/hps3/emc/global/noscrub/Mallory.Row/VRFY/METviewer_AWS
> > > > >> > and
> > > > >> >
/gpfs/dell2/emc/verification/noscrub/emc.metplus/METviewer_AWS
> > > > >> > for interaction between AWS and WCOSS?
> > > > >> >
> > > > >> > Thanks!
> > > > >> >
> > > > >> > Julie
> > > > >> >
> > > > >> > On Tue, Dec 1, 2020 at 10:49 AM Edward Strobach - NOAA
> > > > >> > Affiliate
> > > > >> > via RT
> > > > >> <
> > > > >> > met_help at ucar.edu> wrote:
> > > > >> >
> > > > >> > >
> > > > >> > > <URL:
> > > > >> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
>
> > > > >> > >
> > > > >> > > Perry does not do batch plotting with MET. I was under
the
> > > > >> > > impression
> > > > >> > that
> > > > >> > > the MET helpdesk helped with MET related issues.
> > > > >> > >
> > > > >> > > On Tue, Dec 1, 2020 at 12:13 PM Julie Prestopnik via RT
<
> > > > >> > met_help at ucar.edu
> > > > >> > > >
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > Hi Edward.
> > > > >> > > >
> > > > >> > > > Please consult with Mallory Row and Perry Shafran for
> > > > >> > > > assistance
> > > > >> with
> > > > >> > > this
> > > > >> > > > issue. They are most familiar with these
capabilities and
> > > > >> > > > will be
> > > > >> best
> > > > >> > > > able to assist you.
> > > > >> > > >
> > > > >> > > > Thanks!
> > > > >> > > >
> > > > >> > > > Julie
> > > > >> > > >
> > > > >> > > > On Tue, Dec 1, 2020 at 9:25 AM Edward Strobach - NOAA
> > > > >> > > > Affiliate via
> > > > >> RT
> > > > >> > <
> > > > >> > > > met_help at ucar.edu> wrote:
> > > > >> > > >
> > > > >> > > > >
> > > > >> > > > > Tue Dec 01 09:25:26 2020: Request 97687 was acted
upon.
> > > > >> > > > > Transaction: Ticket created by
edward.strobach at noaa.gov
> > > > >> > > > > Queue: met_help
> > > > >> > > > > Subject: can't copy plots over
> > > > >> > > > > Owner: Nobody
> > > > >> > > > > Requestors: edward.strobach at noaa.gov
> > > > >> > > > > Status: new
> > > > >> > > > > Ticket <URL:
> > > > >> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > > >> > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > I'm trying to copy plots over to Dell from the MET
> > > > >> > > > > server.
> > > > >> > > > > I
> > > > >> would
> > > > >> > > like
> > > > >> > > > to
> > > > >> > > > > store my plots in the usual place but there is an
scp
> > > > >> > > > > error
> > > > >> > > > > which
> > > > >> I'm
> > > > >> > > > > attaching below.
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > AND BINARY h.fcst_var = 'PMTF'
> > > > >> > > > > AND ld.stat_header_id = h.stat_header_id;
> > > > >> > > > >
> > > > >> > > > > Connection to 205.156.8.85 closed by remote host.
> > > > >> > > > > CALLING: scp edward.strobach at 205.156.8.85
> > > > >> > > > > ://data/mv_data//edward.strobach/plots/*
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > > >> > > > >
> > > > >> > > > > You are accessing a U.S. Government information
> > > > >> > > > > system, which includes: 1) this computer, 2) this
> > > > >> > > > > computer network, 3) all computers connected to
this
> > > > >> > > > > network, and 4) all devices and storage media
attached
> > > > >> > > > > to this network or to a computer on this network.
You
> > > > >> > > > > understand and consent to the following: you may
> > > > >> > > > > access this information system for authorized use
only;
> > > > >> > > > > you have no reasonable expectation of privacy
regarding
> > > > >> > > > > any communication of data transiting or stored on
this
> > > > >> > > > > information system; at any time and for any lawful
> > > > >> > > > > Government purpose, the Government may monitor,
> > > > >> > > > > intercept, and search and seize any communication
or
> > > > >> > > > > data transiting or stored on this information
system;
> > > > >> > > > > and
> > > > >> > > > > any communications or data transiting or stored on
this
> > > > >> > > > > information system may be disclosed or used for any
> > > > >> > > > > lawful Government purpose.
> > > > >> > > > >
> > > > >> > > > > scp: //data/mv_data//edward.strobach/plots/*: No
such
> > > > >> > > > > file
> > > > >> > > > > or
> > > > >> > directory
> > > > >> > > > > ERROR: Command returned with non-zero status (1):
scp
> > > > >> > edward.strobach@
> > > > >> > > > >
205.156.8.85://data/mv_data//edward.strobach/plots/*
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > > >> > > > > ++ printf '\033]0;%s@%s:%s\007' Edward.Strobach
v71a1
> > > > >> > > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > NOTE: nothing about this set-up has changed other
than
> > > > >> > > > > the
> > > > >> directory
> > > > >> > > > path
> > > > >> > > > > onto dell. The path has been created so it should
> > > > >> > > > > support
> > > > >> > > > > the
> > > > >> > transfer
> > > > >> > > > of
> > > > >> > > > > plots generated on the MET server. The preceding
> > > > >> > > > > information
> > > > >> > indicates
> > > > >> > > > > that the plot was processed successfully. It's not
> > > > >> > > > > clear
> > > > >> > > > > why I
> > > > >> can't
> > > > >> > > > > transfer my plots over.
> > > > >> > > > > --
> > > > >> > > > > Edward Strobach
> > > > >> > > > > EMC/NCEP/NWS/
> > > > >> > > > > IMSG Contractor
> > > > >> > > > > Cubicle#: 2029
> > > > >> > > > > 301-683-3717
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > > > --
> > > > >> > > > Julie Prestopnik (she/her)
> > > > >> > > > Software Engineer
> > > > >> > > > National Center for Atmospheric Research
> > > > >> > > > Research Applications Laboratory
> > > > >> > > > Email: jpresto at ucar.edu
> > > > >> > > >
> > > > >> > > > My working day may not be your working day. Please
do not
> > > > >> > > > feel
> > > > >> obliged
> > > > >> > > to
> > > > >> > > > reply to this email outside of your normal working
hours.
> > > > >> > > >
> > > > >> > > >
> > > > >> > >
> > > > >> > > --
> > > > >> > > Edward Strobach
> > > > >> > > EMC/NCEP/NWS/
> > > > >> > > IMSG Contractor
> > > > >> > > Cubicle#: 2029
> > > > >> > > 301-683-3717
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >> > --
> > > > >> > Julie Prestopnik (she/her)
> > > > >> > Software Engineer
> > > > >> > National Center for Atmospheric Research
> > > > >> > Research Applications Laboratory
> > > > >> > Email: jpresto at ucar.edu
> > > > >> >
> > > > >> > My working day may not be your working day. Please do
not
> > > > >> > feel
> > > > >> > obliged
> > > > >> to
> > > > >> > reply to this email outside of your normal working hours.
> > > > >> >
> > > > >> >
> > > > >>
> > > > >> --
> > > > >> Edward Strobach
> > > > >> EMC/NCEP/NWS/
> > > > >> IMSG Contractor
> > > > >> Cubicle#: 2029
> > > > >> 301-683-3717
> > > > >>
> > > > >>
> > > > >
> > > > > --
> > > > > Julie Prestopnik (she/her)
> > > > > Software Engineer
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > Email: jpresto at ucar.edu
> > > > >
> > > > > My working day may not be your working day. Please do not
feel
> > > > > obliged to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > >
> > >
> > >
> > >
>
>
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: can't copy plots over
From: Edward Strobach - NOAA Affiliate
Time: Tue Dec 01 15:14:59 2020
maybe there is a script that allows me to check what is going on with
my
database? A lot of these issues I've come across happen because of
things
I'm unaware of or things that go on simultaneously that causes
confusion in
the processing. Right now I have a plotting script set that
simultaneously
submits batch jobs for time series statistics, Taylor diagrams, and
CSI. I
haven't seen too many issues when I do that, but would it make sense
to
insert a while loop after the first submission that uses a script to
check
the activity within my database? In other words, I would use a
script,
perhaps one that has already been developed, to check the status of
processing before submitting the next batch job. Once the processing
for
the first plotting steps have completed, then I would commence to the
next
one.
I guess check_my_db.sh would not do that?
On Tue, Dec 1, 2020 at 4:27 PM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> are there youtube examples or a section in a technical manual that
goes
> over examples illustrating best practices related to this process?
>
> On Tue, Dec 1, 2020 at 4:03 PM Tatiana Burek via RT
<met_help at ucar.edu>
> wrote:
>
>> For your process you need only two scripts:
>> mv_load_to_aws.sh
>> mv_batch_on_aws.sh
>> I don't think, that others would help you.
>>
>> The main work is in creating efficient XMLs for the each step.
>> Like not recalculating indexes or performing stat_header_db_check
during
>> data loading.
>>
>> Tatiana
>>
>>
>> On Tue Dec 01 13:00:15 2020, edward.strobach at noaa.gov wrote:
>> > Thanks. I just made those changes in the xml file, so I 'guess'
I'm
>> > ready
>> > to go.
>> >
>> > I should note that I have several steps that I run each day. I
>> > convert
>> > files into netcdf files that can be read by MET, I generate the
stat
>> > files,
>> > and I use the stat files in the MET batch plotting system. I
have
>> > actually
>> > 5 total steps, but the plot batch system is really the only one
that
>> > gives
>> > me trouble on a regular basis. I wish I knew how to reach a
point
>> > where I
>> > have a plotting package that would produce the plots that I need
each
>> > day.
>> > I'm expected to hand the final product off in the near future,
but
>> > this
>> > part has delayed me extensively. I know that I'm generating a lot
of
>> > plots
>> > which could potentially overwhelm the system, but I'm unaware how
>> > others
>> > work to circumvent that. I hope to move past this soon so I can
>> > continue
>> > development work.
>> >
>> > There are several scripts that I don't use that might be helpful.
Do
>> > any
>> > of these work to help me manage the system better?
>> > check_my_db.sh mv_batch_on_aws.sh mv_create_db_on_aws.sh
>> > mv_db_size_on_aws.sh mv_delete_db_on_aws.sh mv_load_to_aws.sh
>> >
>> >
>> > On Tue, Dec 1, 2020 at 2:44 PM Tatiana Burek via RT
>> > <met_help at ucar.edu>
>> > wrote:
>> >
>> > > Edward
>> > >
>> > > I run a few butch jobs from my laptop and made sure that
METviewer
>> > > code
>> > > and scripts working as expected: I was able to produce and copy
>> > > plots.
>> > > I also got an error like the one that you reported but only
when
>> > > METviewer
>> > > did not generate any plots and the script couldn't copy empty
>> > > directory.
>> > >
>> > > During my testing I noticed a few batch jobs that where
started by
>> > > you
>> > > and were handing. Each of these jobs started a database query
and
>> > > they also
>> > > were piling up and handing. They all were waiting for the
indexing
>> > > that
>> > > takes place after the data loading.
>> > > I had to kill all database and batch processes in order to move
>> > > forward
>> > > with the testing.
>> > > To avoid this gridlock you could:
>> > > - in your loading xml use <drop_indexes>false</drop_indexes>
>> > > <apply_indexes>false</apply_indexes>
>> > >
>> > >
>> > > - do not start a few parallel batch jobs
>> > >
>> > > I will add a info message to the script that would explain why
the
>> > > error
>> > > occurs.
>> > > For now, could you test batch with some simple XML that produce
at
>> > > least
>> > > one image.
>> > >
>> > > Tatiana
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Tue Dec 01 11:35:03 2020, jpresto wrote:
>> > > > Hi Edward.
>> > > >
>> > > > After talking with Tatiana, it sounds like I am misinformed
and the
>> > > > scripts
>> > > > you are using were, in fact, developed by our METviewer team.
My
>> > > > apologies. METplus has a lot of components, and while I
thought I
>> > > > had
>> > > > a
>> > > > good understanding of who did what, it turns out, I did not.
>> > > > Tatiana
>> > > > is
>> > > > working on this issue and will follow up once she knows more.
>> > > > Thank
>> > > > you
>> > > > for your patience and understanding.
>> > > >
>> > > > Julie
>> > > >
>> > > > On Tue, Dec 1, 2020 at 11:22 AM Julie Prestopnik
<jpresto at ucar.edu>
>> > > > wrote:
>> > > >
>> > > > > Thanks, Edward. I directed you to Mallory and Perry
because I
>> > > > > thought you
>> > > > > might be using scripts from them. We did not develop and
have
>> > > > > not
>> > > > > used
>> > > > > those scripts. We have no experience interacting with AWS
and
>> > > > > WCOSS,
>> > > > > so
>> > > > > they seemed best suited to answer your questions related to
that
>> > > > > process.
>> > > > > However, as Tatiana pointed out, perhaps the security
changes on
>> > > > > the
>> > > > > AWS
>> > > > > server have something to do with what you are experiencing.
If
>> > > > > that
>> > > > > turns
>> > > > > out not to be the case, Mallory and Perry would be best
able to
>> > > > > help
>> > > > > you.
>> > > > > If they are unable to help, please let us know, and we can
see
>> > > > > what
>> > > > > they
>> > > > > tried and how we can help you get this problem figured out.
>> > > > >
>> > > > > Julie
>> > > > >
>> > > > > On Tue, Dec 1, 2020 at 11:15 AM Edward Strobach - NOAA
Affiliate
>> > > > > via
>> > > > > RT <
>> > > > > met_help at ucar.edu> wrote:
>> > > > >
>> > > > >>
>> > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
>> > > > >>
>> > > > >> Hi Julie,
>> > > > >>
>> > > > >> I'm actually using the scripts located here:
>> > > > >>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts.
>> > > > >> These
>> > > > >> were scripts that I either got from Mallory or Perry a
while
>> > > > >> back -
>> > > > >> around
>> > > > >> May.
>> > > > >>
>> > > > >> On Tue, Dec 1, 2020 at 12:59 PM Julie Prestopnik via RT <
>> > > > >> met_help at ucar.edu>
>> > > > >> wrote:
>> > > > >>
>> > > > >> > Hi Edward.
>> > > > >> >
>> > > > >> > Are you using the scripts that are located in
>> > > > >> >
/gpfs/hps3/emc/global/noscrub/Mallory.Row/VRFY/METviewer_AWS
>> > > > >> > and
>> > > > >> >
/gpfs/dell2/emc/verification/noscrub/emc.metplus/METviewer_AWS
>> > > > >> > for interaction between AWS and WCOSS?
>> > > > >> >
>> > > > >> > Thanks!
>> > > > >> >
>> > > > >> > Julie
>> > > > >> >
>> > > > >> > On Tue, Dec 1, 2020 at 10:49 AM Edward Strobach - NOAA
>> > > > >> > Affiliate
>> > > > >> > via RT
>> > > > >> <
>> > > > >> > met_help at ucar.edu> wrote:
>> > > > >> >
>> > > > >> > >
>> > > > >> > > <URL:
>> > > > >> > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
>> > > > >> > >
>> > > > >> > > Perry does not do batch plotting with MET. I was
under the
>> > > > >> > > impression
>> > > > >> > that
>> > > > >> > > the MET helpdesk helped with MET related issues.
>> > > > >> > >
>> > > > >> > > On Tue, Dec 1, 2020 at 12:13 PM Julie Prestopnik via
RT <
>> > > > >> > met_help at ucar.edu
>> > > > >> > > >
>> > > > >> > > wrote:
>> > > > >> > >
>> > > > >> > > > Hi Edward.
>> > > > >> > > >
>> > > > >> > > > Please consult with Mallory Row and Perry Shafran
for
>> > > > >> > > > assistance
>> > > > >> with
>> > > > >> > > this
>> > > > >> > > > issue. They are most familiar with these
capabilities and
>> > > > >> > > > will be
>> > > > >> best
>> > > > >> > > > able to assist you.
>> > > > >> > > >
>> > > > >> > > > Thanks!
>> > > > >> > > >
>> > > > >> > > > Julie
>> > > > >> > > >
>> > > > >> > > > On Tue, Dec 1, 2020 at 9:25 AM Edward Strobach -
NOAA
>> > > > >> > > > Affiliate via
>> > > > >> RT
>> > > > >> > <
>> > > > >> > > > met_help at ucar.edu> wrote:
>> > > > >> > > >
>> > > > >> > > > >
>> > > > >> > > > > Tue Dec 01 09:25:26 2020: Request 97687 was acted
upon.
>> > > > >> > > > > Transaction: Ticket created by
edward.strobach at noaa.gov
>> > > > >> > > > > Queue: met_help
>> > > > >> > > > > Subject: can't copy plots over
>> > > > >> > > > > Owner: Nobody
>> > > > >> > > > > Requestors: edward.strobach at noaa.gov
>> > > > >> > > > > Status: new
>> > > > >> > > > > Ticket <URL:
>> > > > >> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
>> > > > >> > > >
>> > > > >> > > > >
>> > > > >> > > > >
>> > > > >> > > > > I'm trying to copy plots over to Dell from the MET
>> > > > >> > > > > server.
>> > > > >> > > > > I
>> > > > >> would
>> > > > >> > > like
>> > > > >> > > > to
>> > > > >> > > > > store my plots in the usual place but there is an
scp
>> > > > >> > > > > error
>> > > > >> > > > > which
>> > > > >> I'm
>> > > > >> > > > > attaching below.
>> > > > >> > > > >
>> > > > >> > > > >
>> > > > >> > > > > AND BINARY h.fcst_var = 'PMTF'
>> > > > >> > > > > AND ld.stat_header_id = h.stat_header_id;
>> > > > >> > > > >
>> > > > >> > > > > Connection to 205.156.8.85 closed by remote host.
>> > > > >> > > > > CALLING: scp edward.strobach at 205.156.8.85
>> > > > >> > > > > ://data/mv_data//edward.strobach/plots/*
>> > > > >> > > > >
>> > > > >> > > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
>> > > > >> > > > >
>> > > > >> > > > > You are accessing a U.S. Government information
>> > > > >> > > > > system, which includes: 1) this computer, 2) this
>> > > > >> > > > > computer network, 3) all computers connected to
this
>> > > > >> > > > > network, and 4) all devices and storage media
attached
>> > > > >> > > > > to this network or to a computer on this network.
You
>> > > > >> > > > > understand and consent to the following: you may
>> > > > >> > > > > access this information system for authorized use
only;
>> > > > >> > > > > you have no reasonable expectation of privacy
regarding
>> > > > >> > > > > any communication of data transiting or stored on
this
>> > > > >> > > > > information system; at any time and for any lawful
>> > > > >> > > > > Government purpose, the Government may monitor,
>> > > > >> > > > > intercept, and search and seize any communication
or
>> > > > >> > > > > data transiting or stored on this information
system;
>> > > > >> > > > > and
>> > > > >> > > > > any communications or data transiting or stored on
this
>> > > > >> > > > > information system may be disclosed or used for
any
>> > > > >> > > > > lawful Government purpose.
>> > > > >> > > > >
>> > > > >> > > > > scp: //data/mv_data//edward.strobach/plots/*: No
such
>> > > > >> > > > > file
>> > > > >> > > > > or
>> > > > >> > directory
>> > > > >> > > > > ERROR: Command returned with non-zero status (1):
scp
>> > > > >> > edward.strobach@
>> > > > >> > > > >
205.156.8.85://data/mv_data//edward.strobach/plots/*
>> > > > >> > > > >
>> > > > >> > > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > >
>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
>> > > > >> > > > > ++ printf '\033]0;%s@%s:%s\007' Edward.Strobach
v71a1
>> > > > >> > > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>> > > > >> > > > >
>> > > > >> > > > >
>> > > > >> > > > > NOTE: nothing about this set-up has changed other
than
>> > > > >> > > > > the
>> > > > >> directory
>> > > > >> > > > path
>> > > > >> > > > > onto dell. The path has been created so it should
>> > > > >> > > > > support
>> > > > >> > > > > the
>> > > > >> > transfer
>> > > > >> > > > of
>> > > > >> > > > > plots generated on the MET server. The preceding
>> > > > >> > > > > information
>> > > > >> > indicates
>> > > > >> > > > > that the plot was processed successfully. It's
not
>> > > > >> > > > > clear
>> > > > >> > > > > why I
>> > > > >> can't
>> > > > >> > > > > transfer my plots over.
>> > > > >> > > > > --
>> > > > >> > > > > Edward Strobach
>> > > > >> > > > > EMC/NCEP/NWS/
>> > > > >> > > > > IMSG Contractor
>> > > > >> > > > > Cubicle#: 2029
>> > > > >> > > > > 301-683-3717
>> > > > >> > > > >
>> > > > >> > > > >
>> > > > >> > > >
>> > > > >> > > > --
>> > > > >> > > > Julie Prestopnik (she/her)
>> > > > >> > > > Software Engineer
>> > > > >> > > > National Center for Atmospheric Research
>> > > > >> > > > Research Applications Laboratory
>> > > > >> > > > Email: jpresto at ucar.edu
>> > > > >> > > >
>> > > > >> > > > My working day may not be your working day. Please
do not
>> > > > >> > > > feel
>> > > > >> obliged
>> > > > >> > > to
>> > > > >> > > > reply to this email outside of your normal working
hours.
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> > > --
>> > > > >> > > Edward Strobach
>> > > > >> > > EMC/NCEP/NWS/
>> > > > >> > > IMSG Contractor
>> > > > >> > > Cubicle#: 2029
>> > > > >> > > 301-683-3717
>> > > > >> > >
>> > > > >> > >
>> > > > >> >
>> > > > >> > --
>> > > > >> > Julie Prestopnik (she/her)
>> > > > >> > Software Engineer
>> > > > >> > National Center for Atmospheric Research
>> > > > >> > Research Applications Laboratory
>> > > > >> > Email: jpresto at ucar.edu
>> > > > >> >
>> > > > >> > My working day may not be your working day. Please do
not
>> > > > >> > feel
>> > > > >> > obliged
>> > > > >> to
>> > > > >> > reply to this email outside of your normal working
hours.
>> > > > >> >
>> > > > >> >
>> > > > >>
>> > > > >> --
>> > > > >> Edward Strobach
>> > > > >> EMC/NCEP/NWS/
>> > > > >> IMSG Contractor
>> > > > >> Cubicle#: 2029
>> > > > >> 301-683-3717
>> > > > >>
>> > > > >>
>> > > > >
>> > > > > --
>> > > > > Julie Prestopnik (she/her)
>> > > > > Software Engineer
>> > > > > National Center for Atmospheric Research
>> > > > > Research Applications Laboratory
>> > > > > Email: jpresto at ucar.edu
>> > > > >
>> > > > > My working day may not be your working day. Please do not
feel
>> > > > > obliged to
>> > > > > reply to this email outside of your normal working hours.
>> > > > >
>> > >
>> > >
>> > >
>> > >
>>
>>
>>
>>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: can't copy plots over
From: Edward Strobach - NOAA Affiliate
Time: Wed Dec 02 06:32:23 2020
I came across this issue again this morning. Is there a way to stop
certain processes and start fresh from my end?
On Tue, Dec 1, 2020 at 5:14 PM Edward Strobach - NOAA Affiliate <
edward.strobach at noaa.gov> wrote:
> maybe there is a script that allows me to check what is going on
with my
> database? A lot of these issues I've come across happen because of
things
> I'm unaware of or things that go on simultaneously that causes
confusion in
> the processing. Right now I have a plotting script set that
simultaneously
> submits batch jobs for time series statistics, Taylor diagrams, and
CSI. I
> haven't seen too many issues when I do that, but would it make sense
to
> insert a while loop after the first submission that uses a script to
check
> the activity within my database? In other words, I would use a
script,
> perhaps one that has already been developed, to check the status of
> processing before submitting the next batch job. Once the
processing for
> the first plotting steps have completed, then I would commence to
the next
> one.
>
> I guess check_my_db.sh would not do that?
>
>
>
>
> On Tue, Dec 1, 2020 at 4:27 PM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
>> are there youtube examples or a section in a technical manual that
goes
>> over examples illustrating best practices related to this process?
>>
>> On Tue, Dec 1, 2020 at 4:03 PM Tatiana Burek via RT
<met_help at ucar.edu>
>> wrote:
>>
>>> For your process you need only two scripts:
>>> mv_load_to_aws.sh
>>> mv_batch_on_aws.sh
>>> I don't think, that others would help you.
>>>
>>> The main work is in creating efficient XMLs for the each step.
>>> Like not recalculating indexes or performing stat_header_db_check
during
>>> data loading.
>>>
>>> Tatiana
>>>
>>>
>>> On Tue Dec 01 13:00:15 2020, edward.strobach at noaa.gov wrote:
>>> > Thanks. I just made those changes in the xml file, so I 'guess'
I'm
>>> > ready
>>> > to go.
>>> >
>>> > I should note that I have several steps that I run each day. I
>>> > convert
>>> > files into netcdf files that can be read by MET, I generate the
stat
>>> > files,
>>> > and I use the stat files in the MET batch plotting system. I
have
>>> > actually
>>> > 5 total steps, but the plot batch system is really the only one
that
>>> > gives
>>> > me trouble on a regular basis. I wish I knew how to reach a
point
>>> > where I
>>> > have a plotting package that would produce the plots that I need
each
>>> > day.
>>> > I'm expected to hand the final product off in the near future,
but
>>> > this
>>> > part has delayed me extensively. I know that I'm generating a
lot of
>>> > plots
>>> > which could potentially overwhelm the system, but I'm unaware
how
>>> > others
>>> > work to circumvent that. I hope to move past this soon so I can
>>> > continue
>>> > development work.
>>> >
>>> > There are several scripts that I don't use that might be
helpful. Do
>>> > any
>>> > of these work to help me manage the system better?
>>> > check_my_db.sh mv_batch_on_aws.sh mv_create_db_on_aws.sh
>>> > mv_db_size_on_aws.sh mv_delete_db_on_aws.sh mv_load_to_aws.sh
>>> >
>>> >
>>> > On Tue, Dec 1, 2020 at 2:44 PM Tatiana Burek via RT
>>> > <met_help at ucar.edu>
>>> > wrote:
>>> >
>>> > > Edward
>>> > >
>>> > > I run a few butch jobs from my laptop and made sure that
METviewer
>>> > > code
>>> > > and scripts working as expected: I was able to produce and
copy
>>> > > plots.
>>> > > I also got an error like the one that you reported but only
when
>>> > > METviewer
>>> > > did not generate any plots and the script couldn't copy empty
>>> > > directory.
>>> > >
>>> > > During my testing I noticed a few batch jobs that where
started by
>>> > > you
>>> > > and were handing. Each of these jobs started a database query
and
>>> > > they also
>>> > > were piling up and handing. They all were waiting for the
indexing
>>> > > that
>>> > > takes place after the data loading.
>>> > > I had to kill all database and batch processes in order to
move
>>> > > forward
>>> > > with the testing.
>>> > > To avoid this gridlock you could:
>>> > > - in your loading xml use <drop_indexes>false</drop_indexes>
>>> > > <apply_indexes>false</apply_indexes>
>>> > >
>>> > >
>>> > > - do not start a few parallel batch jobs
>>> > >
>>> > > I will add a info message to the script that would explain why
the
>>> > > error
>>> > > occurs.
>>> > > For now, could you test batch with some simple XML that
produce at
>>> > > least
>>> > > one image.
>>> > >
>>> > > Tatiana
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > On Tue Dec 01 11:35:03 2020, jpresto wrote:
>>> > > > Hi Edward.
>>> > > >
>>> > > > After talking with Tatiana, it sounds like I am misinformed
and the
>>> > > > scripts
>>> > > > you are using were, in fact, developed by our METviewer
team. My
>>> > > > apologies. METplus has a lot of components, and while I
thought I
>>> > > > had
>>> > > > a
>>> > > > good understanding of who did what, it turns out, I did not.
>>> > > > Tatiana
>>> > > > is
>>> > > > working on this issue and will follow up once she knows
more.
>>> > > > Thank
>>> > > > you
>>> > > > for your patience and understanding.
>>> > > >
>>> > > > Julie
>>> > > >
>>> > > > On Tue, Dec 1, 2020 at 11:22 AM Julie Prestopnik
<jpresto at ucar.edu
>>> >
>>> > > > wrote:
>>> > > >
>>> > > > > Thanks, Edward. I directed you to Mallory and Perry
because I
>>> > > > > thought you
>>> > > > > might be using scripts from them. We did not develop and
have
>>> > > > > not
>>> > > > > used
>>> > > > > those scripts. We have no experience interacting with AWS
and
>>> > > > > WCOSS,
>>> > > > > so
>>> > > > > they seemed best suited to answer your questions related
to that
>>> > > > > process.
>>> > > > > However, as Tatiana pointed out, perhaps the security
changes on
>>> > > > > the
>>> > > > > AWS
>>> > > > > server have something to do with what you are
experiencing. If
>>> > > > > that
>>> > > > > turns
>>> > > > > out not to be the case, Mallory and Perry would be best
able to
>>> > > > > help
>>> > > > > you.
>>> > > > > If they are unable to help, please let us know, and we can
see
>>> > > > > what
>>> > > > > they
>>> > > > > tried and how we can help you get this problem figured
out.
>>> > > > >
>>> > > > > Julie
>>> > > > >
>>> > > > > On Tue, Dec 1, 2020 at 11:15 AM Edward Strobach - NOAA
Affiliate
>>> > > > > via
>>> > > > > RT <
>>> > > > > met_help at ucar.edu> wrote:
>>> > > > >
>>> > > > >>
>>> > > > >> <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
>>> > > > >>
>>> > > > >> Hi Julie,
>>> > > > >>
>>> > > > >> I'm actually using the scripts located here:
>>> > > > >>
>>> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts.
>>> > > > >> These
>>> > > > >> were scripts that I either got from Mallory or Perry a
while
>>> > > > >> back -
>>> > > > >> around
>>> > > > >> May.
>>> > > > >>
>>> > > > >> On Tue, Dec 1, 2020 at 12:59 PM Julie Prestopnik via RT <
>>> > > > >> met_help at ucar.edu>
>>> > > > >> wrote:
>>> > > > >>
>>> > > > >> > Hi Edward.
>>> > > > >> >
>>> > > > >> > Are you using the scripts that are located in
>>> > > > >> >
/gpfs/hps3/emc/global/noscrub/Mallory.Row/VRFY/METviewer_AWS
>>> > > > >> > and
>>> > > > >> >
/gpfs/dell2/emc/verification/noscrub/emc.metplus/METviewer_AWS
>>> > > > >> > for interaction between AWS and WCOSS?
>>> > > > >> >
>>> > > > >> > Thanks!
>>> > > > >> >
>>> > > > >> > Julie
>>> > > > >> >
>>> > > > >> > On Tue, Dec 1, 2020 at 10:49 AM Edward Strobach - NOAA
>>> > > > >> > Affiliate
>>> > > > >> > via RT
>>> > > > >> <
>>> > > > >> > met_help at ucar.edu> wrote:
>>> > > > >> >
>>> > > > >> > >
>>> > > > >> > > <URL:
>>> > > > >> > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
>>> > > > >> > >
>>> > > > >> > > Perry does not do batch plotting with MET. I was
under the
>>> > > > >> > > impression
>>> > > > >> > that
>>> > > > >> > > the MET helpdesk helped with MET related issues.
>>> > > > >> > >
>>> > > > >> > > On Tue, Dec 1, 2020 at 12:13 PM Julie Prestopnik via
RT <
>>> > > > >> > met_help at ucar.edu
>>> > > > >> > > >
>>> > > > >> > > wrote:
>>> > > > >> > >
>>> > > > >> > > > Hi Edward.
>>> > > > >> > > >
>>> > > > >> > > > Please consult with Mallory Row and Perry Shafran
for
>>> > > > >> > > > assistance
>>> > > > >> with
>>> > > > >> > > this
>>> > > > >> > > > issue. They are most familiar with these
capabilities and
>>> > > > >> > > > will be
>>> > > > >> best
>>> > > > >> > > > able to assist you.
>>> > > > >> > > >
>>> > > > >> > > > Thanks!
>>> > > > >> > > >
>>> > > > >> > > > Julie
>>> > > > >> > > >
>>> > > > >> > > > On Tue, Dec 1, 2020 at 9:25 AM Edward Strobach -
NOAA
>>> > > > >> > > > Affiliate via
>>> > > > >> RT
>>> > > > >> > <
>>> > > > >> > > > met_help at ucar.edu> wrote:
>>> > > > >> > > >
>>> > > > >> > > > >
>>> > > > >> > > > > Tue Dec 01 09:25:26 2020: Request 97687 was acted
upon.
>>> > > > >> > > > > Transaction: Ticket created by
edward.strobach at noaa.gov
>>> > > > >> > > > > Queue: met_help
>>> > > > >> > > > > Subject: can't copy plots over
>>> > > > >> > > > > Owner: Nobody
>>> > > > >> > > > > Requestors: edward.strobach at noaa.gov
>>> > > > >> > > > > Status: new
>>> > > > >> > > > > Ticket <URL:
>>> > > > >> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
>>> > > > >> > > >
>>> > > > >> > > > >
>>> > > > >> > > > >
>>> > > > >> > > > > I'm trying to copy plots over to Dell from the
MET
>>> > > > >> > > > > server.
>>> > > > >> > > > > I
>>> > > > >> would
>>> > > > >> > > like
>>> > > > >> > > > to
>>> > > > >> > > > > store my plots in the usual place but there is an
scp
>>> > > > >> > > > > error
>>> > > > >> > > > > which
>>> > > > >> I'm
>>> > > > >> > > > > attaching below.
>>> > > > >> > > > >
>>> > > > >> > > > >
>>> > > > >> > > > > AND BINARY h.fcst_var = 'PMTF'
>>> > > > >> > > > > AND ld.stat_header_id = h.stat_header_id;
>>> > > > >> > > > >
>>> > > > >> > > > > Connection to 205.156.8.85 closed by remote host.
>>> > > > >> > > > > CALLING: scp edward.strobach at 205.156.8.85
>>> > > > >> > > > > ://data/mv_data//edward.strobach/plots/*
>>> > > > >> > > > >
>>> > > > >> > > > >
>>> > > > >> > > >
>>> > > > >> > >
>>> > > > >> >
>>> > > > >>
>>> > >
>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
>>> > > > >> > > > >
>>> > > > >> > > > > You are accessing a U.S. Government information
>>> > > > >> > > > > system, which includes: 1) this computer, 2) this
>>> > > > >> > > > > computer network, 3) all computers connected to
this
>>> > > > >> > > > > network, and 4) all devices and storage media
attached
>>> > > > >> > > > > to this network or to a computer on this network.
You
>>> > > > >> > > > > understand and consent to the following: you may
>>> > > > >> > > > > access this information system for authorized use
only;
>>> > > > >> > > > > you have no reasonable expectation of privacy
regarding
>>> > > > >> > > > > any communication of data transiting or stored on
this
>>> > > > >> > > > > information system; at any time and for any
lawful
>>> > > > >> > > > > Government purpose, the Government may monitor,
>>> > > > >> > > > > intercept, and search and seize any communication
or
>>> > > > >> > > > > data transiting or stored on this information
system;
>>> > > > >> > > > > and
>>> > > > >> > > > > any communications or data transiting or stored
on this
>>> > > > >> > > > > information system may be disclosed or used for
any
>>> > > > >> > > > > lawful Government purpose.
>>> > > > >> > > > >
>>> > > > >> > > > > scp: //data/mv_data//edward.strobach/plots/*: No
such
>>> > > > >> > > > > file
>>> > > > >> > > > > or
>>> > > > >> > directory
>>> > > > >> > > > > ERROR: Command returned with non-zero status (1):
scp
>>> > > > >> > edward.strobach@
>>> > > > >> > > > >
205.156.8.85://data/mv_data//edward.strobach/plots/*
>>> > > > >> > > > >
>>> > > > >> > > > >
>>> > > > >> > > >
>>> > > > >> > >
>>> > > > >> >
>>> > > > >>
>>> > >
>>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
>>> > > > >> > > > > ++ printf '\033]0;%s@%s:%s\007' Edward.Strobach
v71a1
>>> > > > >> > > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
>>> > > > >> > > > >
>>> > > > >> > > > >
>>> > > > >> > > > > NOTE: nothing about this set-up has changed
other than
>>> > > > >> > > > > the
>>> > > > >> directory
>>> > > > >> > > > path
>>> > > > >> > > > > onto dell. The path has been created so it
should
>>> > > > >> > > > > support
>>> > > > >> > > > > the
>>> > > > >> > transfer
>>> > > > >> > > > of
>>> > > > >> > > > > plots generated on the MET server. The preceding
>>> > > > >> > > > > information
>>> > > > >> > indicates
>>> > > > >> > > > > that the plot was processed successfully. It's
not
>>> > > > >> > > > > clear
>>> > > > >> > > > > why I
>>> > > > >> can't
>>> > > > >> > > > > transfer my plots over.
>>> > > > >> > > > > --
>>> > > > >> > > > > Edward Strobach
>>> > > > >> > > > > EMC/NCEP/NWS/
>>> > > > >> > > > > IMSG Contractor
>>> > > > >> > > > > Cubicle#: 2029
>>> > > > >> > > > > 301-683-3717
>>> > > > >> > > > >
>>> > > > >> > > > >
>>> > > > >> > > >
>>> > > > >> > > > --
>>> > > > >> > > > Julie Prestopnik (she/her)
>>> > > > >> > > > Software Engineer
>>> > > > >> > > > National Center for Atmospheric Research
>>> > > > >> > > > Research Applications Laboratory
>>> > > > >> > > > Email: jpresto at ucar.edu
>>> > > > >> > > >
>>> > > > >> > > > My working day may not be your working day. Please
do not
>>> > > > >> > > > feel
>>> > > > >> obliged
>>> > > > >> > > to
>>> > > > >> > > > reply to this email outside of your normal working
hours.
>>> > > > >> > > >
>>> > > > >> > > >
>>> > > > >> > >
>>> > > > >> > > --
>>> > > > >> > > Edward Strobach
>>> > > > >> > > EMC/NCEP/NWS/
>>> > > > >> > > IMSG Contractor
>>> > > > >> > > Cubicle#: 2029
>>> > > > >> > > 301-683-3717
>>> > > > >> > >
>>> > > > >> > >
>>> > > > >> >
>>> > > > >> > --
>>> > > > >> > Julie Prestopnik (she/her)
>>> > > > >> > Software Engineer
>>> > > > >> > National Center for Atmospheric Research
>>> > > > >> > Research Applications Laboratory
>>> > > > >> > Email: jpresto at ucar.edu
>>> > > > >> >
>>> > > > >> > My working day may not be your working day. Please do
not
>>> > > > >> > feel
>>> > > > >> > obliged
>>> > > > >> to
>>> > > > >> > reply to this email outside of your normal working
hours.
>>> > > > >> >
>>> > > > >> >
>>> > > > >>
>>> > > > >> --
>>> > > > >> Edward Strobach
>>> > > > >> EMC/NCEP/NWS/
>>> > > > >> IMSG Contractor
>>> > > > >> Cubicle#: 2029
>>> > > > >> 301-683-3717
>>> > > > >>
>>> > > > >>
>>> > > > >
>>> > > > > --
>>> > > > > Julie Prestopnik (she/her)
>>> > > > > Software Engineer
>>> > > > > National Center for Atmospheric Research
>>> > > > > Research Applications Laboratory
>>> > > > > Email: jpresto at ucar.edu
>>> > > > >
>>> > > > > My working day may not be your working day. Please do not
feel
>>> > > > > obliged to
>>> > > > > reply to this email outside of your normal working hours.
>>> > > > >
>>> > >
>>> > >
>>> > >
>>> > >
>>>
>>>
>>>
>>>
>>
>> --
>> Edward Strobach
>> EMC/NCEP/NWS/
>> IMSG Contractor
>> Cubicle#: 2029
>> 301-683-3717
>>
>
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: can't copy plots over
From: Tatiana Burek
Time: Wed Dec 02 07:34:47 2020
You can kill your processes by :
- collecting what is running:
ps -ef | grep edward
- killing each process
kill -9 2574
kill -9 2594
or all your processes:
pkill -9 -u `id -u edward.strobach`
Right now you have 4 loading scripts and 10 batch scripts running at
the same time which creates a gridlock and none of the processes are
getting executes.
Tatiana
On Wed Dec 02 06:32:23 2020, edward.strobach at noaa.gov wrote:
> I came across this issue again this morning. Is there a way to stop
> certain processes and start fresh from my end?
>
> On Tue, Dec 1, 2020 at 5:14 PM Edward Strobach - NOAA Affiliate <
> edward.strobach at noaa.gov> wrote:
>
> > maybe there is a script that allows me to check what is going on
with
> > my
> > database? A lot of these issues I've come across happen because
of
> > things
> > I'm unaware of or things that go on simultaneously that causes
> > confusion in
> > the processing. Right now I have a plotting script set that
> > simultaneously
> > submits batch jobs for time series statistics, Taylor diagrams,
and
> > CSI. I
> > haven't seen too many issues when I do that, but would it make
sense
> > to
> > insert a while loop after the first submission that uses a script
to
> > check
> > the activity within my database? In other words, I would use a
> > script,
> > perhaps one that has already been developed, to check the status
of
> > processing before submitting the next batch job. Once the
processing
> > for
> > the first plotting steps have completed, then I would commence to
the
> > next
> > one.
> >
> > I guess check_my_db.sh would not do that?
> >
> >
> >
> >
> > On Tue, Dec 1, 2020 at 4:27 PM Edward Strobach - NOAA Affiliate <
> > edward.strobach at noaa.gov> wrote:
> >
> >> are there youtube examples or a section in a technical manual
that
> >> goes
> >> over examples illustrating best practices related to this
process?
> >>
> >> On Tue, Dec 1, 2020 at 4:03 PM Tatiana Burek via RT
> >> <met_help at ucar.edu>
> >> wrote:
> >>
> >>> For your process you need only two scripts:
> >>> mv_load_to_aws.sh
> >>> mv_batch_on_aws.sh
> >>> I don't think, that others would help you.
> >>>
> >>> The main work is in creating efficient XMLs for the each step.
> >>> Like not recalculating indexes or performing
stat_header_db_check
> >>> during
> >>> data loading.
> >>>
> >>> Tatiana
> >>>
> >>>
> >>> On Tue Dec 01 13:00:15 2020, edward.strobach at noaa.gov wrote:
> >>> > Thanks. I just made those changes in the xml file, so I
'guess'
> >>> > I'm
> >>> > ready
> >>> > to go.
> >>> >
> >>> > I should note that I have several steps that I run each day.
I
> >>> > convert
> >>> > files into netcdf files that can be read by MET, I generate
the
> >>> > stat
> >>> > files,
> >>> > and I use the stat files in the MET batch plotting system. I
> >>> > have
> >>> > actually
> >>> > 5 total steps, but the plot batch system is really the only
one
> >>> > that
> >>> > gives
> >>> > me trouble on a regular basis. I wish I knew how to reach a
> >>> > point
> >>> > where I
> >>> > have a plotting package that would produce the plots that I
need
> >>> > each
> >>> > day.
> >>> > I'm expected to hand the final product off in the near future,
> >>> > but
> >>> > this
> >>> > part has delayed me extensively. I know that I'm generating a
lot
> >>> > of
> >>> > plots
> >>> > which could potentially overwhelm the system, but I'm unaware
how
> >>> > others
> >>> > work to circumvent that. I hope to move past this soon so I
can
> >>> > continue
> >>> > development work.
> >>> >
> >>> > There are several scripts that I don't use that might be
helpful.
> >>> > Do
> >>> > any
> >>> > of these work to help me manage the system better?
> >>> > check_my_db.sh mv_batch_on_aws.sh mv_create_db_on_aws.sh
> >>> > mv_db_size_on_aws.sh mv_delete_db_on_aws.sh
mv_load_to_aws.sh
> >>> >
> >>> >
> >>> > On Tue, Dec 1, 2020 at 2:44 PM Tatiana Burek via RT
> >>> > <met_help at ucar.edu>
> >>> > wrote:
> >>> >
> >>> > > Edward
> >>> > >
> >>> > > I run a few butch jobs from my laptop and made sure that
> >>> > > METviewer
> >>> > > code
> >>> > > and scripts working as expected: I was able to produce and
copy
> >>> > > plots.
> >>> > > I also got an error like the one that you reported but only
> >>> > > when
> >>> > > METviewer
> >>> > > did not generate any plots and the script couldn't copy
empty
> >>> > > directory.
> >>> > >
> >>> > > During my testing I noticed a few batch jobs that where
> >>> > > started by
> >>> > > you
> >>> > > and were handing. Each of these jobs started a database
query
> >>> > > and
> >>> > > they also
> >>> > > were piling up and handing. They all were waiting for the
> >>> > > indexing
> >>> > > that
> >>> > > takes place after the data loading.
> >>> > > I had to kill all database and batch processes in order to
move
> >>> > > forward
> >>> > > with the testing.
> >>> > > To avoid this gridlock you could:
> >>> > > - in your loading xml use <drop_indexes>false</drop_indexes>
> >>> > > <apply_indexes>false</apply_indexes>
> >>> > >
> >>> > >
> >>> > > - do not start a few parallel batch jobs
> >>> > >
> >>> > > I will add a info message to the script that would explain
why
> >>> > > the
> >>> > > error
> >>> > > occurs.
> >>> > > For now, could you test batch with some simple XML that
produce
> >>> > > at
> >>> > > least
> >>> > > one image.
> >>> > >
> >>> > > Tatiana
> >>> > >
> >>> > >
> >>> > >
> >>> > >
> >>> > >
> >>> > >
> >>> > >
> >>> > > On Tue Dec 01 11:35:03 2020, jpresto wrote:
> >>> > > > Hi Edward.
> >>> > > >
> >>> > > > After talking with Tatiana, it sounds like I am
misinformed
> >>> > > > and the
> >>> > > > scripts
> >>> > > > you are using were, in fact, developed by our METviewer
team.
> >>> > > > My
> >>> > > > apologies. METplus has a lot of components, and while I
> >>> > > > thought I
> >>> > > > had
> >>> > > > a
> >>> > > > good understanding of who did what, it turns out, I did
not.
> >>> > > > Tatiana
> >>> > > > is
> >>> > > > working on this issue and will follow up once she knows
more.
> >>> > > > Thank
> >>> > > > you
> >>> > > > for your patience and understanding.
> >>> > > >
> >>> > > > Julie
> >>> > > >
> >>> > > > On Tue, Dec 1, 2020 at 11:22 AM Julie Prestopnik
> >>> > > > <jpresto at ucar.edu
> >>> >
> >>> > > > wrote:
> >>> > > >
> >>> > > > > Thanks, Edward. I directed you to Mallory and Perry
> >>> > > > > because I
> >>> > > > > thought you
> >>> > > > > might be using scripts from them. We did not develop
and
> >>> > > > > have
> >>> > > > > not
> >>> > > > > used
> >>> > > > > those scripts. We have no experience interacting with
AWS
> >>> > > > > and
> >>> > > > > WCOSS,
> >>> > > > > so
> >>> > > > > they seemed best suited to answer your questions related
to
> >>> > > > > that
> >>> > > > > process.
> >>> > > > > However, as Tatiana pointed out, perhaps the security
> >>> > > > > changes on
> >>> > > > > the
> >>> > > > > AWS
> >>> > > > > server have something to do with what you are
experiencing.
> >>> > > > > If
> >>> > > > > that
> >>> > > > > turns
> >>> > > > > out not to be the case, Mallory and Perry would be best
> >>> > > > > able to
> >>> > > > > help
> >>> > > > > you.
> >>> > > > > If they are unable to help, please let us know, and we
can
> >>> > > > > see
> >>> > > > > what
> >>> > > > > they
> >>> > > > > tried and how we can help you get this problem figured
out.
> >>> > > > >
> >>> > > > > Julie
> >>> > > > >
> >>> > > > > On Tue, Dec 1, 2020 at 11:15 AM Edward Strobach - NOAA
> >>> > > > > Affiliate
> >>> > > > > via
> >>> > > > > RT <
> >>> > > > > met_help at ucar.edu> wrote:
> >>> > > > >
> >>> > > > >>
> >>> > > > >> <URL:
> >>> > > > >> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
>
> >>> > > > >>
> >>> > > > >> Hi Julie,
> >>> > > > >>
> >>> > > > >> I'm actually using the scripts located here:
> >>> > > > >>
> >>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts.
> >>> > > > >> These
> >>> > > > >> were scripts that I either got from Mallory or Perry a
> >>> > > > >> while
> >>> > > > >> back -
> >>> > > > >> around
> >>> > > > >> May.
> >>> > > > >>
> >>> > > > >> On Tue, Dec 1, 2020 at 12:59 PM Julie Prestopnik via RT
<
> >>> > > > >> met_help at ucar.edu>
> >>> > > > >> wrote:
> >>> > > > >>
> >>> > > > >> > Hi Edward.
> >>> > > > >> >
> >>> > > > >> > Are you using the scripts that are located in
> >>> > > > >> >
/gpfs/hps3/emc/global/noscrub/Mallory.Row/VRFY/METviewer_AWS
> >>> > > > >> > and
> >>> > > > >> >
/gpfs/dell2/emc/verification/noscrub/emc.metplus/METviewer_AWS
> >>> > > > >> > for interaction between AWS and WCOSS?
> >>> > > > >> >
> >>> > > > >> > Thanks!
> >>> > > > >> >
> >>> > > > >> > Julie
> >>> > > > >> >
> >>> > > > >> > On Tue, Dec 1, 2020 at 10:49 AM Edward Strobach -
NOAA
> >>> > > > >> > Affiliate
> >>> > > > >> > via RT
> >>> > > > >> <
> >>> > > > >> > met_help at ucar.edu> wrote:
> >>> > > > >> >
> >>> > > > >> > >
> >>> > > > >> > > <URL:
> >>> > > > >> > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> >>> > > > >> > > >
> >>> > > > >> > >
> >>> > > > >> > > Perry does not do batch plotting with MET. I was
> >>> > > > >> > > under the
> >>> > > > >> > > impression
> >>> > > > >> > that
> >>> > > > >> > > the MET helpdesk helped with MET related issues.
> >>> > > > >> > >
> >>> > > > >> > > On Tue, Dec 1, 2020 at 12:13 PM Julie Prestopnik
via
> >>> > > > >> > > RT <
> >>> > > > >> > met_help at ucar.edu
> >>> > > > >> > > >
> >>> > > > >> > > wrote:
> >>> > > > >> > >
> >>> > > > >> > > > Hi Edward.
> >>> > > > >> > > >
> >>> > > > >> > > > Please consult with Mallory Row and Perry Shafran
> >>> > > > >> > > > for
> >>> > > > >> > > > assistance
> >>> > > > >> with
> >>> > > > >> > > this
> >>> > > > >> > > > issue. They are most familiar with these
> >>> > > > >> > > > capabilities and
> >>> > > > >> > > > will be
> >>> > > > >> best
> >>> > > > >> > > > able to assist you.
> >>> > > > >> > > >
> >>> > > > >> > > > Thanks!
> >>> > > > >> > > >
> >>> > > > >> > > > Julie
> >>> > > > >> > > >
> >>> > > > >> > > > On Tue, Dec 1, 2020 at 9:25 AM Edward Strobach -
> >>> > > > >> > > > NOAA
> >>> > > > >> > > > Affiliate via
> >>> > > > >> RT
> >>> > > > >> > <
> >>> > > > >> > > > met_help at ucar.edu> wrote:
> >>> > > > >> > > >
> >>> > > > >> > > > >
> >>> > > > >> > > > > Tue Dec 01 09:25:26 2020: Request 97687 was
acted
> >>> > > > >> > > > > upon.
> >>> > > > >> > > > > Transaction: Ticket created by
> >>> > > > >> > > > > edward.strobach at noaa.gov
> >>> > > > >> > > > > Queue: met_help
> >>> > > > >> > > > > Subject: can't copy plots over
> >>> > > > >> > > > > Owner: Nobody
> >>> > > > >> > > > > Requestors: edward.strobach at noaa.gov
> >>> > > > >> > > > > Status: new
> >>> > > > >> > > > > Ticket <URL:
> >>> > > > >> >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> >>> > > > >> > > >
> >>> > > > >> > > > >
> >>> > > > >> > > > >
> >>> > > > >> > > > > I'm trying to copy plots over to Dell from the
MET
> >>> > > > >> > > > > server.
> >>> > > > >> > > > > I
> >>> > > > >> would
> >>> > > > >> > > like
> >>> > > > >> > > > to
> >>> > > > >> > > > > store my plots in the usual place but there is
an
> >>> > > > >> > > > > scp
> >>> > > > >> > > > > error
> >>> > > > >> > > > > which
> >>> > > > >> I'm
> >>> > > > >> > > > > attaching below.
> >>> > > > >> > > > >
> >>> > > > >> > > > >
> >>> > > > >> > > > > AND BINARY h.fcst_var = 'PMTF'
> >>> > > > >> > > > > AND ld.stat_header_id = h.stat_header_id;
> >>> > > > >> > > > >
> >>> > > > >> > > > > Connection to 205.156.8.85 closed by remote
host.
> >>> > > > >> > > > > CALLING: scp edward.strobach at 205.156.8.85
> >>> > > > >> > > > > ://data/mv_data//edward.strobach/plots/*
> >>> > > > >> > > > >
> >>> > > > >> > > > >
> >>> > > > >> > > >
> >>> > > > >> > >
> >>> > > > >> >
> >>> > > > >>
> >>> > >
> >>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> >>> > > > >> > > > >
> >>> > > > >> > > > > You are accessing a U.S. Government information
> >>> > > > >> > > > > system, which includes: 1) this computer, 2)
this
> >>> > > > >> > > > > computer network, 3) all computers connected to
> >>> > > > >> > > > > this
> >>> > > > >> > > > > network, and 4) all devices and storage media
> >>> > > > >> > > > > attached
> >>> > > > >> > > > > to this network or to a computer on this
network.
> >>> > > > >> > > > > You
> >>> > > > >> > > > > understand and consent to the following: you
may
> >>> > > > >> > > > > access this information system for authorized
use
> >>> > > > >> > > > > only;
> >>> > > > >> > > > > you have no reasonable expectation of privacy
> >>> > > > >> > > > > regarding
> >>> > > > >> > > > > any communication of data transiting or stored
on
> >>> > > > >> > > > > this
> >>> > > > >> > > > > information system; at any time and for any
lawful
> >>> > > > >> > > > > Government purpose, the Government may monitor,
> >>> > > > >> > > > > intercept, and search and seize any
communication
> >>> > > > >> > > > > or
> >>> > > > >> > > > > data transiting or stored on this information
> >>> > > > >> > > > > system;
> >>> > > > >> > > > > and
> >>> > > > >> > > > > any communications or data transiting or stored
on
> >>> > > > >> > > > > this
> >>> > > > >> > > > > information system may be disclosed or used for
> >>> > > > >> > > > > any
> >>> > > > >> > > > > lawful Government purpose.
> >>> > > > >> > > > >
> >>> > > > >> > > > > scp: //data/mv_data//edward.strobach/plots/*:
No
> >>> > > > >> > > > > such
> >>> > > > >> > > > > file
> >>> > > > >> > > > > or
> >>> > > > >> > directory
> >>> > > > >> > > > > ERROR: Command returned with non-zero status
(1):
> >>> > > > >> > > > > scp
> >>> > > > >> > edward.strobach@
> >>> > > > >> > > > >
205.156.8.85://data/mv_data//edward.strobach/plots/*
> >>> > > > >> > > > >
> >>> > > > >> > > > >
> >>> > > > >> > > >
> >>> > > > >> > >
> >>> > > > >> >
> >>> > > > >>
> >>> > >
> >>>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> >>> > > > >> > > > > ++ printf '\033]0;%s@%s:%s\007' Edward.Strobach
> >>> > > > >> > > > > v71a1
> >>> > > > >> > > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> >>> > > > >> > > > >
> >>> > > > >> > > > >
> >>> > > > >> > > > > NOTE: nothing about this set-up has changed
other
> >>> > > > >> > > > > than
> >>> > > > >> > > > > the
> >>> > > > >> directory
> >>> > > > >> > > > path
> >>> > > > >> > > > > onto dell. The path has been created so it
should
> >>> > > > >> > > > > support
> >>> > > > >> > > > > the
> >>> > > > >> > transfer
> >>> > > > >> > > > of
> >>> > > > >> > > > > plots generated on the MET server. The
preceding
> >>> > > > >> > > > > information
> >>> > > > >> > indicates
> >>> > > > >> > > > > that the plot was processed successfully. It's
> >>> > > > >> > > > > not
> >>> > > > >> > > > > clear
> >>> > > > >> > > > > why I
> >>> > > > >> can't
> >>> > > > >> > > > > transfer my plots over.
> >>> > > > >> > > > > --
> >>> > > > >> > > > > Edward Strobach
> >>> > > > >> > > > > EMC/NCEP/NWS/
> >>> > > > >> > > > > IMSG Contractor
> >>> > > > >> > > > > Cubicle#: 2029
> >>> > > > >> > > > > 301-683-3717
> >>> > > > >> > > > >
> >>> > > > >> > > > >
> >>> > > > >> > > >
> >>> > > > >> > > > --
> >>> > > > >> > > > Julie Prestopnik (she/her)
> >>> > > > >> > > > Software Engineer
> >>> > > > >> > > > National Center for Atmospheric Research
> >>> > > > >> > > > Research Applications Laboratory
> >>> > > > >> > > > Email: jpresto at ucar.edu
> >>> > > > >> > > >
> >>> > > > >> > > > My working day may not be your working day.
Please
> >>> > > > >> > > > do not
> >>> > > > >> > > > feel
> >>> > > > >> obliged
> >>> > > > >> > > to
> >>> > > > >> > > > reply to this email outside of your normal
working
> >>> > > > >> > > > hours.
> >>> > > > >> > > >
> >>> > > > >> > > >
> >>> > > > >> > >
> >>> > > > >> > > --
> >>> > > > >> > > Edward Strobach
> >>> > > > >> > > EMC/NCEP/NWS/
> >>> > > > >> > > IMSG Contractor
> >>> > > > >> > > Cubicle#: 2029
> >>> > > > >> > > 301-683-3717
> >>> > > > >> > >
> >>> > > > >> > >
> >>> > > > >> >
> >>> > > > >> > --
> >>> > > > >> > Julie Prestopnik (she/her)
> >>> > > > >> > Software Engineer
> >>> > > > >> > National Center for Atmospheric Research
> >>> > > > >> > Research Applications Laboratory
> >>> > > > >> > Email: jpresto at ucar.edu
> >>> > > > >> >
> >>> > > > >> > My working day may not be your working day. Please
do
> >>> > > > >> > not
> >>> > > > >> > feel
> >>> > > > >> > obliged
> >>> > > > >> to
> >>> > > > >> > reply to this email outside of your normal working
> >>> > > > >> > hours.
> >>> > > > >> >
> >>> > > > >> >
> >>> > > > >>
> >>> > > > >> --
> >>> > > > >> Edward Strobach
> >>> > > > >> EMC/NCEP/NWS/
> >>> > > > >> IMSG Contractor
> >>> > > > >> Cubicle#: 2029
> >>> > > > >> 301-683-3717
> >>> > > > >>
> >>> > > > >>
> >>> > > > >
> >>> > > > > --
> >>> > > > > Julie Prestopnik (she/her)
> >>> > > > > Software Engineer
> >>> > > > > National Center for Atmospheric Research
> >>> > > > > Research Applications Laboratory
> >>> > > > > Email: jpresto at ucar.edu
> >>> > > > >
> >>> > > > > My working day may not be your working day. Please do
not
> >>> > > > > feel
> >>> > > > > obliged to
> >>> > > > > reply to this email outside of your normal working
hours.
> >>> > > > >
> >>> > >
> >>> > >
> >>> > >
> >>> > >
> >>>
> >>>
> >>>
> >>>
> >>
> >> --
> >> Edward Strobach
> >> EMC/NCEP/NWS/
> >> IMSG Contractor
> >> Cubicle#: 2029
> >> 301-683-3717
> >>
> >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
------------------------------------------------
Subject: can't copy plots over
From: Edward Strobach - NOAA Affiliate
Time: Wed Dec 02 08:51:45 2020
I see this when running ps -ef | grep edward
Edward.+ 115762 154642 0 15:48 pts/31 00:00:00 grep --color=auto
edward
The numbers there change every time I run this step.
pkill -9 -u `id -u edward.strobach` does not seem to work.
On Wed, Dec 2, 2020 at 9:34 AM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> You can kill your processes by :
> - collecting what is running:
> ps -ef | grep edward
> - killing each process
> kill -9 2574
> kill -9 2594
> or all your processes:
> pkill -9 -u `id -u edward.strobach`
>
> Right now you have 4 loading scripts and 10 batch scripts running at
the
> same time which creates a gridlock and none of the processes are
getting
> executes.
>
> Tatiana
>
> On Wed Dec 02 06:32:23 2020, edward.strobach at noaa.gov wrote:
> > I came across this issue again this morning. Is there a way to
stop
> > certain processes and start fresh from my end?
> >
> > On Tue, Dec 1, 2020 at 5:14 PM Edward Strobach - NOAA Affiliate <
> > edward.strobach at noaa.gov> wrote:
> >
> > > maybe there is a script that allows me to check what is going on
with
> > > my
> > > database? A lot of these issues I've come across happen because
of
> > > things
> > > I'm unaware of or things that go on simultaneously that causes
> > > confusion in
> > > the processing. Right now I have a plotting script set that
> > > simultaneously
> > > submits batch jobs for time series statistics, Taylor diagrams,
and
> > > CSI. I
> > > haven't seen too many issues when I do that, but would it make
sense
> > > to
> > > insert a while loop after the first submission that uses a
script to
> > > check
> > > the activity within my database? In other words, I would use a
> > > script,
> > > perhaps one that has already been developed, to check the status
of
> > > processing before submitting the next batch job. Once the
processing
> > > for
> > > the first plotting steps have completed, then I would commence
to the
> > > next
> > > one.
> > >
> > > I guess check_my_db.sh would not do that?
> > >
> > >
> > >
> > >
> > > On Tue, Dec 1, 2020 at 4:27 PM Edward Strobach - NOAA Affiliate
<
> > > edward.strobach at noaa.gov> wrote:
> > >
> > >> are there youtube examples or a section in a technical manual
that
> > >> goes
> > >> over examples illustrating best practices related to this
process?
> > >>
> > >> On Tue, Dec 1, 2020 at 4:03 PM Tatiana Burek via RT
> > >> <met_help at ucar.edu>
> > >> wrote:
> > >>
> > >>> For your process you need only two scripts:
> > >>> mv_load_to_aws.sh
> > >>> mv_batch_on_aws.sh
> > >>> I don't think, that others would help you.
> > >>>
> > >>> The main work is in creating efficient XMLs for the each step.
> > >>> Like not recalculating indexes or performing
stat_header_db_check
> > >>> during
> > >>> data loading.
> > >>>
> > >>> Tatiana
> > >>>
> > >>>
> > >>> On Tue Dec 01 13:00:15 2020, edward.strobach at noaa.gov wrote:
> > >>> > Thanks. I just made those changes in the xml file, so I
'guess'
> > >>> > I'm
> > >>> > ready
> > >>> > to go.
> > >>> >
> > >>> > I should note that I have several steps that I run each day.
I
> > >>> > convert
> > >>> > files into netcdf files that can be read by MET, I generate
the
> > >>> > stat
> > >>> > files,
> > >>> > and I use the stat files in the MET batch plotting system.
I
> > >>> > have
> > >>> > actually
> > >>> > 5 total steps, but the plot batch system is really the only
one
> > >>> > that
> > >>> > gives
> > >>> > me trouble on a regular basis. I wish I knew how to reach a
> > >>> > point
> > >>> > where I
> > >>> > have a plotting package that would produce the plots that I
need
> > >>> > each
> > >>> > day.
> > >>> > I'm expected to hand the final product off in the near
future,
> > >>> > but
> > >>> > this
> > >>> > part has delayed me extensively. I know that I'm generating
a lot
> > >>> > of
> > >>> > plots
> > >>> > which could potentially overwhelm the system, but I'm
unaware how
> > >>> > others
> > >>> > work to circumvent that. I hope to move past this soon so I
can
> > >>> > continue
> > >>> > development work.
> > >>> >
> > >>> > There are several scripts that I don't use that might be
helpful.
> > >>> > Do
> > >>> > any
> > >>> > of these work to help me manage the system better?
> > >>> > check_my_db.sh mv_batch_on_aws.sh mv_create_db_on_aws.sh
> > >>> > mv_db_size_on_aws.sh mv_delete_db_on_aws.sh
mv_load_to_aws.sh
> > >>> >
> > >>> >
> > >>> > On Tue, Dec 1, 2020 at 2:44 PM Tatiana Burek via RT
> > >>> > <met_help at ucar.edu>
> > >>> > wrote:
> > >>> >
> > >>> > > Edward
> > >>> > >
> > >>> > > I run a few butch jobs from my laptop and made sure that
> > >>> > > METviewer
> > >>> > > code
> > >>> > > and scripts working as expected: I was able to produce and
copy
> > >>> > > plots.
> > >>> > > I also got an error like the one that you reported but
only
> > >>> > > when
> > >>> > > METviewer
> > >>> > > did not generate any plots and the script couldn't copy
empty
> > >>> > > directory.
> > >>> > >
> > >>> > > During my testing I noticed a few batch jobs that where
> > >>> > > started by
> > >>> > > you
> > >>> > > and were handing. Each of these jobs started a database
query
> > >>> > > and
> > >>> > > they also
> > >>> > > were piling up and handing. They all were waiting for the
> > >>> > > indexing
> > >>> > > that
> > >>> > > takes place after the data loading.
> > >>> > > I had to kill all database and batch processes in order to
move
> > >>> > > forward
> > >>> > > with the testing.
> > >>> > > To avoid this gridlock you could:
> > >>> > > - in your loading xml use
<drop_indexes>false</drop_indexes>
> > >>> > > <apply_indexes>false</apply_indexes>
> > >>> > >
> > >>> > >
> > >>> > > - do not start a few parallel batch jobs
> > >>> > >
> > >>> > > I will add a info message to the script that would explain
why
> > >>> > > the
> > >>> > > error
> > >>> > > occurs.
> > >>> > > For now, could you test batch with some simple XML that
produce
> > >>> > > at
> > >>> > > least
> > >>> > > one image.
> > >>> > >
> > >>> > > Tatiana
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > > On Tue Dec 01 11:35:03 2020, jpresto wrote:
> > >>> > > > Hi Edward.
> > >>> > > >
> > >>> > > > After talking with Tatiana, it sounds like I am
misinformed
> > >>> > > > and the
> > >>> > > > scripts
> > >>> > > > you are using were, in fact, developed by our METviewer
team.
> > >>> > > > My
> > >>> > > > apologies. METplus has a lot of components, and while I
> > >>> > > > thought I
> > >>> > > > had
> > >>> > > > a
> > >>> > > > good understanding of who did what, it turns out, I did
not.
> > >>> > > > Tatiana
> > >>> > > > is
> > >>> > > > working on this issue and will follow up once she knows
more.
> > >>> > > > Thank
> > >>> > > > you
> > >>> > > > for your patience and understanding.
> > >>> > > >
> > >>> > > > Julie
> > >>> > > >
> > >>> > > > On Tue, Dec 1, 2020 at 11:22 AM Julie Prestopnik
> > >>> > > > <jpresto at ucar.edu
> > >>> >
> > >>> > > > wrote:
> > >>> > > >
> > >>> > > > > Thanks, Edward. I directed you to Mallory and Perry
> > >>> > > > > because I
> > >>> > > > > thought you
> > >>> > > > > might be using scripts from them. We did not develop
and
> > >>> > > > > have
> > >>> > > > > not
> > >>> > > > > used
> > >>> > > > > those scripts. We have no experience interacting with
AWS
> > >>> > > > > and
> > >>> > > > > WCOSS,
> > >>> > > > > so
> > >>> > > > > they seemed best suited to answer your questions
related to
> > >>> > > > > that
> > >>> > > > > process.
> > >>> > > > > However, as Tatiana pointed out, perhaps the security
> > >>> > > > > changes on
> > >>> > > > > the
> > >>> > > > > AWS
> > >>> > > > > server have something to do with what you are
experiencing.
> > >>> > > > > If
> > >>> > > > > that
> > >>> > > > > turns
> > >>> > > > > out not to be the case, Mallory and Perry would be
best
> > >>> > > > > able to
> > >>> > > > > help
> > >>> > > > > you.
> > >>> > > > > If they are unable to help, please let us know, and we
can
> > >>> > > > > see
> > >>> > > > > what
> > >>> > > > > they
> > >>> > > > > tried and how we can help you get this problem figured
out.
> > >>> > > > >
> > >>> > > > > Julie
> > >>> > > > >
> > >>> > > > > On Tue, Dec 1, 2020 at 11:15 AM Edward Strobach - NOAA
> > >>> > > > > Affiliate
> > >>> > > > > via
> > >>> > > > > RT <
> > >>> > > > > met_help at ucar.edu> wrote:
> > >>> > > > >
> > >>> > > > >>
> > >>> > > > >> <URL:
> > >>> > > > >>
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
> > >>> > > > >>
> > >>> > > > >> Hi Julie,
> > >>> > > > >>
> > >>> > > > >> I'm actually using the scripts located here:
> > >>> > > > >>
> > >>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts.
> > >>> > > > >> These
> > >>> > > > >> were scripts that I either got from Mallory or Perry
a
> > >>> > > > >> while
> > >>> > > > >> back -
> > >>> > > > >> around
> > >>> > > > >> May.
> > >>> > > > >>
> > >>> > > > >> On Tue, Dec 1, 2020 at 12:59 PM Julie Prestopnik via
RT <
> > >>> > > > >> met_help at ucar.edu>
> > >>> > > > >> wrote:
> > >>> > > > >>
> > >>> > > > >> > Hi Edward.
> > >>> > > > >> >
> > >>> > > > >> > Are you using the scripts that are located in
> > >>> > > > >> >
> /gpfs/hps3/emc/global/noscrub/Mallory.Row/VRFY/METviewer_AWS
> > >>> > > > >> > and
> > >>> > > > >> >
> /gpfs/dell2/emc/verification/noscrub/emc.metplus/METviewer_AWS
> > >>> > > > >> > for interaction between AWS and WCOSS?
> > >>> > > > >> >
> > >>> > > > >> > Thanks!
> > >>> > > > >> >
> > >>> > > > >> > Julie
> > >>> > > > >> >
> > >>> > > > >> > On Tue, Dec 1, 2020 at 10:49 AM Edward Strobach -
NOAA
> > >>> > > > >> > Affiliate
> > >>> > > > >> > via RT
> > >>> > > > >> <
> > >>> > > > >> > met_help at ucar.edu> wrote:
> > >>> > > > >> >
> > >>> > > > >> > >
> > >>> > > > >> > > <URL:
> > >>> > > > >> > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > >>> > > > >> > > >
> > >>> > > > >> > >
> > >>> > > > >> > > Perry does not do batch plotting with MET. I was
> > >>> > > > >> > > under the
> > >>> > > > >> > > impression
> > >>> > > > >> > that
> > >>> > > > >> > > the MET helpdesk helped with MET related issues.
> > >>> > > > >> > >
> > >>> > > > >> > > On Tue, Dec 1, 2020 at 12:13 PM Julie Prestopnik
via
> > >>> > > > >> > > RT <
> > >>> > > > >> > met_help at ucar.edu
> > >>> > > > >> > > >
> > >>> > > > >> > > wrote:
> > >>> > > > >> > >
> > >>> > > > >> > > > Hi Edward.
> > >>> > > > >> > > >
> > >>> > > > >> > > > Please consult with Mallory Row and Perry
Shafran
> > >>> > > > >> > > > for
> > >>> > > > >> > > > assistance
> > >>> > > > >> with
> > >>> > > > >> > > this
> > >>> > > > >> > > > issue. They are most familiar with these
> > >>> > > > >> > > > capabilities and
> > >>> > > > >> > > > will be
> > >>> > > > >> best
> > >>> > > > >> > > > able to assist you.
> > >>> > > > >> > > >
> > >>> > > > >> > > > Thanks!
> > >>> > > > >> > > >
> > >>> > > > >> > > > Julie
> > >>> > > > >> > > >
> > >>> > > > >> > > > On Tue, Dec 1, 2020 at 9:25 AM Edward Strobach
-
> > >>> > > > >> > > > NOAA
> > >>> > > > >> > > > Affiliate via
> > >>> > > > >> RT
> > >>> > > > >> > <
> > >>> > > > >> > > > met_help at ucar.edu> wrote:
> > >>> > > > >> > > >
> > >>> > > > >> > > > >
> > >>> > > > >> > > > > Tue Dec 01 09:25:26 2020: Request 97687 was
acted
> > >>> > > > >> > > > > upon.
> > >>> > > > >> > > > > Transaction: Ticket created by
> > >>> > > > >> > > > > edward.strobach at noaa.gov
> > >>> > > > >> > > > > Queue: met_help
> > >>> > > > >> > > > > Subject: can't copy plots over
> > >>> > > > >> > > > > Owner: Nobody
> > >>> > > > >> > > > > Requestors: edward.strobach at noaa.gov
> > >>> > > > >> > > > > Status: new
> > >>> > > > >> > > > > Ticket <URL:
> > >>> > > > >> >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > >>> > > > >> > > >
> > >>> > > > >> > > > >
> > >>> > > > >> > > > >
> > >>> > > > >> > > > > I'm trying to copy plots over to Dell from
the MET
> > >>> > > > >> > > > > server.
> > >>> > > > >> > > > > I
> > >>> > > > >> would
> > >>> > > > >> > > like
> > >>> > > > >> > > > to
> > >>> > > > >> > > > > store my plots in the usual place but there
is an
> > >>> > > > >> > > > > scp
> > >>> > > > >> > > > > error
> > >>> > > > >> > > > > which
> > >>> > > > >> I'm
> > >>> > > > >> > > > > attaching below.
> > >>> > > > >> > > > >
> > >>> > > > >> > > > >
> > >>> > > > >> > > > > AND BINARY h.fcst_var = 'PMTF'
> > >>> > > > >> > > > > AND ld.stat_header_id = h.stat_header_id;
> > >>> > > > >> > > > >
> > >>> > > > >> > > > > Connection to 205.156.8.85 closed by remote
host.
> > >>> > > > >> > > > > CALLING: scp edward.strobach at 205.156.8.85
> > >>> > > > >> > > > > ://data/mv_data//edward.strobach/plots/*
> > >>> > > > >> > > > >
> > >>> > > > >> > > > >
> > >>> > > > >> > > >
> > >>> > > > >> > >
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > >
> > >>>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > >>> > > > >> > > > >
> > >>> > > > >> > > > > You are accessing a U.S. Government
information
> > >>> > > > >> > > > > system, which includes: 1) this computer, 2)
this
> > >>> > > > >> > > > > computer network, 3) all computers connected
to
> > >>> > > > >> > > > > this
> > >>> > > > >> > > > > network, and 4) all devices and storage media
> > >>> > > > >> > > > > attached
> > >>> > > > >> > > > > to this network or to a computer on this
network.
> > >>> > > > >> > > > > You
> > >>> > > > >> > > > > understand and consent to the following: you
may
> > >>> > > > >> > > > > access this information system for authorized
use
> > >>> > > > >> > > > > only;
> > >>> > > > >> > > > > you have no reasonable expectation of privacy
> > >>> > > > >> > > > > regarding
> > >>> > > > >> > > > > any communication of data transiting or
stored on
> > >>> > > > >> > > > > this
> > >>> > > > >> > > > > information system; at any time and for any
lawful
> > >>> > > > >> > > > > Government purpose, the Government may
monitor,
> > >>> > > > >> > > > > intercept, and search and seize any
communication
> > >>> > > > >> > > > > or
> > >>> > > > >> > > > > data transiting or stored on this information
> > >>> > > > >> > > > > system;
> > >>> > > > >> > > > > and
> > >>> > > > >> > > > > any communications or data transiting or
stored on
> > >>> > > > >> > > > > this
> > >>> > > > >> > > > > information system may be disclosed or used
for
> > >>> > > > >> > > > > any
> > >>> > > > >> > > > > lawful Government purpose.
> > >>> > > > >> > > > >
> > >>> > > > >> > > > > scp: //data/mv_data//edward.strobach/plots/*:
No
> > >>> > > > >> > > > > such
> > >>> > > > >> > > > > file
> > >>> > > > >> > > > > or
> > >>> > > > >> > directory
> > >>> > > > >> > > > > ERROR: Command returned with non-zero status
(1):
> > >>> > > > >> > > > > scp
> > >>> > > > >> > edward.strobach@
> > >>> > > > >> > > > > 205.156.8.85
> ://data/mv_data//edward.strobach/plots/*
> > >>> > > > >> > > > >
> > >>> > > > >> > > > >
> > >>> > > > >> > > >
> > >>> > > > >> > >
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > >
> > >>>
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > >>> > > > >> > > > > ++ printf '\033]0;%s@%s:%s\007'
Edward.Strobach
> > >>> > > > >> > > > > v71a1
> > >>> > > > >> > > > >
> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > >>> > > > >> > > > >
> > >>> > > > >> > > > >
> > >>> > > > >> > > > > NOTE: nothing about this set-up has changed
other
> > >>> > > > >> > > > > than
> > >>> > > > >> > > > > the
> > >>> > > > >> directory
> > >>> > > > >> > > > path
> > >>> > > > >> > > > > onto dell. The path has been created so it
should
> > >>> > > > >> > > > > support
> > >>> > > > >> > > > > the
> > >>> > > > >> > transfer
> > >>> > > > >> > > > of
> > >>> > > > >> > > > > plots generated on the MET server. The
preceding
> > >>> > > > >> > > > > information
> > >>> > > > >> > indicates
> > >>> > > > >> > > > > that the plot was processed successfully.
It's
> > >>> > > > >> > > > > not
> > >>> > > > >> > > > > clear
> > >>> > > > >> > > > > why I
> > >>> > > > >> can't
> > >>> > > > >> > > > > transfer my plots over.
> > >>> > > > >> > > > > --
> > >>> > > > >> > > > > Edward Strobach
> > >>> > > > >> > > > > EMC/NCEP/NWS/
> > >>> > > > >> > > > > IMSG Contractor
> > >>> > > > >> > > > > Cubicle#: 2029
> > >>> > > > >> > > > > 301-683-3717
> > >>> > > > >> > > > >
> > >>> > > > >> > > > >
> > >>> > > > >> > > >
> > >>> > > > >> > > > --
> > >>> > > > >> > > > Julie Prestopnik (she/her)
> > >>> > > > >> > > > Software Engineer
> > >>> > > > >> > > > National Center for Atmospheric Research
> > >>> > > > >> > > > Research Applications Laboratory
> > >>> > > > >> > > > Email: jpresto at ucar.edu
> > >>> > > > >> > > >
> > >>> > > > >> > > > My working day may not be your working day.
Please
> > >>> > > > >> > > > do not
> > >>> > > > >> > > > feel
> > >>> > > > >> obliged
> > >>> > > > >> > > to
> > >>> > > > >> > > > reply to this email outside of your normal
working
> > >>> > > > >> > > > hours.
> > >>> > > > >> > > >
> > >>> > > > >> > > >
> > >>> > > > >> > >
> > >>> > > > >> > > --
> > >>> > > > >> > > Edward Strobach
> > >>> > > > >> > > EMC/NCEP/NWS/
> > >>> > > > >> > > IMSG Contractor
> > >>> > > > >> > > Cubicle#: 2029
> > >>> > > > >> > > 301-683-3717
> > >>> > > > >> > >
> > >>> > > > >> > >
> > >>> > > > >> >
> > >>> > > > >> > --
> > >>> > > > >> > Julie Prestopnik (she/her)
> > >>> > > > >> > Software Engineer
> > >>> > > > >> > National Center for Atmospheric Research
> > >>> > > > >> > Research Applications Laboratory
> > >>> > > > >> > Email: jpresto at ucar.edu
> > >>> > > > >> >
> > >>> > > > >> > My working day may not be your working day. Please
do
> > >>> > > > >> > not
> > >>> > > > >> > feel
> > >>> > > > >> > obliged
> > >>> > > > >> to
> > >>> > > > >> > reply to this email outside of your normal working
> > >>> > > > >> > hours.
> > >>> > > > >> >
> > >>> > > > >> >
> > >>> > > > >>
> > >>> > > > >> --
> > >>> > > > >> Edward Strobach
> > >>> > > > >> EMC/NCEP/NWS/
> > >>> > > > >> IMSG Contractor
> > >>> > > > >> Cubicle#: 2029
> > >>> > > > >> 301-683-3717
> > >>> > > > >>
> > >>> > > > >>
> > >>> > > > >
> > >>> > > > > --
> > >>> > > > > Julie Prestopnik (she/her)
> > >>> > > > > Software Engineer
> > >>> > > > > National Center for Atmospheric Research
> > >>> > > > > Research Applications Laboratory
> > >>> > > > > Email: jpresto at ucar.edu
> > >>> > > > >
> > >>> > > > > My working day may not be your working day. Please do
not
> > >>> > > > > feel
> > >>> > > > > obliged to
> > >>> > > > > reply to this email outside of your normal working
hours.
> > >>> > > > >
> > >>> > >
> > >>> > >
> > >>> > >
> > >>> > >
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> > >> --
> > >> Edward Strobach
> > >> EMC/NCEP/NWS/
> > >> IMSG Contractor
> > >> Cubicle#: 2029
> > >> 301-683-3717
> > >>
> > >
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
>
>
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: can't copy plots over
From: Julie Prestopnik
Time: Wed Dec 02 09:13:06 2020
Hi Edward.
Because capitals matter when running "grep" and because WCOSS uses
capital
letters for our username you can run either of the following instead:
ps -ef | grep -i edward (the "-i" tells grep to ignore case)
ps -ef | grep Edward
The second column will list the process id and the third column will
list
the parent process id.
The same issue exists with running pkill on WCOSS. You can run pkill
on
WCOSS using the following, however, I don't recommend it and I'll
explain
below:
pkill -9 -u `id -u Edward.Strobach`
First, you'd need to make sure you're logged in appropriately and be
on the
same node (not sure if that is the correct terminology) as the one
running
your processes. For example, I ran a simple check to test this. I
ran
"top" when I was logged into "Julie.Prestopnik@*v71a3*", but my second
log
in was on "Julie.Prestopnik@*v72a1*". v72a1 did not recognize "top"
as
running, which I could see when running "ps -ef | grep Julie". When I
ran
"ssh v71a3" and then reran "ps -ef | grep Julie", I could then see
"top"
was running. Running "pkill -9 -u `id -u Julie.Presotpnik`" also
closed my
connection to WCOSS, so if you don't want your connection to WCOSS
killed,
which I didn't, I would just kill the processes individually using the
kill
command with the process number.
I hope that helps!
Julie
On Wed, Dec 2, 2020 at 8:51 AM Edward Strobach - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
>
> I see this when running ps -ef | grep edward
> Edward.+ 115762 154642 0 15:48 pts/31 00:00:00 grep --color=auto
edward
>
> The numbers there change every time I run this step.
>
> pkill -9 -u `id -u edward.strobach` does not seem to work.
>
> On Wed, Dec 2, 2020 at 9:34 AM Tatiana Burek via RT
<met_help at ucar.edu>
> wrote:
>
> > You can kill your processes by :
> > - collecting what is running:
> > ps -ef | grep edward
> > - killing each process
> > kill -9 2574
> > kill -9 2594
> > or all your processes:
> > pkill -9 -u `id -u edward.strobach`
> >
> > Right now you have 4 loading scripts and 10 batch scripts running
at the
> > same time which creates a gridlock and none of the processes are
getting
> > executes.
> >
> > Tatiana
> >
> > On Wed Dec 02 06:32:23 2020, edward.strobach at noaa.gov wrote:
> > > I came across this issue again this morning. Is there a way to
stop
> > > certain processes and start fresh from my end?
> > >
> > > On Tue, Dec 1, 2020 at 5:14 PM Edward Strobach - NOAA Affiliate
<
> > > edward.strobach at noaa.gov> wrote:
> > >
> > > > maybe there is a script that allows me to check what is going
on with
> > > > my
> > > > database? A lot of these issues I've come across happen
because of
> > > > things
> > > > I'm unaware of or things that go on simultaneously that causes
> > > > confusion in
> > > > the processing. Right now I have a plotting script set that
> > > > simultaneously
> > > > submits batch jobs for time series statistics, Taylor
diagrams, and
> > > > CSI. I
> > > > haven't seen too many issues when I do that, but would it make
sense
> > > > to
> > > > insert a while loop after the first submission that uses a
script to
> > > > check
> > > > the activity within my database? In other words, I would use
a
> > > > script,
> > > > perhaps one that has already been developed, to check the
status of
> > > > processing before submitting the next batch job. Once the
processing
> > > > for
> > > > the first plotting steps have completed, then I would commence
to the
> > > > next
> > > > one.
> > > >
> > > > I guess check_my_db.sh would not do that?
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, Dec 1, 2020 at 4:27 PM Edward Strobach - NOAA
Affiliate <
> > > > edward.strobach at noaa.gov> wrote:
> > > >
> > > >> are there youtube examples or a section in a technical manual
that
> > > >> goes
> > > >> over examples illustrating best practices related to this
process?
> > > >>
> > > >> On Tue, Dec 1, 2020 at 4:03 PM Tatiana Burek via RT
> > > >> <met_help at ucar.edu>
> > > >> wrote:
> > > >>
> > > >>> For your process you need only two scripts:
> > > >>> mv_load_to_aws.sh
> > > >>> mv_batch_on_aws.sh
> > > >>> I don't think, that others would help you.
> > > >>>
> > > >>> The main work is in creating efficient XMLs for the each
step.
> > > >>> Like not recalculating indexes or performing
stat_header_db_check
> > > >>> during
> > > >>> data loading.
> > > >>>
> > > >>> Tatiana
> > > >>>
> > > >>>
> > > >>> On Tue Dec 01 13:00:15 2020, edward.strobach at noaa.gov wrote:
> > > >>> > Thanks. I just made those changes in the xml file, so I
'guess'
> > > >>> > I'm
> > > >>> > ready
> > > >>> > to go.
> > > >>> >
> > > >>> > I should note that I have several steps that I run each
day. I
> > > >>> > convert
> > > >>> > files into netcdf files that can be read by MET, I
generate the
> > > >>> > stat
> > > >>> > files,
> > > >>> > and I use the stat files in the MET batch plotting system.
I
> > > >>> > have
> > > >>> > actually
> > > >>> > 5 total steps, but the plot batch system is really the
only one
> > > >>> > that
> > > >>> > gives
> > > >>> > me trouble on a regular basis. I wish I knew how to reach
a
> > > >>> > point
> > > >>> > where I
> > > >>> > have a plotting package that would produce the plots that
I need
> > > >>> > each
> > > >>> > day.
> > > >>> > I'm expected to hand the final product off in the near
future,
> > > >>> > but
> > > >>> > this
> > > >>> > part has delayed me extensively. I know that I'm
generating a lot
> > > >>> > of
> > > >>> > plots
> > > >>> > which could potentially overwhelm the system, but I'm
unaware how
> > > >>> > others
> > > >>> > work to circumvent that. I hope to move past this soon so
I can
> > > >>> > continue
> > > >>> > development work.
> > > >>> >
> > > >>> > There are several scripts that I don't use that might be
helpful.
> > > >>> > Do
> > > >>> > any
> > > >>> > of these work to help me manage the system better?
> > > >>> > check_my_db.sh mv_batch_on_aws.sh mv_create_db_on_aws.sh
> > > >>> > mv_db_size_on_aws.sh mv_delete_db_on_aws.sh
mv_load_to_aws.sh
> > > >>> >
> > > >>> >
> > > >>> > On Tue, Dec 1, 2020 at 2:44 PM Tatiana Burek via RT
> > > >>> > <met_help at ucar.edu>
> > > >>> > wrote:
> > > >>> >
> > > >>> > > Edward
> > > >>> > >
> > > >>> > > I run a few butch jobs from my laptop and made sure that
> > > >>> > > METviewer
> > > >>> > > code
> > > >>> > > and scripts working as expected: I was able to produce
and copy
> > > >>> > > plots.
> > > >>> > > I also got an error like the one that you reported but
only
> > > >>> > > when
> > > >>> > > METviewer
> > > >>> > > did not generate any plots and the script couldn't copy
empty
> > > >>> > > directory.
> > > >>> > >
> > > >>> > > During my testing I noticed a few batch jobs that where
> > > >>> > > started by
> > > >>> > > you
> > > >>> > > and were handing. Each of these jobs started a database
query
> > > >>> > > and
> > > >>> > > they also
> > > >>> > > were piling up and handing. They all were waiting for
the
> > > >>> > > indexing
> > > >>> > > that
> > > >>> > > takes place after the data loading.
> > > >>> > > I had to kill all database and batch processes in order
to move
> > > >>> > > forward
> > > >>> > > with the testing.
> > > >>> > > To avoid this gridlock you could:
> > > >>> > > - in your loading xml use
<drop_indexes>false</drop_indexes>
> > > >>> > > <apply_indexes>false</apply_indexes>
> > > >>> > >
> > > >>> > >
> > > >>> > > - do not start a few parallel batch jobs
> > > >>> > >
> > > >>> > > I will add a info message to the script that would
explain why
> > > >>> > > the
> > > >>> > > error
> > > >>> > > occurs.
> > > >>> > > For now, could you test batch with some simple XML that
produce
> > > >>> > > at
> > > >>> > > least
> > > >>> > > one image.
> > > >>> > >
> > > >>> > > Tatiana
> > > >>> > >
> > > >>> > >
> > > >>> > >
> > > >>> > >
> > > >>> > >
> > > >>> > >
> > > >>> > >
> > > >>> > > On Tue Dec 01 11:35:03 2020, jpresto wrote:
> > > >>> > > > Hi Edward.
> > > >>> > > >
> > > >>> > > > After talking with Tatiana, it sounds like I am
misinformed
> > > >>> > > > and the
> > > >>> > > > scripts
> > > >>> > > > you are using were, in fact, developed by our
METviewer team.
> > > >>> > > > My
> > > >>> > > > apologies. METplus has a lot of components, and while
I
> > > >>> > > > thought I
> > > >>> > > > had
> > > >>> > > > a
> > > >>> > > > good understanding of who did what, it turns out, I
did not.
> > > >>> > > > Tatiana
> > > >>> > > > is
> > > >>> > > > working on this issue and will follow up once she
knows more.
> > > >>> > > > Thank
> > > >>> > > > you
> > > >>> > > > for your patience and understanding.
> > > >>> > > >
> > > >>> > > > Julie
> > > >>> > > >
> > > >>> > > > On Tue, Dec 1, 2020 at 11:22 AM Julie Prestopnik
> > > >>> > > > <jpresto at ucar.edu
> > > >>> >
> > > >>> > > > wrote:
> > > >>> > > >
> > > >>> > > > > Thanks, Edward. I directed you to Mallory and Perry
> > > >>> > > > > because I
> > > >>> > > > > thought you
> > > >>> > > > > might be using scripts from them. We did not
develop and
> > > >>> > > > > have
> > > >>> > > > > not
> > > >>> > > > > used
> > > >>> > > > > those scripts. We have no experience interacting
with AWS
> > > >>> > > > > and
> > > >>> > > > > WCOSS,
> > > >>> > > > > so
> > > >>> > > > > they seemed best suited to answer your questions
related to
> > > >>> > > > > that
> > > >>> > > > > process.
> > > >>> > > > > However, as Tatiana pointed out, perhaps the
security
> > > >>> > > > > changes on
> > > >>> > > > > the
> > > >>> > > > > AWS
> > > >>> > > > > server have something to do with what you are
experiencing.
> > > >>> > > > > If
> > > >>> > > > > that
> > > >>> > > > > turns
> > > >>> > > > > out not to be the case, Mallory and Perry would be
best
> > > >>> > > > > able to
> > > >>> > > > > help
> > > >>> > > > > you.
> > > >>> > > > > If they are unable to help, please let us know, and
we can
> > > >>> > > > > see
> > > >>> > > > > what
> > > >>> > > > > they
> > > >>> > > > > tried and how we can help you get this problem
figured out.
> > > >>> > > > >
> > > >>> > > > > Julie
> > > >>> > > > >
> > > >>> > > > > On Tue, Dec 1, 2020 at 11:15 AM Edward Strobach -
NOAA
> > > >>> > > > > Affiliate
> > > >>> > > > > via
> > > >>> > > > > RT <
> > > >>> > > > > met_help at ucar.edu> wrote:
> > > >>> > > > >
> > > >>> > > > >>
> > > >>> > > > >> <URL:
> > > >>> > > > >>
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
> > > >>> > > > >>
> > > >>> > > > >> Hi Julie,
> > > >>> > > > >>
> > > >>> > > > >> I'm actually using the scripts located here:
> > > >>> > > > >>
> > > >>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts.
> > > >>> > > > >> These
> > > >>> > > > >> were scripts that I either got from Mallory or
Perry a
> > > >>> > > > >> while
> > > >>> > > > >> back -
> > > >>> > > > >> around
> > > >>> > > > >> May.
> > > >>> > > > >>
> > > >>> > > > >> On Tue, Dec 1, 2020 at 12:59 PM Julie Prestopnik
via RT <
> > > >>> > > > >> met_help at ucar.edu>
> > > >>> > > > >> wrote:
> > > >>> > > > >>
> > > >>> > > > >> > Hi Edward.
> > > >>> > > > >> >
> > > >>> > > > >> > Are you using the scripts that are located in
> > > >>> > > > >> >
> > /gpfs/hps3/emc/global/noscrub/Mallory.Row/VRFY/METviewer_AWS
> > > >>> > > > >> > and
> > > >>> > > > >> >
> > /gpfs/dell2/emc/verification/noscrub/emc.metplus/METviewer_AWS
> > > >>> > > > >> > for interaction between AWS and WCOSS?
> > > >>> > > > >> >
> > > >>> > > > >> > Thanks!
> > > >>> > > > >> >
> > > >>> > > > >> > Julie
> > > >>> > > > >> >
> > > >>> > > > >> > On Tue, Dec 1, 2020 at 10:49 AM Edward Strobach -
NOAA
> > > >>> > > > >> > Affiliate
> > > >>> > > > >> > via RT
> > > >>> > > > >> <
> > > >>> > > > >> > met_help at ucar.edu> wrote:
> > > >>> > > > >> >
> > > >>> > > > >> > >
> > > >>> > > > >> > > <URL:
> > > >>> > > > >> > >
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > >>> > > > >> > > >
> > > >>> > > > >> > >
> > > >>> > > > >> > > Perry does not do batch plotting with MET. I
was
> > > >>> > > > >> > > under the
> > > >>> > > > >> > > impression
> > > >>> > > > >> > that
> > > >>> > > > >> > > the MET helpdesk helped with MET related
issues.
> > > >>> > > > >> > >
> > > >>> > > > >> > > On Tue, Dec 1, 2020 at 12:13 PM Julie
Prestopnik via
> > > >>> > > > >> > > RT <
> > > >>> > > > >> > met_help at ucar.edu
> > > >>> > > > >> > > >
> > > >>> > > > >> > > wrote:
> > > >>> > > > >> > >
> > > >>> > > > >> > > > Hi Edward.
> > > >>> > > > >> > > >
> > > >>> > > > >> > > > Please consult with Mallory Row and Perry
Shafran
> > > >>> > > > >> > > > for
> > > >>> > > > >> > > > assistance
> > > >>> > > > >> with
> > > >>> > > > >> > > this
> > > >>> > > > >> > > > issue. They are most familiar with these
> > > >>> > > > >> > > > capabilities and
> > > >>> > > > >> > > > will be
> > > >>> > > > >> best
> > > >>> > > > >> > > > able to assist you.
> > > >>> > > > >> > > >
> > > >>> > > > >> > > > Thanks!
> > > >>> > > > >> > > >
> > > >>> > > > >> > > > Julie
> > > >>> > > > >> > > >
> > > >>> > > > >> > > > On Tue, Dec 1, 2020 at 9:25 AM Edward
Strobach -
> > > >>> > > > >> > > > NOAA
> > > >>> > > > >> > > > Affiliate via
> > > >>> > > > >> RT
> > > >>> > > > >> > <
> > > >>> > > > >> > > > met_help at ucar.edu> wrote:
> > > >>> > > > >> > > >
> > > >>> > > > >> > > > >
> > > >>> > > > >> > > > > Tue Dec 01 09:25:26 2020: Request 97687 was
acted
> > > >>> > > > >> > > > > upon.
> > > >>> > > > >> > > > > Transaction: Ticket created by
> > > >>> > > > >> > > > > edward.strobach at noaa.gov
> > > >>> > > > >> > > > > Queue: met_help
> > > >>> > > > >> > > > > Subject: can't copy plots over
> > > >>> > > > >> > > > > Owner: Nobody
> > > >>> > > > >> > > > > Requestors: edward.strobach at noaa.gov
> > > >>> > > > >> > > > > Status: new
> > > >>> > > > >> > > > > Ticket <URL:
> > > >>> > > > >> >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > >>> > > > >> > > >
> > > >>> > > > >> > > > >
> > > >>> > > > >> > > > >
> > > >>> > > > >> > > > > I'm trying to copy plots over to Dell from
the MET
> > > >>> > > > >> > > > > server.
> > > >>> > > > >> > > > > I
> > > >>> > > > >> would
> > > >>> > > > >> > > like
> > > >>> > > > >> > > > to
> > > >>> > > > >> > > > > store my plots in the usual place but there
is an
> > > >>> > > > >> > > > > scp
> > > >>> > > > >> > > > > error
> > > >>> > > > >> > > > > which
> > > >>> > > > >> I'm
> > > >>> > > > >> > > > > attaching below.
> > > >>> > > > >> > > > >
> > > >>> > > > >> > > > >
> > > >>> > > > >> > > > > AND BINARY h.fcst_var = 'PMTF'
> > > >>> > > > >> > > > > AND ld.stat_header_id = h.stat_header_id;
> > > >>> > > > >> > > > >
> > > >>> > > > >> > > > > Connection to 205.156.8.85 closed by remote
host.
> > > >>> > > > >> > > > > CALLING: scp edward.strobach at 205.156.8.85
> > > >>> > > > >> > > > > ://data/mv_data//edward.strobach/plots/*
> > > >>> > > > >> > > > >
> > > >>> > > > >> > > > >
> > > >>> > > > >> > > >
> > > >>> > > > >> > >
> > > >>> > > > >> >
> > > >>> > > > >>
> > > >>> > >
> > > >>>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > >>> > > > >> > > > >
> > > >>> > > > >> > > > > You are accessing a U.S. Government
information
> > > >>> > > > >> > > > > system, which includes: 1) this computer,
2) this
> > > >>> > > > >> > > > > computer network, 3) all computers
connected to
> > > >>> > > > >> > > > > this
> > > >>> > > > >> > > > > network, and 4) all devices and storage
media
> > > >>> > > > >> > > > > attached
> > > >>> > > > >> > > > > to this network or to a computer on this
network.
> > > >>> > > > >> > > > > You
> > > >>> > > > >> > > > > understand and consent to the following:
you may
> > > >>> > > > >> > > > > access this information system for
authorized use
> > > >>> > > > >> > > > > only;
> > > >>> > > > >> > > > > you have no reasonable expectation of
privacy
> > > >>> > > > >> > > > > regarding
> > > >>> > > > >> > > > > any communication of data transiting or
stored on
> > > >>> > > > >> > > > > this
> > > >>> > > > >> > > > > information system; at any time and for any
lawful
> > > >>> > > > >> > > > > Government purpose, the Government may
monitor,
> > > >>> > > > >> > > > > intercept, and search and seize any
communication
> > > >>> > > > >> > > > > or
> > > >>> > > > >> > > > > data transiting or stored on this
information
> > > >>> > > > >> > > > > system;
> > > >>> > > > >> > > > > and
> > > >>> > > > >> > > > > any communications or data transiting or
stored on
> > > >>> > > > >> > > > > this
> > > >>> > > > >> > > > > information system may be disclosed or used
for
> > > >>> > > > >> > > > > any
> > > >>> > > > >> > > > > lawful Government purpose.
> > > >>> > > > >> > > > >
> > > >>> > > > >> > > > > scp:
//data/mv_data//edward.strobach/plots/*: No
> > > >>> > > > >> > > > > such
> > > >>> > > > >> > > > > file
> > > >>> > > > >> > > > > or
> > > >>> > > > >> > directory
> > > >>> > > > >> > > > > ERROR: Command returned with non-zero
status (1):
> > > >>> > > > >> > > > > scp
> > > >>> > > > >> > edward.strobach@
> > > >>> > > > >> > > > > 205.156.8.85
> > ://data/mv_data//edward.strobach/plots/*
> > > >>> > > > >> > > > >
> > > >>> > > > >> > > > >
> > > >>> > > > >> > > >
> > > >>> > > > >> > >
> > > >>> > > > >> >
> > > >>> > > > >>
> > > >>> > >
> > > >>>
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > >>> > > > >> > > > > ++ printf '\033]0;%s@%s:%s\007'
Edward.Strobach
> > > >>> > > > >> > > > > v71a1
> > > >>> > > > >> > > > >
> > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > > >>> > > > >> > > > >
> > > >>> > > > >> > > > >
> > > >>> > > > >> > > > > NOTE: nothing about this set-up has
changed other
> > > >>> > > > >> > > > > than
> > > >>> > > > >> > > > > the
> > > >>> > > > >> directory
> > > >>> > > > >> > > > path
> > > >>> > > > >> > > > > onto dell. The path has been created so it
should
> > > >>> > > > >> > > > > support
> > > >>> > > > >> > > > > the
> > > >>> > > > >> > transfer
> > > >>> > > > >> > > > of
> > > >>> > > > >> > > > > plots generated on the MET server. The
preceding
> > > >>> > > > >> > > > > information
> > > >>> > > > >> > indicates
> > > >>> > > > >> > > > > that the plot was processed successfully.
It's
> > > >>> > > > >> > > > > not
> > > >>> > > > >> > > > > clear
> > > >>> > > > >> > > > > why I
> > > >>> > > > >> can't
> > > >>> > > > >> > > > > transfer my plots over.
> > > >>> > > > >> > > > > --
> > > >>> > > > >> > > > > Edward Strobach
> > > >>> > > > >> > > > > EMC/NCEP/NWS/
> > > >>> > > > >> > > > > IMSG Contractor
> > > >>> > > > >> > > > > Cubicle#: 2029
> > > >>> > > > >> > > > > 301-683-3717
> > > >>> > > > >> > > > >
> > > >>> > > > >> > > > >
> > > >>> > > > >> > > >
> > > >>> > > > >> > > > --
> > > >>> > > > >> > > > Julie Prestopnik (she/her)
> > > >>> > > > >> > > > Software Engineer
> > > >>> > > > >> > > > National Center for Atmospheric Research
> > > >>> > > > >> > > > Research Applications Laboratory
> > > >>> > > > >> > > > Email: jpresto at ucar.edu
> > > >>> > > > >> > > >
> > > >>> > > > >> > > > My working day may not be your working day.
Please
> > > >>> > > > >> > > > do not
> > > >>> > > > >> > > > feel
> > > >>> > > > >> obliged
> > > >>> > > > >> > > to
> > > >>> > > > >> > > > reply to this email outside of your normal
working
> > > >>> > > > >> > > > hours.
> > > >>> > > > >> > > >
> > > >>> > > > >> > > >
> > > >>> > > > >> > >
> > > >>> > > > >> > > --
> > > >>> > > > >> > > Edward Strobach
> > > >>> > > > >> > > EMC/NCEP/NWS/
> > > >>> > > > >> > > IMSG Contractor
> > > >>> > > > >> > > Cubicle#: 2029
> > > >>> > > > >> > > 301-683-3717
> > > >>> > > > >> > >
> > > >>> > > > >> > >
> > > >>> > > > >> >
> > > >>> > > > >> > --
> > > >>> > > > >> > Julie Prestopnik (she/her)
> > > >>> > > > >> > Software Engineer
> > > >>> > > > >> > National Center for Atmospheric Research
> > > >>> > > > >> > Research Applications Laboratory
> > > >>> > > > >> > Email: jpresto at ucar.edu
> > > >>> > > > >> >
> > > >>> > > > >> > My working day may not be your working day.
Please do
> > > >>> > > > >> > not
> > > >>> > > > >> > feel
> > > >>> > > > >> > obliged
> > > >>> > > > >> to
> > > >>> > > > >> > reply to this email outside of your normal
working
> > > >>> > > > >> > hours.
> > > >>> > > > >> >
> > > >>> > > > >> >
> > > >>> > > > >>
> > > >>> > > > >> --
> > > >>> > > > >> Edward Strobach
> > > >>> > > > >> EMC/NCEP/NWS/
> > > >>> > > > >> IMSG Contractor
> > > >>> > > > >> Cubicle#: 2029
> > > >>> > > > >> 301-683-3717
> > > >>> > > > >>
> > > >>> > > > >>
> > > >>> > > > >
> > > >>> > > > > --
> > > >>> > > > > Julie Prestopnik (she/her)
> > > >>> > > > > Software Engineer
> > > >>> > > > > National Center for Atmospheric Research
> > > >>> > > > > Research Applications Laboratory
> > > >>> > > > > Email: jpresto at ucar.edu
> > > >>> > > > >
> > > >>> > > > > My working day may not be your working day. Please
do not
> > > >>> > > > > feel
> > > >>> > > > > obliged to
> > > >>> > > > > reply to this email outside of your normal working
hours.
> > > >>> > > > >
> > > >>> > >
> > > >>> > >
> > > >>> > >
> > > >>> > >
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>
> > > >> --
> > > >> Edward Strobach
> > > >> EMC/NCEP/NWS/
> > > >> IMSG Contractor
> > > >> Cubicle#: 2029
> > > >> 301-683-3717
> > > >>
> > > >
> > > >
> > > > --
> > > > Edward Strobach
> > > > EMC/NCEP/NWS/
> > > > IMSG Contractor
> > > > Cubicle#: 2029
> > > > 301-683-3717
> > > >
> >
> >
> >
> >
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
--
Julie Prestopnik (she/her)
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: can't copy plots over
From: Edward Strobach - NOAA Affiliate
Time: Wed Dec 02 09:28:46 2020
Thanks. I was able to determine the processes using ps -ef | grep
Edward.Strobach
I did try pkill -9 -u `id -u Edward.Strobach` before sending your
email
which did log me out unfortunately.
I also noticed that kill -9 jobid also logs me out. I wish there was
a way
to distinguish between MET tasks v. other tasks. When I look at what
I
currently have I can't tell what is MET and what is not MET.
[Edward.Strobach at v71a2 ~]$ metcheck
root 76468 12071 0 16:25 ? 00:00:00 sshd:
Edward.Strobach
[priv]
Edward.+ 76482 76468 0 16:25 ? 00:00:00 sshd:
Edward.Strobach at pts
/19
Edward.+ 85480 82420 0 16:26 pts/19 00:00:00 grep --color=auto
Edward.Strobach
root 141334 12071 0 12:27 ? 00:00:00 sshd:
Edward.Strobach
[priv]
Edward.+ 141337 141334 0 12:27 ? 00:00:00 sshd:
Edward.Strobach at pts
/5
Note that metcheck is just an alias for ps -ef | grep Edward.Strobach
On Wed, Dec 2, 2020 at 11:13 AM Julie Prestopnik via RT
<met_help at ucar.edu>
wrote:
> Hi Edward.
>
> Because capitals matter when running "grep" and because WCOSS uses
capital
> letters for our username you can run either of the following
instead:
>
> ps -ef | grep -i edward (the "-i" tells grep to ignore case)
> ps -ef | grep Edward
>
> The second column will list the process id and the third column will
list
> the parent process id.
>
> The same issue exists with running pkill on WCOSS. You can run
pkill on
> WCOSS using the following, however, I don't recommend it and I'll
explain
> below:
>
> pkill -9 -u `id -u Edward.Strobach`
>
> First, you'd need to make sure you're logged in appropriately and be
on the
> same node (not sure if that is the correct terminology) as the one
running
> your processes. For example, I ran a simple check to test this. I
ran
> "top" when I was logged into "Julie.Prestopnik@*v71a3*", but my
second log
> in was on "Julie.Prestopnik@*v72a1*". v72a1 did not recognize "top"
as
> running, which I could see when running "ps -ef | grep Julie". When
I ran
> "ssh v71a3" and then reran "ps -ef | grep Julie", I could then see
"top"
> was running. Running "pkill -9 -u `id -u Julie.Presotpnik`" also
closed my
> connection to WCOSS, so if you don't want your connection to WCOSS
killed,
> which I didn't, I would just kill the processes individually using
the kill
> command with the process number.
>
> I hope that helps!
>
> Julie
>
> On Wed, Dec 2, 2020 at 8:51 AM Edward Strobach - NOAA Affiliate via
RT <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
> >
> > I see this when running ps -ef | grep edward
> > Edward.+ 115762 154642 0 15:48 pts/31 00:00:00 grep
--color=auto
> edward
> >
> > The numbers there change every time I run this step.
> >
> > pkill -9 -u `id -u edward.strobach` does not seem to work.
> >
> > On Wed, Dec 2, 2020 at 9:34 AM Tatiana Burek via RT
<met_help at ucar.edu>
> > wrote:
> >
> > > You can kill your processes by :
> > > - collecting what is running:
> > > ps -ef | grep edward
> > > - killing each process
> > > kill -9 2574
> > > kill -9 2594
> > > or all your processes:
> > > pkill -9 -u `id -u edward.strobach`
> > >
> > > Right now you have 4 loading scripts and 10 batch scripts
running at
> the
> > > same time which creates a gridlock and none of the processes are
> getting
> > > executes.
> > >
> > > Tatiana
> > >
> > > On Wed Dec 02 06:32:23 2020, edward.strobach at noaa.gov wrote:
> > > > I came across this issue again this morning. Is there a way
to stop
> > > > certain processes and start fresh from my end?
> > > >
> > > > On Tue, Dec 1, 2020 at 5:14 PM Edward Strobach - NOAA
Affiliate <
> > > > edward.strobach at noaa.gov> wrote:
> > > >
> > > > > maybe there is a script that allows me to check what is
going on
> with
> > > > > my
> > > > > database? A lot of these issues I've come across happen
because of
> > > > > things
> > > > > I'm unaware of or things that go on simultaneously that
causes
> > > > > confusion in
> > > > > the processing. Right now I have a plotting script set that
> > > > > simultaneously
> > > > > submits batch jobs for time series statistics, Taylor
diagrams, and
> > > > > CSI. I
> > > > > haven't seen too many issues when I do that, but would it
make
> sense
> > > > > to
> > > > > insert a while loop after the first submission that uses a
script
> to
> > > > > check
> > > > > the activity within my database? In other words, I would
use a
> > > > > script,
> > > > > perhaps one that has already been developed, to check the
status of
> > > > > processing before submitting the next batch job. Once the
> processing
> > > > > for
> > > > > the first plotting steps have completed, then I would
commence to
> the
> > > > > next
> > > > > one.
> > > > >
> > > > > I guess check_my_db.sh would not do that?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Dec 1, 2020 at 4:27 PM Edward Strobach - NOAA
Affiliate <
> > > > > edward.strobach at noaa.gov> wrote:
> > > > >
> > > > >> are there youtube examples or a section in a technical
manual that
> > > > >> goes
> > > > >> over examples illustrating best practices related to this
process?
> > > > >>
> > > > >> On Tue, Dec 1, 2020 at 4:03 PM Tatiana Burek via RT
> > > > >> <met_help at ucar.edu>
> > > > >> wrote:
> > > > >>
> > > > >>> For your process you need only two scripts:
> > > > >>> mv_load_to_aws.sh
> > > > >>> mv_batch_on_aws.sh
> > > > >>> I don't think, that others would help you.
> > > > >>>
> > > > >>> The main work is in creating efficient XMLs for the each
step.
> > > > >>> Like not recalculating indexes or performing
stat_header_db_check
> > > > >>> during
> > > > >>> data loading.
> > > > >>>
> > > > >>> Tatiana
> > > > >>>
> > > > >>>
> > > > >>> On Tue Dec 01 13:00:15 2020, edward.strobach at noaa.gov
wrote:
> > > > >>> > Thanks. I just made those changes in the xml file, so I
> 'guess'
> > > > >>> > I'm
> > > > >>> > ready
> > > > >>> > to go.
> > > > >>> >
> > > > >>> > I should note that I have several steps that I run each
day. I
> > > > >>> > convert
> > > > >>> > files into netcdf files that can be read by MET, I
generate the
> > > > >>> > stat
> > > > >>> > files,
> > > > >>> > and I use the stat files in the MET batch plotting
system. I
> > > > >>> > have
> > > > >>> > actually
> > > > >>> > 5 total steps, but the plot batch system is really the
only one
> > > > >>> > that
> > > > >>> > gives
> > > > >>> > me trouble on a regular basis. I wish I knew how to
reach a
> > > > >>> > point
> > > > >>> > where I
> > > > >>> > have a plotting package that would produce the plots
that I
> need
> > > > >>> > each
> > > > >>> > day.
> > > > >>> > I'm expected to hand the final product off in the near
future,
> > > > >>> > but
> > > > >>> > this
> > > > >>> > part has delayed me extensively. I know that I'm
generating a
> lot
> > > > >>> > of
> > > > >>> > plots
> > > > >>> > which could potentially overwhelm the system, but I'm
unaware
> how
> > > > >>> > others
> > > > >>> > work to circumvent that. I hope to move past this soon
so I
> can
> > > > >>> > continue
> > > > >>> > development work.
> > > > >>> >
> > > > >>> > There are several scripts that I don't use that might be
> helpful.
> > > > >>> > Do
> > > > >>> > any
> > > > >>> > of these work to help me manage the system better?
> > > > >>> > check_my_db.sh mv_batch_on_aws.sh
mv_create_db_on_aws.sh
> > > > >>> > mv_db_size_on_aws.sh mv_delete_db_on_aws.sh
> mv_load_to_aws.sh
> > > > >>> >
> > > > >>> >
> > > > >>> > On Tue, Dec 1, 2020 at 2:44 PM Tatiana Burek via RT
> > > > >>> > <met_help at ucar.edu>
> > > > >>> > wrote:
> > > > >>> >
> > > > >>> > > Edward
> > > > >>> > >
> > > > >>> > > I run a few butch jobs from my laptop and made sure
that
> > > > >>> > > METviewer
> > > > >>> > > code
> > > > >>> > > and scripts working as expected: I was able to produce
and
> copy
> > > > >>> > > plots.
> > > > >>> > > I also got an error like the one that you reported but
only
> > > > >>> > > when
> > > > >>> > > METviewer
> > > > >>> > > did not generate any plots and the script couldn't
copy empty
> > > > >>> > > directory.
> > > > >>> > >
> > > > >>> > > During my testing I noticed a few batch jobs that
where
> > > > >>> > > started by
> > > > >>> > > you
> > > > >>> > > and were handing. Each of these jobs started a
database query
> > > > >>> > > and
> > > > >>> > > they also
> > > > >>> > > were piling up and handing. They all were waiting for
the
> > > > >>> > > indexing
> > > > >>> > > that
> > > > >>> > > takes place after the data loading.
> > > > >>> > > I had to kill all database and batch processes in
order to
> move
> > > > >>> > > forward
> > > > >>> > > with the testing.
> > > > >>> > > To avoid this gridlock you could:
> > > > >>> > > - in your loading xml use
<drop_indexes>false</drop_indexes>
> > > > >>> > > <apply_indexes>false</apply_indexes>
> > > > >>> > >
> > > > >>> > >
> > > > >>> > > - do not start a few parallel batch jobs
> > > > >>> > >
> > > > >>> > > I will add a info message to the script that would
explain
> why
> > > > >>> > > the
> > > > >>> > > error
> > > > >>> > > occurs.
> > > > >>> > > For now, could you test batch with some simple XML
that
> produce
> > > > >>> > > at
> > > > >>> > > least
> > > > >>> > > one image.
> > > > >>> > >
> > > > >>> > > Tatiana
> > > > >>> > >
> > > > >>> > >
> > > > >>> > >
> > > > >>> > >
> > > > >>> > >
> > > > >>> > >
> > > > >>> > >
> > > > >>> > > On Tue Dec 01 11:35:03 2020, jpresto wrote:
> > > > >>> > > > Hi Edward.
> > > > >>> > > >
> > > > >>> > > > After talking with Tatiana, it sounds like I am
misinformed
> > > > >>> > > > and the
> > > > >>> > > > scripts
> > > > >>> > > > you are using were, in fact, developed by our
METviewer
> team.
> > > > >>> > > > My
> > > > >>> > > > apologies. METplus has a lot of components, and
while I
> > > > >>> > > > thought I
> > > > >>> > > > had
> > > > >>> > > > a
> > > > >>> > > > good understanding of who did what, it turns out, I
did
> not.
> > > > >>> > > > Tatiana
> > > > >>> > > > is
> > > > >>> > > > working on this issue and will follow up once she
knows
> more.
> > > > >>> > > > Thank
> > > > >>> > > > you
> > > > >>> > > > for your patience and understanding.
> > > > >>> > > >
> > > > >>> > > > Julie
> > > > >>> > > >
> > > > >>> > > > On Tue, Dec 1, 2020 at 11:22 AM Julie Prestopnik
> > > > >>> > > > <jpresto at ucar.edu
> > > > >>> >
> > > > >>> > > > wrote:
> > > > >>> > > >
> > > > >>> > > > > Thanks, Edward. I directed you to Mallory and
Perry
> > > > >>> > > > > because I
> > > > >>> > > > > thought you
> > > > >>> > > > > might be using scripts from them. We did not
develop and
> > > > >>> > > > > have
> > > > >>> > > > > not
> > > > >>> > > > > used
> > > > >>> > > > > those scripts. We have no experience interacting
with
> AWS
> > > > >>> > > > > and
> > > > >>> > > > > WCOSS,
> > > > >>> > > > > so
> > > > >>> > > > > they seemed best suited to answer your questions
related
> to
> > > > >>> > > > > that
> > > > >>> > > > > process.
> > > > >>> > > > > However, as Tatiana pointed out, perhaps the
security
> > > > >>> > > > > changes on
> > > > >>> > > > > the
> > > > >>> > > > > AWS
> > > > >>> > > > > server have something to do with what you are
> experiencing.
> > > > >>> > > > > If
> > > > >>> > > > > that
> > > > >>> > > > > turns
> > > > >>> > > > > out not to be the case, Mallory and Perry would be
best
> > > > >>> > > > > able to
> > > > >>> > > > > help
> > > > >>> > > > > you.
> > > > >>> > > > > If they are unable to help, please let us know,
and we
> can
> > > > >>> > > > > see
> > > > >>> > > > > what
> > > > >>> > > > > they
> > > > >>> > > > > tried and how we can help you get this problem
figured
> out.
> > > > >>> > > > >
> > > > >>> > > > > Julie
> > > > >>> > > > >
> > > > >>> > > > > On Tue, Dec 1, 2020 at 11:15 AM Edward Strobach -
NOAA
> > > > >>> > > > > Affiliate
> > > > >>> > > > > via
> > > > >>> > > > > RT <
> > > > >>> > > > > met_help at ucar.edu> wrote:
> > > > >>> > > > >
> > > > >>> > > > >>
> > > > >>> > > > >> <URL:
> > > > >>> > > > >>
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> >
> > > > >>> > > > >>
> > > > >>> > > > >> Hi Julie,
> > > > >>> > > > >>
> > > > >>> > > > >> I'm actually using the scripts located here:
> > > > >>> > > > >>
> > > > >>>
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts.
> > > > >>> > > > >> These
> > > > >>> > > > >> were scripts that I either got from Mallory or
Perry a
> > > > >>> > > > >> while
> > > > >>> > > > >> back -
> > > > >>> > > > >> around
> > > > >>> > > > >> May.
> > > > >>> > > > >>
> > > > >>> > > > >> On Tue, Dec 1, 2020 at 12:59 PM Julie Prestopnik
via RT
> <
> > > > >>> > > > >> met_help at ucar.edu>
> > > > >>> > > > >> wrote:
> > > > >>> > > > >>
> > > > >>> > > > >> > Hi Edward.
> > > > >>> > > > >> >
> > > > >>> > > > >> > Are you using the scripts that are located in
> > > > >>> > > > >> >
> > > /gpfs/hps3/emc/global/noscrub/Mallory.Row/VRFY/METviewer_AWS
> > > > >>> > > > >> > and
> > > > >>> > > > >> >
> > > /gpfs/dell2/emc/verification/noscrub/emc.metplus/METviewer_AWS
> > > > >>> > > > >> > for interaction between AWS and WCOSS?
> > > > >>> > > > >> >
> > > > >>> > > > >> > Thanks!
> > > > >>> > > > >> >
> > > > >>> > > > >> > Julie
> > > > >>> > > > >> >
> > > > >>> > > > >> > On Tue, Dec 1, 2020 at 10:49 AM Edward Strobach
- NOAA
> > > > >>> > > > >> > Affiliate
> > > > >>> > > > >> > via RT
> > > > >>> > > > >> <
> > > > >>> > > > >> > met_help at ucar.edu> wrote:
> > > > >>> > > > >> >
> > > > >>> > > > >> > >
> > > > >>> > > > >> > > <URL:
> > > > >>> > > > >> > >
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > > >>> > > > >> > > >
> > > > >>> > > > >> > >
> > > > >>> > > > >> > > Perry does not do batch plotting with MET. I
was
> > > > >>> > > > >> > > under the
> > > > >>> > > > >> > > impression
> > > > >>> > > > >> > that
> > > > >>> > > > >> > > the MET helpdesk helped with MET related
issues.
> > > > >>> > > > >> > >
> > > > >>> > > > >> > > On Tue, Dec 1, 2020 at 12:13 PM Julie
Prestopnik via
> > > > >>> > > > >> > > RT <
> > > > >>> > > > >> > met_help at ucar.edu
> > > > >>> > > > >> > > >
> > > > >>> > > > >> > > wrote:
> > > > >>> > > > >> > >
> > > > >>> > > > >> > > > Hi Edward.
> > > > >>> > > > >> > > >
> > > > >>> > > > >> > > > Please consult with Mallory Row and Perry
Shafran
> > > > >>> > > > >> > > > for
> > > > >>> > > > >> > > > assistance
> > > > >>> > > > >> with
> > > > >>> > > > >> > > this
> > > > >>> > > > >> > > > issue. They are most familiar with these
> > > > >>> > > > >> > > > capabilities and
> > > > >>> > > > >> > > > will be
> > > > >>> > > > >> best
> > > > >>> > > > >> > > > able to assist you.
> > > > >>> > > > >> > > >
> > > > >>> > > > >> > > > Thanks!
> > > > >>> > > > >> > > >
> > > > >>> > > > >> > > > Julie
> > > > >>> > > > >> > > >
> > > > >>> > > > >> > > > On Tue, Dec 1, 2020 at 9:25 AM Edward
Strobach -
> > > > >>> > > > >> > > > NOAA
> > > > >>> > > > >> > > > Affiliate via
> > > > >>> > > > >> RT
> > > > >>> > > > >> > <
> > > > >>> > > > >> > > > met_help at ucar.edu> wrote:
> > > > >>> > > > >> > > >
> > > > >>> > > > >> > > > >
> > > > >>> > > > >> > > > > Tue Dec 01 09:25:26 2020: Request 97687
was
> acted
> > > > >>> > > > >> > > > > upon.
> > > > >>> > > > >> > > > > Transaction: Ticket created by
> > > > >>> > > > >> > > > > edward.strobach at noaa.gov
> > > > >>> > > > >> > > > > Queue: met_help
> > > > >>> > > > >> > > > > Subject: can't copy plots over
> > > > >>> > > > >> > > > > Owner: Nobody
> > > > >>> > > > >> > > > > Requestors: edward.strobach at noaa.gov
> > > > >>> > > > >> > > > > Status: new
> > > > >>> > > > >> > > > > Ticket <URL:
> > > > >>> > > > >> >
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > > >>> > > > >> > > >
> > > > >>> > > > >> > > > >
> > > > >>> > > > >> > > > >
> > > > >>> > > > >> > > > > I'm trying to copy plots over to Dell
from the
> MET
> > > > >>> > > > >> > > > > server.
> > > > >>> > > > >> > > > > I
> > > > >>> > > > >> would
> > > > >>> > > > >> > > like
> > > > >>> > > > >> > > > to
> > > > >>> > > > >> > > > > store my plots in the usual place but
there is
> an
> > > > >>> > > > >> > > > > scp
> > > > >>> > > > >> > > > > error
> > > > >>> > > > >> > > > > which
> > > > >>> > > > >> I'm
> > > > >>> > > > >> > > > > attaching below.
> > > > >>> > > > >> > > > >
> > > > >>> > > > >> > > > >
> > > > >>> > > > >> > > > > AND BINARY h.fcst_var = 'PMTF'
> > > > >>> > > > >> > > > > AND ld.stat_header_id =
h.stat_header_id;
> > > > >>> > > > >> > > > >
> > > > >>> > > > >> > > > > Connection to 205.156.8.85 closed by
remote
> host.
> > > > >>> > > > >> > > > > CALLING: scp edward.strobach at 205.156.8.85
> > > > >>> > > > >> > > > > ://data/mv_data//edward.strobach/plots/*
> > > > >>> > > > >> > > > >
> > > > >>> > > > >> > > > >
> > > > >>> > > > >> > > >
> > > > >>> > > > >> > >
> > > > >>> > > > >> >
> > > > >>> > > > >>
> > > > >>> > >
> > > > >>>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > > >>> > > > >> > > > >
> > > > >>> > > > >> > > > > You are accessing a U.S. Government
information
> > > > >>> > > > >> > > > > system, which includes: 1) this computer,
2)
> this
> > > > >>> > > > >> > > > > computer network, 3) all computers
connected to
> > > > >>> > > > >> > > > > this
> > > > >>> > > > >> > > > > network, and 4) all devices and storage
media
> > > > >>> > > > >> > > > > attached
> > > > >>> > > > >> > > > > to this network or to a computer on this
> network.
> > > > >>> > > > >> > > > > You
> > > > >>> > > > >> > > > > understand and consent to the following:
you may
> > > > >>> > > > >> > > > > access this information system for
authorized
> use
> > > > >>> > > > >> > > > > only;
> > > > >>> > > > >> > > > > you have no reasonable expectation of
privacy
> > > > >>> > > > >> > > > > regarding
> > > > >>> > > > >> > > > > any communication of data transiting or
stored
> on
> > > > >>> > > > >> > > > > this
> > > > >>> > > > >> > > > > information system; at any time and for
any
> lawful
> > > > >>> > > > >> > > > > Government purpose, the Government may
monitor,
> > > > >>> > > > >> > > > > intercept, and search and seize any
> communication
> > > > >>> > > > >> > > > > or
> > > > >>> > > > >> > > > > data transiting or stored on this
information
> > > > >>> > > > >> > > > > system;
> > > > >>> > > > >> > > > > and
> > > > >>> > > > >> > > > > any communications or data transiting or
stored
> on
> > > > >>> > > > >> > > > > this
> > > > >>> > > > >> > > > > information system may be disclosed or
used for
> > > > >>> > > > >> > > > > any
> > > > >>> > > > >> > > > > lawful Government purpose.
> > > > >>> > > > >> > > > >
> > > > >>> > > > >> > > > > scp:
//data/mv_data//edward.strobach/plots/*: No
> > > > >>> > > > >> > > > > such
> > > > >>> > > > >> > > > > file
> > > > >>> > > > >> > > > > or
> > > > >>> > > > >> > directory
> > > > >>> > > > >> > > > > ERROR: Command returned with non-zero
status
> (1):
> > > > >>> > > > >> > > > > scp
> > > > >>> > > > >> > edward.strobach@
> > > > >>> > > > >> > > > > 205.156.8.85
> > > ://data/mv_data//edward.strobach/plots/*
> > > > >>> > > > >> > > > >
> > > > >>> > > > >> > > > >
> > > > >>> > > > >> > > >
> > > > >>> > > > >> > >
> > > > >>> > > > >> >
> > > > >>> > > > >>
> > > > >>> > >
> > > > >>>
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > > >>> > > > >> > > > > ++ printf '\033]0;%s@%s:%s\007'
Edward.Strobach
> > > > >>> > > > >> > > > > v71a1
> > > > >>> > > > >> > > > >
> > > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > > > >>> > > > >> > > > >
> > > > >>> > > > >> > > > >
> > > > >>> > > > >> > > > > NOTE: nothing about this set-up has
changed
> other
> > > > >>> > > > >> > > > > than
> > > > >>> > > > >> > > > > the
> > > > >>> > > > >> directory
> > > > >>> > > > >> > > > path
> > > > >>> > > > >> > > > > onto dell. The path has been created so
it
> should
> > > > >>> > > > >> > > > > support
> > > > >>> > > > >> > > > > the
> > > > >>> > > > >> > transfer
> > > > >>> > > > >> > > > of
> > > > >>> > > > >> > > > > plots generated on the MET server. The
> preceding
> > > > >>> > > > >> > > > > information
> > > > >>> > > > >> > indicates
> > > > >>> > > > >> > > > > that the plot was processed successfully.
It's
> > > > >>> > > > >> > > > > not
> > > > >>> > > > >> > > > > clear
> > > > >>> > > > >> > > > > why I
> > > > >>> > > > >> can't
> > > > >>> > > > >> > > > > transfer my plots over.
> > > > >>> > > > >> > > > > --
> > > > >>> > > > >> > > > > Edward Strobach
> > > > >>> > > > >> > > > > EMC/NCEP/NWS/
> > > > >>> > > > >> > > > > IMSG Contractor
> > > > >>> > > > >> > > > > Cubicle#: 2029
> > > > >>> > > > >> > > > > 301-683-3717
> > > > >>> > > > >> > > > >
> > > > >>> > > > >> > > > >
> > > > >>> > > > >> > > >
> > > > >>> > > > >> > > > --
> > > > >>> > > > >> > > > Julie Prestopnik (she/her)
> > > > >>> > > > >> > > > Software Engineer
> > > > >>> > > > >> > > > National Center for Atmospheric Research
> > > > >>> > > > >> > > > Research Applications Laboratory
> > > > >>> > > > >> > > > Email: jpresto at ucar.edu
> > > > >>> > > > >> > > >
> > > > >>> > > > >> > > > My working day may not be your working day.
> Please
> > > > >>> > > > >> > > > do not
> > > > >>> > > > >> > > > feel
> > > > >>> > > > >> obliged
> > > > >>> > > > >> > > to
> > > > >>> > > > >> > > > reply to this email outside of your normal
working
> > > > >>> > > > >> > > > hours.
> > > > >>> > > > >> > > >
> > > > >>> > > > >> > > >
> > > > >>> > > > >> > >
> > > > >>> > > > >> > > --
> > > > >>> > > > >> > > Edward Strobach
> > > > >>> > > > >> > > EMC/NCEP/NWS/
> > > > >>> > > > >> > > IMSG Contractor
> > > > >>> > > > >> > > Cubicle#: 2029
> > > > >>> > > > >> > > 301-683-3717
> > > > >>> > > > >> > >
> > > > >>> > > > >> > >
> > > > >>> > > > >> >
> > > > >>> > > > >> > --
> > > > >>> > > > >> > Julie Prestopnik (she/her)
> > > > >>> > > > >> > Software Engineer
> > > > >>> > > > >> > National Center for Atmospheric Research
> > > > >>> > > > >> > Research Applications Laboratory
> > > > >>> > > > >> > Email: jpresto at ucar.edu
> > > > >>> > > > >> >
> > > > >>> > > > >> > My working day may not be your working day.
Please do
> > > > >>> > > > >> > not
> > > > >>> > > > >> > feel
> > > > >>> > > > >> > obliged
> > > > >>> > > > >> to
> > > > >>> > > > >> > reply to this email outside of your normal
working
> > > > >>> > > > >> > hours.
> > > > >>> > > > >> >
> > > > >>> > > > >> >
> > > > >>> > > > >>
> > > > >>> > > > >> --
> > > > >>> > > > >> Edward Strobach
> > > > >>> > > > >> EMC/NCEP/NWS/
> > > > >>> > > > >> IMSG Contractor
> > > > >>> > > > >> Cubicle#: 2029
> > > > >>> > > > >> 301-683-3717
> > > > >>> > > > >>
> > > > >>> > > > >>
> > > > >>> > > > >
> > > > >>> > > > > --
> > > > >>> > > > > Julie Prestopnik (she/her)
> > > > >>> > > > > Software Engineer
> > > > >>> > > > > National Center for Atmospheric Research
> > > > >>> > > > > Research Applications Laboratory
> > > > >>> > > > > Email: jpresto at ucar.edu
> > > > >>> > > > >
> > > > >>> > > > > My working day may not be your working day.
Please do
> not
> > > > >>> > > > > feel
> > > > >>> > > > > obliged to
> > > > >>> > > > > reply to this email outside of your normal working
hours.
> > > > >>> > > > >
> > > > >>> > >
> > > > >>> > >
> > > > >>> > >
> > > > >>> > >
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>
> > > > >> --
> > > > >> Edward Strobach
> > > > >> EMC/NCEP/NWS/
> > > > >> IMSG Contractor
> > > > >> Cubicle#: 2029
> > > > >> 301-683-3717
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > Edward Strobach
> > > > > EMC/NCEP/NWS/
> > > > > IMSG Contractor
> > > > > Cubicle#: 2029
> > > > > 301-683-3717
> > > > >
> > >
> > >
> > >
> > >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
>
> --
> Julie Prestopnik (she/her)
> Software Engineer
> National Center for Atmospheric Research
> Research Applications Laboratory
> Email: jpresto at ucar.edu
>
> My working day may not be your working day. Please do not feel
obliged to
> reply to this email outside of your normal working hours.
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: can't copy plots over
From: Julie Prestopnik
Time: Wed Dec 02 09:42:20 2020
Hi Edward.
You should be able to distinguish what processes are MET and what are
not.
>From your "metcheck" results, it seems you have no MET processes
running.
I'll explain below.
[Edward.Strobach at v71a2 ~]$ metcheck
> root 76468 12071 0 16:25 ? 00:00:00 sshd:
Edward.Strobach
> [priv]
> Edward.+ 76482 76468 0 16:25 ? 00:00:00 sshd:
Edward.Strobach at pts
> /19
> Edward.+ 85480 82420 0 16:26 pts/19 00:00:00 grep --color=auto
> Edward.Strobach
> root 141334 12071 0 12:27 ? 00:00:00 sshd:
Edward.Strobach
> [priv]
> Edward.+ 141337 141334 0 12:27 ? 00:00:00 sshd:
Edward.Strobach at pts
> /5
All of the processes that start with "sshd" are your logins, so only
if you
ran "kill -9" on one of those would you get logged out. The "grep"
processes (3rd result) is your running of "ps -ef | grep
Edward.Strobach".
If there were MET processes running you would be able to see them
here.
The fact that there are none listed here could be because when you ran
"pkill -9 -u `id -u Edward.Strobach`", they were all killed, or
because
you're on a different node (v71a2) than the one you ran them on.
Julie
On Wed, Dec 2, 2020 at 9:28 AM Edward Strobach - NOAA Affiliate via RT
<
met_help at ucar.edu> wrote:
>
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
>
> Thanks. I was able to determine the processes using ps -ef | grep
> Edward.Strobach
>
> I did try pkill -9 -u `id -u Edward.Strobach` before sending your
email
> which did log me out unfortunately.
>
> I also noticed that kill -9 jobid also logs me out. I wish there
was a way
> to distinguish between MET tasks v. other tasks. When I look at
what I
> currently have I can't tell what is MET and what is not MET.
> [Edward.Strobach at v71a2 ~]$ metcheck
> root 76468 12071 0 16:25 ? 00:00:00 sshd:
Edward.Strobach
> [priv]
> Edward.+ 76482 76468 0 16:25 ? 00:00:00 sshd:
Edward.Strobach at pts
> /19
> Edward.+ 85480 82420 0 16:26 pts/19 00:00:00 grep --color=auto
> Edward.Strobach
> root 141334 12071 0 12:27 ? 00:00:00 sshd:
Edward.Strobach
> [priv]
> Edward.+ 141337 141334 0 12:27 ? 00:00:00 sshd:
Edward.Strobach at pts
> /5
>
> Note that metcheck is just an alias for ps -ef | grep
Edward.Strobach
>
> On Wed, Dec 2, 2020 at 11:13 AM Julie Prestopnik via RT
<met_help at ucar.edu
> >
> wrote:
>
> > Hi Edward.
> >
> > Because capitals matter when running "grep" and because WCOSS uses
> capital
> > letters for our username you can run either of the following
instead:
> >
> > ps -ef | grep -i edward (the "-i" tells grep to ignore case)
> > ps -ef | grep Edward
> >
> > The second column will list the process id and the third column
will list
> > the parent process id.
> >
> > The same issue exists with running pkill on WCOSS. You can run
pkill on
> > WCOSS using the following, however, I don't recommend it and I'll
explain
> > below:
> >
> > pkill -9 -u `id -u Edward.Strobach`
> >
> > First, you'd need to make sure you're logged in appropriately and
be on
> the
> > same node (not sure if that is the correct terminology) as the one
> running
> > your processes. For example, I ran a simple check to test this.
I ran
> > "top" when I was logged into "Julie.Prestopnik@*v71a3*", but my
second
> log
> > in was on "Julie.Prestopnik@*v72a1*". v72a1 did not recognize
"top" as
> > running, which I could see when running "ps -ef | grep Julie".
When I
> ran
> > "ssh v71a3" and then reran "ps -ef | grep Julie", I could then see
"top"
> > was running. Running "pkill -9 -u `id -u Julie.Presotpnik`" also
closed
> my
> > connection to WCOSS, so if you don't want your connection to WCOSS
> killed,
> > which I didn't, I would just kill the processes individually using
the
> kill
> > command with the process number.
> >
> > I hope that helps!
> >
> > Julie
> >
> > On Wed, Dec 2, 2020 at 8:51 AM Edward Strobach - NOAA Affiliate
via RT <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
> > >
> > > I see this when running ps -ef | grep edward
> > > Edward.+ 115762 154642 0 15:48 pts/31 00:00:00 grep
--color=auto
> > edward
> > >
> > > The numbers there change every time I run this step.
> > >
> > > pkill -9 -u `id -u edward.strobach` does not seem to work.
> > >
> > > On Wed, Dec 2, 2020 at 9:34 AM Tatiana Burek via RT
<met_help at ucar.edu
> >
> > > wrote:
> > >
> > > > You can kill your processes by :
> > > > - collecting what is running:
> > > > ps -ef | grep edward
> > > > - killing each process
> > > > kill -9 2574
> > > > kill -9 2594
> > > > or all your processes:
> > > > pkill -9 -u `id -u edward.strobach`
> > > >
> > > > Right now you have 4 loading scripts and 10 batch scripts
running at
> > the
> > > > same time which creates a gridlock and none of the processes
are
> > getting
> > > > executes.
> > > >
> > > > Tatiana
> > > >
> > > > On Wed Dec 02 06:32:23 2020, edward.strobach at noaa.gov wrote:
> > > > > I came across this issue again this morning. Is there a way
to
> stop
> > > > > certain processes and start fresh from my end?
> > > > >
> > > > > On Tue, Dec 1, 2020 at 5:14 PM Edward Strobach - NOAA
Affiliate <
> > > > > edward.strobach at noaa.gov> wrote:
> > > > >
> > > > > > maybe there is a script that allows me to check what is
going on
> > with
> > > > > > my
> > > > > > database? A lot of these issues I've come across happen
because
> of
> > > > > > things
> > > > > > I'm unaware of or things that go on simultaneously that
causes
> > > > > > confusion in
> > > > > > the processing. Right now I have a plotting script set
that
> > > > > > simultaneously
> > > > > > submits batch jobs for time series statistics, Taylor
diagrams,
> and
> > > > > > CSI. I
> > > > > > haven't seen too many issues when I do that, but would it
make
> > sense
> > > > > > to
> > > > > > insert a while loop after the first submission that uses a
script
> > to
> > > > > > check
> > > > > > the activity within my database? In other words, I would
use a
> > > > > > script,
> > > > > > perhaps one that has already been developed, to check the
status
> of
> > > > > > processing before submitting the next batch job. Once the
> > processing
> > > > > > for
> > > > > > the first plotting steps have completed, then I would
commence to
> > the
> > > > > > next
> > > > > > one.
> > > > > >
> > > > > > I guess check_my_db.sh would not do that?
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Dec 1, 2020 at 4:27 PM Edward Strobach - NOAA
Affiliate <
> > > > > > edward.strobach at noaa.gov> wrote:
> > > > > >
> > > > > >> are there youtube examples or a section in a technical
manual
> that
> > > > > >> goes
> > > > > >> over examples illustrating best practices related to this
> process?
> > > > > >>
> > > > > >> On Tue, Dec 1, 2020 at 4:03 PM Tatiana Burek via RT
> > > > > >> <met_help at ucar.edu>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> For your process you need only two scripts:
> > > > > >>> mv_load_to_aws.sh
> > > > > >>> mv_batch_on_aws.sh
> > > > > >>> I don't think, that others would help you.
> > > > > >>>
> > > > > >>> The main work is in creating efficient XMLs for the each
step.
> > > > > >>> Like not recalculating indexes or performing
> stat_header_db_check
> > > > > >>> during
> > > > > >>> data loading.
> > > > > >>>
> > > > > >>> Tatiana
> > > > > >>>
> > > > > >>>
> > > > > >>> On Tue Dec 01 13:00:15 2020, edward.strobach at noaa.gov
wrote:
> > > > > >>> > Thanks. I just made those changes in the xml file, so
I
> > 'guess'
> > > > > >>> > I'm
> > > > > >>> > ready
> > > > > >>> > to go.
> > > > > >>> >
> > > > > >>> > I should note that I have several steps that I run
each
> day. I
> > > > > >>> > convert
> > > > > >>> > files into netcdf files that can be read by MET, I
generate
> the
> > > > > >>> > stat
> > > > > >>> > files,
> > > > > >>> > and I use the stat files in the MET batch plotting
system. I
> > > > > >>> > have
> > > > > >>> > actually
> > > > > >>> > 5 total steps, but the plot batch system is really the
only
> one
> > > > > >>> > that
> > > > > >>> > gives
> > > > > >>> > me trouble on a regular basis. I wish I knew how to
reach a
> > > > > >>> > point
> > > > > >>> > where I
> > > > > >>> > have a plotting package that would produce the plots
that I
> > need
> > > > > >>> > each
> > > > > >>> > day.
> > > > > >>> > I'm expected to hand the final product off in the near
> future,
> > > > > >>> > but
> > > > > >>> > this
> > > > > >>> > part has delayed me extensively. I know that I'm
generating a
> > lot
> > > > > >>> > of
> > > > > >>> > plots
> > > > > >>> > which could potentially overwhelm the system, but I'm
unaware
> > how
> > > > > >>> > others
> > > > > >>> > work to circumvent that. I hope to move past this
soon so I
> > can
> > > > > >>> > continue
> > > > > >>> > development work.
> > > > > >>> >
> > > > > >>> > There are several scripts that I don't use that might
be
> > helpful.
> > > > > >>> > Do
> > > > > >>> > any
> > > > > >>> > of these work to help me manage the system better?
> > > > > >>> > check_my_db.sh mv_batch_on_aws.sh
mv_create_db_on_aws.sh
> > > > > >>> > mv_db_size_on_aws.sh mv_delete_db_on_aws.sh
> > mv_load_to_aws.sh
> > > > > >>> >
> > > > > >>> >
> > > > > >>> > On Tue, Dec 1, 2020 at 2:44 PM Tatiana Burek via RT
> > > > > >>> > <met_help at ucar.edu>
> > > > > >>> > wrote:
> > > > > >>> >
> > > > > >>> > > Edward
> > > > > >>> > >
> > > > > >>> > > I run a few butch jobs from my laptop and made sure
that
> > > > > >>> > > METviewer
> > > > > >>> > > code
> > > > > >>> > > and scripts working as expected: I was able to
produce and
> > copy
> > > > > >>> > > plots.
> > > > > >>> > > I also got an error like the one that you reported
but only
> > > > > >>> > > when
> > > > > >>> > > METviewer
> > > > > >>> > > did not generate any plots and the script couldn't
copy
> empty
> > > > > >>> > > directory.
> > > > > >>> > >
> > > > > >>> > > During my testing I noticed a few batch jobs that
where
> > > > > >>> > > started by
> > > > > >>> > > you
> > > > > >>> > > and were handing. Each of these jobs started a
database
> query
> > > > > >>> > > and
> > > > > >>> > > they also
> > > > > >>> > > were piling up and handing. They all were waiting
for the
> > > > > >>> > > indexing
> > > > > >>> > > that
> > > > > >>> > > takes place after the data loading.
> > > > > >>> > > I had to kill all database and batch processes in
order to
> > move
> > > > > >>> > > forward
> > > > > >>> > > with the testing.
> > > > > >>> > > To avoid this gridlock you could:
> > > > > >>> > > - in your loading xml use
> <drop_indexes>false</drop_indexes>
> > > > > >>> > > <apply_indexes>false</apply_indexes>
> > > > > >>> > >
> > > > > >>> > >
> > > > > >>> > > - do not start a few parallel batch jobs
> > > > > >>> > >
> > > > > >>> > > I will add a info message to the script that would
explain
> > why
> > > > > >>> > > the
> > > > > >>> > > error
> > > > > >>> > > occurs.
> > > > > >>> > > For now, could you test batch with some simple XML
that
> > produce
> > > > > >>> > > at
> > > > > >>> > > least
> > > > > >>> > > one image.
> > > > > >>> > >
> > > > > >>> > > Tatiana
> > > > > >>> > >
> > > > > >>> > >
> > > > > >>> > >
> > > > > >>> > >
> > > > > >>> > >
> > > > > >>> > >
> > > > > >>> > >
> > > > > >>> > > On Tue Dec 01 11:35:03 2020, jpresto wrote:
> > > > > >>> > > > Hi Edward.
> > > > > >>> > > >
> > > > > >>> > > > After talking with Tatiana, it sounds like I am
> misinformed
> > > > > >>> > > > and the
> > > > > >>> > > > scripts
> > > > > >>> > > > you are using were, in fact, developed by our
METviewer
> > team.
> > > > > >>> > > > My
> > > > > >>> > > > apologies. METplus has a lot of components, and
while I
> > > > > >>> > > > thought I
> > > > > >>> > > > had
> > > > > >>> > > > a
> > > > > >>> > > > good understanding of who did what, it turns out,
I did
> > not.
> > > > > >>> > > > Tatiana
> > > > > >>> > > > is
> > > > > >>> > > > working on this issue and will follow up once she
knows
> > more.
> > > > > >>> > > > Thank
> > > > > >>> > > > you
> > > > > >>> > > > for your patience and understanding.
> > > > > >>> > > >
> > > > > >>> > > > Julie
> > > > > >>> > > >
> > > > > >>> > > > On Tue, Dec 1, 2020 at 11:22 AM Julie Prestopnik
> > > > > >>> > > > <jpresto at ucar.edu
> > > > > >>> >
> > > > > >>> > > > wrote:
> > > > > >>> > > >
> > > > > >>> > > > > Thanks, Edward. I directed you to Mallory and
Perry
> > > > > >>> > > > > because I
> > > > > >>> > > > > thought you
> > > > > >>> > > > > might be using scripts from them. We did not
develop
> and
> > > > > >>> > > > > have
> > > > > >>> > > > > not
> > > > > >>> > > > > used
> > > > > >>> > > > > those scripts. We have no experience
interacting with
> > AWS
> > > > > >>> > > > > and
> > > > > >>> > > > > WCOSS,
> > > > > >>> > > > > so
> > > > > >>> > > > > they seemed best suited to answer your questions
> related
> > to
> > > > > >>> > > > > that
> > > > > >>> > > > > process.
> > > > > >>> > > > > However, as Tatiana pointed out, perhaps the
security
> > > > > >>> > > > > changes on
> > > > > >>> > > > > the
> > > > > >>> > > > > AWS
> > > > > >>> > > > > server have something to do with what you are
> > experiencing.
> > > > > >>> > > > > If
> > > > > >>> > > > > that
> > > > > >>> > > > > turns
> > > > > >>> > > > > out not to be the case, Mallory and Perry would
be best
> > > > > >>> > > > > able to
> > > > > >>> > > > > help
> > > > > >>> > > > > you.
> > > > > >>> > > > > If they are unable to help, please let us know,
and we
> > can
> > > > > >>> > > > > see
> > > > > >>> > > > > what
> > > > > >>> > > > > they
> > > > > >>> > > > > tried and how we can help you get this problem
figured
> > out.
> > > > > >>> > > > >
> > > > > >>> > > > > Julie
> > > > > >>> > > > >
> > > > > >>> > > > > On Tue, Dec 1, 2020 at 11:15 AM Edward Strobach
- NOAA
> > > > > >>> > > > > Affiliate
> > > > > >>> > > > > via
> > > > > >>> > > > > RT <
> > > > > >>> > > > > met_help at ucar.edu> wrote:
> > > > > >>> > > > >
> > > > > >>> > > > >>
> > > > > >>> > > > >> <URL:
> > > > > >>> > > > >>
> https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > >
> > > > > >>> > > > >>
> > > > > >>> > > > >> Hi Julie,
> > > > > >>> > > > >>
> > > > > >>> > > > >> I'm actually using the scripts located here:
> > > > > >>> > > > >>
> > > > > >>>
> /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts.
> > > > > >>> > > > >> These
> > > > > >>> > > > >> were scripts that I either got from Mallory or
Perry a
> > > > > >>> > > > >> while
> > > > > >>> > > > >> back -
> > > > > >>> > > > >> around
> > > > > >>> > > > >> May.
> > > > > >>> > > > >>
> > > > > >>> > > > >> On Tue, Dec 1, 2020 at 12:59 PM Julie
Prestopnik via
> RT
> > <
> > > > > >>> > > > >> met_help at ucar.edu>
> > > > > >>> > > > >> wrote:
> > > > > >>> > > > >>
> > > > > >>> > > > >> > Hi Edward.
> > > > > >>> > > > >> >
> > > > > >>> > > > >> > Are you using the scripts that are located in
> > > > > >>> > > > >> >
> > > > /gpfs/hps3/emc/global/noscrub/Mallory.Row/VRFY/METviewer_AWS
> > > > > >>> > > > >> > and
> > > > > >>> > > > >> >
> > > > /gpfs/dell2/emc/verification/noscrub/emc.metplus/METviewer_AWS
> > > > > >>> > > > >> > for interaction between AWS and WCOSS?
> > > > > >>> > > > >> >
> > > > > >>> > > > >> > Thanks!
> > > > > >>> > > > >> >
> > > > > >>> > > > >> > Julie
> > > > > >>> > > > >> >
> > > > > >>> > > > >> > On Tue, Dec 1, 2020 at 10:49 AM Edward
Strobach -
> NOAA
> > > > > >>> > > > >> > Affiliate
> > > > > >>> > > > >> > via RT
> > > > > >>> > > > >> <
> > > > > >>> > > > >> > met_help at ucar.edu> wrote:
> > > > > >>> > > > >> >
> > > > > >>> > > > >> > >
> > > > > >>> > > > >> > > <URL:
> > > > > >>> > > > >> > >
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > > > >>> > > > >> > > >
> > > > > >>> > > > >> > >
> > > > > >>> > > > >> > > Perry does not do batch plotting with MET.
I was
> > > > > >>> > > > >> > > under the
> > > > > >>> > > > >> > > impression
> > > > > >>> > > > >> > that
> > > > > >>> > > > >> > > the MET helpdesk helped with MET related
issues.
> > > > > >>> > > > >> > >
> > > > > >>> > > > >> > > On Tue, Dec 1, 2020 at 12:13 PM Julie
Prestopnik
> via
> > > > > >>> > > > >> > > RT <
> > > > > >>> > > > >> > met_help at ucar.edu
> > > > > >>> > > > >> > > >
> > > > > >>> > > > >> > > wrote:
> > > > > >>> > > > >> > >
> > > > > >>> > > > >> > > > Hi Edward.
> > > > > >>> > > > >> > > >
> > > > > >>> > > > >> > > > Please consult with Mallory Row and Perry
> Shafran
> > > > > >>> > > > >> > > > for
> > > > > >>> > > > >> > > > assistance
> > > > > >>> > > > >> with
> > > > > >>> > > > >> > > this
> > > > > >>> > > > >> > > > issue. They are most familiar with these
> > > > > >>> > > > >> > > > capabilities and
> > > > > >>> > > > >> > > > will be
> > > > > >>> > > > >> best
> > > > > >>> > > > >> > > > able to assist you.
> > > > > >>> > > > >> > > >
> > > > > >>> > > > >> > > > Thanks!
> > > > > >>> > > > >> > > >
> > > > > >>> > > > >> > > > Julie
> > > > > >>> > > > >> > > >
> > > > > >>> > > > >> > > > On Tue, Dec 1, 2020 at 9:25 AM Edward
Strobach -
> > > > > >>> > > > >> > > > NOAA
> > > > > >>> > > > >> > > > Affiliate via
> > > > > >>> > > > >> RT
> > > > > >>> > > > >> > <
> > > > > >>> > > > >> > > > met_help at ucar.edu> wrote:
> > > > > >>> > > > >> > > >
> > > > > >>> > > > >> > > > >
> > > > > >>> > > > >> > > > > Tue Dec 01 09:25:26 2020: Request 97687
was
> > acted
> > > > > >>> > > > >> > > > > upon.
> > > > > >>> > > > >> > > > > Transaction: Ticket created by
> > > > > >>> > > > >> > > > > edward.strobach at noaa.gov
> > > > > >>> > > > >> > > > > Queue: met_help
> > > > > >>> > > > >> > > > > Subject: can't copy plots over
> > > > > >>> > > > >> > > > > Owner: Nobody
> > > > > >>> > > > >> > > > > Requestors: edward.strobach at noaa.gov
> > > > > >>> > > > >> > > > > Status: new
> > > > > >>> > > > >> > > > > Ticket <URL:
> > > > > >>> > > > >> >
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > > > >>> > > > >> > > >
> > > > > >>> > > > >> > > > >
> > > > > >>> > > > >> > > > >
> > > > > >>> > > > >> > > > > I'm trying to copy plots over to Dell
from the
> > MET
> > > > > >>> > > > >> > > > > server.
> > > > > >>> > > > >> > > > > I
> > > > > >>> > > > >> would
> > > > > >>> > > > >> > > like
> > > > > >>> > > > >> > > > to
> > > > > >>> > > > >> > > > > store my plots in the usual place but
there is
> > an
> > > > > >>> > > > >> > > > > scp
> > > > > >>> > > > >> > > > > error
> > > > > >>> > > > >> > > > > which
> > > > > >>> > > > >> I'm
> > > > > >>> > > > >> > > > > attaching below.
> > > > > >>> > > > >> > > > >
> > > > > >>> > > > >> > > > >
> > > > > >>> > > > >> > > > > AND BINARY h.fcst_var = 'PMTF'
> > > > > >>> > > > >> > > > > AND ld.stat_header_id =
h.stat_header_id;
> > > > > >>> > > > >> > > > >
> > > > > >>> > > > >> > > > > Connection to 205.156.8.85 closed by
remote
> > host.
> > > > > >>> > > > >> > > > > CALLING: scp
edward.strobach at 205.156.8.85
> > > > > >>> > > > >> > > > >
://data/mv_data//edward.strobach/plots/*
> > > > > >>> > > > >> > > > >
> > > > > >>> > > > >> > > > >
> > > > > >>> > > > >> > > >
> > > > > >>> > > > >> > >
> > > > > >>> > > > >> >
> > > > > >>> > > > >>
> > > > > >>> > >
> > > > > >>>
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > > > >>> > > > >> > > > >
> > > > > >>> > > > >> > > > > You are accessing a U.S. Government
> information
> > > > > >>> > > > >> > > > > system, which includes: 1) this
computer, 2)
> > this
> > > > > >>> > > > >> > > > > computer network, 3) all computers
connected
> to
> > > > > >>> > > > >> > > > > this
> > > > > >>> > > > >> > > > > network, and 4) all devices and storage
media
> > > > > >>> > > > >> > > > > attached
> > > > > >>> > > > >> > > > > to this network or to a computer on
this
> > network.
> > > > > >>> > > > >> > > > > You
> > > > > >>> > > > >> > > > > understand and consent to the
following: you
> may
> > > > > >>> > > > >> > > > > access this information system for
authorized
> > use
> > > > > >>> > > > >> > > > > only;
> > > > > >>> > > > >> > > > > you have no reasonable expectation of
privacy
> > > > > >>> > > > >> > > > > regarding
> > > > > >>> > > > >> > > > > any communication of data transiting or
stored
> > on
> > > > > >>> > > > >> > > > > this
> > > > > >>> > > > >> > > > > information system; at any time and for
any
> > lawful
> > > > > >>> > > > >> > > > > Government purpose, the Government may
> monitor,
> > > > > >>> > > > >> > > > > intercept, and search and seize any
> > communication
> > > > > >>> > > > >> > > > > or
> > > > > >>> > > > >> > > > > data transiting or stored on this
information
> > > > > >>> > > > >> > > > > system;
> > > > > >>> > > > >> > > > > and
> > > > > >>> > > > >> > > > > any communications or data transiting
or
> stored
> > on
> > > > > >>> > > > >> > > > > this
> > > > > >>> > > > >> > > > > information system may be disclosed or
used
> for
> > > > > >>> > > > >> > > > > any
> > > > > >>> > > > >> > > > > lawful Government purpose.
> > > > > >>> > > > >> > > > >
> > > > > >>> > > > >> > > > > scp:
//data/mv_data//edward.strobach/plots/*:
> No
> > > > > >>> > > > >> > > > > such
> > > > > >>> > > > >> > > > > file
> > > > > >>> > > > >> > > > > or
> > > > > >>> > > > >> > directory
> > > > > >>> > > > >> > > > > ERROR: Command returned with non-zero
status
> > (1):
> > > > > >>> > > > >> > > > > scp
> > > > > >>> > > > >> > edward.strobach@
> > > > > >>> > > > >> > > > > 205.156.8.85
> > > > ://data/mv_data//edward.strobach/plots/*
> > > > > >>> > > > >> > > > >
> > > > > >>> > > > >> > > > >
> > > > > >>> > > > >> > > >
> > > > > >>> > > > >> > >
> > > > > >>> > > > >> >
> > > > > >>> > > > >>
> > > > > >>> > >
> > > > > >>>
> > > >
> > >
> >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > > > >>> > > > >> > > > > ++ printf '\033]0;%s@%s:%s\007'
> Edward.Strobach
> > > > > >>> > > > >> > > > > v71a1
> > > > > >>> > > > >> > > > >
> > > > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > > > > >>> > > > >> > > > >
> > > > > >>> > > > >> > > > >
> > > > > >>> > > > >> > > > > NOTE: nothing about this set-up has
changed
> > other
> > > > > >>> > > > >> > > > > than
> > > > > >>> > > > >> > > > > the
> > > > > >>> > > > >> directory
> > > > > >>> > > > >> > > > path
> > > > > >>> > > > >> > > > > onto dell. The path has been created
so it
> > should
> > > > > >>> > > > >> > > > > support
> > > > > >>> > > > >> > > > > the
> > > > > >>> > > > >> > transfer
> > > > > >>> > > > >> > > > of
> > > > > >>> > > > >> > > > > plots generated on the MET server. The
> > preceding
> > > > > >>> > > > >> > > > > information
> > > > > >>> > > > >> > indicates
> > > > > >>> > > > >> > > > > that the plot was processed
successfully.
> It's
> > > > > >>> > > > >> > > > > not
> > > > > >>> > > > >> > > > > clear
> > > > > >>> > > > >> > > > > why I
> > > > > >>> > > > >> can't
> > > > > >>> > > > >> > > > > transfer my plots over.
> > > > > >>> > > > >> > > > > --
> > > > > >>> > > > >> > > > > Edward Strobach
> > > > > >>> > > > >> > > > > EMC/NCEP/NWS/
> > > > > >>> > > > >> > > > > IMSG Contractor
> > > > > >>> > > > >> > > > > Cubicle#: 2029
> > > > > >>> > > > >> > > > > 301-683-3717
> > > > > >>> > > > >> > > > >
> > > > > >>> > > > >> > > > >
> > > > > >>> > > > >> > > >
> > > > > >>> > > > >> > > > --
> > > > > >>> > > > >> > > > Julie Prestopnik (she/her)
> > > > > >>> > > > >> > > > Software Engineer
> > > > > >>> > > > >> > > > National Center for Atmospheric Research
> > > > > >>> > > > >> > > > Research Applications Laboratory
> > > > > >>> > > > >> > > > Email: jpresto at ucar.edu
> > > > > >>> > > > >> > > >
> > > > > >>> > > > >> > > > My working day may not be your working
day.
> > Please
> > > > > >>> > > > >> > > > do not
> > > > > >>> > > > >> > > > feel
> > > > > >>> > > > >> obliged
> > > > > >>> > > > >> > > to
> > > > > >>> > > > >> > > > reply to this email outside of your
normal
> working
> > > > > >>> > > > >> > > > hours.
> > > > > >>> > > > >> > > >
> > > > > >>> > > > >> > > >
> > > > > >>> > > > >> > >
> > > > > >>> > > > >> > > --
> > > > > >>> > > > >> > > Edward Strobach
> > > > > >>> > > > >> > > EMC/NCEP/NWS/
> > > > > >>> > > > >> > > IMSG Contractor
> > > > > >>> > > > >> > > Cubicle#: 2029
> > > > > >>> > > > >> > > 301-683-3717
> > > > > >>> > > > >> > >
> > > > > >>> > > > >> > >
> > > > > >>> > > > >> >
> > > > > >>> > > > >> > --
> > > > > >>> > > > >> > Julie Prestopnik (she/her)
> > > > > >>> > > > >> > Software Engineer
> > > > > >>> > > > >> > National Center for Atmospheric Research
> > > > > >>> > > > >> > Research Applications Laboratory
> > > > > >>> > > > >> > Email: jpresto at ucar.edu
> > > > > >>> > > > >> >
> > > > > >>> > > > >> > My working day may not be your working day.
Please
> do
> > > > > >>> > > > >> > not
> > > > > >>> > > > >> > feel
> > > > > >>> > > > >> > obliged
> > > > > >>> > > > >> to
> > > > > >>> > > > >> > reply to this email outside of your normal
working
> > > > > >>> > > > >> > hours.
> > > > > >>> > > > >> >
> > > > > >>> > > > >> >
> > > > > >>> > > > >>
> > > > > >>> > > > >> --
> > > > > >>> > > > >> Edward Strobach
> > > > > >>> > > > >> EMC/NCEP/NWS/
> > > > > >>> > > > >> IMSG Contractor
> > > > > >>> > > > >> Cubicle#: 2029
> > > > > >>> > > > >> 301-683-3717
> > > > > >>> > > > >>
> > > > > >>> > > > >>
> > > > > >>> > > > >
> > > > > >>> > > > > --
> > > > > >>> > > > > Julie Prestopnik (she/her)
> > > > > >>> > > > > Software Engineer
> > > > > >>> > > > > National Center for Atmospheric Research
> > > > > >>> > > > > Research Applications Laboratory
> > > > > >>> > > > > Email: jpresto at ucar.edu
> > > > > >>> > > > >
> > > > > >>> > > > > My working day may not be your working day.
Please do
> > not
> > > > > >>> > > > > feel
> > > > > >>> > > > > obliged to
> > > > > >>> > > > > reply to this email outside of your normal
working
> hours.
> > > > > >>> > > > >
> > > > > >>> > >
> > > > > >>> > >
> > > > > >>> > >
> > > > > >>> > >
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > > >> --
> > > > > >> Edward Strobach
> > > > > >> EMC/NCEP/NWS/
> > > > > >> IMSG Contractor
> > > > > >> Cubicle#: 2029
> > > > > >> 301-683-3717
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Edward Strobach
> > > > > > EMC/NCEP/NWS/
> > > > > > IMSG Contractor
> > > > > > Cubicle#: 2029
> > > > > > 301-683-3717
> > > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> > >
> >
> > --
> > Julie Prestopnik (she/her)
> > Software Engineer
> > National Center for Atmospheric Research
> > Research Applications Laboratory
> > Email: jpresto at ucar.edu
> >
> > My working day may not be your working day. Please do not feel
obliged
> to
> > reply to this email outside of your normal working hours.
> >
> >
>
> --
> Edward Strobach
> EMC/NCEP/NWS/
> IMSG Contractor
> Cubicle#: 2029
> 301-683-3717
>
>
--
Julie Prestopnik (she/her)
Software Engineer
National Center for Atmospheric Research
Research Applications Laboratory
Email: jpresto at ucar.edu
My working day may not be your working day. Please do not feel
obliged to
reply to this email outside of your normal working hours.
------------------------------------------------
Subject: can't copy plots over
From: Tatiana Burek
Time: Wed Dec 02 10:10:40 2020
Edward
I am sorry, I did not mentioned that the commands I sent you should be
ran on AWS server and not on WCOSS
Tatiana
On Wed Dec 02 09:42:20 2020, jpresto wrote:
> Hi Edward.
>
> You should be able to distinguish what processes are MET and what
are
> not.
> From your "metcheck" results, it seems you have no MET processes
> running.
> I'll explain below.
>
> [Edward.Strobach at v71a2 ~]$ metcheck
> > root 76468 12071 0 16:25 ? 00:00:00 sshd:
> > Edward.Strobach
> > [priv]
> > Edward.+ 76482 76468 0 16:25 ? 00:00:00 sshd:
> > Edward.Strobach at pts
> > /19
> > Edward.+ 85480 82420 0 16:26 pts/19 00:00:00 grep
--color=auto
> > Edward.Strobach
> > root 141334 12071 0 12:27 ? 00:00:00 sshd:
> > Edward.Strobach
> > [priv]
> > Edward.+ 141337 141334 0 12:27 ? 00:00:00 sshd:
> > Edward.Strobach at pts
> > /5
>
>
> All of the processes that start with "sshd" are your logins, so only
> if you
> ran "kill -9" on one of those would you get logged out. The "grep"
> processes (3rd result) is your running of "ps -ef | grep
> Edward.Strobach".
> If there were MET processes running you would be able to see them
> here.
> The fact that there are none listed here could be because when you
ran
> "pkill -9 -u `id -u Edward.Strobach`", they were all killed, or
> because
> you're on a different node (v71a2) than the one you ran them on.
>
> Julie
>
> On Wed, Dec 2, 2020 at 9:28 AM Edward Strobach - NOAA Affiliate via
RT
> <
> met_help at ucar.edu> wrote:
>
> >
> > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
> >
> > Thanks. I was able to determine the processes using ps -ef | grep
> > Edward.Strobach
> >
> > I did try pkill -9 -u `id -u Edward.Strobach` before sending your
> > email
> > which did log me out unfortunately.
> >
> > I also noticed that kill -9 jobid also logs me out. I wish there
was
> > a way
> > to distinguish between MET tasks v. other tasks. When I look at
what
> > I
> > currently have I can't tell what is MET and what is not MET.
> > [Edward.Strobach at v71a2 ~]$ metcheck
> > root 76468 12071 0 16:25 ? 00:00:00 sshd:
> > Edward.Strobach
> > [priv]
> > Edward.+ 76482 76468 0 16:25 ? 00:00:00 sshd:
> > Edward.Strobach at pts
> > /19
> > Edward.+ 85480 82420 0 16:26 pts/19 00:00:00 grep
--color=auto
> > Edward.Strobach
> > root 141334 12071 0 12:27 ? 00:00:00 sshd:
> > Edward.Strobach
> > [priv]
> > Edward.+ 141337 141334 0 12:27 ? 00:00:00 sshd:
> > Edward.Strobach at pts
> > /5
> >
> > Note that metcheck is just an alias for ps -ef | grep
Edward.Strobach
> >
> > On Wed, Dec 2, 2020 at 11:13 AM Julie Prestopnik via RT
> > <met_help at ucar.edu
> > >
> > wrote:
> >
> > > Hi Edward.
> > >
> > > Because capitals matter when running "grep" and because WCOSS
uses
> > capital
> > > letters for our username you can run either of the following
> > > instead:
> > >
> > > ps -ef | grep -i edward (the "-i" tells grep to ignore case)
> > > ps -ef | grep Edward
> > >
> > > The second column will list the process id and the third column
> > > will list
> > > the parent process id.
> > >
> > > The same issue exists with running pkill on WCOSS. You can run
> > > pkill on
> > > WCOSS using the following, however, I don't recommend it and
I'll
> > > explain
> > > below:
> > >
> > > pkill -9 -u `id -u Edward.Strobach`
> > >
> > > First, you'd need to make sure you're logged in appropriately
and
> > > be on
> > the
> > > same node (not sure if that is the correct terminology) as the
one
> > running
> > > your processes. For example, I ran a simple check to test this.
I
> > > ran
> > > "top" when I was logged into "Julie.Prestopnik@*v71a3*", but my
> > > second
> > log
> > > in was on "Julie.Prestopnik@*v72a1*". v72a1 did not recognize
> > > "top" as
> > > running, which I could see when running "ps -ef | grep Julie".
> > > When I
> > ran
> > > "ssh v71a3" and then reran "ps -ef | grep Julie", I could then
see
> > > "top"
> > > was running. Running "pkill -9 -u `id -u Julie.Presotpnik`"
also
> > > closed
> > my
> > > connection to WCOSS, so if you don't want your connection to
WCOSS
> > killed,
> > > which I didn't, I would just kill the processes individually
using
> > > the
> > kill
> > > command with the process number.
> > >
> > > I hope that helps!
> > >
> > > Julie
> > >
> > > On Wed, Dec 2, 2020 at 8:51 AM Edward Strobach - NOAA Affiliate
via
> > > RT <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
>
> > > >
> > > > I see this when running ps -ef | grep edward
> > > > Edward.+ 115762 154642 0 15:48 pts/31 00:00:00 grep
> > > > --color=auto
> > > edward
> > > >
> > > > The numbers there change every time I run this step.
> > > >
> > > > pkill -9 -u `id -u edward.strobach` does not seem to work.
> > > >
> > > > On Wed, Dec 2, 2020 at 9:34 AM Tatiana Burek via RT
> > > > <met_help at ucar.edu
> > >
> > > > wrote:
> > > >
> > > > > You can kill your processes by :
> > > > > - collecting what is running:
> > > > > ps -ef | grep edward
> > > > > - killing each process
> > > > > kill -9 2574
> > > > > kill -9 2594
> > > > > or all your processes:
> > > > > pkill -9 -u `id -u edward.strobach`
> > > > >
> > > > > Right now you have 4 loading scripts and 10 batch scripts
> > > > > running at
> > > the
> > > > > same time which creates a gridlock and none of the processes
> > > > > are
> > > getting
> > > > > executes.
> > > > >
> > > > > Tatiana
> > > > >
> > > > > On Wed Dec 02 06:32:23 2020, edward.strobach at noaa.gov wrote:
> > > > > > I came across this issue again this morning. Is there a
way
> > > > > > to
> > stop
> > > > > > certain processes and start fresh from my end?
> > > > > >
> > > > > > On Tue, Dec 1, 2020 at 5:14 PM Edward Strobach - NOAA
> > > > > > Affiliate <
> > > > > > edward.strobach at noaa.gov> wrote:
> > > > > >
> > > > > > > maybe there is a script that allows me to check what is
> > > > > > > going on
> > > with
> > > > > > > my
> > > > > > > database? A lot of these issues I've come across happen
> > > > > > > because
> > of
> > > > > > > things
> > > > > > > I'm unaware of or things that go on simultaneously that
> > > > > > > causes
> > > > > > > confusion in
> > > > > > > the processing. Right now I have a plotting script set
> > > > > > > that
> > > > > > > simultaneously
> > > > > > > submits batch jobs for time series statistics, Taylor
> > > > > > > diagrams,
> > and
> > > > > > > CSI. I
> > > > > > > haven't seen too many issues when I do that, but would
it
> > > > > > > make
> > > sense
> > > > > > > to
> > > > > > > insert a while loop after the first submission that uses
a
> > > > > > > script
> > > to
> > > > > > > check
> > > > > > > the activity within my database? In other words, I
would
> > > > > > > use a
> > > > > > > script,
> > > > > > > perhaps one that has already been developed, to check
the
> > > > > > > status
> > of
> > > > > > > processing before submitting the next batch job. Once
the
> > > processing
> > > > > > > for
> > > > > > > the first plotting steps have completed, then I would
> > > > > > > commence to
> > > the
> > > > > > > next
> > > > > > > one.
> > > > > > >
> > > > > > > I guess check_my_db.sh would not do that?
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Dec 1, 2020 at 4:27 PM Edward Strobach - NOAA
> > > > > > > Affiliate <
> > > > > > > edward.strobach at noaa.gov> wrote:
> > > > > > >
> > > > > > >> are there youtube examples or a section in a technical
> > > > > > >> manual
> > that
> > > > > > >> goes
> > > > > > >> over examples illustrating best practices related to
this
> > process?
> > > > > > >>
> > > > > > >> On Tue, Dec 1, 2020 at 4:03 PM Tatiana Burek via RT
> > > > > > >> <met_help at ucar.edu>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> For your process you need only two scripts:
> > > > > > >>> mv_load_to_aws.sh
> > > > > > >>> mv_batch_on_aws.sh
> > > > > > >>> I don't think, that others would help you.
> > > > > > >>>
> > > > > > >>> The main work is in creating efficient XMLs for the
each
> > > > > > >>> step.
> > > > > > >>> Like not recalculating indexes or performing
> > stat_header_db_check
> > > > > > >>> during
> > > > > > >>> data loading.
> > > > > > >>>
> > > > > > >>> Tatiana
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> On Tue Dec 01 13:00:15 2020, edward.strobach at noaa.gov
> > > > > > >>> wrote:
> > > > > > >>> > Thanks. I just made those changes in the xml file,
so
> > > > > > >>> > I
> > > 'guess'
> > > > > > >>> > I'm
> > > > > > >>> > ready
> > > > > > >>> > to go.
> > > > > > >>> >
> > > > > > >>> > I should note that I have several steps that I run
each
> > day. I
> > > > > > >>> > convert
> > > > > > >>> > files into netcdf files that can be read by MET, I
> > > > > > >>> > generate
> > the
> > > > > > >>> > stat
> > > > > > >>> > files,
> > > > > > >>> > and I use the stat files in the MET batch plotting
> > > > > > >>> > system. I
> > > > > > >>> > have
> > > > > > >>> > actually
> > > > > > >>> > 5 total steps, but the plot batch system is really
the
> > > > > > >>> > only
> > one
> > > > > > >>> > that
> > > > > > >>> > gives
> > > > > > >>> > me trouble on a regular basis. I wish I knew how to
> > > > > > >>> > reach a
> > > > > > >>> > point
> > > > > > >>> > where I
> > > > > > >>> > have a plotting package that would produce the plots
> > > > > > >>> > that I
> > > need
> > > > > > >>> > each
> > > > > > >>> > day.
> > > > > > >>> > I'm expected to hand the final product off in the
near
> > future,
> > > > > > >>> > but
> > > > > > >>> > this
> > > > > > >>> > part has delayed me extensively. I know that I'm
> > > > > > >>> > generating a
> > > lot
> > > > > > >>> > of
> > > > > > >>> > plots
> > > > > > >>> > which could potentially overwhelm the system, but
I'm
> > > > > > >>> > unaware
> > > how
> > > > > > >>> > others
> > > > > > >>> > work to circumvent that. I hope to move past this
soon
> > > > > > >>> > so I
> > > can
> > > > > > >>> > continue
> > > > > > >>> > development work.
> > > > > > >>> >
> > > > > > >>> > There are several scripts that I don't use that
might
> > > > > > >>> > be
> > > helpful.
> > > > > > >>> > Do
> > > > > > >>> > any
> > > > > > >>> > of these work to help me manage the system better?
> > > > > > >>> > check_my_db.sh mv_batch_on_aws.sh
> > > > > > >>> > mv_create_db_on_aws.sh
> > > > > > >>> > mv_db_size_on_aws.sh mv_delete_db_on_aws.sh
> > > mv_load_to_aws.sh
> > > > > > >>> >
> > > > > > >>> >
> > > > > > >>> > On Tue, Dec 1, 2020 at 2:44 PM Tatiana Burek via RT
> > > > > > >>> > <met_help at ucar.edu>
> > > > > > >>> > wrote:
> > > > > > >>> >
> > > > > > >>> > > Edward
> > > > > > >>> > >
> > > > > > >>> > > I run a few butch jobs from my laptop and made
sure
> > > > > > >>> > > that
> > > > > > >>> > > METviewer
> > > > > > >>> > > code
> > > > > > >>> > > and scripts working as expected: I was able to
> > > > > > >>> > > produce and
> > > copy
> > > > > > >>> > > plots.
> > > > > > >>> > > I also got an error like the one that you reported
> > > > > > >>> > > but only
> > > > > > >>> > > when
> > > > > > >>> > > METviewer
> > > > > > >>> > > did not generate any plots and the script couldn't
> > > > > > >>> > > copy
> > empty
> > > > > > >>> > > directory.
> > > > > > >>> > >
> > > > > > >>> > > During my testing I noticed a few batch jobs that
> > > > > > >>> > > where
> > > > > > >>> > > started by
> > > > > > >>> > > you
> > > > > > >>> > > and were handing. Each of these jobs started a
> > > > > > >>> > > database
> > query
> > > > > > >>> > > and
> > > > > > >>> > > they also
> > > > > > >>> > > were piling up and handing. They all were waiting
for
> > > > > > >>> > > the
> > > > > > >>> > > indexing
> > > > > > >>> > > that
> > > > > > >>> > > takes place after the data loading.
> > > > > > >>> > > I had to kill all database and batch processes in
> > > > > > >>> > > order to
> > > move
> > > > > > >>> > > forward
> > > > > > >>> > > with the testing.
> > > > > > >>> > > To avoid this gridlock you could:
> > > > > > >>> > > - in your loading xml use
> > <drop_indexes>false</drop_indexes>
> > > > > > >>> > > <apply_indexes>false</apply_indexes>
> > > > > > >>> > >
> > > > > > >>> > >
> > > > > > >>> > > - do not start a few parallel batch jobs
> > > > > > >>> > >
> > > > > > >>> > > I will add a info message to the script that would
> > > > > > >>> > > explain
> > > why
> > > > > > >>> > > the
> > > > > > >>> > > error
> > > > > > >>> > > occurs.
> > > > > > >>> > > For now, could you test batch with some simple XML
> > > > > > >>> > > that
> > > produce
> > > > > > >>> > > at
> > > > > > >>> > > least
> > > > > > >>> > > one image.
> > > > > > >>> > >
> > > > > > >>> > > Tatiana
> > > > > > >>> > >
> > > > > > >>> > >
> > > > > > >>> > >
> > > > > > >>> > >
> > > > > > >>> > >
> > > > > > >>> > >
> > > > > > >>> > >
> > > > > > >>> > > On Tue Dec 01 11:35:03 2020, jpresto wrote:
> > > > > > >>> > > > Hi Edward.
> > > > > > >>> > > >
> > > > > > >>> > > > After talking with Tatiana, it sounds like I am
> > misinformed
> > > > > > >>> > > > and the
> > > > > > >>> > > > scripts
> > > > > > >>> > > > you are using were, in fact, developed by our
> > > > > > >>> > > > METviewer
> > > team.
> > > > > > >>> > > > My
> > > > > > >>> > > > apologies. METplus has a lot of components, and
> > > > > > >>> > > > while I
> > > > > > >>> > > > thought I
> > > > > > >>> > > > had
> > > > > > >>> > > > a
> > > > > > >>> > > > good understanding of who did what, it turns
out, I
> > > > > > >>> > > > did
> > > not.
> > > > > > >>> > > > Tatiana
> > > > > > >>> > > > is
> > > > > > >>> > > > working on this issue and will follow up once
she
> > > > > > >>> > > > knows
> > > more.
> > > > > > >>> > > > Thank
> > > > > > >>> > > > you
> > > > > > >>> > > > for your patience and understanding.
> > > > > > >>> > > >
> > > > > > >>> > > > Julie
> > > > > > >>> > > >
> > > > > > >>> > > > On Tue, Dec 1, 2020 at 11:22 AM Julie Prestopnik
> > > > > > >>> > > > <jpresto at ucar.edu
> > > > > > >>> >
> > > > > > >>> > > > wrote:
> > > > > > >>> > > >
> > > > > > >>> > > > > Thanks, Edward. I directed you to Mallory and
> > > > > > >>> > > > > Perry
> > > > > > >>> > > > > because I
> > > > > > >>> > > > > thought you
> > > > > > >>> > > > > might be using scripts from them. We did not
> > > > > > >>> > > > > develop
> > and
> > > > > > >>> > > > > have
> > > > > > >>> > > > > not
> > > > > > >>> > > > > used
> > > > > > >>> > > > > those scripts. We have no experience
interacting
> > > > > > >>> > > > > with
> > > AWS
> > > > > > >>> > > > > and
> > > > > > >>> > > > > WCOSS,
> > > > > > >>> > > > > so
> > > > > > >>> > > > > they seemed best suited to answer your
questions
> > related
> > > to
> > > > > > >>> > > > > that
> > > > > > >>> > > > > process.
> > > > > > >>> > > > > However, as Tatiana pointed out, perhaps the
> > > > > > >>> > > > > security
> > > > > > >>> > > > > changes on
> > > > > > >>> > > > > the
> > > > > > >>> > > > > AWS
> > > > > > >>> > > > > server have something to do with what you are
> > > experiencing.
> > > > > > >>> > > > > If
> > > > > > >>> > > > > that
> > > > > > >>> > > > > turns
> > > > > > >>> > > > > out not to be the case, Mallory and Perry
would
> > > > > > >>> > > > > be best
> > > > > > >>> > > > > able to
> > > > > > >>> > > > > help
> > > > > > >>> > > > > you.
> > > > > > >>> > > > > If they are unable to help, please let us
know,
> > > > > > >>> > > > > and we
> > > can
> > > > > > >>> > > > > see
> > > > > > >>> > > > > what
> > > > > > >>> > > > > they
> > > > > > >>> > > > > tried and how we can help you get this problem
> > > > > > >>> > > > > figured
> > > out.
> > > > > > >>> > > > >
> > > > > > >>> > > > > Julie
> > > > > > >>> > > > >
> > > > > > >>> > > > > On Tue, Dec 1, 2020 at 11:15 AM Edward
Strobach -
> > > > > > >>> > > > > NOAA
> > > > > > >>> > > > > Affiliate
> > > > > > >>> > > > > via
> > > > > > >>> > > > > RT <
> > > > > > >>> > > > > met_help at ucar.edu> wrote:
> > > > > > >>> > > > >
> > > > > > >>> > > > >>
> > > > > > >>> > > > >> <URL:
> > > > > > >>> > > > >>
> > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > >
> > > > > > >>> > > > >>
> > > > > > >>> > > > >> Hi Julie,
> > > > > > >>> > > > >>
> > > > > > >>> > > > >> I'm actually using the scripts located here:
> > > > > > >>> > > > >>
> > > > > > >>>
> > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts.
> > > > > > >>> > > > >> These
> > > > > > >>> > > > >> were scripts that I either got from Mallory
or
> > > > > > >>> > > > >> Perry a
> > > > > > >>> > > > >> while
> > > > > > >>> > > > >> back -
> > > > > > >>> > > > >> around
> > > > > > >>> > > > >> May.
> > > > > > >>> > > > >>
> > > > > > >>> > > > >> On Tue, Dec 1, 2020 at 12:59 PM Julie
Prestopnik
> > > > > > >>> > > > >> via
> > RT
> > > <
> > > > > > >>> > > > >> met_help at ucar.edu>
> > > > > > >>> > > > >> wrote:
> > > > > > >>> > > > >>
> > > > > > >>> > > > >> > Hi Edward.
> > > > > > >>> > > > >> >
> > > > > > >>> > > > >> > Are you using the scripts that are located
in
> > > > > > >>> > > > >> >
> > > > > /gpfs/hps3/emc/global/noscrub/Mallory.Row/VRFY/METviewer_AWS
> > > > > > >>> > > > >> > and
> > > > > > >>> > > > >> >
> > > > >
/gpfs/dell2/emc/verification/noscrub/emc.metplus/METviewer_AWS
> > > > > > >>> > > > >> > for interaction between AWS and WCOSS?
> > > > > > >>> > > > >> >
> > > > > > >>> > > > >> > Thanks!
> > > > > > >>> > > > >> >
> > > > > > >>> > > > >> > Julie
> > > > > > >>> > > > >> >
> > > > > > >>> > > > >> > On Tue, Dec 1, 2020 at 10:49 AM Edward
> > > > > > >>> > > > >> > Strobach -
> > NOAA
> > > > > > >>> > > > >> > Affiliate
> > > > > > >>> > > > >> > via RT
> > > > > > >>> > > > >> <
> > > > > > >>> > > > >> > met_help at ucar.edu> wrote:
> > > > > > >>> > > > >> >
> > > > > > >>> > > > >> > >
> > > > > > >>> > > > >> > > <URL:
> > > > > > >>> > > > >> > >
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > > > > >>> > > > >> > > >
> > > > > > >>> > > > >> > >
> > > > > > >>> > > > >> > > Perry does not do batch plotting with
MET.
> > > > > > >>> > > > >> > > I was
> > > > > > >>> > > > >> > > under the
> > > > > > >>> > > > >> > > impression
> > > > > > >>> > > > >> > that
> > > > > > >>> > > > >> > > the MET helpdesk helped with MET related
> > > > > > >>> > > > >> > > issues.
> > > > > > >>> > > > >> > >
> > > > > > >>> > > > >> > > On Tue, Dec 1, 2020 at 12:13 PM Julie
> > > > > > >>> > > > >> > > Prestopnik
> > via
> > > > > > >>> > > > >> > > RT <
> > > > > > >>> > > > >> > met_help at ucar.edu
> > > > > > >>> > > > >> > > >
> > > > > > >>> > > > >> > > wrote:
> > > > > > >>> > > > >> > >
> > > > > > >>> > > > >> > > > Hi Edward.
> > > > > > >>> > > > >> > > >
> > > > > > >>> > > > >> > > > Please consult with Mallory Row and
Perry
> > Shafran
> > > > > > >>> > > > >> > > > for
> > > > > > >>> > > > >> > > > assistance
> > > > > > >>> > > > >> with
> > > > > > >>> > > > >> > > this
> > > > > > >>> > > > >> > > > issue. They are most familiar with
these
> > > > > > >>> > > > >> > > > capabilities and
> > > > > > >>> > > > >> > > > will be
> > > > > > >>> > > > >> best
> > > > > > >>> > > > >> > > > able to assist you.
> > > > > > >>> > > > >> > > >
> > > > > > >>> > > > >> > > > Thanks!
> > > > > > >>> > > > >> > > >
> > > > > > >>> > > > >> > > > Julie
> > > > > > >>> > > > >> > > >
> > > > > > >>> > > > >> > > > On Tue, Dec 1, 2020 at 9:25 AM Edward
> > > > > > >>> > > > >> > > > Strobach -
> > > > > > >>> > > > >> > > > NOAA
> > > > > > >>> > > > >> > > > Affiliate via
> > > > > > >>> > > > >> RT
> > > > > > >>> > > > >> > <
> > > > > > >>> > > > >> > > > met_help at ucar.edu> wrote:
> > > > > > >>> > > > >> > > >
> > > > > > >>> > > > >> > > > >
> > > > > > >>> > > > >> > > > > Tue Dec 01 09:25:26 2020: Request
97687
> > > > > > >>> > > > >> > > > > was
> > > acted
> > > > > > >>> > > > >> > > > > upon.
> > > > > > >>> > > > >> > > > > Transaction: Ticket created by
> > > > > > >>> > > > >> > > > > edward.strobach at noaa.gov
> > > > > > >>> > > > >> > > > > Queue: met_help
> > > > > > >>> > > > >> > > > > Subject: can't copy plots over
> > > > > > >>> > > > >> > > > > Owner: Nobody
> > > > > > >>> > > > >> > > > > Requestors:
edward.strobach at noaa.gov
> > > > > > >>> > > > >> > > > > Status: new
> > > > > > >>> > > > >> > > > > Ticket <URL:
> > > > > > >>> > > > >> >
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > > > > >>> > > > >> > > >
> > > > > > >>> > > > >> > > > >
> > > > > > >>> > > > >> > > > >
> > > > > > >>> > > > >> > > > > I'm trying to copy plots over to Dell
> > > > > > >>> > > > >> > > > > from the
> > > MET
> > > > > > >>> > > > >> > > > > server.
> > > > > > >>> > > > >> > > > > I
> > > > > > >>> > > > >> would
> > > > > > >>> > > > >> > > like
> > > > > > >>> > > > >> > > > to
> > > > > > >>> > > > >> > > > > store my plots in the usual place but
> > > > > > >>> > > > >> > > > > there is
> > > an
> > > > > > >>> > > > >> > > > > scp
> > > > > > >>> > > > >> > > > > error
> > > > > > >>> > > > >> > > > > which
> > > > > > >>> > > > >> I'm
> > > > > > >>> > > > >> > > > > attaching below.
> > > > > > >>> > > > >> > > > >
> > > > > > >>> > > > >> > > > >
> > > > > > >>> > > > >> > > > > AND BINARY h.fcst_var = 'PMTF'
> > > > > > >>> > > > >> > > > > AND ld.stat_header_id =
> > > > > > >>> > > > >> > > > > h.stat_header_id;
> > > > > > >>> > > > >> > > > >
> > > > > > >>> > > > >> > > > > Connection to 205.156.8.85 closed by
> > > > > > >>> > > > >> > > > > remote
> > > host.
> > > > > > >>> > > > >> > > > > CALLING: scp
> > > > > > >>> > > > >> > > > > edward.strobach at 205.156.8.85
> > > > > > >>> > > > >> > > > >
://data/mv_data//edward.strobach/plots/*
> > > > > > >>> > > > >> > > > >
> > > > > > >>> > > > >> > > > >
> > > > > > >>> > > > >> > > >
> > > > > > >>> > > > >> > >
> > > > > > >>> > > > >> >
> > > > > > >>> > > > >>
> > > > > > >>> > >
> > > > > > >>>
> > > > >
> > > >
> > >
> >
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > > > > >>> > > > >> > > > >
> > > > > > >>> > > > >> > > > > You are accessing a U.S. Government
> > information
> > > > > > >>> > > > >> > > > > system, which includes: 1) this
> > > > > > >>> > > > >> > > > > computer, 2)
> > > this
> > > > > > >>> > > > >> > > > > computer network, 3) all computers
> > > > > > >>> > > > >> > > > > connected
> > to
> > > > > > >>> > > > >> > > > > this
> > > > > > >>> > > > >> > > > > network, and 4) all devices and
storage
> > > > > > >>> > > > >> > > > > media
> > > > > > >>> > > > >> > > > > attached
> > > > > > >>> > > > >> > > > > to this network or to a computer on
this
> > > network.
> > > > > > >>> > > > >> > > > > You
> > > > > > >>> > > > >> > > > > understand and consent to the
following:
> > > > > > >>> > > > >> > > > > you
> > may
> > > > > > >>> > > > >> > > > > access this information system for
> > > > > > >>> > > > >> > > > > authorized
> > > use
> > > > > > >>> > > > >> > > > > only;
> > > > > > >>> > > > >> > > > > you have no reasonable expectation of
> > > > > > >>> > > > >> > > > > privacy
> > > > > > >>> > > > >> > > > > regarding
> > > > > > >>> > > > >> > > > > any communication of data transiting
or
> > > > > > >>> > > > >> > > > > stored
> > > on
> > > > > > >>> > > > >> > > > > this
> > > > > > >>> > > > >> > > > > information system; at any time and
for
> > > > > > >>> > > > >> > > > > any
> > > lawful
> > > > > > >>> > > > >> > > > > Government purpose, the Government
may
> > monitor,
> > > > > > >>> > > > >> > > > > intercept, and search and seize any
> > > communication
> > > > > > >>> > > > >> > > > > or
> > > > > > >>> > > > >> > > > > data transiting or stored on this
> > > > > > >>> > > > >> > > > > information
> > > > > > >>> > > > >> > > > > system;
> > > > > > >>> > > > >> > > > > and
> > > > > > >>> > > > >> > > > > any communications or data transiting
or
> > stored
> > > on
> > > > > > >>> > > > >> > > > > this
> > > > > > >>> > > > >> > > > > information system may be disclosed
or
> > > > > > >>> > > > >> > > > > used
> > for
> > > > > > >>> > > > >> > > > > any
> > > > > > >>> > > > >> > > > > lawful Government purpose.
> > > > > > >>> > > > >> > > > >
> > > > > > >>> > > > >> > > > > scp:
> > > > > > >>> > > > >> > > > >
//data/mv_data//edward.strobach/plots/*:
> > No
> > > > > > >>> > > > >> > > > > such
> > > > > > >>> > > > >> > > > > file
> > > > > > >>> > > > >> > > > > or
> > > > > > >>> > > > >> > directory
> > > > > > >>> > > > >> > > > > ERROR: Command returned with non-zero
> > > > > > >>> > > > >> > > > > status
> > > (1):
> > > > > > >>> > > > >> > > > > scp
> > > > > > >>> > > > >> > edward.strobach@
> > > > > > >>> > > > >> > > > > 205.156.8.85
> > > > > ://data/mv_data//edward.strobach/plots/*
> > > > > > >>> > > > >> > > > >
> > > > > > >>> > > > >> > > > >
> > > > > > >>> > > > >> > > >
> > > > > > >>> > > > >> > >
> > > > > > >>> > > > >> >
> > > > > > >>> > > > >>
> > > > > > >>> > >
> > > > > > >>>
> > > > >
> > > >
> > >
> >
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > > > > >>> > > > >> > > > > ++ printf '\033]0;%s@%s:%s\007'
> > Edward.Strobach
> > > > > > >>> > > > >> > > > > v71a1
> > > > > > >>> > > > >> > > > >
> > > > > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > > > > > >>> > > > >> > > > >
> > > > > > >>> > > > >> > > > >
> > > > > > >>> > > > >> > > > > NOTE: nothing about this set-up has
> > > > > > >>> > > > >> > > > > changed
> > > other
> > > > > > >>> > > > >> > > > > than
> > > > > > >>> > > > >> > > > > the
> > > > > > >>> > > > >> directory
> > > > > > >>> > > > >> > > > path
> > > > > > >>> > > > >> > > > > onto dell. The path has been created
so
> > > > > > >>> > > > >> > > > > it
> > > should
> > > > > > >>> > > > >> > > > > support
> > > > > > >>> > > > >> > > > > the
> > > > > > >>> > > > >> > transfer
> > > > > > >>> > > > >> > > > of
> > > > > > >>> > > > >> > > > > plots generated on the MET server.
The
> > > preceding
> > > > > > >>> > > > >> > > > > information
> > > > > > >>> > > > >> > indicates
> > > > > > >>> > > > >> > > > > that the plot was processed
> > > > > > >>> > > > >> > > > > successfully.
> > It's
> > > > > > >>> > > > >> > > > > not
> > > > > > >>> > > > >> > > > > clear
> > > > > > >>> > > > >> > > > > why I
> > > > > > >>> > > > >> can't
> > > > > > >>> > > > >> > > > > transfer my plots over.
> > > > > > >>> > > > >> > > > > --
> > > > > > >>> > > > >> > > > > Edward Strobach
> > > > > > >>> > > > >> > > > > EMC/NCEP/NWS/
> > > > > > >>> > > > >> > > > > IMSG Contractor
> > > > > > >>> > > > >> > > > > Cubicle#: 2029
> > > > > > >>> > > > >> > > > > 301-683-3717
> > > > > > >>> > > > >> > > > >
> > > > > > >>> > > > >> > > > >
> > > > > > >>> > > > >> > > >
> > > > > > >>> > > > >> > > > --
> > > > > > >>> > > > >> > > > Julie Prestopnik (she/her)
> > > > > > >>> > > > >> > > > Software Engineer
> > > > > > >>> > > > >> > > > National Center for Atmospheric
Research
> > > > > > >>> > > > >> > > > Research Applications Laboratory
> > > > > > >>> > > > >> > > > Email: jpresto at ucar.edu
> > > > > > >>> > > > >> > > >
> > > > > > >>> > > > >> > > > My working day may not be your working
> > > > > > >>> > > > >> > > > day.
> > > Please
> > > > > > >>> > > > >> > > > do not
> > > > > > >>> > > > >> > > > feel
> > > > > > >>> > > > >> obliged
> > > > > > >>> > > > >> > > to
> > > > > > >>> > > > >> > > > reply to this email outside of your
normal
> > working
> > > > > > >>> > > > >> > > > hours.
> > > > > > >>> > > > >> > > >
> > > > > > >>> > > > >> > > >
> > > > > > >>> > > > >> > >
> > > > > > >>> > > > >> > > --
> > > > > > >>> > > > >> > > Edward Strobach
> > > > > > >>> > > > >> > > EMC/NCEP/NWS/
> > > > > > >>> > > > >> > > IMSG Contractor
> > > > > > >>> > > > >> > > Cubicle#: 2029
> > > > > > >>> > > > >> > > 301-683-3717
> > > > > > >>> > > > >> > >
> > > > > > >>> > > > >> > >
> > > > > > >>> > > > >> >
> > > > > > >>> > > > >> > --
> > > > > > >>> > > > >> > Julie Prestopnik (she/her)
> > > > > > >>> > > > >> > Software Engineer
> > > > > > >>> > > > >> > National Center for Atmospheric Research
> > > > > > >>> > > > >> > Research Applications Laboratory
> > > > > > >>> > > > >> > Email: jpresto at ucar.edu
> > > > > > >>> > > > >> >
> > > > > > >>> > > > >> > My working day may not be your working day.
> > > > > > >>> > > > >> > Please
> > do
> > > > > > >>> > > > >> > not
> > > > > > >>> > > > >> > feel
> > > > > > >>> > > > >> > obliged
> > > > > > >>> > > > >> to
> > > > > > >>> > > > >> > reply to this email outside of your normal
> > > > > > >>> > > > >> > working
> > > > > > >>> > > > >> > hours.
> > > > > > >>> > > > >> >
> > > > > > >>> > > > >> >
> > > > > > >>> > > > >>
> > > > > > >>> > > > >> --
> > > > > > >>> > > > >> Edward Strobach
> > > > > > >>> > > > >> EMC/NCEP/NWS/
> > > > > > >>> > > > >> IMSG Contractor
> > > > > > >>> > > > >> Cubicle#: 2029
> > > > > > >>> > > > >> 301-683-3717
> > > > > > >>> > > > >>
> > > > > > >>> > > > >>
> > > > > > >>> > > > >
> > > > > > >>> > > > > --
> > > > > > >>> > > > > Julie Prestopnik (she/her)
> > > > > > >>> > > > > Software Engineer
> > > > > > >>> > > > > National Center for Atmospheric Research
> > > > > > >>> > > > > Research Applications Laboratory
> > > > > > >>> > > > > Email: jpresto at ucar.edu
> > > > > > >>> > > > >
> > > > > > >>> > > > > My working day may not be your working day.
> > > > > > >>> > > > > Please do
> > > not
> > > > > > >>> > > > > feel
> > > > > > >>> > > > > obliged to
> > > > > > >>> > > > > reply to this email outside of your normal
> > > > > > >>> > > > > working
> > hours.
> > > > > > >>> > > > >
> > > > > > >>> > >
> > > > > > >>> > >
> > > > > > >>> > >
> > > > > > >>> > >
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>
> > > > > > >> --
> > > > > > >> Edward Strobach
> > > > > > >> EMC/NCEP/NWS/
> > > > > > >> IMSG Contractor
> > > > > > >> Cubicle#: 2029
> > > > > > >> 301-683-3717
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Edward Strobach
> > > > > > > EMC/NCEP/NWS/
> > > > > > > IMSG Contractor
> > > > > > > Cubicle#: 2029
> > > > > > > 301-683-3717
> > > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > Edward Strobach
> > > > EMC/NCEP/NWS/
> > > > IMSG Contractor
> > > > Cubicle#: 2029
> > > > 301-683-3717
> > > >
> > > >
> > >
> > > --
> > > Julie Prestopnik (she/her)
> > > Software Engineer
> > > National Center for Atmospheric Research
> > > Research Applications Laboratory
> > > Email: jpresto at ucar.edu
> > >
> > > My working day may not be your working day. Please do not feel
> > > obliged
> > to
> > > reply to this email outside of your normal working hours.
> > >
> > >
> >
> > --
> > Edward Strobach
> > EMC/NCEP/NWS/
> > IMSG Contractor
> > Cubicle#: 2029
> > 301-683-3717
> >
> >
------------------------------------------------
Subject: can't copy plots over
From: Tatiana Burek
Time: Thu Dec 03 08:04:13 2020
Edward
AWS has 3 loading jobs that wore started by you:
2 for load_mv_cmaq_metplus_CMAQ.xml and 2 for
load_mv_cmaq_metplus_CMAQ_Meteorology.xml
and 25 batch jobs.
None of them are executing because of the database locks created by so
many incerts, create indexes and select statments.
Shoud I kill all your jbs aagain? Have you had a chance to look at my
sugesstuion on how to improve the process?
Tatiana
On Wed Dec 02 10:10:40 2020, tatiana wrote:
> Edward
> I am sorry, I did not mentioned that the commands I sent you should
be
> ran on AWS server and not on WCOSS
>
> Tatiana
>
> On Wed Dec 02 09:42:20 2020, jpresto wrote:
> > Hi Edward.
> >
> > You should be able to distinguish what processes are MET and what
are
> > not.
> > From your "metcheck" results, it seems you have no MET processes
> > running.
> > I'll explain below.
> >
> > [Edward.Strobach at v71a2 ~]$ metcheck
> > > root 76468 12071 0 16:25 ? 00:00:00 sshd:
> > > Edward.Strobach
> > > [priv]
> > > Edward.+ 76482 76468 0 16:25 ? 00:00:00 sshd:
> > > Edward.Strobach at pts
> > > /19
> > > Edward.+ 85480 82420 0 16:26 pts/19 00:00:00 grep
--color=auto
> > > Edward.Strobach
> > > root 141334 12071 0 12:27 ? 00:00:00 sshd:
> > > Edward.Strobach
> > > [priv]
> > > Edward.+ 141337 141334 0 12:27 ? 00:00:00 sshd:
> > > Edward.Strobach at pts
> > > /5
> >
> >
> > All of the processes that start with "sshd" are your logins, so
only
> > if you
> > ran "kill -9" on one of those would you get logged out. The
"grep"
> > processes (3rd result) is your running of "ps -ef | grep
> > Edward.Strobach".
> > If there were MET processes running you would be able to see them
> > here.
> > The fact that there are none listed here could be because when you
> > ran
> > "pkill -9 -u `id -u Edward.Strobach`", they were all killed, or
> > because
> > you're on a different node (v71a2) than the one you ran them on.
> >
> > Julie
> >
> > On Wed, Dec 2, 2020 at 9:28 AM Edward Strobach - NOAA Affiliate
via
> > RT
> > <
> > met_help at ucar.edu> wrote:
> >
> > >
> > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
> > >
> > > Thanks. I was able to determine the processes using ps -ef |
grep
> > > Edward.Strobach
> > >
> > > I did try pkill -9 -u `id -u Edward.Strobach` before sending
your
> > > email
> > > which did log me out unfortunately.
> > >
> > > I also noticed that kill -9 jobid also logs me out. I wish
there
> > > was
> > > a way
> > > to distinguish between MET tasks v. other tasks. When I look at
> > > what
> > > I
> > > currently have I can't tell what is MET and what is not MET.
> > > [Edward.Strobach at v71a2 ~]$ metcheck
> > > root 76468 12071 0 16:25 ? 00:00:00 sshd:
> > > Edward.Strobach
> > > [priv]
> > > Edward.+ 76482 76468 0 16:25 ? 00:00:00 sshd:
> > > Edward.Strobach at pts
> > > /19
> > > Edward.+ 85480 82420 0 16:26 pts/19 00:00:00 grep
--color=auto
> > > Edward.Strobach
> > > root 141334 12071 0 12:27 ? 00:00:00 sshd:
> > > Edward.Strobach
> > > [priv]
> > > Edward.+ 141337 141334 0 12:27 ? 00:00:00 sshd:
> > > Edward.Strobach at pts
> > > /5
> > >
> > > Note that metcheck is just an alias for ps -ef | grep
> > > Edward.Strobach
> > >
> > > On Wed, Dec 2, 2020 at 11:13 AM Julie Prestopnik via RT
> > > <met_help at ucar.edu
> > > >
> > > wrote:
> > >
> > > > Hi Edward.
> > > >
> > > > Because capitals matter when running "grep" and because WCOSS
> > > > uses
> > > capital
> > > > letters for our username you can run either of the following
> > > > instead:
> > > >
> > > > ps -ef | grep -i edward (the "-i" tells grep to ignore case)
> > > > ps -ef | grep Edward
> > > >
> > > > The second column will list the process id and the third
column
> > > > will list
> > > > the parent process id.
> > > >
> > > > The same issue exists with running pkill on WCOSS. You can
run
> > > > pkill on
> > > > WCOSS using the following, however, I don't recommend it and
I'll
> > > > explain
> > > > below:
> > > >
> > > > pkill -9 -u `id -u Edward.Strobach`
> > > >
> > > > First, you'd need to make sure you're logged in appropriately
and
> > > > be on
> > > the
> > > > same node (not sure if that is the correct terminology) as the
> > > > one
> > > running
> > > > your processes. For example, I ran a simple check to test
this.
> > > > I
> > > > ran
> > > > "top" when I was logged into "Julie.Prestopnik@*v71a3*", but
my
> > > > second
> > > log
> > > > in was on "Julie.Prestopnik@*v72a1*". v72a1 did not recognize
> > > > "top" as
> > > > running, which I could see when running "ps -ef | grep Julie".
> > > > When I
> > > ran
> > > > "ssh v71a3" and then reran "ps -ef | grep Julie", I could then
> > > > see
> > > > "top"
> > > > was running. Running "pkill -9 -u `id -u Julie.Presotpnik`"
also
> > > > closed
> > > my
> > > > connection to WCOSS, so if you don't want your connection to
> > > > WCOSS
> > > killed,
> > > > which I didn't, I would just kill the processes individually
> > > > using
> > > > the
> > > kill
> > > > command with the process number.
> > > >
> > > > I hope that helps!
> > > >
> > > > Julie
> > > >
> > > > On Wed, Dec 2, 2020 at 8:51 AM Edward Strobach - NOAA
Affiliate
> > > > via
> > > > RT <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
> > > > >
> > > > > I see this when running ps -ef | grep edward
> > > > > Edward.+ 115762 154642 0 15:48 pts/31 00:00:00 grep
> > > > > --color=auto
> > > > edward
> > > > >
> > > > > The numbers there change every time I run this step.
> > > > >
> > > > > pkill -9 -u `id -u edward.strobach` does not seem to work.
> > > > >
> > > > > On Wed, Dec 2, 2020 at 9:34 AM Tatiana Burek via RT
> > > > > <met_help at ucar.edu
> > > >
> > > > > wrote:
> > > > >
> > > > > > You can kill your processes by :
> > > > > > - collecting what is running:
> > > > > > ps -ef | grep edward
> > > > > > - killing each process
> > > > > > kill -9 2574
> > > > > > kill -9 2594
> > > > > > or all your processes:
> > > > > > pkill -9 -u `id -u edward.strobach`
> > > > > >
> > > > > > Right now you have 4 loading scripts and 10 batch scripts
> > > > > > running at
> > > > the
> > > > > > same time which creates a gridlock and none of the
processes
> > > > > > are
> > > > getting
> > > > > > executes.
> > > > > >
> > > > > > Tatiana
> > > > > >
> > > > > > On Wed Dec 02 06:32:23 2020, edward.strobach at noaa.gov
wrote:
> > > > > > > I came across this issue again this morning. Is there a
> > > > > > > way
> > > > > > > to
> > > stop
> > > > > > > certain processes and start fresh from my end?
> > > > > > >
> > > > > > > On Tue, Dec 1, 2020 at 5:14 PM Edward Strobach - NOAA
> > > > > > > Affiliate <
> > > > > > > edward.strobach at noaa.gov> wrote:
> > > > > > >
> > > > > > > > maybe there is a script that allows me to check what
is
> > > > > > > > going on
> > > > with
> > > > > > > > my
> > > > > > > > database? A lot of these issues I've come across
happen
> > > > > > > > because
> > > of
> > > > > > > > things
> > > > > > > > I'm unaware of or things that go on simultaneously
that
> > > > > > > > causes
> > > > > > > > confusion in
> > > > > > > > the processing. Right now I have a plotting script
set
> > > > > > > > that
> > > > > > > > simultaneously
> > > > > > > > submits batch jobs for time series statistics, Taylor
> > > > > > > > diagrams,
> > > and
> > > > > > > > CSI. I
> > > > > > > > haven't seen too many issues when I do that, but would
it
> > > > > > > > make
> > > > sense
> > > > > > > > to
> > > > > > > > insert a while loop after the first submission that
uses
> > > > > > > > a
> > > > > > > > script
> > > > to
> > > > > > > > check
> > > > > > > > the activity within my database? In other words, I
would
> > > > > > > > use a
> > > > > > > > script,
> > > > > > > > perhaps one that has already been developed, to check
the
> > > > > > > > status
> > > of
> > > > > > > > processing before submitting the next batch job. Once
> > > > > > > > the
> > > > processing
> > > > > > > > for
> > > > > > > > the first plotting steps have completed, then I would
> > > > > > > > commence to
> > > > the
> > > > > > > > next
> > > > > > > > one.
> > > > > > > >
> > > > > > > > I guess check_my_db.sh would not do that?
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Dec 1, 2020 at 4:27 PM Edward Strobach - NOAA
> > > > > > > > Affiliate <
> > > > > > > > edward.strobach at noaa.gov> wrote:
> > > > > > > >
> > > > > > > >> are there youtube examples or a section in a
technical
> > > > > > > >> manual
> > > that
> > > > > > > >> goes
> > > > > > > >> over examples illustrating best practices related to
> > > > > > > >> this
> > > process?
> > > > > > > >>
> > > > > > > >> On Tue, Dec 1, 2020 at 4:03 PM Tatiana Burek via RT
> > > > > > > >> <met_help at ucar.edu>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >>> For your process you need only two scripts:
> > > > > > > >>> mv_load_to_aws.sh
> > > > > > > >>> mv_batch_on_aws.sh
> > > > > > > >>> I don't think, that others would help you.
> > > > > > > >>>
> > > > > > > >>> The main work is in creating efficient XMLs for the
> > > > > > > >>> each
> > > > > > > >>> step.
> > > > > > > >>> Like not recalculating indexes or performing
> > > stat_header_db_check
> > > > > > > >>> during
> > > > > > > >>> data loading.
> > > > > > > >>>
> > > > > > > >>> Tatiana
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> On Tue Dec 01 13:00:15 2020,
edward.strobach at noaa.gov
> > > > > > > >>> wrote:
> > > > > > > >>> > Thanks. I just made those changes in the xml
file,
> > > > > > > >>> > so
> > > > > > > >>> > I
> > > > 'guess'
> > > > > > > >>> > I'm
> > > > > > > >>> > ready
> > > > > > > >>> > to go.
> > > > > > > >>> >
> > > > > > > >>> > I should note that I have several steps that I run
> > > > > > > >>> > each
> > > day. I
> > > > > > > >>> > convert
> > > > > > > >>> > files into netcdf files that can be read by MET, I
> > > > > > > >>> > generate
> > > the
> > > > > > > >>> > stat
> > > > > > > >>> > files,
> > > > > > > >>> > and I use the stat files in the MET batch plotting
> > > > > > > >>> > system. I
> > > > > > > >>> > have
> > > > > > > >>> > actually
> > > > > > > >>> > 5 total steps, but the plot batch system is really
> > > > > > > >>> > the
> > > > > > > >>> > only
> > > one
> > > > > > > >>> > that
> > > > > > > >>> > gives
> > > > > > > >>> > me trouble on a regular basis. I wish I knew how
to
> > > > > > > >>> > reach a
> > > > > > > >>> > point
> > > > > > > >>> > where I
> > > > > > > >>> > have a plotting package that would produce the
plots
> > > > > > > >>> > that I
> > > > need
> > > > > > > >>> > each
> > > > > > > >>> > day.
> > > > > > > >>> > I'm expected to hand the final product off in the
> > > > > > > >>> > near
> > > future,
> > > > > > > >>> > but
> > > > > > > >>> > this
> > > > > > > >>> > part has delayed me extensively. I know that I'm
> > > > > > > >>> > generating a
> > > > lot
> > > > > > > >>> > of
> > > > > > > >>> > plots
> > > > > > > >>> > which could potentially overwhelm the system, but
I'm
> > > > > > > >>> > unaware
> > > > how
> > > > > > > >>> > others
> > > > > > > >>> > work to circumvent that. I hope to move past this
> > > > > > > >>> > soon
> > > > > > > >>> > so I
> > > > can
> > > > > > > >>> > continue
> > > > > > > >>> > development work.
> > > > > > > >>> >
> > > > > > > >>> > There are several scripts that I don't use that
might
> > > > > > > >>> > be
> > > > helpful.
> > > > > > > >>> > Do
> > > > > > > >>> > any
> > > > > > > >>> > of these work to help me manage the system better?
> > > > > > > >>> > check_my_db.sh mv_batch_on_aws.sh
> > > > > > > >>> > mv_create_db_on_aws.sh
> > > > > > > >>> > mv_db_size_on_aws.sh mv_delete_db_on_aws.sh
> > > > mv_load_to_aws.sh
> > > > > > > >>> >
> > > > > > > >>> >
> > > > > > > >>> > On Tue, Dec 1, 2020 at 2:44 PM Tatiana Burek via
RT
> > > > > > > >>> > <met_help at ucar.edu>
> > > > > > > >>> > wrote:
> > > > > > > >>> >
> > > > > > > >>> > > Edward
> > > > > > > >>> > >
> > > > > > > >>> > > I run a few butch jobs from my laptop and made
sure
> > > > > > > >>> > > that
> > > > > > > >>> > > METviewer
> > > > > > > >>> > > code
> > > > > > > >>> > > and scripts working as expected: I was able to
> > > > > > > >>> > > produce and
> > > > copy
> > > > > > > >>> > > plots.
> > > > > > > >>> > > I also got an error like the one that you
reported
> > > > > > > >>> > > but only
> > > > > > > >>> > > when
> > > > > > > >>> > > METviewer
> > > > > > > >>> > > did not generate any plots and the script
couldn't
> > > > > > > >>> > > copy
> > > empty
> > > > > > > >>> > > directory.
> > > > > > > >>> > >
> > > > > > > >>> > > During my testing I noticed a few batch jobs
that
> > > > > > > >>> > > where
> > > > > > > >>> > > started by
> > > > > > > >>> > > you
> > > > > > > >>> > > and were handing. Each of these jobs started a
> > > > > > > >>> > > database
> > > query
> > > > > > > >>> > > and
> > > > > > > >>> > > they also
> > > > > > > >>> > > were piling up and handing. They all were
waiting
> > > > > > > >>> > > for
> > > > > > > >>> > > the
> > > > > > > >>> > > indexing
> > > > > > > >>> > > that
> > > > > > > >>> > > takes place after the data loading.
> > > > > > > >>> > > I had to kill all database and batch processes
in
> > > > > > > >>> > > order to
> > > > move
> > > > > > > >>> > > forward
> > > > > > > >>> > > with the testing.
> > > > > > > >>> > > To avoid this gridlock you could:
> > > > > > > >>> > > - in your loading xml use
> > > <drop_indexes>false</drop_indexes>
> > > > > > > >>> > > <apply_indexes>false</apply_indexes>
> > > > > > > >>> > >
> > > > > > > >>> > >
> > > > > > > >>> > > - do not start a few parallel batch jobs
> > > > > > > >>> > >
> > > > > > > >>> > > I will add a info message to the script that
would
> > > > > > > >>> > > explain
> > > > why
> > > > > > > >>> > > the
> > > > > > > >>> > > error
> > > > > > > >>> > > occurs.
> > > > > > > >>> > > For now, could you test batch with some simple
XML
> > > > > > > >>> > > that
> > > > produce
> > > > > > > >>> > > at
> > > > > > > >>> > > least
> > > > > > > >>> > > one image.
> > > > > > > >>> > >
> > > > > > > >>> > > Tatiana
> > > > > > > >>> > >
> > > > > > > >>> > >
> > > > > > > >>> > >
> > > > > > > >>> > >
> > > > > > > >>> > >
> > > > > > > >>> > >
> > > > > > > >>> > >
> > > > > > > >>> > > On Tue Dec 01 11:35:03 2020, jpresto wrote:
> > > > > > > >>> > > > Hi Edward.
> > > > > > > >>> > > >
> > > > > > > >>> > > > After talking with Tatiana, it sounds like I
am
> > > misinformed
> > > > > > > >>> > > > and the
> > > > > > > >>> > > > scripts
> > > > > > > >>> > > > you are using were, in fact, developed by our
> > > > > > > >>> > > > METviewer
> > > > team.
> > > > > > > >>> > > > My
> > > > > > > >>> > > > apologies. METplus has a lot of components,
and
> > > > > > > >>> > > > while I
> > > > > > > >>> > > > thought I
> > > > > > > >>> > > > had
> > > > > > > >>> > > > a
> > > > > > > >>> > > > good understanding of who did what, it turns
out,
> > > > > > > >>> > > > I
> > > > > > > >>> > > > did
> > > > not.
> > > > > > > >>> > > > Tatiana
> > > > > > > >>> > > > is
> > > > > > > >>> > > > working on this issue and will follow up once
she
> > > > > > > >>> > > > knows
> > > > more.
> > > > > > > >>> > > > Thank
> > > > > > > >>> > > > you
> > > > > > > >>> > > > for your patience and understanding.
> > > > > > > >>> > > >
> > > > > > > >>> > > > Julie
> > > > > > > >>> > > >
> > > > > > > >>> > > > On Tue, Dec 1, 2020 at 11:22 AM Julie
Prestopnik
> > > > > > > >>> > > > <jpresto at ucar.edu
> > > > > > > >>> >
> > > > > > > >>> > > > wrote:
> > > > > > > >>> > > >
> > > > > > > >>> > > > > Thanks, Edward. I directed you to Mallory
and
> > > > > > > >>> > > > > Perry
> > > > > > > >>> > > > > because I
> > > > > > > >>> > > > > thought you
> > > > > > > >>> > > > > might be using scripts from them. We did
not
> > > > > > > >>> > > > > develop
> > > and
> > > > > > > >>> > > > > have
> > > > > > > >>> > > > > not
> > > > > > > >>> > > > > used
> > > > > > > >>> > > > > those scripts. We have no experience
> > > > > > > >>> > > > > interacting
> > > > > > > >>> > > > > with
> > > > AWS
> > > > > > > >>> > > > > and
> > > > > > > >>> > > > > WCOSS,
> > > > > > > >>> > > > > so
> > > > > > > >>> > > > > they seemed best suited to answer your
> > > > > > > >>> > > > > questions
> > > related
> > > > to
> > > > > > > >>> > > > > that
> > > > > > > >>> > > > > process.
> > > > > > > >>> > > > > However, as Tatiana pointed out, perhaps the
> > > > > > > >>> > > > > security
> > > > > > > >>> > > > > changes on
> > > > > > > >>> > > > > the
> > > > > > > >>> > > > > AWS
> > > > > > > >>> > > > > server have something to do with what you
are
> > > > experiencing.
> > > > > > > >>> > > > > If
> > > > > > > >>> > > > > that
> > > > > > > >>> > > > > turns
> > > > > > > >>> > > > > out not to be the case, Mallory and Perry
would
> > > > > > > >>> > > > > be best
> > > > > > > >>> > > > > able to
> > > > > > > >>> > > > > help
> > > > > > > >>> > > > > you.
> > > > > > > >>> > > > > If they are unable to help, please let us
know,
> > > > > > > >>> > > > > and we
> > > > can
> > > > > > > >>> > > > > see
> > > > > > > >>> > > > > what
> > > > > > > >>> > > > > they
> > > > > > > >>> > > > > tried and how we can help you get this
problem
> > > > > > > >>> > > > > figured
> > > > out.
> > > > > > > >>> > > > >
> > > > > > > >>> > > > > Julie
> > > > > > > >>> > > > >
> > > > > > > >>> > > > > On Tue, Dec 1, 2020 at 11:15 AM Edward
Strobach
> > > > > > > >>> > > > > -
> > > > > > > >>> > > > > NOAA
> > > > > > > >>> > > > > Affiliate
> > > > > > > >>> > > > > via
> > > > > > > >>> > > > > RT <
> > > > > > > >>> > > > > met_help at ucar.edu> wrote:
> > > > > > > >>> > > > >
> > > > > > > >>> > > > >>
> > > > > > > >>> > > > >> <URL:
> > > > > > > >>> > > > >>
> > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > > >
> > > > > > > >>> > > > >>
> > > > > > > >>> > > > >> Hi Julie,
> > > > > > > >>> > > > >>
> > > > > > > >>> > > > >> I'm actually using the scripts located
here:
> > > > > > > >>> > > > >>
> > > > > > > >>>
> > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts.
> > > > > > > >>> > > > >> These
> > > > > > > >>> > > > >> were scripts that I either got from Mallory
or
> > > > > > > >>> > > > >> Perry a
> > > > > > > >>> > > > >> while
> > > > > > > >>> > > > >> back -
> > > > > > > >>> > > > >> around
> > > > > > > >>> > > > >> May.
> > > > > > > >>> > > > >>
> > > > > > > >>> > > > >> On Tue, Dec 1, 2020 at 12:59 PM Julie
> > > > > > > >>> > > > >> Prestopnik
> > > > > > > >>> > > > >> via
> > > RT
> > > > <
> > > > > > > >>> > > > >> met_help at ucar.edu>
> > > > > > > >>> > > > >> wrote:
> > > > > > > >>> > > > >>
> > > > > > > >>> > > > >> > Hi Edward.
> > > > > > > >>> > > > >> >
> > > > > > > >>> > > > >> > Are you using the scripts that are
located
> > > > > > > >>> > > > >> > in
> > > > > > > >>> > > > >> >
> > > > > >
/gpfs/hps3/emc/global/noscrub/Mallory.Row/VRFY/METviewer_AWS
> > > > > > > >>> > > > >> > and
> > > > > > > >>> > > > >> >
> > > > > >
/gpfs/dell2/emc/verification/noscrub/emc.metplus/METviewer_AWS
> > > > > > > >>> > > > >> > for interaction between AWS and WCOSS?
> > > > > > > >>> > > > >> >
> > > > > > > >>> > > > >> > Thanks!
> > > > > > > >>> > > > >> >
> > > > > > > >>> > > > >> > Julie
> > > > > > > >>> > > > >> >
> > > > > > > >>> > > > >> > On Tue, Dec 1, 2020 at 10:49 AM Edward
> > > > > > > >>> > > > >> > Strobach -
> > > NOAA
> > > > > > > >>> > > > >> > Affiliate
> > > > > > > >>> > > > >> > via RT
> > > > > > > >>> > > > >> <
> > > > > > > >>> > > > >> > met_help at ucar.edu> wrote:
> > > > > > > >>> > > > >> >
> > > > > > > >>> > > > >> > >
> > > > > > > >>> > > > >> > > <URL:
> > > > > > > >>> > > > >> > >
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > > > > > >>> > > > >> > > >
> > > > > > > >>> > > > >> > >
> > > > > > > >>> > > > >> > > Perry does not do batch plotting with
MET.
> > > > > > > >>> > > > >> > > I was
> > > > > > > >>> > > > >> > > under the
> > > > > > > >>> > > > >> > > impression
> > > > > > > >>> > > > >> > that
> > > > > > > >>> > > > >> > > the MET helpdesk helped with MET
related
> > > > > > > >>> > > > >> > > issues.
> > > > > > > >>> > > > >> > >
> > > > > > > >>> > > > >> > > On Tue, Dec 1, 2020 at 12:13 PM Julie
> > > > > > > >>> > > > >> > > Prestopnik
> > > via
> > > > > > > >>> > > > >> > > RT <
> > > > > > > >>> > > > >> > met_help at ucar.edu
> > > > > > > >>> > > > >> > > >
> > > > > > > >>> > > > >> > > wrote:
> > > > > > > >>> > > > >> > >
> > > > > > > >>> > > > >> > > > Hi Edward.
> > > > > > > >>> > > > >> > > >
> > > > > > > >>> > > > >> > > > Please consult with Mallory Row and
> > > > > > > >>> > > > >> > > > Perry
> > > Shafran
> > > > > > > >>> > > > >> > > > for
> > > > > > > >>> > > > >> > > > assistance
> > > > > > > >>> > > > >> with
> > > > > > > >>> > > > >> > > this
> > > > > > > >>> > > > >> > > > issue. They are most familiar with
> > > > > > > >>> > > > >> > > > these
> > > > > > > >>> > > > >> > > > capabilities and
> > > > > > > >>> > > > >> > > > will be
> > > > > > > >>> > > > >> best
> > > > > > > >>> > > > >> > > > able to assist you.
> > > > > > > >>> > > > >> > > >
> > > > > > > >>> > > > >> > > > Thanks!
> > > > > > > >>> > > > >> > > >
> > > > > > > >>> > > > >> > > > Julie
> > > > > > > >>> > > > >> > > >
> > > > > > > >>> > > > >> > > > On Tue, Dec 1, 2020 at 9:25 AM Edward
> > > > > > > >>> > > > >> > > > Strobach -
> > > > > > > >>> > > > >> > > > NOAA
> > > > > > > >>> > > > >> > > > Affiliate via
> > > > > > > >>> > > > >> RT
> > > > > > > >>> > > > >> > <
> > > > > > > >>> > > > >> > > > met_help at ucar.edu> wrote:
> > > > > > > >>> > > > >> > > >
> > > > > > > >>> > > > >> > > > >
> > > > > > > >>> > > > >> > > > > Tue Dec 01 09:25:26 2020: Request
> > > > > > > >>> > > > >> > > > > 97687
> > > > > > > >>> > > > >> > > > > was
> > > > acted
> > > > > > > >>> > > > >> > > > > upon.
> > > > > > > >>> > > > >> > > > > Transaction: Ticket created by
> > > > > > > >>> > > > >> > > > > edward.strobach at noaa.gov
> > > > > > > >>> > > > >> > > > > Queue: met_help
> > > > > > > >>> > > > >> > > > > Subject: can't copy plots over
> > > > > > > >>> > > > >> > > > > Owner: Nobody
> > > > > > > >>> > > > >> > > > > Requestors:
edward.strobach at noaa.gov
> > > > > > > >>> > > > >> > > > > Status: new
> > > > > > > >>> > > > >> > > > > Ticket <URL:
> > > > > > > >>> > > > >> >
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > > > > > >>> > > > >> > > >
> > > > > > > >>> > > > >> > > > >
> > > > > > > >>> > > > >> > > > >
> > > > > > > >>> > > > >> > > > > I'm trying to copy plots over to
Dell
> > > > > > > >>> > > > >> > > > > from the
> > > > MET
> > > > > > > >>> > > > >> > > > > server.
> > > > > > > >>> > > > >> > > > > I
> > > > > > > >>> > > > >> would
> > > > > > > >>> > > > >> > > like
> > > > > > > >>> > > > >> > > > to
> > > > > > > >>> > > > >> > > > > store my plots in the usual place
but
> > > > > > > >>> > > > >> > > > > there is
> > > > an
> > > > > > > >>> > > > >> > > > > scp
> > > > > > > >>> > > > >> > > > > error
> > > > > > > >>> > > > >> > > > > which
> > > > > > > >>> > > > >> I'm
> > > > > > > >>> > > > >> > > > > attaching below.
> > > > > > > >>> > > > >> > > > >
> > > > > > > >>> > > > >> > > > >
> > > > > > > >>> > > > >> > > > > AND BINARY h.fcst_var = 'PMTF'
> > > > > > > >>> > > > >> > > > > AND ld.stat_header_id =
> > > > > > > >>> > > > >> > > > > h.stat_header_id;
> > > > > > > >>> > > > >> > > > >
> > > > > > > >>> > > > >> > > > > Connection to 205.156.8.85 closed
by
> > > > > > > >>> > > > >> > > > > remote
> > > > host.
> > > > > > > >>> > > > >> > > > > CALLING: scp
> > > > > > > >>> > > > >> > > > > edward.strobach at 205.156.8.85
> > > > > > > >>> > > > >> > > > >
://data/mv_data//edward.strobach/plots/*
> > > > > > > >>> > > > >> > > > >
> > > > > > > >>> > > > >> > > > >
> > > > > > > >>> > > > >> > > >
> > > > > > > >>> > > > >> > >
> > > > > > > >>> > > > >> >
> > > > > > > >>> > > > >>
> > > > > > > >>> > >
> > > > > > > >>>
> > > > > >
> > > > >
> > > >
> > >
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > > > > > >>> > > > >> > > > >
> > > > > > > >>> > > > >> > > > > You are accessing a U.S. Government
> > > information
> > > > > > > >>> > > > >> > > > > system, which includes: 1) this
> > > > > > > >>> > > > >> > > > > computer, 2)
> > > > this
> > > > > > > >>> > > > >> > > > > computer network, 3) all computers
> > > > > > > >>> > > > >> > > > > connected
> > > to
> > > > > > > >>> > > > >> > > > > this
> > > > > > > >>> > > > >> > > > > network, and 4) all devices and
> > > > > > > >>> > > > >> > > > > storage
> > > > > > > >>> > > > >> > > > > media
> > > > > > > >>> > > > >> > > > > attached
> > > > > > > >>> > > > >> > > > > to this network or to a computer on
> > > > > > > >>> > > > >> > > > > this
> > > > network.
> > > > > > > >>> > > > >> > > > > You
> > > > > > > >>> > > > >> > > > > understand and consent to the
> > > > > > > >>> > > > >> > > > > following:
> > > > > > > >>> > > > >> > > > > you
> > > may
> > > > > > > >>> > > > >> > > > > access this information system for
> > > > > > > >>> > > > >> > > > > authorized
> > > > use
> > > > > > > >>> > > > >> > > > > only;
> > > > > > > >>> > > > >> > > > > you have no reasonable expectation
of
> > > > > > > >>> > > > >> > > > > privacy
> > > > > > > >>> > > > >> > > > > regarding
> > > > > > > >>> > > > >> > > > > any communication of data
transiting
> > > > > > > >>> > > > >> > > > > or
> > > > > > > >>> > > > >> > > > > stored
> > > > on
> > > > > > > >>> > > > >> > > > > this
> > > > > > > >>> > > > >> > > > > information system; at any time and
> > > > > > > >>> > > > >> > > > > for
> > > > > > > >>> > > > >> > > > > any
> > > > lawful
> > > > > > > >>> > > > >> > > > > Government purpose, the Government
may
> > > monitor,
> > > > > > > >>> > > > >> > > > > intercept, and search and seize any
> > > > communication
> > > > > > > >>> > > > >> > > > > or
> > > > > > > >>> > > > >> > > > > data transiting or stored on this
> > > > > > > >>> > > > >> > > > > information
> > > > > > > >>> > > > >> > > > > system;
> > > > > > > >>> > > > >> > > > > and
> > > > > > > >>> > > > >> > > > > any communications or data
transiting
> > > > > > > >>> > > > >> > > > > or
> > > stored
> > > > on
> > > > > > > >>> > > > >> > > > > this
> > > > > > > >>> > > > >> > > > > information system may be disclosed
or
> > > > > > > >>> > > > >> > > > > used
> > > for
> > > > > > > >>> > > > >> > > > > any
> > > > > > > >>> > > > >> > > > > lawful Government purpose.
> > > > > > > >>> > > > >> > > > >
> > > > > > > >>> > > > >> > > > > scp:
> > > > > > > >>> > > > >> > > > >
//data/mv_data//edward.strobach/plots/*:
> > > No
> > > > > > > >>> > > > >> > > > > such
> > > > > > > >>> > > > >> > > > > file
> > > > > > > >>> > > > >> > > > > or
> > > > > > > >>> > > > >> > directory
> > > > > > > >>> > > > >> > > > > ERROR: Command returned with non-
zero
> > > > > > > >>> > > > >> > > > > status
> > > > (1):
> > > > > > > >>> > > > >> > > > > scp
> > > > > > > >>> > > > >> > edward.strobach@
> > > > > > > >>> > > > >> > > > > 205.156.8.85
> > > > > > ://data/mv_data//edward.strobach/plots/*
> > > > > > > >>> > > > >> > > > >
> > > > > > > >>> > > > >> > > > >
> > > > > > > >>> > > > >> > > >
> > > > > > > >>> > > > >> > >
> > > > > > > >>> > > > >> >
> > > > > > > >>> > > > >>
> > > > > > > >>> > >
> > > > > > > >>>
> > > > > >
> > > > >
> > > >
> > >
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > > > > > >>> > > > >> > > > > ++ printf '\033]0;%s@%s:%s\007'
> > > Edward.Strobach
> > > > > > > >>> > > > >> > > > > v71a1
> > > > > > > >>> > > > >> > > > >
> > > > > > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > > > > > > >>> > > > >> > > > >
> > > > > > > >>> > > > >> > > > >
> > > > > > > >>> > > > >> > > > > NOTE: nothing about this set-up
has
> > > > > > > >>> > > > >> > > > > changed
> > > > other
> > > > > > > >>> > > > >> > > > > than
> > > > > > > >>> > > > >> > > > > the
> > > > > > > >>> > > > >> directory
> > > > > > > >>> > > > >> > > > path
> > > > > > > >>> > > > >> > > > > onto dell. The path has been
created
> > > > > > > >>> > > > >> > > > > so
> > > > > > > >>> > > > >> > > > > it
> > > > should
> > > > > > > >>> > > > >> > > > > support
> > > > > > > >>> > > > >> > > > > the
> > > > > > > >>> > > > >> > transfer
> > > > > > > >>> > > > >> > > > of
> > > > > > > >>> > > > >> > > > > plots generated on the MET server.
> > > > > > > >>> > > > >> > > > > The
> > > > preceding
> > > > > > > >>> > > > >> > > > > information
> > > > > > > >>> > > > >> > indicates
> > > > > > > >>> > > > >> > > > > that the plot was processed
> > > > > > > >>> > > > >> > > > > successfully.
> > > It's
> > > > > > > >>> > > > >> > > > > not
> > > > > > > >>> > > > >> > > > > clear
> > > > > > > >>> > > > >> > > > > why I
> > > > > > > >>> > > > >> can't
> > > > > > > >>> > > > >> > > > > transfer my plots over.
> > > > > > > >>> > > > >> > > > > --
> > > > > > > >>> > > > >> > > > > Edward Strobach
> > > > > > > >>> > > > >> > > > > EMC/NCEP/NWS/
> > > > > > > >>> > > > >> > > > > IMSG Contractor
> > > > > > > >>> > > > >> > > > > Cubicle#: 2029
> > > > > > > >>> > > > >> > > > > 301-683-3717
> > > > > > > >>> > > > >> > > > >
> > > > > > > >>> > > > >> > > > >
> > > > > > > >>> > > > >> > > >
> > > > > > > >>> > > > >> > > > --
> > > > > > > >>> > > > >> > > > Julie Prestopnik (she/her)
> > > > > > > >>> > > > >> > > > Software Engineer
> > > > > > > >>> > > > >> > > > National Center for Atmospheric
Research
> > > > > > > >>> > > > >> > > > Research Applications Laboratory
> > > > > > > >>> > > > >> > > > Email: jpresto at ucar.edu
> > > > > > > >>> > > > >> > > >
> > > > > > > >>> > > > >> > > > My working day may not be your
working
> > > > > > > >>> > > > >> > > > day.
> > > > Please
> > > > > > > >>> > > > >> > > > do not
> > > > > > > >>> > > > >> > > > feel
> > > > > > > >>> > > > >> obliged
> > > > > > > >>> > > > >> > > to
> > > > > > > >>> > > > >> > > > reply to this email outside of your
> > > > > > > >>> > > > >> > > > normal
> > > working
> > > > > > > >>> > > > >> > > > hours.
> > > > > > > >>> > > > >> > > >
> > > > > > > >>> > > > >> > > >
> > > > > > > >>> > > > >> > >
> > > > > > > >>> > > > >> > > --
> > > > > > > >>> > > > >> > > Edward Strobach
> > > > > > > >>> > > > >> > > EMC/NCEP/NWS/
> > > > > > > >>> > > > >> > > IMSG Contractor
> > > > > > > >>> > > > >> > > Cubicle#: 2029
> > > > > > > >>> > > > >> > > 301-683-3717
> > > > > > > >>> > > > >> > >
> > > > > > > >>> > > > >> > >
> > > > > > > >>> > > > >> >
> > > > > > > >>> > > > >> > --
> > > > > > > >>> > > > >> > Julie Prestopnik (she/her)
> > > > > > > >>> > > > >> > Software Engineer
> > > > > > > >>> > > > >> > National Center for Atmospheric Research
> > > > > > > >>> > > > >> > Research Applications Laboratory
> > > > > > > >>> > > > >> > Email: jpresto at ucar.edu
> > > > > > > >>> > > > >> >
> > > > > > > >>> > > > >> > My working day may not be your working
day.
> > > > > > > >>> > > > >> > Please
> > > do
> > > > > > > >>> > > > >> > not
> > > > > > > >>> > > > >> > feel
> > > > > > > >>> > > > >> > obliged
> > > > > > > >>> > > > >> to
> > > > > > > >>> > > > >> > reply to this email outside of your
normal
> > > > > > > >>> > > > >> > working
> > > > > > > >>> > > > >> > hours.
> > > > > > > >>> > > > >> >
> > > > > > > >>> > > > >> >
> > > > > > > >>> > > > >>
> > > > > > > >>> > > > >> --
> > > > > > > >>> > > > >> Edward Strobach
> > > > > > > >>> > > > >> EMC/NCEP/NWS/
> > > > > > > >>> > > > >> IMSG Contractor
> > > > > > > >>> > > > >> Cubicle#: 2029
> > > > > > > >>> > > > >> 301-683-3717
> > > > > > > >>> > > > >>
> > > > > > > >>> > > > >>
> > > > > > > >>> > > > >
> > > > > > > >>> > > > > --
> > > > > > > >>> > > > > Julie Prestopnik (she/her)
> > > > > > > >>> > > > > Software Engineer
> > > > > > > >>> > > > > National Center for Atmospheric Research
> > > > > > > >>> > > > > Research Applications Laboratory
> > > > > > > >>> > > > > Email: jpresto at ucar.edu
> > > > > > > >>> > > > >
> > > > > > > >>> > > > > My working day may not be your working day.
> > > > > > > >>> > > > > Please do
> > > > not
> > > > > > > >>> > > > > feel
> > > > > > > >>> > > > > obliged to
> > > > > > > >>> > > > > reply to this email outside of your normal
> > > > > > > >>> > > > > working
> > > hours.
> > > > > > > >>> > > > >
> > > > > > > >>> > >
> > > > > > > >>> > >
> > > > > > > >>> > >
> > > > > > > >>> > >
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >> Edward Strobach
> > > > > > > >> EMC/NCEP/NWS/
> > > > > > > >> IMSG Contractor
> > > > > > > >> Cubicle#: 2029
> > > > > > > >> 301-683-3717
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Edward Strobach
> > > > > > > > EMC/NCEP/NWS/
> > > > > > > > IMSG Contractor
> > > > > > > > Cubicle#: 2029
> > > > > > > > 301-683-3717
> > > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Edward Strobach
> > > > > EMC/NCEP/NWS/
> > > > > IMSG Contractor
> > > > > Cubicle#: 2029
> > > > > 301-683-3717
> > > > >
> > > > >
> > > >
> > > > --
> > > > Julie Prestopnik (she/her)
> > > > Software Engineer
> > > > National Center for Atmospheric Research
> > > > Research Applications Laboratory
> > > > Email: jpresto at ucar.edu
> > > >
> > > > My working day may not be your working day. Please do not
feel
> > > > obliged
> > > to
> > > > reply to this email outside of your normal working hours.
> > > >
> > > >
> > >
> > > --
> > > Edward Strobach
> > > EMC/NCEP/NWS/
> > > IMSG Contractor
> > > Cubicle#: 2029
> > > 301-683-3717
> > >
> > >
------------------------------------------------
Subject: can't copy plots over
From: Edward Strobach - NOAA Affiliate
Time: Thu Dec 03 08:28:52 2020
If you mean this suggestions:
<drop_indexes>false</drop_indexes>
<apply_indexes>false</apply_indexes>
then yes. However, until just now I didn't realize that my
load_to_metviewer_AWS.py set the template to have these options as
true. I
have since changed these options to false in the script. There are
some
issues in my set-up with the Taylor Diagrams and CSI that I need to
correct
for. Will the above option permit me to conduct frequent tests if
needed?
On Thu, Dec 3, 2020 at 10:04 AM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> Edward
> AWS has 3 loading jobs that wore started by you:
> 2 for load_mv_cmaq_metplus_CMAQ.xml and 2 for
> load_mv_cmaq_metplus_CMAQ_Meteorology.xml
> and 25 batch jobs.
> None of them are executing because of the database locks created by
so
> many incerts, create indexes and select statments.
> Shoud I kill all your jbs aagain? Have you had a chance to look at
my
> sugesstuion on how to improve the process?
>
> Tatiana
>
> On Wed Dec 02 10:10:40 2020, tatiana wrote:
> > Edward
> > I am sorry, I did not mentioned that the commands I sent you
should be
> > ran on AWS server and not on WCOSS
> >
> > Tatiana
> >
> > On Wed Dec 02 09:42:20 2020, jpresto wrote:
> > > Hi Edward.
> > >
> > > You should be able to distinguish what processes are MET and
what are
> > > not.
> > > From your "metcheck" results, it seems you have no MET processes
> > > running.
> > > I'll explain below.
> > >
> > > [Edward.Strobach at v71a2 ~]$ metcheck
> > > > root 76468 12071 0 16:25 ? 00:00:00 sshd:
> > > > Edward.Strobach
> > > > [priv]
> > > > Edward.+ 76482 76468 0 16:25 ? 00:00:00 sshd:
> > > > Edward.Strobach at pts
> > > > /19
> > > > Edward.+ 85480 82420 0 16:26 pts/19 00:00:00 grep
--color=auto
> > > > Edward.Strobach
> > > > root 141334 12071 0 12:27 ? 00:00:00 sshd:
> > > > Edward.Strobach
> > > > [priv]
> > > > Edward.+ 141337 141334 0 12:27 ? 00:00:00 sshd:
> > > > Edward.Strobach at pts
> > > > /5
> > >
> > >
> > > All of the processes that start with "sshd" are your logins, so
only
> > > if you
> > > ran "kill -9" on one of those would you get logged out. The
"grep"
> > > processes (3rd result) is your running of "ps -ef | grep
> > > Edward.Strobach".
> > > If there were MET processes running you would be able to see
them
> > > here.
> > > The fact that there are none listed here could be because when
you
> > > ran
> > > "pkill -9 -u `id -u Edward.Strobach`", they were all killed, or
> > > because
> > > you're on a different node (v71a2) than the one you ran them on.
> > >
> > > Julie
> > >
> > > On Wed, Dec 2, 2020 at 9:28 AM Edward Strobach - NOAA Affiliate
via
> > > RT
> > > <
> > > met_help at ucar.edu> wrote:
> > >
> > > >
> > > > <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
>
> > > >
> > > > Thanks. I was able to determine the processes using ps -ef |
grep
> > > > Edward.Strobach
> > > >
> > > > I did try pkill -9 -u `id -u Edward.Strobach` before sending
your
> > > > email
> > > > which did log me out unfortunately.
> > > >
> > > > I also noticed that kill -9 jobid also logs me out. I wish
there
> > > > was
> > > > a way
> > > > to distinguish between MET tasks v. other tasks. When I look
at
> > > > what
> > > > I
> > > > currently have I can't tell what is MET and what is not MET.
> > > > [Edward.Strobach at v71a2 ~]$ metcheck
> > > > root 76468 12071 0 16:25 ? 00:00:00 sshd:
> > > > Edward.Strobach
> > > > [priv]
> > > > Edward.+ 76482 76468 0 16:25 ? 00:00:00 sshd:
> > > > Edward.Strobach at pts
> > > > /19
> > > > Edward.+ 85480 82420 0 16:26 pts/19 00:00:00 grep
--color=auto
> > > > Edward.Strobach
> > > > root 141334 12071 0 12:27 ? 00:00:00 sshd:
> > > > Edward.Strobach
> > > > [priv]
> > > > Edward.+ 141337 141334 0 12:27 ? 00:00:00 sshd:
> > > > Edward.Strobach at pts
> > > > /5
> > > >
> > > > Note that metcheck is just an alias for ps -ef | grep
> > > > Edward.Strobach
> > > >
> > > > On Wed, Dec 2, 2020 at 11:13 AM Julie Prestopnik via RT
> > > > <met_help at ucar.edu
> > > > >
> > > > wrote:
> > > >
> > > > > Hi Edward.
> > > > >
> > > > > Because capitals matter when running "grep" and because
WCOSS
> > > > > uses
> > > > capital
> > > > > letters for our username you can run either of the following
> > > > > instead:
> > > > >
> > > > > ps -ef | grep -i edward (the "-i" tells grep to ignore case)
> > > > > ps -ef | grep Edward
> > > > >
> > > > > The second column will list the process id and the third
column
> > > > > will list
> > > > > the parent process id.
> > > > >
> > > > > The same issue exists with running pkill on WCOSS. You can
run
> > > > > pkill on
> > > > > WCOSS using the following, however, I don't recommend it and
I'll
> > > > > explain
> > > > > below:
> > > > >
> > > > > pkill -9 -u `id -u Edward.Strobach`
> > > > >
> > > > > First, you'd need to make sure you're logged in
appropriately and
> > > > > be on
> > > > the
> > > > > same node (not sure if that is the correct terminology) as
the
> > > > > one
> > > > running
> > > > > your processes. For example, I ran a simple check to test
this.
> > > > > I
> > > > > ran
> > > > > "top" when I was logged into "Julie.Prestopnik@*v71a3*", but
my
> > > > > second
> > > > log
> > > > > in was on "Julie.Prestopnik@*v72a1*". v72a1 did not
recognize
> > > > > "top" as
> > > > > running, which I could see when running "ps -ef | grep
Julie".
> > > > > When I
> > > > ran
> > > > > "ssh v71a3" and then reran "ps -ef | grep Julie", I could
then
> > > > > see
> > > > > "top"
> > > > > was running. Running "pkill -9 -u `id -u Julie.Presotpnik`"
also
> > > > > closed
> > > > my
> > > > > connection to WCOSS, so if you don't want your connection to
> > > > > WCOSS
> > > > killed,
> > > > > which I didn't, I would just kill the processes individually
> > > > > using
> > > > > the
> > > > kill
> > > > > command with the process number.
> > > > >
> > > > > I hope that helps!
> > > > >
> > > > > Julie
> > > > >
> > > > > On Wed, Dec 2, 2020 at 8:51 AM Edward Strobach - NOAA
Affiliate
> > > > > via
> > > > > RT <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
> > > > > >
> > > > > > I see this when running ps -ef | grep edward
> > > > > > Edward.+ 115762 154642 0 15:48 pts/31 00:00:00 grep
> > > > > > --color=auto
> > > > > edward
> > > > > >
> > > > > > The numbers there change every time I run this step.
> > > > > >
> > > > > > pkill -9 -u `id -u edward.strobach` does not seem to work.
> > > > > >
> > > > > > On Wed, Dec 2, 2020 at 9:34 AM Tatiana Burek via RT
> > > > > > <met_help at ucar.edu
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > You can kill your processes by :
> > > > > > > - collecting what is running:
> > > > > > > ps -ef | grep edward
> > > > > > > - killing each process
> > > > > > > kill -9 2574
> > > > > > > kill -9 2594
> > > > > > > or all your processes:
> > > > > > > pkill -9 -u `id -u edward.strobach`
> > > > > > >
> > > > > > > Right now you have 4 loading scripts and 10 batch
scripts
> > > > > > > running at
> > > > > the
> > > > > > > same time which creates a gridlock and none of the
processes
> > > > > > > are
> > > > > getting
> > > > > > > executes.
> > > > > > >
> > > > > > > Tatiana
> > > > > > >
> > > > > > > On Wed Dec 02 06:32:23 2020, edward.strobach at noaa.gov
wrote:
> > > > > > > > I came across this issue again this morning. Is there
a
> > > > > > > > way
> > > > > > > > to
> > > > stop
> > > > > > > > certain processes and start fresh from my end?
> > > > > > > >
> > > > > > > > On Tue, Dec 1, 2020 at 5:14 PM Edward Strobach - NOAA
> > > > > > > > Affiliate <
> > > > > > > > edward.strobach at noaa.gov> wrote:
> > > > > > > >
> > > > > > > > > maybe there is a script that allows me to check what
is
> > > > > > > > > going on
> > > > > with
> > > > > > > > > my
> > > > > > > > > database? A lot of these issues I've come across
happen
> > > > > > > > > because
> > > > of
> > > > > > > > > things
> > > > > > > > > I'm unaware of or things that go on simultaneously
that
> > > > > > > > > causes
> > > > > > > > > confusion in
> > > > > > > > > the processing. Right now I have a plotting script
set
> > > > > > > > > that
> > > > > > > > > simultaneously
> > > > > > > > > submits batch jobs for time series statistics,
Taylor
> > > > > > > > > diagrams,
> > > > and
> > > > > > > > > CSI. I
> > > > > > > > > haven't seen too many issues when I do that, but
would it
> > > > > > > > > make
> > > > > sense
> > > > > > > > > to
> > > > > > > > > insert a while loop after the first submission that
uses
> > > > > > > > > a
> > > > > > > > > script
> > > > > to
> > > > > > > > > check
> > > > > > > > > the activity within my database? In other words, I
would
> > > > > > > > > use a
> > > > > > > > > script,
> > > > > > > > > perhaps one that has already been developed, to
check the
> > > > > > > > > status
> > > > of
> > > > > > > > > processing before submitting the next batch job.
Once
> > > > > > > > > the
> > > > > processing
> > > > > > > > > for
> > > > > > > > > the first plotting steps have completed, then I
would
> > > > > > > > > commence to
> > > > > the
> > > > > > > > > next
> > > > > > > > > one.
> > > > > > > > >
> > > > > > > > > I guess check_my_db.sh would not do that?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Dec 1, 2020 at 4:27 PM Edward Strobach -
NOAA
> > > > > > > > > Affiliate <
> > > > > > > > > edward.strobach at noaa.gov> wrote:
> > > > > > > > >
> > > > > > > > >> are there youtube examples or a section in a
technical
> > > > > > > > >> manual
> > > > that
> > > > > > > > >> goes
> > > > > > > > >> over examples illustrating best practices related
to
> > > > > > > > >> this
> > > > process?
> > > > > > > > >>
> > > > > > > > >> On Tue, Dec 1, 2020 at 4:03 PM Tatiana Burek via RT
> > > > > > > > >> <met_help at ucar.edu>
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >>> For your process you need only two scripts:
> > > > > > > > >>> mv_load_to_aws.sh
> > > > > > > > >>> mv_batch_on_aws.sh
> > > > > > > > >>> I don't think, that others would help you.
> > > > > > > > >>>
> > > > > > > > >>> The main work is in creating efficient XMLs for
the
> > > > > > > > >>> each
> > > > > > > > >>> step.
> > > > > > > > >>> Like not recalculating indexes or performing
> > > > stat_header_db_check
> > > > > > > > >>> during
> > > > > > > > >>> data loading.
> > > > > > > > >>>
> > > > > > > > >>> Tatiana
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> On Tue Dec 01 13:00:15 2020,
edward.strobach at noaa.gov
> > > > > > > > >>> wrote:
> > > > > > > > >>> > Thanks. I just made those changes in the xml
file,
> > > > > > > > >>> > so
> > > > > > > > >>> > I
> > > > > 'guess'
> > > > > > > > >>> > I'm
> > > > > > > > >>> > ready
> > > > > > > > >>> > to go.
> > > > > > > > >>> >
> > > > > > > > >>> > I should note that I have several steps that I
run
> > > > > > > > >>> > each
> > > > day. I
> > > > > > > > >>> > convert
> > > > > > > > >>> > files into netcdf files that can be read by MET,
I
> > > > > > > > >>> > generate
> > > > the
> > > > > > > > >>> > stat
> > > > > > > > >>> > files,
> > > > > > > > >>> > and I use the stat files in the MET batch
plotting
> > > > > > > > >>> > system. I
> > > > > > > > >>> > have
> > > > > > > > >>> > actually
> > > > > > > > >>> > 5 total steps, but the plot batch system is
really
> > > > > > > > >>> > the
> > > > > > > > >>> > only
> > > > one
> > > > > > > > >>> > that
> > > > > > > > >>> > gives
> > > > > > > > >>> > me trouble on a regular basis. I wish I knew
how to
> > > > > > > > >>> > reach a
> > > > > > > > >>> > point
> > > > > > > > >>> > where I
> > > > > > > > >>> > have a plotting package that would produce the
plots
> > > > > > > > >>> > that I
> > > > > need
> > > > > > > > >>> > each
> > > > > > > > >>> > day.
> > > > > > > > >>> > I'm expected to hand the final product off in
the
> > > > > > > > >>> > near
> > > > future,
> > > > > > > > >>> > but
> > > > > > > > >>> > this
> > > > > > > > >>> > part has delayed me extensively. I know that I'm
> > > > > > > > >>> > generating a
> > > > > lot
> > > > > > > > >>> > of
> > > > > > > > >>> > plots
> > > > > > > > >>> > which could potentially overwhelm the system,
but I'm
> > > > > > > > >>> > unaware
> > > > > how
> > > > > > > > >>> > others
> > > > > > > > >>> > work to circumvent that. I hope to move past
this
> > > > > > > > >>> > soon
> > > > > > > > >>> > so I
> > > > > can
> > > > > > > > >>> > continue
> > > > > > > > >>> > development work.
> > > > > > > > >>> >
> > > > > > > > >>> > There are several scripts that I don't use that
might
> > > > > > > > >>> > be
> > > > > helpful.
> > > > > > > > >>> > Do
> > > > > > > > >>> > any
> > > > > > > > >>> > of these work to help me manage the system
better?
> > > > > > > > >>> > check_my_db.sh mv_batch_on_aws.sh
> > > > > > > > >>> > mv_create_db_on_aws.sh
> > > > > > > > >>> > mv_db_size_on_aws.sh mv_delete_db_on_aws.sh
> > > > > mv_load_to_aws.sh
> > > > > > > > >>> >
> > > > > > > > >>> >
> > > > > > > > >>> > On Tue, Dec 1, 2020 at 2:44 PM Tatiana Burek via
RT
> > > > > > > > >>> > <met_help at ucar.edu>
> > > > > > > > >>> > wrote:
> > > > > > > > >>> >
> > > > > > > > >>> > > Edward
> > > > > > > > >>> > >
> > > > > > > > >>> > > I run a few butch jobs from my laptop and made
sure
> > > > > > > > >>> > > that
> > > > > > > > >>> > > METviewer
> > > > > > > > >>> > > code
> > > > > > > > >>> > > and scripts working as expected: I was able to
> > > > > > > > >>> > > produce and
> > > > > copy
> > > > > > > > >>> > > plots.
> > > > > > > > >>> > > I also got an error like the one that you
reported
> > > > > > > > >>> > > but only
> > > > > > > > >>> > > when
> > > > > > > > >>> > > METviewer
> > > > > > > > >>> > > did not generate any plots and the script
couldn't
> > > > > > > > >>> > > copy
> > > > empty
> > > > > > > > >>> > > directory.
> > > > > > > > >>> > >
> > > > > > > > >>> > > During my testing I noticed a few batch jobs
that
> > > > > > > > >>> > > where
> > > > > > > > >>> > > started by
> > > > > > > > >>> > > you
> > > > > > > > >>> > > and were handing. Each of these jobs started a
> > > > > > > > >>> > > database
> > > > query
> > > > > > > > >>> > > and
> > > > > > > > >>> > > they also
> > > > > > > > >>> > > were piling up and handing. They all were
waiting
> > > > > > > > >>> > > for
> > > > > > > > >>> > > the
> > > > > > > > >>> > > indexing
> > > > > > > > >>> > > that
> > > > > > > > >>> > > takes place after the data loading.
> > > > > > > > >>> > > I had to kill all database and batch processes
in
> > > > > > > > >>> > > order to
> > > > > move
> > > > > > > > >>> > > forward
> > > > > > > > >>> > > with the testing.
> > > > > > > > >>> > > To avoid this gridlock you could:
> > > > > > > > >>> > > - in your loading xml use
> > > > <drop_indexes>false</drop_indexes>
> > > > > > > > >>> > > <apply_indexes>false</apply_indexes>
> > > > > > > > >>> > >
> > > > > > > > >>> > >
> > > > > > > > >>> > > - do not start a few parallel batch jobs
> > > > > > > > >>> > >
> > > > > > > > >>> > > I will add a info message to the script that
would
> > > > > > > > >>> > > explain
> > > > > why
> > > > > > > > >>> > > the
> > > > > > > > >>> > > error
> > > > > > > > >>> > > occurs.
> > > > > > > > >>> > > For now, could you test batch with some simple
XML
> > > > > > > > >>> > > that
> > > > > produce
> > > > > > > > >>> > > at
> > > > > > > > >>> > > least
> > > > > > > > >>> > > one image.
> > > > > > > > >>> > >
> > > > > > > > >>> > > Tatiana
> > > > > > > > >>> > >
> > > > > > > > >>> > >
> > > > > > > > >>> > >
> > > > > > > > >>> > >
> > > > > > > > >>> > >
> > > > > > > > >>> > >
> > > > > > > > >>> > >
> > > > > > > > >>> > > On Tue Dec 01 11:35:03 2020, jpresto wrote:
> > > > > > > > >>> > > > Hi Edward.
> > > > > > > > >>> > > >
> > > > > > > > >>> > > > After talking with Tatiana, it sounds like I
am
> > > > misinformed
> > > > > > > > >>> > > > and the
> > > > > > > > >>> > > > scripts
> > > > > > > > >>> > > > you are using were, in fact, developed by
our
> > > > > > > > >>> > > > METviewer
> > > > > team.
> > > > > > > > >>> > > > My
> > > > > > > > >>> > > > apologies. METplus has a lot of components,
and
> > > > > > > > >>> > > > while I
> > > > > > > > >>> > > > thought I
> > > > > > > > >>> > > > had
> > > > > > > > >>> > > > a
> > > > > > > > >>> > > > good understanding of who did what, it turns
out,
> > > > > > > > >>> > > > I
> > > > > > > > >>> > > > did
> > > > > not.
> > > > > > > > >>> > > > Tatiana
> > > > > > > > >>> > > > is
> > > > > > > > >>> > > > working on this issue and will follow up
once she
> > > > > > > > >>> > > > knows
> > > > > more.
> > > > > > > > >>> > > > Thank
> > > > > > > > >>> > > > you
> > > > > > > > >>> > > > for your patience and understanding.
> > > > > > > > >>> > > >
> > > > > > > > >>> > > > Julie
> > > > > > > > >>> > > >
> > > > > > > > >>> > > > On Tue, Dec 1, 2020 at 11:22 AM Julie
Prestopnik
> > > > > > > > >>> > > > <jpresto at ucar.edu
> > > > > > > > >>> >
> > > > > > > > >>> > > > wrote:
> > > > > > > > >>> > > >
> > > > > > > > >>> > > > > Thanks, Edward. I directed you to Mallory
and
> > > > > > > > >>> > > > > Perry
> > > > > > > > >>> > > > > because I
> > > > > > > > >>> > > > > thought you
> > > > > > > > >>> > > > > might be using scripts from them. We did
not
> > > > > > > > >>> > > > > develop
> > > > and
> > > > > > > > >>> > > > > have
> > > > > > > > >>> > > > > not
> > > > > > > > >>> > > > > used
> > > > > > > > >>> > > > > those scripts. We have no experience
> > > > > > > > >>> > > > > interacting
> > > > > > > > >>> > > > > with
> > > > > AWS
> > > > > > > > >>> > > > > and
> > > > > > > > >>> > > > > WCOSS,
> > > > > > > > >>> > > > > so
> > > > > > > > >>> > > > > they seemed best suited to answer your
> > > > > > > > >>> > > > > questions
> > > > related
> > > > > to
> > > > > > > > >>> > > > > that
> > > > > > > > >>> > > > > process.
> > > > > > > > >>> > > > > However, as Tatiana pointed out, perhaps
the
> > > > > > > > >>> > > > > security
> > > > > > > > >>> > > > > changes on
> > > > > > > > >>> > > > > the
> > > > > > > > >>> > > > > AWS
> > > > > > > > >>> > > > > server have something to do with what you
are
> > > > > experiencing.
> > > > > > > > >>> > > > > If
> > > > > > > > >>> > > > > that
> > > > > > > > >>> > > > > turns
> > > > > > > > >>> > > > > out not to be the case, Mallory and Perry
would
> > > > > > > > >>> > > > > be best
> > > > > > > > >>> > > > > able to
> > > > > > > > >>> > > > > help
> > > > > > > > >>> > > > > you.
> > > > > > > > >>> > > > > If they are unable to help, please let us
know,
> > > > > > > > >>> > > > > and we
> > > > > can
> > > > > > > > >>> > > > > see
> > > > > > > > >>> > > > > what
> > > > > > > > >>> > > > > they
> > > > > > > > >>> > > > > tried and how we can help you get this
problem
> > > > > > > > >>> > > > > figured
> > > > > out.
> > > > > > > > >>> > > > >
> > > > > > > > >>> > > > > Julie
> > > > > > > > >>> > > > >
> > > > > > > > >>> > > > > On Tue, Dec 1, 2020 at 11:15 AM Edward
Strobach
> > > > > > > > >>> > > > > -
> > > > > > > > >>> > > > > NOAA
> > > > > > > > >>> > > > > Affiliate
> > > > > > > > >>> > > > > via
> > > > > > > > >>> > > > > RT <
> > > > > > > > >>> > > > > met_help at ucar.edu> wrote:
> > > > > > > > >>> > > > >
> > > > > > > > >>> > > > >>
> > > > > > > > >>> > > > >> <URL:
> > > > > > > > >>> > > > >>
> > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > > > >
> > > > > > > > >>> > > > >>
> > > > > > > > >>> > > > >> Hi Julie,
> > > > > > > > >>> > > > >>
> > > > > > > > >>> > > > >> I'm actually using the scripts located
here:
> > > > > > > > >>> > > > >>
> > > > > > > > >>>
> > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts.
> > > > > > > > >>> > > > >> These
> > > > > > > > >>> > > > >> were scripts that I either got from
Mallory or
> > > > > > > > >>> > > > >> Perry a
> > > > > > > > >>> > > > >> while
> > > > > > > > >>> > > > >> back -
> > > > > > > > >>> > > > >> around
> > > > > > > > >>> > > > >> May.
> > > > > > > > >>> > > > >>
> > > > > > > > >>> > > > >> On Tue, Dec 1, 2020 at 12:59 PM Julie
> > > > > > > > >>> > > > >> Prestopnik
> > > > > > > > >>> > > > >> via
> > > > RT
> > > > > <
> > > > > > > > >>> > > > >> met_help at ucar.edu>
> > > > > > > > >>> > > > >> wrote:
> > > > > > > > >>> > > > >>
> > > > > > > > >>> > > > >> > Hi Edward.
> > > > > > > > >>> > > > >> >
> > > > > > > > >>> > > > >> > Are you using the scripts that are
located
> > > > > > > > >>> > > > >> > in
> > > > > > > > >>> > > > >> >
> > > > > > >
/gpfs/hps3/emc/global/noscrub/Mallory.Row/VRFY/METviewer_AWS
> > > > > > > > >>> > > > >> > and
> > > > > > > > >>> > > > >> >
> > > > > > >
/gpfs/dell2/emc/verification/noscrub/emc.metplus/METviewer_AWS
> > > > > > > > >>> > > > >> > for interaction between AWS and WCOSS?
> > > > > > > > >>> > > > >> >
> > > > > > > > >>> > > > >> > Thanks!
> > > > > > > > >>> > > > >> >
> > > > > > > > >>> > > > >> > Julie
> > > > > > > > >>> > > > >> >
> > > > > > > > >>> > > > >> > On Tue, Dec 1, 2020 at 10:49 AM Edward
> > > > > > > > >>> > > > >> > Strobach -
> > > > NOAA
> > > > > > > > >>> > > > >> > Affiliate
> > > > > > > > >>> > > > >> > via RT
> > > > > > > > >>> > > > >> <
> > > > > > > > >>> > > > >> > met_help at ucar.edu> wrote:
> > > > > > > > >>> > > > >> >
> > > > > > > > >>> > > > >> > >
> > > > > > > > >>> > > > >> > > <URL:
> > > > > > > > >>> > > > >> > >
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > > > > > > >>> > > > >> > > >
> > > > > > > > >>> > > > >> > >
> > > > > > > > >>> > > > >> > > Perry does not do batch plotting with
MET.
> > > > > > > > >>> > > > >> > > I was
> > > > > > > > >>> > > > >> > > under the
> > > > > > > > >>> > > > >> > > impression
> > > > > > > > >>> > > > >> > that
> > > > > > > > >>> > > > >> > > the MET helpdesk helped with MET
related
> > > > > > > > >>> > > > >> > > issues.
> > > > > > > > >>> > > > >> > >
> > > > > > > > >>> > > > >> > > On Tue, Dec 1, 2020 at 12:13 PM Julie
> > > > > > > > >>> > > > >> > > Prestopnik
> > > > via
> > > > > > > > >>> > > > >> > > RT <
> > > > > > > > >>> > > > >> > met_help at ucar.edu
> > > > > > > > >>> > > > >> > > >
> > > > > > > > >>> > > > >> > > wrote:
> > > > > > > > >>> > > > >> > >
> > > > > > > > >>> > > > >> > > > Hi Edward.
> > > > > > > > >>> > > > >> > > >
> > > > > > > > >>> > > > >> > > > Please consult with Mallory Row and
> > > > > > > > >>> > > > >> > > > Perry
> > > > Shafran
> > > > > > > > >>> > > > >> > > > for
> > > > > > > > >>> > > > >> > > > assistance
> > > > > > > > >>> > > > >> with
> > > > > > > > >>> > > > >> > > this
> > > > > > > > >>> > > > >> > > > issue. They are most familiar with
> > > > > > > > >>> > > > >> > > > these
> > > > > > > > >>> > > > >> > > > capabilities and
> > > > > > > > >>> > > > >> > > > will be
> > > > > > > > >>> > > > >> best
> > > > > > > > >>> > > > >> > > > able to assist you.
> > > > > > > > >>> > > > >> > > >
> > > > > > > > >>> > > > >> > > > Thanks!
> > > > > > > > >>> > > > >> > > >
> > > > > > > > >>> > > > >> > > > Julie
> > > > > > > > >>> > > > >> > > >
> > > > > > > > >>> > > > >> > > > On Tue, Dec 1, 2020 at 9:25 AM
Edward
> > > > > > > > >>> > > > >> > > > Strobach -
> > > > > > > > >>> > > > >> > > > NOAA
> > > > > > > > >>> > > > >> > > > Affiliate via
> > > > > > > > >>> > > > >> RT
> > > > > > > > >>> > > > >> > <
> > > > > > > > >>> > > > >> > > > met_help at ucar.edu> wrote:
> > > > > > > > >>> > > > >> > > >
> > > > > > > > >>> > > > >> > > > >
> > > > > > > > >>> > > > >> > > > > Tue Dec 01 09:25:26 2020: Request
> > > > > > > > >>> > > > >> > > > > 97687
> > > > > > > > >>> > > > >> > > > > was
> > > > > acted
> > > > > > > > >>> > > > >> > > > > upon.
> > > > > > > > >>> > > > >> > > > > Transaction: Ticket created by
> > > > > > > > >>> > > > >> > > > > edward.strobach at noaa.gov
> > > > > > > > >>> > > > >> > > > > Queue: met_help
> > > > > > > > >>> > > > >> > > > > Subject: can't copy plots
over
> > > > > > > > >>> > > > >> > > > > Owner: Nobody
> > > > > > > > >>> > > > >> > > > > Requestors:
edward.strobach at noaa.gov
> > > > > > > > >>> > > > >> > > > > Status: new
> > > > > > > > >>> > > > >> > > > > Ticket <URL:
> > > > > > > > >>> > > > >> >
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > > > > > > >>> > > > >> > > >
> > > > > > > > >>> > > > >> > > > >
> > > > > > > > >>> > > > >> > > > >
> > > > > > > > >>> > > > >> > > > > I'm trying to copy plots over to
Dell
> > > > > > > > >>> > > > >> > > > > from the
> > > > > MET
> > > > > > > > >>> > > > >> > > > > server.
> > > > > > > > >>> > > > >> > > > > I
> > > > > > > > >>> > > > >> would
> > > > > > > > >>> > > > >> > > like
> > > > > > > > >>> > > > >> > > > to
> > > > > > > > >>> > > > >> > > > > store my plots in the usual place
but
> > > > > > > > >>> > > > >> > > > > there is
> > > > > an
> > > > > > > > >>> > > > >> > > > > scp
> > > > > > > > >>> > > > >> > > > > error
> > > > > > > > >>> > > > >> > > > > which
> > > > > > > > >>> > > > >> I'm
> > > > > > > > >>> > > > >> > > > > attaching below.
> > > > > > > > >>> > > > >> > > > >
> > > > > > > > >>> > > > >> > > > >
> > > > > > > > >>> > > > >> > > > > AND BINARY h.fcst_var = 'PMTF'
> > > > > > > > >>> > > > >> > > > > AND ld.stat_header_id =
> > > > > > > > >>> > > > >> > > > > h.stat_header_id;
> > > > > > > > >>> > > > >> > > > >
> > > > > > > > >>> > > > >> > > > > Connection to 205.156.8.85 closed
by
> > > > > > > > >>> > > > >> > > > > remote
> > > > > host.
> > > > > > > > >>> > > > >> > > > > CALLING: scp
> > > > > > > > >>> > > > >> > > > > edward.strobach at 205.156.8.85
> > > > > > > > >>> > > > >> > > > >
://data/mv_data//edward.strobach/plots/*
> > > > > > > > >>> > > > >> > > > >
> > > > > > > > >>> > > > >> > > > >
> > > > > > > > >>> > > > >> > > >
> > > > > > > > >>> > > > >> > >
> > > > > > > > >>> > > > >> >
> > > > > > > > >>> > > > >>
> > > > > > > > >>> > >
> > > > > > > > >>>
> > > > > > >
> > > > > >
> > > > >
> > > >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > > > > > > >>> > > > >> > > > >
> > > > > > > > >>> > > > >> > > > > You are accessing a U.S.
Government
> > > > information
> > > > > > > > >>> > > > >> > > > > system, which includes: 1) this
> > > > > > > > >>> > > > >> > > > > computer, 2)
> > > > > this
> > > > > > > > >>> > > > >> > > > > computer network, 3) all
computers
> > > > > > > > >>> > > > >> > > > > connected
> > > > to
> > > > > > > > >>> > > > >> > > > > this
> > > > > > > > >>> > > > >> > > > > network, and 4) all devices and
> > > > > > > > >>> > > > >> > > > > storage
> > > > > > > > >>> > > > >> > > > > media
> > > > > > > > >>> > > > >> > > > > attached
> > > > > > > > >>> > > > >> > > > > to this network or to a computer
on
> > > > > > > > >>> > > > >> > > > > this
> > > > > network.
> > > > > > > > >>> > > > >> > > > > You
> > > > > > > > >>> > > > >> > > > > understand and consent to the
> > > > > > > > >>> > > > >> > > > > following:
> > > > > > > > >>> > > > >> > > > > you
> > > > may
> > > > > > > > >>> > > > >> > > > > access this information system
for
> > > > > > > > >>> > > > >> > > > > authorized
> > > > > use
> > > > > > > > >>> > > > >> > > > > only;
> > > > > > > > >>> > > > >> > > > > you have no reasonable
expectation of
> > > > > > > > >>> > > > >> > > > > privacy
> > > > > > > > >>> > > > >> > > > > regarding
> > > > > > > > >>> > > > >> > > > > any communication of data
transiting
> > > > > > > > >>> > > > >> > > > > or
> > > > > > > > >>> > > > >> > > > > stored
> > > > > on
> > > > > > > > >>> > > > >> > > > > this
> > > > > > > > >>> > > > >> > > > > information system; at any time
and
> > > > > > > > >>> > > > >> > > > > for
> > > > > > > > >>> > > > >> > > > > any
> > > > > lawful
> > > > > > > > >>> > > > >> > > > > Government purpose, the
Government may
> > > > monitor,
> > > > > > > > >>> > > > >> > > > > intercept, and search and seize
any
> > > > > communication
> > > > > > > > >>> > > > >> > > > > or
> > > > > > > > >>> > > > >> > > > > data transiting or stored on this
> > > > > > > > >>> > > > >> > > > > information
> > > > > > > > >>> > > > >> > > > > system;
> > > > > > > > >>> > > > >> > > > > and
> > > > > > > > >>> > > > >> > > > > any communications or data
transiting
> > > > > > > > >>> > > > >> > > > > or
> > > > stored
> > > > > on
> > > > > > > > >>> > > > >> > > > > this
> > > > > > > > >>> > > > >> > > > > information system may be
disclosed or
> > > > > > > > >>> > > > >> > > > > used
> > > > for
> > > > > > > > >>> > > > >> > > > > any
> > > > > > > > >>> > > > >> > > > > lawful Government purpose.
> > > > > > > > >>> > > > >> > > > >
> > > > > > > > >>> > > > >> > > > > scp:
> > > > > > > > >>> > > > >> > > > >
//data/mv_data//edward.strobach/plots/*:
> > > > No
> > > > > > > > >>> > > > >> > > > > such
> > > > > > > > >>> > > > >> > > > > file
> > > > > > > > >>> > > > >> > > > > or
> > > > > > > > >>> > > > >> > directory
> > > > > > > > >>> > > > >> > > > > ERROR: Command returned with non-
zero
> > > > > > > > >>> > > > >> > > > > status
> > > > > (1):
> > > > > > > > >>> > > > >> > > > > scp
> > > > > > > > >>> > > > >> > edward.strobach@
> > > > > > > > >>> > > > >> > > > > 205.156.8.85
> > > > > > > ://data/mv_data//edward.strobach/plots/*
> > > > > > > > >>> > > > >> > > > >
> > > > > > > > >>> > > > >> > > > >
> > > > > > > > >>> > > > >> > > >
> > > > > > > > >>> > > > >> > >
> > > > > > > > >>> > > > >> >
> > > > > > > > >>> > > > >>
> > > > > > > > >>> > >
> > > > > > > > >>>
> > > > > > >
> > > > > >
> > > > >
> > > >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > > > > > > >>> > > > >> > > > > ++ printf '\033]0;%s@%s:%s\007'
> > > > Edward.Strobach
> > > > > > > > >>> > > > >> > > > > v71a1
> > > > > > > > >>> > > > >> > > > >
> > > > > > > /gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > > > > > > > >>> > > > >> > > > >
> > > > > > > > >>> > > > >> > > > >
> > > > > > > > >>> > > > >> > > > > NOTE: nothing about this set-up
has
> > > > > > > > >>> > > > >> > > > > changed
> > > > > other
> > > > > > > > >>> > > > >> > > > > than
> > > > > > > > >>> > > > >> > > > > the
> > > > > > > > >>> > > > >> directory
> > > > > > > > >>> > > > >> > > > path
> > > > > > > > >>> > > > >> > > > > onto dell. The path has been
created
> > > > > > > > >>> > > > >> > > > > so
> > > > > > > > >>> > > > >> > > > > it
> > > > > should
> > > > > > > > >>> > > > >> > > > > support
> > > > > > > > >>> > > > >> > > > > the
> > > > > > > > >>> > > > >> > transfer
> > > > > > > > >>> > > > >> > > > of
> > > > > > > > >>> > > > >> > > > > plots generated on the MET
server.
> > > > > > > > >>> > > > >> > > > > The
> > > > > preceding
> > > > > > > > >>> > > > >> > > > > information
> > > > > > > > >>> > > > >> > indicates
> > > > > > > > >>> > > > >> > > > > that the plot was processed
> > > > > > > > >>> > > > >> > > > > successfully.
> > > > It's
> > > > > > > > >>> > > > >> > > > > not
> > > > > > > > >>> > > > >> > > > > clear
> > > > > > > > >>> > > > >> > > > > why I
> > > > > > > > >>> > > > >> can't
> > > > > > > > >>> > > > >> > > > > transfer my plots over.
> > > > > > > > >>> > > > >> > > > > --
> > > > > > > > >>> > > > >> > > > > Edward Strobach
> > > > > > > > >>> > > > >> > > > > EMC/NCEP/NWS/
> > > > > > > > >>> > > > >> > > > > IMSG Contractor
> > > > > > > > >>> > > > >> > > > > Cubicle#: 2029
> > > > > > > > >>> > > > >> > > > > 301-683-3717
> > > > > > > > >>> > > > >> > > > >
> > > > > > > > >>> > > > >> > > > >
> > > > > > > > >>> > > > >> > > >
> > > > > > > > >>> > > > >> > > > --
> > > > > > > > >>> > > > >> > > > Julie Prestopnik (she/her)
> > > > > > > > >>> > > > >> > > > Software Engineer
> > > > > > > > >>> > > > >> > > > National Center for Atmospheric
Research
> > > > > > > > >>> > > > >> > > > Research Applications Laboratory
> > > > > > > > >>> > > > >> > > > Email: jpresto at ucar.edu
> > > > > > > > >>> > > > >> > > >
> > > > > > > > >>> > > > >> > > > My working day may not be your
working
> > > > > > > > >>> > > > >> > > > day.
> > > > > Please
> > > > > > > > >>> > > > >> > > > do not
> > > > > > > > >>> > > > >> > > > feel
> > > > > > > > >>> > > > >> obliged
> > > > > > > > >>> > > > >> > > to
> > > > > > > > >>> > > > >> > > > reply to this email outside of your
> > > > > > > > >>> > > > >> > > > normal
> > > > working
> > > > > > > > >>> > > > >> > > > hours.
> > > > > > > > >>> > > > >> > > >
> > > > > > > > >>> > > > >> > > >
> > > > > > > > >>> > > > >> > >
> > > > > > > > >>> > > > >> > > --
> > > > > > > > >>> > > > >> > > Edward Strobach
> > > > > > > > >>> > > > >> > > EMC/NCEP/NWS/
> > > > > > > > >>> > > > >> > > IMSG Contractor
> > > > > > > > >>> > > > >> > > Cubicle#: 2029
> > > > > > > > >>> > > > >> > > 301-683-3717
> > > > > > > > >>> > > > >> > >
> > > > > > > > >>> > > > >> > >
> > > > > > > > >>> > > > >> >
> > > > > > > > >>> > > > >> > --
> > > > > > > > >>> > > > >> > Julie Prestopnik (she/her)
> > > > > > > > >>> > > > >> > Software Engineer
> > > > > > > > >>> > > > >> > National Center for Atmospheric
Research
> > > > > > > > >>> > > > >> > Research Applications Laboratory
> > > > > > > > >>> > > > >> > Email: jpresto at ucar.edu
> > > > > > > > >>> > > > >> >
> > > > > > > > >>> > > > >> > My working day may not be your working
day.
> > > > > > > > >>> > > > >> > Please
> > > > do
> > > > > > > > >>> > > > >> > not
> > > > > > > > >>> > > > >> > feel
> > > > > > > > >>> > > > >> > obliged
> > > > > > > > >>> > > > >> to
> > > > > > > > >>> > > > >> > reply to this email outside of your
normal
> > > > > > > > >>> > > > >> > working
> > > > > > > > >>> > > > >> > hours.
> > > > > > > > >>> > > > >> >
> > > > > > > > >>> > > > >> >
> > > > > > > > >>> > > > >>
> > > > > > > > >>> > > > >> --
> > > > > > > > >>> > > > >> Edward Strobach
> > > > > > > > >>> > > > >> EMC/NCEP/NWS/
> > > > > > > > >>> > > > >> IMSG Contractor
> > > > > > > > >>> > > > >> Cubicle#: 2029
> > > > > > > > >>> > > > >> 301-683-3717
> > > > > > > > >>> > > > >>
> > > > > > > > >>> > > > >>
> > > > > > > > >>> > > > >
> > > > > > > > >>> > > > > --
> > > > > > > > >>> > > > > Julie Prestopnik (she/her)
> > > > > > > > >>> > > > > Software Engineer
> > > > > > > > >>> > > > > National Center for Atmospheric Research
> > > > > > > > >>> > > > > Research Applications Laboratory
> > > > > > > > >>> > > > > Email: jpresto at ucar.edu
> > > > > > > > >>> > > > >
> > > > > > > > >>> > > > > My working day may not be your working
day.
> > > > > > > > >>> > > > > Please do
> > > > > not
> > > > > > > > >>> > > > > feel
> > > > > > > > >>> > > > > obliged to
> > > > > > > > >>> > > > > reply to this email outside of your normal
> > > > > > > > >>> > > > > working
> > > > hours.
> > > > > > > > >>> > > > >
> > > > > > > > >>> > >
> > > > > > > > >>> > >
> > > > > > > > >>> > >
> > > > > > > > >>> > >
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > > >> --
> > > > > > > > >> Edward Strobach
> > > > > > > > >> EMC/NCEP/NWS/
> > > > > > > > >> IMSG Contractor
> > > > > > > > >> Cubicle#: 2029
> > > > > > > > >> 301-683-3717
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Edward Strobach
> > > > > > > > > EMC/NCEP/NWS/
> > > > > > > > > IMSG Contractor
> > > > > > > > > Cubicle#: 2029
> > > > > > > > > 301-683-3717
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Edward Strobach
> > > > > > EMC/NCEP/NWS/
> > > > > > IMSG Contractor
> > > > > > Cubicle#: 2029
> > > > > > 301-683-3717
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Julie Prestopnik (she/her)
> > > > > Software Engineer
> > > > > National Center for Atmospheric Research
> > > > > Research Applications Laboratory
> > > > > Email: jpresto at ucar.edu
> > > > >
> > > > > My working day may not be your working day. Please do not
feel
> > > > > obliged
> > > > to
> > > > > reply to this email outside of your normal working hours.
> > > > >
> > > > >
> > > >
> > > > --
> > > > Edward Strobach
> > > > EMC/NCEP/NWS/
> > > > IMSG Contractor
> > > > Cubicle#: 2029
> > > > 301-683-3717
> > > >
> > > >
>
>
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
Subject: can't copy plots over
From: Tatiana Burek
Time: Thu Dec 03 09:08:55 2020
Since your database is quite large, not creating the indexes would
help to speed up the loading. You also need to make sure that only one
loading process is running at a time. Multiple loading could create
invalid entities and would stall the inserts.
Tatiana
On Thu Dec 03 08:28:52 2020, edward.strobach at noaa.gov wrote:
> If you mean this suggestions:
>
> <drop_indexes>false</drop_indexes>
> <apply_indexes>false</apply_indexes>
>
> then yes. However, until just now I didn't realize that my
> load_to_metviewer_AWS.py set the template to have these options as
> true. I
> have since changed these options to false in the script. There are
> some
> issues in my set-up with the Taylor Diagrams and CSI that I need to
> correct
> for. Will the above option permit me to conduct frequent tests if
> needed?
>
> On Thu, Dec 3, 2020 at 10:04 AM Tatiana Burek via RT
> <met_help at ucar.edu>
> wrote:
>
> > Edward
> > AWS has 3 loading jobs that wore started by you:
> > 2 for load_mv_cmaq_metplus_CMAQ.xml and 2 for
> > load_mv_cmaq_metplus_CMAQ_Meteorology.xml
> > and 25 batch jobs.
> > None of them are executing because of the database locks created
by
> > so
> > many incerts, create indexes and select statments.
> > Shoud I kill all your jbs aagain? Have you had a chance to look at
my
> > sugesstuion on how to improve the process?
> >
> > Tatiana
> >
> > On Wed Dec 02 10:10:40 2020, tatiana wrote:
> > > Edward
> > > I am sorry, I did not mentioned that the commands I sent you
should
> > > be
> > > ran on AWS server and not on WCOSS
> > >
> > > Tatiana
> > >
> > > On Wed Dec 02 09:42:20 2020, jpresto wrote:
> > > > Hi Edward.
> > > >
> > > > You should be able to distinguish what processes are MET and
what
> > > > are
> > > > not.
> > > > From your "metcheck" results, it seems you have no MET
processes
> > > > running.
> > > > I'll explain below.
> > > >
> > > > [Edward.Strobach at v71a2 ~]$ metcheck
> > > > > root 76468 12071 0 16:25 ? 00:00:00 sshd:
> > > > > Edward.Strobach
> > > > > [priv]
> > > > > Edward.+ 76482 76468 0 16:25 ? 00:00:00 sshd:
> > > > > Edward.Strobach at pts
> > > > > /19
> > > > > Edward.+ 85480 82420 0 16:26 pts/19 00:00:00 grep
> > > > > --color=auto
> > > > > Edward.Strobach
> > > > > root 141334 12071 0 12:27 ? 00:00:00 sshd:
> > > > > Edward.Strobach
> > > > > [priv]
> > > > > Edward.+ 141337 141334 0 12:27 ? 00:00:00 sshd:
> > > > > Edward.Strobach at pts
> > > > > /5
> > > >
> > > >
> > > > All of the processes that start with "sshd" are your logins,
so
> > > > only
> > > > if you
> > > > ran "kill -9" on one of those would you get logged out. The
> > > > "grep"
> > > > processes (3rd result) is your running of "ps -ef | grep
> > > > Edward.Strobach".
> > > > If there were MET processes running you would be able to see
them
> > > > here.
> > > > The fact that there are none listed here could be because when
> > > > you
> > > > ran
> > > > "pkill -9 -u `id -u Edward.Strobach`", they were all killed,
or
> > > > because
> > > > you're on a different node (v71a2) than the one you ran them
on.
> > > >
> > > > Julie
> > > >
> > > > On Wed, Dec 2, 2020 at 9:28 AM Edward Strobach - NOAA
Affiliate
> > > > via
> > > > RT
> > > > <
> > > > met_help at ucar.edu> wrote:
> > > >
> > > > >
> > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
> > > > >
> > > > > Thanks. I was able to determine the processes using ps -ef
|
> > > > > grep
> > > > > Edward.Strobach
> > > > >
> > > > > I did try pkill -9 -u `id -u Edward.Strobach` before sending
> > > > > your
> > > > > email
> > > > > which did log me out unfortunately.
> > > > >
> > > > > I also noticed that kill -9 jobid also logs me out. I wish
> > > > > there
> > > > > was
> > > > > a way
> > > > > to distinguish between MET tasks v. other tasks. When I
look
> > > > > at
> > > > > what
> > > > > I
> > > > > currently have I can't tell what is MET and what is not MET.
> > > > > [Edward.Strobach at v71a2 ~]$ metcheck
> > > > > root 76468 12071 0 16:25 ? 00:00:00 sshd:
> > > > > Edward.Strobach
> > > > > [priv]
> > > > > Edward.+ 76482 76468 0 16:25 ? 00:00:00 sshd:
> > > > > Edward.Strobach at pts
> > > > > /19
> > > > > Edward.+ 85480 82420 0 16:26 pts/19 00:00:00 grep
> > > > > --color=auto
> > > > > Edward.Strobach
> > > > > root 141334 12071 0 12:27 ? 00:00:00 sshd:
> > > > > Edward.Strobach
> > > > > [priv]
> > > > > Edward.+ 141337 141334 0 12:27 ? 00:00:00 sshd:
> > > > > Edward.Strobach at pts
> > > > > /5
> > > > >
> > > > > Note that metcheck is just an alias for ps -ef | grep
> > > > > Edward.Strobach
> > > > >
> > > > > On Wed, Dec 2, 2020 at 11:13 AM Julie Prestopnik via RT
> > > > > <met_help at ucar.edu
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Edward.
> > > > > >
> > > > > > Because capitals matter when running "grep" and because
WCOSS
> > > > > > uses
> > > > > capital
> > > > > > letters for our username you can run either of the
following
> > > > > > instead:
> > > > > >
> > > > > > ps -ef | grep -i edward (the "-i" tells grep to ignore
case)
> > > > > > ps -ef | grep Edward
> > > > > >
> > > > > > The second column will list the process id and the third
> > > > > > column
> > > > > > will list
> > > > > > the parent process id.
> > > > > >
> > > > > > The same issue exists with running pkill on WCOSS. You
can
> > > > > > run
> > > > > > pkill on
> > > > > > WCOSS using the following, however, I don't recommend it
and
> > > > > > I'll
> > > > > > explain
> > > > > > below:
> > > > > >
> > > > > > pkill -9 -u `id -u Edward.Strobach`
> > > > > >
> > > > > > First, you'd need to make sure you're logged in
appropriately
> > > > > > and
> > > > > > be on
> > > > > the
> > > > > > same node (not sure if that is the correct terminology) as
> > > > > > the
> > > > > > one
> > > > > running
> > > > > > your processes. For example, I ran a simple check to test
> > > > > > this.
> > > > > > I
> > > > > > ran
> > > > > > "top" when I was logged into "Julie.Prestopnik@*v71a3*",
but
> > > > > > my
> > > > > > second
> > > > > log
> > > > > > in was on "Julie.Prestopnik@*v72a1*". v72a1 did not
> > > > > > recognize
> > > > > > "top" as
> > > > > > running, which I could see when running "ps -ef | grep
> > > > > > Julie".
> > > > > > When I
> > > > > ran
> > > > > > "ssh v71a3" and then reran "ps -ef | grep Julie", I could
> > > > > > then
> > > > > > see
> > > > > > "top"
> > > > > > was running. Running "pkill -9 -u `id -u
Julie.Presotpnik`"
> > > > > > also
> > > > > > closed
> > > > > my
> > > > > > connection to WCOSS, so if you don't want your connection
to
> > > > > > WCOSS
> > > > > killed,
> > > > > > which I didn't, I would just kill the processes
individually
> > > > > > using
> > > > > > the
> > > > > kill
> > > > > > command with the process number.
> > > > > >
> > > > > > I hope that helps!
> > > > > >
> > > > > > Julie
> > > > > >
> > > > > > On Wed, Dec 2, 2020 at 8:51 AM Edward Strobach - NOAA
> > > > > > Affiliate
> > > > > > via
> > > > > > RT <
> > > > > > met_help at ucar.edu> wrote:
> > > > > >
> > > > > > >
> > > > > > > <URL:
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
>
> > > > > > >
> > > > > > > I see this when running ps -ef | grep edward
> > > > > > > Edward.+ 115762 154642 0 15:48 pts/31 00:00:00 grep
> > > > > > > --color=auto
> > > > > > edward
> > > > > > >
> > > > > > > The numbers there change every time I run this step.
> > > > > > >
> > > > > > > pkill -9 -u `id -u edward.strobach` does not seem to
work.
> > > > > > >
> > > > > > > On Wed, Dec 2, 2020 at 9:34 AM Tatiana Burek via RT
> > > > > > > <met_help at ucar.edu
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > You can kill your processes by :
> > > > > > > > - collecting what is running:
> > > > > > > > ps -ef | grep edward
> > > > > > > > - killing each process
> > > > > > > > kill -9 2574
> > > > > > > > kill -9 2594
> > > > > > > > or all your processes:
> > > > > > > > pkill -9 -u `id -u edward.strobach`
> > > > > > > >
> > > > > > > > Right now you have 4 loading scripts and 10 batch
scripts
> > > > > > > > running at
> > > > > > the
> > > > > > > > same time which creates a gridlock and none of the
> > > > > > > > processes
> > > > > > > > are
> > > > > > getting
> > > > > > > > executes.
> > > > > > > >
> > > > > > > > Tatiana
> > > > > > > >
> > > > > > > > On Wed Dec 02 06:32:23 2020, edward.strobach at noaa.gov
> > > > > > > > wrote:
> > > > > > > > > I came across this issue again this morning. Is
there
> > > > > > > > > a
> > > > > > > > > way
> > > > > > > > > to
> > > > > stop
> > > > > > > > > certain processes and start fresh from my end?
> > > > > > > > >
> > > > > > > > > On Tue, Dec 1, 2020 at 5:14 PM Edward Strobach -
NOAA
> > > > > > > > > Affiliate <
> > > > > > > > > edward.strobach at noaa.gov> wrote:
> > > > > > > > >
> > > > > > > > > > maybe there is a script that allows me to check
what
> > > > > > > > > > is
> > > > > > > > > > going on
> > > > > > with
> > > > > > > > > > my
> > > > > > > > > > database? A lot of these issues I've come across
> > > > > > > > > > happen
> > > > > > > > > > because
> > > > > of
> > > > > > > > > > things
> > > > > > > > > > I'm unaware of or things that go on simultaneously
> > > > > > > > > > that
> > > > > > > > > > causes
> > > > > > > > > > confusion in
> > > > > > > > > > the processing. Right now I have a plotting
script
> > > > > > > > > > set
> > > > > > > > > > that
> > > > > > > > > > simultaneously
> > > > > > > > > > submits batch jobs for time series statistics,
Taylor
> > > > > > > > > > diagrams,
> > > > > and
> > > > > > > > > > CSI. I
> > > > > > > > > > haven't seen too many issues when I do that, but
> > > > > > > > > > would it
> > > > > > > > > > make
> > > > > > sense
> > > > > > > > > > to
> > > > > > > > > > insert a while loop after the first submission
that
> > > > > > > > > > uses
> > > > > > > > > > a
> > > > > > > > > > script
> > > > > > to
> > > > > > > > > > check
> > > > > > > > > > the activity within my database? In other words,
I
> > > > > > > > > > would
> > > > > > > > > > use a
> > > > > > > > > > script,
> > > > > > > > > > perhaps one that has already been developed, to
check
> > > > > > > > > > the
> > > > > > > > > > status
> > > > > of
> > > > > > > > > > processing before submitting the next batch job.
> > > > > > > > > > Once
> > > > > > > > > > the
> > > > > > processing
> > > > > > > > > > for
> > > > > > > > > > the first plotting steps have completed, then I
would
> > > > > > > > > > commence to
> > > > > > the
> > > > > > > > > > next
> > > > > > > > > > one.
> > > > > > > > > >
> > > > > > > > > > I guess check_my_db.sh would not do that?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Dec 1, 2020 at 4:27 PM Edward Strobach -
NOAA
> > > > > > > > > > Affiliate <
> > > > > > > > > > edward.strobach at noaa.gov> wrote:
> > > > > > > > > >
> > > > > > > > > >> are there youtube examples or a section in a
> > > > > > > > > >> technical
> > > > > > > > > >> manual
> > > > > that
> > > > > > > > > >> goes
> > > > > > > > > >> over examples illustrating best practices related
to
> > > > > > > > > >> this
> > > > > process?
> > > > > > > > > >>
> > > > > > > > > >> On Tue, Dec 1, 2020 at 4:03 PM Tatiana Burek via
RT
> > > > > > > > > >> <met_help at ucar.edu>
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >>> For your process you need only two scripts:
> > > > > > > > > >>> mv_load_to_aws.sh
> > > > > > > > > >>> mv_batch_on_aws.sh
> > > > > > > > > >>> I don't think, that others would help you.
> > > > > > > > > >>>
> > > > > > > > > >>> The main work is in creating efficient XMLs for
the
> > > > > > > > > >>> each
> > > > > > > > > >>> step.
> > > > > > > > > >>> Like not recalculating indexes or performing
> > > > > stat_header_db_check
> > > > > > > > > >>> during
> > > > > > > > > >>> data loading.
> > > > > > > > > >>>
> > > > > > > > > >>> Tatiana
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> On Tue Dec 01 13:00:15 2020,
> > > > > > > > > >>> edward.strobach at noaa.gov
> > > > > > > > > >>> wrote:
> > > > > > > > > >>> > Thanks. I just made those changes in the xml
> > > > > > > > > >>> > file,
> > > > > > > > > >>> > so
> > > > > > > > > >>> > I
> > > > > > 'guess'
> > > > > > > > > >>> > I'm
> > > > > > > > > >>> > ready
> > > > > > > > > >>> > to go.
> > > > > > > > > >>> >
> > > > > > > > > >>> > I should note that I have several steps that I
> > > > > > > > > >>> > run
> > > > > > > > > >>> > each
> > > > > day. I
> > > > > > > > > >>> > convert
> > > > > > > > > >>> > files into netcdf files that can be read by
MET,
> > > > > > > > > >>> > I
> > > > > > > > > >>> > generate
> > > > > the
> > > > > > > > > >>> > stat
> > > > > > > > > >>> > files,
> > > > > > > > > >>> > and I use the stat files in the MET batch
> > > > > > > > > >>> > plotting
> > > > > > > > > >>> > system. I
> > > > > > > > > >>> > have
> > > > > > > > > >>> > actually
> > > > > > > > > >>> > 5 total steps, but the plot batch system is
> > > > > > > > > >>> > really
> > > > > > > > > >>> > the
> > > > > > > > > >>> > only
> > > > > one
> > > > > > > > > >>> > that
> > > > > > > > > >>> > gives
> > > > > > > > > >>> > me trouble on a regular basis. I wish I knew
how
> > > > > > > > > >>> > to
> > > > > > > > > >>> > reach a
> > > > > > > > > >>> > point
> > > > > > > > > >>> > where I
> > > > > > > > > >>> > have a plotting package that would produce the
> > > > > > > > > >>> > plots
> > > > > > > > > >>> > that I
> > > > > > need
> > > > > > > > > >>> > each
> > > > > > > > > >>> > day.
> > > > > > > > > >>> > I'm expected to hand the final product off in
the
> > > > > > > > > >>> > near
> > > > > future,
> > > > > > > > > >>> > but
> > > > > > > > > >>> > this
> > > > > > > > > >>> > part has delayed me extensively. I know that
I'm
> > > > > > > > > >>> > generating a
> > > > > > lot
> > > > > > > > > >>> > of
> > > > > > > > > >>> > plots
> > > > > > > > > >>> > which could potentially overwhelm the system,
but
> > > > > > > > > >>> > I'm
> > > > > > > > > >>> > unaware
> > > > > > how
> > > > > > > > > >>> > others
> > > > > > > > > >>> > work to circumvent that. I hope to move past
> > > > > > > > > >>> > this
> > > > > > > > > >>> > soon
> > > > > > > > > >>> > so I
> > > > > > can
> > > > > > > > > >>> > continue
> > > > > > > > > >>> > development work.
> > > > > > > > > >>> >
> > > > > > > > > >>> > There are several scripts that I don't use
that
> > > > > > > > > >>> > might
> > > > > > > > > >>> > be
> > > > > > helpful.
> > > > > > > > > >>> > Do
> > > > > > > > > >>> > any
> > > > > > > > > >>> > of these work to help me manage the system
> > > > > > > > > >>> > better?
> > > > > > > > > >>> > check_my_db.sh mv_batch_on_aws.sh
> > > > > > > > > >>> > mv_create_db_on_aws.sh
> > > > > > > > > >>> > mv_db_size_on_aws.sh mv_delete_db_on_aws.sh
> > > > > > mv_load_to_aws.sh
> > > > > > > > > >>> >
> > > > > > > > > >>> >
> > > > > > > > > >>> > On Tue, Dec 1, 2020 at 2:44 PM Tatiana Burek
via
> > > > > > > > > >>> > RT
> > > > > > > > > >>> > <met_help at ucar.edu>
> > > > > > > > > >>> > wrote:
> > > > > > > > > >>> >
> > > > > > > > > >>> > > Edward
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > I run a few butch jobs from my laptop and
made
> > > > > > > > > >>> > > sure
> > > > > > > > > >>> > > that
> > > > > > > > > >>> > > METviewer
> > > > > > > > > >>> > > code
> > > > > > > > > >>> > > and scripts working as expected: I was able
to
> > > > > > > > > >>> > > produce and
> > > > > > copy
> > > > > > > > > >>> > > plots.
> > > > > > > > > >>> > > I also got an error like the one that you
> > > > > > > > > >>> > > reported
> > > > > > > > > >>> > > but only
> > > > > > > > > >>> > > when
> > > > > > > > > >>> > > METviewer
> > > > > > > > > >>> > > did not generate any plots and the script
> > > > > > > > > >>> > > couldn't
> > > > > > > > > >>> > > copy
> > > > > empty
> > > > > > > > > >>> > > directory.
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > During my testing I noticed a few batch
jobs
> > > > > > > > > >>> > > that
> > > > > > > > > >>> > > where
> > > > > > > > > >>> > > started by
> > > > > > > > > >>> > > you
> > > > > > > > > >>> > > and were handing. Each of these jobs started
a
> > > > > > > > > >>> > > database
> > > > > query
> > > > > > > > > >>> > > and
> > > > > > > > > >>> > > they also
> > > > > > > > > >>> > > were piling up and handing. They all were
> > > > > > > > > >>> > > waiting
> > > > > > > > > >>> > > for
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > indexing
> > > > > > > > > >>> > > that
> > > > > > > > > >>> > > takes place after the data loading.
> > > > > > > > > >>> > > I had to kill all database and batch
processes
> > > > > > > > > >>> > > in
> > > > > > > > > >>> > > order to
> > > > > > move
> > > > > > > > > >>> > > forward
> > > > > > > > > >>> > > with the testing.
> > > > > > > > > >>> > > To avoid this gridlock you could:
> > > > > > > > > >>> > > - in your loading xml use
> > > > > <drop_indexes>false</drop_indexes>
> > > > > > > > > >>> > > <apply_indexes>false</apply_indexes>
> > > > > > > > > >>> > >
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > - do not start a few parallel batch jobs
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > I will add a info message to the script that
> > > > > > > > > >>> > > would
> > > > > > > > > >>> > > explain
> > > > > > why
> > > > > > > > > >>> > > the
> > > > > > > > > >>> > > error
> > > > > > > > > >>> > > occurs.
> > > > > > > > > >>> > > For now, could you test batch with some
simple
> > > > > > > > > >>> > > XML
> > > > > > > > > >>> > > that
> > > > > > produce
> > > > > > > > > >>> > > at
> > > > > > > > > >>> > > least
> > > > > > > > > >>> > > one image.
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > Tatiana
> > > > > > > > > >>> > >
> > > > > > > > > >>> > >
> > > > > > > > > >>> > >
> > > > > > > > > >>> > >
> > > > > > > > > >>> > >
> > > > > > > > > >>> > >
> > > > > > > > > >>> > >
> > > > > > > > > >>> > > On Tue Dec 01 11:35:03 2020, jpresto wrote:
> > > > > > > > > >>> > > > Hi Edward.
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > > > After talking with Tatiana, it sounds like
I
> > > > > > > > > >>> > > > am
> > > > > misinformed
> > > > > > > > > >>> > > > and the
> > > > > > > > > >>> > > > scripts
> > > > > > > > > >>> > > > you are using were, in fact, developed by
our
> > > > > > > > > >>> > > > METviewer
> > > > > > team.
> > > > > > > > > >>> > > > My
> > > > > > > > > >>> > > > apologies. METplus has a lot of
components,
> > > > > > > > > >>> > > > and
> > > > > > > > > >>> > > > while I
> > > > > > > > > >>> > > > thought I
> > > > > > > > > >>> > > > had
> > > > > > > > > >>> > > > a
> > > > > > > > > >>> > > > good understanding of who did what, it
turns
> > > > > > > > > >>> > > > out,
> > > > > > > > > >>> > > > I
> > > > > > > > > >>> > > > did
> > > > > > not.
> > > > > > > > > >>> > > > Tatiana
> > > > > > > > > >>> > > > is
> > > > > > > > > >>> > > > working on this issue and will follow up
once
> > > > > > > > > >>> > > > she
> > > > > > > > > >>> > > > knows
> > > > > > more.
> > > > > > > > > >>> > > > Thank
> > > > > > > > > >>> > > > you
> > > > > > > > > >>> > > > for your patience and understanding.
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > > > Julie
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > > > On Tue, Dec 1, 2020 at 11:22 AM Julie
> > > > > > > > > >>> > > > Prestopnik
> > > > > > > > > >>> > > > <jpresto at ucar.edu
> > > > > > > > > >>> >
> > > > > > > > > >>> > > > wrote:
> > > > > > > > > >>> > > >
> > > > > > > > > >>> > > > > Thanks, Edward. I directed you to
Mallory
> > > > > > > > > >>> > > > > and
> > > > > > > > > >>> > > > > Perry
> > > > > > > > > >>> > > > > because I
> > > > > > > > > >>> > > > > thought you
> > > > > > > > > >>> > > > > might be using scripts from them. We
did
> > > > > > > > > >>> > > > > not
> > > > > > > > > >>> > > > > develop
> > > > > and
> > > > > > > > > >>> > > > > have
> > > > > > > > > >>> > > > > not
> > > > > > > > > >>> > > > > used
> > > > > > > > > >>> > > > > those scripts. We have no experience
> > > > > > > > > >>> > > > > interacting
> > > > > > > > > >>> > > > > with
> > > > > > AWS
> > > > > > > > > >>> > > > > and
> > > > > > > > > >>> > > > > WCOSS,
> > > > > > > > > >>> > > > > so
> > > > > > > > > >>> > > > > they seemed best suited to answer your
> > > > > > > > > >>> > > > > questions
> > > > > related
> > > > > > to
> > > > > > > > > >>> > > > > that
> > > > > > > > > >>> > > > > process.
> > > > > > > > > >>> > > > > However, as Tatiana pointed out, perhaps
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > security
> > > > > > > > > >>> > > > > changes on
> > > > > > > > > >>> > > > > the
> > > > > > > > > >>> > > > > AWS
> > > > > > > > > >>> > > > > server have something to do with what
you
> > > > > > > > > >>> > > > > are
> > > > > > experiencing.
> > > > > > > > > >>> > > > > If
> > > > > > > > > >>> > > > > that
> > > > > > > > > >>> > > > > turns
> > > > > > > > > >>> > > > > out not to be the case, Mallory and
Perry
> > > > > > > > > >>> > > > > would
> > > > > > > > > >>> > > > > be best
> > > > > > > > > >>> > > > > able to
> > > > > > > > > >>> > > > > help
> > > > > > > > > >>> > > > > you.
> > > > > > > > > >>> > > > > If they are unable to help, please let
us
> > > > > > > > > >>> > > > > know,
> > > > > > > > > >>> > > > > and we
> > > > > > can
> > > > > > > > > >>> > > > > see
> > > > > > > > > >>> > > > > what
> > > > > > > > > >>> > > > > they
> > > > > > > > > >>> > > > > tried and how we can help you get this
> > > > > > > > > >>> > > > > problem
> > > > > > > > > >>> > > > > figured
> > > > > > out.
> > > > > > > > > >>> > > > >
> > > > > > > > > >>> > > > > Julie
> > > > > > > > > >>> > > > >
> > > > > > > > > >>> > > > > On Tue, Dec 1, 2020 at 11:15 AM Edward
> > > > > > > > > >>> > > > > Strobach
> > > > > > > > > >>> > > > > -
> > > > > > > > > >>> > > > > NOAA
> > > > > > > > > >>> > > > > Affiliate
> > > > > > > > > >>> > > > > via
> > > > > > > > > >>> > > > > RT <
> > > > > > > > > >>> > > > > met_help at ucar.edu> wrote:
> > > > > > > > > >>> > > > >
> > > > > > > > > >>> > > > >>
> > > > > > > > > >>> > > > >> <URL:
> > > > > > > > > >>> > > > >>
> > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > > > > >
> > > > > > > > > >>> > > > >>
> > > > > > > > > >>> > > > >> Hi Julie,
> > > > > > > > > >>> > > > >>
> > > > > > > > > >>> > > > >> I'm actually using the scripts located
> > > > > > > > > >>> > > > >> here:
> > > > > > > > > >>> > > > >>
> > > > > > > > > >>>
> > > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts.
> > > > > > > > > >>> > > > >> These
> > > > > > > > > >>> > > > >> were scripts that I either got from
> > > > > > > > > >>> > > > >> Mallory or
> > > > > > > > > >>> > > > >> Perry a
> > > > > > > > > >>> > > > >> while
> > > > > > > > > >>> > > > >> back -
> > > > > > > > > >>> > > > >> around
> > > > > > > > > >>> > > > >> May.
> > > > > > > > > >>> > > > >>
> > > > > > > > > >>> > > > >> On Tue, Dec 1, 2020 at 12:59 PM Julie
> > > > > > > > > >>> > > > >> Prestopnik
> > > > > > > > > >>> > > > >> via
> > > > > RT
> > > > > > <
> > > > > > > > > >>> > > > >> met_help at ucar.edu>
> > > > > > > > > >>> > > > >> wrote:
> > > > > > > > > >>> > > > >>
> > > > > > > > > >>> > > > >> > Hi Edward.
> > > > > > > > > >>> > > > >> >
> > > > > > > > > >>> > > > >> > Are you using the scripts that are
> > > > > > > > > >>> > > > >> > located
> > > > > > > > > >>> > > > >> > in
> > > > > > > > > >>> > > > >> >
> > > > > > > >
/gpfs/hps3/emc/global/noscrub/Mallory.Row/VRFY/METviewer_AWS
> > > > > > > > > >>> > > > >> > and
> > > > > > > > > >>> > > > >> >
> > > > > > > >
/gpfs/dell2/emc/verification/noscrub/emc.metplus/METviewer_AWS
> > > > > > > > > >>> > > > >> > for interaction between AWS and
WCOSS?
> > > > > > > > > >>> > > > >> >
> > > > > > > > > >>> > > > >> > Thanks!
> > > > > > > > > >>> > > > >> >
> > > > > > > > > >>> > > > >> > Julie
> > > > > > > > > >>> > > > >> >
> > > > > > > > > >>> > > > >> > On Tue, Dec 1, 2020 at 10:49 AM
Edward
> > > > > > > > > >>> > > > >> > Strobach -
> > > > > NOAA
> > > > > > > > > >>> > > > >> > Affiliate
> > > > > > > > > >>> > > > >> > via RT
> > > > > > > > > >>> > > > >> <
> > > > > > > > > >>> > > > >> > met_help at ucar.edu> wrote:
> > > > > > > > > >>> > > > >> >
> > > > > > > > > >>> > > > >> > >
> > > > > > > > > >>> > > > >> > > <URL:
> > > > > > > > > >>> > > > >> > >
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > > > > > > > >>> > > > >> > > >
> > > > > > > > > >>> > > > >> > >
> > > > > > > > > >>> > > > >> > > Perry does not do batch plotting
with
> > > > > > > > > >>> > > > >> > > MET.
> > > > > > > > > >>> > > > >> > > I was
> > > > > > > > > >>> > > > >> > > under the
> > > > > > > > > >>> > > > >> > > impression
> > > > > > > > > >>> > > > >> > that
> > > > > > > > > >>> > > > >> > > the MET helpdesk helped with MET
> > > > > > > > > >>> > > > >> > > related
> > > > > > > > > >>> > > > >> > > issues.
> > > > > > > > > >>> > > > >> > >
> > > > > > > > > >>> > > > >> > > On Tue, Dec 1, 2020 at 12:13 PM
Julie
> > > > > > > > > >>> > > > >> > > Prestopnik
> > > > > via
> > > > > > > > > >>> > > > >> > > RT <
> > > > > > > > > >>> > > > >> > met_help at ucar.edu
> > > > > > > > > >>> > > > >> > > >
> > > > > > > > > >>> > > > >> > > wrote:
> > > > > > > > > >>> > > > >> > >
> > > > > > > > > >>> > > > >> > > > Hi Edward.
> > > > > > > > > >>> > > > >> > > >
> > > > > > > > > >>> > > > >> > > > Please consult with Mallory Row
and
> > > > > > > > > >>> > > > >> > > > Perry
> > > > > Shafran
> > > > > > > > > >>> > > > >> > > > for
> > > > > > > > > >>> > > > >> > > > assistance
> > > > > > > > > >>> > > > >> with
> > > > > > > > > >>> > > > >> > > this
> > > > > > > > > >>> > > > >> > > > issue. They are most familiar
with
> > > > > > > > > >>> > > > >> > > > these
> > > > > > > > > >>> > > > >> > > > capabilities and
> > > > > > > > > >>> > > > >> > > > will be
> > > > > > > > > >>> > > > >> best
> > > > > > > > > >>> > > > >> > > > able to assist you.
> > > > > > > > > >>> > > > >> > > >
> > > > > > > > > >>> > > > >> > > > Thanks!
> > > > > > > > > >>> > > > >> > > >
> > > > > > > > > >>> > > > >> > > > Julie
> > > > > > > > > >>> > > > >> > > >
> > > > > > > > > >>> > > > >> > > > On Tue, Dec 1, 2020 at 9:25 AM
> > > > > > > > > >>> > > > >> > > > Edward
> > > > > > > > > >>> > > > >> > > > Strobach -
> > > > > > > > > >>> > > > >> > > > NOAA
> > > > > > > > > >>> > > > >> > > > Affiliate via
> > > > > > > > > >>> > > > >> RT
> > > > > > > > > >>> > > > >> > <
> > > > > > > > > >>> > > > >> > > > met_help at ucar.edu> wrote:
> > > > > > > > > >>> > > > >> > > >
> > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > >>> > > > >> > > > > Tue Dec 01 09:25:26 2020:
Request
> > > > > > > > > >>> > > > >> > > > > 97687
> > > > > > > > > >>> > > > >> > > > > was
> > > > > > acted
> > > > > > > > > >>> > > > >> > > > > upon.
> > > > > > > > > >>> > > > >> > > > > Transaction: Ticket created by
> > > > > > > > > >>> > > > >> > > > > edward.strobach at noaa.gov
> > > > > > > > > >>> > > > >> > > > > Queue: met_help
> > > > > > > > > >>> > > > >> > > > > Subject: can't copy plots
> > > > > > > > > >>> > > > >> > > > > over
> > > > > > > > > >>> > > > >> > > > > Owner: Nobody
> > > > > > > > > >>> > > > >> > > > > Requestors:
> > > > > > > > > >>> > > > >> > > > > edward.strobach at noaa.gov
> > > > > > > > > >>> > > > >> > > > > Status: new
> > > > > > > > > >>> > > > >> > > > > Ticket <URL:
> > > > > > > > > >>> > > > >> >
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > > > > > > > >>> > > > >> > > >
> > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > >>> > > > >> > > > > I'm trying to copy plots over
to
> > > > > > > > > >>> > > > >> > > > > Dell
> > > > > > > > > >>> > > > >> > > > > from the
> > > > > > MET
> > > > > > > > > >>> > > > >> > > > > server.
> > > > > > > > > >>> > > > >> > > > > I
> > > > > > > > > >>> > > > >> would
> > > > > > > > > >>> > > > >> > > like
> > > > > > > > > >>> > > > >> > > > to
> > > > > > > > > >>> > > > >> > > > > store my plots in the usual
place
> > > > > > > > > >>> > > > >> > > > > but
> > > > > > > > > >>> > > > >> > > > > there is
> > > > > > an
> > > > > > > > > >>> > > > >> > > > > scp
> > > > > > > > > >>> > > > >> > > > > error
> > > > > > > > > >>> > > > >> > > > > which
> > > > > > > > > >>> > > > >> I'm
> > > > > > > > > >>> > > > >> > > > > attaching below.
> > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > >>> > > > >> > > > > AND BINARY h.fcst_var =
'PMTF'
> > > > > > > > > >>> > > > >> > > > > AND ld.stat_header_id =
> > > > > > > > > >>> > > > >> > > > > h.stat_header_id;
> > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > >>> > > > >> > > > > Connection to 205.156.8.85
closed
> > > > > > > > > >>> > > > >> > > > > by
> > > > > > > > > >>> > > > >> > > > > remote
> > > > > > host.
> > > > > > > > > >>> > > > >> > > > > CALLING: scp
> > > > > > > > > >>> > > > >> > > > > edward.strobach at 205.156.8.85
> > > > > > > > > >>> > > > >> > > > >
://data/mv_data//edward.strobach/plots/*
> > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > >>> > > > >> > > >
> > > > > > > > > >>> > > > >> > >
> > > > > > > > > >>> > > > >> >
> > > > > > > > > >>> > > > >>
> > > > > > > > > >>> > >
> > > > > > > > > >>>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> >
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > >>> > > > >> > > > > You are accessing a U.S.
> > > > > > > > > >>> > > > >> > > > > Government
> > > > > information
> > > > > > > > > >>> > > > >> > > > > system, which includes: 1) this
> > > > > > > > > >>> > > > >> > > > > computer, 2)
> > > > > > this
> > > > > > > > > >>> > > > >> > > > > computer network, 3) all
computers
> > > > > > > > > >>> > > > >> > > > > connected
> > > > > to
> > > > > > > > > >>> > > > >> > > > > this
> > > > > > > > > >>> > > > >> > > > > network, and 4) all devices and
> > > > > > > > > >>> > > > >> > > > > storage
> > > > > > > > > >>> > > > >> > > > > media
> > > > > > > > > >>> > > > >> > > > > attached
> > > > > > > > > >>> > > > >> > > > > to this network or to a
computer
> > > > > > > > > >>> > > > >> > > > > on
> > > > > > > > > >>> > > > >> > > > > this
> > > > > > network.
> > > > > > > > > >>> > > > >> > > > > You
> > > > > > > > > >>> > > > >> > > > > understand and consent to the
> > > > > > > > > >>> > > > >> > > > > following:
> > > > > > > > > >>> > > > >> > > > > you
> > > > > may
> > > > > > > > > >>> > > > >> > > > > access this information system
for
> > > > > > > > > >>> > > > >> > > > > authorized
> > > > > > use
> > > > > > > > > >>> > > > >> > > > > only;
> > > > > > > > > >>> > > > >> > > > > you have no reasonable
expectation
> > > > > > > > > >>> > > > >> > > > > of
> > > > > > > > > >>> > > > >> > > > > privacy
> > > > > > > > > >>> > > > >> > > > > regarding
> > > > > > > > > >>> > > > >> > > > > any communication of data
> > > > > > > > > >>> > > > >> > > > > transiting
> > > > > > > > > >>> > > > >> > > > > or
> > > > > > > > > >>> > > > >> > > > > stored
> > > > > > on
> > > > > > > > > >>> > > > >> > > > > this
> > > > > > > > > >>> > > > >> > > > > information system; at any time
> > > > > > > > > >>> > > > >> > > > > and
> > > > > > > > > >>> > > > >> > > > > for
> > > > > > > > > >>> > > > >> > > > > any
> > > > > > lawful
> > > > > > > > > >>> > > > >> > > > > Government purpose, the
Government
> > > > > > > > > >>> > > > >> > > > > may
> > > > > monitor,
> > > > > > > > > >>> > > > >> > > > > intercept, and search and seize
> > > > > > > > > >>> > > > >> > > > > any
> > > > > > communication
> > > > > > > > > >>> > > > >> > > > > or
> > > > > > > > > >>> > > > >> > > > > data transiting or stored on
this
> > > > > > > > > >>> > > > >> > > > > information
> > > > > > > > > >>> > > > >> > > > > system;
> > > > > > > > > >>> > > > >> > > > > and
> > > > > > > > > >>> > > > >> > > > > any communications or data
> > > > > > > > > >>> > > > >> > > > > transiting
> > > > > > > > > >>> > > > >> > > > > or
> > > > > stored
> > > > > > on
> > > > > > > > > >>> > > > >> > > > > this
> > > > > > > > > >>> > > > >> > > > > information system may be
> > > > > > > > > >>> > > > >> > > > > disclosed or
> > > > > > > > > >>> > > > >> > > > > used
> > > > > for
> > > > > > > > > >>> > > > >> > > > > any
> > > > > > > > > >>> > > > >> > > > > lawful Government purpose.
> > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > >>> > > > >> > > > > scp:
> > > > > > > > > >>> > > > >> > > > >
//data/mv_data//edward.strobach/plots/*:
> > > > > No
> > > > > > > > > >>> > > > >> > > > > such
> > > > > > > > > >>> > > > >> > > > > file
> > > > > > > > > >>> > > > >> > > > > or
> > > > > > > > > >>> > > > >> > directory
> > > > > > > > > >>> > > > >> > > > > ERROR: Command returned with
non-
> > > > > > > > > >>> > > > >> > > > > zero
> > > > > > > > > >>> > > > >> > > > > status
> > > > > > (1):
> > > > > > > > > >>> > > > >> > > > > scp
> > > > > > > > > >>> > > > >> > edward.strobach@
> > > > > > > > > >>> > > > >> > > > > 205.156.8.85
> > > > > > > > ://data/mv_data//edward.strobach/plots/*
> > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > >>> > > > >> > > >
> > > > > > > > > >>> > > > >> > >
> > > > > > > > > >>> > > > >> >
> > > > > > > > > >>> > > > >>
> > > > > > > > > >>> > >
> > > > > > > > > >>>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> >
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > > > > > > > >>> > > > >> > > > > ++ printf '\033]0;%s@%s:%s\007'
> > > > > Edward.Strobach
> > > > > > > > > >>> > > > >> > > > > v71a1
> > > > > > > > > >>> > > > >> > > > >
> > > > > > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > >>> > > > >> > > > > NOTE: nothing about this set-
up
> > > > > > > > > >>> > > > >> > > > > has
> > > > > > > > > >>> > > > >> > > > > changed
> > > > > > other
> > > > > > > > > >>> > > > >> > > > > than
> > > > > > > > > >>> > > > >> > > > > the
> > > > > > > > > >>> > > > >> directory
> > > > > > > > > >>> > > > >> > > > path
> > > > > > > > > >>> > > > >> > > > > onto dell. The path has been
> > > > > > > > > >>> > > > >> > > > > created
> > > > > > > > > >>> > > > >> > > > > so
> > > > > > > > > >>> > > > >> > > > > it
> > > > > > should
> > > > > > > > > >>> > > > >> > > > > support
> > > > > > > > > >>> > > > >> > > > > the
> > > > > > > > > >>> > > > >> > transfer
> > > > > > > > > >>> > > > >> > > > of
> > > > > > > > > >>> > > > >> > > > > plots generated on the MET
server.
> > > > > > > > > >>> > > > >> > > > > The
> > > > > > preceding
> > > > > > > > > >>> > > > >> > > > > information
> > > > > > > > > >>> > > > >> > indicates
> > > > > > > > > >>> > > > >> > > > > that the plot was processed
> > > > > > > > > >>> > > > >> > > > > successfully.
> > > > > It's
> > > > > > > > > >>> > > > >> > > > > not
> > > > > > > > > >>> > > > >> > > > > clear
> > > > > > > > > >>> > > > >> > > > > why I
> > > > > > > > > >>> > > > >> can't
> > > > > > > > > >>> > > > >> > > > > transfer my plots over.
> > > > > > > > > >>> > > > >> > > > > --
> > > > > > > > > >>> > > > >> > > > > Edward Strobach
> > > > > > > > > >>> > > > >> > > > > EMC/NCEP/NWS/
> > > > > > > > > >>> > > > >> > > > > IMSG Contractor
> > > > > > > > > >>> > > > >> > > > > Cubicle#: 2029
> > > > > > > > > >>> > > > >> > > > > 301-683-3717
> > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > >>> > > > >> > > >
> > > > > > > > > >>> > > > >> > > > --
> > > > > > > > > >>> > > > >> > > > Julie Prestopnik (she/her)
> > > > > > > > > >>> > > > >> > > > Software Engineer
> > > > > > > > > >>> > > > >> > > > National Center for Atmospheric
> > > > > > > > > >>> > > > >> > > > Research
> > > > > > > > > >>> > > > >> > > > Research Applications Laboratory
> > > > > > > > > >>> > > > >> > > > Email: jpresto at ucar.edu
> > > > > > > > > >>> > > > >> > > >
> > > > > > > > > >>> > > > >> > > > My working day may not be your
> > > > > > > > > >>> > > > >> > > > working
> > > > > > > > > >>> > > > >> > > > day.
> > > > > > Please
> > > > > > > > > >>> > > > >> > > > do not
> > > > > > > > > >>> > > > >> > > > feel
> > > > > > > > > >>> > > > >> obliged
> > > > > > > > > >>> > > > >> > > to
> > > > > > > > > >>> > > > >> > > > reply to this email outside of
your
> > > > > > > > > >>> > > > >> > > > normal
> > > > > working
> > > > > > > > > >>> > > > >> > > > hours.
> > > > > > > > > >>> > > > >> > > >
> > > > > > > > > >>> > > > >> > > >
> > > > > > > > > >>> > > > >> > >
> > > > > > > > > >>> > > > >> > > --
> > > > > > > > > >>> > > > >> > > Edward Strobach
> > > > > > > > > >>> > > > >> > > EMC/NCEP/NWS/
> > > > > > > > > >>> > > > >> > > IMSG Contractor
> > > > > > > > > >>> > > > >> > > Cubicle#: 2029
> > > > > > > > > >>> > > > >> > > 301-683-3717
> > > > > > > > > >>> > > > >> > >
> > > > > > > > > >>> > > > >> > >
> > > > > > > > > >>> > > > >> >
> > > > > > > > > >>> > > > >> > --
> > > > > > > > > >>> > > > >> > Julie Prestopnik (she/her)
> > > > > > > > > >>> > > > >> > Software Engineer
> > > > > > > > > >>> > > > >> > National Center for Atmospheric
Research
> > > > > > > > > >>> > > > >> > Research Applications Laboratory
> > > > > > > > > >>> > > > >> > Email: jpresto at ucar.edu
> > > > > > > > > >>> > > > >> >
> > > > > > > > > >>> > > > >> > My working day may not be your
working
> > > > > > > > > >>> > > > >> > day.
> > > > > > > > > >>> > > > >> > Please
> > > > > do
> > > > > > > > > >>> > > > >> > not
> > > > > > > > > >>> > > > >> > feel
> > > > > > > > > >>> > > > >> > obliged
> > > > > > > > > >>> > > > >> to
> > > > > > > > > >>> > > > >> > reply to this email outside of your
> > > > > > > > > >>> > > > >> > normal
> > > > > > > > > >>> > > > >> > working
> > > > > > > > > >>> > > > >> > hours.
> > > > > > > > > >>> > > > >> >
> > > > > > > > > >>> > > > >> >
> > > > > > > > > >>> > > > >>
> > > > > > > > > >>> > > > >> --
> > > > > > > > > >>> > > > >> Edward Strobach
> > > > > > > > > >>> > > > >> EMC/NCEP/NWS/
> > > > > > > > > >>> > > > >> IMSG Contractor
> > > > > > > > > >>> > > > >> Cubicle#: 2029
> > > > > > > > > >>> > > > >> 301-683-3717
> > > > > > > > > >>> > > > >>
> > > > > > > > > >>> > > > >>
> > > > > > > > > >>> > > > >
> > > > > > > > > >>> > > > > --
> > > > > > > > > >>> > > > > Julie Prestopnik (she/her)
> > > > > > > > > >>> > > > > Software Engineer
> > > > > > > > > >>> > > > > National Center for Atmospheric Research
> > > > > > > > > >>> > > > > Research Applications Laboratory
> > > > > > > > > >>> > > > > Email: jpresto at ucar.edu
> > > > > > > > > >>> > > > >
> > > > > > > > > >>> > > > > My working day may not be your working
day.
> > > > > > > > > >>> > > > > Please do
> > > > > > not
> > > > > > > > > >>> > > > > feel
> > > > > > > > > >>> > > > > obliged to
> > > > > > > > > >>> > > > > reply to this email outside of your
normal
> > > > > > > > > >>> > > > > working
> > > > > hours.
> > > > > > > > > >>> > > > >
> > > > > > > > > >>> > >
> > > > > > > > > >>> > >
> > > > > > > > > >>> > >
> > > > > > > > > >>> > >
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > > >> --
> > > > > > > > > >> Edward Strobach
> > > > > > > > > >> EMC/NCEP/NWS/
> > > > > > > > > >> IMSG Contractor
> > > > > > > > > >> Cubicle#: 2029
> > > > > > > > > >> 301-683-3717
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Edward Strobach
> > > > > > > > > > EMC/NCEP/NWS/
> > > > > > > > > > IMSG Contractor
> > > > > > > > > > Cubicle#: 2029
> > > > > > > > > > 301-683-3717
> > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Edward Strobach
> > > > > > > EMC/NCEP/NWS/
> > > > > > > IMSG Contractor
> > > > > > > Cubicle#: 2029
> > > > > > > 301-683-3717
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Julie Prestopnik (she/her)
> > > > > > Software Engineer
> > > > > > National Center for Atmospheric Research
> > > > > > Research Applications Laboratory
> > > > > > Email: jpresto at ucar.edu
> > > > > >
> > > > > > My working day may not be your working day. Please do not
> > > > > > feel
> > > > > > obliged
> > > > > to
> > > > > > reply to this email outside of your normal working hours.
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Edward Strobach
> > > > > EMC/NCEP/NWS/
> > > > > IMSG Contractor
> > > > > Cubicle#: 2029
> > > > > 301-683-3717
> > > > >
> > > > >
> >
> >
> >
> >
------------------------------------------------
Subject: can't copy plots over
From: Edward Strobach - NOAA Affiliate
Time: Thu Dec 03 09:17:36 2020
That's why I had made some suggestions but didn't get feedback. I
don't
believe I'm loading in parallel, however. I first load the chemistry
verification followed by meteorology verification. Afterwards I
simultaneously submit a 'batch' job for time series statistics, taylor
diagrams, and CSI. I need to make sure that those work and test the
simultaneity over multiple days so I can understand this issue better.
It
may be that I have to insert logic to separate these processes in
time.
However, since I aggregate statistics over a month, this could become
problematic as the processing time increases substantially and could
interfere with processing the following day.
On Thu, Dec 3, 2020 at 11:09 AM Tatiana Burek via RT
<met_help at ucar.edu>
wrote:
> Since your database is quite large, not creating the indexes would
help to
> speed up the loading. You also need to make sure that only one
loading
> process is running at a time. Multiple loading could create invalid
> entities and would stall the inserts.
>
> Tatiana
>
> On Thu Dec 03 08:28:52 2020, edward.strobach at noaa.gov wrote:
> > If you mean this suggestions:
> >
> > <drop_indexes>false</drop_indexes>
> > <apply_indexes>false</apply_indexes>
> >
> > then yes. However, until just now I didn't realize that my
> > load_to_metviewer_AWS.py set the template to have these options as
> > true. I
> > have since changed these options to false in the script. There
are
> > some
> > issues in my set-up with the Taylor Diagrams and CSI that I need
to
> > correct
> > for. Will the above option permit me to conduct frequent tests if
> > needed?
> >
> > On Thu, Dec 3, 2020 at 10:04 AM Tatiana Burek via RT
> > <met_help at ucar.edu>
> > wrote:
> >
> > > Edward
> > > AWS has 3 loading jobs that wore started by you:
> > > 2 for load_mv_cmaq_metplus_CMAQ.xml and 2 for
> > > load_mv_cmaq_metplus_CMAQ_Meteorology.xml
> > > and 25 batch jobs.
> > > None of them are executing because of the database locks created
by
> > > so
> > > many incerts, create indexes and select statments.
> > > Shoud I kill all your jbs aagain? Have you had a chance to look
at my
> > > sugesstuion on how to improve the process?
> > >
> > > Tatiana
> > >
> > > On Wed Dec 02 10:10:40 2020, tatiana wrote:
> > > > Edward
> > > > I am sorry, I did not mentioned that the commands I sent you
should
> > > > be
> > > > ran on AWS server and not on WCOSS
> > > >
> > > > Tatiana
> > > >
> > > > On Wed Dec 02 09:42:20 2020, jpresto wrote:
> > > > > Hi Edward.
> > > > >
> > > > > You should be able to distinguish what processes are MET and
what
> > > > > are
> > > > > not.
> > > > > From your "metcheck" results, it seems you have no MET
processes
> > > > > running.
> > > > > I'll explain below.
> > > > >
> > > > > [Edward.Strobach at v71a2 ~]$ metcheck
> > > > > > root 76468 12071 0 16:25 ? 00:00:00 sshd:
> > > > > > Edward.Strobach
> > > > > > [priv]
> > > > > > Edward.+ 76482 76468 0 16:25 ? 00:00:00 sshd:
> > > > > > Edward.Strobach at pts
> > > > > > /19
> > > > > > Edward.+ 85480 82420 0 16:26 pts/19 00:00:00 grep
> > > > > > --color=auto
> > > > > > Edward.Strobach
> > > > > > root 141334 12071 0 12:27 ? 00:00:00 sshd:
> > > > > > Edward.Strobach
> > > > > > [priv]
> > > > > > Edward.+ 141337 141334 0 12:27 ? 00:00:00 sshd:
> > > > > > Edward.Strobach at pts
> > > > > > /5
> > > > >
> > > > >
> > > > > All of the processes that start with "sshd" are your logins,
so
> > > > > only
> > > > > if you
> > > > > ran "kill -9" on one of those would you get logged out. The
> > > > > "grep"
> > > > > processes (3rd result) is your running of "ps -ef | grep
> > > > > Edward.Strobach".
> > > > > If there were MET processes running you would be able to see
them
> > > > > here.
> > > > > The fact that there are none listed here could be because
when
> > > > > you
> > > > > ran
> > > > > "pkill -9 -u `id -u Edward.Strobach`", they were all killed,
or
> > > > > because
> > > > > you're on a different node (v71a2) than the one you ran them
on.
> > > > >
> > > > > Julie
> > > > >
> > > > > On Wed, Dec 2, 2020 at 9:28 AM Edward Strobach - NOAA
Affiliate
> > > > > via
> > > > > RT
> > > > > <
> > > > > met_help at ucar.edu> wrote:
> > > > >
> > > > > >
> > > > > > <URL:
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
> > > > > >
> > > > > > Thanks. I was able to determine the processes using ps
-ef |
> > > > > > grep
> > > > > > Edward.Strobach
> > > > > >
> > > > > > I did try pkill -9 -u `id -u Edward.Strobach` before
sending
> > > > > > your
> > > > > > email
> > > > > > which did log me out unfortunately.
> > > > > >
> > > > > > I also noticed that kill -9 jobid also logs me out. I
wish
> > > > > > there
> > > > > > was
> > > > > > a way
> > > > > > to distinguish between MET tasks v. other tasks. When I
look
> > > > > > at
> > > > > > what
> > > > > > I
> > > > > > currently have I can't tell what is MET and what is not
MET.
> > > > > > [Edward.Strobach at v71a2 ~]$ metcheck
> > > > > > root 76468 12071 0 16:25 ? 00:00:00 sshd:
> > > > > > Edward.Strobach
> > > > > > [priv]
> > > > > > Edward.+ 76482 76468 0 16:25 ? 00:00:00 sshd:
> > > > > > Edward.Strobach at pts
> > > > > > /19
> > > > > > Edward.+ 85480 82420 0 16:26 pts/19 00:00:00 grep
> > > > > > --color=auto
> > > > > > Edward.Strobach
> > > > > > root 141334 12071 0 12:27 ? 00:00:00 sshd:
> > > > > > Edward.Strobach
> > > > > > [priv]
> > > > > > Edward.+ 141337 141334 0 12:27 ? 00:00:00 sshd:
> > > > > > Edward.Strobach at pts
> > > > > > /5
> > > > > >
> > > > > > Note that metcheck is just an alias for ps -ef | grep
> > > > > > Edward.Strobach
> > > > > >
> > > > > > On Wed, Dec 2, 2020 at 11:13 AM Julie Prestopnik via RT
> > > > > > <met_help at ucar.edu
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Edward.
> > > > > > >
> > > > > > > Because capitals matter when running "grep" and because
WCOSS
> > > > > > > uses
> > > > > > capital
> > > > > > > letters for our username you can run either of the
following
> > > > > > > instead:
> > > > > > >
> > > > > > > ps -ef | grep -i edward (the "-i" tells grep to ignore
case)
> > > > > > > ps -ef | grep Edward
> > > > > > >
> > > > > > > The second column will list the process id and the third
> > > > > > > column
> > > > > > > will list
> > > > > > > the parent process id.
> > > > > > >
> > > > > > > The same issue exists with running pkill on WCOSS. You
can
> > > > > > > run
> > > > > > > pkill on
> > > > > > > WCOSS using the following, however, I don't recommend it
and
> > > > > > > I'll
> > > > > > > explain
> > > > > > > below:
> > > > > > >
> > > > > > > pkill -9 -u `id -u Edward.Strobach`
> > > > > > >
> > > > > > > First, you'd need to make sure you're logged in
appropriately
> > > > > > > and
> > > > > > > be on
> > > > > > the
> > > > > > > same node (not sure if that is the correct terminology)
as
> > > > > > > the
> > > > > > > one
> > > > > > running
> > > > > > > your processes. For example, I ran a simple check to
test
> > > > > > > this.
> > > > > > > I
> > > > > > > ran
> > > > > > > "top" when I was logged into "Julie.Prestopnik@*v71a3*",
but
> > > > > > > my
> > > > > > > second
> > > > > > log
> > > > > > > in was on "Julie.Prestopnik@*v72a1*". v72a1 did not
> > > > > > > recognize
> > > > > > > "top" as
> > > > > > > running, which I could see when running "ps -ef | grep
> > > > > > > Julie".
> > > > > > > When I
> > > > > > ran
> > > > > > > "ssh v71a3" and then reran "ps -ef | grep Julie", I
could
> > > > > > > then
> > > > > > > see
> > > > > > > "top"
> > > > > > > was running. Running "pkill -9 -u `id -u
Julie.Presotpnik`"
> > > > > > > also
> > > > > > > closed
> > > > > > my
> > > > > > > connection to WCOSS, so if you don't want your
connection to
> > > > > > > WCOSS
> > > > > > killed,
> > > > > > > which I didn't, I would just kill the processes
individually
> > > > > > > using
> > > > > > > the
> > > > > > kill
> > > > > > > command with the process number.
> > > > > > >
> > > > > > > I hope that helps!
> > > > > > >
> > > > > > > Julie
> > > > > > >
> > > > > > > On Wed, Dec 2, 2020 at 8:51 AM Edward Strobach - NOAA
> > > > > > > Affiliate
> > > > > > > via
> > > > > > > RT <
> > > > > > > met_help at ucar.edu> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > <URL:
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687 >
> > > > > > > >
> > > > > > > > I see this when running ps -ef | grep edward
> > > > > > > > Edward.+ 115762 154642 0 15:48 pts/31 00:00:00 grep
> > > > > > > > --color=auto
> > > > > > > edward
> > > > > > > >
> > > > > > > > The numbers there change every time I run this step.
> > > > > > > >
> > > > > > > > pkill -9 -u `id -u edward.strobach` does not seem to
work.
> > > > > > > >
> > > > > > > > On Wed, Dec 2, 2020 at 9:34 AM Tatiana Burek via RT
> > > > > > > > <met_help at ucar.edu
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > You can kill your processes by :
> > > > > > > > > - collecting what is running:
> > > > > > > > > ps -ef | grep edward
> > > > > > > > > - killing each process
> > > > > > > > > kill -9 2574
> > > > > > > > > kill -9 2594
> > > > > > > > > or all your processes:
> > > > > > > > > pkill -9 -u `id -u edward.strobach`
> > > > > > > > >
> > > > > > > > > Right now you have 4 loading scripts and 10 batch
scripts
> > > > > > > > > running at
> > > > > > > the
> > > > > > > > > same time which creates a gridlock and none of the
> > > > > > > > > processes
> > > > > > > > > are
> > > > > > > getting
> > > > > > > > > executes.
> > > > > > > > >
> > > > > > > > > Tatiana
> > > > > > > > >
> > > > > > > > > On Wed Dec 02 06:32:23 2020,
edward.strobach at noaa.gov
> > > > > > > > > wrote:
> > > > > > > > > > I came across this issue again this morning. Is
there
> > > > > > > > > > a
> > > > > > > > > > way
> > > > > > > > > > to
> > > > > > stop
> > > > > > > > > > certain processes and start fresh from my end?
> > > > > > > > > >
> > > > > > > > > > On Tue, Dec 1, 2020 at 5:14 PM Edward Strobach -
NOAA
> > > > > > > > > > Affiliate <
> > > > > > > > > > edward.strobach at noaa.gov> wrote:
> > > > > > > > > >
> > > > > > > > > > > maybe there is a script that allows me to check
what
> > > > > > > > > > > is
> > > > > > > > > > > going on
> > > > > > > with
> > > > > > > > > > > my
> > > > > > > > > > > database? A lot of these issues I've come
across
> > > > > > > > > > > happen
> > > > > > > > > > > because
> > > > > > of
> > > > > > > > > > > things
> > > > > > > > > > > I'm unaware of or things that go on
simultaneously
> > > > > > > > > > > that
> > > > > > > > > > > causes
> > > > > > > > > > > confusion in
> > > > > > > > > > > the processing. Right now I have a plotting
script
> > > > > > > > > > > set
> > > > > > > > > > > that
> > > > > > > > > > > simultaneously
> > > > > > > > > > > submits batch jobs for time series statistics,
Taylor
> > > > > > > > > > > diagrams,
> > > > > > and
> > > > > > > > > > > CSI. I
> > > > > > > > > > > haven't seen too many issues when I do that, but
> > > > > > > > > > > would it
> > > > > > > > > > > make
> > > > > > > sense
> > > > > > > > > > > to
> > > > > > > > > > > insert a while loop after the first submission
that
> > > > > > > > > > > uses
> > > > > > > > > > > a
> > > > > > > > > > > script
> > > > > > > to
> > > > > > > > > > > check
> > > > > > > > > > > the activity within my database? In other
words, I
> > > > > > > > > > > would
> > > > > > > > > > > use a
> > > > > > > > > > > script,
> > > > > > > > > > > perhaps one that has already been developed, to
check
> > > > > > > > > > > the
> > > > > > > > > > > status
> > > > > > of
> > > > > > > > > > > processing before submitting the next batch job.
> > > > > > > > > > > Once
> > > > > > > > > > > the
> > > > > > > processing
> > > > > > > > > > > for
> > > > > > > > > > > the first plotting steps have completed, then I
would
> > > > > > > > > > > commence to
> > > > > > > the
> > > > > > > > > > > next
> > > > > > > > > > > one.
> > > > > > > > > > >
> > > > > > > > > > > I guess check_my_db.sh would not do that?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Dec 1, 2020 at 4:27 PM Edward Strobach -
NOAA
> > > > > > > > > > > Affiliate <
> > > > > > > > > > > edward.strobach at noaa.gov> wrote:
> > > > > > > > > > >
> > > > > > > > > > >> are there youtube examples or a section in a
> > > > > > > > > > >> technical
> > > > > > > > > > >> manual
> > > > > > that
> > > > > > > > > > >> goes
> > > > > > > > > > >> over examples illustrating best practices
related to
> > > > > > > > > > >> this
> > > > > > process?
> > > > > > > > > > >>
> > > > > > > > > > >> On Tue, Dec 1, 2020 at 4:03 PM Tatiana Burek
via RT
> > > > > > > > > > >> <met_help at ucar.edu>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >>> For your process you need only two scripts:
> > > > > > > > > > >>> mv_load_to_aws.sh
> > > > > > > > > > >>> mv_batch_on_aws.sh
> > > > > > > > > > >>> I don't think, that others would help you.
> > > > > > > > > > >>>
> > > > > > > > > > >>> The main work is in creating efficient XMLs
for the
> > > > > > > > > > >>> each
> > > > > > > > > > >>> step.
> > > > > > > > > > >>> Like not recalculating indexes or performing
> > > > > > stat_header_db_check
> > > > > > > > > > >>> during
> > > > > > > > > > >>> data loading.
> > > > > > > > > > >>>
> > > > > > > > > > >>> Tatiana
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>> On Tue Dec 01 13:00:15 2020,
> > > > > > > > > > >>> edward.strobach at noaa.gov
> > > > > > > > > > >>> wrote:
> > > > > > > > > > >>> > Thanks. I just made those changes in the
xml
> > > > > > > > > > >>> > file,
> > > > > > > > > > >>> > so
> > > > > > > > > > >>> > I
> > > > > > > 'guess'
> > > > > > > > > > >>> > I'm
> > > > > > > > > > >>> > ready
> > > > > > > > > > >>> > to go.
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > I should note that I have several steps that
I
> > > > > > > > > > >>> > run
> > > > > > > > > > >>> > each
> > > > > > day. I
> > > > > > > > > > >>> > convert
> > > > > > > > > > >>> > files into netcdf files that can be read by
MET,
> > > > > > > > > > >>> > I
> > > > > > > > > > >>> > generate
> > > > > > the
> > > > > > > > > > >>> > stat
> > > > > > > > > > >>> > files,
> > > > > > > > > > >>> > and I use the stat files in the MET batch
> > > > > > > > > > >>> > plotting
> > > > > > > > > > >>> > system. I
> > > > > > > > > > >>> > have
> > > > > > > > > > >>> > actually
> > > > > > > > > > >>> > 5 total steps, but the plot batch system is
> > > > > > > > > > >>> > really
> > > > > > > > > > >>> > the
> > > > > > > > > > >>> > only
> > > > > > one
> > > > > > > > > > >>> > that
> > > > > > > > > > >>> > gives
> > > > > > > > > > >>> > me trouble on a regular basis. I wish I
knew how
> > > > > > > > > > >>> > to
> > > > > > > > > > >>> > reach a
> > > > > > > > > > >>> > point
> > > > > > > > > > >>> > where I
> > > > > > > > > > >>> > have a plotting package that would produce
the
> > > > > > > > > > >>> > plots
> > > > > > > > > > >>> > that I
> > > > > > > need
> > > > > > > > > > >>> > each
> > > > > > > > > > >>> > day.
> > > > > > > > > > >>> > I'm expected to hand the final product off
in the
> > > > > > > > > > >>> > near
> > > > > > future,
> > > > > > > > > > >>> > but
> > > > > > > > > > >>> > this
> > > > > > > > > > >>> > part has delayed me extensively. I know that
I'm
> > > > > > > > > > >>> > generating a
> > > > > > > lot
> > > > > > > > > > >>> > of
> > > > > > > > > > >>> > plots
> > > > > > > > > > >>> > which could potentially overwhelm the
system, but
> > > > > > > > > > >>> > I'm
> > > > > > > > > > >>> > unaware
> > > > > > > how
> > > > > > > > > > >>> > others
> > > > > > > > > > >>> > work to circumvent that. I hope to move
past
> > > > > > > > > > >>> > this
> > > > > > > > > > >>> > soon
> > > > > > > > > > >>> > so I
> > > > > > > can
> > > > > > > > > > >>> > continue
> > > > > > > > > > >>> > development work.
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > There are several scripts that I don't use
that
> > > > > > > > > > >>> > might
> > > > > > > > > > >>> > be
> > > > > > > helpful.
> > > > > > > > > > >>> > Do
> > > > > > > > > > >>> > any
> > > > > > > > > > >>> > of these work to help me manage the system
> > > > > > > > > > >>> > better?
> > > > > > > > > > >>> > check_my_db.sh mv_batch_on_aws.sh
> > > > > > > > > > >>> > mv_create_db_on_aws.sh
> > > > > > > > > > >>> > mv_db_size_on_aws.sh
mv_delete_db_on_aws.sh
> > > > > > > mv_load_to_aws.sh
> > > > > > > > > > >>> >
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > On Tue, Dec 1, 2020 at 2:44 PM Tatiana Burek
via
> > > > > > > > > > >>> > RT
> > > > > > > > > > >>> > <met_help at ucar.edu>
> > > > > > > > > > >>> > wrote:
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > > Edward
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > I run a few butch jobs from my laptop and
made
> > > > > > > > > > >>> > > sure
> > > > > > > > > > >>> > > that
> > > > > > > > > > >>> > > METviewer
> > > > > > > > > > >>> > > code
> > > > > > > > > > >>> > > and scripts working as expected: I was
able to
> > > > > > > > > > >>> > > produce and
> > > > > > > copy
> > > > > > > > > > >>> > > plots.
> > > > > > > > > > >>> > > I also got an error like the one that you
> > > > > > > > > > >>> > > reported
> > > > > > > > > > >>> > > but only
> > > > > > > > > > >>> > > when
> > > > > > > > > > >>> > > METviewer
> > > > > > > > > > >>> > > did not generate any plots and the script
> > > > > > > > > > >>> > > couldn't
> > > > > > > > > > >>> > > copy
> > > > > > empty
> > > > > > > > > > >>> > > directory.
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > During my testing I noticed a few batch
jobs
> > > > > > > > > > >>> > > that
> > > > > > > > > > >>> > > where
> > > > > > > > > > >>> > > started by
> > > > > > > > > > >>> > > you
> > > > > > > > > > >>> > > and were handing. Each of these jobs
started a
> > > > > > > > > > >>> > > database
> > > > > > query
> > > > > > > > > > >>> > > and
> > > > > > > > > > >>> > > they also
> > > > > > > > > > >>> > > were piling up and handing. They all were
> > > > > > > > > > >>> > > waiting
> > > > > > > > > > >>> > > for
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > indexing
> > > > > > > > > > >>> > > that
> > > > > > > > > > >>> > > takes place after the data loading.
> > > > > > > > > > >>> > > I had to kill all database and batch
processes
> > > > > > > > > > >>> > > in
> > > > > > > > > > >>> > > order to
> > > > > > > move
> > > > > > > > > > >>> > > forward
> > > > > > > > > > >>> > > with the testing.
> > > > > > > > > > >>> > > To avoid this gridlock you could:
> > > > > > > > > > >>> > > - in your loading xml use
> > > > > > <drop_indexes>false</drop_indexes>
> > > > > > > > > > >>> > > <apply_indexes>false</apply_indexes>
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > - do not start a few parallel batch jobs
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > I will add a info message to the script
that
> > > > > > > > > > >>> > > would
> > > > > > > > > > >>> > > explain
> > > > > > > why
> > > > > > > > > > >>> > > the
> > > > > > > > > > >>> > > error
> > > > > > > > > > >>> > > occurs.
> > > > > > > > > > >>> > > For now, could you test batch with some
simple
> > > > > > > > > > >>> > > XML
> > > > > > > > > > >>> > > that
> > > > > > > produce
> > > > > > > > > > >>> > > at
> > > > > > > > > > >>> > > least
> > > > > > > > > > >>> > > one image.
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > Tatiana
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > > On Tue Dec 01 11:35:03 2020, jpresto
wrote:
> > > > > > > > > > >>> > > > Hi Edward.
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > After talking with Tatiana, it sounds
like I
> > > > > > > > > > >>> > > > am
> > > > > > misinformed
> > > > > > > > > > >>> > > > and the
> > > > > > > > > > >>> > > > scripts
> > > > > > > > > > >>> > > > you are using were, in fact, developed
by our
> > > > > > > > > > >>> > > > METviewer
> > > > > > > team.
> > > > > > > > > > >>> > > > My
> > > > > > > > > > >>> > > > apologies. METplus has a lot of
components,
> > > > > > > > > > >>> > > > and
> > > > > > > > > > >>> > > > while I
> > > > > > > > > > >>> > > > thought I
> > > > > > > > > > >>> > > > had
> > > > > > > > > > >>> > > > a
> > > > > > > > > > >>> > > > good understanding of who did what, it
turns
> > > > > > > > > > >>> > > > out,
> > > > > > > > > > >>> > > > I
> > > > > > > > > > >>> > > > did
> > > > > > > not.
> > > > > > > > > > >>> > > > Tatiana
> > > > > > > > > > >>> > > > is
> > > > > > > > > > >>> > > > working on this issue and will follow up
once
> > > > > > > > > > >>> > > > she
> > > > > > > > > > >>> > > > knows
> > > > > > > more.
> > > > > > > > > > >>> > > > Thank
> > > > > > > > > > >>> > > > you
> > > > > > > > > > >>> > > > for your patience and understanding.
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > Julie
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > On Tue, Dec 1, 2020 at 11:22 AM Julie
> > > > > > > > > > >>> > > > Prestopnik
> > > > > > > > > > >>> > > > <jpresto at ucar.edu
> > > > > > > > > > >>> >
> > > > > > > > > > >>> > > > wrote:
> > > > > > > > > > >>> > > >
> > > > > > > > > > >>> > > > > Thanks, Edward. I directed you to
Mallory
> > > > > > > > > > >>> > > > > and
> > > > > > > > > > >>> > > > > Perry
> > > > > > > > > > >>> > > > > because I
> > > > > > > > > > >>> > > > > thought you
> > > > > > > > > > >>> > > > > might be using scripts from them. We
did
> > > > > > > > > > >>> > > > > not
> > > > > > > > > > >>> > > > > develop
> > > > > > and
> > > > > > > > > > >>> > > > > have
> > > > > > > > > > >>> > > > > not
> > > > > > > > > > >>> > > > > used
> > > > > > > > > > >>> > > > > those scripts. We have no experience
> > > > > > > > > > >>> > > > > interacting
> > > > > > > > > > >>> > > > > with
> > > > > > > AWS
> > > > > > > > > > >>> > > > > and
> > > > > > > > > > >>> > > > > WCOSS,
> > > > > > > > > > >>> > > > > so
> > > > > > > > > > >>> > > > > they seemed best suited to answer your
> > > > > > > > > > >>> > > > > questions
> > > > > > related
> > > > > > > to
> > > > > > > > > > >>> > > > > that
> > > > > > > > > > >>> > > > > process.
> > > > > > > > > > >>> > > > > However, as Tatiana pointed out,
perhaps
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > security
> > > > > > > > > > >>> > > > > changes on
> > > > > > > > > > >>> > > > > the
> > > > > > > > > > >>> > > > > AWS
> > > > > > > > > > >>> > > > > server have something to do with what
you
> > > > > > > > > > >>> > > > > are
> > > > > > > experiencing.
> > > > > > > > > > >>> > > > > If
> > > > > > > > > > >>> > > > > that
> > > > > > > > > > >>> > > > > turns
> > > > > > > > > > >>> > > > > out not to be the case, Mallory and
Perry
> > > > > > > > > > >>> > > > > would
> > > > > > > > > > >>> > > > > be best
> > > > > > > > > > >>> > > > > able to
> > > > > > > > > > >>> > > > > help
> > > > > > > > > > >>> > > > > you.
> > > > > > > > > > >>> > > > > If they are unable to help, please let
us
> > > > > > > > > > >>> > > > > know,
> > > > > > > > > > >>> > > > > and we
> > > > > > > can
> > > > > > > > > > >>> > > > > see
> > > > > > > > > > >>> > > > > what
> > > > > > > > > > >>> > > > > they
> > > > > > > > > > >>> > > > > tried and how we can help you get this
> > > > > > > > > > >>> > > > > problem
> > > > > > > > > > >>> > > > > figured
> > > > > > > out.
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > Julie
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > On Tue, Dec 1, 2020 at 11:15 AM Edward
> > > > > > > > > > >>> > > > > Strobach
> > > > > > > > > > >>> > > > > -
> > > > > > > > > > >>> > > > > NOAA
> > > > > > > > > > >>> > > > > Affiliate
> > > > > > > > > > >>> > > > > via
> > > > > > > > > > >>> > > > > RT <
> > > > > > > > > > >>> > > > > met_help at ucar.edu> wrote:
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > >>
> > > > > > > > > > >>> > > > >> <URL:
> > > > > > > > > > >>> > > > >>
> > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > > > > > >
> > > > > > > > > > >>> > > > >>
> > > > > > > > > > >>> > > > >> Hi Julie,
> > > > > > > > > > >>> > > > >>
> > > > > > > > > > >>> > > > >> I'm actually using the scripts
located
> > > > > > > > > > >>> > > > >> here:
> > > > > > > > > > >>> > > > >>
> > > > > > > > > > >>>
> > > > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer/scripts.
> > > > > > > > > > >>> > > > >> These
> > > > > > > > > > >>> > > > >> were scripts that I either got from
> > > > > > > > > > >>> > > > >> Mallory or
> > > > > > > > > > >>> > > > >> Perry a
> > > > > > > > > > >>> > > > >> while
> > > > > > > > > > >>> > > > >> back -
> > > > > > > > > > >>> > > > >> around
> > > > > > > > > > >>> > > > >> May.
> > > > > > > > > > >>> > > > >>
> > > > > > > > > > >>> > > > >> On Tue, Dec 1, 2020 at 12:59 PM Julie
> > > > > > > > > > >>> > > > >> Prestopnik
> > > > > > > > > > >>> > > > >> via
> > > > > > RT
> > > > > > > <
> > > > > > > > > > >>> > > > >> met_help at ucar.edu>
> > > > > > > > > > >>> > > > >> wrote:
> > > > > > > > > > >>> > > > >>
> > > > > > > > > > >>> > > > >> > Hi Edward.
> > > > > > > > > > >>> > > > >> >
> > > > > > > > > > >>> > > > >> > Are you using the scripts that are
> > > > > > > > > > >>> > > > >> > located
> > > > > > > > > > >>> > > > >> > in
> > > > > > > > > > >>> > > > >> >
> > > > > > > > >
> /gpfs/hps3/emc/global/noscrub/Mallory.Row/VRFY/METviewer_AWS
> > > > > > > > > > >>> > > > >> > and
> > > > > > > > > > >>> > > > >> >
> > > > > > > > >
> /gpfs/dell2/emc/verification/noscrub/emc.metplus/METviewer_AWS
> > > > > > > > > > >>> > > > >> > for interaction between AWS and
WCOSS?
> > > > > > > > > > >>> > > > >> >
> > > > > > > > > > >>> > > > >> > Thanks!
> > > > > > > > > > >>> > > > >> >
> > > > > > > > > > >>> > > > >> > Julie
> > > > > > > > > > >>> > > > >> >
> > > > > > > > > > >>> > > > >> > On Tue, Dec 1, 2020 at 10:49 AM
Edward
> > > > > > > > > > >>> > > > >> > Strobach -
> > > > > > NOAA
> > > > > > > > > > >>> > > > >> > Affiliate
> > > > > > > > > > >>> > > > >> > via RT
> > > > > > > > > > >>> > > > >> <
> > > > > > > > > > >>> > > > >> > met_help at ucar.edu> wrote:
> > > > > > > > > > >>> > > > >> >
> > > > > > > > > > >>> > > > >> > >
> > > > > > > > > > >>> > > > >> > > <URL:
> > > > > > > > > > >>> > > > >> > >
> > > > > > > >
https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > > > > > > > > >>> > > > >> > > >
> > > > > > > > > > >>> > > > >> > >
> > > > > > > > > > >>> > > > >> > > Perry does not do batch plotting
with
> > > > > > > > > > >>> > > > >> > > MET.
> > > > > > > > > > >>> > > > >> > > I was
> > > > > > > > > > >>> > > > >> > > under the
> > > > > > > > > > >>> > > > >> > > impression
> > > > > > > > > > >>> > > > >> > that
> > > > > > > > > > >>> > > > >> > > the MET helpdesk helped with MET
> > > > > > > > > > >>> > > > >> > > related
> > > > > > > > > > >>> > > > >> > > issues.
> > > > > > > > > > >>> > > > >> > >
> > > > > > > > > > >>> > > > >> > > On Tue, Dec 1, 2020 at 12:13 PM
Julie
> > > > > > > > > > >>> > > > >> > > Prestopnik
> > > > > > via
> > > > > > > > > > >>> > > > >> > > RT <
> > > > > > > > > > >>> > > > >> > met_help at ucar.edu
> > > > > > > > > > >>> > > > >> > > >
> > > > > > > > > > >>> > > > >> > > wrote:
> > > > > > > > > > >>> > > > >> > >
> > > > > > > > > > >>> > > > >> > > > Hi Edward.
> > > > > > > > > > >>> > > > >> > > >
> > > > > > > > > > >>> > > > >> > > > Please consult with Mallory Row
and
> > > > > > > > > > >>> > > > >> > > > Perry
> > > > > > Shafran
> > > > > > > > > > >>> > > > >> > > > for
> > > > > > > > > > >>> > > > >> > > > assistance
> > > > > > > > > > >>> > > > >> with
> > > > > > > > > > >>> > > > >> > > this
> > > > > > > > > > >>> > > > >> > > > issue. They are most familiar
with
> > > > > > > > > > >>> > > > >> > > > these
> > > > > > > > > > >>> > > > >> > > > capabilities and
> > > > > > > > > > >>> > > > >> > > > will be
> > > > > > > > > > >>> > > > >> best
> > > > > > > > > > >>> > > > >> > > > able to assist you.
> > > > > > > > > > >>> > > > >> > > >
> > > > > > > > > > >>> > > > >> > > > Thanks!
> > > > > > > > > > >>> > > > >> > > >
> > > > > > > > > > >>> > > > >> > > > Julie
> > > > > > > > > > >>> > > > >> > > >
> > > > > > > > > > >>> > > > >> > > > On Tue, Dec 1, 2020 at 9:25 AM
> > > > > > > > > > >>> > > > >> > > > Edward
> > > > > > > > > > >>> > > > >> > > > Strobach -
> > > > > > > > > > >>> > > > >> > > > NOAA
> > > > > > > > > > >>> > > > >> > > > Affiliate via
> > > > > > > > > > >>> > > > >> RT
> > > > > > > > > > >>> > > > >> > <
> > > > > > > > > > >>> > > > >> > > > met_help at ucar.edu> wrote:
> > > > > > > > > > >>> > > > >> > > >
> > > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > > >>> > > > >> > > > > Tue Dec 01 09:25:26 2020:
Request
> > > > > > > > > > >>> > > > >> > > > > 97687
> > > > > > > > > > >>> > > > >> > > > > was
> > > > > > > acted
> > > > > > > > > > >>> > > > >> > > > > upon.
> > > > > > > > > > >>> > > > >> > > > > Transaction: Ticket created
by
> > > > > > > > > > >>> > > > >> > > > > edward.strobach at noaa.gov
> > > > > > > > > > >>> > > > >> > > > > Queue: met_help
> > > > > > > > > > >>> > > > >> > > > > Subject: can't copy
plots
> > > > > > > > > > >>> > > > >> > > > > over
> > > > > > > > > > >>> > > > >> > > > > Owner: Nobody
> > > > > > > > > > >>> > > > >> > > > > Requestors:
> > > > > > > > > > >>> > > > >> > > > > edward.strobach at noaa.gov
> > > > > > > > > > >>> > > > >> > > > > Status: new
> > > > > > > > > > >>> > > > >> > > > > Ticket <URL:
> > > > > > > > > > >>> > > > >> >
> > > > > > > https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=97687
> > > > > > > > > > >>> > > > >> > > >
> > > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > > >>> > > > >> > > > > I'm trying to copy plots over
to
> > > > > > > > > > >>> > > > >> > > > > Dell
> > > > > > > > > > >>> > > > >> > > > > from the
> > > > > > > MET
> > > > > > > > > > >>> > > > >> > > > > server.
> > > > > > > > > > >>> > > > >> > > > > I
> > > > > > > > > > >>> > > > >> would
> > > > > > > > > > >>> > > > >> > > like
> > > > > > > > > > >>> > > > >> > > > to
> > > > > > > > > > >>> > > > >> > > > > store my plots in the usual
place
> > > > > > > > > > >>> > > > >> > > > > but
> > > > > > > > > > >>> > > > >> > > > > there is
> > > > > > > an
> > > > > > > > > > >>> > > > >> > > > > scp
> > > > > > > > > > >>> > > > >> > > > > error
> > > > > > > > > > >>> > > > >> > > > > which
> > > > > > > > > > >>> > > > >> I'm
> > > > > > > > > > >>> > > > >> > > > > attaching below.
> > > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > > >>> > > > >> > > > > AND BINARY h.fcst_var =
'PMTF'
> > > > > > > > > > >>> > > > >> > > > > AND ld.stat_header_id =
> > > > > > > > > > >>> > > > >> > > > > h.stat_header_id;
> > > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > > >>> > > > >> > > > > Connection to 205.156.8.85
closed
> > > > > > > > > > >>> > > > >> > > > > by
> > > > > > > > > > >>> > > > >> > > > > remote
> > > > > > > host.
> > > > > > > > > > >>> > > > >> > > > > CALLING: scp
> > > > > > > > > > >>> > > > >> > > > > edward.strobach at 205.156.8.85
> > > > > > > > > > >>> > > > >> > > > >
> ://data/mv_data//edward.strobach/plots/*
> > > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > > >>> > > > >> > > >
> > > > > > > > > > >>> > > > >> > >
> > > > > > > > > > >>> > > > >> >
> > > > > > > > > > >>> > > > >>
> > > > > > > > > > >>> > >
> > > > > > > > > > >>>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > > >>> > > > >> > > > > You are accessing a U.S.
> > > > > > > > > > >>> > > > >> > > > > Government
> > > > > > information
> > > > > > > > > > >>> > > > >> > > > > system, which includes: 1)
this
> > > > > > > > > > >>> > > > >> > > > > computer, 2)
> > > > > > > this
> > > > > > > > > > >>> > > > >> > > > > computer network, 3) all
computers
> > > > > > > > > > >>> > > > >> > > > > connected
> > > > > > to
> > > > > > > > > > >>> > > > >> > > > > this
> > > > > > > > > > >>> > > > >> > > > > network, and 4) all devices
and
> > > > > > > > > > >>> > > > >> > > > > storage
> > > > > > > > > > >>> > > > >> > > > > media
> > > > > > > > > > >>> > > > >> > > > > attached
> > > > > > > > > > >>> > > > >> > > > > to this network or to a
computer
> > > > > > > > > > >>> > > > >> > > > > on
> > > > > > > > > > >>> > > > >> > > > > this
> > > > > > > network.
> > > > > > > > > > >>> > > > >> > > > > You
> > > > > > > > > > >>> > > > >> > > > > understand and consent to the
> > > > > > > > > > >>> > > > >> > > > > following:
> > > > > > > > > > >>> > > > >> > > > > you
> > > > > > may
> > > > > > > > > > >>> > > > >> > > > > access this information
system for
> > > > > > > > > > >>> > > > >> > > > > authorized
> > > > > > > use
> > > > > > > > > > >>> > > > >> > > > > only;
> > > > > > > > > > >>> > > > >> > > > > you have no reasonable
expectation
> > > > > > > > > > >>> > > > >> > > > > of
> > > > > > > > > > >>> > > > >> > > > > privacy
> > > > > > > > > > >>> > > > >> > > > > regarding
> > > > > > > > > > >>> > > > >> > > > > any communication of data
> > > > > > > > > > >>> > > > >> > > > > transiting
> > > > > > > > > > >>> > > > >> > > > > or
> > > > > > > > > > >>> > > > >> > > > > stored
> > > > > > > on
> > > > > > > > > > >>> > > > >> > > > > this
> > > > > > > > > > >>> > > > >> > > > > information system; at any
time
> > > > > > > > > > >>> > > > >> > > > > and
> > > > > > > > > > >>> > > > >> > > > > for
> > > > > > > > > > >>> > > > >> > > > > any
> > > > > > > lawful
> > > > > > > > > > >>> > > > >> > > > > Government purpose, the
Government
> > > > > > > > > > >>> > > > >> > > > > may
> > > > > > monitor,
> > > > > > > > > > >>> > > > >> > > > > intercept, and search and
seize
> > > > > > > > > > >>> > > > >> > > > > any
> > > > > > > communication
> > > > > > > > > > >>> > > > >> > > > > or
> > > > > > > > > > >>> > > > >> > > > > data transiting or stored on
this
> > > > > > > > > > >>> > > > >> > > > > information
> > > > > > > > > > >>> > > > >> > > > > system;
> > > > > > > > > > >>> > > > >> > > > > and
> > > > > > > > > > >>> > > > >> > > > > any communications or data
> > > > > > > > > > >>> > > > >> > > > > transiting
> > > > > > > > > > >>> > > > >> > > > > or
> > > > > > stored
> > > > > > > on
> > > > > > > > > > >>> > > > >> > > > > this
> > > > > > > > > > >>> > > > >> > > > > information system may be
> > > > > > > > > > >>> > > > >> > > > > disclosed or
> > > > > > > > > > >>> > > > >> > > > > used
> > > > > > for
> > > > > > > > > > >>> > > > >> > > > > any
> > > > > > > > > > >>> > > > >> > > > > lawful Government purpose.
> > > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > > >>> > > > >> > > > > scp:
> > > > > > > > > > >>> > > > >> > > > >
> //data/mv_data//edward.strobach/plots/*:
> > > > > > No
> > > > > > > > > > >>> > > > >> > > > > such
> > > > > > > > > > >>> > > > >> > > > > file
> > > > > > > > > > >>> > > > >> > > > > or
> > > > > > > > > > >>> > > > >> > directory
> > > > > > > > > > >>> > > > >> > > > > ERROR: Command returned with
non-
> > > > > > > > > > >>> > > > >> > > > > zero
> > > > > > > > > > >>> > > > >> > > > > status
> > > > > > > (1):
> > > > > > > > > > >>> > > > >> > > > > scp
> > > > > > > > > > >>> > > > >> > edward.strobach@
> > > > > > > > > > >>> > > > >> > > > > 205.156.8.85
> > > > > > > > > ://data/mv_data//edward.strobach/plots/*
> > > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > > >>> > > > >> > > >
> > > > > > > > > > >>> > > > >> > >
> > > > > > > > > > >>> > > > >> >
> > > > > > > > > > >>> > > > >>
> > > > > > > > > > >>> > >
> > > > > > > > > > >>>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > >
>
/gpfs/dell2/emc/modeling/noscrub/Edward.Strobach/metplus_aq/plots/MET/202009/Time_Series/PMTF/fcst_lead/FBAR_OBAR/Aggregate
> > > > > > > > > > >>> > > > >> > > > > ++ printf
'\033]0;%s@%s:%s\007'
> > > > > > Edward.Strobach
> > > > > > > > > > >>> > > > >> > > > > v71a1
> > > > > > > > > > >>> > > > >> > > > >
> > > > > > > > >
/gpfs/dell2/emc/modeling/save/Edward.Strobach/Met_Viewer
> > > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > > >>> > > > >> > > > > NOTE: nothing about this
set-up
> > > > > > > > > > >>> > > > >> > > > > has
> > > > > > > > > > >>> > > > >> > > > > changed
> > > > > > > other
> > > > > > > > > > >>> > > > >> > > > > than
> > > > > > > > > > >>> > > > >> > > > > the
> > > > > > > > > > >>> > > > >> directory
> > > > > > > > > > >>> > > > >> > > > path
> > > > > > > > > > >>> > > > >> > > > > onto dell. The path has been
> > > > > > > > > > >>> > > > >> > > > > created
> > > > > > > > > > >>> > > > >> > > > > so
> > > > > > > > > > >>> > > > >> > > > > it
> > > > > > > should
> > > > > > > > > > >>> > > > >> > > > > support
> > > > > > > > > > >>> > > > >> > > > > the
> > > > > > > > > > >>> > > > >> > transfer
> > > > > > > > > > >>> > > > >> > > > of
> > > > > > > > > > >>> > > > >> > > > > plots generated on the MET
server.
> > > > > > > > > > >>> > > > >> > > > > The
> > > > > > > preceding
> > > > > > > > > > >>> > > > >> > > > > information
> > > > > > > > > > >>> > > > >> > indicates
> > > > > > > > > > >>> > > > >> > > > > that the plot was processed
> > > > > > > > > > >>> > > > >> > > > > successfully.
> > > > > > It's
> > > > > > > > > > >>> > > > >> > > > > not
> > > > > > > > > > >>> > > > >> > > > > clear
> > > > > > > > > > >>> > > > >> > > > > why I
> > > > > > > > > > >>> > > > >> can't
> > > > > > > > > > >>> > > > >> > > > > transfer my plots over.
> > > > > > > > > > >>> > > > >> > > > > --
> > > > > > > > > > >>> > > > >> > > > > Edward Strobach
> > > > > > > > > > >>> > > > >> > > > > EMC/NCEP/NWS/
> > > > > > > > > > >>> > > > >> > > > > IMSG Contractor
> > > > > > > > > > >>> > > > >> > > > > Cubicle#: 2029
> > > > > > > > > > >>> > > > >> > > > > 301-683-3717
> > > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > > >>> > > > >> > > > >
> > > > > > > > > > >>> > > > >> > > >
> > > > > > > > > > >>> > > > >> > > > --
> > > > > > > > > > >>> > > > >> > > > Julie Prestopnik (she/her)
> > > > > > > > > > >>> > > > >> > > > Software Engineer
> > > > > > > > > > >>> > > > >> > > > National Center for Atmospheric
> > > > > > > > > > >>> > > > >> > > > Research
> > > > > > > > > > >>> > > > >> > > > Research Applications
Laboratory
> > > > > > > > > > >>> > > > >> > > > Email: jpresto at ucar.edu
> > > > > > > > > > >>> > > > >> > > >
> > > > > > > > > > >>> > > > >> > > > My working day may not be your
> > > > > > > > > > >>> > > > >> > > > working
> > > > > > > > > > >>> > > > >> > > > day.
> > > > > > > Please
> > > > > > > > > > >>> > > > >> > > > do not
> > > > > > > > > > >>> > > > >> > > > feel
> > > > > > > > > > >>> > > > >> obliged
> > > > > > > > > > >>> > > > >> > > to
> > > > > > > > > > >>> > > > >> > > > reply to this email outside of
your
> > > > > > > > > > >>> > > > >> > > > normal
> > > > > > working
> > > > > > > > > > >>> > > > >> > > > hours.
> > > > > > > > > > >>> > > > >> > > >
> > > > > > > > > > >>> > > > >> > > >
> > > > > > > > > > >>> > > > >> > >
> > > > > > > > > > >>> > > > >> > > --
> > > > > > > > > > >>> > > > >> > > Edward Strobach
> > > > > > > > > > >>> > > > >> > > EMC/NCEP/NWS/
> > > > > > > > > > >>> > > > >> > > IMSG Contractor
> > > > > > > > > > >>> > > > >> > > Cubicle#: 2029
> > > > > > > > > > >>> > > > >> > > 301-683-3717
> > > > > > > > > > >>> > > > >> > >
> > > > > > > > > > >>> > > > >> > >
> > > > > > > > > > >>> > > > >> >
> > > > > > > > > > >>> > > > >> > --
> > > > > > > > > > >>> > > > >> > Julie Prestopnik (she/her)
> > > > > > > > > > >>> > > > >> > Software Engineer
> > > > > > > > > > >>> > > > >> > National Center for Atmospheric
Research
> > > > > > > > > > >>> > > > >> > Research Applications Laboratory
> > > > > > > > > > >>> > > > >> > Email: jpresto at ucar.edu
> > > > > > > > > > >>> > > > >> >
> > > > > > > > > > >>> > > > >> > My working day may not be your
working
> > > > > > > > > > >>> > > > >> > day.
> > > > > > > > > > >>> > > > >> > Please
> > > > > > do
> > > > > > > > > > >>> > > > >> > not
> > > > > > > > > > >>> > > > >> > feel
> > > > > > > > > > >>> > > > >> > obliged
> > > > > > > > > > >>> > > > >> to
> > > > > > > > > > >>> > > > >> > reply to this email outside of your
> > > > > > > > > > >>> > > > >> > normal
> > > > > > > > > > >>> > > > >> > working
> > > > > > > > > > >>> > > > >> > hours.
> > > > > > > > > > >>> > > > >> >
> > > > > > > > > > >>> > > > >> >
> > > > > > > > > > >>> > > > >>
> > > > > > > > > > >>> > > > >> --
> > > > > > > > > > >>> > > > >> Edward Strobach
> > > > > > > > > > >>> > > > >> EMC/NCEP/NWS/
> > > > > > > > > > >>> > > > >> IMSG Contractor
> > > > > > > > > > >>> > > > >> Cubicle#: 2029
> > > > > > > > > > >>> > > > >> 301-683-3717
> > > > > > > > > > >>> > > > >>
> > > > > > > > > > >>> > > > >>
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > --
> > > > > > > > > > >>> > > > > Julie Prestopnik (she/her)
> > > > > > > > > > >>> > > > > Software Engineer
> > > > > > > > > > >>> > > > > National Center for Atmospheric
Research
> > > > > > > > > > >>> > > > > Research Applications Laboratory
> > > > > > > > > > >>> > > > > Email: jpresto at ucar.edu
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > > > > My working day may not be your working
day.
> > > > > > > > > > >>> > > > > Please do
> > > > > > > not
> > > > > > > > > > >>> > > > > feel
> > > > > > > > > > >>> > > > > obliged to
> > > > > > > > > > >>> > > > > reply to this email outside of your
normal
> > > > > > > > > > >>> > > > > working
> > > > > > hours.
> > > > > > > > > > >>> > > > >
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > >
> > > > > > > > > > >>> > >
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>
> > > > > > > > > > >> --
> > > > > > > > > > >> Edward Strobach
> > > > > > > > > > >> EMC/NCEP/NWS/
> > > > > > > > > > >> IMSG Contractor
> > > > > > > > > > >> Cubicle#: 2029
> > > > > > > > > > >> 301-683-3717
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Edward Strobach
> > > > > > > > > > > EMC/NCEP/NWS/
> > > > > > > > > > > IMSG Contractor
> > > > > > > > > > > Cubicle#: 2029
> > > > > > > > > > > 301-683-3717
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Edward Strobach
> > > > > > > > EMC/NCEP/NWS/
> > > > > > > > IMSG Contractor
> > > > > > > > Cubicle#: 2029
> > > > > > > > 301-683-3717
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Julie Prestopnik (she/her)
> > > > > > > Software Engineer
> > > > > > > National Center for Atmospheric Research
> > > > > > > Research Applications Laboratory
> > > > > > > Email: jpresto at ucar.edu
> > > > > > >
> > > > > > > My working day may not be your working day. Please do
not
> > > > > > > feel
> > > > > > > obliged
> > > > > > to
> > > > > > > reply to this email outside of your normal working
hours.
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Edward Strobach
> > > > > > EMC/NCEP/NWS/
> > > > > > IMSG Contractor
> > > > > > Cubicle#: 2029
> > > > > > 301-683-3717
> > > > > >
> > > > > >
> > >
> > >
> > >
> > >
>
>
>
>
--
Edward Strobach
EMC/NCEP/NWS/
IMSG Contractor
Cubicle#: 2029
301-683-3717
------------------------------------------------
More information about the Met_help
mailing list