[Met_help] [rt.rap.ucar.edu #80149] History for stuck mv_load job

[John Halley Gotway via RT]

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hello.  I've been running /usr1/metviewer/metviewer/bin/mv_load.sh to 
add 7-days' worth of VSDB files to the database mv_ylin_pcp.  The job 
has been running for nearly 26 hours now (since 17:54Z yesterday) and 
it's likely 'stuck' - the output file 
/metviewer/staging/wd22yl/out.load/loadlog.20170328-20170403 seems to 
have had no new entries for the last ~20 hours.  Should I just kill the 
job?  How would it affect the database if a mv_load job is killed before 
it's complete?

In the past, loading additional VSDBs to an existing database of this 
size would take 1-2 hours (no strong dependence on the amount of new 
data - 2 days to 2 weeks).

 From /metviewer/staging/wd22yl/, I'm running

     /usr1/metviewer/metviewer/bin/mv_load.sh load_test.xml_ying > 
out.load/loadlog.20170328-20170403

The last lines of the loadlog are:

     ==== indexes ====

              stat_header_model_idx: 0:00:00.927
           stat_header_fcst_var_idx: 0:00:00.882
           stat_header_fcst_lev_idx: 0:00:00.958
             stat_header_obtype_idx: 0:00:00.947
            stat_header_vx_mask_idx: 0:00:00.924
        stat_header_interp_mthd_idx: 0:00:00.948
        stat_header_interp_pnts_idx: 0:00:00.976
        stat_header_fcst_thresh_idx: 0:00:00.982
              mode_header_model_idx: 0:00:00.007
          mode_header_fcst_lead_idx: 0:00:00.007
         mode_header_fcst_valid_idx: 0:00:00.006
          mode_header_fcst_init_idx: 0:00:00.007
           mode_header_fcst_rad_idx: 0:00:00.007
           mode_header_fcst_thr_idx: 0:00:00.005
           mode_header_fcst_var_idx: 0:00:00.005
           mode_header_fcst_lev_idx: 0:00:00.007
        line_data_fho_fcst_lead_idx: 0:00:00.028
   line_data_fho_fcst_valid_beg_idx: 0:00:00.013
    line_data_fho_fcst_init_beg_idx: 0:00:00.006

Thanks,

Ying

-- 
Ying Lin
NCEP/EMC/Mesoscale Modeling Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov




----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: stuck mv_load job
From: Tatiana Burek
Time: Thu Apr 13 15:03:56 2017

Ying,

Looks like your data loading is finished successfully.
The log messages indicated that  1385277 lines were inserted.
I do see that the loading module is still on executing state. It is
safe to kill it now without deleting or messing any data in the
database.
I suspect that MySQL reached a space limit and is not excepting any
data at this point.
This might caused the loading execution to get 'stuck'.

Tatiana


On Thu Apr 13 13:45:12 2017, ying.lin at noaa.gov wrote:
> Hello.  I've been running /usr1/metviewer/metviewer/bin/mv_load.sh
to
> add 7-days' worth of VSDB files to the database mv_ylin_pcp.  The
job
> has been running for nearly 26 hours now (since 17:54Z yesterday)
and
> it's likely 'stuck' - the output file
> /metviewer/staging/wd22yl/out.load/loadlog.20170328-20170403 seems
to
> have had no new entries for the last ~20 hours.  Should I just kill
the
> job?  How would it affect the database if a mv_load job is killed
before
> it's complete?
>
> In the past, loading additional VSDBs to an existing database of
this
> size would take 1-2 hours (no strong dependence on the amount of new
> data - 2 days to 2 weeks).
>
>  From /metviewer/staging/wd22yl/, I'm running
>
>      /usr1/metviewer/metviewer/bin/mv_load.sh load_test.xml_ying >
> out.load/loadlog.20170328-20170403
>
> The last lines of the loadlog are:
>
>      ==== indexes ====
>
>               stat_header_model_idx: 0:00:00.927
>            stat_header_fcst_var_idx: 0:00:00.882
>            stat_header_fcst_lev_idx: 0:00:00.958
>              stat_header_obtype_idx: 0:00:00.947
>             stat_header_vx_mask_idx: 0:00:00.924
>         stat_header_interp_mthd_idx: 0:00:00.948
>         stat_header_interp_pnts_idx: 0:00:00.976
>         stat_header_fcst_thresh_idx: 0:00:00.982
>               mode_header_model_idx: 0:00:00.007
>           mode_header_fcst_lead_idx: 0:00:00.007
>          mode_header_fcst_valid_idx: 0:00:00.006
>           mode_header_fcst_init_idx: 0:00:00.007
>            mode_header_fcst_rad_idx: 0:00:00.007
>            mode_header_fcst_thr_idx: 0:00:00.005
>            mode_header_fcst_var_idx: 0:00:00.005
>            mode_header_fcst_lev_idx: 0:00:00.007
>         line_data_fho_fcst_lead_idx: 0:00:00.028
>    line_data_fho_fcst_valid_beg_idx: 0:00:00.013
>     line_data_fho_fcst_init_beg_idx: 0:00:00.006
>
> Thanks,
>
> Ying
>



------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #80149] stuck mv_load job
From: Ying Lin
Time: Thu Apr 13 15:20:47 2017

Hi Tatiana,

     Thank you very much for looking into this inquiry and for the
quick
reply!  I've killed the job.

Ying

On 04/13/2017 05:03 PM, Tatiana Burek via RT wrote:
> Ying,
>
> Looks like your data loading is finished successfully.
> The log messages indicated that  1385277 lines were inserted.
> I do see that the loading module is still on executing state. It is
safe to kill it now without deleting or messing any data in the
database.
> I suspect that MySQL reached a space limit and is not excepting any
data at this point.
> This might caused the loading execution to get 'stuck'.
>
> Tatiana
>
>
> On Thu Apr 13 13:45:12 2017, ying.lin at noaa.gov wrote:
>> Hello.  I've been running /usr1/metviewer/metviewer/bin/mv_load.sh
to
>> add 7-days' worth of VSDB files to the database mv_ylin_pcp.  The
job
>> has been running for nearly 26 hours now (since 17:54Z yesterday)
and
>> it's likely 'stuck' - the output file
>> /metviewer/staging/wd22yl/out.load/loadlog.20170328-20170403 seems
to
>> have had no new entries for the last ~20 hours.  Should I just kill
the
>> job?  How would it affect the database if a mv_load job is killed
before
>> it's complete?
>>
>> In the past, loading additional VSDBs to an existing database of
this
>> size would take 1-2 hours (no strong dependence on the amount of
new
>> data - 2 days to 2 weeks).
>>
>>   From /metviewer/staging/wd22yl/, I'm running
>>
>>       /usr1/metviewer/metviewer/bin/mv_load.sh load_test.xml_ying >
>> out.load/loadlog.20170328-20170403
>>
>> The last lines of the loadlog are:
>>
>>       ==== indexes ====
>>
>>                stat_header_model_idx: 0:00:00.927
>>             stat_header_fcst_var_idx: 0:00:00.882
>>             stat_header_fcst_lev_idx: 0:00:00.958
>>               stat_header_obtype_idx: 0:00:00.947
>>              stat_header_vx_mask_idx: 0:00:00.924
>>          stat_header_interp_mthd_idx: 0:00:00.948
>>          stat_header_interp_pnts_idx: 0:00:00.976
>>          stat_header_fcst_thresh_idx: 0:00:00.982
>>                mode_header_model_idx: 0:00:00.007
>>            mode_header_fcst_lead_idx: 0:00:00.007
>>           mode_header_fcst_valid_idx: 0:00:00.006
>>            mode_header_fcst_init_idx: 0:00:00.007
>>             mode_header_fcst_rad_idx: 0:00:00.007
>>             mode_header_fcst_thr_idx: 0:00:00.005
>>             mode_header_fcst_var_idx: 0:00:00.005
>>             mode_header_fcst_lev_idx: 0:00:00.007
>>          line_data_fho_fcst_lead_idx: 0:00:00.028
>>     line_data_fho_fcst_valid_beg_idx: 0:00:00.013
>>      line_data_fho_fcst_init_beg_idx: 0:00:00.006
>>
>> Thanks,
>>
>> Ying
>>
>
>


--
Ying Lin
NCEP/EMC/Mesoscale Modeling Branch
NCWCP Cubicle No. 2015
Ying.Lin at noaa.gov



------------------------------------------------