[Met_help] [rt.rap.ucar.edu #38383] History for Problem to assign tmp dir using point_stat and stat_analysis

Wed Jun 9 10:12:38 MDT 2010

----------------------------------------------------------------
  Initial Request
----------------------------------------------------------------

Hi,

We run MET at our Linux cluster. At this cluster the /tmp is really small, but each node has a rather large /local disk.

Therefore I want point_stat and stat_analysis to use /local instead of /tmp.

This should be straight forward by either setting "tmp_dir = "/local";" in the point_stat config file or to give the option "-tmp_dir /local" to the stat_analysis GLI.

Only I still see *.stat files in the /tmp directory and nothing in the /local directory. Am I missing something?

Kind regards,

Daniël

------------------------------------------------------------------------

  Daniël van Dijke

  Meteorological Researcher  

  Meteo Consult BV, Wageningen, the Netherlands

  Tel: +31 (0)317 399874

  Email: vandijke at meteogroup.com

------------------------------------------------------------------------

-- 

This e-mail is from Meteo Consult B.V., a MeteoGroup company. For more information, see http://www.weer.nl/gebruiksvoorwaarden.

This e-mail may contain confidential information. Only the addressee is permitted to read, copy, distribute or otherwise use this e-mail or any attachments. If you have received it in error, please contact the sender immediately. Any opinion expressed in this e-mail is personal to the sender and may not reflect the opinion of MeteoGroup.

Any e-mail reply to this address may be subject to interception or monitoring for operational reasons or for lawful business practices.

----------------------------------------------------------------
  Complete Ticket History
----------------------------------------------------------------

Subject: Problem to assign tmp dir using point_stat and stat_analysis
From: John Halley Gotway
Time: Tue Jun 08 13:14:08 2010

Daniel,

I suspect that that "tmp_dir" option is actually working as expected,
and you're being confused by old files.  I double-checked it here to
make sure it works.

Point-Stat writes temp files when computing bootstrap confidence
intervals.  Once it is finished with it's computations, it deletes
those temp files.  However, if you kill Point-Stat while it's running,
it doesn't have a chance to delete those temp files.

I suspect that the temp files you're seeing in /tmp are old ones
hanging around from a previous Point-Stat run that was killed or
errored out.

Please try the following:
(1) Look at the timestamp on the files in /tmp:
   ls -la /tmp/tmp*.txt
(2) Remove those files:
   rm /tmp/tmp*.txt
(3) Make sure you have the tmp_dir set in the Point-Stat config file:
   tmp_dir = "/local";
(4) Rerun Point-Stat with bootstrap CI's turned on (i.e. n_boot_rep =
1000;).
(5) In another window, while Point-Stat is running, do "ls
/local/tmp*.txt" to check for the existence of those temp files.
You'll need to check a lot since Point-Stat will remove them once it's
done with them.

Please let me know if you continue to experience problems.

Thanks,
John Halley Gotway
johnhg at ucar.edu

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #38383] Problem to assign tmp dir using point_stat and stat_analysis 
From: Daniel van Dijke
Time: Tue Jun 08 13:25:42 2010

I double checked it. I've to say that point_stat is doing it
correctly.
Only stat_analysis still produces *.stat files in /tmp.
This is the command I run:

/cm/shared/apps/met/2.0/METv2.0/bin/stat_analysis -lookin
${maindir}/out/point_stat/%MODEL%_${year_2}${month_2}${day_2}_%ISSUE%
\
  -job aggregate_stat -line_type MPR -out_line_type CNT -tmp_dir
/local -vx_mask %AREA% -obs_var %FAMILY1% -fcst_thresh NA -obs_thresh
NA \
  -out
${maindir}/out/stat_analysis/${year_2}${month_2}${day_2}/%MODEL%_%AREA%_${year_2}${month_2}${day_2}_%ISSUE%_%FAMILY1%.2.txt
-v 2

Could you check if the -tmp_dir option is working correctly on your
sides as well?

Thanks,

Daniel

-----Oorspronkelijk bericht-----
Van: RAL HelpDesk {for John Halley Gotway} [mailto:met_help at ucar.edu]
Verzonden: dinsdag 8 juni 2010 21:14
Aan: Daniel van Dijke
Onderwerp: [rt.rap.ucar.edu #38383] Problem to assign tmp dir using
point_stat and stat_analysis

Daniel,

I suspect that that "tmp_dir" option is actually working as expected,
and you're being confused by old files.  I double-checked it here to
make sure it works.

Point-Stat writes temp files when computing bootstrap confidence
intervals.  Once it is finished with it's computations, it deletes
those temp files.  However, if you kill Point-Stat while it's running,
it doesn't have a chance to delete those temp files.

I suspect that the temp files you're seeing in /tmp are old ones
hanging around from a previous Point-Stat run that was killed or
errored out.

Please try the following:
(1) Look at the timestamp on the files in /tmp:
   ls -la /tmp/tmp*.txt
(2) Remove those files:
   rm /tmp/tmp*.txt
(3) Make sure you have the tmp_dir set in the Point-Stat config file:
   tmp_dir = "/local";
(4) Rerun Point-Stat with bootstrap CI's turned on (i.e. n_boot_rep =
1000;).
(5) In another window, while Point-Stat is running, do "ls
/local/tmp*.txt" to check for the existence of those temp files.
You'll need to check a lot since Point-Stat will remove them once it's
done with them.

Please let me know if you continue to experience problems.

Thanks,
John Halley Gotway
johnhg at ucar.edu

--

This e-mail is from Meteo Consult B.V., a MeteoGroup company. For more
information, see http://www.weer.nl/gebruiksvoorwaarden.

This e-mail may contain confidential information. Only the addressee
is permitted to read, copy, distribute or otherwise use this e-mail or
any attachments. If you have received it in error, please contact the
sender immediately. Any opinion expressed in this e-mail is personal
to the sender and may not reflect the opinion of MeteoGroup.

Any e-mail reply to this address may be subject to interception or
monitoring for operational reasons or for lawful business practices.

------------------------------------------------
Subject: Re: [rt.rap.ucar.edu #38383] Problem to assign tmp dir using point_stat and stat_analysis
From: John Halley Gotway
Time: Tue Jun 08 14:33:32 2010

Daniel,

Ah yes, I see the problem.  Thank you very much for pointing out this
bug.  Using "-tmp_dir" on the command line when defining a job will
NOT have the desired effect.  However, if you pass a
configuration file to STAT-Analysis and set the "tmp_dir" parameter in
the config file, it should work as expected.

For now, I'd suggest moving your job(s) into a config file and setting
the "tmp_dir" parameter.  However, we're working hard on the next
release of MET, METv3.0, and I'll fix this bug in that release.

Thanks again for catching this.  And please let me know if you
continue to have problems when using the STATAnalysisConfig file.

Thanks,
John Halley Gotway
met_help at ucar.edu

RAL HelpDesk {for Daniel van Dijke} wrote:
> <URL: https://rt.rap.ucar.edu/rt/Ticket/Display.html?id=38383 >
>
> I double checked it. I've to say that point_stat is doing it
correctly.
> Only stat_analysis still produces *.stat files in /tmp.
> This is the command I run:
>
> /cm/shared/apps/met/2.0/METv2.0/bin/stat_analysis -lookin
${maindir}/out/point_stat/%MODEL%_${year_2}${month_2}${day_2}_%ISSUE%
\
>   -job aggregate_stat -line_type MPR -out_line_type CNT -tmp_dir
/local -vx_mask %AREA% -obs_var %FAMILY1% -fcst_thresh NA -obs_thresh
NA \
>   -out
${maindir}/out/stat_analysis/${year_2}${month_2}${day_2}/%MODEL%_%AREA%_${year_2}${month_2}${day_2}_%ISSUE%_%FAMILY1%.2.txt
-v 2
>
> Could you check if the -tmp_dir option is working correctly on your
sides as well?
>
> Thanks,
>
> Daniel
>
>
> -----Oorspronkelijk bericht-----
> Van: RAL HelpDesk {for John Halley Gotway}
[mailto:met_help at ucar.edu]
> Verzonden: dinsdag 8 juni 2010 21:14
> Aan: Daniel van Dijke
> Onderwerp: [rt.rap.ucar.edu #38383] Problem to assign tmp dir using
point_stat and stat_analysis
>
> Daniel,
>
> I suspect that that "tmp_dir" option is actually working as
expected, and you're being confused by old files.  I double-checked it
here to make sure it works.
>
> Point-Stat writes temp files when computing bootstrap confidence
intervals.  Once it is finished with it's computations, it deletes
those temp files.  However, if you kill Point-Stat while it's running,
it doesn't have a chance to delete those temp files.
>
>
> I suspect that the temp files you're seeing in /tmp are old ones
hanging around from a previous Point-Stat run that was killed or
errored out.
>
> Please try the following:
> (1) Look at the timestamp on the files in /tmp:
>    ls -la /tmp/tmp*.txt
> (2) Remove those files:
>    rm /tmp/tmp*.txt
> (3) Make sure you have the tmp_dir set in the Point-Stat config
file:
>    tmp_dir = "/local";
> (4) Rerun Point-Stat with bootstrap CI's turned on (i.e. n_boot_rep
= 1000;).
> (5) In another window, while Point-Stat is running, do "ls
/local/tmp*.txt" to check for the existence of those temp files.
You'll need to check a lot since Point-Stat will remove them once it's
done with them.
>
> Please let me know if you continue to experience problems.
>
> Thanks,
> John Halley Gotway
> johnhg at ucar.edu
>
>
>

------------------------------------------------
Subject: RE: [rt.rap.ucar.edu #38383] Problem to assign tmp dir using point_stat and stat_analysis
From: Daniel van Dijke
Time: Wed Jun 09 02:06:07 2010

John,

It Works, thanks for your input!

Cheers,

Daniël

-----Oorspronkelijk bericht-----
Van: RAL HelpDesk {for John Halley Gotway} [mailto:met_help at ucar.edu]
Verzonden: dinsdag 8 juni 2010 22:34
Aan: Daniel van Dijke
Onderwerp: Re: [rt.rap.ucar.edu #38383] Problem to assign tmp dir
using point_stat and stat_analysis

Daniel,