[ncl-talk] readAsciiTable not thread safe

[Dennis Shea]

I have opened a JIRA ticket on this. Will try for 6.3.0 but may not make it
in for that release.

NCL-2174:  readAsciiTable: make thread safe

THX
D

On Thu, Mar 12, 2015 at 4:59 AM, <Daniel.Leuenberger at meteoswiss.ch> wrote:

>  Kyle,
>
>
>
> Thanks for this very helpful solution. I implemented and tested your
> suggestion and it works like a charm! Additionally I was made aware of the
> unix utility “mktemp” (default available on most linux distributions) which
> also is intended to generate unique file names and which could also be a
> good solution.
>
>
>
> Best regards
>
> Daniel
>
>
>
> *From:* windrunnerxc at gmail.com [mailto:windrunnerxc at gmail.com] *On Behalf
> Of *Kyle Griffin
> *Sent:* Dienstag, 10. März 2015 16:38
> *To:* Leuenberger Daniel
> *Cc:* ncl-talk
> *Subject:* Re: [ncl-talk] readAsciiTable not thread safe
>
>
>
> Hi Daniel,
>
>
>
> That's an excellent point. It looks like the function is designed to try
> and call a random file name with the 'echo tmp$$' call, but that returns a
> single number for each shell. I assume if multiple sub-processes are being
> run from a single shell, this would cause the files to overlap.
>
>
>
> However, since this function is in the contributed.ncl file, you can
> change it on your system as a temporary fix. Although NCL does have a
> rand() function, it is always seeded with the same numbers upon
> initialization of a new NCL process and therefore can be expected to
> produce predictable numbers with each call. You could try to use srand(x)
> or random_setallseed(x,y) with some component of the time, but for the sake
> of producing consistently unique numbers, I would recommend using a
> component of the date as the actual random number.
>
>
>
> I've used the nanosecond return from the UNIX date function,
> systemfunc("date +%N") in NCL, to get the job done. Producing a very large
> number of random numbers a day and up to five simultaneously and I have yet
> to have any issues with duplicate numbers. If you are using NCL on Mac OS
> X, you can get the same result by installing the GNU-compatible version of
> date, typically referenced as "gdate" in MacPort or Homebrew or similar
> UNIX-ish package manager.
>
>
>
> Anyway, the fix you'd be looking for is below. It's not particularly
> robust, but it should fix your problem while the developers might look to
> address this in the future. This can be used in place of the line
>
>
>
> tmpfile = systemfunc("echo tmp$$")
>
>
>
> near the end of the readAsciiTable function. This is line 9062 in
> $NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl
> <https://www.ncl.ucar.edu/Document/Functions/Contributed/contrib.shtml> if
> you're using version 6.2.1 and have write permissions in the directory
> where NCL is installed. If not, find the readAsciiTable function in the
> contributed.ncl file, copy it into a new file and make the changes. Then
> you can "load" the new file at the top of your script after your loading of
> the contributed.ncl file.
>
>
>
>       tmpnum = systemfunc("date +%N")
>
>       if(tmpnum.eq."N")then
>
>         delete(tmpnum)
>
>         srand(toint(systemfunc("date +%s")))
>
>         tmpnum = rand()
>
>       end if
>
>       tmpfile = "tmp"+tmpnum
>
>
>
> Hope this has made sense. If not, feel free to reply back and include the
> list with questions. I'm sure the developers will look at this a bit closer
> in the future, as making functions parallel-safe appears to be one of the
> things they are working on alongside the ongoing development.
>
>
>
> Kyle
>
>
>   ----------------------------------------
>
> Kyle S. Griffin
>
> Department of Atmospheric and Oceanic Sciences
>
> University of Wisconsin - Madison
>
> Room 1421
>
> 1225 W Dayton St, Madison, WI 53706
>
> Email: ksgriffin2 at wisc.edu
>
>
>
> On Tue, Mar 10, 2015 at 9:46 AM, <Daniel.Leuenberger at meteoswiss.ch> wrote:
>
> Dear NCL Team,
>
>
>
> When running a large number of parallel NCL jobs using the readAsciiTable
> function (contributed.ncl) I realized that the function is not thread safe,
> e.g. may fail. The reason is that it writes a temporary ascii file and
> deletes it again at the end of the function. If now two parallel jobs want
> to write to the temporary file at exactly the same time, one is not allowed
> and will crash. One method to circumvent the problem would be to use a
> random number and/or the process ID in the file name of the temporary file.
>
>
>
> Best regards
>
>
>
> *Daniel Leuenberger*
>
>
>
> Federal Department of Home Affairs FDHA
>
> *Federal Office of Meteorology and Climatology MeteoSwiss*
>
>
>
> All about weather and climate, please visit our new website
>
> www.meteoswiss.ch <http://www.meteoschweiz.admin.ch/web/en.html> and MeteoSwiss
> App
> <http://www.meteoswiss.admin.ch/home/services-and-publications/beratung-und-service/meteoswiss-app.html>
>
>
>
>
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150312/eeedfa45/attachment.html