[ncl-talk] readAsciiTable not thread safe

[Kyle Griffin]

Hi Daniel,

That's an excellent point. It looks like the function is designed to try
and call a random file name with the 'echo tmp$$' call, but that returns a
single number for each shell. I assume if multiple sub-processes are being
run from a single shell, this would cause the files to overlap.

However, since this function is in the contributed.ncl file, you can change
it on your system as a temporary fix. Although NCL does have a rand()
function, it is always seeded with the same numbers upon initialization of
a new NCL process and therefore can be expected to produce predictable
numbers with each call. You could try to use srand(x) or
random_setallseed(x,y) with some component of the time, but for the sake of
producing consistently unique numbers, I would recommend using a component
of the date as the actual random number.

I've used the nanosecond return from the UNIX date function,
systemfunc("date +%N") in NCL, to get the job done. Producing a very large
number of random numbers a day and up to five simultaneously and I have yet
to have any issues with duplicate numbers. If you are using NCL on Mac OS
X, you can get the same result by installing the GNU-compatible version of
date, typically referenced as "gdate" in MacPort or Homebrew or similar
UNIX-ish package manager.

Anyway, the fix you'd be looking for is below. It's not particularly
robust, but it should fix your problem while the developers might look to
address this in the future. This can be used in place of the line

tmpfile = systemfunc("echo tmp$$")

near the end of the readAsciiTable function. This is line 9062 in
$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl
<https://www.ncl.ucar.edu/Document/Functions/Contributed/contrib.shtml> if
you're using version 6.2.1 and have write permissions in the directory
where NCL is installed. If not, find the readAsciiTable function in the
contributed.ncl file, copy it into a new file and make the changes. Then
you can "load" the new file at the top of your script after your loading of
the contributed.ncl file.

      tmpnum = systemfunc("date +%N")

      if(tmpnum.eq."N")then

        delete(tmpnum)

        srand(toint(systemfunc("date +%s")))

        tmpnum = rand()

      end if

      tmpfile = "tmp"+tmpnum


Hope this has made sense. If not, feel free to reply back and include the
list with questions. I'm sure the developers will look at this a bit closer
in the future, as making functions parallel-safe appears to be one of the
things they are working on alongside the ongoing development.


Kyle

----------------------------------------
Kyle S. Griffin
Department of Atmospheric and Oceanic Sciences
University of Wisconsin - Madison
Room 1421
1225 W Dayton St, Madison, WI 53706
Email: ksgriffin2 at wisc.edu

On Tue, Mar 10, 2015 at 9:46 AM, <Daniel.Leuenberger at meteoswiss.ch> wrote:

>  Dear NCL Team,
>
>
>
> When running a large number of parallel NCL jobs using the readAsciiTable
> function (contributed.ncl) I realized that the function is not thread safe,
> e.g. may fail. The reason is that it writes a temporary ascii file and
> deletes it again at the end of the function. If now two parallel jobs want
> to write to the temporary file at exactly the same time, one is not allowed
> and will crash. One method to circumvent the problem would be to use a
> random number and/or the process ID in the file name of the temporary file.
>
>
>
> Best regards
>
>
>
> *Daniel Leuenberger*
>
>
>
> Federal Department of Home Affairs FDHA
>
> *Federal Office of Meteorology and Climatology MeteoSwiss*
>
>
>
> All about weather and climate, please visit our new website
>
> www.meteoswiss.ch <http://www.meteoschweiz.admin.ch/web/en.html> and MeteoSwiss
> App
> <http://www.meteoswiss.admin.ch/home/services-and-publications/beratung-und-service/meteoswiss-app.html>
>
>
>
>
>
> _______________________________________________
> ncl-talk mailing list
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150310/38c5bd08/attachment.html