[ncl-talk] readAsciiTable not thread safe

Thu Mar 12 04:59:34 MDT 2015

Kyle,

Thanks for this very helpful solution. I implemented and tested your suggestion and it works like a charm! Additionally I was made aware of the unix utility “mktemp” (default available on most linux distributions) which also is intended to generate unique file names and which could also be a good solution.

Best regards
Daniel

From: windrunnerxc at gmail.com [mailto:windrunnerxc at gmail.com] On Behalf Of Kyle Griffin
Sent: Dienstag, 10. März 2015 16:38
To: Leuenberger Daniel
Cc: ncl-talk
Subject: Re: [ncl-talk] readAsciiTable not thread safe

Hi Daniel,

That's an excellent point. It looks like the function is designed to try and call a random file name with the 'echo tmp$$' call, but that returns a single number for each shell. I assume if multiple sub-processes are being run from a single shell, this would cause the files to overlap.

However, since this function is in the contributed.ncl file, you can change it on your system as a temporary fix. Although NCL does have a rand() function, it is always seeded with the same numbers upon initialization of a new NCL process and therefore can be expected to produce predictable numbers with each call. You could try to use srand(x) or random_setallseed(x,y) with some component of the time, but for the sake of producing consistently unique numbers, I would recommend using a component of the date as the actual random number.

I've used the nanosecond return from the UNIX date function, systemfunc("date +%N") in NCL, to get the job done. Producing a very large number of random numbers a day and up to five simultaneously and I have yet to have any issues with duplicate numbers. If you are using NCL on Mac OS X, you can get the same result by installing the GNU-compatible version of date, typically referenced as "gdate" in MacPort or Homebrew or similar UNIX-ish package manager.

Anyway, the fix you'd be looking for is below. It's not particularly robust, but it should fix your problem while the developers might look to address this in the future. This can be used in place of the line

tmpfile = systemfunc("echo tmp$$")

near the end of the readAsciiTable function. This is line 9062 in $NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl<https://www.ncl.ucar.edu/Document/Functions/Contributed/contrib.shtml> if you're using version 6.2.1 and have write permissions in the directory where NCL is installed. If not, find the readAsciiTable function in the contributed.ncl file, copy it into a new file and make the changes. Then you can "load" the new file at the top of your script after your loading of the contributed.ncl file.

      tmpnum = systemfunc("date +%N")
      if(tmpnum.eq."N")then
        delete(tmpnum)
        srand(toint(systemfunc("date +%s")))
        tmpnum = rand()
      end if
      tmpfile = "tmp"+tmpnum

Hope this has made sense. If not, feel free to reply back and include the list with questions. I'm sure the developers will look at this a bit closer in the future, as making functions parallel-safe appears to be one of the things they are working on alongside the ongoing development.

Kyle

----------------------------------------
Kyle S. Griffin
Department of Atmospheric and Oceanic Sciences
University of Wisconsin - Madison
Room 1421
1225 W Dayton St, Madison, WI 53706
Email: ksgriffin2 at wisc.edu<mailto:ksgriffin2 at wisc.edu>

On Tue, Mar 10, 2015 at 9:46 AM, <Daniel.Leuenberger at meteoswiss.ch<mailto:Daniel.Leuenberger at meteoswiss.ch>> wrote:
Dear NCL Team,

When running a large number of parallel NCL jobs using the readAsciiTable function (contributed.ncl) I realized that the function is not thread safe, e.g. may fail. The reason is that it writes a temporary ascii file and deletes it again at the end of the function. If now two parallel jobs want to write to the temporary file at exactly the same time, one is not allowed and will crash. One method to circumvent the problem would be to use a random number and/or the process ID in the file name of the temporary file.

Best regards

Daniel Leuenberger

Federal Department of Home Affairs FDHA
Federal Office of Meteorology and Climatology MeteoSwiss

All about weather and climate, please visit our new website
www.meteoswiss.ch<http://www.meteoschweiz.admin.ch/web/en.html> and MeteoSwiss App<http://www.meteoswiss.admin.ch/home/services-and-publications/beratung-und-service/meteoswiss-app.html>

_______________________________________________
ncl-talk mailing list
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150312/413b67ea/attachment.html 