[ncl-talk] readAsciiTable not thread safe

Thu Mar 12 13:38:47 MDT 2015

FYI: The code suggested by KG has been included with 'readAsciiTable'.
It will be in the 6.3.0 release

See 'Improvements':    https://www.ncl.ucar.edu/future_release.shtml

THX
D

On Thu, Mar 12, 2015 at 8:24 AM, Dennis Shea <shea at ucar.edu> wrote:

> I have opened a JIRA ticket on this. Will try for 6.3.0 but may not make
> it in for that release.
>
> NCL-2174:  readAsciiTable: make thread safe
>
> THX
> D
>
> On Thu, Mar 12, 2015 at 4:59 AM, <Daniel.Leuenberger at meteoswiss.ch> wrote:
>
>>  Kyle,
>>
>>
>>
>> Thanks for this very helpful solution. I implemented and tested your
>> suggestion and it works like a charm! Additionally I was made aware of the
>> unix utility “mktemp” (default available on most linux distributions) which
>> also is intended to generate unique file names and which could also be a
>> good solution.
>>
>>
>>
>> Best regards
>>
>> Daniel
>>
>>
>>
>> *From:* windrunnerxc at gmail.com [mailto:windrunnerxc at gmail.com] *On
>> Behalf Of *Kyle Griffin
>> *Sent:* Dienstag, 10. März 2015 16:38
>> *To:* Leuenberger Daniel
>> *Cc:* ncl-talk
>> *Subject:* Re: [ncl-talk] readAsciiTable not thread safe
>>
>>
>>
>> Hi Daniel,
>>
>>
>>
>> That's an excellent point. It looks like the function is designed to try
>> and call a random file name with the 'echo tmp$$' call, but that returns a
>> single number for each shell. I assume if multiple sub-processes are being
>> run from a single shell, this would cause the files to overlap.
>>
>>
>>
>> However, since this function is in the contributed.ncl file, you can
>> change it on your system as a temporary fix. Although NCL does have a
>> rand() function, it is always seeded with the same numbers upon
>> initialization of a new NCL process and therefore can be expected to
>> produce predictable numbers with each call. You could try to use srand(x)
>> or random_setallseed(x,y) with some component of the time, but for the sake
>> of producing consistently unique numbers, I would recommend using a
>> component of the date as the actual random number.
>>
>>
>>
>> I've used the nanosecond return from the UNIX date function,
>> systemfunc("date +%N") in NCL, to get the job done. Producing a very large
>> number of random numbers a day and up to five simultaneously and I have yet
>> to have any issues with duplicate numbers. If you are using NCL on Mac OS
>> X, you can get the same result by installing the GNU-compatible version of
>> date, typically referenced as "gdate" in MacPort or Homebrew or similar
>> UNIX-ish package manager.
>>
>>
>>
>> Anyway, the fix you'd be looking for is below. It's not particularly
>> robust, but it should fix your problem while the developers might look to
>> address this in the future. This can be used in place of the line
>>
>>
>>
>> tmpfile = systemfunc("echo tmp$$")
>>
>>
>>
>> near the end of the readAsciiTable function. This is line 9062 in
>> $NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl
>> <https://www.ncl.ucar.edu/Document/Functions/Contributed/contrib.shtml> if
>> you're using version 6.2.1 and have write permissions in the directory
>> where NCL is installed. If not, find the readAsciiTable function in the
>> contributed.ncl file, copy it into a new file and make the changes. Then
>> you can "load" the new file at the top of your script after your loading of
>> the contributed.ncl file.
>>
>>
>>
>>       tmpnum = systemfunc("date +%N")
>>
>>       if(tmpnum.eq."N")then
>>
>>         delete(tmpnum)
>>
>>         srand(toint(systemfunc("date +%s")))
>>
>>         tmpnum = rand()
>>
>>       end if
>>
>>       tmpfile = "tmp"+tmpnum
>>
>>
>>
>> Hope this has made sense. If not, feel free to reply back and include the
>> list with questions. I'm sure the developers will look at this a bit closer
>> in the future, as making functions parallel-safe appears to be one of the
>> things they are working on alongside the ongoing development.
>>
>>
>>
>> Kyle
>>
>>
>>   ----------------------------------------
>>
>> Kyle S. Griffin
>>
>> Department of Atmospheric and Oceanic Sciences
>>
>> University of Wisconsin - Madison
>>
>> Room 1421
>>
>> 1225 W Dayton St, Madison, WI 53706
>>
>> Email: ksgriffin2 at wisc.edu
>>
>>
>>
>> On Tue, Mar 10, 2015 at 9:46 AM, <Daniel.Leuenberger at meteoswiss.ch>
>> wrote:
>>
>> Dear NCL Team,
>>
>>
>>
>> When running a large number of parallel NCL jobs using the readAsciiTable
>> function (contributed.ncl) I realized that the function is not thread safe,
>> e.g. may fail. The reason is that it writes a temporary ascii file and
>> deletes it again at the end of the function. If now two parallel jobs want
>> to write to the temporary file at exactly the same time, one is not allowed
>> and will crash. One method to circumvent the problem would be to use a
>> random number and/or the process ID in the file name of the temporary file.
>>
>>
>>
>> Best regards
>>
>>
>>
>> *Daniel Leuenberger*
>>
>>
>>
>> Federal Department of Home Affairs FDHA
>>
>> *Federal Office of Meteorology and Climatology MeteoSwiss*
>>
>>
>>
>> All about weather and climate, please visit our new website
>>
>> www.meteoswiss.ch <http://www.meteoschweiz.admin.ch/web/en.html> and MeteoSwiss
>> App
>> <http://www.meteoswiss.admin.ch/home/services-and-publications/beratung-und-service/meteoswiss-app.html>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>>
>> _______________________________________________
>> ncl-talk mailing list
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150312/ce2743db/attachment.html 