[ncl-talk] Fwd: Error: wrf_tk Unable to allocate memory
Berny Chaimite
bchaimite at gmail.com
Thu May 17 16:07:55 MDT 2018
*-----*
----------------------------------------
Berny A. Chaimite
----------------------------------------
Atmospheric and Climate Sciences Student
Department of Physics
Eduardo Mondlane University
*P. Contact*:
Cell: +258 828 957 675(Whatsapp)/846041931/870434233
Tell: +258 2145 6389
Institutional: *berny.chaimite at uem.ac.mz <berny.chaimite at uem.ac.mz>*
Maputo, Mozambique
---------- Forwarded message ---------
From: Berny Chaimite <bchaimite at gmail.com>
Date: segunda, 14/05/2018 à(s) 07:05
Subject: Re: [ncl-talk] Error: wrf_tk Unable to allocate memory
To: <hallock at ucar.edu>
Good Morning!
Sorry for the delay on reply this mail. I was see if there would be any
other way out of problem.
1) Files are not the same size. the files are like this (see attached).
2) For a smaller domain (d02) it showed the graph successfully (using this
script wrf_tc_d02.ncl, see attached). So my goal is to try to do the same
for d01. The idea is to read and graph all of these output files. Reading
the data in steps will not be useful to discuss my results. I do not know
if I'm clear?
If you need more clarification and details you can ask me please.
Help me,
Best regards,
BA Chaimite
*-----*
----------------------------------------
Berny A. Chaimite
----------------------------------------
Atmospheric and Climate Sciences Student
Department of Physics
Eduardo Mondlane University
*P. Contact*:
Cell: +258 828 957 675(Whatsapp)/846041931/870434233
Tell: +258 2145 6389
Institutional: *berny.chaimite at uem.ac.mz <berny.chaimite at uem.ac.mz>*
Maputo, Mozambique
Kevin Hallock <hallock at ucar.edu> escreveu no dia sexta, 11/05/2018 à(s)
01:49:
> Hi BA,
>
> Thanks for the screenshot.
>
> Based on that information, it does seem like you might be running out of
> memory on your system. In particular, the output below shows 308304800
> bytes (29 * 137 * 194 * 100 time steps * 4 byte float ≈ 300 MB), but that
> should actually be multiplied by the number of files (308304800 * 53 =
> 16,340,154,400 ≈ 16 GB), assuming they’re all the same size. To add to the
> issue, that’s just the raw amount of memory needed to store the output
> array — unfortunately, in order to calculate “tc” from WRF data, NCL has to
> extract several other variables from your files (I believe 3 other
> variables are needed), with each variable consuming an additional ~16GB of
> memory. (Read this article from our wrf-python documentation
> <http://wrf-python.readthedocs.io/en/latest/basic_usage.html#memory-issues-with-wrf-all-times> for
> more background information on this issue.)
>
> Are all of the “wrfout_oldfiles/wrfout_d01*” files the same size? You
> could try opening fewer files to see if that helps at all.
> For example, instead of:
> f = systemfunc("ls -1
> /home/bchaimite/lustre/WRFChem/WRFV3/run/wrfout_oldfiles/wrfout_d01*") ;
> Open a file
>
> try being more specific:
> f = systemfunc("ls -1
> /home/bchaimite/lustre/WRFChem/WRFV3/run/wrfout_oldfiles/wrfout_d01_2018-05-10*") ;
> Only matches files from 2018-05-10
>
>
>
> You may want to try iterating over each file in f and extract “tc”
> separately for each file. While the output array will still need to be
> ~16GB, the temporary arrays used for other variables should use
> significantly less space.
>
> Something like this:
> t_dims = dimsizes(a[0]->T)
> time_steps = t_dims(0) ; time steps per file
> t_dims(0) = dimsizes(f) * time_steps ; multiply time steps per file by
> number of files
>
> t = new(t_dims, float)
> do n=0,dimsizes(f) - 1 ; loop over each file in f
> ; 0:99 for first loop, 100:199 for second loop, etc
> t((time_steps * n):(time_steps * (n + 1) - 1),:,:,:) =
> wrf_user_getvar(a[n], "tc", -1) ; could also loop over each time step,
> but this should be faster
> end do
>
>
> If this still causes the “fatal:wrf_tk: Unable to allocate memory for
> output array” error, then we might need to figure out a way of breaking the
> calculation into smaller steps.
>
> I hope this helps,
> Kevin
>
>
> On May 9, 2018, at 7:24 PM, Berny Chaimite <bchaimite at gmail.com> wrote:
>
> Variable: T (file variable)
> Type: float
> Total Size: 308304800 bytes
> 77076200 values
> Number of Dimensions: 4
> Dimensions and sizes: [Time | 100] x [bottom_top | 29] x [south_north |
> 137] x [west_east | 194]
> Coordinates:
> Number Of Attributes: 6
> FieldType : 104
> MemoryOrder : XYZ
> description : perturbation potential temperature (theta-t0)
> units : K
> stagger :
> coordinates : XLONG XLAT XTIME
>
> Variable: T (file variable)
> Type: float
> Total Size: 524118160 bytes
> 131029540 values
> Number of Dimensions: 4
> Dimensions and sizes: [Time | 170] x [bottom_top | 29] x [south_north |
> 137] x [west_east | 194]
> Coordinates:
> Number Of Attributes: 6
> FieldType : 104
> MemoryOrder : XYZ
> description : perturbation potential temperature (theta-t0)
> units : K
> stagger :
> coordinates : XLONG XLAT XTIME
> fatal:wrf_tk: Unable to allocate memory for output array
> ^Mfatal:["Execute.c":8575]:Execute: Error occurred at or near line 1046 in
> file $NCARG_ROOT/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl
>
> ^Mfatal:["Execute.c":8575]:Execute: Error occurred at or near line 48 in
> file plotd01_pvt.ncl
>
>
> *-----*
>
> ----------------------------------------
> Berny A. Chaimite
> ----------------------------------------
>
> Atmospheric and Climate Sciences Student
> Department of Physics
> Eduardo Mondlane University
>
>
> *P. Contact*:
>
> Cell: +258 828 957 675(Whatsapp)/846041931/870434233
> Tell: +258 2145 6389
> Institutional: *berny.chaimite at uem.ac.mz <berny.chaimite at uem.ac.mz>*
>
>
> Maputo, Mozambique
>
>
>
> ---------- Forwarded message ---------
> From: Berny Chaimite <bchaimite at gmail.com>
> Date: quinta, 10/05/2018 à(s) 01:56
> Subject: Re: [ncl-talk] Error: wrf_tk Unable to allocate memory
> To: <hallock at ucar.edu>
>
>
> Hi,
> ok I changed the line, and I send the result attached.
>
>
>
> Regards,
> BA Chaimite
> *-----*
>
> ----------------------------------------
> Berny A. Chaimite
> ----------------------------------------
>
> Atmospheric and Climate Sciences Student
> Department of Physics
> Eduardo Mondlane University
>
>
> *P. Contact*:
>
> Cell: +258 828 957 675(Whatsapp)/846041931/870434233
> Tell: +258 2145 6389
> Institutional: *berny.chaimite at uem.ac.mz <berny.chaimite at uem.ac.mz>*
>
>
> Maputo, Mozambique
>
>
>
> Kevin Hallock <hallock at ucar.edu> escreveu no dia quinta, 10/05/2018 à(s)
> 01:24:
>
>> Hi BA,
>>
>> Thanks for the information about the number of files.
>>
>> Sorry, I gave you the wrong NCL commands to add… I forgot that you are
>> using a list of files and not a single file. Could you please change those
>> printVarSummary lines to this:
>> printVarSummary(a[0]->T)
>> printVarSummary(a1[0]->T)
>>
>> Really, the information I’m looking for is something like this:
>> Variable: T (file variable)
>> Type: float
>> Total Size: 5194496 bytes
>> 1298624 values
>> Number of Dimensions: 4
>> Dimensions and sizes: [Time | 1] x [bottom_top | 32] x [south_north |
>> 197] x [west_east | 206]
>>
>> Depending on the number of files, number of time steps per file, and the
>> sizes of bottom_top, south_north, and west_east, it’s possible that you’re
>> running out of memory.
>>
>> Thanks,
>> Kevin
>>
>>
>> On May 9, 2018, at 5:12 PM, Berny Chaimite <bchaimite at gmail.com> wrote:
>>
>> Hi Kevin,
>>
>> 1) There are 53 files I try to open in *f* and *f1* are 3 files.
>>
>> 2) I added the lines (exactly like this) that you recommended in the
>> previous email. And it shows me the following:
>>
>> fatal:(a) not reference to a valid file
>> ^Mfatal:["Execute.c":8575]:Execute: Error occurred at or near line 19 in
>> file plotd01_pvt.ncl
>>
>> line 19 is where I added: printVarSummary(a->T)
>> printVarSummary(a1->T)
>>
>>
>>
>> Regards,
>> BA Chaimite
>> *-----*
>>
>> ----------------------------------------
>> Berny A. Chaimite
>> ----------------------------------------
>>
>> Atmospheric and Climate Sciences Student
>> Department of Physics
>> Eduardo Mondlane University
>>
>>
>> *P. Contact*:
>>
>> Cell: +258 828 957 675(Whatsapp)/846041931/870434233
>> Tell: +258 2145 6389
>> Institutional: *berny.chaimite at uem.ac.mz <berny.chaimite at uem.ac.mz>*
>>
>>
>> Maputo, Mozambique
>>
>>
>>
>> Kevin Hallock <hallock at ucar.edu> escreveu no dia quarta, 9/05/2018 à(s)
>> 18:15:
>>
>>> Hi BA,
>>>
>>> How many files are you trying to open with this:
>>> f = systemfunc("ls -1
>>> /home/bchaimite/lustre/WRFChem/WRFV3/run/wrfout_oldfiles/wrfout_d01*") ;
>>> Open a file
>>> a = addfiles(f,"r")
>>>
>>> Also, could you please add the following lines to your script after both
>>> instances of “addfiles” and reply with the output:
>>> printVarSummary(a->T)
>>> printVarSummary(a1->T)
>>>
>>> Thanks,
>>> Kevin
>>>
>>> On May 9, 2018, at 3:10 AM, Berny Chaimite <bchaimite at gmail.com> wrote:
>>>
>>> I used the attached script to generate temperature maps at different
>>> vertical levels. They are data from simulations made with wrf. I used
>>> the same script to plot a smaller domain (d02) and did not give any error,
>>> now to plot d01 data shows me:
>>>
>>> fatal:wrf_tk: Unable to allocate memory for output array
>>> ^Mfatal:["Execute.c":8575]:Execute: Error occurred at or near line 1046
>>> in file $NCARG_ROOT/lib/ncarg/nclscripts/wrf/WRFUserARW.ncl
>>>
>>> ^Mfatal:["Execute.c":8575]:Execute: Error occurred at or near line 46 in
>>> file plotd01_pvt.ncl
>>>
>>>
>>>
>>>
>>> Regards,
>>> BA Chaimite
>>> *-----*
>>>
>>> ----------------------------------------
>>> Berny A. Chaimite
>>> ----------------------------------------
>>>
>>> Atmospheric and Climate Sciences Student
>>> Department of Physics
>>> Eduardo Mondlane University
>>>
>>>
>>> *P. Contact*:
>>>
>>> Cell: +258 828 957 675(Whatsapp)/846041931/870434233
>>> Tell: +258 2145 6389
>>> Institutional: *berny.chaimite at uem.ac.mz <berny.chaimite at uem.ac.mz>*
>>>
>>>
>>> Maputo, Mozambique
>>>
>>> <plotd01_pvt.ncl>_______________________________________________
>>> ncl-talk mailing list
>>> ncl-talk at ucar.edu
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>>
>>>
>> <Screenshot from 2018-05-10 01:52:19.png>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180518/2e460416/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screenshot from 2018-05-14 06:43:29.png
Type: image/png
Size: 260929 bytes
Desc: not available
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180518/2e460416/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: wrf_tc_d02.ncl
Type: application/octet-stream
Size: 3216 bytes
Desc: not available
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180518/2e460416/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screenshot from 2018-05-14 07:00:12.png
Type: image/png
Size: 286091 bytes
Desc: not available
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180518/2e460416/attachment-0003.png>
More information about the ncl-talk
mailing list