[ncl-talk] Fwd: [ncl-install] Code/loop problems

[Mariama Barr - Dallas]

The variable information is below (xco2). It can range from 6000 to 8000
indices and is a 1D float array, so in other words it varies from file to
file (all HDF5).

​

On Mon, Aug 24, 2015 at 1:23 PM, Mary Haley <haley at ucar.edu> wrote:

> This is likely due you to running out of memory, since you are now trying
> to read all the data at once. You may have to read the variable in
> sections. I can help with this, but I need to know more about the variable
> you are trying to read.
>
> How big is the variable on *one* of the files, and what type is it?  That
> is, what are the dimension sizes of the array? You can use "ncl_filedump"
> to get this information.
>
> Is this variable the same size on all 107 files?
>
> --Mary
>
> On Mon, Aug 24, 2015 at 10:35 AM, Mariama Barr - Dallas <bama4 at umbc.edu>
> wrote:
>
>>
>> ---------- Forwarded message ----------
>> From: Mariama Barr - Dallas <bama4 at umbc.edu>
>> Date: Mon, Aug 24, 2015 at 12:31 PM
>> Subject: Re: [ncl-install] Code/loop problems
>> To: Mary Haley <haley at ucar.edu>, ncl-talk at ucar.edu
>>
>>
>> When I try to use the addfiles function for 107 HDF5 files that are
>> approx. 60,000 Kb large, I get a segmentation fault: failure to allocate
>> memory for NclHDF5datatype..why is this?
>>
>>
>> On Sun, Aug 23, 2015 at 11:20 PM, Mary Haley <haley at ucar.edu> wrote:
>>
>>> Mariama,
>>>
>>> In the future, please post NCL questions to ncl-talk, and not
>>> ncl-install:
>>>
>>> http://www.ncl.ucar.edu/Support/email_lists.shtml
>>>
>>> Without knowing anything about your data, it's almost impossible to tell
>>> you how to fix your code so you can correctly "gather" all the data. We
>>> need to know what structure the data is on each file.
>>>
>>> Depending on your data, you might be able to use "addfiles" instead of
>>> "addfile", as it allows you to read in a series of files with a single
>>> call.You can then read data off all the files in either "join" or "cat"
>>> mode.  See examples 1 and 2 on the "addfiles" function page:
>>>
>>> http://www.ncl.ucar.edu/Document/Functions/Built-in/addfiles.shtml
>>>
>>> Your code might look something like this:
>>>
>>>   files_0 = addfiles(names,"r")    ; Open all the HDF5 files in one call.
>>>   longs = files_0[:]->retrieval_longitude   ; Read retrieval_longitude
>>> across all files
>>>   lats  = files_0[:]->retrieval_latitude    ; ...
>>>   data  = files_0[:]->xco2
>>>
>>>
>>> Also, you don't need to redirect your "find" output to an ASCII file,
>>> and then read in that ASCII file to get the list of names. You should be
>>> able to do something like this:
>>>
>>> function get_file_names()
>>> begin
>>>   names = systemfunc("find /home/Bama4/OCO2_DATA/ -name '*.h5' -print')
>>>   return names
>>> end
>>>
>>> Please look at your data files carefully, and determine whether
>>> "addfiles" might work for you. If you continue to have questions, post them
>>> to ncl-talk, but include more information about the structure of your
>>> data.  You can either use "ncl_filedump" to look at single files from the
>>> UNIX command line:
>>>
>>>     ncl_filedump xxxxx.h5
>>>
>>> or "printVarSummary" to examine the contents of a variable after you
>>> read it off a file:
>>>
>>>    data = files_0->xco2
>>>    printVarSummary(data)
>>>
>>> --Mary
>>>
>>>
>>> On Sat, Aug 22, 2015 at 5:09 PM, Mariama Barr - Dallas <bama4 at umbc.edu>
>>> wrote:
>>>
>>>> The following code is giving me problems, Im trying to gather all of
>>>> the data into three arrays: longs, lats, and data. It  keeps saying that
>>>> the arrays are not the same dimension when I try array_append_record, and
>>>> when I try copy_MetaVar for the arrays it just gives me an empty array. Is
>>>> there an easier way to gather up all of the latitudes, longitudes, and data
>>>> from each file into three arrays so I can in the future display it  on one
>>>> map?
>>>>
>>>> ​
>>>>
>>>> --
>>>> Mariama Barr-Dallas
>>>> Meyerhoff Scholar/ NSA Scholar M23
>>>> Computer Science Major, History Major
>>>> University of Maryland Baltimore County
>>>> Class of 2015
>>>>
>>>> _______________________________________________
>>>> ncl-install mailing list
>>>> List instructions, subscriber options, unsubscribe:
>>>> http://mailman.ucar.edu/mailman/listinfo/ncl-install
>>>>
>>>>
>>>
>>
>>
>> --
>> Mariama Barr-Dallas
>> Meyerhoff Scholar/ NSA Scholar M23
>> Computer Science Major, History Major
>> University of Maryland Baltimore County
>> Class of 2015
>>
>>
>>
>> --
>> Mariama Barr-Dallas
>> Meyerhoff Scholar/ NSA Scholar M23
>> Computer Science Major, History Major
>> University of Maryland Baltimore County
>> Class of 2015
>>
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>
>>
>


-- 
Mariama Barr-Dallas
Meyerhoff Scholar/ NSA Scholar M23
Computer Science Major, History Major
University of Maryland Baltimore County
Class of 2015
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150824/deda5369/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: code.PNG
Type: image/png
Size: 58715 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150824/deda5369/attachment.png 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: co2DryMolarFrac.PNG
Type: image/png
Size: 2594 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150824/deda5369/attachment-0001.png