[ncl-talk] Unable to Allocate Memory

[Wei Huang]

It is likely that you have run out of memory.
If your variable is 6000*6000, with 70 files, 3 arrays,
They you are talking about 28G of data.

First, you do not need lat/lon in 70 files, as you only need one.
This will cut 2/3 of the memory required.

Second, you can just use a loop to do the accumulation, and then average it.
That will be easier to manipulate the data, as in total,
You only need  0.5G of memory.

Wei Huang

From: ncl-talk-bounces at ucar.edu [mailto:ncl-talk-bounces at ucar.edu] On Behalf Of Mariama Barr - Dallas
Sent: Saturday, August 29, 2015 7:11 PM
To: ncl-talk at ucar.edu
Subject: [ncl-talk] Unable to Allocate Memory

Dear ncl-talk users,
I keep getting the following errors when I try to open about 70 files (HDF5 format) and opening up three arrays each of approximately length 6000:

    minor: Unable to initialize object
  #003: H5Dchunk.c line 1451 in H5D__create_chunk_mem_map_hyper(): unable to copy memory space
    major: Dataspace
    minor: Unable to copy object
  #004: H5S.c line 594 in H5S_copy(): can't copy select
    major: Dataspace
    minor: Unable to copy object
  #005: H5Sselect.c line 123 in H5S_select_copy(): can't copy selection specific information
    major: Dataspace
    minor: Unable to copy object
  #006: H5Shyper.c line 1631 in H5S_hyper_copy(): can't allocate hyperslab info
    major: Resource unavailable
    minor: No space available for allocation
  #007: H5FL.c line 400 in H5FL_reg_malloc(): memory allocation failed
    major: Resource unavailable
    minor: No space available for allocation
  #008: H5FL.c line 206 in H5FL_malloc(): memory allocation failed for chunk
    major: Resource unavailable
    minor: No space available for allocation

-----------------------------------------------------------------------
Here is the script that accesses the files

load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/contributed.ncl"

DEFAULT_CONT = 0.001
SCALE = 1000000

;--adjust contour levels
;--returns contour levels based on the default scale
function adjContour(data)
local max_n,min_n

begin
max_n = max(data)
min_n = min(data)


end

;--retrieves file namess
function get_file_names()
local names,tmp_str
begin
            tmp_str = systemfunc("find /home/Bama4/OCO2_DATA/July_Test_Week -name '*.h5' ") ; July

            return tmp_str
end

;--gets data from all files in the given directory
function processFiles(names)
local longs,lats,data,files_0,tmp, avg_data

begin


            files_0 = addfiles(names,"r")
            longs = files_0[:]->retrieval_longitude
            lats  = files_0[:]->retrieval_latitude
            data  = files_0[:]->xco2
            avg_data = avg(data)

            data = abs(data - avg_data)*SCALE ;--scale values
            delete(files_0)
            delete(avg_data)
            return ([/longs,lats,data/])


end

function get_data_()
local n,tmp_list
begin
            n = get_file_names()
            tmp_list = processFiles(n)

            return tmp_list
end


Here is the main script

load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl"

load "$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl"

load "/home/Bama4/OCO2_Data/OCO2_Script.ncl"



begin



            ;-- Bounds for Region

            lon0 = -153.0

            lon1 = -163.0

            lat0 = 72.4

            lat1 = 69.0



            ;-- Region coordinates and data



            list_tmp = get_data_()



            longitudes = list_tmp[0]

            latitudes = list_tmp[1]

            var = list_tmp[2]




            ;-- define the workstation (plot type and name)

            wks = gsn_open_wks("png","unstructured_grid")



            ;-- set resources

            res = True

            res at gsnMaximize = True ;-- maximize plot output

            res at cnFillOn = True ;-- turn on contour fill

            res at cnFillPalette = "rainbow" ;-- choose a colormap

            res at tiMainString = "World Map"

            res at tiMainFontHeightF = 0.02



            res at sfXArray = list_tmp[0]

            res at sfYArray = list_tmp[1]



            ;--res at mpMinLatF = lat1

            ;--res at mpMaxLatF = lat0

            ;--res at mpMinLonF = lon1

            ;--res at mpMaxLonF = lon0

            ;--manually set the contour levels with the following 3 resources
  res at cnLevelSelectionMode = "ManualLevels"
  res at cnMinLevelValF  = 0.
  res at cnMaxLevelValF  = 500.                                  ; set the maximum contour level
  res at cnLevelSpacingF = 100.



            ;-- draw the contour map

            plot = gsn_csm_contour_map(wks, var, res)



end

--
Mariama Barr-Dallas
Meyerhoff Scholar/ NSA Scholar M23
Computer Science Major, History Major
University of Maryland Baltimore County
Class of 2015
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150831/15df4d35/attachment.html