[ncl-talk] Sub: Killed
David Brown
dbrown at ucar.edu
Tue Mar 3 22:32:53 MST 2015
>>Hi,
>>I have the hard drive space of 700GB. Do you speak about RAM?
>>I have 20 files in total.
Thanks
Hard drive space is not the issue. RAM is what I am talking about. So
if you try to read the cl variable from all files, you will have 1.5
GB x 20 in the aggregated variable. That is 30 GB of RAM you would
need just to hold the variable, not to mention what is needed for the
operating system, as well as space to do work on the data. So unless
you are on a pretty big machine you do not have enough space to
operate on the whole variable at once.
How you solve this depends on what you are trying to do. Perhaps you
could process just one level at a time. If you do the arithmetic you
will find that one level with all the timesteps is 491 MB. That might
work.
-dave
On Tue, Mar 3, 2015 at 8:34 PM, Adv <advita6 at gmail.com> wrote:
> Hi,
> I have the hard drive space of 700GB. Do you speak about RAM?
> I have 20 files in total.
>
> Thanks
>
> On 03-Mar-2015 5:02 PM, "David Brown" <dbrown at ucar.edu> wrote:
>>
>> Hi Adv,
>> Well my questions would be how many of these files are you trying to
>> read and how much memory do you have available on your system. Since
>> the size of the 'cl' variable in each file is ~1.5 GB, it would
>> require 15 GB of memory to read 10 files. 15 GB is more than an
>> average desktop computer could give you. Check your numbers. If you
>> are exceeding the capacity of your system you will have to figure out
>> a way to do your processing in smaller chunks.
>> -dave
>>
>>
>> On Tue, Mar 3, 2015 at 3:50 PM, Adv <advita6 at gmail.com> wrote:
>> > Hi,
>> > I am trying to handle cluster of netcdf files with each file size of
>> > 1.5GB.
>> > I also printed below the code i used. When i try to run my code, outputs
>> > Killed.
>> > Could someone help me to fix the bug.?
>> >
>> >
>> >
>> > fils = systemfunc ("ls /home/use/downloadesgf/set1/ncl/pr/*.nc") ; file
>> > paths
>> > f = addfiles (fils, "r")
>> > ListSetType (f, "cat")
>> > cl= f[:]->cl ; read T from all files
>> > printVarSummary (cl)
>> >
>> >
>> > ncdump -c of my netcdf is
>> > ncdump -c cl_Amon_EC-EARTH_decadal1960_r3i1p1_196101-197012.nc
>> > netcdf cl_Amon_EC-EARTH_decadal1960_r3i1p1_196101-197012 {
>> > dimensions:
>> > time = UNLIMITED ; // (120 currently)
>> > lev = 62 ;
>> > lat = 160 ;
>> > lon = 320 ;
>> > bnds = 2 ;
>> > variables:
>> > double time(time) ;
>> > time:bounds = "time_bnds" ;
>> > time:units = "days since 1960-1-1" ;
>> > time:calendar = "gregorian" ;
>> > time:axis = "T" ;
>> > time:long_name = "time" ;
>> > time:standard_name = "time" ;
>> > double time_bnds(time, bnds) ;
>> > double lev(lev) ;
>> > lev:bounds = "lev_bnds" ;
>> > lev:units = "1" ;
>> > lev:axis = "Z" ;
>> > lev:positive = "down" ;
>> > lev:long_name = "hybrid sigma pressure coordinate" ;
>> > lev:standard_name =
>> > "atmosphere_hybrid_sigma_pressure_coordinate" ;
>> > lev:formula = "p = a*p0 + b*ps" ;
>> > lev:formula_terms = "p0: p0 a: a b: b ps: ps" ;
>> > double lev_bnds(lev, bnds) ;
>> > lev_bnds:formula = "p = a*p0 + b*ps" ;
>> > lev_bnds:standard_name =
>> > "atmosphere_hybrid_sigma_pressure_coordinate" ;
>> > lev_bnds:units = "1" ;
>> > lev_bnds:formula_terms = "p0: p0 a: a_bnds b: b_bnds ps: ps" ;
>> > float p0 ;
>> > p0:long_name = "vertical coordinate formula term: reference
>> > pressure" ;
>> > p0:units = "Pa" ;
>> > double a(lev) ;
>> > a:long_name = "vertical coordinate formula term: a(k)" ;
>> > double b(lev) ;
>> > b:long_name = "vertical coordinate formula term: b(k)" ;
>> > float ps(time, lat, lon) ;
>> > ps:standard_name = "surface_air_pressure" ;
>> > ps:long_name = "Surface Air Pressure" ;
>> > ps:comment = "not, in general, the same as mean sea-level
>> > pressure"
>> > ;
>> > ps:units = "Pa" ;
>> > ps:cell_methods = "time: mean" ;
>> > ps:cell_measures = "area: areacella" ;
>> > double a_bnds(lev, bnds) ;
>> > a_bnds:long_name = "vertical coordinate formula term: a(k+1/2)"
>> > ;
>> > double b_bnds(lev, bnds) ;
>> > b_bnds:long_name = "vertical coordinate formula term: b(k+1/2)"
>> > ;
>> > double lat(lat) ;
>> > lat:bounds = "lat_bnds" ;
>> > lat:units = "degrees_north" ;
>> > lat:axis = "Y" ;
>> > lat:long_name = "latitude" ;
>> > lat:standard_name = "latitude" ;
>> > double lat_bnds(lat, bnds) ;
>> > double lon(lon) ;
>> > lon:bounds = "lon_bnds" ;
>> > lon:units = "degrees_east" ;
>> > lon:axis = "X" ;
>> > lon:long_name = "longitude" ;
>> > lon:standard_name = "longitude" ;
>> > double lon_bnds(lon, bnds) ;
>> > float cl(time, lev, lat, lon) ;
>> > cl:standard_name = "cloud_area_fraction_in_atmosphere_layer" ;
>> > cl:long_name = "Cloud Area Fraction" ;
>> > cl:comment = "Includes both large-scale and convective cloud." ;
>> > cl:units = "%" ;
>> > cl:original_name = "CC" ;
>> > cl:cell_methods = "time: mean (interval: 6 hours)" ;
>> > cl:cell_measures = "area: areacella" ;
>> > cl:history = "2012-04-02T08:53:38Z altered by CMOR: replaced
>> > missing
>> > value flag (1e+28) with standard missing value (1e+20).
>> > 2012-04-02T08:53:38Z
>> > altered by CMOR: Converted type from \'d\' to \'f\'.
>> > 2012-04-02T08:53:38Z
>> > altered by CMOR: Inverted axis: lev. 2012-04-02T08:53:38Z altered by
>> > CMOR:
>> > Inverted axis: lat." ;
>> > cl:missing_value = 1.e+20f ;
>> > cl:_FillValue = 1.e+20f ;
>> > cl:associated_files = "baseURL:
>> > http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile:
>> > gridspec_atmos_fx_EC-Earth_decadal1960_r0i0p0.nc areacella:
>> > areacella_fx_EC-Earth_decadal1960_r0i0p0.nc" ;
>> >
>> > // global attributes:
>> > :institution = "EC-Earth (European Earth System Model)" ;
>> > :institute_id = "ICHEC" ;
>> > :experiment_id = "decadal1960" ;
>> > :source = "EC-Earth 2.3 (2011); atmosphere: IFS
>> > (cy31R1+modifications, T159L62); ocean: NEMO (version2+modifications,
>> > ORCA1-42lev); sea ice: LIM2; land: HTessel" ;
>> > :model_id = "EC-EARTH" ;
>> > :forcing = "GHG, Oz, SA, Sl, Vl, (GHG = CO2, N2O, CH4, CFCs)" ;
>> > :parent_experiment_id = "N/A" ;
>> > :parent_experiment_rip = "N/A" ;
>> > :branch_time = 0. ;
>> > :contact = "Alastair McKinstry <alastair.mckinstry at ichec.ie>" ;
>> > :history = "IC3 Decadal Experiment 2012-04-02T08:47:41Z CMOR
>> > rewrote
>> > data to comply with CF standards and CMIP5 requirements." ;
>> > :comment = "IC3 contact f.doblas-reyes at ic3.cat" ;
>> > :references = "Du, H., F.J. Doblas-Reyes, J. Garcia-Serrano, V.
>> > Guemas, Y. Soufflet, B. Wouters, 2012. Sensitivity of decadal
>> > predictions to
>> > the initial atmospheric and oceanic perturbations. Climate Dynamics, in
>> > press" ;
>> > :initialization_method = 1 ;
>> > :physics_version = 1 ;
>> > :tracking_id = "27b8c0a8-25a8-4207-aef0-5c580252df3e" ;
>> > :product = "output" ;
>> > :experiment = "10- or 30-year run initialized in year 1960" ;
>> > :frequency = "mon" ;
>> > :creation_date = "2012-04-02T08:53:38Z" ;
>> > :Conventions = "CF-1.4" ;
>> > :project_id = "CMIP5" ;
>> > :table_id = "Table Amon (26 July 2011)
>> > b26379e76858ab98b927917878a63d01" ;
>> > :title = "EC-Earth model output prepared for CMIP5 10- or
>> > 30-year
>> > run initialized in year 1960" ;
>> > :parent_experiment = "N/A" ;
>> > :modeling_realm = "atmos" ;
>> > :realization = 3 ;
>> > :cmor_version = "2.8.0" ;
>> > data:
>> >
>> > time = 381.5, 411, 440.5, 471, 501.5, 532, 562.5, 593.5, 624, 654.5,
>> > 685,
>> > 715.5, 746.5, 776, 805.5, 836, 866.5, 897, 927.5, 958.5, 989,
>> > 1019.5,
>> > 1050, 1080.5, 1111.5, 1141, 1170.5, 1201, 1231.5, 1262, 1292.5,
>> > 1323.5,
>> > 1354, 1384.5, 1415, 1445.5, 1476.5, 1506.5, 1536.5, 1567, 1597.5,
>> > 1628,
>> > 1658.5, 1689.5, 1720, 1750.5, 1781, 1811.5, 1842.5, 1872, 1901.5,
>> > 1932,
>> > 1962.5, 1993, 2023.5, 2054.5, 2085, 2115.5, 2146, 2176.5, 2207.5,
>> > 2237,
>> > 2266.5, 2297, 2327.5, 2358, 2388.5, 2419.5, 2450, 2480.5, 2511,
>> > 2541.5,
>> > 2572.5, 2602, 2631.5, 2662, 2692.5, 2723, 2753.5, 2784.5, 2815,
>> > 2845.5,
>> > 2876, 2906.5, 2937.5, 2967.5, 2997.5, 3028, 3058.5, 3089, 3119.5,
>> > 3150.5,
>> > 3181, 3211.5, 3242, 3272.5, 3303.5, 3333, 3362.5, 3393, 3423.5,
>> > 3454,
>> > 3484.5, 3515.5, 3546, 3576.5, 3607, 3637.5, 3668.5, 3698, 3727.5,
>> > 3758,
>> > 3788.5, 3819, 3849.5, 3880.5, 3911, 3941.5, 3972, 4002.5 ;
>> >
>> > lev = 0.998814937344117, 0.995949133192705, 0.991977550709468,
>> > 0.986761005882505, 0.98031097707606, 0.972738617371053,
>> > 0.963896752354567, 0.953529666328856, 0.941606471269923,
>> > 0.92813291389402, 0.913110538468371, 0.896545012353933,
>> > 0.878475201530559, 0.858976029448204, 0.838120207706876,
>> > 0.815972842517533, 0.7926217749721, 0.768187587358379,
>> > 0.742788084545164,
>> > 0.716525528571299, 0.689516580275552, 0.661905753898445,
>> > 0.63383208242183, 0.605413205707871, 0.576774847794481,
>> > 0.548064217305543, 0.519501453051902, 0.491366554478158,
>> > 0.463933650692722, 0.437458895338325, 0.41218007358813,
>> > 0.388198451687388, 0.36547741407194, 0.343964864673422,
>> > 0.323604011499092, 0.304340414099066, 0.286121358005341,
>> > 0.268896466407701, 0.252617382048276, 0.237237738281987,
>> > 0.222713168620164, 0.209001182237978, 0.19606110463038,
>> > 0.183854030567662, 0.172342708649639, 0.161491573394506,
>> > 0.151266613407328, 0.14142902940564, 0.131674244431666,
>> > 0.121919806943467, 0.112165666514245, 0.102411788016295,
>> > 0.0926581119445705, 0.0829045477863824, 0.0731510471469045,
>> > 0.0633975753094143, 0.0536441023448784, 0.0438906173627338,
>> > 0.0341371295830871, 0.0243836516742719, 0.01463017363562,
>> > 0.00487671730798125 ;
>> >
>> > lat = -89.14152, -88.02943, -86.91077, -85.79063, -84.66992, -83.54895,
>> > -82.42782, -81.30659, -80.18531, -79.06398, -77.94262, -76.82124,
>> > -75.69984, -74.57843, -73.45701, -72.33558, -71.21414, -70.09269,
>> > -68.97124, -67.84978, -66.72833, -65.60686, -64.4854, -63.36393,
>> > -62.24246, -61.12099, -59.99952, -58.87804, -57.75657, -56.63509,
>> > -55.51361, -54.39214, -53.27066, -52.14917, -51.02769, -49.90621,
>> > -48.78473, -47.66325, -46.54176, -45.42028, -44.29879, -43.17731,
>> > -42.05582, -40.93434, -39.81285, -38.69137, -37.56988, -36.44839,
>> > -35.32691, -34.20542, -33.08393, -31.96244, -30.84096, -29.71947,
>> > -28.59798, -27.47649, -26.355, -25.23351, -24.11203, -22.99054,
>> > -21.86905, -20.74756, -19.62607, -18.50458, -17.38309, -16.2616,
>> > -15.14011, -14.01862, -12.89713, -11.77564, -10.65415, -9.53266,
>> > -8.41117, -7.28968, -6.16819, -5.0467, -3.92521, -2.80372, -1.68223,
>> > -0.56074, 0.56074, 1.68223, 2.80372, 3.92521, 5.0467, 6.16819,
>> > 7.28968,
>> > 8.41117, 9.53266, 10.65415, 11.77564, 12.89713, 14.01862, 15.14011,
>> > 16.2616, 17.38309, 18.50458, 19.62607, 20.74756, 21.86905, 22.99054,
>> > 24.11203, 25.23351, 26.355, 27.47649, 28.59798, 29.71947, 30.84096,
>> > 31.96244, 33.08393, 34.20542, 35.32691, 36.44839, 37.56988,
>> > 38.69137,
>> > 39.81285, 40.93434, 42.05582, 43.17731, 44.29879, 45.42028,
>> > 46.54176,
>> > 47.66325, 48.78473, 49.90621, 51.02769, 52.14917, 53.27066,
>> > 54.39214,
>> > 55.51361, 56.63509, 57.75657, 58.87804, 59.99952, 61.12099,
>> > 62.24246,
>> > 63.36393, 64.4854, 65.60686, 66.72833, 67.84978, 68.97124, 70.09269,
>> > 71.21414, 72.33558, 73.45701, 74.57843, 75.69984, 76.82124,
>> > 77.94262,
>> > 79.06398, 80.18531, 81.30659, 82.42782, 83.54895, 84.66992,
>> > 85.79063,
>> > 86.91077, 88.02943, 89.14152 ;
>> >
>> > lon = 0, 1.125, 2.25, 3.375, 4.5, 5.625, 6.75, 7.875, 9, 10.125, 11.25,
>> > 12.375, 13.5, 14.625, 15.75, 16.875, 18, 19.125, 20.25, 21.375,
>> > 22.5,
>> > 23.625, 24.75, 25.875, 27, 28.125, 29.25, 30.375, 31.5, 32.625,
>> > 33.75,
>> > 34.875, 36, 37.125, 38.25, 39.375, 40.5, 41.625, 42.75, 43.875, 45,
>> > 46.125, 47.25, 48.375, 49.5, 50.625, 51.75, 52.875, 54, 55.125,
>> > 56.25,
>> > 57.375, 58.5, 59.625, 60.75, 61.875, 63, 64.125, 65.25, 66.375,
>> > 67.5,
>> > 68.625, 69.75, 70.875, 72, 73.125, 74.25, 75.375, 76.5, 77.625,
>> > 78.75,
>> > 79.875, 81, 82.125, 83.25, 84.375, 85.5, 86.625, 87.75, 88.875, 90,
>> > 91.125, 92.25, 93.375, 94.5, 95.625, 96.75, 97.875, 99, 100.125,
>> > 101.25,
>> > 102.375, 103.5, 104.625, 105.75, 106.875, 108, 109.125, 110.25,
>> > 111.375,
>> > 112.5, 113.625, 114.75, 115.875, 117, 118.125, 119.25, 120.375,
>> > 121.5,
>> > 122.625, 123.75, 124.875, 126, 127.125, 128.25, 129.375, 130.5,
>> > 131.625,
>> > 132.75, 133.875, 135, 136.125, 137.25, 138.375, 139.5, 140.625,
>> > 141.75,
>> > 142.875, 144, 145.125, 146.25, 147.375, 148.5, 149.625, 150.75,
>> > 151.875,
>> > 153, 154.125, 155.25, 156.375, 157.5, 158.625, 159.75, 160.875, 162,
>> > 163.125, 164.25, 165.375, 166.5, 167.625, 168.75, 169.875, 171,
>> > 172.125,
>> > 173.25, 174.375, 175.5, 176.625, 177.75, 178.875, 180, 181.125,
>> > 182.25,
>> > 183.375, 184.5, 185.625, 186.75, 187.875, 189, 190.125, 191.25,
>> > 192.375,
>> > 193.5, 194.625, 195.75, 196.875, 198, 199.125, 200.25, 201.375,
>> > 202.5,
>> > 203.625, 204.75, 205.875, 207, 208.125, 209.25, 210.375, 211.5,
>> > 212.625,
>> > 213.75, 214.875, 216, 217.125, 218.25, 219.375, 220.5, 221.625,
>> > 222.75,
>> > 223.875, 225, 226.125, 227.25, 228.375, 229.5, 230.625, 231.75,
>> > 232.875,
>> > 234, 235.125, 236.25, 237.375, 238.5, 239.625, 240.75, 241.875, 243,
>> > 244.125, 245.25, 246.375, 247.5, 248.625, 249.75, 250.875, 252,
>> > 253.125,
>> > 254.25, 255.375, 256.5, 257.625, 258.75, 259.875, 261, 262.125,
>> > 263.25,
>> > 264.375, 265.5, 266.625, 267.75, 268.875, 270, 271.125, 272.25,
>> > 273.375,
>> > 274.5, 275.625, 276.75, 277.875, 279, 280.125, 281.25, 282.375,
>> > 283.5,
>> > 284.625, 285.75, 286.875, 288, 289.125, 290.25, 291.375, 292.5,
>> > 293.625,
>> > 294.75, 295.875, 297, 298.125, 299.25, 300.375, 301.5, 302.625,
>> > 303.75,
>> > 304.875, 306, 307.125, 308.25, 309.375, 310.5, 311.625, 312.75,
>> > 313.875,
>> > 315, 316.125, 317.25, 318.375, 319.5, 320.625, 321.75, 322.875, 324,
>> > 325.125, 326.25, 327.375, 328.5, 329.625, 330.75, 331.875, 333,
>> > 334.125,
>> > 335.25, 336.375, 337.5, 338.625, 339.75, 340.875, 342, 343.125,
>> > 344.25,
>> > 345.375, 346.5, 347.625, 348.75, 349.875, 351, 352.125, 353.25,
>> > 354.375,
>> > 355.5, 356.625, 357.75, 358.875 ;
>> > }
>> > Thanks
>> > Adv
>> >
>> >
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > ncl-talk mailing list
>> > List instructions, subscriber options, unsubscribe:
>> > http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>> >
More information about the ncl-talk
mailing list