[ncl-talk] Reading a large netcdf file
Tomoko Koyama
Tomoko.Koyama at Colorado.EDU
Mon Dec 18 15:47:34 MST 2017
Thank you very much for your response.
I’m using NCL ver. 6.3.0 and the system info is the following:
Linux login02 2.6.32-696.3.2.el6.x86_64 #1 SMP Wed Jun 7 11:51:39 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
The header of the target file is:
netcdf tas_day_IPSL-CM5A-LR_rcp85_r1i1p1_20060101-22051231.rgrd {
dimensions:
time = 73000 ;
i = 180 ;
j = 180 ;
variables:
int time(time) ;
string time:long_name = "time" ;
string time:units = "month" ;
float i(i) ;
string i:units = "none" ;
string i:long_name = "i index" ;
float j(j) ;
string j:units = "none" ;
string j:long_name = "j index" ;
float tas(time, i, j) ;
string tas:associated_files = "baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_atmos_fx_IPSL-CM5A-LR_rcp85_r0i0p0.nc areacella: areacella_fx_IPSL-CM5A-LR_rcp85_r0i0p0.nc" ;
string tas:coordinates = "height" ;
string tas:history = "2011-08-16T22:13:26Z altered by CMOR: Treated scalar dimension: \'height\'. 2011-08-16T22:13:26Z altered by CMOR: replaced missing value flag (9.96921e+36) with standard missing value (1e+20). 2011-08-16T22:13:45Z altered by CMOR: Inverted axis: lat." ;
string tas:cell_measures = "area: areacella" ;
string tas:cell_methods = "time: mean (interval: 30 minutes)" ;
string tas:original_name = "t2m" ;
string tas:units = "K" ;
string tas:long_name = "Near-Surface Air Temperature" ;
string tas:standard_name = "air_temperature" ;
string tas:remap = "remapped via ESMF_regrid_with_weights: Bilinear" ;
tas:missing_value = 1.e+20f ;
tas:_FillValue = 1.e+20f ;
float lat(j, i) ;
string lat:long_name = "latitude" ;
string lat:units = "degrees_north" ;
lat:_FillValue = -999.f ;
float lon(j, i) ;
string lon:long_name = "longitude" ;
string lon:units = "degrees_east" ;
lon:_FillValue = -999.f ;
}
> On Dec 18, 2017, at 8:05 AM, Dennis Shea <shea at ucar.edu> wrote:
>
> When you have a file issue, you should include some information:
>
> (a) what what either of the fillowing show?
>
> %> ncl_filedump tas_day_IPSL-CM5A-LR_rcp85_r1i1p1_20060101-22051231.rgrd.nc <http://secure-web.cisco.com/1Vrb-DzQ8Nq6PmsZlx-VCfXUfqpPMn1kXTYcQdEn-y4fG8MjIf5FObpsBYxAL2dmU2ZkqNJfPs2m48ExK2fva0vSjYwNLr6rs3gkQ6g5UEmk8YZ2M20H3syb-rPE3GNx2tTdfo_z1ua7u5k0Wb08hDwLEukpmfhGBv7kt8YDlQbBwjUBMS_HdToqmK_XiP8JoUcRtxtarP0x80S1_Z-v8v5Z8PvxcBc0moozZZf5dmaDSt4FFu4fIrM23KZHUbY46HPXuKt9NI2DWfPerKnJJLwBnFnXSQPpfgpK-FLJBenIoiKsGlrVbaphjCtP6hAq7mvHF1zP6hieqFXhU995Jask72xYsh1-1fl1VwZb4r3sJuqZFamfSy6SNO2CHfgcNy90HUGvXz8Tr1-oKAdgRxdzO1wcnFpPoAjegHXfheH2PZAQj9b07_gdKkzY6eidU/http%3A%2F%2Ftas_day_IPSL-CM5A-LR_rcp85_r1i1p1_20060101-22051231.rgrd.nc>
>
> or
>
> %> ncdump -h tas_day_IPSL-CM5A-LR_rcp85_r1i1p1_20060101-22051231.rgrd.nc <http://secure-web.cisco.com/1Vrb-DzQ8Nq6PmsZlx-VCfXUfqpPMn1kXTYcQdEn-y4fG8MjIf5FObpsBYxAL2dmU2ZkqNJfPs2m48ExK2fva0vSjYwNLr6rs3gkQ6g5UEmk8YZ2M20H3syb-rPE3GNx2tTdfo_z1ua7u5k0Wb08hDwLEukpmfhGBv7kt8YDlQbBwjUBMS_HdToqmK_XiP8JoUcRtxtarP0x80S1_Z-v8v5Z8PvxcBc0moozZZf5dmaDSt4FFu4fIrM23KZHUbY46HPXuKt9NI2DWfPerKnJJLwBnFnXSQPpfgpK-FLJBenIoiKsGlrVbaphjCtP6hAq7mvHF1zP6hieqFXhU995Jask72xYsh1-1fl1VwZb4r3sJuqZFamfSy6SNO2CHfgcNy90HUGvXz8Tr1-oKAdgRxdzO1wcnFpPoAjegHXfheH2PZAQj9b07_gdKkzY6eidU/http%3A%2F%2Ftas_day_IPSL-CM5A-LR_rcp85_r1i1p1_20060101-22051231.rgrd.nc>
>
> (b) what version of NCL are you using>
>
> %> ncl -V
>
> (c) your system info
>
> %> uname -a
>
> -----------
>
> fdir=“/root/dir4ncl/“
> fili="tas_day_IPSL-CM5A-LR_rcp85_r1i1p1_20060101-22051231.rgrd.nc <http://secure-web.cisco.com/1Vrb-DzQ8Nq6PmsZlx-VCfXUfqpPMn1kXTYcQdEn-y4fG8MjIf5FObpsBYxAL2dmU2ZkqNJfPs2m48ExK2fva0vSjYwNLr6rs3gkQ6g5UEmk8YZ2M20H3syb-rPE3GNx2tTdfo_z1ua7u5k0Wb08hDwLEukpmfhGBv7kt8YDlQbBwjUBMS_HdToqmK_XiP8JoUcRtxtarP0x80S1_Z-v8v5Z8PvxcBc0moozZZf5dmaDSt4FFu4fIrM23KZHUbY46HPXuKt9NI2DWfPerKnJJLwBnFnXSQPpfgpK-FLJBenIoiKsGlrVbaphjCtP6hAq7mvHF1zP6hieqFXhU995Jask72xYsh1-1fl1VwZb4r3sJuqZFamfSy6SNO2CHfgcNy90HUGvXz8Tr1-oKAdgRxdzO1wcnFpPoAjegHXfheH2PZAQj9b07_gdKkzY6eidU/http%3A%2F%2Ftas_day_IPSL-CM5A-LR_rcp85_r1i1p1_20060101-22051231.rgrd.nc>"
> fn=addfile(fdir+fili,"r")
> buff=fn->tas
> print(“Data is stored”)
>
> On Mon, Dec 18, 2017 at 3:35 AM, Guido Cioni <guidocioni at gmail.com <mailto:guidocioni at gmail.com>> wrote:
> Tomoko,
> 9 GB is anything but "large", although the concept of "large" is highly subjective :P
>
> I've successfully read SINGLE netcdf files in NCL whose size was ~500GB so that shouldn't be the problem. For some reason a netcdf file of some size, say 400 GB, which has many timesteps is read more slowly than a file with the same size but with less timesteps; that was my impression.
>
> You are setting a lot of options which I think are not needed. Did you just try to read the file with this line?
>
>> fn=addfile(fdir+fili,"r")
>
> If it still takes a lot of time it could be system-dependent. When creating the variable NCL stores it into the RAM. If the system does not have enough RAM, some virtual memory will be created on your hard drive, which can slow down everything. But honestly I don't think you're even close to saturate your system's RAM. The problem may lie somewhere else...
>
> Let us know.
>
>
>> On 18. Dec 2017, at 07:17, Tomoko Koyama <Tomoko.Koyama at Colorado.EDU <mailto:Tomoko.Koyama at Colorado.EDU>> wrote:
>>
>> I’d like to extract some data from a large netcdf file, which size is about 9GB.
>>
>> The following shows the partial script, but a submitted job was killed before “Data is stored” message appeared.
>> (I attempted several times with 3-hr max walltime )
>>
>> setfileoption("nc", "FileStructure", "Advanced")
>> fdir=“/root/dir4ncl/“
>> fili="tas_day_IPSL-CM5A-LR_rcp85_r1i1p1_20060101-22051231.rgrd.nc <http://secure-web.cisco.com/1Vrb-DzQ8Nq6PmsZlx-VCfXUfqpPMn1kXTYcQdEn-y4fG8MjIf5FObpsBYxAL2dmU2ZkqNJfPs2m48ExK2fva0vSjYwNLr6rs3gkQ6g5UEmk8YZ2M20H3syb-rPE3GNx2tTdfo_z1ua7u5k0Wb08hDwLEukpmfhGBv7kt8YDlQbBwjUBMS_HdToqmK_XiP8JoUcRtxtarP0x80S1_Z-v8v5Z8PvxcBc0moozZZf5dmaDSt4FFu4fIrM23KZHUbY46HPXuKt9NI2DWfPerKnJJLwBnFnXSQPpfgpK-FLJBenIoiKsGlrVbaphjCtP6hAq7mvHF1zP6hieqFXhU995Jask72xYsh1-1fl1VwZb4r3sJuqZFamfSy6SNO2CHfgcNy90HUGvXz8Tr1-oKAdgRxdzO1wcnFpPoAjegHXfheH2PZAQj9b07_gdKkzY6eidU/http%3A%2F%2Ftas_day_IPSL-CM5A-LR_rcp85_r1i1p1_20060101-22051231.rgrd.nc>"
>> fn=addfile(fdir+fili,"r")
>> setfileoption("nc","Format","NetCDF4Classic")
>> buff=fn->tas
>> print(“Data is stored”)
>>
>> Does it simply take a long time to read?
>> Is there anyway to speed up to read a large netcdf file?
>>
>>
>> Thank you,
>> Tomoko
>> _______________________________________________
>> ncl-talk mailing list
>> ncl-talk at ucar.edu <mailto:ncl-talk at ucar.edu>
>> List instructions, subscriber options, unsubscribe:
>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk <http://mailman.ucar.edu/mailman/listinfo/ncl-talk>
>
>
> Guido Cioni
> http://guidocioni.altervista <http://secure-web.cisco.com/1VTkTCOmq-V7se95ulfH2_V2onf2c5MHHSTkyctv2YOqoL1PVEAl_MqESQrE3TCnUXt5-Plwk6X6Ijz7gW23j1IO1btnfLdwl5uueV2Byy2e8E0u3WyoeKH12w2VLjTkl-D2S3X_BVWJH-e346H9PjUFHsGPt8LKQoSWmxYYoaBF17tgPyud58ZrGEy1Qz0AIEN_hEEVgf3aZxxopkxDkeph0fLWlG48ghQ4jtD4XBzTLqXCYE6HY0OpDrszaiyfBqlA8nNl2lBEzsl0vnu9k5hLcDideVoHSjwf4q4FmI7Eyzo3SPdcG_z7wCvLKzMRx2gLbgEOIgS35uovnkmWg4MteYUkJvSQjsVXdhrfOe7LnBXeHA6rs2JZYtZCdLzsqT7rE_dzKlyPUK2cav6PhfYhz7MwFpKBJmj0joQ3s4RlnsHFtLB-nxRtZq9PTt9JL/http%3A%2F%2Fguidocioni.altervista>.org
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu <mailto:ncl-talk at ucar.edu>
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk <http://mailman.ucar.edu/mailman/listinfo/ncl-talk>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20171218/b06c81bd/attachment.html>
More information about the ncl-talk
mailing list