[ncl-talk] short2flt error

[Dave Allured - NOAA Affiliate]

Ehsan, a standard strategy for large array problems is to add a loop and
process the data one time step at a time.  Do not try to read the whole
array into memory at any time.  This assumes that you are interpolating
across X and Y, but not time.

open input files
start a new output file
do itime = 0, ntimes-1
    read grids for one time step
    interpolate
    write result grids for one time step
end do
close files


On Mon, Sep 12, 2022 at 10:54 AM Ehsan Taghizadeh via ncl-talk <
ncl-talk at mailman.ucar.edu> wrote:

> Dear Rick,
> The nc file is included a domain including (25-40N, 40-65E) which was
> downloaded with the attach api script (temp2m09_18.py). As you mentioned
> the downloaded file has a size more than 6 GB, because of the domain and
> long term hourly data. Should I break the time period to have smaller size?
>
> On the other hand I would like to interpolate temperature on stations
> location with the attached script (if I assume short2flt has fixed). How
> could I write these long data (34*10*365), which 34 is number of stations,
> 10*365 is number of days?
> In other words, the desired quantity "t2m_int2p_mima" has 3 dimensions;
> 1st dimension is related to (min, max, avg), 2nd dimension is related to
> days which is about 10*365,
> and 3rd dimension is related to stations which is about 34 stations.
> May I ask how could I write "t2m_int2p_mima" in a table with headers same
> as below:
>
>  "stid  min(day1) max(day1) avg(day1) min(day2) max(day2) avg(day2) ...
> min(day10*365) max(day10*365) avg(day10*365)"
>
> I'll be thankful for any help.
>
>
> *-----------------------------------------------------------------------------*
> *Sincerely,*
>
> *Ehsan Taghizadeh*
>
>
> On Monday, September 12, 2022 at 08:14:54 PM GMT+4:30, Rick Brownrigg <
> brownrig at ucar.edu> wrote:
>
> Hi Ehsan,
>
> That errno=12 is an out of memory error, so I suspect short2flt() is not
> the issue. Are you working with really large datasets/arrays?  That short
> array converted to floats will be 6.6GB
>
> Rick
>
> On Mon, Sep 12, 2022 at 9:30 AM Ehsan Taghizadeh via ncl-talk <
> ncl-talk at mailman.ucar.edu> wrote:
>
> Hi everybody,
> I hope you are doing well.
> I tried to read "2 metre temperature" from ERA5-Land nc file. This nc file
> include about 10 years, all months (12 months), all days (365 days), and
> all hours (24 hours) with size about 7 GB. I used following simple lines:
>
> dr = "./"
> fn = "era5land.nc"
> fin = addfile(dr+fn, "r")
> t2mera = fin->t2m(:,::-1,:)
> printVarSummary(t2mera)
>
> Output is like following lines:
>
> Type: short
> Dimensions and sizes:   [time | 87648] x [latitude | 151] x [longitude |
> 251]
> Coordinates:
>             time: [955488..1043135]
>             latitude: [25..40]
>             longitude: [40..65]
>
> However, if I use short2flt as below:
> t2meras = short2flt(fin->t2m(:,::-1,:))
>
> Following error occurred:
>
> fatal:NclMalloc Failed:[errno=12]
> fatal:Could not coerce values for operation
> fatal:["Execute.c":8637]:Execute: Error occurred at or near line 2013 in
> file /usr/local/ncl6.6/lib/ncarg/nclscripts/csm/contributed.ncl
>
> Then it mentioned the line with the "short2flt" command.
>
> I'll be thankful if I have any help.
>
>
> *-----------------------------------------------------------------------------*
> *Sincerely,*
> *Ehsan Taghizadeh*
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20220912/cc28a730/attachment.htm>