<div dir="ltr"><div>Ehsan, a standard strategy for large array problems is to add a loop and process the data one time step at a time. Do not try to read the whole array into memory at any time. This assumes that you are interpolating across X and Y, but not time.</div><div><br></div><div>open input files</div><div>start a new output file</div><div>do itime = 0, ntimes-1</div><div> read grids for one time step</div><div> interpolate</div><div> write result grids for one time step</div><div>end do</div><div>close files</div><div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Sep 12, 2022 at 10:54 AM Ehsan Taghizadeh via ncl-talk <<a href="mailto:ncl-talk@mailman.ucar.edu">ncl-talk@mailman.ucar.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div><div style="font-size:16px;font-family:"bookman old style","new york",times,serif"><div><div dir="ltr" style="font-family:"bookman old style","new york",times,serif">Dear Rick,</div><div dir="ltr" style="font-family:"bookman old style","new york",times,serif">The nc file is included a domain including (25-40N, 40-65E) which was downloaded with the attach api script (temp2m09_18.py). As you mentioned the downloaded file has a size more than 6 GB, because of the domain and long term hourly data. Should I break the time period to have smaller size?</div><div dir="ltr" style="font-family:"bookman old style","new york",times,serif"><br></div><div dir="ltr" style="font-family:"bookman old style","new york",times,serif">On the other hand I would like to interpolate temperature on stations location with the attached script (if I assume short2flt has fixed). How could I write these long data (34*10*365), which 34 is number of stations, 10*365 is number of days?</div><div dir="ltr">In other words, the desired quantity "<span style="color:rgb(0,0,0);font-size:16px"><font face="courier new, courier, monaco, monospace, sans-serif" style="background-color:inherit">t2m_int2p_mima</font></span>" has 3 dimensions; 1st dimension is related to (min, max, avg), 2nd dimension is related to days which is about 10*365,</div><div dir="ltr"><div><div style="font-family:"bookman old style","new york",times,serif">and 3rd dimension is related to stations which is about 34 stations.</div><div><span style="font-family:"bookman old style","new york",times,serif">May I ask how could I write "</span><font face="courier new, courier, monaco, monospace, sans-serif">t2m_int2p_mima</font>" in a table with headers same as below:</div><div style="font-family:"bookman old style","new york",times,serif"><br></div><div> "<font face="courier new, courier, monaco, monospace, sans-serif">stid min(day1) max(day1) avg(day1) min(day2) max(day2) avg(day2) ... min(day10*365) max(day10*365) avg(day10*365)</font>"</div></div><br></div><div dir="ltr">I'll be thankful for any help.</div><div style="font-family:"bookman old style","new york",times,serif"><br></div><div style="font-family:"bookman old style","new york",times,serif"><div dir="ltr" style="font-family:"old times",serif;font-size:16px"><div><div dir="ltr" style="color:rgb(0,0,0);font-family:"old times",serif;font-size:16px"><i><font size="2">-----------------------------------------------------------------------------</font></i></div><div dir="ltr" style="color:rgb(0,0,0);font-family:"old times",serif;font-size:16px"><i><font size="2">Sincerely,</font></i></div><div dir="ltr" style="color:rgb(0,0,0);font-family:"old times",serif;font-size:16px"><i><font size="2">Ehsan Taghizadeh<br></font></i></div></div></div></div></div><div style="font-family:"bookman old style","new york",times,serif"><br></div><div style="font-family:"bookman old style","new york",times,serif"><br></div>
</div><div id="m_-3950798734659794963yahoo_quoted_3177203399">
<div style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:13px;color:rgb(38,40,42)">
<div>
On Monday, September 12, 2022 at 08:14:54 PM GMT+4:30, Rick Brownrigg <<a href="mailto:brownrig@ucar.edu" target="_blank">brownrig@ucar.edu</a>> wrote:</div>
<div><br></div>
<div><div id="m_-3950798734659794963yiv9955788550"><div><div dir="ltr"><div>Hi Ehsan,</div><div><br clear="none"></div><div>That errno=12 is an out of memory error, so I suspect short2flt() is not the issue. Are you working with really large datasets/arrays? That short array converted to floats will be 6.6GB</div><div><br clear="none"></div><div>Rick<br clear="none"></div></div><br clear="none"><div><div id="m_-3950798734659794963yiv9955788550yqt58430"><div dir="ltr">On Mon, Sep 12, 2022 at 9:30 AM Ehsan Taghizadeh via ncl-talk <<a rel="nofollow noopener noreferrer" shape="rect" href="mailto:ncl-talk@mailman.ucar.edu" target="_blank">ncl-talk@mailman.ucar.edu</a>> wrote:<br clear="none"></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div><div style="font-family:"bookman old style","new york",times,serif;font-size:16px"><div><div dir="ltr" style="font-size:16px;font-family:"bookman old style","new york",times,serif">Hi everybody,</div><div dir="ltr" style="font-size:16px;font-family:"bookman old style","new york",times,serif">I hope you are doing well.</div><div dir="ltr" style="font-size:16px;font-family:"bookman old style","new york",times,serif">I tried to read "2 metre temperature" from ERA5-Land nc file. This nc file include about 10 years, all months (12 months), all days (365 days), and all hours (24 hours) with size about 7 GB. I used following simple lines:</div><div dir="ltr" style="font-size:16px;font-family:"bookman old style","new york",times,serif"><br clear="none"></div><div dir="ltr" style="font-size:16px"><font face="courier new, courier, monaco, monospace, sans-serif">dr = "./"</font></div><div dir="ltr" style="font-size:16px"><font face="courier new, courier, monaco, monospace, sans-serif">fn = "<a rel="nofollow noopener noreferrer" shape="rect" href="http://era5land.nc" target="_blank">era5land.nc</a>"</font></div><div dir="ltr" style="font-size:16px"><font face="courier new, courier, monaco, monospace, sans-serif">fin = addfile(dr+fn, "r")</font></div><div dir="ltr" style="font-size:16px"><font face="courier new, courier, monaco, monospace, sans-serif">t2mera = fin->t2m(:,::-1,:)</font></div><div dir="ltr" style="font-size:16px"><font face="courier new, courier, monaco, monospace, sans-serif">printVarSummary(t2mera)</font></div><div dir="ltr" style="font-size:16px;font-family:"bookman old style","new york",times,serif"><br clear="none"></div><div dir="ltr" style="font-size:16px;font-family:"bookman old style","new york",times,serif">Output is like following lines:</div><div dir="ltr" style="font-size:16px;font-family:"bookman old style","new york",times,serif"><br clear="none"></div><div dir="ltr" style="font-size:16px"><div><div dir="ltr"><font face="courier new, courier, monaco, monospace, sans-serif">Type: short</font></div><div><font face="courier new, courier, monaco, monospace, sans-serif">Dimensions and sizes: [time | 87648] x [latitude | 151] x [longitude | 251]</font></div><div><font face="courier new, courier, monaco, monospace, sans-serif">Coordinates:</font></div><div><font face="courier new, courier, monaco, monospace, sans-serif"> time: [955488..1043135]</font></div><div><font face="courier new, courier, monaco, monospace, sans-serif"> latitude: [25..40]</font></div><div><font face="courier new, courier, monaco, monospace, sans-serif"> longitude: [40..65]</font></div></div><br clear="none"></div><div dir="ltr" style="font-size:16px;font-family:"bookman old style","new york",times,serif">However, if I use short2flt as below:</div><div dir="ltr" style="font-size:16px"><font face="courier new, courier, monaco, monospace, sans-serif">t<span style="color:rgb(0,0,0);font-size:16px">2meras = short2flt(fin->t2m(:,::-1,:))</span></font></div><div dir="ltr" style="font-size:16px;font-family:"bookman old style","new york",times,serif"><span><span style="color:rgb(0,0,0);font-family:"bookman old style","new york",times,serif;font-size:16px"><br clear="none"></span></span></div><div dir="ltr">Following error occurred:</div><div dir="ltr" style="font-size:16px;font-family:"bookman old style","new york",times,serif"><span><span style="color:rgb(0,0,0);font-family:"bookman old style","new york",times,serif;font-size:16px"><br clear="none"></span></span></div><div dir="ltr" style="font-size:16px"><div><div><font face="courier new, courier, monaco, monospace, sans-serif">fatal:NclMalloc Failed:[errno=12]</font></div><div><font face="courier new, courier, monaco, monospace, sans-serif">fatal:Could not coerce values for operation</font></div><div><font face="courier new, courier, monaco, monospace, sans-serif">fatal:["Execute.c":8637]:Execute: Error occurred at or near line 2013 in file /usr/local/ncl6.6/lib/ncarg/nclscripts/csm/contributed.ncl</font></div></div><div style="font-family:"bookman old style","new york",times,serif"><br clear="none"></div><div dir="ltr"><span style="font-family:"bookman old style","new york",times,serif">Then it mentioned the line with the "</span><font face="courier new, courier, monaco, monospace, sans-serif">short2flt</font>" command.</div><div dir="ltr" style="font-family:"bookman old style","new york",times,serif"><br clear="none"></div><div dir="ltr" style="font-family:"bookman old style","new york",times,serif">I'll be thankful if I have any help.<br clear="none"></div></div><div style="font-size:16px;font-family:"bookman old style","new york",times,serif"><br clear="none"></div><div style="font-size:16px;font-family:"bookman old style","new york",times,serif"><div dir="ltr" style="font-family:"old times",serif;font-size:16px"><div><div dir="ltr" style="color:rgb(0,0,0);font-family:"old times",serif;font-size:16px"><i><font size="2">-----------------------------------------------------------------------------</font></i></div><div dir="ltr" style="color:rgb(0,0,0);font-family:"old times",serif;font-size:16px"><i><font size="2">Sincerely,</font></i></div><div dir="ltr" style="color:rgb(0,0,0);font-family:"old times",serif;font-size:16px"><i><font size="2">Ehsan Taghizadeh</font></i></div></div></div></div></div></div></div></blockquote></div></div></div></div></div></div></div></div>
</blockquote></div></div>