<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:445588541;
mso-list-type:hybrid;
mso-list-template-ids:1287934552 505428192 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1
{mso-list-id:1426223156;
mso-list-type:hybrid;
mso-list-template-ids:337426194 -1932261458 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:35.25pt;
text-indent:-18.0pt;
font-family:Wingdings;
mso-fareast-font-family:"Times New Roman";
mso-bidi-font-family:Calibri;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:71.25pt;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:107.25pt;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:143.25pt;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:179.25pt;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:215.25pt;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:251.25pt;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:287.25pt;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:323.25pt;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-AU" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi Dennis<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks for the reply; I’m already saving the profiles to file for later work, but I need to do this for 3 model experiments and ~15 CMIP5 models. The script is slow but low on memory and CPU burden, so maybe I just need to dip my toe into
parallel processing…<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:18.0pt">>I'm not sure how the underlying netCDF software would implement something like<o:p></o:p></p>
<p class="MsoListParagraph"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:17.25pt"><span lang="EN-US">>Tprof = fi->T(pInd(<b><span style="color:red">:</span></b>,0),<b>:</b>,pInd(<b><span style="color:red">:</span></b>,1),pInd(<b><span style="color:red">:</span></b>,2))
</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">For future reference, it returns a HUGE array, because each time index will have every profile’s lat and every profile’s lon, so the array is (nprofile x nlev nprofile x nprofile). With a small number of profiles (i.e. less than the time,
lat, and lon dimensions) it would be fine, but I have ~50,000.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Will<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Dennis Shea <shea@ucar.edu><br>
<b>Date: </b>Wednesday, 17 October 2018 at 12:45 PM<br>
<b>To: </b>Will Hobbs <will.hobbs@utas.edu.au><br>
<b>Cc: </b>Ncl-talk <ncl-talk@ucar.edu><br>
<b>Subject: </b>Re: [ncl-talk] Advice on extracting point locations from array<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">'Brain Trust' ... "great idea" ===> Nope ... Sorry<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">---<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">IMHO: <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I don't think the loop can be replaced by array operations. I'm not sure how the underlying netCDF software would implement something like<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Tprof = fi->T(pInd(<b><span style="color:red">:</span></b>,0),<b>:</b>,pInd(<b><span style="color:red">:</span></b>,1),pInd(<b><span style="color:red">:</span></b>,2))
</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">---</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> nprofile = ...</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> nlev = ...</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> Tprof = <b>new</b>((/<b><span style="color:blue">nprofile,nlev</span></b>/),float) ;
<span style="color:fuchsia">reorder</span> from your code<br>
begTime = <b>get_cpu_time</b>()<br>
do i = 0, nprofile-1<br>
Tprof(i,:) = fi->T(pInd(i,0),:,pInd(i,1),pInd(i,2)) ;pInd is an array with coord triplets<br>
end do<br>
<br>
printVarSummary(Tprof)<br>
printMinMax(Tprof,0)<br>
print("Tprof generation time: " + (<b>get_cpu_time</b>() - begTime)+" seconds; nprofile="+nprofile)<br>
print("---")</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">---</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">If it takes a long time, I'd suggest saving the created variable(s) in a netCDF file. Untested:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">;------------------<br>
; netCDF<br>
;-----------------<br>
netCDF = True<br>
if (netCDF) then<br>
profile = <b>ispan</b>(0,nprofile-1,1) ; or, ispan(1,nprofile,1) <br>
profile@long_name = "Profile Number"<br>
profile!0 = "profile"<br>
profile&profile = profile<br>
<br>
TIME = fi->time(pInd(:,0))<br>
LAT = fi->lat(pInd(:,1))<br>
LON = fi->lon(pInd(:,2))<br>
<br>
Tprof!0= "profile"<br>
TIME!0 = "profile"<br>
LAT!0 = "profile"<br>
LON!0 = "profile"<br>
Tprof&profile = profile<br>
TIME&profile = profile<br>
LAT&profile = profile<br>
LON&profile = profile<br>
<br>
dir_nc = "./"<br>
fil_nc = "<a href="http://FOO.nc">FOO.nc</a>"<br>
pth_nc = dir_nc + fil_nc<br>
print(pth_nc)<br>
<br>
<b>system</b>("/bin/rm -f "+pth_nc) ; remove any pre-existing file<br>
ncdf = <b>addfile</b>(pth_nc ,<b><span style="color:blue">"c"</span></b>) ; open output netCDF file<br>
<br>
;===================================================================<br>
; Create global attributes of the file (optional)<br>
;===================================================================<br>
fAtt = True ; assign file attributes<br>
fAtt@title = "Selected Profiles"<br>
;fAtt@source_info = "..."<br>
fAtt@Conventions = "None"<br>
fAtt@creation_date = <b>systemfunc </b>("date")<br>
<b>fileattdef(</b> ncdf, fAtt ) ; copy file attributes<br>
<br>
;===================================================================<br>
; Make profile an UNLIMITED dimension; recommended for most applications</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">; Allows for future profiles to be appended</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">;===================================================================<br>
<b>filedimdef</b>f(ncdf,"profile",-1,True)<br>
<br>
;===================================================================<br>
; output variable(s) directly; NCL will call appropriate functions<br>
;===================================================================<br>
ncdf->TIME = TIME<br>
ncdf->LAT = LAT<br>
ncdf->LON = LON<br>
ncdf->TPROF = Tprof<br>
end if ; netCDF</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Mon, Oct 15, 2018 at 11:53 PM Will Hobbs <<a href="mailto:will.hobbs@utas.edu.au" target="_blank">will.hobbs@utas.edu.au</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Hi all</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">This one is for the NCL ‘Brains Trust’. I’m extracting a large number of ocean profiles from a 4-d array (dimensioned time,depth,lat,lon), using coordinate triplets
based on time,lat and lon, (i.e. a 2-d array of indices dimensioned nprofile, 3), but it’s a slow process and I wonder if anyone has any smart ideas on speeding it up.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">At the moment (partly to manage memory), I’m reading each profile in a loop, viz:</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Tprof = new((/nlev,nprofile/),float)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">do i = 0, nprofile-1</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> Tprof(:,i) = fi->T(pInd(i,0),:,pInd(i,1),pInd(i,2)) ;pInd is an array with coord triplets</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">end do</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Obviously one obvious way of speeding things up would be to read the entire array of input data and extract the locations, to avoid multiple i/o calls. What I
really want to do though is somehow get rid of the loop entirely. Is there any way of extracting coordinate pairs (or in this case triplets) without looping through each profile?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">The only way I can think of is to turn the input array (T in my example above) into a 1-d array, and somehow convert ‘pInd’ into the elements of that 1-d array
– this seems fraught with opportunity for error though. Catastrophic error I can deal with – it’s the sneaky, inobvious error that scares me….</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Hoping someone has a great idea….</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Will</span><o:p></o:p></p>
</div>
<p style="line-height:10.0pt"><span style="font-size:10.0pt"><br>
<br>
University of Tasmania Electronic Communications Policy (December, 2014). <br>
This email is confidential, and is for the intended recipient only. Access, disclosure, copying, distribution, or reliance on any of it by anyone outside the intended recipient organisation is prohibited and may be a criminal offence. Please delete if obtained
in error and email confirmation to the sender. The views expressed in this email are not necessarily the views of the University of Tasmania, unless clearly intended otherwise.
<o:p></o:p></span></p>
</div>
<p class="MsoNormal">_______________________________________________<br>
ncl-talk mailing list<br>
<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br>
<a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" target="_blank">http://mailman.ucar.edu/mailman/listinfo/ncl-talk</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</body>
</html>