<div dir="ltr"><div class="gmail_default" style="font-size:small">Tabish,</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">It's a bit hard to tell from your script what arrays might be getting referenced multiple times on the file, but it might go faster if you read these arrays to local variables first. For example, instead of:</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"><span style="color:rgb(11,83,148);font-size:12.8px">so4_sim = a[:]->so4_a01(:,0,J,I) + a[:]->so4_a02(:,0,J,I) +a[:]->so4_a03(:,0,J,I) +a[:]->so4_a04(:,0,J,I) +a[:]->so4_a05(:,0,J,I) +a[:]->so4_a06(:,0,J,I)</span><br style="color:rgb(11,83,148);font-size:12.8px"><br style="color:rgb(11,83,148);font-size:12.8px"><span style="color:rgb(11,83,148);font-size:12.8px">so4_sim_260 = a[:]->so4_a01(:,3,J,I) + a[:]->so4_a02(:,3,J,I) +a[:]->so4_a03(:,3,J,I) +a[:]->so4_a04(:,3,J,I) +a[:]->so4_a05(:,3,J,I) +a[:]->so4_a06(:,3,J,I)</span></font><br></div><div class="gmail_default" style="font-size:small"><span style="color:rgb(11,83,148);font-family:"trebuchet ms",sans-serif;font-size:12.8px"><br></span></div><div class="gmail_default" style="font-size:small"><span style="color:rgb(11,83,148);font-family:"trebuchet ms",sans-serif;font-size:12.8px">You would have:</span></div><div class="gmail_default" style="font-size:small"><span style="color:rgb(11,83,148);font-size:12.8px"><font face="monospace, monospace"><br></font></span></div><div class="gmail_default" style="font-size:small"><span style="color:rgb(11,83,148);font-size:12.8px"><font face="monospace, monospace">so4_a01 = a[:]->so4_a01 ; store to local variable</font></span></div><div class="gmail_default" style="font-size:small"><div class="gmail_default"><span style="color:rgb(11,83,148);font-size:12.8px"><font face="monospace, monospace">so4_a02 = a[:]->so4_a02</font></span></div><div><div class="gmail_default"><span style="color:rgb(11,83,148);font-size:12.8px"><font face="monospace, monospace">so4_a03 = a[:]->so4_a03</font></span></div></div><div><div class="gmail_default"><span style="color:rgb(11,83,148);font-size:12.8px"><font face="monospace, monospace">so4_a04 = a[:]->so4_a04</font></span></div></div><div><span style="color:rgb(11,83,148);font-family:"trebuchet ms",sans-serif;font-size:12.8px">so4_sim = so4_a01(:,0,J,I) + </span><span style="color:rgb(11,83,148);font-family:"trebuchet ms",sans-serif;font-size:12.8px">so4_a02(:,0,J,I) + </span><span style="color:rgb(11,83,148);font-family:"trebuchet ms",sans-serif;font-size:12.8px">so4_a03(:,0,J,I) + </span><span style="color:rgb(11,83,148);font-family:"trebuchet ms",sans-serif;font-size:12.8px">so4_a04(:,0,J,I) </span></div><div><div class="gmail_default"><span style="color:rgb(11,83,148);font-family:"trebuchet ms",sans-serif;font-size:12.8px">so4_sim_260 = so4_a01(:,3,J,I) + </span><span style="color:rgb(11,83,148);font-family:"trebuchet ms",sans-serif;font-size:12.8px">so4_a02(:,3,J,I) + </span><span style="color:rgb(11,83,148);font-family:"trebuchet ms",sans-serif;font-size:12.8px">so4_a03(:,3,J,I) + </span><span style="color:rgb(11,83,148);font-family:"trebuchet ms",sans-serif;font-size:12.8px">so4_a04(:,3,J,I)</span></div></div><div><span style="color:rgb(11,83,148);font-family:"trebuchet ms",sans-serif;font-size:12.8px"><br></span></div></div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">Every time you do a "a[:]->" type of operation, NCL has to go out to the file, find the correct location, grab the data, and then return it back to NCL.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">If you save it to a local variable, then you won't be jumping out to the file every time.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">The other thing about this script is that it seems prone to user error. You have to be careful that you are not subscripting something incorrectly, or pulling the wrong variable off the file. Sometimes, even though something might take longer, it can be better to do something inefficiently but correctly. By this I mean instead of listing out all the variables for a summation on a line, you have a function.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">For example:</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace">function add_vars(f,var_prefix,k,j,i)</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace">local n, var_names, var_sum</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace">begin</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> var_names = var_prefix + (1,2,3,4/)</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> var_sum = 0.0</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> do n=0,3</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> var_sum = var_sum + f[:]->$var_names(n)$(:,k,j,i)</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> end do</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> return(var_sum)</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace">end</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"><br></font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace">begin</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> . . .</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> so4_sim = add_vars(a,"so4_a",0,J,I)</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace"> so4_sim_260 = add_vars(a,"so4_a",3,J,I)</font></div><div class="gmail_default" style="font-size:small"><font face="monospace, monospace">end</font></div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">Note that this is going out to the file every time, but it might be easier to read. This is something you will have to decide with your script. Also, I don't know what you are doing with all these variables afterwards, so that could make a difference how you write all this code.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">The other possibility might be to use the NCO (NetCDF operators). They are not part of NCL, but they are frequently used in lieu NCL if you need to do quick operations across files from the UNIX command line, and then write the results to a new file.</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small">--Mary</div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small"><br></div><div class="gmail_default" style="font-size:small"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 16, 2018 at 9:58 AM, Tabish Ansari via ncl-talk <span dir="ltr"><<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">Hi<br><br></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">I'm reading in some wrf data from multiple simulations and storing the relevant variables as 1d arrays. This process is taking several minutes. Is there I can changes something structurally in the algorithm to make this faster?<br><br></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">Here are the lines of code:<br><span style="color:rgb(0,0,255)"><br><span style="color:rgb(11,83,148)">print("reading wrf directories...")<br>DATADir1 = "/data2/tabish/control-run-<wbr>so4-ECMWF/"<br>DATADir2 = "/data2/tabish/best-guess_run/<wbr>"<br>DATADir3 = "/data2/tabish/what-if_run/"<br>;DATADir4 = "/data2/tabish/NH3-<wbr>sensitivity_run/"<br>FILES1 = systemfunc (" ls -1 " + DATADir1 + "wrfout_d03_2014-* ")<br>FILES2 = systemfunc (" ls -1 " + DATADir2 + "wrfout_d03_2014-* ")<br>FILES3 = systemfunc (" ls -1 " + DATADir3 + "wrfout_d03_2014-* ")<br>;FILES4 = systemfunc (" ls -1 " + DATADir4 + "wrfout_d03_2014-* ")<br>a = addfiles(FILES1,"r")<br>b = addfiles(FILES2,"r")<br>c = addfiles(FILES3,"r")<br>;d = addfiles(FILES4,"r")<br>times = wrf_times_c(a[:]->Times,0)<br>;print(times)<br>;times_crop = times(16:135)<br>I=79<br>J=144<br>so4_sim = a[:]->so4_a01(:,0,J,I) + a[:]->so4_a02(:,0,J,I) +a[:]->so4_a03(:,0,J,I) +a[:]->so4_a04(:,0,J,I) +a[:]->so4_a05(:,0,J,I) +a[:]->so4_a06(:,0,J,I)<br>no3_sim = a[:]->no3_a01(:,0,J,I) + a[:]->no3_a02(:,0,J,I) +a[:]->no3_a03(:,0,J,I) +a[:]->no3_a04(:,0,J,I) +a[:]->no3_a05(:,0,J,I) +a[:]->no3_a06(:,0,J,I)<br>nh4_sim = a[:]->nh4_a01(:,0,J,I) + a[:]->nh4_a02(:,0,J,I) +a[:]->nh4_a03(:,0,J,I) +a[:]->nh4_a04(:,0,J,I) +a[:]->nh4_a05(:,0,J,I) +a[:]->nh4_a06(:,0,J,I)<br>chl_sim = a[:]->cl_a01(:,0,J,I) + a[:]->cl_a02(:,0,J,I) +a[:]->cl_a03(:,0,J,I) +a[:]->cl_a04(:,0,J,I) +a[:]->cl_a05(:,0,J,I) +a[:]->cl_a06(:,0,J,I)<br>oc_sim = a[:]->oc_a01(:,0,J,I) + a[:]->oc_a02(:,0,J,I) +a[:]->oc_a03(:,0,J,I) +a[:]->oc_a04(:,0,J,I) +a[:]->oc_a05(:,0,J,I) +a[:]->oc_a06(:,0,J,I)<br>bc_sim = a[:]->bc_a01(:,0,J,I) + a[:]->bc_a02(:,0,J,I) +a[:]->bc_a03(:,0,J,I) +a[:]->bc_a04(:,0,J,I) +a[:]->bc_a05(:,0,J,I) +a[:]->bc_a06(:,0,J,I)<br>co_sim = a[:]->co(:,0,J,I)<br>no2_sim = a[:]->no2(:,0,J,I)<br>so2_sim = a[:]->so2(:,0,J,I)<br>o3_sim = a[:]->o3(:,0,J,I)<br><br>so4_sim_260 = a[:]->so4_a01(:,3,J,I) + a[:]->so4_a02(:,3,J,I) +a[:]->so4_a03(:,3,J,I) +a[:]->so4_a04(:,3,J,I) +a[:]->so4_a05(:,3,J,I) +a[:]->so4_a06(:,3,J,I)<br>no3_sim_260 = a[:]->no3_a01(:,3,J,I) + a[:]->no3_a02(:,3,J,I) +a[:]->no3_a03(:,3,J,I) +a[:]->no3_a04(:,3,J,I) +a[:]->no3_a05(:,3,J,I) +a[:]->no3_a06(:,3,J,I)<br>nh4_sim_260 = a[:]->nh4_a01(:,3,J,I) + a[:]->nh4_a02(:,3,J,I) +a[:]->nh4_a03(:,3,J,I) +a[:]->nh4_a04(:,3,J,I) +a[:]->nh4_a05(:,3,J,I) +a[:]->nh4_a06(:,3,J,I)<br>chl_sim_260 = a[:]->cl_a01(:,3,J,I) + a[:]->cl_a02(:,3,J,I) +a[:]->cl_a03(:,3,J,I) +a[:]->cl_a04(:,3,J,I) +a[:]->cl_a05(:,3,J,I) +a[:]->cl_a06(:,3,J,I)<br>oc_sim_260 = a[:]->oc_a01(:,3,J,I) + a[:]->oc_a02(:,3,J,I) +a[:]->oc_a03(:,3,J,I) +a[:]->oc_a04(:,3,J,I) +a[:]->oc_a05(:,3,J,I) +a[:]->oc_a06(:,3,J,I)<br><br><br>so4_sim_b = b[:]->so4_a01(:,0,J,I) + b[:]->so4_a02(:,0,J,I) +b[:]->so4_a03(:,0,J,I) +b[:]->so4_a04(:,0,J,I) +b[:]->so4_a05(:,0,J,I) +b[:]->so4_a06(:,0,J,I)<br>no3_sim_b = b[:]->no3_a01(:,0,J,I) + b[:]->no3_a02(:,0,J,I) +b[:]->no3_a03(:,0,J,I) +b[:]->no3_a04(:,0,J,I) +b[:]->no3_a05(:,0,J,I) +b[:]->no3_a06(:,0,J,I)<br>nh4_sim_b = b[:]->nh4_a01(:,0,J,I) + b[:]->nh4_a02(:,0,J,I) +b[:]->nh4_a03(:,0,J,I) +b[:]->nh4_a04(:,0,J,I) +b[:]->nh4_a05(:,0,J,I) +b[:]->nh4_a06(:,0,J,I)<br>chl_sim_b = b[:]->cl_a01(:,0,J,I) + b[:]->cl_a02(:,0,J,I) +b[:]->cl_a03(:,0,J,I) +b[:]->cl_a04(:,0,J,I) +b[:]->cl_a05(:,0,J,I) +b[:]->cl_a06(:,0,J,I)<br>oc_sim_b = b[:]->oc_a01(:,0,J,I) + b[:]->oc_a02(:,0,J,I) +b[:]->oc_a03(:,0,J,I) +b[:]->oc_a04(:,0,J,I) +b[:]->oc_a05(:,0,J,I) +b[:]->oc_a06(:,0,J,I)<br>bc_sim_b = b[:]->bc_a01(:,0,J,I) + b[:]->bc_a02(:,0,J,I) +b[:]->bc_a03(:,0,J,I) +b[:]->bc_a04(:,0,J,I) +b[:]->bc_a05(:,0,J,I) +b[:]->bc_a06(:,0,J,I)<br>co_sim_b = b[:]->co(:,0,J,I)<br>no2_sim_b = b[:]->no2(:,0,J,I)<br>so2_sim_b = b[:]->so2(:,0,J,I)<br>o3_sim_b = b[:]->o3(:,0,J,I)<br>so4_sim_260_b = b[:]->so4_a01(:,3,J,I) + b[:]->so4_a02(:,3,J,I) +b[:]->so4_a03(:,3,J,I) +b[:]->so4_a04(:,3,J,I) +b[:]->so4_a05(:,3,J,I) +b[:]->so4_a06(:,3,J,I)<br>no3_sim_260_b = b[:]->no3_a01(:,3,J,I) + b[:]->no3_a02(:,3,J,I) +b[:]->no3_a03(:,3,J,I) +b[:]->no3_a04(:,3,J,I) +b[:]->no3_a05(:,3,J,I) +b[:]->no3_a06(:,3,J,I)<br>nh4_sim_260_b = b[:]->nh4_a01(:,3,J,I) + b[:]->nh4_a02(:,3,J,I) +b[:]->nh4_a03(:,3,J,I) +b[:]->nh4_a04(:,3,J,I) +b[:]->nh4_a05(:,3,J,I) +b[:]->nh4_a06(:,3,J,I)<br>chl_sim_260_b = b[:]->cl_a01(:,3,J,I) + b[:]->cl_a02(:,3,J,I) +b[:]->cl_a03(:,3,J,I) +b[:]->cl_a04(:,3,J,I) +b[:]->cl_a05(:,3,J,I) +b[:]->cl_a06(:,3,J,I)<br>oc_sim_260_b = b[:]->oc_a01(:,3,J,I) + b[:]->oc_a02(:,3,J,I) +b[:]->oc_a03(:,3,J,I) +b[:]->oc_a04(:,3,J,I) +b[:]->oc_a05(:,3,J,I) +b[:]->oc_a06(:,3,J,I)<br><br>so4_sim_c = c[:]->so4_a01(:,0,J,I) + c[:]->so4_a02(:,0,J,I) +c[:]->so4_a03(:,0,J,I) +c[:]->so4_a04(:,0,J,I) +c[:]->so4_a05(:,0,J,I) +c[:]->so4_a06(:,0,J,I)<br>no3_sim_c = c[:]->no3_a01(:,0,J,I) + c[:]->no3_a02(:,0,J,I) +c[:]->no3_a03(:,0,J,I) +c[:]->no3_a04(:,0,J,I) +c[:]->no3_a05(:,0,J,I) +c[:]->no3_a06(:,0,J,I)<br>nh4_sim_c = c[:]->nh4_a01(:,0,J,I) + c[:]->nh4_a02(:,0,J,I) +c[:]->nh4_a03(:,0,J,I) +c[:]->nh4_a04(:,0,J,I) +c[:]->nh4_a05(:,0,J,I) +c[:]->nh4_a06(:,0,J,I)<br>chl_sim_c = c[:]->cl_a01(:,0,J,I) + c[:]->cl_a02(:,0,J,I) +c[:]->cl_a03(:,0,J,I) +c[:]->cl_a04(:,0,J,I) +c[:]->cl_a05(:,0,J,I) +c[:]->cl_a06(:,0,J,I)<br>oc_sim_c = c[:]->oc_a01(:,0,J,I) + c[:]->oc_a02(:,0,J,I) +c[:]->oc_a03(:,0,J,I) +c[:]->oc_a04(:,0,J,I) +c[:]->oc_a05(:,0,J,I) +c[:]->oc_a06(:,0,J,I)<br>bc_sim_c = c[:]->bc_a01(:,0,J,I) + c[:]->bc_a02(:,0,J,I) +c[:]->bc_a03(:,0,J,I) +c[:]->bc_a04(:,0,J,I) +c[:]->bc_a05(:,0,J,I) +c[:]->bc_a06(:,0,J,I)<br>co_sim_c = c[:]->co(:,0,J,I)<br>no2_sim_c = c[:]->no2(:,0,J,I)<br>so2_sim_c = c[:]->so2(:,0,J,I)<br>o3_sim_c = c[:]->o3(:,0,J,I)<br><br>so4_sim_260_c = c[:]->so4_a01(:,3,J,I) + c[:]->so4_a02(:,3,J,I) +c[:]->so4_a03(:,3,J,I) +c[:]->so4_a04(:,3,J,I) +c[:]->so4_a05(:,3,J,I) +c[:]->so4_a06(:,3,J,I)<br>no3_sim_260_c = c[:]->no3_a01(:,3,J,I) + c[:]->no3_a02(:,3,J,I) +c[:]->no3_a03(:,3,J,I) +c[:]->no3_a04(:,3,J,I) +c[:]->no3_a05(:,3,J,I) +c[:]->no3_a06(:,3,J,I)<br>nh4_sim_260_c = c[:]->nh4_a01(:,3,J,I) + c[:]->nh4_a02(:,3,J,I) +c[:]->nh4_a03(:,3,J,I) +c[:]->nh4_a04(:,3,J,I) +c[:]->nh4_a05(:,3,J,I) +c[:]->nh4_a06(:,3,J,I)<br>chl_sim_260_c = c[:]->cl_a01(:,3,J,I) + c[:]->cl_a02(:,3,J,I) +c[:]->cl_a03(:,3,J,I) +c[:]->cl_a04(:,3,J,I) +c[:]->cl_a05(:,3,J,I) +c[:]->cl_a06(:,3,J,I)<br>oc_sim_260_c = c[:]->oc_a01(:,3,J,I) + c[:]->oc_a02(:,3,J,I) +c[:]->oc_a03(:,3,J,I) +c[:]->oc_a04(:,3,J,I) +c[:]->oc_a05(:,3,J,I) +c[:]->oc_a06(:,3,J,I)<br><br>;so4_sim_d = d[:]->so4_a01(:,0,J,I) + d[:]->so4_a02(:,0,J,I) +d[:]->so4_a03(:,0,J,I) +d[:]->so4_a04(:,0,J,I) +d[:]->so4_a05(:,0,J,I) +d[:]->so4_a06(:,0,J,I)<br>;no3_sim_d = d[:]->no3_a01(:,0,J,I) + d[:]->no3_a02(:,0,J,I) +d[:]->no3_a03(:,0,J,I) +d[:]->no3_a04(:,0,J,I) +d[:]->no3_a05(:,0,J,I) +d[:]->no3_a06(:,0,J,I)<br>;nh4_sim_d = d[:]->nh4_a01(:,0,J,I) + d[:]->nh4_a02(:,0,J,I) +d[:]->nh4_a03(:,0,J,I) +d[:]->nh4_a04(:,0,J,I) +d[:]->nh4_a05(:,0,J,I) +d[:]->nh4_a06(:,0,J,I)<br><br>;so4_sim_260_d = d[:]->so4_a01(:,3,J,I) + d[:]->so4_a02(:,3,J,I) +d[:]->so4_a03(:,3,J,I) +d[:]->so4_a04(:,3,J,I) +d[:]->so4_a05(:,3,J,I) +d[:]->so4_a06(:,3,J,I)<br>;no3_sim_260_d = d[:]->no3_a01(:,3,J,I) + d[:]->no3_a02(:,3,J,I) +d[:]->no3_a03(:,3,J,I) +d[:]->no3_a04(:,3,J,I) +d[:]->no3_a05(:,3,J,I) +d[:]->no3_a06(:,3,J,I)<br>;nh4_sim_260_d = d[:]->nh4_a01(:,3,J,I) + d[:]->nh4_a02(:,3,J,I) +d[:]->nh4_a03(:,3,J,I) +d[:]->nh4_a04(:,3,J,I) +d[:]->nh4_a05(:,3,J,I) +d[:]->nh4_a06(:,3,J,I)<br><br>print("Stored relevant pollutant arrays.")</span></span><br><br></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">Please note that I've commented out DataDir4 and its related arrays. When I try to load that too, I get a segmentation fault.<br></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">I just need those arrays for further processing and ultimately for a combined time-series plot. Is there anyway to speed this up, and probably also read in DataDir4 arrays without a seg fault?<br><br></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">Thanks in advance!<br></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif"><br></div><div class="gmail_default" style="font-family:trebuchet ms,sans-serif">Tabish<br><br><br clear="all"></div><div><div class="m_8987311986348572150gmail_signature"><div dir="ltr"><div><div><font size="1"><span style="font-family:tahoma,sans-serif">Tabish U Ansari<br></span></font></div><font size="1"><span style="font-family:tahoma,sans-serif">PhD student, Lancaster Environment Center<br></span></font></div><font size="1"><span style="font-family:tahoma,sans-serif">Lancaster Univeristy<br> <span>Bailrigg</span>, <span>Lancaster</span>, <br><span>LA1 4YW</span>, <span>United Kingdom</span></span></font><br></div></div></div>
</div>
<br>______________________________<wbr>_________________<br>
ncl-talk mailing list<br>
<a href="mailto:ncl-talk@ucar.edu">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br>
<a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" rel="noreferrer" target="_blank">http://mailman.ucar.edu/<wbr>mailman/listinfo/ncl-talk</a><br>
<br></blockquote></div><br></div>