[ncl-talk] Fwd: efficient way of reading numbered directories?
Will Hobbs
will.hobbs at utas.edu.au
Thu Feb 21 16:40:03 MST 2019
If the variable in the files is the same dimension, you should be able to to use addfiles to read all of them without looping (I think). In this example I’ve put all your directory paths into a single string array, DATADir. Depending on how many files are in each directory you **may** hit file list limits with addfiles.
I’m also assuming that you’re looping though i and j to get local sums – here I’ve used the range of i and j as ‘imin, imax, jmin and jmax
FILES = systemfunc(“ls -1 {“+str_join(DATADir,”,”)+”}/subset_d03_2014-*”) ;uses the unix {} operator to search all the required directories
fi = addfiles(FILES,”r”)
ListSetType(fi, “join”)
tmp_data = fi[:]->PM2_5_DRY(:,0,imin:imax,jmin:jmax) ;read all files across the range of i and j, tmp_data is dimensioned (nfile, Ni, Nj)
pm25_sim_a = dim_sum_n(tmp_data, (/1,2/)) ;sum all i, j dimensions
pm25_sim_a = pm25_sim_a/12.
That gives you what I **think** you’re after without loops....
Will
From: ncl-talk <ncl-talk-bounces at ucar.edu> on behalf of Dennis Shea <shea at ucar.edu>
Date: Friday, 22 February 2019 at 10:21 AM
To: Tabish Ansari <tabishumaransari at gmail.com>
Cc: "ncl-talk at ucar.edu USERS" <ncl-talk at ucar.edu>
Subject: Re: [ncl-talk] Fwd: efficient way of reading numbered directories?
Nothin of which I am aware. You can nt automatically create variables on the left hand side.
Maybe Adam has an idea..
pm25_sim_a1 = a1[:]->PM2_5_DRY(:,0,145,79)
On Thu, Feb 21, 2019 at 3:53 PM Tabish Ansari <tabishumaransari at gmail.com<mailto:tabishumaransari at gmail.com>> wrote:
Tabish U Ansari
PhD student, Lancaster Environment Center
Lancaster Univeristy
Bailrigg, Lancaster,
LA1 4YW, United Kingdom
---------- Forwarded message ---------
From: Tabish Ansari <tabishumaransari at gmail.com<mailto:tabishumaransari at gmail.com>>
Date: Thu, 21 Feb 2019 at 22:52
Subject: Re: [ncl-talk] efficient way of reading numbered directories?
To: Adam Phillips <asphilli at ucar.edu<mailto:asphilli at ucar.edu>>
Cc: Dennis Shea <shea at ucar.edu<mailto:shea at ucar.edu>>
Hi Adam, Dennis,
Thanks very much.
But this only solves part of the problem that is reading in the directories.
I still have these other lines which have to be repeated 60 times with changing numbers as part of variable names.
FILES1 = systemfunc (" ls -1 " + DATADir1 + "subset_d03_2014-* ")
FILES2 = systemfunc (" ls -1 " + DATADir2 + "subset_d03_2014-* ")
FILES3 = systemfunc (" ls -1 " + DATADir3 + "subset_d03_2014-* ")
FILES4 = systemfunc (" ls -1 " + DATADir4 + "subset_d03_2014-* ")
FILES5 = systemfunc (" ls -1 " + DATADir5 + "subset_d03_2014-* ")
FILES6 = systemfunc (" ls -1 " + DATADir6 + "subset_d03_2014-* ")
FILES7 = systemfunc (" ls -1 " + DATADir7 + "subset_d03_2014-* ")
..... and so on up to FILES60
a1 = addfiles(FILES1,"r")
a2 = addfiles(FILES2,"r")
a3 = addfiles(FILES3,"r")
a4 = addfiles(FILES4,"r")
a5 = addfiles(FILES5,"r")
a6 = addfiles(FILES6,"r")
a7 = addfiles(FILES7,"r")
a8 = addfiles(FILES8,"r")
.... and so on up to a60
pm25_sim_a1 = a1[:]->PM2_5_DRY(:,0,145,79)
pm25_sim_a2 = a2[:]->PM2_5_DRY(:,0,145,79)
pm25_sim_a3 = a3[:]->PM2_5_DRY(:,0,145,79)
pm25_sim_a4 = a4[:]->PM2_5_DRY(:,0,145,79)
pm25_sim_a5 = a5[:]->PM2_5_DRY(:,0,145,79)
pm25_sim_a6 = a6[:]->PM2_5_DRY(:,0,145,79)
pm25_sim_a7 = a7[:]->PM2_5_DRY(:,0,145,79)
..... and so on up to
pm25_sim_a60 = a60[:]->PM2_5_DRY(:,0,145,79)
Further inside a loop...
pm25_sim_a1 = pm25_sim_a1 + a1[:]->PM2_5_DRY(:,0,J,I)
pm25_sim_a2 = pm25_sim_a2 + a2[:]->PM2_5_DRY(:,0,J,I)
pm25_sim_a3 = pm25_sim_a3 + a3[:]->PM2_5_DRY(:,0,J,I)
pm25_sim_a4 = pm25_sim_a4 + a4[:]->PM2_5_DRY(:,0,J,I)
pm25_sim_a5 = pm25_sim_a5 + a5[:]->PM2_5_DRY(:,0,J,I)
pm25_sim_a6 = pm25_sim_a6 + a6[:]->PM2_5_DRY(:,0,J,I)
and so on... up to
pm25_sim_a60 = pm25_sim_a60 + a60[:]->PM2_5_DRY(:,0,J,I)
pm25_sim_a1 = pm25_sim_a1 / 12.0
pm25_sim_a2 = pm25_sim_a2 / 12.0
pm25_sim_a3 = pm25_sim_a3 / 12.0
pm25_sim_a4 = pm25_sim_a4 / 12.0
pm25_sim_a5 = pm25_sim_a5 / 12.0
pm25_sim_a6 = pm25_sim_a6 / 12.0
pm25_sim_a7 = pm25_sim_a7 / 12.0
... and so on up to
pm25_sim_a60 = pm25_sim_a60 / 12.0
Is there a way to deal with that in a loop?
Cheers,
Tabish
Tabish U Ansari
PhD student, Lancaster Environment Center
Lancaster Univeristy
Bailrigg, Lancaster,
LA1 4YW, United Kingdom
On Thu, 21 Feb 2019 at 22:39, Adam Phillips <asphilli at ucar.edu<mailto:asphilli at ucar.edu>> wrote:
Dennis: Agreed, your way is more elegant.
On Thu, Feb 21, 2019 at 3:37 PM Dennis Shea <shea at ucar.edu<mailto:shea at ucar.edu>> wrote:
offline!
I was just typing when I saw Adam's response.
Originally, I was going to use a 'do loop' but then
N = 60
dirRoot = "/mnt/rasher/"+N+"sensruns/run" +ispan(1,N,1)+"/"
print(dirRoot)
On Thu, Feb 21, 2019 at 3:30 PM Adam Phillips <asphilli at ucar.edu<mailto:asphilli at ucar.edu>> wrote:
Hi Tabish,
Yes, a do loop could be used:
DATADir = "/mnt/rasher/60sensruns/run"
do gg = 1,60
FILES1 = systemfunc (" ls -1 " + DATADir +gg+ "subset_d03_2014-* ")
a1 = addfiles(FILES1,"r")
.....
Also note, you could set up a string array that houses the directories using a do loop
DATADir = new(60,string)
do gg = 1,dimsizes(DATADir)-1
DATADir(gg-1) = "/mnt/rasher/60sensruns/run"+gg
end do
Adam
On Thu, Feb 21, 2019 at 1:15 PM Tabish Ansari <tabishumaransari at gmail.com<mailto:tabishumaransari at gmail.com>> wrote:
Hi
I am currently reading in files in a very inefficient way from 60 directories:
DATADir1 = "/mnt/rasher/60sensruns/run1/"
DATADir2 = "/mnt/rasher/60sensruns/run2/"
DATADir3 = "/mnt/rasher/60sensruns/run3/"
DATADir4 = "/mnt/rasher/60sensruns/run4/"
DATADir5 = "/mnt/rasher/60sensruns/run5/"
DATADir6 = "/mnt/rasher/60sensruns/run6/"
DATADir7 = "/mnt/rasher/60sensruns/run7/"
DATADir8 = "/mnt/rasher/60sensruns/run8/"
DATADir9 = "/mnt/rasher/60sensruns/run9/"
DATADir10 = "/mnt/rasher/60sensruns/run10/"
DATADir11 = "/mnt/rasher/60sensruns/run11/"
DATADir12 = "/mnt/rasher/60sensruns/run12/"
DATADir13 = "/mnt/rasher/60sensruns/run13/"
DATADir14 = "/mnt/rasher/60sensruns/run14/"
DATADir15 = "/mnt/rasher/60sensruns/run15/"
DATADir16 = "/mnt/rasher/60sensruns/run16/"
DATADir17 = "/mnt/rasher/60sensruns/run17/"
DATADir18 = "/mnt/rasher/60sensruns/run18/"
DATADir19 = "/mnt/rasher/60sensruns/run19/"
DATADir20 = "/mnt/rasher/60sensruns/run20/"
DATADir21 = "/mnt/rasher/60sensruns/run21/"
DATADir22 = "/mnt/rasher/60sensruns/run22/"
DATADir23 = "/mnt/rasher/60sensruns/run23/"
DATADir24 = "/mnt/rasher/60sensruns/run24/"
DATADir25 = "/mnt/rasher/60sensruns/run25/"
DATADir26 = "/mnt/rasher/60sensruns/run26/"
DATADir27 = "/mnt/rasher/60sensruns/run27/"
DATADir28 = "/mnt/rasher/60sensruns/run28/"
DATADir29 = "/mnt/rasher/60sensruns/run29/"
DATADir30 = "/mnt/rasher/60sensruns/run30/"
DATADir31 = "/mnt/rasher/60sensruns/run31/"
DATADir32 = "/mnt/rasher/60sensruns/run32/"
DATADir33 = "/mnt/rasher/60sensruns/run33/"
DATADir34 = "/mnt/rasher/60sensruns/run34/"
DATADir35 = "/mnt/rasher/60sensruns/run35/"
DATADir36 = "/mnt/rasher/60sensruns/run36/"
DATADir37 = "/mnt/rasher/60sensruns/run37/"
DATADir38 = "/mnt/rasher/60sensruns/run38/"
DATADir39 = "/mnt/rasher/60sensruns/run39/"
DATADir40 = "/mnt/rasher/60sensruns/run40/"
DATADir41 = "/mnt/rasher/60sensruns/run41/"
DATADir42 = "/mnt/rasher/60sensruns/run42/"
DATADir43 = "/mnt/rasher/60sensruns/run43/"
DATADir44 = "/mnt/rasher/60sensruns/run44/"
DATADir45 = "/mnt/rasher/60sensruns/run45/"
DATADir46 = "/mnt/rasher/60sensruns/run46/"
DATADir47 = "/mnt/rasher/60sensruns/run47/"
DATADir48 = "/mnt/rasher/60sensruns/run48/"
DATADir49 = "/mnt/rasher/60sensruns/run49/"
DATADir50 = "/mnt/rasher/60sensruns/run50/"
DATADir51 = "/mnt/rasher/60sensruns/run51/"
DATADir52 = "/mnt/rasher/60sensruns/run52/"
DATADir53 = "/mnt/rasher/60sensruns/run53/"
DATADir54 = "/mnt/rasher/60sensruns/run54/"
DATADir55 = "/mnt/rasher/60sensruns/run55/"
DATADir56 = "/mnt/rasher/60sensruns/run56/"
DATADir57 = "/mnt/rasher/60sensruns/run57/"
DATADir58 = "/mnt/rasher/60sensruns/run58/"
DATADir59 = "/mnt/rasher/60sensruns/run59/"
DATADir60 = "/mnt/rasher/60sensruns/run60/"
For each directory I will further have to do this:
FILES1 = systemfunc (" ls -1 " + DATADir1 + "subset_d03_2014-* ")
a1 = addfiles(FILES1,"r")
pm25_sim_a1 = a1[:]->PM2_5_DRY(:,0,145,79)
(I will have to type this another 59 times for the other directories)
Is there a way to do this more efficiently perhaps in a loop?
Thanks
Tabish
Tabish U Ansari
PhD student, Lancaster Environment Center
Lancaster Univeristy
Bailrigg, Lancaster,
LA1 4YW, United Kingdom
_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk<http://mailman.ucar.edu/mailman/listinfo/ncl-talk>
--
Adam Phillips
Associate Scientist, Climate and Global Dynamics Laboratory, NCAR
www.cgd.ucar.edu/staff/asphilli/<http://www.cgd.ucar.edu/staff/asphilli/> 303-497-1726
_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk<http://mailman.ucar.edu/mailman/listinfo/ncl-talk>
--
Adam Phillips
Associate Scientist, Climate and Global Dynamics Laboratory, NCAR
www.cgd.ucar.edu/staff/asphilli/<http://www.cgd.ucar.edu/staff/asphilli/> 303-497-1726
_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk<http://mailman.ucar.edu/mailman/listinfo/ncl-talk>
University of Tasmania Electronic Communications Policy (December, 2014).
This email is confidential, and is for the intended recipient only. Access, disclosure, copying, distribution, or reliance on any of it by anyone outside the intended recipient organisation is prohibited and may be a criminal offence. Please delete if obtained in error and email confirmation to the sender. The views expressed in this email are not necessarily the views of the University of Tasmania, unless clearly intended otherwise.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190221/70add974/attachment.html>
More information about the ncl-talk
mailing list