[ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file
Dennis Shea
shea at ucar.edu
Sun Jan 6 08:34:33 MST 2019
What to say?
After :
nl=13 Elapsed Time=73.0269
start at nl=14
keep going till u get all the file and then concatenate.
On Sun, Jan 6, 2019 at 12:32 AM Muhammad Omer Mughal <
m.mughal1 at graduate.curtin.edu.au> wrote:
> Dear Dennis
>
>
> Kindly see the output of two iterations attached.
>
>
> They seem to have the same CPU time consumed when the process gets killed.
>
>
> Regards
>
>
>
> Muhammad Omer Mughal
> MSc BSc Mechanical Engineering
> PhD Research Scholar
> Remote Sensing and Satellite Research Group
> Department of Imaging and Applied Physics
> Curtin University
>
> Curtin University
> Tel | +61 8 9266 7962
> Fax | +61 8 9266 2377
> Mobile | 0470 237 525
>
> Email | m.mughal1 at postgrad.curtin.edu.au <m.lynch at curtin.edu.au>
> Web | http://curtin.edu.au
>
> Curtin University is a trademark of Curtin University of Technology.
> CRICOS Provider Code 00301J (WA), 02637B (NSW)
>
>
> ------------------------------
> *From:* Dennis Shea <shea at ucar.edu>
> *Sent:* Saturday, 5 January 2019 10:30:38 PM
> *To:* Muhammad Omer Mughal
> *Cc:* Rick Brownrigg; ncl-talk at ucar.edu
> *Subject:* Re: [ncl-talk] Extracting all values of rh2 from WRF file and
> writing it to csv file
>
> Depending on your system, the attached will run for 'a while'. It create
> one asciifile for each latitude. Then uses the unix 'cat' to combine them.
>
> The loop:
> do nl=*0*,nlat-1
>
> If it fails [ 'killed' ] after (say) iteration 25, then rerun with
>
> do nl=*26*,nlat-1
>
> etc
> ====
> Of course, you could create a netCDF file from NCL and then read that into
> Matlab. Then use MATLAB to create the required text file.
>
> On Sat, Jan 5, 2019 at 4:40 AM Muhammad Omer Mughal <
> m.mughal1 at graduate.curtin.edu.au> wrote:
>
> Hi all
>
> If there is no solution to this problem could some one kindly confirm the
> method used to calculate rh2 in NCL so that I can prepare the required data
> in MATLAB.
>
> Regards
>
> Omer
>
>
>
> Muhammad Omer Mughal
> MSc BSc Mechanical Engineering
> PhD Research Scholar
> Remote Sensing and Satellite Research Group
> Department of Imaging and Applied Physics
> Curtin University
>
> Curtin University
> Tel | +61 8 9266 7962
> Fax | +61 8 9266 2377
> Mobile | 0470 237 525
>
> Email | m.mughal1 at postgrad.curtin.edu.au
> Web | http://curtin.edu.au
>
> Curtin University is a trademark of Curtin University of Technology.
> CRICOS Provider Code 00301J (WA), 02637B (NSW)
>
>
> ------------------------------
> *From:* Muhammad Omer Mughal
> *Sent:* Friday, 4 January 2019 2:51:16 PM
> *To:* Rick Brownrigg
> *Subject:* Re: [ncl-talk] Extracting all values of rh2 from WRF file and
> writing it to csv file
>
>
> Dear Rick and Dennis
>
>
> After doing various experiments, I found that increasing the value of ntim
> beyond 200 in this particular case causes the process to get killed.
>
>
> Regards
>
>
>
> Muhammad Omer Mughal
> MSc BSc Mechanical Engineering
> PhD Research Scholar
> Remote Sensing and Satellite Research Group
> Department of Imaging and Applied Physics
> Curtin University
>
> Curtin University
> Tel | +61 8 9266 7962
> Fax | +61 8 9266 2377
> Mobile | 0470 237 525
>
> Email | m.mughal1 at postgrad.curtin.edu.au <m.lynch at curtin.edu.au>
> Web | http://curtin.edu.au
>
> Curtin University is a trademark of Curtin University of Technology.
> CRICOS Provider Code 00301J (WA), 02637B (NSW)
>
>
> ------------------------------
> *From:* Muhammad Omer Mughal
> *Sent:* Friday, 4 January 2019 11:42:19 AM
> *To:* Rick Brownrigg
> *Subject:* Re: [ncl-talk] Extracting all values of rh2 from WRF file and
> writing it to csv file
>
>
> Dear Rick
>
>
> Thank you for the reply again.
>
>
> It always happens at the same stage.
>
>
> However, as per Dennis suggestion if I assign a value of say 10 to ntim
> than the code runs fine and I obtain the attached csv file. I am now
> increasing the value step wise to see exactly at which value the code gets
> killed.
>
>
> Regards
>
>
>
>
> Muhammad Omer Mughal
> MSc BSc Mechanical Engineering
> PhD Research Scholar
> Remote Sensing and Satellite Research Group
> Department of Imaging and Applied Physics
> Curtin University
>
> Curtin University
> Tel | +61 8 9266 7962
> Fax | +61 8 9266 2377
> Mobile | 0470 237 525
>
> Email | m.mughal1 at postgrad.curtin.edu.au <m.lynch at curtin.edu.au>
> Web | http://curtin.edu.au
>
> Curtin University is a trademark of Curtin University of Technology.
> CRICOS Provider Code 00301J (WA), 02637B (NSW)
>
>
> ------------------------------
> *From:* Rick Brownrigg <brownrig at ucar.edu>
> *Sent:* Friday, 4 January 2019 11:19:13 AM
> *To:* Muhammad Omer Mughal
> *Subject:* Re: [ncl-talk] Extracting all values of rh2 from WRF file and
> writing it to csv file
>
> Hi Muhammad,
> I've never seen a simple terminating message of "Killed" on a Linux
> system. But a quick google indicates that, yes, it may indeed be a
> possibility and that it may have to do with memory over-allocation on
> behalf of the Linux kernel. I don't fully understand, at this point, the
> circumstances that might lead up to it -- it will take me some research
> into the matter.
>
> In the mean time, given the print statements that Dennis had you insert in
> the loops , does it always happen at the same point, or does it vary?
>
>
>
> On Thu, Jan 3, 2019 at 7:57 PM Muhammad Omer Mughal <
> m.mughal1 at graduate.curtin.edu.au> wrote:
>
> Dear Rick
>
>
> The WRF model and NCL are both installed on the cluster,however, I am not
> using parallel computing for running NCL. I am running it interactively.
> Please see below:
>
>
> Copyright (C) 1995-2014 - All Rights Reserved
> University Corporation for Atmospheric Research
> NCAR Command Language Version 6.2.1
> The use of this software is governed by a License Agreement.
> See http://www.ncl.ucar.edu/ for more details.
> Killed
>
>
> Regards
>
> Omer
>
>
>
>
> Muhammad Omer Mughal
> MSc BSc Mechanical Engineering
> PhD Research Scholar
> Remote Sensing and Satellite Research Group
> Department of Imaging and Applied Physics
> Curtin University
>
> Curtin University
> Tel | +61 8 9266 7962
> Fax | +61 8 9266 2377
> Mobile | 0470 237 525
>
> Email | m.mughal1 at postgrad.curtin.edu.au <m.lynch at curtin.edu.au>
> Web | http://curtin.edu.au
>
> Curtin University is a trademark of Curtin University of Technology.
> CRICOS Provider Code 00301J (WA), 02637B (NSW)
>
>
> ------------------------------
> *From:* Rick Brownrigg <brownrig at ucar.edu>
> *Sent:* Friday, 4 January 2019 10:24:04 AM
> *To:* Muhammad Omer Mughal
> *Cc:* Dennis Shea; ncl-talk at ucar.edu
> *Subject:* Re: [ncl-talk] Extracting all values of rh2 from WRF file and
> writing it to csv file
>
> Hi Muhammad,
>
> It would help to see the exact wording of the errors you are seeing. The
> phrase "killed" makes me think perhaps you are running on a shared
> computing resource, and the job is getting terminated because of over use
> of memory, processor, etc.? Is this the case?
>
> On Thu, Jan 3, 2019 at 7:12 PM Muhammad Omer Mughal <
> m.mughal1 at graduate.curtin.edu.au> wrote:
>
> Dear Dennis
>
>
> Thank you again for the reply and suggestions.
>
>
>
> Using *print("nl="+nl+"; nt="+nt)* provides
>
>
> (0) nl=0; nt=0
> (0) nl=0; nt=1
> (0) nl=0; nt=2
> (0) nl=0; nt=3
> (0) nl=0; nt=4
> :
> :
> :
> (0) nl=0; nt=405
> (0) nl=0; nt=406
> (0) nl=0; nt=407
> (0) nl=0; nt=408
> (0) nl=0; nt=409
> (0) nl=0; nt=410 and so on
>
> *using **print(wrftxt(nl)) *yields
>
> 98.57 98.68 98.72 98.81 98.75 98.75 98.75 98.71
> 98.74 98.57 98.12 97.50 96.66 95.69 94.67 93.80
> 92.96 92.26 91.93 91.47 91.11 90.95 90.88 90.97
> 91.38 91.84 92.68 93.84 94.49 94.72 95.09 95.22
> 95.61 95.84 95.95 95.95 96.13 95.52 94.99 94.28
> 93.33 92.13 91.21 90.88 90.77 90.92 91.21 91.63
> 91.78 91.51 91.12 90.74 90.42 90.19 90.09 90.23
> 90.43 90.52 90.58 90.47 90.46 90.48 90.56 90.62
> 90.67 90.72 90.71 90.69 90.62 90.60 90.56 98.37
>
> Even after changing %9.2f" to "%6.2f" or "%5.1f" , I am still getting the
> "killed" message.
>
>
> Regards
>
>
> Muhammad Omer Mughal
> MSc BSc Mechanical Engineering
> PhD Research Scholar
> Remote Sensing and Satellite Research Group
> Department of Imaging and Applied Physics
> Curtin University
>
> Curtin University
> Tel | +61 8 9266 7962
> Fax | +61 8 9266 2377
> Mobile | 0470 237 525
>
> Email | m.mughal1 at postgrad.curtin.edu.au <m.lynch at curtin.edu.au>
> Web | http://curtin.edu.au
>
> Curtin University is a trademark of Curtin University of Technology.
> CRICOS Provider Code 00301J (WA), 02637B (NSW)
>
>
> ------------------------------
> *From:* Dennis Shea <shea at ucar.edu>
> *Sent:* Friday, 4 January 2019 3:10:59 AM
> *To:* Muhammad Omer Mughal
> *Cc:* Rick Brownrigg; ncl-talk at ucar.edu
> *Subject:* Re: [ncl-talk] Extracting all values of rh2 from WRF file and
> writing it to csv file
>
> [1] I doubt it is the size of the ascii file.
>
> [2] I tried the script on an available wrf file that had only one time
> dimension.
>
> Variable: rh2
> Dimensions and sizes: [Time | 1] x [south_north | 108] x [west_east |
> 114]
>
> and the script worked fine. It put out 108 latiitudes and 1*114
> Time/longitudes
> ---
> You can add a print to *provide more information*. It will tell you when
> it failed.
> Did it fail on the 1st iteration or later? Really, just saying it
> terminated provides little information.
>
> wrftxt = new(nlat, "string")
> wrftxt = ""
>
> do nl=0,nlat-1
>
> do nt=0,ntim-1
> *print("nl="+nl+"; nt="+nt)*
> do ml=0,mlon-1
> wrftxt(nl) = wrftxt(nl) +sprintf("%9.2f", rh2(nt,nl,ml)) ;
> append to previous value
> end do
> end do
>
> ----
> *If *it failed on the 1st iteration, I speculate that the underlying
> C-code has some string length limit.
> [Time | 673] x [south_north | 71] x [west_east | 82]
>
> Each single string has 673*82*(nchar) where nchar is the length of each
> string.
> So: "%9.2f" yields string lengths of 673*82*9= 496674 characters. That is
> a LONG string length.
>
> Change "%9.2f" to "%6.2f" or "%5.1f" ... see if it works. Still, they are
> all LONG.
>
> ---
> Play with these
>
> NTIM = 50 ; whatever
> MLON = ???
> do nt=0,NTIM
> *print("nl="+nl+"; nt="+nt)*
> do ml=0,MLON
>
> DO various combinations. That will provides some insight into the issue.
>
>
> On Thu, Jan 3, 2019 at 4:53 AM Muhammad Omer Mughal <
> m.mughal1 at graduate.curtin.edu.au> wrote:
>
> Dear Dennis and all
>
> Thank you for reply and sending the code.
>
> However, the script gets killed by itself without giving an error at the
> end of do loop. This I suspect might be due the fact that string is
> getting overflow.
>
> Is there a way to fix this ?
>
>
> Muhammad Omer Mughal
> MSc BSc Mechanical Engineering
> PhD Research Scholar
> Remote Sensing and Satellite Research Group
> Department of Imaging and Applied Physics
> Curtin University
>
> Curtin University
> Tel | +61 8 9266 7962
> Fax | +61 8 9266 2377
> Mobile | 0470 237 525
>
> Email | m.mughal1 at postgrad.curtin.edu.au
> Web | http://curtin.edu.au
>
> Curtin University is a trademark of Curtin University of Technology.
> CRICOS Provider Code 00301J (WA), 02637B (NSW)
>
>
> ________________________________
> From: Dennis Shea <shea at ucar.edu>
> Sent: Thursday, 3 January 2019 12:17:16 PM
> To: Muhammad Omer Mughal
> Cc: Rick Brownrigg; ncl-talk at ucar.edu
> Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and
> writing it to csv file
>
> This series of questions is a bit frustrating.
>
> You have used NCL quite a bit and you have sent numerous questions.
> As you know, ncl-talk always recommends that users use 'printVarSummary'
> to look at variables. You should do that.... ALWAYS. I don't have your
> file but
>
> rh2 = wrf_user_getvar(a,"rh2",-1) ; [Time | 673] x [south_north |
> 71] x [west_east | 82]
> lat=wrf_user_getvar(a,"XLAT",-1) ; "
> lon=wrf_user_getvar(a,"XLONG",-1) ; "
>
> Your initial question:
>
> lat and lon are variables with a 'Time' dimension.
> You can not use multidimensional variables. The following is not correct
> syntax.
>
> allRH = rh2(:, lon, lat) ; ??????????????????
>
> Please read the User Manual on subscripting and 'do' loops
> http://www.ncl.ucar.edu/Document/Manuals/
>
> also
>
> http://www.ncl.ucar.edu/Applications/write_ascii.shtml<
> http://www.ncl.ucar.edu/Applications/write_ascii.shtml>
>
> ========
> rh2 = wrf_user_getvar(a,"rh2",-1)
> printVarSummary(rh2)
>
> lat = wrf_user_getvar(a,"XLAT",-1)
> lon = wrf_user_getvar(a,"XLONG",-1)
> printVarSummary(lat)
>
>
> dimrh2 = dimsizes(rh2)
> ntim = dimrh2(0)
> nlat = dimrh2(1)
> mlon = dimrh2(2)
>
> wrftxt = new(nlat, "string")
> wrftxt = ""
>
> do nl=0,nlat-1
>
> do nt=0,ntim-1
> do ml=0,mlon-1
> wrftxt(nl) = wrftxt(nl) +sprintf("%9.2f", rh2(nt,nl,ml))
> end do
> end do
>
> end do
>
>
> filo = "rh2.csv"
> system("/bin/rm -f "+filo)
> asciiwrite(filo,wrftxt)
>
>
> On Wed, Jan 2, 2019 at 8:28 PM Muhammad Omer Mughal <
> m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>>
> wrote:
> Hi Rick
>
> Thank you for the reply.
>
> Yes , I want to write rh2 to the ascii format. The size of the rh2 in this
> particular case is [Time|673]*[south_north|81]*[west_east|72]. I want to
> save the file in the format so that it would have 72 rows and 54513 columns
> i.e. (673*81) for a particular code to read it.
>
> Regards
>
> Omer
>
>
> Muhammad Omer Mughal
> MSc BSc Mechanical Engineering
> PhD Research Scholar
> Remote Sensing and Satellite Research Group
> Department of Imaging and Applied Physics
> Curtin University
>
> Curtin University
> Tel | +61 8 9266 7962
> Fax | +61 8 9266 2377
> Mobile | 0470 237 525
>
> Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.lynch at curtin.edu.au>
> Web | http://curtin.edu.au<http://curtin.edu.au/>
>
> Curtin University is a trademark of Curtin University of Technology.
> CRICOS Provider Code 00301J (WA), 02637B (NSW)
>
>
> ________________________________
> From: Rick Brownrigg <brownrig at ucar.edu<mailto:brownrig at ucar.edu>>
> Sent: Thursday, 3 January 2019 10:59:30 AM
> To: Muhammad Omer Mughal
> Cc: ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
> Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and
> writing it to csv file
>
> HI,
>
> That error message makes sense, since in a WRF file, the variables XLAT
> and XLONG are 2D (curvilinear).
>
> I guess I'm confused as to what you are trying to do. If you just simply
> want the variable "rh2" written to an ascii file, instead of this:
>
> allRH = rh2(:, lon, lat)
> asciiwrite("rh2.csv",allRH)
>
> what about just:
>
> asciiwrite("rh2.csv",rh2)
>
> A simple experiment revealed that this will write the 3D array, 1 value
> per line, in "row major" format. This means to read such a file later, you
> must have some way of knowing the shape and dimensionality.
>
> Hope that helps....
> Rick
>
>
> On Wed, Jan 2, 2019 at 7:42 PM Muhammad Omer Mughal <
> m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>>
> wrote:
> Dear Rick
> Thanks for the reply. I changed the script to remove the SEGV error but
> now I am getting the following error messages
>
>
> Please see the script below
>
> a=addfile("wrfout_d03_2017-02-01_00:00:00.nc<http://00.nc>","r")
>
> rh2 = wrf_user_getvar(a,"rh2",-1)
>
> lat=wrf_user_getvar(a,"XLAT",-1)
> lon=wrf_user_getvar(a,"XLONG",-1)
>
>
> allRH = rh2(:, lon, lat)
>
> asciiwrite("rh2.csv",allRH)
>
> fatal:Illegal subscript. Subscripts must be scalar or one dimensional
> vectors
>
>
>
>
>
> Muhammad Omer Mughal
> MSc BSc Mechanical Engineering
> PhD Research Scholar
> Remote Sensing and Satellite Research Group
> Department of Imaging and Applied Physics
> Curtin University
>
> Curtin University
> Tel | +61 8 9266 7962
> Fax | +61 8 9266 2377
> Mobile | 0470 237 525
>
> Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.lynch at curtin.edu.au>
> Web | http://curtin.edu.au<http://curtin.edu.au/>
>
> Curtin University is a trademark of Curtin University of Technology.
> CRICOS Provider Code 00301J (WA), 02637B (NSW)
>
>
> ________________________________
> From: Rick Brownrigg <brownrig at ucar.edu<mailto:brownrig at ucar.edu>>
> Sent: Wednesday, 2 January 2019 10:18:33 PM
> To: Muhammad Omer Mughal
> Cc: ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
> Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and
> writing it to csv file
>
> Hi,
>
> Can you share your script and dataset with me? I would like to
> investigate why there's a SEGV.
>
> Rick
>
>
> On Wed, Jan 2, 2019 at 4:49 AM Muhammad Omer Mughal <
> m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>>
> wrote:
> Dear NCL team
>
> Is there a way to extract all values from WRF output files from all grind
> points and write to an ascii file.
> In my case grid size is 71x82 and the number of times is 673. I tried to
> write using a table format but I am getting a segmentation fault.
>
> I will appreciate help. Kindly note that I have used Lambert conformal
> projection method.
>
> Regards
>
>
>
> Muhammad Omer Mughal
> MSc BSc Mechanical Engineering
> PhD Research Scholar
> Remote Sensing and Satellite Research Group
> Department of Imaging and Applied Physics
> Curtin University
>
> Curtin University
> Tel | +61 8 9266 7962
> Fax | +61 8 9266 2377
> Mobile | 0470 237 525
>
> Email | m.mughal1 at postgrad.curtin.edu.au<mailto:
> m.mughal1 at postgrad.curtin.edu.au>
> Web | http://curtin.edu.au
>
> Curtin University is a trademark of Curtin University of Technology.
> CRICOS Provider Code 00301J (WA), 02637B (NSW)
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190106/e62ecc8e/attachment.html>
More information about the ncl-talk
mailing list