[ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file

Dennis Shea shea at ucar.edu
Sun Jan 6 08:34:33 MST 2019


What to say?

After :

nl=13 Elapsed Time=73.0269
start at nl=14

keep going till u get all the file and then concatenate.



On Sun, Jan 6, 2019 at 12:32 AM Muhammad Omer Mughal <
m.mughal1 at graduate.curtin.edu.au> wrote:

> Dear Dennis
>
>
> Kindly see the output of two iterations attached.
>
>
> They seem to have the same CPU time consumed when the process gets killed.
>
>
> Regards
>
>
>
> Muhammad Omer Mughal
> MSc BSc Mechanical Engineering
> PhD  Research Scholar
> Remote Sensing and Satellite Research Group
> Department of Imaging and Applied Physics
> Curtin University
>
> Curtin University
> Tel | +61 8 9266 7962
> Fax | +61 8 9266 2377
> Mobile | 0470 237 525
>
> Email | m.mughal1 at postgrad.curtin.edu.au <m.lynch at curtin.edu.au>
> Web | http://curtin.edu.au
>
> Curtin University is a trademark of Curtin University of Technology.
> CRICOS Provider Code 00301J (WA), 02637B (NSW)
>
>
> ------------------------------
> *From:* Dennis Shea <shea at ucar.edu>
> *Sent:* Saturday, 5 January 2019 10:30:38 PM
> *To:* Muhammad Omer Mughal
> *Cc:* Rick Brownrigg; ncl-talk at ucar.edu
> *Subject:* Re: [ncl-talk] Extracting all values of rh2 from WRF file and
> writing it to csv file
>
> Depending on your system, the attached will run for 'a while'. It create
> one asciifile for each latitude. Then uses the unix 'cat' to combine them.
>
> The loop:
>    do nl=*0*,nlat-1
>
> If it fails [ 'killed' ] after (say) iteration 25, then rerun with
>
>    do nl=*26*,nlat-1
>
> etc
> ====
> Of course, you could create a netCDF file from NCL and then read that into
> Matlab. Then  use MATLAB to create the required text file.
>
> On Sat, Jan 5, 2019 at 4:40 AM Muhammad Omer Mughal <
> m.mughal1 at graduate.curtin.edu.au> wrote:
>
> Hi all
>
> If there is no solution to this problem could some one kindly confirm the
> method used to calculate rh2 in NCL so that I can prepare the required data
> in MATLAB.
>
> Regards
>
> Omer
>
>
>> Muhammad Omer Mughal
> MSc BSc Mechanical Engineering
> PhD  Research Scholar
> Remote Sensing and Satellite Research Group
> Department of Imaging and Applied Physics
> Curtin University
>
> Curtin University
> Tel | +61 8 9266 7962
> Fax | +61 8 9266 2377
> Mobile | 0470 237 525
>
> Email | m.mughal1 at postgrad.curtin.edu.au
> Web | http://curtin.edu.au
>
> ​ Curtin University is a trademark of Curtin University of Technology.
> CRICOS Provider Code 00301J (WA), 02637B (NSW)
>
>> ------------------------------
> *From:* Muhammad Omer Mughal
> *Sent:* Friday, 4 January 2019 2:51:16 PM
> *To:* Rick Brownrigg
> *Subject:* Re: [ncl-talk] Extracting all values of rh2 from WRF file and
> writing it to csv file
>
>
> Dear Rick and Dennis
>
>
> After doing various experiments, I found that increasing the value of ntim
> beyond 200 in this particular case causes the process to get killed.
>
>
> Regards
>
>
>
> Muhammad Omer Mughal
> MSc BSc Mechanical Engineering
> PhD  Research Scholar
> Remote Sensing and Satellite Research Group
> Department of Imaging and Applied Physics
> Curtin University
>
> Curtin University
> Tel | +61 8 9266 7962
> Fax | +61 8 9266 2377
> Mobile | 0470 237 525
>
> Email | m.mughal1 at postgrad.curtin.edu.au <m.lynch at curtin.edu.au>
> Web | http://curtin.edu.au
>
> Curtin University is a trademark of Curtin University of Technology.
> CRICOS Provider Code 00301J (WA), 02637B (NSW)
>
>
> ------------------------------
> *From:* Muhammad Omer Mughal
> *Sent:* Friday, 4 January 2019 11:42:19 AM
> *To:* Rick Brownrigg
> *Subject:* Re: [ncl-talk] Extracting all values of rh2 from WRF file and
> writing it to csv file
>
>
> Dear Rick
>
>
> Thank you for the reply again.
>
>
> It always happens at the same stage.
>
>
> However, as per Dennis suggestion if I assign a value of say 10 to ntim
> than the code runs fine and I obtain the attached csv file. I am now
> increasing the value step wise to see exactly at which value the code gets
> killed.
>
>
> Regards
>
>
>
>
> Muhammad Omer Mughal
> MSc BSc Mechanical Engineering
> PhD  Research Scholar
> Remote Sensing and Satellite Research Group
> Department of Imaging and Applied Physics
> Curtin University
>
> Curtin University
> Tel | +61 8 9266 7962
> Fax | +61 8 9266 2377
> Mobile | 0470 237 525
>
> Email | m.mughal1 at postgrad.curtin.edu.au <m.lynch at curtin.edu.au>
> Web | http://curtin.edu.au
>
> Curtin University is a trademark of Curtin University of Technology.
> CRICOS Provider Code 00301J (WA), 02637B (NSW)
>
>
> ------------------------------
> *From:* Rick Brownrigg <brownrig at ucar.edu>
> *Sent:* Friday, 4 January 2019 11:19:13 AM
> *To:* Muhammad Omer Mughal
> *Subject:* Re: [ncl-talk] Extracting all values of rh2 from WRF file and
> writing it to csv file
>
> Hi Muhammad,
> I've never seen a simple terminating message of "Killed" on a Linux
> system. But a quick google indicates that, yes, it may indeed be a
> possibility and that it may have to do with memory over-allocation on
> behalf of the Linux kernel. I don't fully understand, at this point, the
> circumstances that might lead up to it -- it will take me some research
> into the matter.
>
> In the mean time, given the print statements that Dennis had you insert in
> the loops , does it always happen at the same point, or does it vary?
>
>
>
> On Thu, Jan 3, 2019 at 7:57 PM Muhammad Omer Mughal <
> m.mughal1 at graduate.curtin.edu.au> wrote:
>
> Dear Rick
>
>
> The WRF model and NCL are both installed on the cluster,however,  I am not
> using parallel computing for running NCL. I am running it interactively.
> Please see below:
>
>
>  Copyright (C) 1995-2014 - All Rights Reserved
>  University Corporation for Atmospheric Research
>  NCAR Command Language Version 6.2.1
>  The use of this software is governed by a License Agreement.
>  See http://www.ncl.ucar.edu/ for more details.
> Killed
>
>
> Regards
>
> Omer
>
>
>
>
> Muhammad Omer Mughal
> MSc BSc Mechanical Engineering
> PhD  Research Scholar
> Remote Sensing and Satellite Research Group
> Department of Imaging and Applied Physics
> Curtin University
>
> Curtin University
> Tel | +61 8 9266 7962
> Fax | +61 8 9266 2377
> Mobile | 0470 237 525
>
> Email | m.mughal1 at postgrad.curtin.edu.au <m.lynch at curtin.edu.au>
> Web | http://curtin.edu.au
>
> Curtin University is a trademark of Curtin University of Technology.
> CRICOS Provider Code 00301J (WA), 02637B (NSW)
>
>
> ------------------------------
> *From:* Rick Brownrigg <brownrig at ucar.edu>
> *Sent:* Friday, 4 January 2019 10:24:04 AM
> *To:* Muhammad Omer Mughal
> *Cc:* Dennis Shea; ncl-talk at ucar.edu
> *Subject:* Re: [ncl-talk] Extracting all values of rh2 from WRF file and
> writing it to csv file
>
> Hi Muhammad,
>
> It would help to see the exact wording of the errors you are seeing. The
> phrase "killed" makes me think perhaps you are running on a shared
> computing resource, and the job is getting terminated because of over use
> of memory, processor, etc.?  Is this the case?
>
> On Thu, Jan 3, 2019 at 7:12 PM Muhammad Omer Mughal <
> m.mughal1 at graduate.curtin.edu.au> wrote:
>
> Dear Dennis
>
>
> Thank you again for the reply and suggestions.
>
>
>
> Using *print("nl="+nl+";   nt="+nt)* provides
>
>
> (0)    nl=0;   nt=0
> (0)    nl=0;   nt=1
> (0)    nl=0;   nt=2
> (0)    nl=0;   nt=3
> (0)    nl=0;   nt=4
> :
> :
> :
> (0)    nl=0;   nt=405
> (0)    nl=0;   nt=406
> (0)    nl=0;   nt=407
> (0)    nl=0;   nt=408
> (0)    nl=0;   nt=409
> (0)    nl=0;   nt=410 and so on
>
> *using **print(wrftxt(nl)) *yields
>
> 98.57    98.68    98.72    98.81    98.75    98.75    98.75    98.71
> 98.74    98.57    98.12    97.50    96.66    95.69    94.67    93.80
> 92.96    92.26    91.93    91.47    91.11    90.95    90.88    90.97
> 91.38    91.84    92.68    93.84    94.49    94.72    95.09    95.22
> 95.61    95.84    95.95    95.95    96.13    95.52    94.99    94.28
> 93.33    92.13    91.21    90.88    90.77    90.92    91.21    91.63
> 91.78    91.51    91.12    90.74    90.42    90.19    90.09    90.23
> 90.43    90.52    90.58    90.47    90.46    90.48    90.56    90.62
> 90.67    90.72    90.71    90.69    90.62    90.60    90.56    98.37
>
> Even after changing %9.2f" to "%6.2f" or "%5.1f" , I am still getting the
> "killed" message.
>
>
> Regards
>
>
> Muhammad Omer Mughal
> MSc BSc Mechanical Engineering
> PhD  Research Scholar
> Remote Sensing and Satellite Research Group
> Department of Imaging and Applied Physics
> Curtin University
>
> Curtin University
> Tel | +61 8 9266 7962
> Fax | +61 8 9266 2377
> Mobile | 0470 237 525
>
> Email | m.mughal1 at postgrad.curtin.edu.au <m.lynch at curtin.edu.au>
> Web | http://curtin.edu.au
>
> Curtin University is a trademark of Curtin University of Technology.
> CRICOS Provider Code 00301J (WA), 02637B (NSW)
>
>
> ------------------------------
> *From:* Dennis Shea <shea at ucar.edu>
> *Sent:* Friday, 4 January 2019 3:10:59 AM
> *To:* Muhammad Omer Mughal
> *Cc:* Rick Brownrigg; ncl-talk at ucar.edu
> *Subject:* Re: [ncl-talk] Extracting all values of rh2 from WRF file and
> writing it to csv file
>
> [1] I doubt it is the size of the ascii file.
>
> [2] I tried the script on an available wrf file that had only one time
> dimension.
>
> Variable: rh2
> Dimensions and sizes:    [Time | 1] x [south_north | 108] x [west_east |
> 114]
>
> and the script worked fine. It put out 108 latiitudes and 1*114
> Time/longitudes
> ---
> You can add a print to *provide more information*. It will tell you when
> it failed.
> Did it fail on the 1st iteration or later? Really, just saying it
> terminated provides little information.
>
>   wrftxt = new(nlat, "string")
>   wrftxt = ""
>
>   do nl=0,nlat-1
>
>     do nt=0,ntim-1
>          *print("nl="+nl+";   nt="+nt)*
>       do ml=0,mlon-1
>          wrftxt(nl) = wrftxt(nl) +sprintf("%9.2f", rh2(nt,nl,ml))  ;
> append to previous value
>       end do
>     end do
>
>  ----
> *If *it failed on the 1st iteration, I speculate that the underlying
> C-code has some string length limit.
> [Time | 673] x [south_north | 71] x [west_east | 82]
>
> Each single string has 673*82*(nchar)  where nchar is the length of each
> string.
> So: "%9.2f" yields string lengths of  673*82*9= 496674 characters. That is
> a LONG string length.
>
> Change "%9.2f" to "%6.2f" or "%5.1f" ... see if it works. Still, they are
> all LONG.
>
>   ---
> Play with these
>
>    NTIM = 50   ; whatever
>    MLON = ???
>    do nt=0,NTIM
>          *print("nl="+nl+";   nt="+nt)*
>       do ml=0,MLON
>
> DO various combinations. That will provides some insight into the issue.
>
>
> On Thu, Jan 3, 2019 at 4:53 AM Muhammad Omer Mughal <
> m.mughal1 at graduate.curtin.edu.au> wrote:
>
> Dear Dennis and all
>
> Thank you for reply and sending the code.
>
> However, the script gets killed by itself without giving an error at the
> end of do loop. This I suspect might be due the fact that string is
> getting  overflow.
>
> Is there a way to fix this ?
>
>
> Muhammad Omer Mughal
> MSc BSc Mechanical Engineering
> PhD  Research Scholar
> Remote Sensing and Satellite Research Group
> Department of Imaging and Applied Physics
> Curtin University
>
> Curtin University
> Tel | +61 8 9266 7962
> Fax | +61 8 9266 2377
> Mobile | 0470 237 525
>
> Email | m.mughal1 at postgrad.curtin.edu.au
> Web | http://curtin.edu.au
>
> Curtin University is a trademark of Curtin University of Technology.
> CRICOS Provider Code 00301J (WA), 02637B (NSW)
>
>
> ________________________________
> From: Dennis Shea <shea at ucar.edu>
> Sent: Thursday, 3 January 2019 12:17:16 PM
> To: Muhammad Omer Mughal
> Cc: Rick Brownrigg; ncl-talk at ucar.edu
> Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and
> writing it to csv file
>
> This series of questions is a bit frustrating.
>
> You have used NCL quite a bit and you have sent numerous questions.
> As you know, ncl-talk always recommends  that users use 'printVarSummary'
> to look at variables. You should do that.... ALWAYS.   I don't have your
> file but
>
> rh2 = wrf_user_getvar(a,"rh2",-1)       ; [Time | 673] x [south_north |
> 71] x [west_east | 82]
> lat=wrf_user_getvar(a,"XLAT",-1)       ;                                 "
> lon=wrf_user_getvar(a,"XLONG",-1)  ;                                 "
>
> Your initial question:
>
> lat and lon are variables with a 'Time' dimension.
> You can not use multidimensional variables. The following is not correct
> syntax.
>
> allRH = rh2(:, lon, lat)   ; ??????????????????
>
> Please read the User Manual on subscripting and 'do' loops
>   http://www.ncl.ucar.edu/Document/Manuals/
>
> also
>
>   http://www.ncl.ucar.edu/Applications/write_ascii.shtml<
> http://www.ncl.ucar.edu/Applications/write_ascii.shtml>
>
> ========
>   rh2 = wrf_user_getvar(a,"rh2",-1)
>   printVarSummary(rh2)
>
>   lat = wrf_user_getvar(a,"XLAT",-1)
>   lon = wrf_user_getvar(a,"XLONG",-1)
>   printVarSummary(lat)
>
>
>   dimrh2 = dimsizes(rh2)
>   ntim   = dimrh2(0)
>   nlat   = dimrh2(1)
>   mlon   = dimrh2(2)
>
>   wrftxt = new(nlat, "string")
>   wrftxt = ""
>
>   do nl=0,nlat-1
>
>     do nt=0,ntim-1
>       do ml=0,mlon-1
>          wrftxt(nl) = wrftxt(nl) +sprintf("%9.2f", rh2(nt,nl,ml))
>       end do
>     end do
>
>   end do
>
>
>   filo = "rh2.csv"
>   system("/bin/rm -f "+filo)
>   asciiwrite(filo,wrftxt)
>
>
> On Wed, Jan 2, 2019 at 8:28 PM Muhammad Omer Mughal <
> m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>>
> wrote:
> Hi Rick
>
> Thank you for the reply.
>
> Yes , I want to write rh2 to the ascii format. The size of the rh2 in this
> particular case is [Time|673]*[south_north|81]*[west_east|72]. I want to
> save the file in the format so that it would have 72 rows and 54513 columns
> i.e. (673*81) for a particular code to read it.
>
> Regards
>
> Omer
>
>
> Muhammad Omer Mughal
> MSc BSc Mechanical Engineering
> PhD  Research Scholar
> Remote Sensing and Satellite Research Group
> Department of Imaging and Applied Physics
> Curtin University
>
> Curtin University
> Tel | +61 8 9266 7962
> Fax | +61 8 9266 2377
> Mobile | 0470 237 525
>
> Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.lynch at curtin.edu.au>
> Web | http://curtin.edu.au<http://curtin.edu.au/>
>
> Curtin University is a trademark of Curtin University of Technology.
> CRICOS Provider Code 00301J (WA), 02637B (NSW)
>
>
> ________________________________
> From: Rick Brownrigg <brownrig at ucar.edu<mailto:brownrig at ucar.edu>>
> Sent: Thursday, 3 January 2019 10:59:30 AM
> To: Muhammad Omer Mughal
> Cc: ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
> Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and
> writing it to csv file
>
> HI,
>
> That error message makes sense, since in a WRF file, the variables XLAT
> and XLONG are 2D (curvilinear).
>
> I guess I'm confused as to what you are trying to do.  If you just simply
> want the variable "rh2" written to an ascii file, instead of this:
>
>     allRH = rh2(:, lon, lat)
>     asciiwrite("rh2.csv",allRH)
>
> what about just:
>
>    asciiwrite("rh2.csv",rh2)
>
> A simple experiment revealed that this will write the 3D array, 1 value
> per line, in "row major" format. This means to read such a file later, you
> must have some way of knowing the shape and dimensionality.
>
> Hope that helps....
> Rick
>
>
> On Wed, Jan 2, 2019 at 7:42 PM Muhammad Omer Mughal <
> m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>>
> wrote:
> Dear Rick
> Thanks for the reply. I changed the script to remove the SEGV error but
> now I am getting the following error messages
>
>
> Please see the script below
>
> a=addfile("wrfout_d03_2017-02-01_00:00:00.nc<http://00.nc>","r")
>
> rh2 = wrf_user_getvar(a,"rh2",-1)
>
> lat=wrf_user_getvar(a,"XLAT",-1)
> lon=wrf_user_getvar(a,"XLONG",-1)
>
>
>   allRH = rh2(:, lon, lat)
>
>   asciiwrite("rh2.csv",allRH)
>
> fatal:Illegal subscript. Subscripts must be scalar or one dimensional
> vectors
>
>
>
>
>
> Muhammad Omer Mughal
> MSc BSc Mechanical Engineering
> PhD  Research Scholar
> Remote Sensing and Satellite Research Group
> Department of Imaging and Applied Physics
> Curtin University
>
> Curtin University
> Tel | +61 8 9266 7962
> Fax | +61 8 9266 2377
> Mobile | 0470 237 525
>
> Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.lynch at curtin.edu.au>
> Web | http://curtin.edu.au<http://curtin.edu.au/>
>
> Curtin University is a trademark of Curtin University of Technology.
> CRICOS Provider Code 00301J (WA), 02637B (NSW)
>
>
> ________________________________
> From: Rick Brownrigg <brownrig at ucar.edu<mailto:brownrig at ucar.edu>>
> Sent: Wednesday, 2 January 2019 10:18:33 PM
> To: Muhammad Omer Mughal
> Cc: ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
> Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and
> writing it to csv file
>
> Hi,
>
> Can you share your script and dataset with me?  I would like to
> investigate why there's a SEGV.
>
> Rick
>
>
> On Wed, Jan 2, 2019 at 4:49 AM Muhammad Omer Mughal <
> m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>>
> wrote:
> Dear NCL team
>
> Is there a way to extract all values from WRF output files from all grind
> points and write to an ascii file.
> In my case grid size is 71x82 and the number of times is 673. I tried to
> write using a table format but I am getting a segmentation fault.
>
> I will appreciate help. Kindly note that I have used Lambert conformal
> projection method.
>
> Regards
>
>
>
> Muhammad Omer Mughal
> MSc BSc Mechanical Engineering
> PhD  Research Scholar
> Remote Sensing and Satellite Research Group
> Department of Imaging and Applied Physics
> Curtin University
>
> Curtin University
> Tel | +61 8 9266 7962
> Fax | +61 8 9266 2377
> Mobile | 0470 237 525
>
> Email | m.mughal1 at postgrad.curtin.edu.au<mailto:
> m.mughal1 at postgrad.curtin.edu.au>
> Web | http://curtin.edu.au
>
> Curtin University is a trademark of Curtin University of Technology.
> CRICOS Provider Code 00301J (WA), 02637B (NSW)
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190106/e62ecc8e/attachment.html>


More information about the ncl-talk mailing list