[ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file
Muhammad Omer Mughal
m.mughal1 at graduate.curtin.edu.au
Sun Jan 6 00:32:26 MST 2019
Dear Dennis
Kindly see the output of two iterations attached.
They seem to have the same CPU time consumed when the process gets killed.
Regards
Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University
Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525
Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.lynch at curtin.edu.au>
Web | http://curtin.edu.au<http://curtin.edu.au/>
Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)
________________________________
From: Dennis Shea <shea at ucar.edu>
Sent: Saturday, 5 January 2019 10:30:38 PM
To: Muhammad Omer Mughal
Cc: Rick Brownrigg; ncl-talk at ucar.edu
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file
Depending on your system, the attached will run for 'a while'. It create one asciifile for each latitude. Then uses the unix 'cat' to combine them.
The loop:
do nl=0,nlat-1
If it fails [ 'killed' ] after (say) iteration 25, then rerun with
do nl=26,nlat-1
etc
====
Of course, you could create a netCDF file from NCL and then read that into Matlab. Then use MATLAB to create the required text file.
On Sat, Jan 5, 2019 at 4:40 AM Muhammad Omer Mughal <m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>> wrote:
Hi all
If there is no solution to this problem could some one kindly confirm the method used to calculate rh2 in NCL so that I can prepare the required data in MATLAB.
Regards
Omer
Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University
Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525
Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.mughal1 at postgrad.curtin.edu.au>
Web | http://curtin.edu.au
Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)
________________________________
From: Muhammad Omer Mughal
Sent: Friday, 4 January 2019 2:51:16 PM
To: Rick Brownrigg
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file
Dear Rick and Dennis
After doing various experiments, I found that increasing the value of ntim beyond 200 in this particular case causes the process to get killed.
Regards
Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University
Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525
Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.lynch at curtin.edu.au>
Web | http://curtin.edu.au<http://curtin.edu.au/>
Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)
________________________________
From: Muhammad Omer Mughal
Sent: Friday, 4 January 2019 11:42:19 AM
To: Rick Brownrigg
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file
Dear Rick
Thank you for the reply again.
It always happens at the same stage.
However, as per Dennis suggestion if I assign a value of say 10 to ntim than the code runs fine and I obtain the attached csv file. I am now increasing the value step wise to see exactly at which value the code gets killed.
Regards
Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University
Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525
Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.lynch at curtin.edu.au>
Web | http://curtin.edu.au<http://curtin.edu.au/>
Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)
________________________________
From: Rick Brownrigg <brownrig at ucar.edu<mailto:brownrig at ucar.edu>>
Sent: Friday, 4 January 2019 11:19:13 AM
To: Muhammad Omer Mughal
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file
Hi Muhammad,
I've never seen a simple terminating message of "Killed" on a Linux system. But a quick google indicates that, yes, it may indeed be a possibility and that it may have to do with memory over-allocation on behalf of the Linux kernel. I don't fully understand, at this point, the circumstances that might lead up to it -- it will take me some research into the matter.
In the mean time, given the print statements that Dennis had you insert in the loops , does it always happen at the same point, or does it vary?
On Thu, Jan 3, 2019 at 7:57 PM Muhammad Omer Mughal <m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>> wrote:
Dear Rick
The WRF model and NCL are both installed on the cluster,however, I am not using parallel computing for running NCL. I am running it interactively. Please see below:
Copyright (C) 1995-2014 - All Rights Reserved
University Corporation for Atmospheric Research
NCAR Command Language Version 6.2.1
The use of this software is governed by a License Agreement.
See http://www.ncl.ucar.edu/ for more details.
Killed
Regards
Omer
Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University
Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525
Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.lynch at curtin.edu.au>
Web | http://curtin.edu.au<http://curtin.edu.au/>
Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)
________________________________
From: Rick Brownrigg <brownrig at ucar.edu<mailto:brownrig at ucar.edu>>
Sent: Friday, 4 January 2019 10:24:04 AM
To: Muhammad Omer Mughal
Cc: Dennis Shea; ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file
Hi Muhammad,
It would help to see the exact wording of the errors you are seeing. The phrase "killed" makes me think perhaps you are running on a shared computing resource, and the job is getting terminated because of over use of memory, processor, etc.? Is this the case?
On Thu, Jan 3, 2019 at 7:12 PM Muhammad Omer Mughal <m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>> wrote:
Dear Dennis
Thank you again for the reply and suggestions.
Using print("nl="+nl+"; nt="+nt) provides
(0) nl=0; nt=0
(0) nl=0; nt=1
(0) nl=0; nt=2
(0) nl=0; nt=3
(0) nl=0; nt=4
:
:
:
(0) nl=0; nt=405
(0) nl=0; nt=406
(0) nl=0; nt=407
(0) nl=0; nt=408
(0) nl=0; nt=409
(0) nl=0; nt=410 and so on
using print(wrftxt(nl)) yields
98.57 98.68 98.72 98.81 98.75 98.75 98.75 98.71 98.74 98.57 98.12 97.50 96.66 95.69 94.67 93.80 92.96 92.26 91.93 91.47 91.11 90.95 90.88 90.97 91.38 91.84 92.68 93.84 94.49 94.72 95.09 95.22 95.61 95.84 95.95 95.95 96.13 95.52 94.99 94.28 93.33 92.13 91.21 90.88 90.77 90.92 91.21 91.63 91.78 91.51 91.12 90.74 90.42 90.19 90.09 90.23 90.43 90.52 90.58 90.47 90.46 90.48 90.56 90.62 90.67 90.72 90.71 90.69 90.62 90.60 90.56 98.37
Even after changing %9.2f" to "%6.2f" or "%5.1f" , I am still getting the "killed" message.
Regards
Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University
Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525
Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.lynch at curtin.edu.au>
Web | http://curtin.edu.au<http://curtin.edu.au/>
Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)
________________________________
From: Dennis Shea <shea at ucar.edu<mailto:shea at ucar.edu>>
Sent: Friday, 4 January 2019 3:10:59 AM
To: Muhammad Omer Mughal
Cc: Rick Brownrigg; ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file
[1] I doubt it is the size of the ascii file.
[2] I tried the script on an available wrf file that had only one time dimension.
Variable: rh2
Dimensions and sizes: [Time | 1] x [south_north | 108] x [west_east | 114]
and the script worked fine. It put out 108 latiitudes and 1*114 Time/longitudes
---
You can add a print to provide more information. It will tell you when it failed.
Did it fail on the 1st iteration or later? Really, just saying it terminated provides little information.
wrftxt = new(nlat, "string")
wrftxt = ""
do nl=0,nlat-1
do nt=0,ntim-1
print("nl="+nl+"; nt="+nt)
do ml=0,mlon-1
wrftxt(nl) = wrftxt(nl) +sprintf("%9.2f", rh2(nt,nl,ml)) ; append to previous value
end do
end do
----
If it failed on the 1st iteration, I speculate that the underlying C-code has some string length limit.
[Time | 673] x [south_north | 71] x [west_east | 82]
Each single string has 673*82*(nchar) where nchar is the length of each string.
So: "%9.2f" yields string lengths of 673*82*9= 496674 characters. That is a LONG string length.
Change "%9.2f" to "%6.2f" or "%5.1f" ... see if it works. Still, they are all LONG.
---
Play with these
NTIM = 50 ; whatever
MLON = ???
do nt=0,NTIM
print("nl="+nl+"; nt="+nt)
do ml=0,MLON
DO various combinations. That will provides some insight into the issue.
On Thu, Jan 3, 2019 at 4:53 AM Muhammad Omer Mughal <m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>> wrote:
Dear Dennis and all
Thank you for reply and sending the code.
However, the script gets killed by itself without giving an error at the end of do loop. This I suspect might be due the fact that string is getting overflow.
Is there a way to fix this ?
Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University
Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525
Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.mughal1 at postgrad.curtin.edu.au>
Web | http://curtin.edu.au
Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)
________________________________
From: Dennis Shea <shea at ucar.edu<mailto:shea at ucar.edu>>
Sent: Thursday, 3 January 2019 12:17:16 PM
To: Muhammad Omer Mughal
Cc: Rick Brownrigg; ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file
This series of questions is a bit frustrating.
You have used NCL quite a bit and you have sent numerous questions.
As you know, ncl-talk always recommends that users use 'printVarSummary' to look at variables. You should do that.... ALWAYS. I don't have your file but
rh2 = wrf_user_getvar(a,"rh2",-1) ; [Time | 673] x [south_north | 71] x [west_east | 82]
lat=wrf_user_getvar(a,"XLAT",-1) ; "
lon=wrf_user_getvar(a,"XLONG",-1) ; "
Your initial question:
lat and lon are variables with a 'Time' dimension.
You can not use multidimensional variables. The following is not correct syntax.
allRH = rh2(:, lon, lat) ; ??????????????????
Please read the User Manual on subscripting and 'do' loops
http://www.ncl.ucar.edu/Document/Manuals/
also
http://www.ncl.ucar.edu/Applications/write_ascii.shtml<http://www.ncl.ucar.edu/Applications/write_ascii.shtml>
========
rh2 = wrf_user_getvar(a,"rh2",-1)
printVarSummary(rh2)
lat = wrf_user_getvar(a,"XLAT",-1)
lon = wrf_user_getvar(a,"XLONG",-1)
printVarSummary(lat)
dimrh2 = dimsizes(rh2)
ntim = dimrh2(0)
nlat = dimrh2(1)
mlon = dimrh2(2)
wrftxt = new(nlat, "string")
wrftxt = ""
do nl=0,nlat-1
do nt=0,ntim-1
do ml=0,mlon-1
wrftxt(nl) = wrftxt(nl) +sprintf("%9.2f", rh2(nt,nl,ml))
end do
end do
end do
filo = "rh2.csv"
system("/bin/rm -f "+filo)
asciiwrite(filo,wrftxt)
On Wed, Jan 2, 2019 at 8:28 PM Muhammad Omer Mughal <m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au><mailto:m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>>> wrote:
Hi Rick
Thank you for the reply.
Yes , I want to write rh2 to the ascii format. The size of the rh2 in this particular case is [Time|673]*[south_north|81]*[west_east|72]. I want to save the file in the format so that it would have 72 rows and 54513 columns i.e. (673*81) for a particular code to read it.
Regards
Omer
Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University
Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525
Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.mughal1 at postgrad.curtin.edu.au><mailto:m.lynch at curtin.edu.au<mailto:m.lynch at curtin.edu.au>>
Web | http://curtin.edu.au<http://curtin.edu.au/>
Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)
________________________________
From: Rick Brownrigg <brownrig at ucar.edu<mailto:brownrig at ucar.edu><mailto:brownrig at ucar.edu<mailto:brownrig at ucar.edu>>>
Sent: Thursday, 3 January 2019 10:59:30 AM
To: Muhammad Omer Mughal
Cc: ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu><mailto:ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>>
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file
HI,
That error message makes sense, since in a WRF file, the variables XLAT and XLONG are 2D (curvilinear).
I guess I'm confused as to what you are trying to do. If you just simply want the variable "rh2" written to an ascii file, instead of this:
allRH = rh2(:, lon, lat)
asciiwrite("rh2.csv",allRH)
what about just:
asciiwrite("rh2.csv",rh2)
A simple experiment revealed that this will write the 3D array, 1 value per line, in "row major" format. This means to read such a file later, you must have some way of knowing the shape and dimensionality.
Hope that helps....
Rick
On Wed, Jan 2, 2019 at 7:42 PM Muhammad Omer Mughal <m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au><mailto:m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>>> wrote:
Dear Rick
Thanks for the reply. I changed the script to remove the SEGV error but now I am getting the following error messages
Please see the script below
a=addfile("wrfout_d03_2017-02-01_00:00:00.nc<http://00.nc><http://00.nc>","r")
rh2 = wrf_user_getvar(a,"rh2",-1)
lat=wrf_user_getvar(a,"XLAT",-1)
lon=wrf_user_getvar(a,"XLONG",-1)
allRH = rh2(:, lon, lat)
asciiwrite("rh2.csv",allRH)
fatal:Illegal subscript. Subscripts must be scalar or one dimensional vectors
Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University
Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525
Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.mughal1 at postgrad.curtin.edu.au><mailto:m.lynch at curtin.edu.au<mailto:m.lynch at curtin.edu.au>>
Web | http://curtin.edu.au<http://curtin.edu.au/>
Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)
________________________________
From: Rick Brownrigg <brownrig at ucar.edu<mailto:brownrig at ucar.edu><mailto:brownrig at ucar.edu<mailto:brownrig at ucar.edu>>>
Sent: Wednesday, 2 January 2019 10:18:33 PM
To: Muhammad Omer Mughal
Cc: ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu><mailto:ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>>
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file
Hi,
Can you share your script and dataset with me? I would like to investigate why there's a SEGV.
Rick
On Wed, Jan 2, 2019 at 4:49 AM Muhammad Omer Mughal <m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au><mailto:m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>>> wrote:
Dear NCL team
Is there a way to extract all values from WRF output files from all grind points and write to an ascii file.
In my case grid size is 71x82 and the number of times is 673. I tried to write using a table format but I am getting a segmentation fault.
I will appreciate help. Kindly note that I have used Lambert conformal projection method.
Regards
Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University
Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525
Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.mughal1 at postgrad.curtin.edu.au><mailto:m.mughal1 at postgrad.curtin.edu.au<mailto:m.mughal1 at postgrad.curtin.edu.au>>
Web | http://curtin.edu.au
Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)
_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu><mailto:ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>>
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu><mailto:ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>>
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190106/4408a0a4/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: response_NCL.docx
Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document
Size: 12033 bytes
Desc: response_NCL.docx
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190106/4408a0a4/attachment.docx>
More information about the ncl-talk
mailing list