[ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file

Mon Jan 7 00:47:25 MST 2019

Dear Dennis

I changed values of

nl =0

nl=14

nl=28

nl=42

nl=56

nl=70

After which  I received the success message and the concatenated RH2.csv file.

Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD  Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University

Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525

Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.lynch at curtin.edu.au>
Web | http://curtin.edu.au<http://curtin.edu.au/>

Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)

________________________________
From: Dennis Shea <shea at ucar.edu>
Sent: Sunday, 6 January 2019 11:34:33 PM
To: Muhammad Omer Mughal
Cc: Rick Brownrigg; ncl-talk at ucar.edu
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file

What to say?

After :

nl=13 Elapsed Time=73.0269

start at nl=14

keep going till u get all the file and then concatenate.

On Sun, Jan 6, 2019 at 12:32 AM Muhammad Omer Mughal <m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>> wrote:

Dear Dennis

Kindly see the output of two iterations attached.

They seem to have the same CPU time consumed when the process gets killed.

Regards

Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD  Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University

Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525

Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.lynch at curtin.edu.au>
Web | http://curtin.edu.au<http://curtin.edu.au/>

Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)

________________________________
From: Dennis Shea <shea at ucar.edu<mailto:shea at ucar.edu>>
Sent: Saturday, 5 January 2019 10:30:38 PM
To: Muhammad Omer Mughal
Cc: Rick Brownrigg; ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file

Depending on your system, the attached will run for 'a while'. It create one asciifile for each latitude. Then uses the unix 'cat' to combine them.

The loop:
   do nl=0,nlat-1

If it fails [ 'killed' ] after (say) iteration 25, then rerun with

   do nl=26,nlat-1

etc
====
Of course, you could create a netCDF file from NCL and then read that into Matlab. Then  use MATLAB to create the required text file.

On Sat, Jan 5, 2019 at 4:40 AM Muhammad Omer Mughal <m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>> wrote:
Hi all

If there is no solution to this problem could some one kindly confirm the method used to calculate rh2 in NCL so that I can prepare the required data in MATLAB.

Regards

Omer

Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD  Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University

Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525

Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.mughal1 at postgrad.curtin.edu.au>
Web | http://curtin.edu.au

 Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)

________________________________
From: Muhammad Omer Mughal
Sent: Friday, 4 January 2019 2:51:16 PM
To: Rick Brownrigg
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file

Dear Rick and Dennis

After doing various experiments, I found that increasing the value of ntim beyond 200 in this particular case causes the process to get killed.

Regards

Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD  Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University

Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525

Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.lynch at curtin.edu.au>
Web | http://curtin.edu.au<http://curtin.edu.au/>

Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)

________________________________
From: Muhammad Omer Mughal
Sent: Friday, 4 January 2019 11:42:19 AM
To: Rick Brownrigg
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file

Dear Rick

Thank you for the reply again.

It always happens at the same stage.

However, as per Dennis suggestion if I assign a value of say 10 to ntim than the code runs fine and I obtain the attached csv file. I am now increasing the value step wise to see exactly at which value the code gets killed.

Regards

Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD  Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University

Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525

Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.lynch at curtin.edu.au>
Web | http://curtin.edu.au<http://curtin.edu.au/>

Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)

________________________________
From: Rick Brownrigg <brownrig at ucar.edu<mailto:brownrig at ucar.edu>>
Sent: Friday, 4 January 2019 11:19:13 AM
To: Muhammad Omer Mughal
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file

Hi Muhammad,
I've never seen a simple terminating message of "Killed" on a Linux system. But a quick google indicates that, yes, it may indeed be a possibility and that it may have to do with memory over-allocation on behalf of the Linux kernel. I don't fully understand, at this point, the circumstances that might lead up to it -- it will take me some research into the matter.

In the mean time, given the print statements that Dennis had you insert in the loops , does it always happen at the same point, or does it vary?

On Thu, Jan 3, 2019 at 7:57 PM Muhammad Omer Mughal <m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>> wrote:

Dear Rick

The WRF model and NCL are both installed on the cluster,however,  I am not using parallel computing for running NCL. I am running it interactively. Please see below:

 Copyright (C) 1995-2014 - All Rights Reserved
 University Corporation for Atmospheric Research
 NCAR Command Language Version 6.2.1
 The use of this software is governed by a License Agreement.
 See http://www.ncl.ucar.edu/ for more details.
Killed

Regards

Omer

Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD  Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University

Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525

Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.lynch at curtin.edu.au>
Web | http://curtin.edu.au<http://curtin.edu.au/>

Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)

________________________________
From: Rick Brownrigg <brownrig at ucar.edu<mailto:brownrig at ucar.edu>>
Sent: Friday, 4 January 2019 10:24:04 AM
To: Muhammad Omer Mughal
Cc: Dennis Shea; ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file

Hi Muhammad,

It would help to see the exact wording of the errors you are seeing. The phrase "killed" makes me think perhaps you are running on a shared computing resource, and the job is getting terminated because of over use of memory, processor, etc.?  Is this the case?

On Thu, Jan 3, 2019 at 7:12 PM Muhammad Omer Mughal <m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>> wrote:

Dear Dennis

Thank you again for the reply and suggestions.

Using print("nl="+nl+";   nt="+nt) provides

(0)    nl=0;   nt=0
(0)    nl=0;   nt=1
(0)    nl=0;   nt=2
(0)    nl=0;   nt=3
(0)    nl=0;   nt=4
:
:
:
(0)    nl=0;   nt=405
(0)    nl=0;   nt=406
(0)    nl=0;   nt=407
(0)    nl=0;   nt=408
(0)    nl=0;   nt=409
(0)    nl=0;   nt=410 and so on

using print(wrftxt(nl)) yields

98.57    98.68    98.72    98.81    98.75    98.75    98.75    98.71    98.74    98.57    98.12    97.50    96.66    95.69    94.67    93.80    92.96    92.26    91.93    91.47    91.11    90.95    90.88    90.97    91.38    91.84    92.68    93.84    94.49    94.72    95.09    95.22    95.61    95.84    95.95    95.95    96.13    95.52    94.99    94.28    93.33    92.13    91.21    90.88    90.77    90.92    91.21    91.63    91.78    91.51    91.12    90.74    90.42    90.19    90.09    90.23    90.43    90.52    90.58    90.47    90.46    90.48    90.56    90.62    90.67    90.72    90.71    90.69    90.62    90.60    90.56    98.37

Even after changing %9.2f" to "%6.2f" or "%5.1f" , I am still getting the "killed" message.

Regards

Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD  Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University

Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525

Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.lynch at curtin.edu.au>
Web | http://curtin.edu.au<http://curtin.edu.au/>

Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)

________________________________
From: Dennis Shea <shea at ucar.edu<mailto:shea at ucar.edu>>
Sent: Friday, 4 January 2019 3:10:59 AM
To: Muhammad Omer Mughal
Cc: Rick Brownrigg; ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file

[1] I doubt it is the size of the ascii file.

[2] I tried the script on an available wrf file that had only one time dimension.

Variable: rh2
Dimensions and sizes:    [Time | 1] x [south_north | 108] x [west_east | 114]

and the script worked fine. It put out 108 latiitudes and 1*114 Time/longitudes
---
You can add a print to provide more information. It will tell you when it failed.
Did it fail on the 1st iteration or later? Really, just saying it terminated provides little information.

  wrftxt = new(nlat, "string")
  wrftxt = ""

  do nl=0,nlat-1

    do nt=0,ntim-1
         print("nl="+nl+";   nt="+nt)
      do ml=0,mlon-1
         wrftxt(nl) = wrftxt(nl) +sprintf("%9.2f", rh2(nt,nl,ml))  ; append to previous value
      end do
    end do

 ----
If it failed on the 1st iteration, I speculate that the underlying C-code has some string length limit.
[Time | 673] x [south_north | 71] x [west_east | 82]

Each single string has 673*82*(nchar)  where nchar is the length of each string.
So: "%9.2f" yields string lengths of  673*82*9= 496674 characters. That is a LONG string length.

Change "%9.2f" to "%6.2f" or "%5.1f" ... see if it works. Still, they are all LONG.

  ---
Play with these

   NTIM = 50   ; whatever
   MLON = ???
   do nt=0,NTIM
         print("nl="+nl+";   nt="+nt)
      do ml=0,MLON

DO various combinations. That will provides some insight into the issue.

On Thu, Jan 3, 2019 at 4:53 AM Muhammad Omer Mughal <m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>> wrote:
Dear Dennis and all

Thank you for reply and sending the code.

However, the script gets killed by itself without giving an error at the end of do loop. This I suspect might be due the fact that string is getting  overflow.

Is there a way to fix this ?

Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD  Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University

Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525

Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.mughal1 at postgrad.curtin.edu.au>
Web | http://curtin.edu.au

Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)

________________________________
From: Dennis Shea <shea at ucar.edu<mailto:shea at ucar.edu>>
Sent: Thursday, 3 January 2019 12:17:16 PM
To: Muhammad Omer Mughal
Cc: Rick Brownrigg; ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file

This series of questions is a bit frustrating.

You have used NCL quite a bit and you have sent numerous questions.
As you know, ncl-talk always recommends  that users use 'printVarSummary' to look at variables. You should do that.... ALWAYS.   I don't have your file but

rh2 = wrf_user_getvar(a,"rh2",-1)       ; [Time | 673] x [south_north | 71] x [west_east | 82]
lat=wrf_user_getvar(a,"XLAT",-1)       ;                                 "
lon=wrf_user_getvar(a,"XLONG",-1)  ;                                 "

Your initial question:

lat and lon are variables with a 'Time' dimension.
You can not use multidimensional variables. The following is not correct syntax.

allRH = rh2(:, lon, lat)   ; ??????????????????

Please read the User Manual on subscripting and 'do' loops
  http://www.ncl.ucar.edu/Document/Manuals/

also

  http://www.ncl.ucar.edu/Applications/write_ascii.shtml<http://www.ncl.ucar.edu/Applications/write_ascii.shtml>

========
  rh2 = wrf_user_getvar(a,"rh2",-1)
  printVarSummary(rh2)

  lat = wrf_user_getvar(a,"XLAT",-1)
  lon = wrf_user_getvar(a,"XLONG",-1)
  printVarSummary(lat)

  dimrh2 = dimsizes(rh2)
  ntim   = dimrh2(0)
  nlat   = dimrh2(1)
  mlon   = dimrh2(2)

  wrftxt = new(nlat, "string")
  wrftxt = ""

  do nl=0,nlat-1

    do nt=0,ntim-1
      do ml=0,mlon-1
         wrftxt(nl) = wrftxt(nl) +sprintf("%9.2f", rh2(nt,nl,ml))
      end do
    end do

  end do

  filo = "rh2.csv"
  system("/bin/rm -f "+filo)
  asciiwrite(filo,wrftxt)

On Wed, Jan 2, 2019 at 8:28 PM Muhammad Omer Mughal <m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au><mailto:m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>>> wrote:
Hi Rick

Thank you for the reply.

Yes , I want to write rh2 to the ascii format. The size of the rh2 in this particular case is [Time|673]*[south_north|81]*[west_east|72]. I want to save the file in the format so that it would have 72 rows and 54513 columns i.e. (673*81) for a particular code to read it.

Regards

Omer

Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD  Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University

Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525

Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.mughal1 at postgrad.curtin.edu.au><mailto:m.lynch at curtin.edu.au<mailto:m.lynch at curtin.edu.au>>
Web | http://curtin.edu.au<http://curtin.edu.au/>

Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)

________________________________
From: Rick Brownrigg <brownrig at ucar.edu<mailto:brownrig at ucar.edu><mailto:brownrig at ucar.edu<mailto:brownrig at ucar.edu>>>
Sent: Thursday, 3 January 2019 10:59:30 AM
To: Muhammad Omer Mughal
Cc: ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu><mailto:ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>>
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file

HI,

That error message makes sense, since in a WRF file, the variables XLAT and XLONG are 2D (curvilinear).

I guess I'm confused as to what you are trying to do.  If you just simply want the variable "rh2" written to an ascii file, instead of this:

    allRH = rh2(:, lon, lat)
    asciiwrite("rh2.csv",allRH)

what about just:

   asciiwrite("rh2.csv",rh2)

A simple experiment revealed that this will write the 3D array, 1 value per line, in "row major" format. This means to read such a file later, you must have some way of knowing the shape and dimensionality.

Hope that helps....
Rick

On Wed, Jan 2, 2019 at 7:42 PM Muhammad Omer Mughal <m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au><mailto:m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>>> wrote:
Dear Rick
Thanks for the reply. I changed the script to remove the SEGV error but now I am getting the following error messages

Please see the script below

a=addfile("wrfout_d03_2017-02-01_00:00:00.nc<http://00.nc><http://00.nc>","r")

rh2 = wrf_user_getvar(a,"rh2",-1)

lat=wrf_user_getvar(a,"XLAT",-1)
lon=wrf_user_getvar(a,"XLONG",-1)

  allRH = rh2(:, lon, lat)

  asciiwrite("rh2.csv",allRH)

fatal:Illegal subscript. Subscripts must be scalar or one dimensional vectors

Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD  Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University

Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525

Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.mughal1 at postgrad.curtin.edu.au><mailto:m.lynch at curtin.edu.au<mailto:m.lynch at curtin.edu.au>>
Web | http://curtin.edu.au<http://curtin.edu.au/>

Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)

________________________________
From: Rick Brownrigg <brownrig at ucar.edu<mailto:brownrig at ucar.edu><mailto:brownrig at ucar.edu<mailto:brownrig at ucar.edu>>>
Sent: Wednesday, 2 January 2019 10:18:33 PM
To: Muhammad Omer Mughal
Cc: ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu><mailto:ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>>
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file

Hi,

Can you share your script and dataset with me?  I would like to investigate why there's a SEGV.

Rick

On Wed, Jan 2, 2019 at 4:49 AM Muhammad Omer Mughal <m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au><mailto:m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>>> wrote:
Dear NCL team

Is there a way to extract all values from WRF output files from all grind points and write to an ascii file.
In my case grid size is 71x82 and the number of times is 673. I tried to write using a table format but I am getting a segmentation fault.

I will appreciate help. Kindly note that I have used Lambert conformal projection method.

Regards

Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD  Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University

Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525

Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.mughal1 at postgrad.curtin.edu.au><mailto:m.mughal1 at postgrad.curtin.edu.au<mailto:m.mughal1 at postgrad.curtin.edu.au>>
Web | http://curtin.edu.au

Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)

_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu><mailto:ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>>
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu><mailto:ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>>
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190107/147a0e4b/attachment.html>