[ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file

Muhammad Omer Mughal m.mughal1 at graduate.curtin.edu.au
Thu Jan 3 19:57:18 MST 2019


Dear Rick


The WRF model and NCL are both installed on the cluster,however,  I am not using parallel computing for running NCL. I am running it interactively. Please see below:


 Copyright (C) 1995-2014 - All Rights Reserved
 University Corporation for Atmospheric Research
 NCAR Command Language Version 6.2.1
 The use of this software is governed by a License Agreement.
 See http://www.ncl.ucar.edu/ for more details.
Killed


Regards

Omer



Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD  Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University

Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525

Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.lynch at curtin.edu.au>
Web | http://curtin.edu.au<http://curtin.edu.au/>

Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)


________________________________
From: Rick Brownrigg <brownrig at ucar.edu>
Sent: Friday, 4 January 2019 10:24:04 AM
To: Muhammad Omer Mughal
Cc: Dennis Shea; ncl-talk at ucar.edu
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file

Hi Muhammad,

It would help to see the exact wording of the errors you are seeing. The phrase "killed" makes me think perhaps you are running on a shared computing resource, and the job is getting terminated because of over use of memory, processor, etc.?  Is this the case?

On Thu, Jan 3, 2019 at 7:12 PM Muhammad Omer Mughal <m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>> wrote:

Dear Dennis


Thank you again for the reply and suggestions.



Using print("nl="+nl+";   nt="+nt) provides


(0)    nl=0;   nt=0
(0)    nl=0;   nt=1
(0)    nl=0;   nt=2
(0)    nl=0;   nt=3
(0)    nl=0;   nt=4
:
:
:
(0)    nl=0;   nt=405
(0)    nl=0;   nt=406
(0)    nl=0;   nt=407
(0)    nl=0;   nt=408
(0)    nl=0;   nt=409
(0)    nl=0;   nt=410 and so on

using print(wrftxt(nl)) yields

98.57    98.68    98.72    98.81    98.75    98.75    98.75    98.71    98.74    98.57    98.12    97.50    96.66    95.69    94.67    93.80    92.96    92.26    91.93    91.47    91.11    90.95    90.88    90.97    91.38    91.84    92.68    93.84    94.49    94.72    95.09    95.22    95.61    95.84    95.95    95.95    96.13    95.52    94.99    94.28    93.33    92.13    91.21    90.88    90.77    90.92    91.21    91.63    91.78    91.51    91.12    90.74    90.42    90.19    90.09    90.23    90.43    90.52    90.58    90.47    90.46    90.48    90.56    90.62    90.67    90.72    90.71    90.69    90.62    90.60    90.56    98.37


Even after changing %9.2f" to "%6.2f" or "%5.1f" , I am still getting the "killed" message.


Regards


Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD  Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University

Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525

Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.lynch at curtin.edu.au>
Web | http://curtin.edu.au<http://curtin.edu.au/>

Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)


________________________________
From: Dennis Shea <shea at ucar.edu<mailto:shea at ucar.edu>>
Sent: Friday, 4 January 2019 3:10:59 AM
To: Muhammad Omer Mughal
Cc: Rick Brownrigg; ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file

[1] I doubt it is the size of the ascii file.

[2] I tried the script on an available wrf file that had only one time dimension.

Variable: rh2
Dimensions and sizes:    [Time | 1] x [south_north | 108] x [west_east | 114]

and the script worked fine. It put out 108 latiitudes and 1*114 Time/longitudes
---
You can add a print to provide more information. It will tell you when it failed.
Did it fail on the 1st iteration or later? Really, just saying it terminated provides little information.

  wrftxt = new(nlat, "string")
  wrftxt = ""

  do nl=0,nlat-1

    do nt=0,ntim-1
         print("nl="+nl+";   nt="+nt)
      do ml=0,mlon-1
         wrftxt(nl) = wrftxt(nl) +sprintf("%9.2f", rh2(nt,nl,ml))  ; append to previous value
      end do
    end do

 ----
If it failed on the 1st iteration, I speculate that the underlying C-code has some string length limit.
[Time | 673] x [south_north | 71] x [west_east | 82]

Each single string has 673*82*(nchar)  where nchar is the length of each string.
So: "%9.2f" yields string lengths of  673*82*9= 496674 characters. That is a LONG string length.

Change "%9.2f" to "%6.2f" or "%5.1f" ... see if it works. Still, they are all LONG.

  ---
Play with these

   NTIM = 50   ; whatever
   MLON = ???
   do nt=0,NTIM
         print("nl="+nl+";   nt="+nt)
      do ml=0,MLON

DO various combinations. That will provides some insight into the issue.


On Thu, Jan 3, 2019 at 4:53 AM Muhammad Omer Mughal <m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>> wrote:
Dear Dennis and all

Thank you for reply and sending the code.

However, the script gets killed by itself without giving an error at the end of do loop. This I suspect might be due the fact that string is getting  overflow.

Is there a way to fix this ?


Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD  Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University

Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525

Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.mughal1 at postgrad.curtin.edu.au>
Web | http://curtin.edu.au

Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)


________________________________
From: Dennis Shea <shea at ucar.edu<mailto:shea at ucar.edu>>
Sent: Thursday, 3 January 2019 12:17:16 PM
To: Muhammad Omer Mughal
Cc: Rick Brownrigg; ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file

This series of questions is a bit frustrating.

You have used NCL quite a bit and you have sent numerous questions.
As you know, ncl-talk always recommends  that users use 'printVarSummary' to look at variables. You should do that.... ALWAYS.   I don't have your file but

rh2 = wrf_user_getvar(a,"rh2",-1)       ; [Time | 673] x [south_north | 71] x [west_east | 82]
lat=wrf_user_getvar(a,"XLAT",-1)       ;                                 "
lon=wrf_user_getvar(a,"XLONG",-1)  ;                                 "

Your initial question:

lat and lon are variables with a 'Time' dimension.
You can not use multidimensional variables. The following is not correct syntax.

allRH = rh2(:, lon, lat)   ; ??????????????????

Please read the User Manual on subscripting and 'do' loops
  http://www.ncl.ucar.edu/Document/Manuals/

also

  http://www.ncl.ucar.edu/Applications/write_ascii.shtml<http://www.ncl.ucar.edu/Applications/write_ascii.shtml>

========
  rh2 = wrf_user_getvar(a,"rh2",-1)
  printVarSummary(rh2)

  lat = wrf_user_getvar(a,"XLAT",-1)
  lon = wrf_user_getvar(a,"XLONG",-1)
  printVarSummary(lat)


  dimrh2 = dimsizes(rh2)
  ntim   = dimrh2(0)
  nlat   = dimrh2(1)
  mlon   = dimrh2(2)

  wrftxt = new(nlat, "string")
  wrftxt = ""

  do nl=0,nlat-1

    do nt=0,ntim-1
      do ml=0,mlon-1
         wrftxt(nl) = wrftxt(nl) +sprintf("%9.2f", rh2(nt,nl,ml))
      end do
    end do

  end do


  filo = "rh2.csv"
  system("/bin/rm -f "+filo)
  asciiwrite(filo,wrftxt)


On Wed, Jan 2, 2019 at 8:28 PM Muhammad Omer Mughal <m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au><mailto:m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>>> wrote:
Hi Rick

Thank you for the reply.

Yes , I want to write rh2 to the ascii format. The size of the rh2 in this particular case is [Time|673]*[south_north|81]*[west_east|72]. I want to save the file in the format so that it would have 72 rows and 54513 columns i.e. (673*81) for a particular code to read it.

Regards

Omer


Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD  Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University

Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525

Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.mughal1 at postgrad.curtin.edu.au><mailto:m.lynch at curtin.edu.au<mailto:m.lynch at curtin.edu.au>>
Web | http://curtin.edu.au<http://curtin.edu.au/>

Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)


________________________________
From: Rick Brownrigg <brownrig at ucar.edu<mailto:brownrig at ucar.edu><mailto:brownrig at ucar.edu<mailto:brownrig at ucar.edu>>>
Sent: Thursday, 3 January 2019 10:59:30 AM
To: Muhammad Omer Mughal
Cc: ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu><mailto:ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>>
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file

HI,

That error message makes sense, since in a WRF file, the variables XLAT and XLONG are 2D (curvilinear).

I guess I'm confused as to what you are trying to do.  If you just simply want the variable "rh2" written to an ascii file, instead of this:

    allRH = rh2(:, lon, lat)
    asciiwrite("rh2.csv",allRH)

what about just:

   asciiwrite("rh2.csv",rh2)

A simple experiment revealed that this will write the 3D array, 1 value per line, in "row major" format. This means to read such a file later, you must have some way of knowing the shape and dimensionality.

Hope that helps....
Rick


On Wed, Jan 2, 2019 at 7:42 PM Muhammad Omer Mughal <m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au><mailto:m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>>> wrote:
Dear Rick
Thanks for the reply. I changed the script to remove the SEGV error but now I am getting the following error messages


Please see the script below

a=addfile("wrfout_d03_2017-02-01_00:00:00.nc<http://00.nc><http://00.nc>","r")

rh2 = wrf_user_getvar(a,"rh2",-1)

lat=wrf_user_getvar(a,"XLAT",-1)
lon=wrf_user_getvar(a,"XLONG",-1)


  allRH = rh2(:, lon, lat)

  asciiwrite("rh2.csv",allRH)

fatal:Illegal subscript. Subscripts must be scalar or one dimensional vectors





Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD  Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University

Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525

Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.mughal1 at postgrad.curtin.edu.au><mailto:m.lynch at curtin.edu.au<mailto:m.lynch at curtin.edu.au>>
Web | http://curtin.edu.au<http://curtin.edu.au/>

Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)


________________________________
From: Rick Brownrigg <brownrig at ucar.edu<mailto:brownrig at ucar.edu><mailto:brownrig at ucar.edu<mailto:brownrig at ucar.edu>>>
Sent: Wednesday, 2 January 2019 10:18:33 PM
To: Muhammad Omer Mughal
Cc: ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu><mailto:ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>>
Subject: Re: [ncl-talk] Extracting all values of rh2 from WRF file and writing it to csv file

Hi,

Can you share your script and dataset with me?  I would like to investigate why there's a SEGV.

Rick


On Wed, Jan 2, 2019 at 4:49 AM Muhammad Omer Mughal <m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au><mailto:m.mughal1 at graduate.curtin.edu.au<mailto:m.mughal1 at graduate.curtin.edu.au>>> wrote:
Dear NCL team

Is there a way to extract all values from WRF output files from all grind points and write to an ascii file.
In my case grid size is 71x82 and the number of times is 673. I tried to write using a table format but I am getting a segmentation fault.

I will appreciate help. Kindly note that I have used Lambert conformal projection method.

Regards



Muhammad Omer Mughal
MSc BSc Mechanical Engineering
PhD  Research Scholar
Remote Sensing and Satellite Research Group
Department of Imaging and Applied Physics
Curtin University

Curtin University
Tel | +61 8 9266 7962
Fax | +61 8 9266 2377
Mobile | 0470 237 525

Email | m.mughal1 at postgrad.curtin.edu.au<mailto:m.mughal1 at postgrad.curtin.edu.au><mailto:m.mughal1 at postgrad.curtin.edu.au<mailto:m.mughal1 at postgrad.curtin.edu.au>>
Web | http://curtin.edu.au

Curtin University is a trademark of Curtin University of Technology.
CRICOS Provider Code 00301J (WA), 02637B (NSW)


_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu><mailto:ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>>
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu><mailto:ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>>
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20190104/d3917506/attachment-0001.html>


More information about the ncl-talk mailing list