[ncl-talk] Era-Interim daily precipitation

Ehsan Taghizadeh ehsantaghizadeh at yahoo.com
Sun Jun 10 16:20:29 MDT 2018


 
Thank you so much for your helpful comments.However could it be related to do not using statements likesetfileoption("nc", "FileStructure", "Advanced")or setfileoption("nc", "Format",  "NetCDF4Classic")Because in the script we don't use these statements while reading input file:[SNIP]
   ;diri = "/Users/shea/Data/ECMWF/ERAI/"
   diri = "/home/taghizade/97_1_17/5_Era-Interim/"   fili = "2_ERAI_Prec_2017_Jan_00_12_12.nc"   pthi = diri+fili
   f    = addfile(pthi, "r")   tp   = short2flt(f->tp)[SNIP]

However I've used them like below:[SNIP] setfileoption("nc", "FileStructure", "Advanced")  ;diri = "/Users/shea/Data/ECMWF/ERAI/"
   diri = "/home/taghizade/97_1_17/5_Era-Interim/"   fili = "2_ERAI_Prec_2017_Jan_00_12_12.nc"   pthi = diri+fili
   f    = addfile(pthi, "r")   tp   = short2flt(f->tp)setfileoption("nc", "Format",  "NetCDF4Classic")[SNIP]
and I didn't see any change, but I'm not expert and I thought may be I don't use correct options.
SincerelyEhsan
    On Sunday, June 10, 2018, 6:45:31 PM GMT+4:30, Dennis Shea <shea at ucar.edu> wrote:  
 
 There appears to be an unusual bug with 6.4.0 with this file. 
This has not been encountered before.The work-around is to not use the 'unlimited' designation.

Please remove the following line:

filedimdef(ncdf,"time",-1,True )      ; make time UNLIMITED

====
Writing the file as NetCDF-4  with time not set to unlimited results in a 14 MB file
Writing the file as NetCDF-3, both limited and unlimited,results in 14 MB files

==== 
 A detail: 
 
Specifically: The unusual thing about the 474 MB fie is that it seems to think it has chunked variables with a dimension size of 1024:
    Variable: TP    Type: float    Total Size: 3586080 values                14344320 bytes    Number of Dimensions: 3    Dimensions and sizes: [ 31 <time | unlimited> x 241 <latitude> x 480 <longitude> ]    Chunking Info: [ 1024 <time> x 241 <latitude> x 480 <longitude> ]
The large size is coming from a time dimension that it thinks is size 1024:This has been reported as  NCL 2757.​ 


On Sat, Jun 9, 2018 at 12:43 PM, Ehsan Taghizadeh <ehsantaghizadeh at yahoo.com> wrote:

 Dears,
In the script to write nc format, fn = "3_ERAI_Prec_2017_Jan_00_12_ 24_ncl.nc4",which is created, has size of about 450 MB! However input filefili = "2_ERAI_Prec_2017_Jan_00_12_12.nc"
is less than 10 MB!Also the file which was created with cdo: F_cdo = "2_2_ERAI_Prec_2015_Jan_24_ cdo.nc4", is less than 15 MB!Could I ask why the created file in ncl has this large size?Also how could I have a nc file with smaller size?
SincerelyEhsan

    On Saturday, June 9, 2018, 1:23:31 AM GMT+4:30, Ehsan Taghizadeh <ehsantaghizadeh at yahoo.com> wrote:  
 
  Thanks a lot. It works correctly.
SincerelyEhsan
    On Saturday, June 9, 2018, 1:12:23 AM GMT+4:30, Dennis Shea <shea at ucar.edu> wrote:  
 
 offline:

Please add the following 'filedimdef'  to what I previously sent.

[SNIP] 
fileattdef(fo, fAtt)

 filedimdef(ncdf,"time",-1, True)      ; make time UNLIMITED

 var_name = "TP"
[SNIP]

On Fri, Jun 8, 2018 at 2:00 PM, Dennis Shea <shea at ucar.edu> wrote:

Please use the 'simple method' to write netCDF:
https://www.ncl.ucar.edu/ Applications/method_1.shtml

===
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;;;;;;;;;;;   
;;;Start: writing tp3_ncl in nc file
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;;;;;;;;;;;   
 setfileoption("nc", "Format",  "NetCDF4")
 dn = "./"
 fn = "3_ERAI_Prec_2017_Jan_00_12_ 24_ncl.nc4"
 pn = dn + fn                          ; path name
 system("/bin/rm -f " + pn)

 fo = addfile(pn, "c")

 fAtt               = True            ; assign file attributes
 fAtt at title         = "ERAI: two-timestep sum"
 fAtt at source_file   = fili
 fAtt at Conventions   = "None"
 fAtt at creation_date = systemfunc("date")
 fileattdef(fo, fAtt)

;============================= ============================== ========
 var_name = "TP"
 fo->$var_name$ = tp3_ncl


On Fri, Jun 8, 2018 at 8:31 AM, Ehsan Taghizadeh <ehsantaghizadeh at yahoo.com> wrote:

 Dears,I did some editions on previous script, and it seems works better.However the problem about time dimensions. I'll be thankful if discard previous script and for any help about new attached file.
SincerelyEhsan
    On Friday, June 8, 2018, 5:54:04 PM GMT+4:30, Ehsan Taghizadeh <ehsantaghizadeh at yahoo.com> wrote:  
 
  Dear Dennis,
Thank you so much for your script.In your script, "tp3_ncl" is exactly what I need, as its values are the same as "tp_cdo" values.However could I have any help to write this variable (tp3_ncl) in nc file, just like (tp) in (2_ERAI_Prec_2015_Jan_00_12_12 .nc)?I've added between 185-218, I've attached the script. However it is absolutely wrong.

I'll be thankful for any help.
SincerelyEhsan

    On Tuesday, June 5, 2018, 4:26:15 AM GMT+4:30, Dennis Shea <shea at ucar.edu> wrote:  
 
 I am going to add a few more items to a previous response:

The Golden Rules of data processing are:

(a) Look at your data!
(b) Know your data!
(c) Make sure *you* know what a function or tool is doing to your data
---

The source netcDF '2_2_ERAI_Prec_2015_Jan_00_12_ 12.nc' was created from a GRIB file:

history attribute:
2018-05-16 09:15:25 GMT by grib_to_netcdf-2.7.3: grib_to_netcdf /data/data05/scratch/3c/71/_ma rs-atls00-a562cefde8a29a7288fa 0b8b7f9413f7-b4mPzK.grib -o /data/data04/scratch/41/14/_gr ib2netcdf-atls17-a562cefde8a29 a7288fa0b8b7f9413f7-PZaKeY.nc -utime" ;

---
There are 62 time steps on the source ERAI file:

                  yyyymmddhh     tp value
(0)    ERAI: 2015010112   0.000330344    <=== only one value for this calendar day

(1)    ERAI: 2015010200   0.0051588        ...... two time steps per calendar day ...
(2)    ERAI: 2015010212   0.00401913
(3)    ERAI: 2015010300   0
(4)    ERAI: 2015010312   0
(5)    ERAI: 2015010400   0
(6)    ERAI: 2015010412   0.000478998
(7)    ERAI: 2015010500   0.00344653
[SNIP]
(55)    ERAI: 2015012900   0
(56)    ERAI: 2015012912   0
(57)    ERAI: 2015013000   9.35942e-05
(58)    ERAI: 2015013012   7.15703e-05
(59)    ERAI: 2015013100   0.00390901
(60)    ERAI: 2015013112   0

(61)    ERAI: 2015020100   0.00183889      <=== only one value for this calendar day---
To what 'time' does the source ERAI 'time stamp apply"? 
That is ***USER RESPONSIBILITY*** to know or find the answer.

Three possibilities:

[1] beginning
[2] end
[3] 'center-of-mass'  ... mid temporal time?

---
NCL's 'calculate_daily_values' with 'ncrit=2' requires two values on the same calendar day
           https://www.ncl.ucar.edu/ Document/Functions/ Contributed/calculate_daily_ values.shtml

With ncrit=2 the function returns 30 values ... not 31 or 32. Why?
Because it requires 2 values per calendar date. It ignores the 1st and last times because there is only one time step per calendar day

tp2_ncl
  NCL_tag :     calculate_daily_values: arith=sum

     ----
The function 'tags' it with the 1st time step of each *complete*  calendar day

(0)     NCL: 2015010200   0.00917792
(1)     NCL: 2015010300   0
(2)     NCL: 2015010400   0.000478998
(3)     NCL: 2015010500   0.00879252
[SNIP]
(28)    NCL: 2015013000   0.000165164
(29)    NCL: 2015013100   0.00390901

---
The CDO takes the first two values [0,1], then the next two [2,3], etc. 
It 'tags' the time as the 2nd time step

(0)     CDO: 2015010200   0.00548913
(1)     CDO: 2015010300   0.00401912
(2)     CDO: 2015010400   -1.08611e-10
[SNIP]
(29)    CDO: 2015013100   0.00398058
(30)    CDO: 2015020100   0.00183889

Even though the time 'tags' are the same, they refer to different sums.
---
 
NCL can be made to handle the time and data any way the user wants.

I'll let you look at the attached script and output


On Fri, Jun 1, 2018 at 4:03 PM, Ehsan Taghizadeh <ehsantaghizadeh at yahoo.com> wrote:

 Dears,About summation of 12 hours precipitation to obtain daily precipitation, as suggested me I used "cdo" and it seems work correctly:"cdo -b F32 -f nc4c -z zip timselsum,2 2_2_ERAI_Prec_2015_Jan_00_12_1 2.nc 2_2_ERAI_Prec_2015_Jan_24_cdo. nc4"However I like to know how writing nc files with "Geo2D".
SincerelyEhsan

    On Thursday, May 31, 2018, 7:12:17 PM GMT+4:30, Ehsan Taghizadeh <ehsantaghizadeh at yahoo.com> wrote:  
 
  Thank you so much for your nice replies.With you comment, it seems work correctly. However I've added some changes to the script to have an output file same as input file, I mean about dimensions, attributes, and so on.However I have some difficulties about it and I'll be thankful if I could have your help, again.For example aboutfout->tp24_1!1 = tp!1fout->tp24_1!2 = tp!2I encountered error. It seems they have conflict with following lines:fout->latitude = fin->latitudefout->longitude = fin->longitude
Beside that I want to have "tp24_1" with dimension "Geo2D" to plot.I attached new version of script, and I'll be thankful if I hear from you.
SincerleyEhsan    On Thursday, May 31, 2018, 6:44:48 PM GMT+4:30, Dennis Shea <shea at ucar.edu> wrote:  
 
 The golden rule of  data processing is look at your data

You have:

tp=f->tp     ; <=== type 'short' [ packed data]

However, your file has:

      short tp ( time, latitude, longitude )
         scale_factor : 5.505649013956001e-06
         add_offset :   0.1803980955369768
         _FillValue :   -32767
         missing_value :        -32767
         units :        m
         long_name :    Total precipitation

You *MUST* unpack the variable before computing the sums.

tp = short2flt(f->tp)

printVarSummary(tp)
printMinMax(tp,0)

===
https://www.ncl.ucar.edu/ Document/Functions/ Contributed/short2flt.shtml

 




On Thu, May 31, 2018 at 7:19 AM, Ehsan Taghizadeh <ehsantaghizadeh at yahoo.com> wrote:

 Dears,
I've tried sum 12 hour precipitation to obtain 24 hour precipitation using attached script.However it seems doesn't work correctly!I had to sum tp(0,:,:)+tp(1,:,:), also tp(2,:,;)+tp(3,;,:), and so on.I uploaded "2_2_ERAI_Prec_2015_Jan_00_12_ 12.nc" in ftp.cgd.ucar.edu.I'll be thankful if I hear from you.
SincerelyEhsan
    On Wednesday, May 16, 2018, 8:23:32 AM GMT+4:30, Dennis Shea <shea at ucar.edu> wrote:  
 
 This is book keeping. Something like:

loop  each year
   loop each month
      loop each day
   
        sum all values occurring between 00:00 and 11:59 inclusive ==>PRC(time,lat,lon,0)
        sum all values occurring between 12:00 and 23:59 inclusive ==>PRC(time,lat,lon,1)

     end day
   end month
 end year


On Sun, May 13, 2018 at 9:07 AM, Ehsan Taghizadeh <ehsantaghizadeh at yahoo.com> wrote:

Hi every body;Could I have any help to get daily precipitation from Era-Interim, please?I mean, a way to sum "accumulated precipitation from 00:00 to 12:00" with "accumulated precipitation from 12:00 to 24:00"?
SincerelyEhsan
______________________________ _________________
ncl-talk mailing list
ncl-talk at ucar.edu
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/ mailman/listinfo/ncl-talk



  
______________________________ _________________
ncl-talk mailing list
ncl-talk at ucar.edu
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/ mailman/listinfo/ncl-talk



    
______________________________ _________________
ncl-talk mailing list
ncl-talk at ucar.edu
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/ mailman/listinfo/ncl-talk



    
______________________________ _________________
ncl-talk mailing list
ncl-talk at ucar.edu
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailma n/listinfo/ncl-talk





    
______________________________ _________________
ncl-talk mailing list
ncl-talk at ucar.edu
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/ mailman/listinfo/ncl-talk



  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180610/31a05533/attachment.html>


More information about the ncl-talk mailing list