[ncl-talk] Era-Interim daily precipitation
Dennis Shea
shea at ucar.edu
Sun Jun 10 08:15:29 MDT 2018
There appears to be an unusual bug with 6.4.0 with this file.
This has not been encountered before.The work-around is to not use the
'unlimited' designation.
Please remove the following line:
*filedimdef(ncdf,"time",-1,True) ; make time UNLIMITED*====
Writing the file as NetCDF-4 with time *not* set to unlimited results in a
14 MB file
Writing the file as NetCDF-3, *both *limited and unlimited,results in 14 MB
files
====
A detail:
Specifically: The unusual thing about the 474 MB fie is that it seems to
think it has *chunked *variables with a dimension *size of 1024*:
Variable: TP
Type: float
Total Size: 3586080 values
14344320 bytes
Number of Dimensions: 3
Dimensions and sizes: [ 31 <time | unlimited> x 241 <latitude> x 480
<longitude> ]
* Chunking Info: [ 1024 <time> x 241 <latitude> x 480 <longitude> ]*
The large size is coming from a *time dimension* that it thinks is size
1024:
This has been reported as NCL 2757 <https://jira.ucar.edu/browse/NCL-2757>.
On Sat, Jun 9, 2018 at 12:43 PM, Ehsan Taghizadeh <ehsantaghizadeh at yahoo.com
> wrote:
> Dears,
> In the script to write nc format, fn = "3_ERAI_Prec_2017_Jan_00_12_
> 24_ncl.nc4",
> which is created, has size of about 450 MB! However input file
> fili = "2_ERAI_Prec_2017_Jan_00_12_12.nc"
> is less than 10 MB!
> Also the file which was created with cdo: F_cdo = "
> 2_2_ERAI_Prec_2015_Jan_24_cdo.nc4", is less than 15 MB!
> Could I ask why the created file in ncl has this large size?
> Also how could I have a nc file with smaller size?
>
> Sincerely
> Ehsan
>
>
> On Saturday, June 9, 2018, 1:23:31 AM GMT+4:30, Ehsan Taghizadeh <
> ehsantaghizadeh at yahoo.com> wrote:
>
>
> Thanks a lot. It works correctly.
>
> Sincerely
> Ehsan
>
> On Saturday, June 9, 2018, 1:12:23 AM GMT+4:30, Dennis Shea <shea at ucar.edu>
> wrote:
>
>
> offline:
>
> Please add the following *'filedimdef'* to what I previously sent.
>
> [SNIP]
> fileattdef(fo, fAtt)
>
> *filedimdef(ncdf,"time",-1,True) ; make time UNLIMITED*
>
> var_name = "TP"
> [SNIP]
>
> On Fri, Jun 8, 2018 at 2:00 PM, Dennis Shea <shea at ucar.edu> wrote:
>
> Please use the 'simple method' to write netCDF:
> *https://www.ncl.ucar.edu/ Applications/method_1.shtml
> <https://www.ncl.ucar.edu/Applications/method_1.shtml>*
>
> ===
> ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
> ;;;;;;;;;;;
> ;;;Start: writing tp3_ncl in nc file
> ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
> ;;;;;;;;;;;
> setfileoption("nc", "Format", "NetCDF4")
> dn = "./"
> fn = "3_ERAI_Prec_2017_Jan_00_12_ 24_ncl.nc4"
> pn = dn + fn ; path name
> system("/bin/rm -f " + pn)
>
> fo = addfile(pn, "c")
>
> fAtt = True ; assign file attributes
> fAtt at title = "ERAI: two-timestep sum"
> fAtt at source_file = fili
> fAtt at Conventions = "None"
> fAtt at creation_date = systemfunc("date")
> fileattdef(fo, fAtt)
>
> ;============================= ============================== ========
> var_name = "TP"
> fo->$var_name$ = tp3_ncl
>
>
> On Fri, Jun 8, 2018 at 8:31 AM, Ehsan Taghizadeh <
> ehsantaghizadeh at yahoo.com> wrote:
>
> Dears,
> I did some editions on previous script, and it seems works better.
> However the problem about time dimensions. I'll be thankful if discard
> previous script and for any help about new attached file.
>
> Sincerely
> Ehsan
>
> On Friday, June 8, 2018, 5:54:04 PM GMT+4:30, Ehsan Taghizadeh <
> ehsantaghizadeh at yahoo.com> wrote:
>
>
> Dear Dennis,
> Thank you so much for your script.
> In your script, "tp3_ncl" is exactly what I need, as its values are the
> same as "tp_cdo" values.
> However could I have any help to write this variable (tp3_ncl) in nc file,
> just like (tp) in (2_ERAI_Prec_2015_Jan_00_12_12 .nc <http://12.nc>)?
> I've added between 185-218, I've attached the script. However it is
> absolutely wrong.
>
> I'll be thankful for any help.
>
> Sincerely
> Ehsan
>
>
> On Tuesday, June 5, 2018, 4:26:15 AM GMT+4:30, Dennis Shea <shea at ucar.edu>
> wrote:
>
>
> I am going to add a few more items to a previous response:
>
> The
>
>
> *Golden Rules of data processing are:(a) Look at your data!*
>
> *(b) Know your data!*
>
> *(c) Make sure *you* know what a function or tool is doing to your data*
>
>
> *---*
> The source netcDF '2_2_ERAI_Prec_2015_Jan_00_12_ 12.nc' was created from
> a GRIB file:
>
> history attribute:
> 2018-05-16 09:15:25 GMT by grib_to_netcdf-2.7.3: grib_to_netcdf
> /data/data05/scratch/3c/71/_ma rs-atls00-a562cefde8a29a7288fa
> 0b8b7f9413f7-b4mPzK.grib -o /data/data04/scratch/41/14/_gr
> ib2netcdf-atls17-a562cefde8a29 a7288fa0b8b7f9413f7-PZaKeY.nc -utime" ;
>
> ---
> There are *62 time steps* on the source ERAI file:
>
> yyyymmddhh tp value
> (0) ERAI: 2015010112 0.000330344
>
> *<=== only one value for this calendar day*(1) ERAI: 2015010200
> 0.0051588 ......* two time steps per calendar day* ...
> (2) ERAI: 2015010212 0.00401913
> (3) ERAI: 2015010300 0
> (4) ERAI: 2015010312 0
> (5) ERAI: 2015010400 0
> (6) ERAI: 2015010412 0.000478998
> (7) ERAI: 2015010500 0.00344653
> [SNIP]
> (55) ERAI: 2015012900 0
> (56) ERAI: 2015012912 0
> (57) ERAI: 2015013000 9.35942e-05
> (58) ERAI: 2015013012 7.15703e-05
> (59) ERAI: 2015013100 0.00390901
> (60) ERAI: 2015013112 0
>
> (61) ERAI: 2015020100 0.00183889* <=== only one value for this
> calendar day*
> ---
> To what 'time' does the source ERAI 'time stamp apply"?
> That is ****USER RESPONSIBILITY*** *to know or find the answer.
>
> Three possibilities:
>
> [1] beginning
> [2] end
> [3] 'center-of-mass' ... mid temporal time?
>
> ---
> NCL's '*calculate_daily_values*' with *'ncrit=2*' requires two values on
> the *same calendar day*
> https://www.ncl.ucar.edu/ Document/Functions/
> Contributed/calculate_daily_ values.shtml
> <https://www.ncl.ucar.edu/Document/Functions/Contributed/calculate_daily_values.shtml>
>
> With *ncrit=2* the function returns 30 values ... not 31 or 32. Why?
> Because it requires 2 values per calendar date. It ignores the 1st and
> last times because there is only one time step per calendar day
>
> tp2_ncl
> NCL_tag : calculate_daily_values: arith=sum
>
> ----
> The function 'tags' it with the *1st time step of each *complete*
> calendar day*
>
> (0) NCL: 2015010200 0.00917792
> (1) NCL: 2015010300 0
> (2) NCL: 2015010400 0.000478998
> (3) NCL: 2015010500 0.00879252
> [SNIP]
> (28) NCL: 2015013000 0.000165164
> (29) NCL: 2015013100 0.00390901
>
> ---
> The CDO takes the first two values [0,1], then the next two [2,3], etc.
> It 'tags' the time as the 2nd time step
>
> (0) CDO: 2015010200 0.00548913
> (1) CDO: 2015010300 0.00401912
> (2) CDO: 2015010400 -1.08611e-10
> [SNIP]
> (29) CDO: 2015013100 0.00398058
> (30) CDO: 2015020100 0.00183889
>
> Even though the time 'tags' are the same, they refer to different sums.
> ---
>
> NCL can be made to handle the time and data any way the user wants.
>
> I'll let you look at the attached script and output
>
>
> On Fri, Jun 1, 2018 at 4:03 PM, Ehsan Taghizadeh <
> ehsantaghizadeh at yahoo.com> wrote:
>
> Dears,
> About summation of 12 hours precipitation to obtain daily precipitation,
> as suggested me I used "cdo" and it seems work correctly:
> "cdo -b F32 -f nc4c -z zip timselsum,2 2_2_ERAI_Prec_2015_Jan_00_12_1 2.nc
> <http://12.nc> 2_2_ERAI_Prec_2015_Jan_24_cdo. nc4"
> However I like to know how writing nc files with "Geo2D".
>
> Sincerely
> Ehsan
>
> On Thursday, May 31, 2018, 7:12:17 PM GMT+4:30, Ehsan Taghizadeh <
> ehsantaghizadeh at yahoo.com> wrote:
>
>
> Thank you so much for your nice replies.
> With you comment, it seems work correctly. However I've added some changes
> to the script to have an output file same as input file, I mean about
> dimensions, attributes, and so on.
> However I have some difficulties about it and I'll be thankful if I could
> have your help, again.
> For example about
> fout->tp24_1!1 = tp!1
> fout->tp24_1!2 = tp!2
> I encountered error. It seems they have conflict with following lines:
> fout->latitude = fin->latitude
> fout->longitude = fin->longitude
>
> Beside that I want to have "tp24_1" with dimension "Geo2D" to plot.
> I attached new version of script, and I'll be thankful if I hear from you.
>
> Sincerley
> Ehsan
> On Thursday, May 31, 2018, 6:44:48 PM GMT+4:30, Dennis Shea <shea at ucar.edu>
> wrote:
>
>
> The *golden rule* of data processing is *look at your data*
>
> You have:
>
> tp=f->tp ; <=== type 'short' [ packed data]
>
> However, your file has:
>
> *short* tp ( time, latitude, longitude )
> * scale_factor *: 5.505649013956001e-06
> *add_offset *: 0.1803980955369768
> _FillValue : -32767
> missing_value : -32767
> units : m
> long_name : Total precipitation
>
> You **MUST* *unpack the variable before computing the sums.
>
> tp = *short2flt*(f->tp)
>
> printVarSummary(tp)
> printMinMax(tp,0)
>
> ===
> https://www.ncl.ucar.edu/ Document/Functions/ Contributed/short2flt.shtml
> <https://www.ncl.ucar.edu/Document/Functions/Contributed/short2flt.shtml>
>
>
>
>
>
>
> On Thu, May 31, 2018 at 7:19 AM, Ehsan Taghizadeh <
> ehsantaghizadeh at yahoo.com> wrote:
>
> Dears,
> I've tried sum 12 hour precipitation to obtain 24 hour precipitation using
> attached script.
> However it seems doesn't work correctly!
> I had to sum tp(0,:,:)+tp(1,:,:), also tp(2,:,;)+tp(3,;,:), and so on.
> I uploaded "2_2_ERAI_Prec_2015_Jan_00_12_ 12.nc" in ftp.cgd.ucar.edu.
> I'll be thankful if I hear from you.
>
> Sincerely
> Ehsan
>
> On Wednesday, May 16, 2018, 8:23:32 AM GMT+4:30, Dennis Shea <
> shea at ucar.edu> wrote:
>
>
> This is book keeping. Something like:
>
> loop each year
> loop each month
> loop each day
>
> sum all values occurring between 00:00 and 11:59 inclusive
> ==>PRC(time,lat,lon,0)
> sum all values occurring between 12:00 and 23:59 inclusive
> ==>PRC(time,lat,lon,1)
>
> end day
> end month
> end year
>
>
> On Sun, May 13, 2018 at 9:07 AM, Ehsan Taghizadeh <
> ehsantaghizadeh at yahoo.com> wrote:
>
> Hi every body;
> Could I have any help to get daily precipitation from Era-Interim, please?
> I mean, a way to sum "accumulated precipitation from 00:00 to 12:00" with
> "accumulated precipitation from 12:00 to 24:00"?
>
> Sincerely
> Ehsan
>
> ______________________________ _________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/ mailman/listinfo/ncl-talk
> <http://mailman.ucar.edu/mailman/listinfo/ncl-talk>
>
>
>
> ______________________________ _________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/ mailman/listinfo/ncl-talk
> <http://mailman.ucar.edu/mailman/listinfo/ncl-talk>
>
>
>
> ______________________________ _________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/ mailman/listinfo/ncl-talk
> <http://mailman.ucar.edu/mailman/listinfo/ncl-talk>
>
>
>
> ______________________________ _________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailma n/listinfo/ncl-talk
> <http://mailman.ucar.edu/mailman/listinfo/ncl-talk>
>
>
>
>
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180610/6f709c77/attachment.html>
More information about the ncl-talk
mailing list