[ncl-talk] convert the double missing_value to float one
Beáta Szabó-Takács
szabo.b at czechglobe.cz
Wed Jun 21 01:05:35 MDT 2017
Hi Dennis,
Thank you for your suggestion. I printed from MPI-ESM-LR-CLM4.8.17_dry_power_sm.nc type of pr values as you suggested:
print(typeof(pr))
print( typeof(pr at _FillValue))
and I got the following results:
[szabo.b at fenix data]$ ncl taylor_1.ncl
Copyright (C) 1995-2017 - All Rights Reserved
University Corporation for Atmospheric Research
NCAR Command Language Version 6.4.0
The use of this software is governed by a License Agreement.
See http://www.ncl.ucar.edu/ for more details.
(0) float
(0) float
I also printed CD_biasn values which resulted _FillValues :
Variable: CD_biasn
Type: float
Total Size: 20 bytes
5 values
Number of Dimensions: 1
Dimensions and sizes: [5]
Coordinates:
Number Of Attributes: 1
_FillValue : -999
(0) -999
(1) -999
(2) -999
(3) -7.358609
(4) -3.67009
In the resulted taylor_bias_wet.png table by taylor_metric_table.ncl these _FillValues are double. I attched the table. Moreover I used taylor_diagram_lh.ncl for taylor diagram which printed the X, Y coordinate values of diagram:
(0) X(n,i) = 0.91817 Y(n,i) = 0.338616
(0) X(n,i) = 0.728601 Y(n,i) = 0.418846
(0) X(n,i) = 0.618642 Y(n,i) = 0.295433
(0) X(n,i) = 1.09463 Y(n,i) = 0.428762
(0) X(n,i) = 0.7988 Y(n,i) = 0.328584
(0) X(n,i) = 1.27678 Y(n,i) = 0.44035
(0) X(n,i) = 1.06292 Y(n,i) = 0.606676
(0) X(n,i) = 0.920748 Y(n,i) = 0.41256
(0) X(n,i) = 1.39532 Y(n,i) = 0.583074
(0) X(n,i) = 0.842161 Y(n,i) = 0.348939
(0) X(n,i) = 0.856069 Y(n,i) = 0.117605
(0) X(n,i) = 0.85604 Y(n,i) = 0.117578
(0) X(n,i) = 0.856069 Y(n,i) = 0.117605
(0) X(n,i) = 0.856069 Y(n,i) = 0.117605
(0) X(n,i) = 0.856069 Y(n,i) = 0.117605
(0) X(n,i) = 0.602354 Y(n,i) = 0.316136
(0) X(n,i) = 1.03864 Y(n,i) = 0.355518
(0) X(n,i) = 0.902709 Y(n,i) = 0.284563
(0) X(n,i) = 0.754752 Y(n,i) = 0.122997
(0) X(n,i) = 0.791836 Y(n,i) = 0.275488
(0) X(n,i) = 0.561078 Y(n,i) = 0.542241
(0) X(n,i) = 0.55108 Y(n,i) = 0.558689
(0) X(n,i) = 0.93554 Y(n,i) = 0.691576
(0) X(n,i) = 0.857712 Y(n,i) = 0.495699
(0) X(n,i) = 0.835235 Y(n,i) = 0.531927
(0) X(n,i) = 0.929597 Y(n,i) = 0.663454
(0) X(n,i) = 0.864055 Y(n,i) = 0.867048
(0) X(n,i) = 1.27496 Y(n,i) = 0.861098
(0) X(n,i) = 1.19209 Y(n,i) = 0.731529
(0) X(n,i) = 0.903928 Y(n,i) = 0.5948
(0) X(n,i) = 0.94527 Y(n,i) = 0.370124
(0) X(n,i) = 0.94527 Y(n,i) = 0.370124
(0) X(n,i) = 0.945275 Y(n,i) = 0.37013
(0) X(n,i) = 0.94527 Y(n,i) = 0.370124
(0) X(n,i) = 0.94527 Y(n,i) = 0.370124
(0) X(n,i) = 9.96921e+36 Y(n,i) = 9.96921e+36
(0) X(n,i) = 9.96921e+36 Y(n,i) = 9.96921e+36
(0) X(n,i) = 9.96921e+36 Y(n,i) = 9.96921e+36
(0) X(n,i) = 0.878889 Y(n,i) = 0.492841
(0) X(n,i) = 0.953258 Y(n,i) = 0.563082
The _FillValues are also double. I attached my script, taylor_diagram_lh.ncl and taylor_metrics_table.ncl. I have sent the MPI-ESM-LR-CLM4.8.17_dry_power_sm.nc by ftp as you suggested.
Thank you for your help in advance!
Have a nice day,
Beata
From: Dennis Shea [mailto:shea at ucar.edu]
Sent: Tuesday, June 20, 2017 3:35 PM
To: Beáta Szabó-Takács <szabo.b at czechglobe.cz>
Cc: Mary Haley <haley at ucar.edu>; ncl-talk (ncl-talk at ucar.edu) <ncl-talk at ucar.edu>
Subject: Re: [ncl-talk] convert the double missing_value to float one
I am Matlab ignorant so I can't answer what Matlab does/does-not do..
NCL has some 'netCDF rules' built into the language:
ncl 0> x := fspan(1,10,10) ; float
ncl 1> x at _FillValue = -999d ; double
fatal:Type Mismatch: The type of missing value could not be converted to type of variable (x)
fatal:["Execute.c":8573]:Execute: Error occurred at or near line 1
NCL will 'help' in some cases
ncl 5> x at _FillValue = -999 ; integer ... here NCL silently 'promotes' the integer to a float
ncl 6> print(x)
ncl 7> print(typeof(x at _FillValue)) ; float
====
ncdump -h MPI-ESM-LR-CLM4.8.17_dry_power.nc<http://MPI-ESM-LR-CLM4.8.17_dry_power.nc>
[snip]
float pr(time, latitude, longitude) ;
pr:_FillValue = -999.f ;
[snip]
Don't be misled by the 'f' ... extension (-999.f).
f = addfile("MPI-ESM-LR-CLM4.8.17_dry_power","r")
pr = f->pr
print(typeof(pr)) ; float
print( typeof(pr at _FillValue)) ; I speculate this will be type double.
; Note the := syntax
pr at _FillValue := totype( pr at _FillValue, typeof(pr) ) ; MH's suggestion
print( typeof(pr at _FillValue))
[snip]
====
You can send the file to
ftp ftp.cgd.ucar.edu<http://ftp.cgd.ucar.edu>
anonymous
your_email
cd incoming
put .....
quit
Then notify ncl-talk after a successful file transfer.
You can attach a clean version of your script to the email.
Cheers
D
On Tue, Jun 20, 2017 at 1:32 AM, Beáta Szabó-Takács <szabo.b at czechglobe.cz<mailto:szabo.b at czechglobe.cz>> wrote:
Dear Dennis,
You are right I agree with you. To be honest I created the netCDF files in Matlab. The missing values were originally NaN values but I converted them to single -999 by:
data(isnan(data))=single(-999);
pr = permute(data,[2,3,1]);
pr = single(pr);
and I defined -999 as _FillValue by:
netcdf.putAtt(ncid1,3,'_FillValue', cast(-999, class(pr)));
I checked the netCDF files with ncdump -h which denoted that _FillValue and pr value are both float:
$ ncdump -h MPI-ESM-LR-CLM4.8.17_dry_power.nc<http://MPI-ESM-LR-CLM4.8.17_dry_power.nc>
netcdf MPI-ESM-LR-CLM4.8.17_dry_power {
dimensions:
longitude = 464 ;
latitude = 201 ;
time = 5479 ;
variables:
float longitude(longitude) ;
longitude:long_name = "longitude" ;
longitude:standard_name = "lon" ;
longitude:units = "degrees_east" ;
float latitude(latitude) ;
latitude:long_name = "latitude" ;
latitude:standard_name = "lat" ;
latitude:units = "degrees_north" ;
double time(time) ;
time:long_name = "time" ;
time:standard_name = "time" ;
time:units = "days since 1949-12-1 00:00:00" ;
float pr(time, latitude, longitude) ;
pr:_FillValue = -999.f ;
pr:long_name = "precipitation" ;
pr:standard_name = "precipitation" ;
pr:units = "mm" ;
pr:cell_methods = "time: mean" ;
}
When I tried to calculate the multi year seasonal mean values with CDO I got the following error message:
cdo yseasmean MPI-ESM-LR-CLM4.8.17_dry_power.nc<http://MPI-ESM-LR-CLM4.8.17_dry_power.nc> MPI-ESM-LR-CLM4.8.17_dry_power_sm.nc<http://MPI-ESM-LR-CLM4.8.17_dry_power_sm.nc>
cdf_put_vara_double : ncid = 131072 varid = 3 val0 = -999.000000
cdf_put_vara_double : varname = pr
Error (cdf_put_vara_double) : NetCDF: Numeric conversion not representable
I could carry out this operator without error message if I use –b 64 option but due to this option the values become double precision. I do not understand why CDO considers _Fill Value to be double?
Beata
From: Dennis Shea [mailto:shea at ucar.edu<mailto:shea at ucar.edu>]
Sent: Monday, June 19, 2017 6:39 PM
To: Mary Haley <haley at ucar.edu<mailto:haley at ucar.edu>>
Cc: Beáta Szabó-Takács <szabo.b at czechglobe.cz<mailto:szabo.b at czechglobe.cz>>; ncl-talk (ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>) <ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>>
Subject: Re: [ncl-talk] convert the double missing_value to float one
A comment about netCDF variables. Since NCL variables follow the netCDF variable model, it holds for NCL variables also.
By rule, the type of the _FillValue should match the type of the variable with which it is associated.
short variable must have a short _Fillvalue
integer variable ... integer _FillValue
float variable ... float _FillValue
double variable ... double _FillValue
---
On Mon, Jun 19, 2017 at 9:38 AM, Mary Haley <haley at ucar.edu<mailto:haley at ucar.edu>> wrote:
Beata,
When going from a 'higher' type to a 'lower' type (double to float, for example), you need to use the reassignment operator (':=') to force the conversion:
p8 at missing_value := doubletofloat(p8 at missing_value)
p8 at _FillValue := p8 at missing_value
It might be better to use the "totype" function:
p8 at _FillValue := totype(p8 at _FillValue,typeof(p8))
p8 at missing_value := totype(p8 at missing_value,typeof(p8))
--Mary
On Mon, Jun 19, 2017 at 1:04 AM, Beáta Szabó-Takács <szabo.b at czechglobe.cz<mailto:szabo.b at czechglobe.cz>> wrote:
Dear NCL Users,
I would like to create a taylor_metrics_table where the missing values are denoted by -999. This part of script is:
season = (/ "DJF","JJA" /)
nSeason = dimsizes(season)
table = new ( (/nCase,nSeason,nSource/), typeof(ratio) )
table(0,:,:) = (/CA_bias, CA_biasn/)
table(1,:,:) = (/CB_bias, CB_biasn/)
table(2,:,:) = (/CC_bias, CC_biasn/)
table(3,:,:) = (/CD_bias, CD_biasn/)
tt_opt = True
tt_opt at tableTitle = "Bias(%)"
tt_opt at pltType= "png" ; "eps" [default], "pdf", "ps"
; "png", "gif" [if you have ImageMajik 'convert']
; tt_opt at color0 = "palegreen2"
; tt_opt at color1 = "tomato2"
tt_opt at color0 = "white"
tt_opt at color1 = "white"
taylor_metrics_table("taylor_bias_dry", source, case ,season, table, tt_opt)
I have some netcdf files where the precipitation (pr) values are float data type, but the _FillValue and missing_value are double. I tried convert these data to float with:
ncap2 -s 'pr=float(pr)' CM5A-MR_RCA4_dry_power_sm.nc CM5A-MR_RCA4_dry_power_sm2.nc
It converted the _FillValue to float but the missing_value are still double:
netcdf CM5A-MR_RCA4_dry_power_sm2 {
dimensions:
time = UNLIMITED ; // (4 currently)
latitude = 201 ;
longitude = 464 ;
variables:
float pr(time, latitude, longitude) ;
pr:_FillValue = -999.f ;
pr:cell_methods = "time: mean" ;
pr:long_name = "precipitation" ;
pr:missing_value = -999. ;
pr:standard_name = "precipitation_flux" ;
pr:units = "mm" ;
float longitude(longitude) ;
longitude:standard_name = "longitude" ;
longitude:long_name = "longitude" ;
longitude:units = "degrees_east" ;
longitude:axis = "X" ;
float latitude(latitude) ;
latitude:standard_name = "latitude" ;
latitude:long_name = "latitude" ;
latitude:units = "degrees_north" ;
latitude:axis = "Y" ;
double time(time) ;
time:standard_name = "time" ;
time:long_name = "time" ;
time:units = "days since 1949-12-1 00:00:00" ;
time:calendar = "standard" ;
time:axis = "T" ;
The resulted table is attached. In the table the missing values are denoted by double values in CD_biasn despite the fact that the printed CD_biasn contains _FillValues:
Variable: CD_biasn
Type: float
Total Size: 20 bytes
5 values
Number of Dimensions: 1
Dimensions and sizes: [5]
Coordinates:
Number Of Attributes: 1
_FillValue : -999
(0) -999
(1) -4.541727
(2) 12.45617
(3) 10.47692
(4) -999
I also tried to convert the missing values to float type by ncl doubletofloat function but it does not work:
p8 at missing_value=doubletofloat(p8 at missing_value)
p8 at _FillValue = p8 at missing_value
delete(p8 at missing_value)
fatal:Type Mismatch: The type of missing value could not be converted to type of variable (p8)
p8_out = addfile("p8.nc<http://p8.nc>","c")
p8_out ->p8 = p8
ncdump -h p8.nc<http://p8.nc>
netcdf p8 {
dimensions:
time = 4 ;
latitude = 201 ;
longitude = 464 ;
variables:
float p8(time, latitude, longitude) ;
p8:_FillValue = -999.f ;
p8:cell_methods = "time: mean" ;
p8:long_name = "precipitation" ;
p8:missing_value = -999. ;
p8:standard_name = "precipitation_flux" ;
p8:units = "mm" ;
double time(time) ;
time:standard_name = "time" ;
time:long_name = "time" ;
time:units = "days since 1949-12-1 00:00:00" ;
time:calendar = "standard" ;
time:axis = "T" ;
float latitude(latitude) ;
latitude:standard_name = "latitude" ;
latitude:long_name = "latitude" ;
latitude:units = "degrees_north" ;
latitude:axis = "Y" ;
float longitude(longitude) ;
longitude:standard_name = "longitude" ;
longitude:long_name = "longitude" ;
longitude:units = "degrees_east" ;
longitude:axis = "X" ;
}
Can someone suggest me a solution?
Thank you for your help in advance!
Kind regards,
Beata
_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20170621/375c4d19/attachment-0001.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: taylor_metrics_table.ncl
Type: application/octet-stream
Size: 6998 bytes
Desc: taylor_metrics_table.ncl
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20170621/375c4d19/attachment-0003.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: taylor_diagram_lh.ncl
Type: application/octet-stream
Size: 24840 bytes
Desc: taylor_diagram_lh.ncl
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20170621/375c4d19/attachment-0004.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: taylor_1.ncl
Type: application/octet-stream
Size: 21178 bytes
Desc: taylor_1.ncl
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20170621/375c4d19/attachment-0005.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: taylor_bias_wet.png
Type: image/png
Size: 92339 bytes
Desc: taylor_bias_wet.png
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20170621/375c4d19/attachment-0001.png
More information about the ncl-talk
mailing list