[ncl-talk] convert the double missing_value to float one

Beáta Szabó-Takács szabo.b at czechglobe.cz
Wed Jun 21 01:05:35 MDT 2017


Hi Dennis,



Thank you for your suggestion. I printed from MPI-ESM-LR-CLM4.8.17_dry_power_sm.nc type of pr values as you suggested:


print(typeof(pr))

print( typeof(pr at _FillValue))

and I got the following results:



[szabo.b at fenix data]$ ncl taylor_1.ncl

Copyright (C) 1995-2017 - All Rights Reserved

University Corporation for Atmospheric Research

NCAR Command Language Version 6.4.0

The use of this software is governed by a License Agreement.

See http://www.ncl.ucar.edu/ for more details.

(0)     float

(0)     float





I also printed CD_biasn values which resulted _FillValues :

Variable: CD_biasn

Type: float

Total Size: 20 bytes

            5 values

Number of Dimensions: 1

Dimensions and sizes:   [5]

Coordinates:

Number Of Attributes: 1

  _FillValue :  -999

(0)     -999

(1)     -999

(2)     -999

(3)     -7.358609

(4)     -3.67009



In the resulted taylor_bias_wet.png table by taylor_metric_table.ncl these _FillValues are double. I attched the table. Moreover I used taylor_diagram_lh.ncl for taylor diagram which printed the X, Y coordinate values of diagram:

(0)     X(n,i) = 0.91817 Y(n,i) = 0.338616

(0)     X(n,i) = 0.728601 Y(n,i) = 0.418846

(0)     X(n,i) = 0.618642 Y(n,i) = 0.295433

(0)     X(n,i) = 1.09463 Y(n,i) = 0.428762

(0)     X(n,i) = 0.7988 Y(n,i) = 0.328584

(0)     X(n,i) = 1.27678 Y(n,i) = 0.44035

(0)     X(n,i) = 1.06292 Y(n,i) = 0.606676

(0)     X(n,i) = 0.920748 Y(n,i) = 0.41256

(0)     X(n,i) = 1.39532 Y(n,i) = 0.583074

(0)     X(n,i) = 0.842161 Y(n,i) = 0.348939

(0)     X(n,i) = 0.856069 Y(n,i) = 0.117605

(0)     X(n,i) = 0.85604 Y(n,i) = 0.117578

(0)     X(n,i) = 0.856069 Y(n,i) = 0.117605

(0)     X(n,i) = 0.856069 Y(n,i) = 0.117605

(0)     X(n,i) = 0.856069 Y(n,i) = 0.117605

(0)     X(n,i) = 0.602354 Y(n,i) = 0.316136

(0)     X(n,i) = 1.03864 Y(n,i) = 0.355518

(0)     X(n,i) = 0.902709 Y(n,i) = 0.284563

(0)     X(n,i) = 0.754752 Y(n,i) = 0.122997

(0)     X(n,i) = 0.791836 Y(n,i) = 0.275488

(0)     X(n,i) = 0.561078 Y(n,i) = 0.542241

(0)     X(n,i) = 0.55108 Y(n,i) = 0.558689

(0)     X(n,i) = 0.93554 Y(n,i) = 0.691576

(0)     X(n,i) = 0.857712 Y(n,i) = 0.495699

(0)     X(n,i) = 0.835235 Y(n,i) = 0.531927

(0)     X(n,i) = 0.929597 Y(n,i) = 0.663454

(0)     X(n,i) = 0.864055 Y(n,i) = 0.867048

(0)     X(n,i) = 1.27496 Y(n,i) = 0.861098

(0)     X(n,i) = 1.19209 Y(n,i) = 0.731529

(0)     X(n,i) = 0.903928 Y(n,i) = 0.5948

(0)     X(n,i) = 0.94527 Y(n,i) = 0.370124

(0)     X(n,i) = 0.94527 Y(n,i) = 0.370124

(0)     X(n,i) = 0.945275 Y(n,i) = 0.37013

(0)     X(n,i) = 0.94527 Y(n,i) = 0.370124

(0)     X(n,i) = 0.94527 Y(n,i) = 0.370124

(0)     X(n,i) = 9.96921e+36 Y(n,i) = 9.96921e+36

(0)     X(n,i) = 9.96921e+36 Y(n,i) = 9.96921e+36

(0)     X(n,i) = 9.96921e+36 Y(n,i) = 9.96921e+36

(0)     X(n,i) = 0.878889 Y(n,i) = 0.492841

(0)     X(n,i) = 0.953258 Y(n,i) = 0.563082



The _FillValues are also double. I attached my script, taylor_diagram_lh.ncl and taylor_metrics_table.ncl. I have sent the MPI-ESM-LR-CLM4.8.17_dry_power_sm.nc by ftp as you suggested.

Thank you for your help in advance!

Have a nice day,

Beata


From: Dennis Shea [mailto:shea at ucar.edu]
Sent: Tuesday, June 20, 2017 3:35 PM
To: Beáta Szabó-Takács <szabo.b at czechglobe.cz>
Cc: Mary Haley <haley at ucar.edu>; ncl-talk (ncl-talk at ucar.edu) <ncl-talk at ucar.edu>
Subject: Re: [ncl-talk] convert the double missing_value to float one

I am Matlab ignorant so I can't answer what Matlab does/does-not do..

 NCL has some 'netCDF rules' built into the language:

ncl 0> x := fspan(1,10,10)          ; float
ncl 1> x at _FillValue = -999d      ; double
fatal:Type Mismatch: The type of missing value could not be converted to type of variable (x)
fatal:["Execute.c":8573]:Execute: Error occurred at or near line 1
NCL will 'help' in some cases

ncl 5> x at _FillValue = -999       ; integer ... here NCL silently 'promotes' the integer to a float
ncl 6> print(x)
ncl 7> print(typeof(x at _FillValue))   ; float

====
ncdump -h MPI-ESM-LR-CLM4.8.17_dry_power.nc<http://MPI-ESM-LR-CLM4.8.17_dry_power.nc>
[snip]
float pr(time, latitude, longitude) ;
pr:_FillValue = -999.f ;
[snip]

Don't be misled by the 'f' ... extension  (-999.f).
  f = addfile("MPI-ESM-LR-CLM4.8.17_dry_power","r")
  pr = f->pr
  print(typeof(pr))                      ; float
  print( typeof(pr at _FillValue))  ; I speculate this will be type double.

                                                 ; Note the := syntax
  pr at _FillValue := totype( pr at _FillValue, typeof(pr) ) ; MH's suggestion
  print( typeof(pr at _FillValue))
[snip]


====
You can send the file to
ftp ftp.cgd.ucar.edu<http://ftp.cgd.ucar.edu>
anonymous
your_email
cd incoming
put .....
quit
Then notify ncl-talk after a successful file transfer.
You can attach a clean version of your script to the email.

Cheers
D





On Tue, Jun 20, 2017 at 1:32 AM, Beáta Szabó-Takács <szabo.b at czechglobe.cz<mailto:szabo.b at czechglobe.cz>> wrote:

Dear Dennis,



You are right I agree with you. To be honest I created the netCDF files in Matlab. The missing values were originally NaN values but I converted them to single -999 by:

data(isnan(data))=single(-999);

pr = permute(data,[2,3,1]);

pr = single(pr);



and I defined -999 as _FillValue by:

netcdf.putAtt(ncid1,3,'_FillValue', cast(-999, class(pr)));



I checked the netCDF files with ncdump -h which denoted that _FillValue and pr value are both float:

$ ncdump -h MPI-ESM-LR-CLM4.8.17_dry_power.nc<http://MPI-ESM-LR-CLM4.8.17_dry_power.nc>

netcdf MPI-ESM-LR-CLM4.8.17_dry_power {

dimensions:

        longitude = 464 ;

        latitude = 201 ;

        time = 5479 ;

variables:

        float longitude(longitude) ;

                longitude:long_name = "longitude" ;

                longitude:standard_name = "lon" ;

                longitude:units = "degrees_east" ;

        float latitude(latitude) ;

                latitude:long_name = "latitude" ;

                latitude:standard_name = "lat" ;

                latitude:units = "degrees_north" ;

        double time(time) ;

                time:long_name = "time" ;

                time:standard_name = "time" ;

                time:units = "days since 1949-12-1 00:00:00" ;

        float pr(time, latitude, longitude) ;

                pr:_FillValue = -999.f ;

                pr:long_name = "precipitation" ;

                pr:standard_name = "precipitation" ;

                pr:units = "mm" ;

                pr:cell_methods = "time: mean" ;

}

When I tried to calculate the multi year seasonal mean values with CDO I got the following error message:

cdo yseasmean MPI-ESM-LR-CLM4.8.17_dry_power.nc<http://MPI-ESM-LR-CLM4.8.17_dry_power.nc> MPI-ESM-LR-CLM4.8.17_dry_power_sm.nc<http://MPI-ESM-LR-CLM4.8.17_dry_power_sm.nc>

cdf_put_vara_double : ncid = 131072 varid = 3 val0 = -999.000000

cdf_put_vara_double : varname = pr



Error (cdf_put_vara_double) : NetCDF: Numeric conversion not representable

I could carry out this operator without error message if I use –b 64 option but due to this option the values become double precision. I do not understand why CDO considers _Fill Value to be double?



Beata


From: Dennis Shea [mailto:shea at ucar.edu<mailto:shea at ucar.edu>]
Sent: Monday, June 19, 2017 6:39 PM
To: Mary Haley <haley at ucar.edu<mailto:haley at ucar.edu>>
Cc: Beáta Szabó-Takács <szabo.b at czechglobe.cz<mailto:szabo.b at czechglobe.cz>>; ncl-talk (ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>) <ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>>
Subject: Re: [ncl-talk] convert the double missing_value to float one

A comment about netCDF variables. Since NCL variables follow the netCDF variable model, it holds for NCL variables also.
By rule, the type of the _FillValue should match the type of the variable with which it is associated.
short variable must have a short _Fillvalue
integer variable ... integer _FillValue
float variable ... float _FillValue
double variable ... double _FillValue

---


On Mon, Jun 19, 2017 at 9:38 AM, Mary Haley <haley at ucar.edu<mailto:haley at ucar.edu>> wrote:
Beata,

When going from a 'higher' type to a 'lower' type (double to float, for example), you need to use the reassignment operator (':=') to force the conversion:


  p8 at missing_value := doubletofloat(p8 at missing_value)

   p8 at _FillValue := p8 at missing_value



It might be better to use the "totype" function:



  p8 at _FillValue := totype(p8 at _FillValue,typeof(p8))

  p8 at missing_value := totype(p8 at missing_value,typeof(p8))


--Mary



On Mon, Jun 19, 2017 at 1:04 AM, Beáta Szabó-Takács <szabo.b at czechglobe.cz<mailto:szabo.b at czechglobe.cz>> wrote:

Dear NCL Users,



I would like to create a taylor_metrics_table where the missing values are denoted by -999. This part of script is:



  season    = (/ "DJF","JJA" /)

  nSeason   = dimsizes(season)



  table     = new ( (/nCase,nSeason,nSource/), typeof(ratio) )

  table(0,:,:) = (/CA_bias, CA_biasn/)

  table(1,:,:) = (/CB_bias, CB_biasn/)

  table(2,:,:) = (/CC_bias, CC_biasn/)

  table(3,:,:) = (/CD_bias, CD_biasn/)





  tt_opt        = True

  tt_opt at tableTitle  =  "Bias(%)"

  tt_opt at pltType= "png"                  ; "eps" [default], "pdf", "ps"

                                         ; "png", "gif" [if you have ImageMajik 'convert']

;  tt_opt at color0 = "palegreen2"

;  tt_opt at color1 = "tomato2"

  tt_opt at color0 = "white"

  tt_opt at color1 = "white"





  taylor_metrics_table("taylor_bias_dry", source, case ,season, table, tt_opt)



I have some netcdf files where the precipitation (pr) values are float data type, but the _FillValue and missing_value are double. I tried convert these data to float with:



ncap2 -s 'pr=float(pr)' CM5A-MR_RCA4_dry_power_sm.nc CM5A-MR_RCA4_dry_power_sm2.nc



It converted the _FillValue to float but the missing_value are still double:

netcdf CM5A-MR_RCA4_dry_power_sm2 {

dimensions:

        time = UNLIMITED ; // (4 currently)

        latitude = 201 ;

        longitude = 464 ;

variables:

        float pr(time, latitude, longitude) ;

                pr:_FillValue = -999.f ;

                pr:cell_methods = "time: mean" ;

                pr:long_name = "precipitation" ;

                pr:missing_value = -999. ;

                pr:standard_name = "precipitation_flux" ;

                pr:units = "mm" ;

        float longitude(longitude) ;

                longitude:standard_name = "longitude" ;

                longitude:long_name = "longitude" ;

                longitude:units = "degrees_east" ;

                longitude:axis = "X" ;

        float latitude(latitude) ;

                latitude:standard_name = "latitude" ;

                latitude:long_name = "latitude" ;

                latitude:units = "degrees_north" ;

                latitude:axis = "Y" ;

        double time(time) ;

                time:standard_name = "time" ;

                time:long_name = "time" ;

                time:units = "days since 1949-12-1 00:00:00" ;

                time:calendar = "standard" ;

                time:axis = "T" ;



The resulted table is attached. In the table the missing values are denoted by double values in CD_biasn despite the fact  that the printed CD_biasn contains _FillValues:



Variable: CD_biasn

Type: float

Total Size: 20 bytes

            5 values

Number of Dimensions: 1

Dimensions and sizes:   [5]

Coordinates:

Number Of Attributes: 1

  _FillValue :  -999

(0)     -999

(1)     -4.541727

(2)     12.45617

(3)     10.47692

(4)     -999



I also tried to convert the missing values to float type by ncl doubletofloat function but it does not work:



p8 at missing_value=doubletofloat(p8 at missing_value)

   p8 at _FillValue = p8 at missing_value

   delete(p8 at missing_value)



fatal:Type Mismatch: The type of missing value could not be converted to type of variable (p8)



   p8_out = addfile("p8.nc<http://p8.nc>","c")

   p8_out ->p8 = p8





ncdump -h p8.nc<http://p8.nc>

netcdf p8 {

dimensions:

        time = 4 ;

        latitude = 201 ;

        longitude = 464 ;

variables:

        float p8(time, latitude, longitude) ;

                p8:_FillValue = -999.f ;

                p8:cell_methods = "time: mean" ;

                p8:long_name = "precipitation" ;

                p8:missing_value = -999. ;

                p8:standard_name = "precipitation_flux" ;

                p8:units = "mm" ;

        double time(time) ;

                time:standard_name = "time" ;

                time:long_name = "time" ;

                time:units = "days since 1949-12-1 00:00:00" ;

                time:calendar = "standard" ;

                time:axis = "T" ;

        float latitude(latitude) ;

                latitude:standard_name = "latitude" ;

                latitude:long_name = "latitude" ;

                latitude:units = "degrees_north" ;

                latitude:axis = "Y" ;

        float longitude(longitude) ;

                longitude:standard_name = "longitude" ;

                longitude:long_name = "longitude" ;

                longitude:units = "degrees_east" ;

                longitude:axis = "X" ;

}





Can someone suggest me a solution?

Thank you for your help in advance!

Kind regards,

Beata


_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk


_______________________________________________
ncl-talk mailing list
ncl-talk at ucar.edu<mailto:ncl-talk at ucar.edu>
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/mailman/listinfo/ncl-talk


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20170621/375c4d19/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: taylor_metrics_table.ncl
Type: application/octet-stream
Size: 6998 bytes
Desc: taylor_metrics_table.ncl
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20170621/375c4d19/attachment-0003.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: taylor_diagram_lh.ncl
Type: application/octet-stream
Size: 24840 bytes
Desc: taylor_diagram_lh.ncl
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20170621/375c4d19/attachment-0004.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: taylor_1.ncl
Type: application/octet-stream
Size: 21178 bytes
Desc: taylor_1.ncl
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20170621/375c4d19/attachment-0005.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: taylor_bias_wet.png
Type: image/png
Size: 92339 bytes
Desc: taylor_bias_wet.png
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20170621/375c4d19/attachment-0001.png 


More information about the ncl-talk mailing list