[ncl-talk] smops interpolation

Fri Sep 14 22:27:23 MDT 2018

 Dear DennisThank you so much for your helpful replies.
Pardon me if I ask some questions with my responsibility to solve them. However sometimes some problems take much time and finally I couldn't find any solution for them. I'll be thankful if I could have ncl-talk help to overcome them. Beside that, solving some problems need more experiences, which I still don't have enough experience with ncl and also some data format. I always query my data with "panoply" or "ncl_filedump" or "ncdump -h", but the problem is that I don't know what should I do. It is obvious helping in these problems are priceless and until now I've got many helps from ncl-talk which kicking me forward much at doing my thesis. I just could appreciate for these helps.
However, it could be disturbing (!), but I encountered same problem with another data set, AMSR2.
First I want to know whether is there any ncl script for AMSR2. I couldn't find anything in (https://www.ncl.ucar.edu/Applications/). If not, may I ask some help for AMSR2 data? I've attached three samples data of AMSR2. These data set don't have _FillValue, although it includes "-32767" (default _FillValue of short type) and "-32768" (default missing Value).So I've tried
sm at _FillValue := getVarFillValue(sm)
But what should I do about "-32768" values (missing_value)?Also, since that ( Geophysical_Data) in AMSR2 is 3 dimensional I've changed some lines as below:     sm_new  = where(lsm.eq.0 , sm_new at _FillValue , sm_new(:,:,0))     sm_new  = sm_new(::-1,:,:)
     sm_region := sm_new({latS:latN},{lonL:lonR},:)
     AMSR2_fo = linint2_points(sm_region&longitude, sm_region&latitude, sm_region(time | :, latitude | :, longitude | :), True , stlon, stlat, 0)

Are they true?
Another problem that is related to handle many h5 data, which resulted to following error:HDF5-DIAG: Error detected in HDF5 (1.10.1) thread 0:  #000: H5F.c line 408 in H5Fis_hdf5(): unable open file    major: File accessibilty    minor: Not an HDF5 file  #001: H5Fint.c line 532 in H5F__is_hdf5(): unable to open file...

Could I have any reference page in ncl website or any suggestion to overcome these problems.
SincerelyEhsan    On Friday, September 14, 2018, 8:10:40 AM GMT+4:30, Dennis Shea <shea at ucar.edu> wrote:  

 SMOPS: soil Moisture Operational Products System 
https://www.ospo.noaa.gov/Products/land/smops/
Values over land only

[1] 
The most important rule in data processing is *Look at your data". 
Sometimes you have to look carefully!This is a user responsibility.

netcdf NPR_SMOPS_CMAP_D20170410 {
dimensions:
        Longitude = 1440 ;
        Latitude = 720 ;
variables:
        short Blended_SM(Latitude, Longitude) ;
                Blended_SM:long_name = "Blended Soil Moisture" ;
                Blended_SM:units = "m^3/m^3" ;
                Blended_SM:FillValue = -999s ;    <=======
                Blended_SM:valid_range = 0s, 10000s ;
                Blended_SM:scale_factor = 0.0001f ;
                Blended_SM:add_offset = 0.f ;[SNIP]

The NPR_SMOPS_CMAP file does not conform to any netCDF convention. Likely an oversight. It uses FillValue rather than _FillValue.  The _ is important!!! NCL  recognizes only _FillValue (Climate and Forecast netCDF Convention) . Hence, NCL's short2flt will not properly unpack the  "short Blended_SM" variable. Likely, this is the source of some of the interpolation issues.
How to fix this? 

Replace:
           sm  = short2flt( SMOPS_fi->Blended_SM )
with 

           sm  = SMOPS_fi->Blended_SM    ; Read variable of type 'short'
           sm at _FillValue = sm at FillValue  ; add correct attribute to original variable
           printVarSummary(sm)                     ; Note _FillValue attribute
           sm  := short2flt( sm )                    ; Use NCL's overwrite syntax  :=

Add coordinates which are not on the file. Another, oversight by the file creator!

          sm!0                          = "latitude"
          sm!1                          = "longitude"
          sm&latitude               = latitude
          sm&longitude            = longitude
          printVarSummary(sm)
          printMinMax(sm, 0)
          print("---")

============[2] Using 'poisson_grid_fill' 

   [a] will 'fill-in' gaps in the satellite mosaic swaths     :-)
   [b] it will also put 'soil moisture' over the oceans and lakes.   :-(

You can use an NCL function to mask out the oceans/lakes/ice/etc
   http://www.ncl.ucar.edu/Document/Functions/Shea_util/landsea_mask.shtml

%> ncl 20_SMOPS.poisson_fill_mask.ncl
creates: SMOPS_FILL_MASK.png
============
[3] The following uses linint2_points_Wrap to perform the station interpolations.
%> 20_SMOPS.poisson_fill_mask_linint2.ncl

On Wed, Sep 12, 2018 at 5:15 AM, Ehsan Taghizadeh <ehsantaghizadeh at yahoo.com> wrote:

 Hi,Again Thank you so much for your helpful comments.
Unfortunately after a few days I still have problem with interpolating data on station points. In spite of using "poisson_grid_fill" and "linmsg" to avoid getting Fill_Value on unstructured points, these don't work; all values for all dates and all points are Fill_value (-0.099900) (attached file: 20_SMOPS_Interp.txt). Could I have any suggestion to get interpolated values other than "-0.099900". I've used this script for some data set and it helps me a lot. But for some data set, like SMOPS data, it gives me just Fill_Value.I know using "knnsearch" in matlab gives values except "-0.099900", however I want to interpolate with ncl.I've attached the script (20_SMOPS_Interp.ncl) and also station points (2_stations.dat) and output file (20_SMOPS_Interp.txt). I've also put zipped input data (SMOPS_Data.tgz) on ftp.
I'll be thankful for any help.
SincerelyEhsan

    On Friday, September 7, 2018, 8:25:16 PM GMT+4:30, Dennis Shea <shea at ucar.edu> wrote:  

 [1] No[2] Yes

      nlat = 720
      latitude = fspan(-49.875, 49.875, nlat)
      latitude at units = "degrees_north"
      latitude at long_name = "latitude"
      latitude!0        = "latitude"
      latitude&latitude =  latitude
      printVarSummary(latitude)
      printMinMax(latitude,0)

      mlon = 1440
      longitude  = lonGlobeFo(mlon, "longitude", "longitude", "degrees_east")  ; 0->360
      longitude  = (/ longitude - 180. /)  ; subtract 180 from all values 
      longitude&longitude = longitude            ; update coordinates       printVarSummary(lon)

      printVarSummary(longitude)
      printMinMax(longitude,0)

      f = addfile("NPR_SMOPS_CMAP_D20170 410.nc","r")
 x = short2flt( f->Blended_SM )
      x&latitude = latitude
      x&longitude = longitude 
      printVarSummary(x)

[3]/[4]
      xo = .... longitudes of land points...
      yo = ... latitudes ....
      xpts = linint2_points(x&longitude,x& latitude,x, False, xo,yo, 0)
      printVarSummary(xpts)
      printMinMax(xpts,0)

On Tue, Sep 4, 2018 at 5:50 AM, Ehsan Taghizadeh <ehsantaghizadeh at yahoo.com> wrote:

Thank you so much for your nice reply, as always.
I did your comments, however I encountered some difficulties and I'll be thankful if I could have any help about them, too.
1. Should I changelatitude = fspan(-49.875, 49.875, nlat)tolatitude = fspan(-89.875, 89.875, nlat)or like longitude, uselat  = latGlobeFo(nlat, "lat", "latitude", "degrees_north")
2. Should I changelatitude at units = "degrees_east"tolatitude at units = "degrees_north"
3. How should I control FillValue (-999s), because after running ncl script, it writes -0.1 for Nan value of the data, both in grided and interpolated output files.4. However the most difficulty is the interpolating part, because all interpolated values for all station points got -0.1, means Nan. There is no value except -0.1 for any station points. Could anybody help me about this?I've attached the output of interpolated file. I'll be thankful for any help.
Sincerely
Ehsan
   On Friday, August 31, 2018, 6:14:53 PM GMT+4:30, Dennis Shea <shea at ucar.edu> wrote:  

 Sometimes, you have to read documentation associated with the file(s)
Soil Moisture Operational Products System (SMOPS)

The file has:

netcdf NPR_SMOPS_CMAP_D20170410 {
dimensions:
        Longitude = 1440 ;
        Latitude = 720 ;
variables:
        short Blended_SM(Latitude, Longitude) ;
                Blended_SM:long_name = "Blended Soil Moisture" ;
                Blended_SM:units = "m^3/m^3" ;
                Blended_SM:FillValue = -999s ;
                Blended_SM:valid_range = 0s, 10000s ;
                Blended_SM:scale_factor = 0.0001f ;
                Blended_SM:add_offset = 0.f ;

[SNIP]
                :Product_Resolution = "0.25 degree" ;
                :Date_Start = "20170410" ;
                :Date_End = "20170410" ;
[SNIP]
 ====
It does not have Latitude[*] and Longitude[*] coordinates. However, *you* can create them. Then, use ESMF or (say) linint2_Wrap or linint2_points_Wrap to perform the interpolation.

====
      nlat = 720
      latitude = fspan(-49.875, 49.875, nlat)
      latitude at units = "degrees_east"
      latitude at long_name = "latitude"
      latitude!0        = "latitude"
      latitude&latitude =  latitude
      printVarSummary(latitude)
      printMinMax(latitude,0)

      mlon = 1440
      longitude  = lonGlobeFo(mlon, "longitude", "longitude", "degrees_east")  ; 0->360
      longitude  = (/ longitude - 180. /)  ; subtract 180 from all values 
      longitude&longitude = longitude            ; update coordinates       printVarSummary(lon)

      printVarSummary(longitude)
      printMinMax(longitude,0)

      f = addfile("NPR_SMOPS_CMAP_D20170 410.nc","r")
 x = short2flt( f->Blended_SM )
      x&latitude = latitude
      x&longitude = longitude 
      printVarSummary(x)

On Fri, Aug 31, 2018 at 5:45 AM, Ehsan Taghizadeh <ehsantaghizadeh at yahoo.com> wrote:

Hi,Until now I've asked some questions related to interpolating some dataset on some unstructured points (synoptic stations). Now I have a new data set which I'm not sure about appropriate method to interpolating. I put one of my data (NPR_SMOPS_CMAP_D20170410.nc) in ftp.cgd.ucar.edu. I'll be thankful if some one could refer me to correct script, like on (https://www.ncl.ucar.edu/ Applications/ESMF.shtml) or (https://www.ncl.ucar.edu/ Applications/regrid.shtml), which interpolates my data on some special unstructured points.However the printVarSummary(Blended_SM) result is as below:
Variable: Blended_SMType: floatTotal Size: 4147200 bytes            1036800 valuesNumber of Dimensions: 2Dimensions and sizes:   [Latitude | 720] x [Longitude | 1440]Coordinates:Number Of Attributes: 5  _FillValue :  1e+20  long_name :   Blended Soil Moisture  units :       m^3/m^3  FillValue :   -999  valid_range : (  0,  1 )

Sincerely
Ehsan
______________________________ _________________
ncl-talk mailing list
ncl-talk at ucar.edu
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/ mailman/listinfo/ncl-talk

______________________________ _________________
ncl-talk mailing list
ncl-talk at ucar.edu
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/ mailman/listinfo/ncl-talk

______________________________ _________________
ncl-talk mailing list
ncl-talk at ucar.edu
List instructions, subscriber options, unsubscribe:
http://mailman.ucar.edu/ mailman/listinfo/ncl-talk

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180915/df9a86a9/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: GW1AM2_1.rar
Type: application/octet-stream
Size: 726114 bytes
Desc: not available
URL: <http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20180915/df9a86a9/attachment-0001.obj>