[ncl-talk] pixel data to regular grid data
Dennis Shea
shea at ucar.edu
Thu Jul 2 08:20:09 MDT 2015
The "problem" is the SEVERI data. 75% of the values are outside the
valid_range. This is why ncl-talk says *Know your data* .... If you have
problems, contact the SEVERI project.
The attached plot shows all values plotted. The stat_dispersion shows many
bogus values if you do not filter the data using the 'valid_range'
attribute.
Again ... it is *your* responsibility to 'know your data'
(0) ctt_1: min=-476.9899893384427 max=176.569996053353
(0) ===> Robust Dispersion Statistics <===
(0) [0] Mean=-216.7663134131998
(0) [1] StdDev=9907.737962740564
(0) [2] Min=-476.9899893384427
(0) [3] LowDec=-299.9999932944775
(0) [4] LowOct=-299.9999932944775
(0) [5] LowSex=-299.9999932944775
(0) [6] LowQuartile=-299.9999932944775
(0) [7] LowTri=-249.9899944122881
(0) [8] Median=-249.9899944122881
(0) [9] HighTri=-249.9899944122881
(0) [10] HighQuartile=-249.9899944122881
(0) [11] HighSex=-67.70999848656356
(0) [12] HighOct=-52.53999882563949
(0) [13] HighDec=-36.61999918147922
(0) [14] Max=176.569996053353
(0) [15] Range=653.5599853917956
(0) [16] Dispersion=0.0659646013903072
(0) [17] RMS Anomaly=99.53761730878492
(0) [18] # Total=13778944
(0) [19] # Used=13778644
(0) [20] # Missing=300
(0) [21] % Missing=0.002177235062428284
(0) [22] Lower 0.1%=-299.9999932944775
(0) [23] Lower 1.0%=-299.9999932944775
(0) [24] Lower 5.0%=-299.9999932944775
(0) [25] Upper 5.0%=-15.39999965578318
(0) [26] Upper 1.0%=-5.399999879300594
(0) [27] Upper 0.1%=-0.119999997317791
(0) [28] Skewness=1.096054549822515
(0) [29] Kurtosis=-0.4490254704621148
To plot this data took about 90 sec on a MAC.
On Thu, Jul 2, 2015 at 12:01 AM, Zhifeng Yang <yangzf01 at gmail.com> wrote:
> Hi Dennis,
> Thank you so much for plotting it out. From the image you sent, it seems
> unreasonable around the line of the 0 degree longitude. I am not sure
> whether it's the problem caused by the SEVIRI data. So how long will it
> take you to plot this image out? I was trying to plot the pixel level
> image. Unfortunately, it took me so long time (more than 2 hours). So the
> alternative way is that I am interpolating the pixel data into
> 0.25degree*0.25degree. Here I attached a figure of COT. BTW, do you mind
> send me a copy of your code? I'd like to plot the pixel level image. Thank
> you, Zhifeng
> [image: Inline image 1]
>
>
>
> On Tue, Jun 30, 2015 at 2:58 PM, Dennis Shea <shea at ucar.edu> wrote:
>
>> [1] Well the way to go is ESMF regridding. See last comment.
>>
>> [2] Did you look at the data? You should.
>>
>> [3]
>> The files are poorly constructed. The whole point of netCDF/HDF is to be
>> self describing. This file has issues. Consider the
>>
>> f = addfile("SEV..., "r")
>> lat2d := f->MSG_Latitude
>> printVarSummary(lat2d)Variable: lat2d
>> Type: float
>> Total Size: 55115776 bytes
>> 13778944 values
>> Number of Dimensions: 2
>> Dimensions and sizes: [fakeDim0 | 3712] x [fakeDim1 | 3712]
>> Coordinates:
>> Number Of Attributes: 1
>> hdf_name : MSG_Latitude
>>
>> There is *no* indication of a missing values. Yet ..
>>
>> printMinMax(lat2d,0) ===> lat2d: min=-999.999 max=81.2637
>>
>> -999.999 should be flagged to indicate missing (_FillValue)
>>
>> lat2d at _FillValue = -999.999
>>
>> If you now count the number of _FillValue, you would find that 25% of the
>> points have missing coordinates. This should be accounted for in any
>> regridding.
>> dim2d = dimsizes(lat2d)
>> kFill = num(ismissing(lat2d))
>> print("kFill="+kFill+" TotalGridPoints="+product(dim2d))
>>
>> The data for many variables are packed with scale and offset. Yet there
>> is* no indication on the file* of how to unpack the variables. Maybe it is
>> ancillary documentation. *If so, that is your responsibility to read and
>> account for it in the script.*
>>
>> *shor*t Cloud_Top_Temperature ( fakeDim69, fakeDim70 ) <= type 'short'
>> units : K
>> long_name : Cloud Top Temperature from in-house ...
>> actual_range : ( -32699, 32657 )
>> Num_Fill : 0
>> Num_Valid : 13778944
>> *valid_range* : ( 0, 20000 )
>> _FillValue : -999
>> scale_factor : 0.009999999776482582 <=========== 0.01
>> scale_factor_err : 0
>> add_offset : -15000 <===========
>> add_offset_err : 0
>>
>> There are 4 ways to unpack
>>
>> ctt_1 = CTT at scale_factor*(CTT + CTT at add_offset)
>> ctt_2 = CTT at scale_factor*(CTT - CTT at add_offset)
>> ctt_3 = CTT at scale_factor*CTT + CTT at add_offset
>> ctt_4 = CTT at scale_factor*CTT - CTT at add_offset
>>
>> Print out the min/max of each. Then select the one most appropriate.
>> Hint: it is this one
>>
>> ctt := CTT at scale_factor*(CTT + CTT at add_offset) ; unpacked
>>
>> ; valid_range : ( 0, 20000 ); type short
>> vrng = CTT at valid_range
>>
>> ; set all outside valid range to _FillValue
>> CTT = where(CTT.ge.vrng(0) .and. CTT.le.vrng(1), CTT, CTT at _FillValue)
>> print("CTT: min="+min(CTT)+" max="+max(CTT))
>>
>> Look at the unpacked values. Often, satellite data have outliers. Do you
>> want to keep them or not??
>> opt = True
>> opt at PrintStat = True
>> ctt_stat = stat_dispersion(ctt, opt ) ; **READ** documentation
>>
>> ;*********************************
>> ; Use information from stat_dispersion to filter some outliers
>> ;*********************************
>> ctt = where(ctt.lt.-80.0 .or. ctt.gt.-5.0, ctt at _FillValue, ctt)
>>
>> PLOT the raw data ....
>>
>> You said 0.5x0.5 .... do you really think you can get that from the
>> available data?
>>
>> ====
>> Rememer the 1st rule of data processing is the ** know your data **.
>>
>> On Mon, Jun 29, 2015 at 4:54 PM, Zhifeng Yang <yangzf01 at gmail.com> wrote:
>>
>>> Hi
>>>
>>> I am using SEVIRI data now, and trying to interpolate the pixel data
>>> into regular grid data, like 0.5deg*0.5. I wrote a procedure to interpolate
>>> the data. Unfortunately, I need to do a lot of loops to complete the
>>> interpolation, which costs a great amount of time. So the process is super
>>> slow. I searched on the NCL website to find a similar function or procedure
>>> which has the similar function. However, I did not find one. So do we have
>>> this kind of function or procedure in NCL to interpolate the pixel data
>>> into regular grid data?
>>>
>>> Thank you,
>>> Zhifeng
>>>
>>> _______________________________________________
>>> ncl-talk mailing list
>>> ncl-talk at ucar.edu
>>> List instructions, subscriber options, unsubscribe:
>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150702/95294186/attachment-0001.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 63379 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150702/95294186/attachment-0002.png
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SEVERI_SOURCE_0.png
Type: image/png
Size: 148370 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150702/95294186/attachment-0003.png
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SEV.ncl
Type: application/octet-stream
Size: 4531 bytes
Desc: not available
Url : http://mailman.ucar.edu/pipermail/ncl-talk/attachments/20150702/95294186/attachment-0001.obj
More information about the ncl-talk
mailing list