<div dir="ltr">The "problem" is the SEVERI data. 75% of the values are outside the valid_range. This is why ncl-talk says *Know your data* .... If you have problems, contact the SEVERI project.<div><br></div><div>The attached plot shows all values plotted. The stat_dispersion shows many bogus values if you do not filter the data using the 'valid_range' attribute.</div><div><br></div><div>Again ... it is *your* responsibility to 'know your data'</div><div><br></div>(0) ctt_1: min=-476.9899893384427 max=176.569996053353<br><br>(0) ===> Robust Dispersion Statistics <===<br><br>(0) [0] Mean=-216.7663134131998<br>(0) [1] StdDev=9907.737962740564<br>(0) [2] Min=-476.9899893384427<br>(0) [3] LowDec=-299.9999932944775<br>(0) [4] LowOct=-299.9999932944775<br>(0) [5] LowSex=-299.9999932944775<br>(0) [6] LowQuartile=-299.9999932944775<br>(0) [7] LowTri=-249.9899944122881<br>(0) [8] Median=-249.9899944122881<br>(0) [9] HighTri=-249.9899944122881<br>(0) [10] HighQuartile=-249.9899944122881<br>(0) [11] HighSex=-67.70999848656356<br>(0) [12] HighOct=-52.53999882563949<br>(0) [13] HighDec=-36.61999918147922<br>(0) [14] Max=176.569996053353<br>(0) [15] Range=653.5599853917956<br>(0) [16] Dispersion=0.0659646013903072<br>(0) [17] RMS Anomaly=99.53761730878492<br>(0) [18] # Total=13778944<br>(0) [19] # Used=13778644<br>(0) [20] # Missing=300<br>(0) [21] % Missing=0.002177235062428284<br>(0) [22] Lower 0.1%=-299.9999932944775<br>(0) [23] Lower 1.0%=-299.9999932944775<br>(0) [24] Lower 5.0%=-299.9999932944775<br>(0) [25] Upper 5.0%=-15.39999965578318<br>(0) [26] Upper 1.0%=-5.399999879300594<br>(0) [27] Upper 0.1%=-0.119999997317791<br>(0) [28] Skewness=1.096054549822515<br>(0) [29] Kurtosis=-0.4490254704621148<div><br></div><div>To plot this data took about 90 sec on a MAC.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jul 2, 2015 at 12:01 AM, Zhifeng Yang <span dir="ltr"><<a href="mailto:yangzf01@gmail.com" target="_blank">yangzf01@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Dennis,<br></div>Thank you so much for plotting it out. From the image you sent, it seems unreasonable around the line of the 0 degree longitude. I am not sure whether it's the problem caused by the SEVIRI data. So how long will it take you to plot this image out? I was trying to plot the pixel level image. Unfortunately, it took me so long time (more than 2 hours). So the alternative way is that I am interpolating the pixel data into 0.25degree*0.25degree. Here I attached a figure of COT. BTW, do you mind send me a copy of your code? I'd like to plot the pixel level image. Thank you, Zhifeng<br><img alt="Inline image 1" src="cid:ii_14e4d5a4c67c81e7" height="563" width="561"><br><br><br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jun 30, 2015 at 2:58 PM, Dennis Shea <span dir="ltr"><<a href="mailto:shea@ucar.edu" target="_blank">shea@ucar.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">[1] Well the way to go is ESMF regridding. See last comment.<div><br></div><div>[2] Did you look at the data? You should.</div><div><br></div><div>[3] </div><div>The files are poorly constructed. The whole point of netCDF/HDF is to be self describing. This file has issues. Consider the </div><div><br></div><div><p> f = addfile("SEV..., "r")<br> lat2d := f->MSG_Latitude<br> printVarSummary(lat2d)Variable: lat2d<br></p>Type: float<br>Total Size: 55115776 bytes<br> 13778944 values<br>Number of Dimensions: 2<br>Dimensions and sizes: [fakeDim0 | 3712] x [fakeDim1 | 3712]<br>Coordinates: <br>Number Of Attributes: 1<br> hdf_name : MSG_Latitude<br><br>There is *no* indication of a missing values. Yet ..</div><div><br></div><div>printMinMax(lat2d,0) ===> lat2d: min=-999.999 max=81.2637</div><div>
<p>-999.999 should be flagged to indicate missing (_FillValue)</p><p>lat2d@_FillValue = -999.999</p><p>If you now count the number of _FillValue, you would find that 25% of the points have missing coordinates. This should be accounted for in any regridding.</p>
dim2d = dimsizes(lat2d)<br>kFill = num(ismissing(lat2d)) <br>print("kFill="+kFill+" TotalGridPoints="+product(dim2d))<p>The data for many variables are packed with scale and offset. Yet there is* no indication on the file* of how to unpack the variables. Maybe it is ancillary documentation. *If so, that is your responsibility to read and account for it in the script.*</p></div><div><br></div><b>shor</b>t Cloud_Top_Temperature ( fakeDim69, fakeDim70 ) <= type 'short'<br>units : K<br>long_name : Cloud Top Temperature from in-house ...<br>actual_range : ( -32699, 32657 )<br>Num_Fill : 0<br>Num_Valid : 13778944<br><b>valid_range</b> : ( 0, 20000 )<br>_FillValue : -999<br>scale_factor : 0.009999999776482582 <=========== 0.01<div>scale_factor_err : 0<br>add_offset : -15000 <===========<br>add_offset_err : 0</div><div><br></div><div><div><div>There are 4 ways to unpack</div></div></div><br> ctt_1 = CTT@scale_factor*(CTT + CTT@add_offset)<br> ctt_2 = CTT@scale_factor*(CTT - CTT@add_offset)<br> ctt_3 = CTT@scale_factor*CTT + CTT@add_offset<div> ctt_4 = CTT@scale_factor*CTT - CTT@add_offset</div><div><br></div><div>Print out the min/max of each. Then select the one most appropriate.</div><div>Hint: it is this one</div><div>
<p> ctt := CTT@scale_factor*(CTT + CTT@add_offset) ; unpacked </p><br> ; valid_range : ( 0, 20000 ); type short<br> vrng = CTT@valid_range</div><div><br> ; set all outside valid range to _FillValue<br> CTT = where(CTT.ge.vrng(0) .and. CTT.le.vrng(1), CTT, CTT@_FillValue) print("CTT: min="+min(CTT)+" max="+max(CTT)) <br><p>Look at the unpacked values. Often, satellite data have outliers. Do you want to keep them or not??<br></p> opt = True<br> opt@PrintStat = True<br> ctt_stat = stat_dispersion(ctt, opt ) ; **READ** documentation</div><div><br></div>;*********************************<br>; Use information from stat_dispersion to filter some outliers<br>;*********************************<br>ctt = where(ctt.lt.-80.0 .or. ctt.gt.-5.0, ctt@_FillValue, ctt)<div><br></div><div>PLOT the raw data ....</div><div><br></div><div>You said 0.5x0.5 .... do you really think you can get that from the available data?</div><div><br></div><div>====</div><div>Rememer the 1st rule of data processing is the ** know your data **.</div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Mon, Jun 29, 2015 at 4:54 PM, Zhifeng Yang <span dir="ltr"><<a href="mailto:yangzf01@gmail.com" target="_blank">yangzf01@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr"><div><div><div>Hi <br><br></div>I am using SEVIRI data now, and trying to interpolate the pixel data into regular grid data, like 0.5deg*0.5. I wrote a procedure to interpolate the data. Unfortunately, I need to do a lot of loops to complete the interpolation, which costs a great amount of time. So the process is super slow. I searched on the NCL website to find a similar function or procedure which has the similar function. However, I did not find one. So do we have this kind of function or procedure in NCL to interpolate the pixel data into regular grid data?<br><br></div>Thank you,<br></div>Zhifeng<br></div>
<br></div></div>_______________________________________________<br>
ncl-talk mailing list<br>
<a href="mailto:ncl-talk@ucar.edu" target="_blank">ncl-talk@ucar.edu</a><br>
List instructions, subscriber options, unsubscribe:<br>
<a href="http://mailman.ucar.edu/mailman/listinfo/ncl-talk" rel="noreferrer" target="_blank">http://mailman.ucar.edu/mailman/listinfo/ncl-talk</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>