[ncl-talk] Missing data in ERA5?

[Kyle Griffin]

Anne and all,

For more information about ERA5 'expver', please read over the ERA5
documentation. The page below is directly relevant and discusses how the
"near-real-time" analysis transitions to the final analysis after a period
of time. If you re-downloaded your data and did not see this variable, you
likely had your time period of interest age out of the "tentative" analysis
and into the final.

https://confluence.ecmwf.int/pages/viewpage.action?pageId=173385064


Kyle

On Tue, Feb 9, 2021 at 12:06 PM Dennis Shea via ncl-talk <
ncl-talk at mailman.ucar.edu> wrote:

> Hi Anne,
> THX for the update and posting to ncl-talk.  It is good information to
> have.
> Stay Healthy
> D
>
> On Tue, Feb 9, 2021 at 4:03 AM Anne Seidenglanz <annework985 at gmail.com>
> wrote:
>
>> The data has been closely examined. The problem was related to
>> downloading ERA5 data, for some reason a dimension "expver" (presumably
>> experiment version?) appeared in the variable. This dimension did not
>> appear after a fresh download of the data (minor changes in the script but
>> the exact reason remains obscure to me).
>> In any case, if someone happens to have problems plotting ERA5 data, pls
>> check if the dimension "expver" appears, and if necessary re-download the
>> data to end up with e.g. z(time, lev, lat, lon).
>>
>> On Mon, Feb 8, 2021 at 8:38 PM Dennis Shea <shea at ucar.edu> wrote:
>>
>>> The 'golden rule' of data processing is to look at the data.
>>>
>>> netcdf geop_500_975_89_20 {
>>> dimensions:
>>>         longitude = 1440 ;
>>>         latitude = 721 ;                * ; has pole points*
>>>         level = 2 ;
>>>         expver = 2 ;                      *; not sure what this is BUT,
>>> possibly, two different physics packages*
>>>                                                  *; OR, possibly, an
>>> ensemble of size two  OR, ???*
>>>         time = 160 ;
>>> variables:
>>>         float longitude(longitude) ;
>>>                 longitude:units = "degrees_east" ;
>>>                 longitude:long_name = "longitude" ;
>>>         float latitude(latitude) ;
>>>                 latitude:units = "degrees_north" ;
>>>                 latitude:long_name = "latitude" ;
>>>         int level(level) ;
>>>                 level:units = "millibars" ;
>>>                 level:long_name = "pressure_level" ;
>>>         int expver(expver) ;
>>>                 expver:long_name = "expver" ;
>>>         int time(time) ;
>>>                 time:units = "hours since 1900-01-01 00:00:00.0" ;
>>>                 time:long_name = "time" ;
>>>                 time:calendar = "gregorian" ;
>>>         *short z(time, expver, level, latitude, longitude)* ;
>>> <====== rank 5
>>>
>>> *                z:scale_factor = 0.898208868468462          ; netCDF CF
>>> [and COARDS] conventions*                *z:add_offset =
>>> 28656.6907881439 *;             *; scale_factor and add_offet must be
>>> present*
>>>                 *z:_FillValue = -32767s ;
>>>    ; CF                    *
>>>                 z:missing_value = -32767s ;                          ;
>>> COARDS
>>>                 z:units = "m**2 s**-2"
>>> ;                                   ; COARDS and CF conventions
>>>                 z:long_name = "Geopotential" ;                     ;
>>> COARDS convention
>>>                 z:standard_name = "geopotential" ;              ; CF
>>> convention
>>>
>>> ======================================================
>>>
>>> dir="/Users/nuovoaccount/Documents/PYTHON/"
>>> fil ="geop_500_975_89_20.nc"
>>> f  = addfile(dir+fil,"r")
>>> z = *short2flt*
>>> <http://www.ncl.ucar.edu/Document/Functions/Contributed/short2flt.shtml>
>>> (f->z)
>>> printVarSummary(z)    ;  *(time, expver, level, latitude, longitude)*
>>> printMinMax(z,0)
>>>
>>> ---
>>> or,
>>>     [a] noting that NCL* eliminates degenerate dimensions* [size 1]
>>>     [b] no information is lost, since NCL will add variable attributes
>>> for the degenerate dimensions: *expver, level*
>>>
>>> z = *short2flt*
>>> <http://www.ncl.ucar.edu/Document/Functions/Contributed/short2flt.shtml>(
>>> f->z(:,1,1,:,:) )    ; second ensemble member and second level
>>> printVarSummary(z)    ; *(time, latitude, longitude) *with attributes *level
>>> *and* expver*
>>> printMinMax(z,0)
>>>
>>> HTH
>>>
>>>
>>> On Mon, Feb 8, 2021 at 8:46 AM Anne Seidenglanz via ncl-talk <
>>> ncl-talk at mailman.ucar.edu> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am not new to NCL but encounter a rather new problem when dealing
>>>> with ERA5 data (in .nc format, see script attached). I am trying to produce
>>>> a simple test plot for geopotential height, and somehow all my data seems
>>>> to be missing? The error message appears:
>>>> warning:ContourPlotInitialize: no valid values in scalar field;
>>>> ContourPlot not possible:[errno=1101]
>>>> When I printed a subset of the dataset, all the values seemed to be
>>>> missing values.
>>>>
>>>> I have:
>>>> - tested that not all data is missing in Z with "print("Is Z all
>>>> missing ? " + all(ismissing(Z)))". It gave me False.
>>>> - I have tried both, reading in the data using "short2flt" to account
>>>> for scale factor and offset value, and without short2flt. In both cases
>>>> appears the missing value problem and the same error message.
>>>>
>>>> Where have I gone wrong?
>>>>
>>>> Thx for any help
>>>> Anne
>>>> _______________________________________________
>>>> ncl-talk mailing list
>>>> ncl-talk at mailman.ucar.edu
>>>> List instructions, subscriber options, unsubscribe:
>>>> https://mailman.ucar.edu/mailman/listinfo/ncl-talk
>>>
>>> _______________________________________________
> ncl-talk mailing list
> ncl-talk at mailman.ucar.edu
> List instructions, subscriber options, unsubscribe:
> https://mailman.ucar.edu/mailman/listinfo/ncl-talk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.ucar.edu/pipermail/ncl-talk/attachments/20210209/030a72e5/attachment.html>