[ncl-talk] NCL to replace CDO

Marston Johnston shejo284 at gmail.com
Tue Jul 4 03:16:15 MDT 2017


Hi guys,

Apologies for my delayed response. It’s been a very busy time for me with almost no end in sight.

Dave: 

Thanks for looking into the problem for me. My thought was to use NCL in the entire conversion process from grib to netcdf. Your solution does not offer me possibility in a manner where I can 
Semi-automate the process. The experimental grib variables are important for my model group because it is how we output diagnostics and data for modules we have added to the model. 
These variables were only in the ICMGGglx2+200701 and ICMGGECE3+199001. I threw in the spectral grib file to show you all types of grib files my mode produces. The best way that I currently can convert these grib files to netcdf, convert from reduced Gaussian to regular lat lon, and get the experimental variables is using the following bash function, where ${i} is the above file names. Gribset comes from the grib_api library. 

GRIBGG () {
    i=${1}
    echo "Processing file = $i"
    grib_set -s editionNumber=1 $i ${i}.grb
    cdo -R copy ${i}.grb ${i}_R.grb
    cdo -v splitzaxis ${i}_R.grb ${i}_R
    # no need to split axis here as all are 3D variables
    grib_to_netcdf -k 2 -D NC_FLOAT -o ${i}_3d.nc ${i}_R02.grb
    grib_to_netcdf -k 2 -D NC_FLOAT -o ${i}_2d.nc ${i}_R01.grb
    rm -vf ${i}.grb ${i}_R.grb ${i}_R02.grb ${i}_R01.grb
    # merge the 3d and 2d variables
    cdo -v -O merge ${i}_2d.nc ${i}_3d.nc ${i}.nc
    rm -vf ${i}_2d.nc ${i}_3d.nc
    return 0
}

Karin is right, there OpenIFS the newer version of IFS, does mix grib 1 and grib 2 variables, but even when this issue is addressed with grib_set -s editionNumber=1, still need CDO to split the grib file in order to convert the reduced grid to regular lat lon. Further, CDO 1.8.2 miss-translates the variables when converting to netcdf. Therefore I need at the moment a mixture of grib_api and cdo to convert these grib files. The issue I have finally is that I cannot control and examine the data during this process, especially when I want to regrid the data. This is why I asked if NCL could be used for this task.

What is the grib engine that NCL uses under the hood? Perhaps my function above can give a possible future upgrade to NCL to handle this type of grib files?


Karin: 

Thanks for looking into the CDO 1.8.2 issue. Is this project now dead? They use to update the grib tables in CDO but now this project seems to be at a stand still. 


Best,
/M
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Marston S. Johnston, PhD
Department of Earth Sciences
University of Gothenburg, Sweden
Email: marston.johnston at gu.se
SkypeID: marston.johnston 
Phone: +46-31-7864901 
Only the fruitful thing is true!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

On 2017-07-03, 18:38, "David Brown" <ncl-talk-bounces at ucar.edu on behalf of dbrown at ucar.edu> wrote:

    I noticed that wgrib said there was a GRIB2 record in one of the
    files. Oddly, NCL did not seem to notice this and I just filed it away
    as a mystery for now. Ordinarily NCL also fails when GRIB1 and GRIB2
    records are mixed in a single file. So it seems that further
    investigation is warranted.
     -dave
    
    On Mon, Jul 3, 2017 at 9:57 AM, Karin Meier-Fleischer
    <meier-fleischer at dkrz.de> wrote:
    > Hi Marston and Dave,
    >
    > I've tried to remap and convert your files with CDO version 1.8.2 but it
    > fails for two of the three files. The file ICMGGECE3+199001 is a GRIB 1 file
    > and the others seem to have GRIB 1 and GRIB 2 mixed records which can't be
    > handled by CDO.
    >
    > I tried to convert the files to netCDF using the ECMWF grib-api/eccodes GRIB
    > tools but those do not like reduced Gaussian grids.
    >
    > Sorry, can't help yet but maybe we'll find a way to do it with CDO. ;-)
    >
    > Bye,
    > Karin
    >
    > Am 30.06.17 um 18:11 schrieb David Brown:
    >
    >> Ben,
    >> I think Marston cannot use ncl_convert2nc because he wants to
    >> interpolate to a lat/lon grid before saving the data to netcdf.
    >>
    >> Karin,
    >> The files that Marston made available to me are still on
    >> ftp.cgd.ucar.edu in the zip file ifs_grib_types.zip.
    >>   -dave
    >>
    >> On Fri, Jun 30, 2017 at 4:29 AM, Benjamin Lamptey <bllamptey at gmail.com>
    >> wrote:
    >>>
    >>> Hi Marston,
    >>> Maybe I am missing something. Why not use "ncl_convert2nc" command
    >>> instead
    >>> of an NCL script?
    >>>
    >>> Even if you have many files, you can use the command in a shell script.
    >>>
    >>> Sorry if I missed something earlier but the above command is easier to
    >>> convert grib to netcdf.
    >>>
    >>> Regards
    >>> Ben
    >>>
    >>> On Fri, Jun 30, 2017 at 11:10 AM, Karin Meier-Fleischer
    >>> <meier-fleischer at dkrz.de> wrote:
    >>>>
    >>>> Hi Marston,
    >>>>
    >>>> as I'm also working in the CDO consultancy here in Hamburg, I would like
    >>>> to have a closer look at your IFS file, too, to see if the CDO version
    >>>> 1.8.2
    >>>> could handle it.
    >>>>
    >>>> If the file is not too large you can attach it to an 'offline' email to
    >>>> me
    >>>> or upload it creating a new request on
    >>>> https://code.mpimet.mpg.de/projects/cdo/boards/1 .
    >>>>
    >>>> Thanks in advance,
    >>>> Karin
    >>>>
    >>>> Am 27.06.17 um 20:49 schrieb Marston Johnston:
    >>>>
    >>>> Hi,
    >>>>
    >>>>
    >>>>
    >>>> For years I’ve been using CDO to convert ECMWF’s IFS grib files to
    >>>> netcdf
    >>>> and converting it from reduced Gaussian grid to regular lat lon.
    >>>>
    >>>> But CDO has not kept up the pace with updates and they do not respond to
    >>>> many questions.
    >>>>
    >>>> Therefore, I’m attempting to move this part of my processing chain to
    >>>> NCL.
    >>>>
    >>>> I’m using NCL 6.5 (beta) to read the grib file (see attached log file).
    >>>> There 1 or 2 problems and then I have a question.
    >>>>
    >>>>
    >>>>
    >>>> 1.)     My NCL script is not finding the experimental variables in the
    >>>> grib file. These variables are given the generic names varXXX where XXX
    >>>> can
    >>>> be between 88 and 111, for example.
    >>>>
    >>>> 2.)     The script also doesn’t find the 3D records. I’m not sure how
    >>>> NCL
    >>>> finds these records in the grib file and then lists them as a 3D
    >>>> (time,lev,lat,lon), not counting the time.
    >>>>
    >>>> 3.)     If 1 and 2 can be resolved, how can I then convert the reduced
    >>>> Gaussian grid to regular lat lon? Would ESMF work if I setup a regular
    >>>> lat
    >>>> lon grid?
    >>>>
    >>>>
    >>>>
    >>>> Appreciate your thoughts on this.
    >>>>
    >>>> /M
    >>>>
    >>>>
    >>>>
    >>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    >>>>
    >>>> Marston S. Johnston, PhD
    >>>>
    >>>> Department of Earth Sciences
    >>>>
    >>>> University of Gothenburg, Sweden
    >>>>
    >>>> Email: marston.johnston at gu.se
    >>>>
    >>>> SkypeID: marston.johnston
    >>>>
    >>>> Phone: +46-31-7864901
    >>>>
    >>>> Only the fruitful thing is true!
    >>>>
    >>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    >>>>
    >>>>
    >>>>
    >>>>
    >>>>
    >>>> _______________________________________________
    >>>> ncl-talk mailing list
    >>>> ncl-talk at ucar.edu
    >>>> List instructions, subscriber options, unsubscribe:
    >>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
    >>>>
    >>>> --
    >>>> Dipl. Geophys. Karin Meier-Fleischer
    >>>> Visualization, NCL, CDO
    >>>> Application Support
    >>>>
    >>>> Deutsches Klimarechenzentrum GmbH (DKRZ)
    >>>> Bundesstrasse 45a - D20146 Hamburg - Germany
    >>>>
    >>>> Phone:    +49 (0)40 460094 126
    >>>> Fax:      +49 (0)40 460094 270
    >>>> E-Mail:   meier-fleischer at dkrz.de
    >>>> URL:      www.dkrz.de
    >>>>
    >>>> Geschäftsführer: Prof. Dr. Thomas Ludwig
    >>>> Sitz der Gesellschaft: Hamburg
    >>>> Amtsgericht Hamburg HRB 39784
    >>>>
    >>>>
    >>>> _______________________________________________
    >>>> ncl-talk mailing list
    >>>> ncl-talk at ucar.edu
    >>>> List instructions, subscriber options, unsubscribe:
    >>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
    >>>>
    >>>
    >>>
    >>> --
    >>> Benjamin L. Lamptey, PhD (Physicist, Meteorologist, Geoscientist, HPC &
    >>> Info
    >>> Systems)
    >>> Acting Director-General
    >>> African Centre of Meteorological Applications for Development (ACMAD)
    >>> 55, Avenue des Ministères, PL6
    >>> BP 13184, 1er Arrondissement, Niamey Plateau, Niger,
    >>> Tel.(+227) 20 73.49.92
    >>> Fax: (+227) 20723627
    >>> www.acmad.org
    >>>
    >>> Cell: +22796645655; +233265121750
    >>> Email: b.lamptey at acmad.org; bllamptey at gmail.com
    >>> Skype: benlamptey
    >>>
    >>>
    >>> _______________________________________________
    >>> ncl-talk mailing list
    >>> ncl-talk at ucar.edu
    >>> List instructions, subscriber options, unsubscribe:
    >>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
    >>>
    >
    _______________________________________________
    ncl-talk mailing list
    ncl-talk at ucar.edu
    List instructions, subscriber options, unsubscribe:
    http://mailman.ucar.edu/mailman/listinfo/ncl-talk
    




More information about the ncl-talk mailing list