[ncl-talk] NCL to replace CDO
Karin Meier-Fleischer
meier-fleischer at dkrz.de
Tue Jul 4 04:02:34 MDT 2017
Hi Marston,
the CDOs won't be dead! :-D
CDO is still an active project and the developers are working on it all
the day (already 3 releases this year):
https://code.mpimet.mpg.de/projects/cdo/files
They are updating the GRIB tables if they were asked to do it - it is
impossible to recognize all changes. Feel free to ask them via the
support/issue web page to update a table.
The mixture of GRIB 1 and GRIB 2 records in one file is a problem, but
as you already said, the grib_set program of the grib-api can correct it.
Bye,
Karin
Am 04.07.17 um 11:16 schrieb Marston Johnston:
> Hi guys,
>
> Apologies for my delayed response. It’s been a very busy time for me with almost no end in sight.
>
> Dave:
>
> Thanks for looking into the problem for me. My thought was to use NCL in the entire conversion process from grib to netcdf. Your solution does not offer me possibility in a manner where I can
> Semi-automate the process. The experimental grib variables are important for my model group because it is how we output diagnostics and data for modules we have added to the model.
> These variables were only in the ICMGGglx2+200701 and ICMGGECE3+199001. I threw in the spectral grib file to show you all types of grib files my mode produces. The best way that I currently can convert these grib files to netcdf, convert from reduced Gaussian to regular lat lon, and get the experimental variables is using the following bash function, where ${i} is the above file names. Gribset comes from the grib_api library.
>
> GRIBGG () {
> i=${1}
> echo "Processing file = $i"
> grib_set -s editionNumber=1 $i ${i}.grb
> cdo -R copy ${i}.grb ${i}_R.grb
> cdo -v splitzaxis ${i}_R.grb ${i}_R
> # no need to split axis here as all are 3D variables
> grib_to_netcdf -k 2 -D NC_FLOAT -o ${i}_3d.nc ${i}_R02.grb
> grib_to_netcdf -k 2 -D NC_FLOAT -o ${i}_2d.nc ${i}_R01.grb
> rm -vf ${i}.grb ${i}_R.grb ${i}_R02.grb ${i}_R01.grb
> # merge the 3d and 2d variables
> cdo -v -O merge ${i}_2d.nc ${i}_3d.nc ${i}.nc
> rm -vf ${i}_2d.nc ${i}_3d.nc
> return 0
> }
>
> Karin is right, there OpenIFS the newer version of IFS, does mix grib 1 and grib 2 variables, but even when this issue is addressed with grib_set -s editionNumber=1, still need CDO to split the grib file in order to convert the reduced grid to regular lat lon. Further, CDO 1.8.2 miss-translates the variables when converting to netcdf. Therefore I need at the moment a mixture of grib_api and cdo to convert these grib files. The issue I have finally is that I cannot control and examine the data during this process, especially when I want to regrid the data. This is why I asked if NCL could be used for this task.
>
> What is the grib engine that NCL uses under the hood? Perhaps my function above can give a possible future upgrade to NCL to handle this type of grib files?
>
>
> Karin:
>
> Thanks for looking into the CDO 1.8.2 issue. Is this project now dead? They use to update the grib tables in CDO but now this project seems to be at a stand still.
>
>
> Best,
> /M
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Marston S. Johnston, PhD
> Department of Earth Sciences
> University of Gothenburg, Sweden
> Email: marston.johnston at gu.se
> SkypeID: marston.johnston
> Phone: +46-31-7864901
> Only the fruitful thing is true!
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>
> On 2017-07-03, 18:38, "David Brown" <ncl-talk-bounces at ucar.edu on behalf of dbrown at ucar.edu> wrote:
>
> I noticed that wgrib said there was a GRIB2 record in one of the
> files. Oddly, NCL did not seem to notice this and I just filed it away
> as a mystery for now. Ordinarily NCL also fails when GRIB1 and GRIB2
> records are mixed in a single file. So it seems that further
> investigation is warranted.
> -dave
>
> On Mon, Jul 3, 2017 at 9:57 AM, Karin Meier-Fleischer
> <meier-fleischer at dkrz.de> wrote:
> > Hi Marston and Dave,
> >
> > I've tried to remap and convert your files with CDO version 1.8.2 but it
> > fails for two of the three files. The file ICMGGECE3+199001 is a GRIB 1 file
> > and the others seem to have GRIB 1 and GRIB 2 mixed records which can't be
> > handled by CDO.
> >
> > I tried to convert the files to netCDF using the ECMWF grib-api/eccodes GRIB
> > tools but those do not like reduced Gaussian grids.
> >
> > Sorry, can't help yet but maybe we'll find a way to do it with CDO. ;-)
> >
> > Bye,
> > Karin
> >
> > Am 30.06.17 um 18:11 schrieb David Brown:
> >
> >> Ben,
> >> I think Marston cannot use ncl_convert2nc because he wants to
> >> interpolate to a lat/lon grid before saving the data to netcdf.
> >>
> >> Karin,
> >> The files that Marston made available to me are still on
> >> ftp.cgd.ucar.edu in the zip file ifs_grib_types.zip.
> >> -dave
> >>
> >> On Fri, Jun 30, 2017 at 4:29 AM, Benjamin Lamptey <bllamptey at gmail.com>
> >> wrote:
> >>>
> >>> Hi Marston,
> >>> Maybe I am missing something. Why not use "ncl_convert2nc" command
> >>> instead
> >>> of an NCL script?
> >>>
> >>> Even if you have many files, you can use the command in a shell script.
> >>>
> >>> Sorry if I missed something earlier but the above command is easier to
> >>> convert grib to netcdf.
> >>>
> >>> Regards
> >>> Ben
> >>>
> >>> On Fri, Jun 30, 2017 at 11:10 AM, Karin Meier-Fleischer
> >>> <meier-fleischer at dkrz.de> wrote:
> >>>>
> >>>> Hi Marston,
> >>>>
> >>>> as I'm also working in the CDO consultancy here in Hamburg, I would like
> >>>> to have a closer look at your IFS file, too, to see if the CDO version
> >>>> 1.8.2
> >>>> could handle it.
> >>>>
> >>>> If the file is not too large you can attach it to an 'offline' email to
> >>>> me
> >>>> or upload it creating a new request on
> >>>> https://code.mpimet.mpg.de/projects/cdo/boards/1 .
> >>>>
> >>>> Thanks in advance,
> >>>> Karin
> >>>>
> >>>> Am 27.06.17 um 20:49 schrieb Marston Johnston:
> >>>>
> >>>> Hi,
> >>>>
> >>>>
> >>>>
> >>>> For years I’ve been using CDO to convert ECMWF’s IFS grib files to
> >>>> netcdf
> >>>> and converting it from reduced Gaussian grid to regular lat lon.
> >>>>
> >>>> But CDO has not kept up the pace with updates and they do not respond to
> >>>> many questions.
> >>>>
> >>>> Therefore, I’m attempting to move this part of my processing chain to
> >>>> NCL.
> >>>>
> >>>> I’m using NCL 6.5 (beta) to read the grib file (see attached log file).
> >>>> There 1 or 2 problems and then I have a question.
> >>>>
> >>>>
> >>>>
> >>>> 1.) My NCL script is not finding the experimental variables in the
> >>>> grib file. These variables are given the generic names varXXX where XXX
> >>>> can
> >>>> be between 88 and 111, for example.
> >>>>
> >>>> 2.) The script also doesn’t find the 3D records. I’m not sure how
> >>>> NCL
> >>>> finds these records in the grib file and then lists them as a 3D
> >>>> (time,lev,lat,lon), not counting the time.
> >>>>
> >>>> 3.) If 1 and 2 can be resolved, how can I then convert the reduced
> >>>> Gaussian grid to regular lat lon? Would ESMF work if I setup a regular
> >>>> lat
> >>>> lon grid?
> >>>>
> >>>>
> >>>>
> >>>> Appreciate your thoughts on this.
> >>>>
> >>>> /M
> >>>>
> >>>>
> >>>>
> >>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >>>>
> >>>> Marston S. Johnston, PhD
> >>>>
> >>>> Department of Earth Sciences
> >>>>
> >>>> University of Gothenburg, Sweden
> >>>>
> >>>> Email: marston.johnston at gu.se
> >>>>
> >>>> SkypeID: marston.johnston
> >>>>
> >>>> Phone: +46-31-7864901
> >>>>
> >>>> Only the fruitful thing is true!
> >>>>
> >>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> ncl-talk mailing list
> >>>> ncl-talk at ucar.edu
> >>>> List instructions, subscriber options, unsubscribe:
> >>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
> >>>>
> >>>> --
> >>>> Dipl. Geophys. Karin Meier-Fleischer
> >>>> Visualization, NCL, CDO
> >>>> Application Support
> >>>>
> >>>> Deutsches Klimarechenzentrum GmbH (DKRZ)
> >>>> Bundesstrasse 45a - D20146 Hamburg - Germany
> >>>>
> >>>> Phone: +49 (0)40 460094 126
> >>>> Fax: +49 (0)40 460094 270
> >>>> E-Mail: meier-fleischer at dkrz.de
> >>>> URL: www.dkrz.de
> >>>>
> >>>> Geschäftsführer: Prof. Dr. Thomas Ludwig
> >>>> Sitz der Gesellschaft: Hamburg
> >>>> Amtsgericht Hamburg HRB 39784
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> ncl-talk mailing list
> >>>> ncl-talk at ucar.edu
> >>>> List instructions, subscriber options, unsubscribe:
> >>>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
> >>>>
> >>>
> >>>
> >>> --
> >>> Benjamin L. Lamptey, PhD (Physicist, Meteorologist, Geoscientist, HPC &
> >>> Info
> >>> Systems)
> >>> Acting Director-General
> >>> African Centre of Meteorological Applications for Development (ACMAD)
> >>> 55, Avenue des Ministères, PL6
> >>> BP 13184, 1er Arrondissement, Niamey Plateau, Niger,
> >>> Tel.(+227) 20 73.49.92
> >>> Fax: (+227) 20723627
> >>> www.acmad.org
> >>>
> >>> Cell: +22796645655; +233265121750
> >>> Email: b.lamptey at acmad.org; bllamptey at gmail.com
> >>> Skype: benlamptey
> >>>
> >>>
> >>> _______________________________________________
> >>> ncl-talk mailing list
> >>> ncl-talk at ucar.edu
> >>> List instructions, subscriber options, unsubscribe:
> >>> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
> >>>
> >
> _______________________________________________
> ncl-talk mailing list
> ncl-talk at ucar.edu
> List instructions, subscriber options, unsubscribe:
> http://mailman.ucar.edu/mailman/listinfo/ncl-talk
>
>
>
More information about the ncl-talk
mailing list