[Go-essp-tech] Global attributes and DRS extensions for downscaled datasets
Mattmann, Chris A (388J)
chris.a.mattmann at jpl.nasa.gov
Fri Mar 29 19:08:57 MDT 2013
Hi Guys,
This is great to hear that Bruce and Chris and you guys have a new project.
Looking forward to collaborating and to its outcomes.
Thanks!
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann at nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: "martin.juckes at stfc.ac.uk" <martin.juckes at stfc.ac.uk>
Date: Wednesday, March 27, 2013 2:23 AM
To: "taylor13 at llnl.gov" <taylor13 at llnl.gov>, "cjack at csag.uct.ac.za"
<cjack at csag.uct.ac.za>
Cc: "ncpp_tech at list.woc.noaa.gov" <ncpp_tech at list.woc.noaa.gov>,
"denis.nadeau at nasa.gov" <denis.nadeau at nasa.gov>, "go-essp-tech at ucar.edu"
<go-essp-tech at ucar.edu>, "gerald.potter at nasa.gov"
<gerald.potter at nasa.gov>, "obc at dmi.dk" <obc at dmi.dk>,
"ncpp_core at list.woc.noaa.gov" <ncpp_core at list.woc.noaa.gov>,
"galina at ucar.edu" <galina at ucar.edu>, "laura.e.carriere at nasa.gov"
<laura.e.carriere at nasa.gov>, "colin.jones at smhi.se" <colin.jones at smhi.se>
Subject: Re: [Go-essp-tech] Global attributes and DRS extensions
for downscaled datasets
>Hello Karl,
>
>Thanks for clear response I probably should have been able to work that
>out if I had followed the email thread carefully.
>
>
>There is a separate document for CORDEX which describes the intended
>mapping of attributes onto facets in the ESGF User Interface I’ll try
>to send that to
> you later.
>
>I understand your reservations about some aspects of the CORDEX data
>requirements, but the aim was to use terms which are in usage in the
>community (particularly
> for the “region” attribute which actually combines region and
>resolution) and thus, hopefully, improve compliance and acceptance of the
>standard.
>
>There is an aspiration to include statistically downscaled data in the
>CORDEX archive (and it is another shortcoming of the CORDEX document you
>refer to, I
> think, that it only deals with dynamically downscaled data and does not
>leave a hook to allow extension to statistically downscaled). The system
>you’ve described could presumably be used for statistically downscaled
>data in CORDEX. We have a new European Union
> project starting next week which funds Bruce Hewitson’s group in Cape
>Town to do some coordination and networking on data standards for CORDEX
>downscaling, so I’ve copied in Chris Jack who is, I think, leading their
>effort.
>
>
>Regards,
>Martin
>From:
> Karl Taylor [mailto:taylor13 at llnl.gov]
>Sent: 27 March 2013 00:51
>To: Juckes, Martin (STFC,RAL,RALSP)
>Cc: galina at ucar.edu; obc at dmi.dk; colin.jones at smhi.se;
>ncpp_core at list.woc.noaa.gov; ncpp_tech at list.woc.noaa.gov;
>go-essp-tech at ucar.edu; laura.e.carriere at nasa.gov; gerald.potter at nasa.gov;
>williams13 at llnl.gov; denis.nadeau at nasa.gov; Pascoe, Stephen
>(STFC,RAL,RALSP)
>Subject: Re: Global attributes and DRS extensions for downscaled datasets
>
>
>
>Dear Martin,
>
>I'm not advocating changing the CORDEX requirements; it's probably much
>too late for that. There are are limitations to the generality of the
>CORDEX specifications, which means they might not be applicable to
>downscaling efforts outside of CORDEX. The document
> I prepared was to try to address the more general issue of what
>descriptors are needed for downscaled datasets.
>
>
>I have proposed that a single additional "descriptor" be added to the
>already defined components of the DRS:
>
>Source of predictor data ⇒ driving_model_id - driving_model_rip (e.g.
>“GFDL-CM3-r1i1p1”) In some cases the driving_model_rip might be omitted
>(e.g., when using reanalysis output to drive the downscaling).
>
>
>In CORDEX this descriptor could be formed by joining with a hyphen your
>GCMModelName and CMIP5EnsembleMember.
>
>I have also proposed expanding the "ensemble member" descriptor to
>include an indication of the "nominal resolution". The idea here is that
>output might need to be regridded or be made available at various
>resolutions, so we would like to be able to distinguish
> among these closely related datasets. Here is the description of the
>'riph' designator:
>
> Ensemble member⇒ ‘riph’ designator, where the “rip” form is defined as
>in CMIP5 (which for downscaled data would usually be “r1i1p1”), and the
>“h” is followed by nominal resolution expressed in kilometers.
> (For backward compatibility the DRS would consider the “h” segment as
>optional, but it is required for downscaled datasets.) The last part of
>the 'riph' designator is of the form “hnXXXX” or “hiXXXX” where XXXX is
>the nominal horizontal resolution of the
> downscaled data, expressed in kilometers (rounded to the nearest km with
>leading zeros dropped). “hn” indicates that the data is stored on the
>model’s “native” grid, while “hi” indicates that the data has been
>interpolated from a model’s native grid to a
> different grid. (Statistically downscaled data would normally be
>recorded on a so-called “native” grid.) Data on a native grid at a
>nominal resolution of 5 km, for example, would be identified as “hn5”,
>while regridded data at 11 km resolution would be identified
> as “hi11”. The XXXX should be calculated as follows: XXXX = sqrt(domain
>area / (number of grid cells)), expressed in km/grid cell and rounded off
>to the nearest km.
>
>
>CORDEX has chosen to include resolution information as part of a domain
>name (e.g., CAM-44 or SAM-44i), but the resolution doesn't seem to me to
>belong as part of the region identification.
>
>I should note also that CORDEX specifies a directory structure and/or
>filenames where in the CORDEX document some of the DRS categories are
>renamed. I've attached a table that shows the DRS elements and
>corresponding CORDEX identifiers, along with global attributes.
> (I'm going to try to get NASA / NOAA to be consistent with the DRS.) I
>also provide a table of additional global attributes. CORDEX is mostly
>consistent with this table, except for using "CORDEX_domain and omitting
>driving_model_tracking_ids.
>
>
>Finally, I note that in the example found in the CORDEX document for
>global attributes:
>
>1) experiment_id = "evaluation", but in the directory structure and
>filename templates, this is presumably used as "CMIP5ExperimentName",
>but of course "evaluation" is not a CMIP5 experiment. I think a better
>term for "CMIP5ExperimentName" is simply "experiment",
> which in the case of CORDEX is usually the same as the CMIP5
>experiment_id.
>
>2) CORDEX requires "contact", but this was left out of the example.
>
>Please let me know what you think.
>
>Best regards,
>Karl
>On 3/26/13 4:58 AM,
>martin.juckes at stfc.ac.uk <mailto:martin.juckes at stfc.ac.uk> wrote:
>
>
>Hello Karl,
>
>I’m puzzled about how this fits in with CORDEX. We went through this
>discussion some time ago, and agreed on some data requirements
> in the document you cite below which we believed to be appropriately
>consistent with the CMIP5 requirements. This document was then discussed
>at a WCRP meeting and has been circulated as the requirements for groups
>submitting CORDEX data to ESGF. Since then,
> modelling groups have been preparing data and we are expecting to start
>publication soon. Do you think there are problems with uniformity in the
>way the CORDEX requirements are specified?
>
>Regards,
>Martin
>
>
>
>From:
> Karl Taylor [mailto:taylor13 at llnl.gov]
>Sent: 25 March 2013 21:50
>To: Galia Guentchev
>Cc: ncpp_core at list.woc.noaa.gov; NCPP TECHNICAL TEAM;
>go-essp-tech at ucar.edu;
>laura.e.carriere at nasa.gov <mailto:laura.e.carriere at nasa.gov>; Potter,
>Gerald Lee. (GSFC-606.2)[UNIVERSITY OF MARYLAND]; Dean Williams; Nadeau,
>Denis (GSFC-610.1)[R S INFORMATION SYSTEMS, ]; Juckes, Martin
>(STFC,RAL,RALSP); Pascoe, Stephen (STFC,RAL,RALSP)
>Subject: Global attributes and DRS extensions for downscaled datasets
>
>
>
>Dear all,
>I have spent considerable time reviewing the following four documents:
>A. The email (copied below) sent by Galia and Aparna, which proposed
>attributes, filenames, and directory structures for downscaled data.
>B.
>http://cmip-pcmdi.llnl.gov/cmip5/docs/cmip5_data_reference_syntax.pdf
><http://cmip-pcmdi.llnl.gov/cmip5/docs/cmip5_data_reference_syntax.pdf>
>which describes the corresponding CMIP5 metadata.
>C.
>http://cordex.dmi.dk/joomla/images/CORDEX/cordex_archive_specifications.pd
>f
><http://cordex.dmi.dk/joomla/images/CORDEX/cordex_archive_specifications_1
>21022.pdf> which describes the corresponding CORDEX metadata.
>D.
>http://cmip-pcmdi.llnl.gov/cmip5/docs/CMIP5_output_metadata_requirements.p
>df
><http://cmip-pcmdi.llnl.gov/cmip5/docs/CMIP5_output_metadata_requirements.
>pdf> which specifies all the CMIP5 metadata requirements.
>I hope that document A above could be made compatible with the others and
>in general could provide a sound basis for establishing more uniformity
>moving forward. Toward that end, I have prepared the attached document
>describing for downscaled data a minimal
> set of global attributes needed to augment those used in CMIP5 and also
>the extensions needed to the DRS document to accommodate downscaled data.
>I hope at least a few of you will take the time to study this document
>and provide feedback.
>
>Best regards,
>Karl
>
>Mail sent by Galia Guentchev 3/12/13
>
>**********************************************************************
>Details of each element of the proposed directory structure:
>
>Proposed elements -
>/projectID/sub-project/product/institution/predictorModel/experimentID/fre
>quency/realm/MIPtable/Pred
>ictor_experiment_rip/predictorversion/downscalingMethod/predictand
>(variableName)/region/DownscaledDataversion/file_name.nc
>
>Example:
>
>/ncpp2013/perfectModel/downscaled/NOAA-GFDL/GFDL-HIRAM-C360-coarsened/amip
>/day/atmos/day/r1i1p1/v20121024/GFDL-ARRMv1/tasmax/US48/v20120227/tasmax_d
>ay_amip_r1i1p1_downscaled_US48_GFDLARRMv1_19790101-19831231.nc
>
>The new element sub-project (in blue above) gives the opportunity to
>indicate to users that in the one case the method was trained on
>observations (standard setting), and in the other on model that was
>considered to be the
> truth (perfect model setting);
>The options there could be: PerfectModel or Standard - where possibly
>there could be a different name instead of 'standard' for the standard
>downscaling setting.
>For NASA datasets some of the directories could be:
>
>project = NEX
>product = downscaled
>institution = NASA-Ames
>predictorModel - original model value
>experimentID = historical
>frequency = mon
>realm = atmos
>Predictor_experiment_rip - original model value
>variable = precipitation or temperature
>region = CONUS
>
>DownscalingMethod will also be included as a directory to allow for
>search on method.
>
>**********************
>There are a set of sub-directories that refer to the PredictorModel -
>presented in bold -
>/predictorModel/experimentID/frequency/realm/MIPtable/Pred
>ictor_experiment_rip/predictorversion
>
>Where:
>·
>predictor model - is the specific GCM which is the source of the
>predictor data set - GFDL-HIRAM-C360-coarsened - in the above example
>·
>experimentID - the specific experiment - amip in this case
>·
>frequency - refers to the temporal scale of the predictor fields - daily
>·
>realm - the realm of the predictors - in this case atmos(phere)
>·
>MIPtable - name of the model intercomparison table - daily in this
>example, could be amon - for atm monthly data;
>·
>Predictor-Experiment-rip - follows the standard notation from CMIP5
>·
>version - the version date of the global model that provided the
>predictor dataset
>The elements above follow quite closely the structure for CMIP5 model
>output directory elements.
>There is a set of sub-directories that refer to the Downscaling method -
>presented in italics -
>
>downscalingMethod/predictand (variableName)/region/DownscaledDataversion
>
>Where:
>·
>downscalingMethod - is the downscaling method abbreviation - in this case
>GFDL-ARRMv1 - the GFDL in the name indicates that this is a setting
>applied by GFDL where there were two sets of predictors, based on the
>ARRM method of
> K.Hayhoe; also v.1 indicates which version of the ARRM method was used
>(the original version) - more details about the method are given in the
>global attributes of the file;
>·
>Predictand (variableName) - the specific predictand variable that was
>downscaled; tasmax in this case;
>·
>region - indicates that the method was applied to the US48
>·
>DownscaledDataversion - the version of the downscaled dataset
>For the purposes of standardization there are two directions to consider:
>1) One is to have one standard directory structure that will be used by
>all - for example, following the example of GFDL to have the details of
>the predictor model first and then the downscaling method details:
>·
>ProjectID - sub-project - product - Institution - Predictor dataset
>details - Downscaling method details - Filename
>Having a standardized approach would help any automated service/web
>service to detect the directory path for a particular dataset.
>
>2) During our last teleconference there was a proposal to follow the
>downscaling practice and describe the downscaling method first and then
>the predictor model. This leads to
>two paths:
> ? ProjectID - Standard or Perfect Model sub-project facet-
>product - Institution - then see below:
> - (if Perfect model setting) Predictor dataset details -
>Downscaling method details,
>
> - (if Standard setting) - Downscaling method details -
>Predictor dataset details
>
>The NCPP Core team accepts that it may be reasonable to have a directory
>structure - where the method description is first; and another directory
>structure - where the predictor description is first and then the methods
>that are applied are described;
>NCPP will support either approach (one overall directory structure, or
>two separate pathways) and if the second approach is chosen (with two
>different sub-directory sequences) - we would like to promote and to
>support the standardization of these different
> directory pathways - meaning - we will support two standardized
>directory structures to accommodate two common practices.
>
>******************
>Additional details:
>
>Variable level attributes-
>The published dataset should also conform to CF-standards.
>eg-
>
> tasmax:long_name = "Downscaled Daily Maximum Near-Surface
>Air Temperature" ;
>
> tasmax:units = "K" ;
>
>
> tasmax:missing_value = 1.e+20f ;
>
>
> tasmax:_FillValue = 1.e+20f ;
>
>
> tasmax:standard_name = "air_temperature" ;
>
>
> tasmax:original_units = "K" ;
>
>
> tasmax:downscaling_method: GFDL-ARRMv1
>
>Global attributes- listing a few here, several CMIP-style attributes will
>be inherited.
>
>
>"predictorModel" will replace "model_id"
> For the 'downscaling model', as agreed with Luca on the call it would
>be 'downscalingMethod'
>
>
> :Conventions = "CF-1.4" ;
> :references = "info about model, training datasets etc
>will be provided here"
> :info = "additional info about the downscaling method"
> :creation_date = "2011-08-19T21:57:06Z" ;
> :institution = "NOAA GFDL(201 Forrestal Rd, Princeton,
>NJ, 08540)" ;
> :history = "info on file processing. Eg" processed by
>toolX." ;
> :projectID = ncpp2013
> :subprojectID = perfectModel
> :product = downscaled
> :institution = NOAA-GFDL
> :predictorModel = GFDL-HIRAM-C360-coarsened
> :experimentID = amip
> :frequency = day
> :modeling_realm = atmos
> :Predictor_experiment_rip = r1i1p1
> :region = US48
> :table_id = day
> :version = v20120227
> :downscalingMethod = GFDL-ARRMv1
>**************************************************
>
>Best regards,
>Galia and Aparna
>
>
>
>
>-- Galia Guentchev, PhDProject ScientistNational CLimatePredictions and
>Projections Platform (NCPP)NCAR RAL CSAPFL2 31033450 Mitchell
>LaneBoulder, CO, 80301phone: 303 497 2743
>
>--
>Scanned by iCritical.
>
>
>
>
>
More information about the GO-ESSP-TECH
mailing list