[Go-essp-tech] NetCDF metadata in THREDDS catalogues
Bob Drach
drach at llnl.gov
Thu Nov 5 11:53:59 MST 2009
Hi Bryan,
Bryan Lawrence wrote:
> Hi Bob
>
> thanks for this, so this means the gateway can only know this stuff and no more.
>
> There is a considerable amount of "cmip5" metadata prescribed in the global
> file attributes. Is that propagated to the gateway?
>
It depends on how the publisher is configured. The global fields to be
published can be defined as categories in the publisher configuration,
extracted and included as dataset properties. Also there will be an
IPCC5Handler (Python class) that understands about CMOR tables, DRS, and
other CMIP5-specific conventions. Handlers of this class will know how
to extract and publish CMIP5-specific metadata.
> Where then is the tracking id that is in each file? What service would expose that?
> I have understood that a user could always go from the tracking identifier
> in the file to find the appropriate metadata (e.g. which version it is).
>
Each file generated by CMOR will have a global attribute 'tracking_id'
that is unique for each file. Tracking IDs will be exposed in the TDS
catalogs. This is being implemented now, so will appear in the next
major release of the publisher. Presumably the gateway will expose these
IDs, but just how is not yet clear.
>
>> The database does contain all the metadata in the files, with the
>> possible exception of coordinate data if you consider that as metadata.
>> However coordinate ranges for aggregate variables are kept in the
>> variable table. The postgres database would be the logical place to
>> obtain the desired information.
>>
>
> Will it be possible to access the postgres database on a data node from
> outside the data node (I would assume not following our earlier conversation
> about what is exposed, but I want to be sure :-)
>
By default the postgres database is configured for localhost queries
only. But it can be configured to allow external queries on a per-IP
basis. It's up to you.
Bob
> (I guess the gateway can also query the services which expose more infomation
> about the data, but to do that, the data node would actually have to read
> the data ...)
>
> Bryan
>
>
>> Some of the table definitions will change in the next version, but I
>> would expect the tables containing attribute and dimension information
>> to remain stable.
>>
>> Bob
>>
>>> For instance:
>>>
>>> esgcet=> select name, value from file_var_attr limit 20;
>>> name |
>>> value
>>> ---------------+------------------------------------------------------------------
>>> axis | Y
>>> bounds | lat_bnds
>>> getitem | <built-in method getitem of CdunifVariable object at
>>> 0x19f8150>
>>> getslice | <built-in method getslice of CdunifVariable object at
>>> 0x19f8150>
>>> long_name | latitude
>>> setitem | <built-in method setitem of CdunifVariable object at
>>> 0x19f8150>
>>> setslice | <built-in method setslice of CdunifVariable object at
>>> 0x19f8150>
>>> standard_name | latitude
>>> units | degrees_north
>>> getitem | <built-in method getitem of CdunifVariable object at
>>> 0x19f8030>
>>> getslice | <built-in method getslice of CdunifVariable object at
>>> 0x19f8030>
>>> setitem | <built-in method setitem of CdunifVariable object at
>>> 0x19f8030>
>>> setslice | <built-in method setslice of CdunifVariable object at
>>> 0x19f8030>
>>> axis | X
>>> bounds | lon_bnds
>>> getitem | <built-in method getitem of CdunifVariable object at
>>> 0x19f8108>
>>> getslice | <built-in method getslice of CdunifVariable object at
>>> 0x19f8108>
>>> long_name | longitude
>>> setitem | <built-in method setitem of CdunifVariable object at
>>> 0x19f8108>
>>> setslice | <built-in method setslice of CdunifVariable object at
>>> 0x19f8108>
>>> (20 rows)
>>>
>>> ---
>>> Stephen Pascoe +44 (0)1235 445980
>>> British Atmospheric Data Centre
>>> Rutherford Appleton Laboratory
>>>
>>>
>>> --
>>> Scanned by iCritical.
>>>
>>>
>>> ------------------------------------------------------------------------
>>>
>>> <?xml version='1.0' encoding='UTF-8'?>
>>> <catalog xmlns:xsi="http://*www.*w3.org/2001/XMLSchema-instance" xmlns:xlink="http://*www.*w3.org/1999/xlink" xmlns="http://*www.*unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" name="TDS configuration file" xsi:schemaLocation="http://*www.*unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0 http://*www.*unidata.ucar.edu/schemas/thredds/InvCatalog.1.0.2.xsd">
>>> <service serviceType="OpenDAP" base="/thredds/dodsC/" name="gridded"/>
>>> <service serviceType="HTTPServer" base="/thredds/fileServer/" name="HTTPServer"/>
>>> <service serviceType="SRM" base="srm://host.sample.gov:6288/srm/v2/server?SFN=/archive.sample.gov/" name="HRMatPCMDI"/>
>>> <dataset restrictAccess="esg-user" ID="cmip3" name="cmip3">
>>> <property name="project" value="test"/>
>>> <documentation type="summary"/>
>>> <metadata inherited="true">
>>> <variables vocabulary="CF-1.0">
>>> <variable name="clivi" vocabulary_name="atmosphere_cloud_ice_content" units="kg m-2">Column Integerated Cloud Ice Content</variable>
>>> <variable name="clwvi" vocabulary_name="atmosphere_cloud_condensed_water_content" units="kg m-2">Column Integerated Cloud Water Content</variable>
>>> <variable name="mrsos" vocabulary_name="moisture_content_of_soil_layer" units="kg m-2">Moisture in Upper 0.1 m of Soil Column</variable>
>>> <variable name="prsn" vocabulary_name="snowfall_flux" units="kg m-2 s-1">Snowfall Flux</variable>
>>> <variable name="rlut" vocabulary_name="toa_outgoing_longwave_flux" units="W m-2">Outgoing Longwave Radiation</variable>
>>> <variable name="rsftoai_co2" vocabulary_name="toa_instantaneous_shortwave_forcing" units="W m-2">Instantaneous Top of the Atmosphere Shortwave Forcing Due to Anthropogenic Carbon Dioxide</variable>
>>> <variable name="rsftoaics_co2" vocabulary_name="Instantaneous Top of the Atmosphere Shortwave Clear-Sky Forcing Due to Anthropogenic Carbon Dioxide" units="W m-2">Instantaneous Top of the Atmosphere Shortwave Clear-Sky Forcing Due to Anthropogenic Carbon Dioxide</variable>
>>> <variable name="rsftropacs_co2" vocabulary_name="Adjusted Tropopause Shortwave Clear-Sky Forcing Due to Anthropogenic Carbon Dioxide" units="W m-2">Adjusted Tropopause Shortwave Clear-Sky Forcing Due to Anthropogenic Carbon Dioxide</variable>
>>> <variable name="rsntp" vocabulary_name="net_downward_shortwave_flux_in_air" units="W m-2">Net Downward Shortwave Flux at 200 hPa</variable>
>>> <variable name="rsuscs" vocabulary_name="surface_upwelling_shortwave_flux_in_air_assuming_clear_sky" units="W m-2">Reflected Surface Clear-Sky Shortwave</variable>
>>> <variable name="rsut" vocabulary_name="toa_outgoing_shortwave_flux" units="W m-2">TOA Reflected Shortwave Radiation</variable>
>>> <variable name="tauv" vocabulary_name="surface_downward_northward_stress" units="Pa">Meridional Surface Wind Stress</variable>
>>> <variable name="ts" vocabulary_name="surface_temperature" units="K">Surface Skin Temperature</variable>
>>> <variable name="uas" vocabulary_name="eastward_wind" units="m s-1">Zonal Surface Wind Speed</variable>
>>> <variable name="va" vocabulary_name="northward_wind" units="m s-1">Meridional Wind Component</variable>
>>> <variable name="sic" vocabulary_name="sea_ice_area_fraction" units="%">Sea Ice Concentration</variable>
>>> <variable name="orog" vocabulary_name="surface_altitude" units="m">Surface Altitude</variable>
>>> <variable name="sftlf" vocabulary_name="land_area_fraction" units="%">Land Area Fraction</variable>
>>> <variable name="mrro" vocabulary_name="runoff_flux" units="kg m-2 s-1">Surface and Subsurface Runoff</variable>
>>> <variable name="mrros" vocabulary_name="surface_runoff_flux" units="kg m-2 s-1">Surface Runoff</variable>
>>> <variable name="mrso" vocabulary_name="soil_moisture_content" units="kg m-2">Total Soil Moisture Content</variable>
>>> <variable name="snm" vocabulary_name="surface_snow_melt_flux_where_land" units="kg m-2 s-1">Snow Melt</variable>
>>> <variable name="snw" vocabulary_name="surface_snow_amount_where_land" units="kg m-2">Snow Amount</variable>
>>> <variable name="zobt" vocabulary_name="sea_floor_depth_below_geoid" units="m">Sea Floor Depth</variable>
>>> <variable name="tos" vocabulary_name="sea_surface_temperature" units="K">Sea Surface Temperature</variable>
>>> <variable name="usi" vocabulary_name="eastward_sea_ice_velocity" units="m s-1">Eastward Sea Ice Velocity</variable>
>>> <variable name="vsi" vocabulary_name="northward_sea_ice_velocity" units="m s-1">Northward Sea Ice Velocity</variable>
>>> </variables>
>>> </metadata>
>>> <metadata inherited="true">
>>> <dataType>Grid</dataType>
>>> <dataFormat>NetCDF</dataFormat>
>>> </metadata>
>>> <dataset ID="cmip3.files" name="cmip3.files">
>>> <dataset name="clivi_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.clivi_A1_2006_Dec_to_2076_Dec.nc">
>>> <dataSize units="bytes">93685040</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/atm/mo/clivi/ukmo_hadgem1/run1/clivi_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="clwvi_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.clwvi_A1_2006_Dec_to_2076_Dec.nc">
>>> <dataSize units="bytes">93685056</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/atm/mo/clwvi/ukmo_hadgem1/run1/clwvi_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="mrsos_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.mrsos_A1_2006_Dec_to_2076_Dec.nc">
>>> <dataSize units="bytes">93685352</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/atm/mo/mrsos/ukmo_hadgem1/run1/mrsos_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="prsn_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.prsn_A1_2006_Dec_to_2076_Dec.nc">
>>> <dataSize units="bytes">93685008</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/atm/mo/prsn/ukmo_hadgem1/run1/prsn_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="rlut_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.rlut_A1_2006_Dec_to_2076_Dec.nc">
>>> <dataSize units="bytes">93685028</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/atm/mo/rlut/ukmo_hadgem1/run1/rlut_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="rsftoai_co2_A5.nc" ID="cmip3.files.rsftoai_co2_A5.nc">
>>> <dataSize units="bytes">680224</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/atm/mo/rsftoai_co2/ukmo_hadcm3/run1/rsftoai_co2_A5.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="rsftoai_co2_A5.nc" ID="cmip3.files.rsftoai_co2_A5.nc">
>>> <dataSize units="bytes">2684460</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/atm/mo/rsftoai_co2/ukmo_hadgem1/run1/rsftoai_co2_A5.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="rsftoaics_co2_A5.nc" ID="cmip3.files.rsftoaics_co2_A5.nc">
>>> <dataSize units="bytes">680256</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/atm/mo/rsftoaics_co2/ukmo_hadcm3/run1/rsftoaics_co2_A5.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="rsftoaics_co2_A5.nc" ID="cmip3.files.rsftoaics_co2_A5.nc">
>>> <dataSize units="bytes">2684496</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/atm/mo/rsftoaics_co2/ukmo_hadgem1/run1/rsftoaics_co2_A5.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="rsftropacs_co2_A5.nc" ID="cmip3.files.rsftropacs_co2_A5.nc">
>>> <dataSize units="bytes">680240</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/atm/mo/rsftropacs_co2/ukmo_hadcm3/run1/rsftropacs_co2_A5.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="rsftropacs_co2_A5.nc" ID="cmip3.files.rsftropacs_co2_A5.nc">
>>> <dataSize units="bytes">2684480</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/atm/mo/rsftropacs_co2/ukmo_hadgem1/run1/rsftropacs_co2_A5.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="rsntp_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.rsntp_A1_2006_Dec_to_2076_Dec.nc">
>>> <dataSize units="bytes">93685260</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/atm/mo/rsntp/ukmo_hadgem1/run1/rsntp_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="rsuscs_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.rsuscs_A1_2006_Dec_to_2076_Dec.nc">
>>> <dataSize units="bytes">93685076</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/atm/mo/rsuscs/ukmo_hadgem1/run1/rsuscs_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="rsut_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.rsut_A1_2006_Dec_to_2076_Dec.nc">
>>> <dataSize units="bytes">93685036</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/atm/mo/rsut/ukmo_hadgem1/run1/rsut_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="tauv_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.tauv_A1_2006_Dec_to_2076_Dec.nc">
>>> <dataSize units="bytes">93039124</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/atm/mo/tauv/ukmo_hadgem1/run1/tauv_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="ts_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.ts_A1_2006_Dec_to_2076_Dec.nc">
>>> <dataSize units="bytes">93685012</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/atm/mo/ts/ukmo_hadgem1/run1/ts_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="uas_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.uas_A1_2006_Dec_to_2076_Dec.nc">
>>> <dataSize units="bytes">93039312</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/atm/mo/uas/ukmo_hadgem1/run1/uas_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="va_A1_2006_Dec_to_2039_Dec.nc" ID="cmip3.files.va_A1_2006_Dec_to_2039_Dec.nc">
>>> <dataSize units="bytes">702501312</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/atm/mo/va/ukmo_hadgem1/run1/va_A1_2006_Dec_to_2039_Dec.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="va_A1_2040_Jan_to_2076_Dec.nc" ID="cmip3.files.va_A1_2040_Jan_to_2076_Dec.nc">
>>> <dataSize units="bytes">785667624</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/atm/mo/va/ukmo_hadgem1/run1/va_A1_2040_Jan_to_2076_Dec.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="sic_O1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.sic_O1_2006_Dec_to_2076_Dec.nc">
>>> <dataSize units="bytes">93685016</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/ice/mo/sic/ukmo_hadgem1/run1/sic_O1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="orog_A1.nc" ID="cmip3.files.orog_A1.nc">
>>> <dataSize units="bytes">122080</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/land/fixed/orog/ukmo_hadgem1/run1/orog_A1.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="sftlf_A1.nc" ID="cmip3.files.sftlf_A1.nc">
>>> <dataSize units="bytes">122068</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/land/fixed/sftlf/ukmo_hadgem1/run1/sftlf_A1.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="mrro_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.mrro_A1_2006_Dec_to_2076_Dec.nc">
>>> <dataSize units="bytes">93685020</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/land/mo/mrro/ukmo_hadgem1/run1/mrro_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="mrros_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.mrros_A1_2006_Dec_to_2076_Dec.nc">
>>> <dataSize units="bytes">93685016</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/land/mo/mrros/ukmo_hadgem1/run1/mrros_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="mrso_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.mrso_A1_2006_Dec_to_2076_Dec.nc">
>>> <dataSize units="bytes">93685024</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/land/mo/mrso/ukmo_hadgem1/run1/mrso_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="snm_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.snm_A1_2006_Dec_to_2076_Dec.nc">
>>> <dataSize units="bytes">93685024</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/land/mo/snm/ukmo_hadgem1/run1/snm_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="snw_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.snw_A1_2006_Dec_to_2076_Dec.nc">
>>> <dataSize units="bytes">93685016</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/land/mo/snw/ukmo_hadgem1/run1/snw_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="zobt_O1.nc" ID="cmip3.files.zobt_O1.nc">
>>> <dataSize units="bytes">327560</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/ocn/fixed/zobt/ukmo_hadgem1/run1/zobt_O1.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="tos_O1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.tos_O1_2006_Dec_to_2076_Dec.nc">
>>> <dataSize units="bytes">93685016</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/ocn/mo/tos/ukmo_hadgem1/run1/tos_O1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="usi_O1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.usi_O1_2006_Dec_to_2076_Dec.nc">
>>> <dataSize units="bytes">93685028</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/ocn/mo/usi/ukmo_hadgem1/run1/usi_O1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> <dataset name="vsi_O1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.vsi_O1_2006_Dec_to_2076_Dec.nc">
>>> <dataSize units="bytes">93039116</dataSize>
>>> <access urlPath="cmip3_datadir/2xco2/ocn/mo/vsi/ukmo_hadgem1/run1/vsi_O1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
>>> </dataset>
>>> </dataset>
>>> </dataset>
>>> </catalog>
>>>
>>>
>> _______________________________________________
>> GO-ESSP-TECH mailing list
>> GO-ESSP-TECH at ucar.edu
>> http://*mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>
>>
>
>
>
>
More information about the GO-ESSP-TECH
mailing list