[Go-essp-tech] NetCDF metadata in THREDDS catalogues
Bob Drach
drach at llnl.gov
Wed Nov 4 13:25:58 MST 2009
Hi Stephen,
stephen.pascoe at stfc.ac.uk wrote:
> Hi Bob,
>
> We are evaluating how best to extract NetCDF metadata on our esg data
> node to reformulate into METAFOR CIM documents. It seems obvious
> there are 3 possible sources of this:
>
> 1. THREDDS catalogues generated by esg publisher
> 2. The Postgres database produced by esg publisher
> 3. The NetCDF files themselves
>
> Looking at the THREDDS xml generated on our data node it appears only
> selected metadata from the NetCDF is exposed in the catalogue. I can
> see variable names, standard_names and units with no global metadata.
> Is that what you expect? I attach a catalogue for a small portion of
> CMIP3.
That is what I would expect. The catalog will also define aggregations -
typically on the time dimension - if the variable_per_file config option
equals true, but this only contains additional time metadata, not all
the file metadata.
>
> I'm guessing that with a correct configuration in esg.ini
> more metadata get's put in the THREDDS xml but does this come from the
> NetCDF? Is there a way to configure the publisher to put all NetCDF
> attributes in the THREDDS xml?
Not in any straightforward way. If you know in advance which attributes
are to be generated, they can be defined as THREDDS properties on a
per-project basis. But that's not completely general.
>
> Turning to the Postgres database it appears to contain all attributes
> from the NetCDF. On my system it also contains lots of Python object
> references that I assume is a bug.
Yes, it's a bug - on my list of things to fix.
The database does contain all the metadata in the files, with the
possible exception of coordinate data if you consider that as metadata.
However coordinate ranges for aggregate variables are kept in the
variable table. The postgres database would be the logical place to
obtain the desired information.
Some of the table definitions will change in the next version, but I
would expect the tables containing attribute and dimension information
to remain stable.
Bob
> For instance:
>
> esgcet=> select name, value from file_var_attr limit 20;
> name |
> value
> ---------------+------------------------------------------------------------------
> axis | Y
> bounds | lat_bnds
> getitem | <built-in method getitem of CdunifVariable object at
> 0x19f8150>
> getslice | <built-in method getslice of CdunifVariable object at
> 0x19f8150>
> long_name | latitude
> setitem | <built-in method setitem of CdunifVariable object at
> 0x19f8150>
> setslice | <built-in method setslice of CdunifVariable object at
> 0x19f8150>
> standard_name | latitude
> units | degrees_north
> getitem | <built-in method getitem of CdunifVariable object at
> 0x19f8030>
> getslice | <built-in method getslice of CdunifVariable object at
> 0x19f8030>
> setitem | <built-in method setitem of CdunifVariable object at
> 0x19f8030>
> setslice | <built-in method setslice of CdunifVariable object at
> 0x19f8030>
> axis | X
> bounds | lon_bnds
> getitem | <built-in method getitem of CdunifVariable object at
> 0x19f8108>
> getslice | <built-in method getslice of CdunifVariable object at
> 0x19f8108>
> long_name | longitude
> setitem | <built-in method setitem of CdunifVariable object at
> 0x19f8108>
> setslice | <built-in method setslice of CdunifVariable object at
> 0x19f8108>
> (20 rows)
>
> ---
> Stephen Pascoe +44 (0)1235 445980
> British Atmospheric Data Centre
> Rutherford Appleton Laboratory
>
>
> --
> Scanned by iCritical.
>
>
> ------------------------------------------------------------------------
>
> <?xml version='1.0' encoding='UTF-8'?>
> <catalog xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" name="TDS configuration file" xsi:schemaLocation="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0 http://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.0.2.xsd">
> <service serviceType="OpenDAP" base="/thredds/dodsC/" name="gridded"/>
> <service serviceType="HTTPServer" base="/thredds/fileServer/" name="HTTPServer"/>
> <service serviceType="SRM" base="srm://host.sample.gov:6288/srm/v2/server?SFN=/archive.sample.gov/" name="HRMatPCMDI"/>
> <dataset restrictAccess="esg-user" ID="cmip3" name="cmip3">
> <property name="project" value="test"/>
> <documentation type="summary"/>
> <metadata inherited="true">
> <variables vocabulary="CF-1.0">
> <variable name="clivi" vocabulary_name="atmosphere_cloud_ice_content" units="kg m-2">Column Integerated Cloud Ice Content</variable>
> <variable name="clwvi" vocabulary_name="atmosphere_cloud_condensed_water_content" units="kg m-2">Column Integerated Cloud Water Content</variable>
> <variable name="mrsos" vocabulary_name="moisture_content_of_soil_layer" units="kg m-2">Moisture in Upper 0.1 m of Soil Column</variable>
> <variable name="prsn" vocabulary_name="snowfall_flux" units="kg m-2 s-1">Snowfall Flux</variable>
> <variable name="rlut" vocabulary_name="toa_outgoing_longwave_flux" units="W m-2">Outgoing Longwave Radiation</variable>
> <variable name="rsftoai_co2" vocabulary_name="toa_instantaneous_shortwave_forcing" units="W m-2">Instantaneous Top of the Atmosphere Shortwave Forcing Due to Anthropogenic Carbon Dioxide</variable>
> <variable name="rsftoaics_co2" vocabulary_name="Instantaneous Top of the Atmosphere Shortwave Clear-Sky Forcing Due to Anthropogenic Carbon Dioxide" units="W m-2">Instantaneous Top of the Atmosphere Shortwave Clear-Sky Forcing Due to Anthropogenic Carbon Dioxide</variable>
> <variable name="rsftropacs_co2" vocabulary_name="Adjusted Tropopause Shortwave Clear-Sky Forcing Due to Anthropogenic Carbon Dioxide" units="W m-2">Adjusted Tropopause Shortwave Clear-Sky Forcing Due to Anthropogenic Carbon Dioxide</variable>
> <variable name="rsntp" vocabulary_name="net_downward_shortwave_flux_in_air" units="W m-2">Net Downward Shortwave Flux at 200 hPa</variable>
> <variable name="rsuscs" vocabulary_name="surface_upwelling_shortwave_flux_in_air_assuming_clear_sky" units="W m-2">Reflected Surface Clear-Sky Shortwave</variable>
> <variable name="rsut" vocabulary_name="toa_outgoing_shortwave_flux" units="W m-2">TOA Reflected Shortwave Radiation</variable>
> <variable name="tauv" vocabulary_name="surface_downward_northward_stress" units="Pa">Meridional Surface Wind Stress</variable>
> <variable name="ts" vocabulary_name="surface_temperature" units="K">Surface Skin Temperature</variable>
> <variable name="uas" vocabulary_name="eastward_wind" units="m s-1">Zonal Surface Wind Speed</variable>
> <variable name="va" vocabulary_name="northward_wind" units="m s-1">Meridional Wind Component</variable>
> <variable name="sic" vocabulary_name="sea_ice_area_fraction" units="%">Sea Ice Concentration</variable>
> <variable name="orog" vocabulary_name="surface_altitude" units="m">Surface Altitude</variable>
> <variable name="sftlf" vocabulary_name="land_area_fraction" units="%">Land Area Fraction</variable>
> <variable name="mrro" vocabulary_name="runoff_flux" units="kg m-2 s-1">Surface and Subsurface Runoff</variable>
> <variable name="mrros" vocabulary_name="surface_runoff_flux" units="kg m-2 s-1">Surface Runoff</variable>
> <variable name="mrso" vocabulary_name="soil_moisture_content" units="kg m-2">Total Soil Moisture Content</variable>
> <variable name="snm" vocabulary_name="surface_snow_melt_flux_where_land" units="kg m-2 s-1">Snow Melt</variable>
> <variable name="snw" vocabulary_name="surface_snow_amount_where_land" units="kg m-2">Snow Amount</variable>
> <variable name="zobt" vocabulary_name="sea_floor_depth_below_geoid" units="m">Sea Floor Depth</variable>
> <variable name="tos" vocabulary_name="sea_surface_temperature" units="K">Sea Surface Temperature</variable>
> <variable name="usi" vocabulary_name="eastward_sea_ice_velocity" units="m s-1">Eastward Sea Ice Velocity</variable>
> <variable name="vsi" vocabulary_name="northward_sea_ice_velocity" units="m s-1">Northward Sea Ice Velocity</variable>
> </variables>
> </metadata>
> <metadata inherited="true">
> <dataType>Grid</dataType>
> <dataFormat>NetCDF</dataFormat>
> </metadata>
> <dataset ID="cmip3.files" name="cmip3.files">
> <dataset name="clivi_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.clivi_A1_2006_Dec_to_2076_Dec.nc">
> <dataSize units="bytes">93685040</dataSize>
> <access urlPath="cmip3_datadir/2xco2/atm/mo/clivi/ukmo_hadgem1/run1/clivi_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="clwvi_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.clwvi_A1_2006_Dec_to_2076_Dec.nc">
> <dataSize units="bytes">93685056</dataSize>
> <access urlPath="cmip3_datadir/2xco2/atm/mo/clwvi/ukmo_hadgem1/run1/clwvi_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="mrsos_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.mrsos_A1_2006_Dec_to_2076_Dec.nc">
> <dataSize units="bytes">93685352</dataSize>
> <access urlPath="cmip3_datadir/2xco2/atm/mo/mrsos/ukmo_hadgem1/run1/mrsos_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="prsn_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.prsn_A1_2006_Dec_to_2076_Dec.nc">
> <dataSize units="bytes">93685008</dataSize>
> <access urlPath="cmip3_datadir/2xco2/atm/mo/prsn/ukmo_hadgem1/run1/prsn_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="rlut_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.rlut_A1_2006_Dec_to_2076_Dec.nc">
> <dataSize units="bytes">93685028</dataSize>
> <access urlPath="cmip3_datadir/2xco2/atm/mo/rlut/ukmo_hadgem1/run1/rlut_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="rsftoai_co2_A5.nc" ID="cmip3.files.rsftoai_co2_A5.nc">
> <dataSize units="bytes">680224</dataSize>
> <access urlPath="cmip3_datadir/2xco2/atm/mo/rsftoai_co2/ukmo_hadcm3/run1/rsftoai_co2_A5.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="rsftoai_co2_A5.nc" ID="cmip3.files.rsftoai_co2_A5.nc">
> <dataSize units="bytes">2684460</dataSize>
> <access urlPath="cmip3_datadir/2xco2/atm/mo/rsftoai_co2/ukmo_hadgem1/run1/rsftoai_co2_A5.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="rsftoaics_co2_A5.nc" ID="cmip3.files.rsftoaics_co2_A5.nc">
> <dataSize units="bytes">680256</dataSize>
> <access urlPath="cmip3_datadir/2xco2/atm/mo/rsftoaics_co2/ukmo_hadcm3/run1/rsftoaics_co2_A5.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="rsftoaics_co2_A5.nc" ID="cmip3.files.rsftoaics_co2_A5.nc">
> <dataSize units="bytes">2684496</dataSize>
> <access urlPath="cmip3_datadir/2xco2/atm/mo/rsftoaics_co2/ukmo_hadgem1/run1/rsftoaics_co2_A5.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="rsftropacs_co2_A5.nc" ID="cmip3.files.rsftropacs_co2_A5.nc">
> <dataSize units="bytes">680240</dataSize>
> <access urlPath="cmip3_datadir/2xco2/atm/mo/rsftropacs_co2/ukmo_hadcm3/run1/rsftropacs_co2_A5.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="rsftropacs_co2_A5.nc" ID="cmip3.files.rsftropacs_co2_A5.nc">
> <dataSize units="bytes">2684480</dataSize>
> <access urlPath="cmip3_datadir/2xco2/atm/mo/rsftropacs_co2/ukmo_hadgem1/run1/rsftropacs_co2_A5.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="rsntp_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.rsntp_A1_2006_Dec_to_2076_Dec.nc">
> <dataSize units="bytes">93685260</dataSize>
> <access urlPath="cmip3_datadir/2xco2/atm/mo/rsntp/ukmo_hadgem1/run1/rsntp_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="rsuscs_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.rsuscs_A1_2006_Dec_to_2076_Dec.nc">
> <dataSize units="bytes">93685076</dataSize>
> <access urlPath="cmip3_datadir/2xco2/atm/mo/rsuscs/ukmo_hadgem1/run1/rsuscs_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="rsut_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.rsut_A1_2006_Dec_to_2076_Dec.nc">
> <dataSize units="bytes">93685036</dataSize>
> <access urlPath="cmip3_datadir/2xco2/atm/mo/rsut/ukmo_hadgem1/run1/rsut_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="tauv_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.tauv_A1_2006_Dec_to_2076_Dec.nc">
> <dataSize units="bytes">93039124</dataSize>
> <access urlPath="cmip3_datadir/2xco2/atm/mo/tauv/ukmo_hadgem1/run1/tauv_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="ts_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.ts_A1_2006_Dec_to_2076_Dec.nc">
> <dataSize units="bytes">93685012</dataSize>
> <access urlPath="cmip3_datadir/2xco2/atm/mo/ts/ukmo_hadgem1/run1/ts_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="uas_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.uas_A1_2006_Dec_to_2076_Dec.nc">
> <dataSize units="bytes">93039312</dataSize>
> <access urlPath="cmip3_datadir/2xco2/atm/mo/uas/ukmo_hadgem1/run1/uas_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="va_A1_2006_Dec_to_2039_Dec.nc" ID="cmip3.files.va_A1_2006_Dec_to_2039_Dec.nc">
> <dataSize units="bytes">702501312</dataSize>
> <access urlPath="cmip3_datadir/2xco2/atm/mo/va/ukmo_hadgem1/run1/va_A1_2006_Dec_to_2039_Dec.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="va_A1_2040_Jan_to_2076_Dec.nc" ID="cmip3.files.va_A1_2040_Jan_to_2076_Dec.nc">
> <dataSize units="bytes">785667624</dataSize>
> <access urlPath="cmip3_datadir/2xco2/atm/mo/va/ukmo_hadgem1/run1/va_A1_2040_Jan_to_2076_Dec.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="sic_O1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.sic_O1_2006_Dec_to_2076_Dec.nc">
> <dataSize units="bytes">93685016</dataSize>
> <access urlPath="cmip3_datadir/2xco2/ice/mo/sic/ukmo_hadgem1/run1/sic_O1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="orog_A1.nc" ID="cmip3.files.orog_A1.nc">
> <dataSize units="bytes">122080</dataSize>
> <access urlPath="cmip3_datadir/2xco2/land/fixed/orog/ukmo_hadgem1/run1/orog_A1.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="sftlf_A1.nc" ID="cmip3.files.sftlf_A1.nc">
> <dataSize units="bytes">122068</dataSize>
> <access urlPath="cmip3_datadir/2xco2/land/fixed/sftlf/ukmo_hadgem1/run1/sftlf_A1.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="mrro_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.mrro_A1_2006_Dec_to_2076_Dec.nc">
> <dataSize units="bytes">93685020</dataSize>
> <access urlPath="cmip3_datadir/2xco2/land/mo/mrro/ukmo_hadgem1/run1/mrro_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="mrros_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.mrros_A1_2006_Dec_to_2076_Dec.nc">
> <dataSize units="bytes">93685016</dataSize>
> <access urlPath="cmip3_datadir/2xco2/land/mo/mrros/ukmo_hadgem1/run1/mrros_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="mrso_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.mrso_A1_2006_Dec_to_2076_Dec.nc">
> <dataSize units="bytes">93685024</dataSize>
> <access urlPath="cmip3_datadir/2xco2/land/mo/mrso/ukmo_hadgem1/run1/mrso_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="snm_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.snm_A1_2006_Dec_to_2076_Dec.nc">
> <dataSize units="bytes">93685024</dataSize>
> <access urlPath="cmip3_datadir/2xco2/land/mo/snm/ukmo_hadgem1/run1/snm_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="snw_A1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.snw_A1_2006_Dec_to_2076_Dec.nc">
> <dataSize units="bytes">93685016</dataSize>
> <access urlPath="cmip3_datadir/2xco2/land/mo/snw/ukmo_hadgem1/run1/snw_A1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="zobt_O1.nc" ID="cmip3.files.zobt_O1.nc">
> <dataSize units="bytes">327560</dataSize>
> <access urlPath="cmip3_datadir/2xco2/ocn/fixed/zobt/ukmo_hadgem1/run1/zobt_O1.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="tos_O1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.tos_O1_2006_Dec_to_2076_Dec.nc">
> <dataSize units="bytes">93685016</dataSize>
> <access urlPath="cmip3_datadir/2xco2/ocn/mo/tos/ukmo_hadgem1/run1/tos_O1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="usi_O1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.usi_O1_2006_Dec_to_2076_Dec.nc">
> <dataSize units="bytes">93685028</dataSize>
> <access urlPath="cmip3_datadir/2xco2/ocn/mo/usi/ukmo_hadgem1/run1/usi_O1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
> </dataset>
> <dataset name="vsi_O1_2006_Dec_to_2076_Dec.nc" ID="cmip3.files.vsi_O1_2006_Dec_to_2076_Dec.nc">
> <dataSize units="bytes">93039116</dataSize>
> <access urlPath="cmip3_datadir/2xco2/ocn/mo/vsi/ukmo_hadgem1/run1/vsi_O1_2006_Dec_to_2076_Dec.nc" serviceName="HTTPServer"/>
> </dataset>
> </dataset>
> </dataset>
> </catalog>
>
More information about the GO-ESSP-TECH
mailing list