[Go-essp-tech] [esg-gateway-dev] publishing error (invalid hour)
stephen.pascoe at stfc.ac.uk
stephen.pascoe at stfc.ac.uk
Wed Jan 4 02:18:36 MST 2012
Muhammad,
These problems are difficult to diagnose. These errors suggest an inconsistency problem with your Gateway database and/or THREDDS catalogs. I suspect you are missing a root THREDDS catalog in the TDS contents directory or it is somehow unreadable. Check what the TDS looks like in your browser. Can you view the top-level catalog as XML? Is the XML valid?
I'm also concerned that your Gateway database may be inconsistent. Check the the tables metadata.persistent_identifier, metadata.resource and metadata.dataset for the id "cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1". They all share the same uuid: resource_id==dataset_id.
Cheers,
Stephen.
---
Stephen Pascoe +44 (0)1235 445980
Centre of Environmental Data Archival
STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
From: esg-gateway-dev-bounces at mailman.earthsystemgrid.org [mailto:esg-gateway-dev-bounces at mailman.earthsystemgrid.org] On Behalf Of Muhammad Atif
Sent: 04 January 2012 02:40
To: Karl Taylor
Cc: Pascoe, Stephen (STFC,RAL,RALSP); go-essp-tech at ucar.edu; esg-gateway-dev at earthsystemgrid.org
Subject: Re: [esg-gateway-dev] [Go-essp-tech] publishing error (invalid hour)
Thanks for the tips; however I am now facing the problem that I stated in the previous email.
I wanted to pin point the file which was causing the issue and for that tried to publish the datasets one by one. Now I cannot even delete the said data.
[root at esgnode1 debug-historicalAntNoAA]# esgunpublish --skip-thredds cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
INFO 2012-01-04 13:39:49,576 Deleting cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
WARNING 2012-01-04 13:39:49,687 Deletion/retraction failed for dataset cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1 with message: Java ServiceException: The persistent identifier=cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1 does not correspond to a dataset at sgf.gateway.service.security.impl.acegi.AcegiCatalogAccessDecisionVoter.vote(AcegiCatalogAccessDecisionVoter.java:74)
INFO 2012-01-04 13:39:49,706 Result: UNSUCCESSFUL
[root at esgnode1 debug-historicalAntNoAA]# esgunpublish --skip-gateway cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
INFO 2012-01-04 13:40:35,331 Writing THREDDS ESG master catalog /data/esg-node/tds-content/thredds/esgcet/catalog.xml
INFO 2012-01-04 13:40:35,719 Reinitializing THREDDS server
Traceback (most recent call last):
File "/usr/local/cdat/bin/esgunpublish", line 5, in <module>
pkg_resources.run_script('esgcet==2.8.4', 'esgunpublish')
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 489, in run_script
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1207, in run_script
File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgunpublish", line 221, in <module>
main(sys.argv[1:])
File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgunpublish", line 184, in main
result = deleteDatasetList(datasetNames, Session, gatewayOp, thredds, las, deleteDset, deleteAll=deleteAll, republish=republish)
File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/unpublish.py", line 272, in deleteDatasetList
result = reinitializeThredds()
File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/thredds.py", line 1041, in reinitializeThredds
raise ESGPublishError("Error reinitializing the THREDDS Data Server: Fatal error: %s\n%s"%(errorMessage, str(result)))
esgcet.exceptions.ESGPublishError: Error reinitializing the THREDDS Data Server: Fatal error: **Fatal: InvCatalogFactory.readXML failed
Catalog init catalog.xml
[2012-01-04T02:40:35GMT]
[root at esgnode1 debug-historicalAntNoAA]# esgunpublish --database-delete cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
INFO 2012-01-04 13:34:34,130 Deleting cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
WARNING 2012-01-04 13:34:34,257 Deletion/retraction failed for dataset cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1 with message: Java ServiceException: The persistent identifier=cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1 does not correspond to a dataset at sgf.gateway.service.security.impl.acegi.AcegiCatalogAccessDecisionVoter.vote(AcegiCatalogAccessDecisionVoter.java:74)
INFO 2012-01-04 13:34:34,274 Result: UNSUCCESSFUL
INFO 2012-01-04 13:34:34,286 Writing THREDDS ESG master catalog /data/esg-node/tds-content/thredds/esgcet/catalog.xml
INFO 2012-01-04 13:34:34,661 Reinitializing THREDDS server
Traceback (most recent call last):
File "/usr/local/cdat/bin/esgunpublish", line 5, in <module>
pkg_resources.run_script('esgcet==2.8.4', 'esgunpublish')
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 489, in run_script
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1207, in run_script
File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgunpublish", line 221, in <module>
main(sys.argv[1:])
File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgunpublish", line 184, in main
result = deleteDatasetList(datasetNames, Session, gatewayOp, thredds, las, deleteDset, deleteAll=deleteAll, republish=republish)
File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/unpublish.py", line 272, in deleteDatasetList
result = reinitializeThredds()
File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/thredds.py", line 1041, in reinitializeThredds
raise ESGPublishError("Error reinitializing the THREDDS Data Server: Fatal error: %s\n%s"%(errorMessage, str(result)))
esgcet.exceptions.ESGPublishError: Error reinitializing the THREDDS Data Server: Fatal error: **Fatal: InvCatalogFactory.readXML failed
Catalog init catalog.xml
[2012-01-04T02:34:34GMT]
readCatalog(): full path=/data/esg-node/tds-content/thredds/catalog.xml; path=catalog.xml
readCatalog(): valid catalog -- ----Catalog Validation version 1.0.01
[root at esgnode1 debug-historicalAntNoAA]# esglist_datasets cmip5 | grep CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
| 4422 | cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1 | cmip5 | CSIRO-Mk3-6-0 | historicalAntNoAA | r10i1p1 | False | None | restricted | mon | ocean | Omon | r10i1p1 | CSIRO-QCCCE | 2012-01-04 13:39:49 | DELETE_DATASET_FAILED | cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0 | 20120104 | cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1.v20120104 | None |
Thanks for your help in advance.
Regards,
On 04/01/12 04:41, Karl Taylor wrote:
Hi Muhammad,
Better yet, execute "ncdump -c". This will return both the header information and the coordinate values. It would be good to see what the time coordinate values are in the file.
thanks,
Karl
On 1/3/12 6:34 AM, stephen.pascoe at stfc.ac.uk<mailto:stephen.pascoe at stfc.ac.uk> wrote:
Hi Muhammad,
Can you send a dump of the NetCDF metadata for this file. The command is "ncdump -h". Whether a time value is valid depends on the time:units attribute and sometimes other attributes.
Cheers,
Stephen.
---
Stephen Pascoe +44 (0)1235 445980
Centre of Environmental Data Archival
STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
-----Original Message-----
From: esg-gateway-dev-bounces at mailman.earthsystemgrid.org<mailto:esg-gateway-dev-bounces at mailman.earthsystemgrid.org> [mailto:esg-gateway-dev-bounces at mailman.earthsystemgrid.org] On Behalf Of Muhammad Atif
Sent: 03 January 2012 04:37
To: go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>; esg-gateway-dev at earthsystemgrid.org<mailto:esg-gateway-dev at earthsystemgrid.org>
Subject: [esg-gateway-dev] publishing error (invalid hour)
Dear All,
In an unrelated problem from the previous email, we are facing another
error that gives "CDMS error: Error on time conversion: invalid hour =
28864363504146.773438"
Please let me know what to do with it. The modelers suggest that they
were not able to find any funny stuff.
INFO 2011-12-28 12:24:32,436 Writing THREDDS catalog
/data/esg-node/tds-content/thredds/esgcet/9/cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1.v20111228.xml
CDMS error: Error on time conversion: invalid hour = 28864363504146.773438
Traceback (most recent call last):
File "/usr/local/cdat/bin/esgpublish", line 5, in <module>
pkg_resources.run_script('esgcet==2.8.4', 'esgpublish')
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 489, in
run_script
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1207, in
run_script
File
"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgpublish",
line 434, in <module>
main(sys.argv[1:])
File
"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgpublish",
line 420, in main
result = publishDatasetList(datasetNames, Session, publish=publish,
thredds=thredds, las=las, parentId=parent, service=service,
perVariable=perVariable)
File
"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/publish.py",
line 244, in publishDatasetList
generateThredds(datasetName, Session, threddsOutput, handler,
service=service, perVariable=perVariable, versionNumber=versionno)
File
"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/thredds.py",
line 726, in generateThredds
_generateThreddsV2(datasetName, outputFile, handler, session, dset,
context, project, model, experiment, config, section, genRoot=genRoot,
service=service, perVariable=perVariable, versionNumber=versionNumber)
File
"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/thredds.py",
line 950, in _generateThreddsV2
_genPerVariableDatasetsV2(datasetElem, dset, datasetName,
resolution, filesRootLoc, filesRootPath, datasetRootDict,
excludeVariables, offline, serviceName, serviceDict, aggServiceName,
handler, project, model, experiment, lasConfigure, lasTimeDelta,
versionNumber, variables, variableElemDict, lasServiceSpecs,
lasServiceHash, gridftpMap=gridftpMapDatasetRoots)
File
"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/thredds.py",
line 563, in _genPerVariableDatasetsV2
timeFirst = mdhandler.genTime(variable.aggdim_first,
dataset.aggdim_units, dataset.calendar)
File
"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/config/cf_handler.py",
line 179, in genTime
c = t.tocomp(_tagToCalendar[calendarTag])
Cdtime error: Invalid component time
Regards,
--
Muhammad Atif
ANU Supercomputer Facility NCI National Facility
Leonard Huxley, Mills Road Room 318, Bldg # 56
Australian National University Ph: +61 2 6125 5031
Canberra, ACT 0200, Australia M: 0430 393863
CRICOS Provider #00120C http://anusf.anu.edu.au
--
Scanned by iCritical.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20120104/bd5186a1/attachment-0001.html
More information about the GO-ESSP-TECH
mailing list