[Go-essp-tech] [esg-gateway-dev] publishing error (invalid hour)
Drach, Bob
drach1 at llnl.gov
Fri Jan 6 13:49:51 MST 2012
Hi Muhammad,
I suspect that because of the original error (bad time value?) the catalog for that dataset is invalid, and this is causing the reinitialization error. Normally esgunpublish would allow you to remove it, but I've seen instances before where that didn't work for various reasons. To correct the problem,
- Find the 'bad' catalog and manually remove it. From your earlier email the file to remove is:
/data/esg-node/tds-content/thredds/esgcet/9/cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1.v20111228.xml
- Reinitialize the TDS:
>From esg.ini get the value of thredds_reinit_url. (On our system it's at: https://localhost:443/thredds/admin/debug?catalogs/reinit). Open the URL in a browser, authenticate with the values of thredds_username and thredds_password in esg.ini. If successful, you will see a screen with links to various logs. Check the catalog init log to see if any other errors occurred (or look directly at
/data/esg-node/tds-content/thredds/logs/catalogInit.log
You've already run esgunpublish --database-delete, so subsequent publish / unpublish operations won't try to regenerate the bogus catalog.
- Find the source of the original time error. This may take a bit of detective work
Run the script esgcheck_times on each file in the erroneous dataset. If the time range is bad you should be able to find which file is bad. One possibility is that one or more files are truncated for some reason, so the time dimension value is corrupted.
- I don't suspect a problem with the gateway database based on the error message below.
Regards,
--Bob
________________________________________
From: esg-gateway-dev-bounces at mailman.earthsystemgrid.org [esg-gateway-dev-bounces at mailman.earthsystemgrid.org] On Behalf Of Muhammad Atif [muhammad.atif at anu.edu.au]
Sent: Thursday, January 05, 2012 9:06 PM
To: stephen.pascoe at stfc.ac.uk
Cc: Taylor, Karl Taylor; go-essp-tech at ucar.edu; esg-gateway-dev at earthsystemgrid.org
Subject: Re: [esg-gateway-dev] [Go-essp-tech] publishing error (invalid hour)
Thanks for that Stephen,
Will try to look in deeper next week. Just for the heck of it, is there a nice recommended way to remove everything from the database related to a model/experiment? A clean start?
Regards,
On 04/01/12 20:18, stephen.pascoe at stfc.ac.uk<mailto:stephen.pascoe at stfc.ac.uk> wrote:
Muhammad,
These problems are difficult to diagnose. These errors suggest an inconsistency problem with your Gateway database and/or THREDDS catalogs. I suspect you are missing a root THREDDS catalog in the TDS contents directory or it is somehow unreadable. Check what the TDS looks like in your browser. Can you view the top-level catalog as XML? Is the XML valid?
I'm also concerned that your Gateway database may be inconsistent. Check the the tables metadata.persistent_identifier, metadata.resource and metadata.dataset for the id "cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1". They all share the same uuid: resource_id==dataset_id.
Cheers,
Stephen.
---
Stephen Pascoe +44 (0)1235 445980
Centre of Environmental Data Archival
STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
From: esg-gateway-dev-bounces at mailman.earthsystemgrid.org<mailto:esg-gateway-dev-bounces at mailman.earthsystemgrid.org> [mailto:esg-gateway-dev-bounces at mailman.earthsystemgrid.org] On Behalf Of Muhammad Atif
Sent: 04 January 2012 02:40
To: Karl Taylor
Cc: Pascoe, Stephen (STFC,RAL,RALSP); go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>; esg-gateway-dev at earthsystemgrid.org<mailto:esg-gateway-dev at earthsystemgrid.org>
Subject: Re: [esg-gateway-dev] [Go-essp-tech] publishing error (invalid hour)
Thanks for the tips; however I am now facing the problem that I stated in the previous email.
I wanted to pin point the file which was causing the issue and for that tried to publish the datasets one by one. Now I cannot even delete the said data.
[root at esgnode1 debug-historicalAntNoAA]# esgunpublish --skip-thredds cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
INFO 2012-01-04 13:39:49,576 Deleting cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
WARNING 2012-01-04 13:39:49,687 Deletion/retraction failed for dataset cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1 with message: Java ServiceException: The persistent identifier=cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1 does not correspond to a dataset at sgf.gateway.service.security.impl.acegi.AcegiCatalogAccessDecisionVoter.vote(AcegiCatalogAccessDecisionVoter.java:74)
INFO 2012-01-04 13:39:49,706 Result: UNSUCCESSFUL
[root at esgnode1 debug-historicalAntNoAA]# esgunpublish --skip-gateway cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
INFO 2012-01-04 13:40:35,331 Writing THREDDS ESG master catalog /data/esg-node/tds-content/thredds/esgcet/catalog.xml
INFO 2012-01-04 13:40:35,719 Reinitializing THREDDS server
Traceback (most recent call last):
File "/usr/local/cdat/bin/esgunpublish", line 5, in <module>
pkg_resources.run_script('esgcet==2.8.4', 'esgunpublish')
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 489, in run_script
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1207, in run_script
File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgunpublish", line 221, in <module>
main(sys.argv[1:])
File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgunpublish", line 184, in main
result = deleteDatasetList(datasetNames, Session, gatewayOp, thredds, las, deleteDset, deleteAll=deleteAll, republish=republish)
File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/unpublish.py", line 272, in deleteDatasetList
result = reinitializeThredds()
File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/thredds.py", line 1041, in reinitializeThredds
raise ESGPublishError("Error reinitializing the THREDDS Data Server: Fatal error: %s\n%s"%(errorMessage, str(result)))
esgcet.exceptions.ESGPublishError: Error reinitializing the THREDDS Data Server: Fatal error: **Fatal: InvCatalogFactory.readXML failed
Catalog init catalog.xml
[2012-01-04T02:40:35GMT]
[root at esgnode1 debug-historicalAntNoAA]# esgunpublish --database-delete cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
INFO 2012-01-04 13:34:34,130 Deleting cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
WARNING 2012-01-04 13:34:34,257 Deletion/retraction failed for dataset cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1 with message: Java ServiceException: The persistent identifier=cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1 does not correspond to a dataset at sgf.gateway.service.security.impl.acegi.AcegiCatalogAccessDecisionVoter.vote(AcegiCatalogAccessDecisionVoter.java:74)
INFO 2012-01-04 13:34:34,274 Result: UNSUCCESSFUL
INFO 2012-01-04 13:34:34,286 Writing THREDDS ESG master catalog /data/esg-node/tds-content/thredds/esgcet/catalog.xml
INFO 2012-01-04 13:34:34,661 Reinitializing THREDDS server
Traceback (most recent call last):
File "/usr/local/cdat/bin/esgunpublish", line 5, in <module>
pkg_resources.run_script('esgcet==2.8.4', 'esgunpublish')
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 489, in run_script
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1207, in run_script
File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgunpublish", line 221, in <module>
main(sys.argv[1:])
File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgunpublish", line 184, in main
result = deleteDatasetList(datasetNames, Session, gatewayOp, thredds, las, deleteDset, deleteAll=deleteAll, republish=republish)
File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/unpublish.py", line 272, in deleteDatasetList
result = reinitializeThredds()
File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/thredds.py", line 1041, in reinitializeThredds
raise ESGPublishError("Error reinitializing the THREDDS Data Server: Fatal error: %s\n%s"%(errorMessage, str(result)))
esgcet.exceptions.ESGPublishError: Error reinitializing the THREDDS Data Server: Fatal error: **Fatal: InvCatalogFactory.readXML failed
Catalog init catalog.xml
[2012-01-04T02:34:34GMT]
readCatalog(): full path=/data/esg-node/tds-content/thredds/catalog.xml; path=catalog.xml
readCatalog(): valid catalog -- ----Catalog Validation version 1.0.01
[root at esgnode1 debug-historicalAntNoAA]# esglist_datasets cmip5 | grep CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
| 4422 | cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1 | cmip5 | CSIRO-Mk3-6-0 | historicalAntNoAA | r10i1p1 | False | None | restricted | mon | ocean | Omon | r10i1p1 | CSIRO-QCCCE | 2012-01-04 13:39:49 | DELETE_DATASET_FAILED | cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0 | 20120104 | cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1.v20120104 | None |
Thanks for your help in advance.
Regards,
On 04/01/12 04:41, Karl Taylor wrote:
Hi Muhammad,
Better yet, execute "ncdump -c". This will return both the header information and the coordinate values. It would be good to see what the time coordinate values are in the file.
thanks,
Karl
On 1/3/12 6:34 AM, stephen.pascoe at stfc.ac.uk<mailto:stephen.pascoe at stfc.ac.uk> wrote:
Hi Muhammad,
Can you send a dump of the NetCDF metadata for this file. The command is "ncdump -h". Whether a time value is valid depends on the time:units attribute and sometimes other attributes.
Cheers,
Stephen.
---
Stephen Pascoe +44 (0)1235 445980
Centre of Environmental Data Archival
STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
-----Original Message-----
From: esg-gateway-dev-bounces at mailman.earthsystemgrid.org<mailto:esg-gateway-dev-bounces at mailman.earthsystemgrid.org> [mailto:esg-gateway-dev-bounces at mailman.earthsystemgrid.org] On Behalf Of Muhammad Atif
Sent: 03 January 2012 04:37
To: go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>; esg-gateway-dev at earthsystemgrid.org<mailto:esg-gateway-dev at earthsystemgrid.org>
Subject: [esg-gateway-dev] publishing error (invalid hour)
Dear All,
In an unrelated problem from the previous email, we are facing another
error that gives "CDMS error: Error on time conversion: invalid hour =
28864363504146.773438"
Please let me know what to do with it. The modelers suggest that they
were not able to find any funny stuff.
INFO 2011-12-28 12:24:32,436 Writing THREDDS catalog
/data/esg-node/tds-content/thredds/esgcet/9/cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1.v20111228.xml
CDMS error: Error on time conversion: invalid hour = 28864363504146.773438
Traceback (most recent call last):
File "/usr/local/cdat/bin/esgpublish", line 5, in <module>
pkg_resources.run_script('esgcet==2.8.4', 'esgpublish')
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 489, in
run_script
File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1207, in
run_script
File
"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgpublish",
line 434, in <module>
main(sys.argv[1:])
File
"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgpublish",
line 420, in main
result = publishDatasetList(datasetNames, Session, publish=publish,
thredds=thredds, las=las, parentId=parent, service=service,
perVariable=perVariable)
File
"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/publish.py",
line 244, in publishDatasetList
generateThredds(datasetName, Session, threddsOutput, handler,
service=service, perVariable=perVariable, versionNumber=versionno)
File
"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/thredds.py",
line 726, in generateThredds
_generateThreddsV2(datasetName, outputFile, handler, session, dset,
context, project, model, experiment, config, section, genRoot=genRoot,
service=service, perVariable=perVariable, versionNumber=versionNumber)
File
"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/thredds.py",
line 950, in _generateThreddsV2
_genPerVariableDatasetsV2(datasetElem, dset, datasetName,
resolution, filesRootLoc, filesRootPath, datasetRootDict,
excludeVariables, offline, serviceName, serviceDict, aggServiceName,
handler, project, model, experiment, lasConfigure, lasTimeDelta,
versionNumber, variables, variableElemDict, lasServiceSpecs,
lasServiceHash, gridftpMap=gridftpMapDatasetRoots)
File
"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/thredds.py",
line 563, in _genPerVariableDatasetsV2
timeFirst = mdhandler.genTime(variable.aggdim_first,
dataset.aggdim_units, dataset.calendar)
File
"/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/config/cf_handler.py",
line 179, in genTime
c = t.tocomp(_tagToCalendar[calendarTag])
Cdtime error: Invalid component time
Regards,
--
Muhammad Atif
ANU Supercomputer Facility NCI National Facility
Leonard Huxley, Mills Road Room 318, Bldg # 56
Australian National University Ph: +61 2 6125 5031
Canberra, ACT 0200, Australia M: 0430 393863
CRICOS Provider #00120C http://anusf.anu.edu.au
--
Scanned by iCritical.
--
Muhammad Atif
ANU Supercomputer Facility NCI National Facility
Leonard Huxley, Mills Road Room 318, Bldg # 56
Australian National University Ph: +61 2 6125 5031
Canberra, ACT 0200, Australia M: 0430 393863
CRICOS Provider #00120C http://anusf.anu.edu.au
More information about the GO-ESSP-TECH
mailing list