[Go-essp-tech] [esg-gateway-dev] publishing error (invalid hour)
Muhammad Atif
muhammad.atif at anu.edu.au
Wed Jan 18 17:25:20 MST 2012
Hi All,
Sorry for not keeping you in loop. We had some dangling thredd catalogs
and by removing them everything got back to normal.
Regarding the invalid hour, there were few files that were indeed
incorrect and the modelers are looking into that.
Thanks for all the help.
Regards
On 07/01/12 07:49, Drach, Bob wrote:
> Hi Muhammad,
>
> I suspect that because of the original error (bad time value?) the catalog for that dataset is invalid, and this is causing the reinitialization error. Normally esgunpublish would allow you to remove it, but I've seen instances before where that didn't work for various reasons. To correct the problem,
>
> - Find the 'bad' catalog and manually remove it. From your earlier email the file to remove is:
>
> /data/esg-node/tds-content/thredds/esgcet/9/cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1.v20111228.xml
>
> - Reinitialize the TDS:
>
> From esg.ini get the value of thredds_reinit_url. (On our system it's at: https://localhost:443/thredds/admin/debug?catalogs/reinit). Open the URL in a browser, authenticate with the values of thredds_username and thredds_password in esg.ini. If successful, you will see a screen with links to various logs. Check the catalog init log to see if any other errors occurred (or look directly at
>
> /data/esg-node/tds-content/thredds/logs/catalogInit.log
>
> You've already run esgunpublish --database-delete, so subsequent publish / unpublish operations won't try to regenerate the bogus catalog.
>
> - Find the source of the original time error. This may take a bit of detective work
>
> Run the script esgcheck_times on each file in the erroneous dataset. If the time range is bad you should be able to find which file is bad. One possibility is that one or more files are truncated for some reason, so the time dimension value is corrupted.
>
> - I don't suspect a problem with the gateway database based on the error message below.
>
> Regards,
>
> --Bob
> ________________________________________
> From: esg-gateway-dev-bounces at mailman.earthsystemgrid.org [esg-gateway-dev-bounces at mailman.earthsystemgrid.org] On Behalf Of Muhammad Atif [muhammad.atif at anu.edu.au]
> Sent: Thursday, January 05, 2012 9:06 PM
> To: stephen.pascoe at stfc.ac.uk
> Cc: Taylor, Karl Taylor; go-essp-tech at ucar.edu; esg-gateway-dev at earthsystemgrid.org
> Subject: Re: [esg-gateway-dev] [Go-essp-tech] publishing error (invalid hour)
>
> Thanks for that Stephen,
> Will try to look in deeper next week. Just for the heck of it, is there a nice recommended way to remove everything from the database related to a model/experiment? A clean start?
>
> Regards,
>
> On 04/01/12 20:18, stephen.pascoe at stfc.ac.uk<mailto:stephen.pascoe at stfc.ac.uk> wrote:
> Muhammad,
>
> These problems are difficult to diagnose. These errors suggest an inconsistency problem with your Gateway database and/or THREDDS catalogs. I suspect you are missing a root THREDDS catalog in the TDS contents directory or it is somehow unreadable. Check what the TDS looks like in your browser. Can you view the top-level catalog as XML? Is the XML valid?
>
> I'm also concerned that your Gateway database may be inconsistent. Check the the tables metadata.persistent_identifier, metadata.resource and metadata.dataset for the id "cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1". They all share the same uuid: resource_id==dataset_id.
>
> Cheers,
> Stephen.
>
> ---
> Stephen Pascoe +44 (0)1235 445980
> Centre of Environmental Data Archival
> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
>
> From: esg-gateway-dev-bounces at mailman.earthsystemgrid.org<mailto:esg-gateway-dev-bounces at mailman.earthsystemgrid.org> [mailto:esg-gateway-dev-bounces at mailman.earthsystemgrid.org] On Behalf Of Muhammad Atif
> Sent: 04 January 2012 02:40
> To: Karl Taylor
> Cc: Pascoe, Stephen (STFC,RAL,RALSP); go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>; esg-gateway-dev at earthsystemgrid.org<mailto:esg-gateway-dev at earthsystemgrid.org>
> Subject: Re: [esg-gateway-dev] [Go-essp-tech] publishing error (invalid hour)
>
> Thanks for the tips; however I am now facing the problem that I stated in the previous email.
> I wanted to pin point the file which was causing the issue and for that tried to publish the datasets one by one. Now I cannot even delete the said data.
>
>
> [root at esgnode1 debug-historicalAntNoAA]# esgunpublish --skip-thredds cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
> INFO 2012-01-04 13:39:49,576 Deleting cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
> WARNING 2012-01-04 13:39:49,687 Deletion/retraction failed for dataset cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1 with message: Java ServiceException: The persistent identifier=cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1 does not correspond to a dataset at sgf.gateway.service.security.impl.acegi.AcegiCatalogAccessDecisionVoter.vote(AcegiCatalogAccessDecisionVoter.java:74)
> INFO 2012-01-04 13:39:49,706 Result: UNSUCCESSFUL
>
> [root at esgnode1 debug-historicalAntNoAA]# esgunpublish --skip-gateway cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
> INFO 2012-01-04 13:40:35,331 Writing THREDDS ESG master catalog /data/esg-node/tds-content/thredds/esgcet/catalog.xml
> INFO 2012-01-04 13:40:35,719 Reinitializing THREDDS server
> Traceback (most recent call last):
> File "/usr/local/cdat/bin/esgunpublish", line 5, in<module>
> pkg_resources.run_script('esgcet==2.8.4', 'esgunpublish')
> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 489, in run_script
> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1207, in run_script
> File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgunpublish", line 221, in<module>
> main(sys.argv[1:])
> File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgunpublish", line 184, in main
> result = deleteDatasetList(datasetNames, Session, gatewayOp, thredds, las, deleteDset, deleteAll=deleteAll, republish=republish)
> File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/unpublish.py", line 272, in deleteDatasetList
> result = reinitializeThredds()
> File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/thredds.py", line 1041, in reinitializeThredds
> raise ESGPublishError("Error reinitializing the THREDDS Data Server: Fatal error: %s\n%s"%(errorMessage, str(result)))
> esgcet.exceptions.ESGPublishError: Error reinitializing the THREDDS Data Server: Fatal error: **Fatal: InvCatalogFactory.readXML failed
> Catalog init catalog.xml
> [2012-01-04T02:40:35GMT]
>
> [root at esgnode1 debug-historicalAntNoAA]# esgunpublish --database-delete cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
> INFO 2012-01-04 13:34:34,130 Deleting cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
> WARNING 2012-01-04 13:34:34,257 Deletion/retraction failed for dataset cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1 with message: Java ServiceException: The persistent identifier=cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1 does not correspond to a dataset at sgf.gateway.service.security.impl.acegi.AcegiCatalogAccessDecisionVoter.vote(AcegiCatalogAccessDecisionVoter.java:74)
> INFO 2012-01-04 13:34:34,274 Result: UNSUCCESSFUL
> INFO 2012-01-04 13:34:34,286 Writing THREDDS ESG master catalog /data/esg-node/tds-content/thredds/esgcet/catalog.xml
> INFO 2012-01-04 13:34:34,661 Reinitializing THREDDS server
> Traceback (most recent call last):
> File "/usr/local/cdat/bin/esgunpublish", line 5, in<module>
> pkg_resources.run_script('esgcet==2.8.4', 'esgunpublish')
> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 489, in run_script
> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1207, in run_script
> File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgunpublish", line 221, in<module>
> main(sys.argv[1:])
> File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgunpublish", line 184, in main
> result = deleteDatasetList(datasetNames, Session, gatewayOp, thredds, las, deleteDset, deleteAll=deleteAll, republish=republish)
> File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/unpublish.py", line 272, in deleteDatasetList
> result = reinitializeThredds()
> File "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/thredds.py", line 1041, in reinitializeThredds
> raise ESGPublishError("Error reinitializing the THREDDS Data Server: Fatal error: %s\n%s"%(errorMessage, str(result)))
> esgcet.exceptions.ESGPublishError: Error reinitializing the THREDDS Data Server: Fatal error: **Fatal: InvCatalogFactory.readXML failed
> Catalog init catalog.xml
> [2012-01-04T02:34:34GMT]
> readCatalog(): full path=/data/esg-node/tds-content/thredds/catalog.xml; path=catalog.xml
> readCatalog(): valid catalog -- ----Catalog Validation version 1.0.01
>
> [root at esgnode1 debug-historicalAntNoAA]# esglist_datasets cmip5 | grep CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
> | 4422 | cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1 | cmip5 | CSIRO-Mk3-6-0 | historicalAntNoAA | r10i1p1 | False | None | restricted | mon | ocean | Omon | r10i1p1 | CSIRO-QCCCE | 2012-01-04 13:39:49 | DELETE_DATASET_FAILED | cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0 | 20120104 | cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1.v20120104 | None |
>
>
> Thanks for your help in advance.
>
> Regards,
>
> On 04/01/12 04:41, Karl Taylor wrote:
> Hi Muhammad,
>
> Better yet, execute "ncdump -c". This will return both the header information and the coordinate values. It would be good to see what the time coordinate values are in the file.
>
> thanks,
> Karl
>
> On 1/3/12 6:34 AM, stephen.pascoe at stfc.ac.uk<mailto:stephen.pascoe at stfc.ac.uk> wrote:
>
> Hi Muhammad,
>
>
>
> Can you send a dump of the NetCDF metadata for this file. The command is "ncdump -h". Whether a time value is valid depends on the time:units attribute and sometimes other attributes.
>
>
>
> Cheers,
>
> Stephen.
>
>
>
> ---
>
> Stephen Pascoe +44 (0)1235 445980
>
> Centre of Environmental Data Archival
>
> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
>
>
>
>
>
> -----Original Message-----
>
> From: esg-gateway-dev-bounces at mailman.earthsystemgrid.org<mailto:esg-gateway-dev-bounces at mailman.earthsystemgrid.org> [mailto:esg-gateway-dev-bounces at mailman.earthsystemgrid.org] On Behalf Of Muhammad Atif
>
> Sent: 03 January 2012 04:37
>
> To: go-essp-tech at ucar.edu<mailto:go-essp-tech at ucar.edu>; esg-gateway-dev at earthsystemgrid.org<mailto:esg-gateway-dev at earthsystemgrid.org>
>
> Subject: [esg-gateway-dev] publishing error (invalid hour)
>
>
>
> Dear All,
>
>
>
> In an unrelated problem from the previous email, we are facing another
>
> error that gives "CDMS error: Error on time conversion: invalid hour =
>
> 28864363504146.773438"
>
> Please let me know what to do with it. The modelers suggest that they
>
> were not able to find any funny stuff.
>
>
>
> INFO 2011-12-28 12:24:32,436 Writing THREDDS catalog
>
> /data/esg-node/tds-content/thredds/esgcet/9/cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1.v20111228.xml
>
> CDMS error: Error on time conversion: invalid hour = 28864363504146.773438
>
>
>
> Traceback (most recent call last):
>
> File "/usr/local/cdat/bin/esgpublish", line 5, in<module>
>
> pkg_resources.run_script('esgcet==2.8.4', 'esgpublish')
>
> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 489, in
>
> run_script
>
> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1207, in
>
> run_script
>
> File
>
> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgpublish",
>
> line 434, in<module>
>
> main(sys.argv[1:])
>
> File
>
> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgpublish",
>
> line 420, in main
>
> result = publishDatasetList(datasetNames, Session, publish=publish,
>
> thredds=thredds, las=las, parentId=parent, service=service,
>
> perVariable=perVariable)
>
> File
>
> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/publish.py",
>
> line 244, in publishDatasetList
>
> generateThredds(datasetName, Session, threddsOutput, handler,
>
> service=service, perVariable=perVariable, versionNumber=versionno)
>
> File
>
> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/thredds.py",
>
> line 726, in generateThredds
>
> _generateThreddsV2(datasetName, outputFile, handler, session, dset,
>
> context, project, model, experiment, config, section, genRoot=genRoot,
>
> service=service, perVariable=perVariable, versionNumber=versionNumber)
>
> File
>
> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/thredds.py",
>
> line 950, in _generateThreddsV2
>
> _genPerVariableDatasetsV2(datasetElem, dset, datasetName,
>
> resolution, filesRootLoc, filesRootPath, datasetRootDict,
>
> excludeVariables, offline, serviceName, serviceDict, aggServiceName,
>
> handler, project, model, experiment, lasConfigure, lasTimeDelta,
>
> versionNumber, variables, variableElemDict, lasServiceSpecs,
>
> lasServiceHash, gridftpMap=gridftpMapDatasetRoots)
>
> File
>
> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/thredds.py",
>
> line 563, in _genPerVariableDatasetsV2
>
> timeFirst = mdhandler.genTime(variable.aggdim_first,
>
> dataset.aggdim_units, dataset.calendar)
>
> File
>
> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/config/cf_handler.py",
>
> line 179, in genTime
>
> c = t.tocomp(_tagToCalendar[calendarTag])
>
> Cdtime error: Invalid component time
>
>
>
> Regards,
>
>
>
>
>
>
> --
>
> Muhammad Atif
>
> ANU Supercomputer Facility NCI National Facility
>
> Leonard Huxley, Mills Road Room 318, Bldg # 56
>
> Australian National University Ph: +61 2 6125 5031
>
> Canberra, ACT 0200, Australia M: 0430 393863
>
> CRICOS Provider #00120C http://anusf.anu.edu.au
>
>
> --
> Scanned by iCritical.
>
>
>
>
> --
> Muhammad Atif
> ANU Supercomputer Facility NCI National Facility
> Leonard Huxley, Mills Road Room 318, Bldg # 56
> Australian National University Ph: +61 2 6125 5031
> Canberra, ACT 0200, Australia M: 0430 393863
> CRICOS Provider #00120C http://anusf.anu.edu.au
>
--
Muhammad Atif
ANU Supercomputer Facility NCI National Facility
Leonard Huxley, Mills Road Room 318, Bldg # 56
Australian National University Ph: +61 2 6125 5031
Canberra, ACT 0200, Australia M: 0430 393863
CRICOS Provider #00120C http://anusf.anu.edu.au
More information about the GO-ESSP-TECH
mailing list