[Go-essp-tech] [esg-gateway-dev] publishing error (invalid hour)
Muhammad Atif
muhammad.atif at anu.edu.au
Thu Jan 5 22:06:22 MST 2012
Thanks for that Stephen,
Will try to look in deeper next week. Just for the heck of it, is there
a nice recommended way to remove everything from the database related to
a model/experiment? A clean start?
Regards,
On 04/01/12 20:18, stephen.pascoe at stfc.ac.uk wrote:
>
> Muhammad,
>
> These problems are difficult to diagnose. These errors suggest an
> inconsistency problem with your Gateway database and/or THREDDS
> catalogs. I suspect you are missing a root THREDDS catalog in the TDS
> contents directory or it is somehow unreadable. Check what the TDS
> looks like in your browser. Can you view the top-level catalog as
> XML? Is the XML valid?
>
> I'm also concerned that your Gateway database may be inconsistent.
> Check the the tables metadata.persistent_identifier, metadata.resource
> and metadata.dataset for the id
> "cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1".
> They all share the same uuid: resource_id==dataset_id.
>
> Cheers,
>
> Stephen.
>
> ---
>
> Stephen Pascoe +44 (0)1235 445980
>
> Centre of Environmental Data Archival
>
> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
>
> *From:*esg-gateway-dev-bounces at mailman.earthsystemgrid.org
> [mailto:esg-gateway-dev-bounces at mailman.earthsystemgrid.org] *On
> Behalf Of * Muhammad Atif
> *Sent:* 04 January 2012 02:40
> *To:* Karl Taylor
> *Cc:* Pascoe, Stephen (STFC,RAL,RALSP); go-essp-tech at ucar.edu;
> esg-gateway-dev at earthsystemgrid.org
> *Subject:* Re: [esg-gateway-dev] [Go-essp-tech] publishing error
> (invalid hour)
>
> Thanks for the tips; however I am now facing the problem that I stated
> in the previous email.
> I wanted to pin point the file which was causing the issue and for
> that tried to publish the datasets one by one. Now I cannot even
> delete the said data.
>
>
> *[root at esgnode1 debug-historicalAntNoAA]# esgunpublish --skip-thredds
> cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1*
> INFO 2012-01-04 13:39:49,576 Deleting
> cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
> WARNING 2012-01-04 13:39:49,687 Deletion/retraction failed for
> dataset
> cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
> with message: Java ServiceException: The persistent
> identifier=cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
> does not correspond to a dataset at
> sgf.gateway.service.security.impl.acegi.AcegiCatalogAccessDecisionVoter.vote(AcegiCatalogAccessDecisionVoter.java:74)
> INFO 2012-01-04 13:39:49,706 Result: UNSUCCESSFUL
>
> *[root at esgnode1 debug-historicalAntNoAA]# esgunpublish --skip-gateway
> cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1*
> INFO 2012-01-04 13:40:35,331 Writing THREDDS ESG master catalog
> /data/esg-node/tds-content/thredds/esgcet/catalog.xml
> INFO 2012-01-04 13:40:35,719 Reinitializing THREDDS server
> Traceback (most recent call last):
> File "/usr/local/cdat/bin/esgunpublish", line 5, in <module>
> pkg_resources.run_script('esgcet==2.8.4', 'esgunpublish')
> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 489, in
> run_script
> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1207, in
> run_script
> File
> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgunpublish",
> line 221, in <module>
> main(sys.argv[1:])
> File
> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgunpublish",
> line 184, in main
> result = deleteDatasetList(datasetNames, Session, gatewayOp,
> thredds, las, deleteDset, deleteAll=deleteAll, republish=republish)
> File
> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/unpublish.py",
> line 272, in deleteDatasetList
> result = reinitializeThredds()
> File
> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/thredds.py",
> line 1041, in reinitializeThredds
> raise ESGPublishError("Error reinitializing the THREDDS Data
> Server: Fatal error: %s\n%s"%(errorMessage, str(result)))
> esgcet.exceptions.ESGPublishError: Error reinitializing the THREDDS
> Data Server: Fatal error: **Fatal: InvCatalogFactory.readXML failed
> Catalog init catalog.xml
> [2012-01-04T02:40:35GMT]
>
> *[root at esgnode1 debug-historicalAntNoAA]# esgunpublish
> --database-delete
> cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1*
> INFO 2012-01-04 13:34:34,130 Deleting
> cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
> WARNING 2012-01-04 13:34:34,257 Deletion/retraction failed for
> dataset
> cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
> with message: Java ServiceException: The persistent
> identifier=cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
> does not correspond to a dataset at
> sgf.gateway.service.security.impl.acegi.AcegiCatalogAccessDecisionVoter.vote(AcegiCatalogAccessDecisionVoter.java:74)
> INFO 2012-01-04 13:34:34,274 Result: UNSUCCESSFUL
> INFO 2012-01-04 13:34:34,286 Writing THREDDS ESG master catalog
> /data/esg-node/tds-content/thredds/esgcet/catalog.xml
> INFO 2012-01-04 13:34:34,661 Reinitializing THREDDS server
> Traceback (most recent call last):
> File "/usr/local/cdat/bin/esgunpublish", line 5, in <module>
> pkg_resources.run_script('esgcet==2.8.4', 'esgunpublish')
> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 489, in
> run_script
> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1207, in
> run_script
> File
> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgunpublish",
> line 221, in <module>
> main(sys.argv[1:])
> File
> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgunpublish",
> line 184, in main
> result = deleteDatasetList(datasetNames, Session, gatewayOp,
> thredds, las, deleteDset, deleteAll=deleteAll, republish=republish)
> File
> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/unpublish.py",
> line 272, in deleteDatasetList
> result = reinitializeThredds()
> File
> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/thredds.py",
> line 1041, in reinitializeThredds
> raise ESGPublishError("Error reinitializing the THREDDS Data
> Server: Fatal error: %s\n%s"%(errorMessage, str(result)))
> esgcet.exceptions.ESGPublishError: Error reinitializing the THREDDS
> Data Server: Fatal error: **Fatal: InvCatalogFactory.readXML failed
> Catalog init catalog.xml
> [2012-01-04T02:34:34GMT]
> readCatalog(): full
> path=/data/esg-node/tds-content/thredds/catalog.xml; path=catalog.xml
> readCatalog(): valid catalog -- ----Catalog Validation version 1.0.01
>
> *[root at esgnode1 debug-historicalAntNoAA]# esglist_datasets cmip5 |
> grep CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1*
> | 4422 |
> cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1
> | cmip5 | CSIRO-Mk3-6-0 | historicalAntNoAA | r10i1p1 | False |
> None | restricted | mon | ocean | Omon |
> r10i1p1 | CSIRO-QCCCE | 2012-01-04 13:39:49 | DELETE_DATASET_FAILED |
> cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0 | 20120104 |
> cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1.v20120104
> | None |
>
>
> Thanks for your help in advance.
>
> Regards,
>
> On 04/01/12 04:41, Karl Taylor wrote:
>
> Hi Muhammad,
>
> Better yet, execute "ncdump -c". This will return both the header
> information and the coordinate values. It would be good to see what
> the time coordinate values are in the file.
>
> thanks,
> Karl
>
> On 1/3/12 6:34 AM, stephen.pascoe at stfc.ac.uk
> <mailto:stephen.pascoe at stfc.ac.uk> wrote:
>
> Hi Muhammad,
>
> Can you send a dump of the NetCDF metadata for this file. The command is "ncdump -h". Whether a time value is valid depends on the time:units attribute and sometimes other attributes.
>
> Cheers,
> Stephen.
>
> ---
> Stephen Pascoe +44 (0)1235 445980
> Centre of Environmental Data Archival
> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot OX11 0QX, UK
>
>
> -----Original Message-----
> From:esg-gateway-dev-bounces at mailman.earthsystemgrid.org <mailto:esg-gateway-dev-bounces at mailman.earthsystemgrid.org> [mailto:esg-gateway-dev-bounces at mailman.earthsystemgrid.org] On Behalf Of Muhammad Atif
> Sent: 03 January 2012 04:37
> To:go-essp-tech at ucar.edu <mailto:go-essp-tech at ucar.edu>;esg-gateway-dev at earthsystemgrid.org <mailto:esg-gateway-dev at earthsystemgrid.org>
> Subject: [esg-gateway-dev] publishing error (invalid hour)
>
> Dear All,
>
> In an unrelated problem from the previous email, we are facing another
> error that gives "CDMS error: Error on time conversion: invalid hour =
> 28864363504146.773438"
> Please let me know what to do with it. The modelers suggest that they
> were not able to find any funny stuff.
>
> INFO 2011-12-28 12:24:32,436 Writing THREDDS catalog
> /data/esg-node/tds-content/thredds/esgcet/9/cmip5.restricted.CSIRO-QCCCE.CSIRO-Mk3-6-0.historicalAntNoAA.mon.ocean.Omon.r10i1p1.v20111228.xml
> CDMS error: Error on time conversion: invalid hour = 28864363504146.773438
>
> Traceback (most recent call last):
> File "/usr/local/cdat/bin/esgpublish", line 5, in<module>
> pkg_resources.run_script('esgcet==2.8.4', 'esgpublish')
> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 489, in
> run_script
> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1207, in
> run_script
> File
> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgpublish",
> line 434, in<module>
> main(sys.argv[1:])
> File
> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/EGG-INFO/scripts/esgpublish",
> line 420, in main
> result = publishDatasetList(datasetNames, Session, publish=publish,
> thredds=thredds, las=las, parentId=parent, service=service,
> perVariable=perVariable)
> File
> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/publish.py",
> line 244, in publishDatasetList
> generateThredds(datasetName, Session, threddsOutput, handler,
> service=service, perVariable=perVariable, versionNumber=versionno)
> File
> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/thredds.py",
> line 726, in generateThredds
> _generateThreddsV2(datasetName, outputFile, handler, session, dset,
> context, project, model, experiment, config, section, genRoot=genRoot,
> service=service, perVariable=perVariable, versionNumber=versionNumber)
> File
> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/thredds.py",
> line 950, in _generateThreddsV2
> _genPerVariableDatasetsV2(datasetElem, dset, datasetName,
> resolution, filesRootLoc, filesRootPath, datasetRootDict,
> excludeVariables, offline, serviceName, serviceDict, aggServiceName,
> handler, project, model, experiment, lasConfigure, lasTimeDelta,
> versionNumber, variables, variableElemDict, lasServiceSpecs,
> lasServiceHash, gridftpMap=gridftpMapDatasetRoots)
> File
> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/publish/thredds.py",
> line 563, in _genPerVariableDatasetsV2
> timeFirst = mdhandler.genTime(variable.aggdim_first,
> dataset.aggdim_units, dataset.calendar)
> File
> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.8.4-py2.6.egg/esgcet/config/cf_handler.py",
> line 179, in genTime
> c = t.tocomp(_tagToCalendar[calendarTag])
> Cdtime error: Invalid component time
>
> Regards,
>
>
>
>
>
> --
> Muhammad Atif
> ANU Supercomputer Facility NCI National Facility
> Leonard Huxley, Mills Road Room 318, Bldg # 56
> Australian National University Ph: +61 2 6125 5031
> Canberra, ACT 0200, Australia M: 0430 393863
> CRICOS Provider #00120Chttp://anusf.anu.edu.au
>
> --
> Scanned by iCritical.
>
>
--
Muhammad Atif
ANU Supercomputer Facility NCI National Facility
Leonard Huxley, Mills Road Room 318, Bldg # 56
Australian National University Ph: +61 2 6125 5031
Canberra, ACT 0200, Australia M: 0430 393863
CRICOS Provider #00120C http://anusf.anu.edu.au
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ucar.edu/pipermail/go-essp-tech/attachments/20120106/933a78c4/attachment-0001.html
More information about the GO-ESSP-TECH
mailing list