[Go-essp-tech] GO-ESSP-TECH Digest, Vol 36, Issue 63
Mark Elkington
mark.elkington at blueyonder.co.uk
Fri Feb 24 13:57:50 MST 2012
Phew - just completed all of the metadata entries for the Hadley Centre. Not a task I want to have
to repeat any time soon (actually ever ;-)).
Anyway the next step is to review the content of what has been entered through the gateway and to
get the 'owning' scientists to sign it off (i.e. did I interpret their input correctly).
The current Gateway software does not display all of the content entered into the questionnaire and
has problems displaying other parts of it. My understanding is that neither of these issues are
addressed completely in Gateway2 or P2P Gateway.
Can I someone give me the timeline when the metadata will be correctly and completely represented in
the gateway presentation so that I can schedule in the work we still need to do - approximate dates
are fine.
Regards
Mark
--------------------------------------------------------------
Dr Mark Elkington
Elmswood Barn, Ogwell, Newton Abbot, Devon TQ12 6AF
tel: 01626 212545 mob: 07804 795826
email: mark.elkington at blueyonder.co.uk
-----Original Message-----
From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of
go-essp-tech-request at ucar.edu
Sent: 24 February 2012 17:06
To: go-essp-tech at ucar.edu
Subject: GO-ESSP-TECH Digest, Vol 36, Issue 63
Send GO-ESSP-TECH mailing list submissions to
go-essp-tech at ucar.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
or, via email, send a message with subject or body 'help' to
go-essp-tech-request at ucar.edu
You can reach the person managing the list at
go-essp-tech-owner at ucar.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of GO-ESSP-TECH digest..."
Today's Topics:
1. Re: Corrected THREDDS group wiki page link (Bryan Lawrence)
2. Re: Error reinitializing THREDDS (Serguei Nikonov)
3. Re: Error reinitializing THREDDS (Serguei Nikonov)
4. Re: Error reinitializing THREDDS (Estanislao Gonzalez)
5. Re: Error reinitializing THREDDS (Serguei Nikonov)
----------------------------------------------------------------------
Message: 1
Date: Fri, 24 Feb 2012 14:21:33 +0000
From: Bryan Lawrence <bryan.lawrence at ncas.ac.uk>
Subject: Re: [Go-essp-tech] Corrected THREDDS group wiki page link
To: go-essp-tech at ucar.edu
Cc: Luca.Cinquini at jpl.nasa.gov
Message-ID: <201202241421.34037.bryan.lawrence at ncas.ac.uk>
Content-Type: Text/Plain; charset="us-ascii"
> I think ISO metadata is at a much higher level like who owns and maintains the dataset. We've had
lots of experience with ISO and I think it's a completely different activity to consider expressing
ESGF metadata in ISO.
and it's been done in Metafor :-)
B
--
Bryan Lawrence
University of Reading: Professor of Weather and Climate Computing.
National Centre for Atmospheric Science: Director of Models and Data.
STFC: Director of the Centre for Environmental Data Archival.
Ph: +44 118 3786507 or 1235 445012; Web:home.badc.rl.ac.uk/lawrence
------------------------------
Message: 2
Date: Fri, 24 Feb 2012 10:13:23 -0500
From: Serguei Nikonov <serguei.nikonov at noaa.gov>
Subject: Re: [Go-essp-tech] Error reinitializing THREDDS
To: "Drach, Bob" <drach1 at llnl.gov>
Cc: "go-essp-tech at ucar.edu" <go-essp-tech at ucar.edu>, Hans Vahlenkamp
<hans.vahlenkamp at noaa.gov>
Message-ID: <4F47A913.8020708 at noaa.gov>
Content-Type: text/plain; charset=UTF-8; format=flowed
Thanks Bob for the help. It happened that 2 datasets xmls were empty. Very
strange thing. I am 100% confident that I didn't touch anything there. I removed
them and corresponding references in catalog.xml, unpublished them and it
started working.
Regards,
Sergey
On 02/23/2012 02:10 PM, Drach, Bob wrote:
> Hi Sergey,
>
> Look in the TDS catalogInit.log. On our system it's in
> /esg/content/thredds/logs. Check for errors in the most recent
> initialization. If you can spot the dataset(s) that caused the
> initialization error, just move those catalogs aside then rerun the
> esgunpublish. If that succeeds, then fix the 'broken' catalogs/datasets
> and regenerate them.
>
> --Bob
>
> On 2/23/12 9:48 AM, "Serguei Nikonov"<serguei.nikonov at noaa.gov> wrote:
>
>> Hi Bob,
>>
>> I have constantly this error when try to publish/unpublish any data:
>>
>> ERROR 2012-02-23 12:36:21,046 Error reading url
>> https://localhost:443/thredds/admin/debug?catalogs/reinit: HTTPError()
>> Traceback (most recent call last):
>> File "/usr/local/cdat/bin/esgunpublish", line 5, in<module>
>> pkg_resources.run_script('esgcet==2.9.0', 'esgunpublish')
>> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 489, in
>> run_script
>> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1207, in
>> run_script
>> File
>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/EGG-IN
>> FO/scripts/esgunpublish",
>> line 221, in<module>
>> main(sys.argv[1:])
>> File
>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/EGG-IN
>> FO/scripts/esgunpublish",
>> line 184, in main
>> result = deleteDatasetList(datasetNames, Session, gatewayOp,
>> thredds, las,
>> deleteDset, deleteAll=deleteAll, republish=republish)
>> File
>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet
>> /publish/unpublish.py",
>> line 272, in deleteDatasetList
>> result = reinitializeThredds()
>> File
>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet
>> /publish/thredds.py",
>> line 1030, in reinitializeThredds
>> raise ESGPublishError("Error reinitializing the THREDDS Data Server:
>> %s"%msg)
>> esgcet.exceptions.ESGPublishError: Error reinitializing the THREDDS Data
>> Server:
>> HTTPError()
>>
>> Do you have any idea what is it?
>>
>> Thanks,
>> Sergey
>>
>>
>>
>> On 02/23/2012 12:35 AM, Hans Vahlenkamp wrote:
>>> Recall the discussion about how to determine the version number of the
>>> installed ESGF software. I just realized that it is listed at the
>>> bottom
>>> of the home page:
>>>
>>> http://esgdata.gfdl.noaa.gov/esgf-node-manager/
>>>
>>> We have "v0.5.6 - bay_ridge release".
>>>
>>> Hans
>>>
>>
>> _______________________________________________
>> GO-ESSP-TECH mailing list
>> GO-ESSP-TECH at ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
>
------------------------------
Message: 3
Date: Fri, 24 Feb 2012 10:28:18 -0500
From: Serguei Nikonov <serguei.nikonov at noaa.gov>
Subject: Re: [Go-essp-tech] Error reinitializing THREDDS
To: Estanislao Gonzalez <gonzalez at dkrz.de>
Cc: "Drach, Bob" <drach1 at llnl.gov>, "go-essp-tech at ucar.edu"
<go-essp-tech at ucar.edu>, Hans Vahlenkamp <hans.vahlenkamp at noaa.gov>
Message-ID: <4F47AC92.2060303 at noaa.gov>
Content-Type: text/plain; charset=UTF-8; format=flowed
Hi Estani,
have to admit that we regularly experience problem with too many open files
error and lack of memory. We even increased yesterday memory allocation to 20G
(from 16G) and open files limit to 8000. But this was another case, some
datasets xmls happened to be empty (?!).
Thanks,
Sergey
On 02/24/2012 08:25 AM, Estanislao Gonzalez wrote:
> Hi Sergei, Hans,
>
> This also happens if your TDS crashed because of the already mentioned problems
> of memory or open files.
>
> Try restarting the node again. If it doesn't work immediately thereafter, then
> is indeed something different. But I've seen this happening and it's kind of
> frustrating to recheck everything for the n-th time, just to find out that the
> node was under a lot of stress...
>
> Thanks,
> Estani
>
> Am 23.02.2012 20:10, schrieb Drach, Bob:
>> Hi Sergey,
>>
>> Look in the TDS catalogInit.log. On our system it's in
>> /esg/content/thredds/logs. Check for errors in the most recent
>> initialization. If you can spot the dataset(s) that caused the
>> initialization error, just move those catalogs aside then rerun the
>> esgunpublish. If that succeeds, then fix the 'broken' catalogs/datasets
>> and regenerate them.
>>
>> --Bob
>>
>> On 2/23/12 9:48 AM, "Serguei Nikonov"<serguei.nikonov at noaa.gov> wrote:
>>
>>> Hi Bob,
>>>
>>> I have constantly this error when try to publish/unpublish any data:
>>>
>>> ERROR 2012-02-23 12:36:21,046 Error reading url
>>> https://localhost:443/thredds/admin/debug?catalogs/reinit: HTTPError()
>>> Traceback (most recent call last):
>>> File "/usr/local/cdat/bin/esgunpublish", line 5, in<module>
>>> pkg_resources.run_script('esgcet==2.9.0', 'esgunpublish')
>>> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 489, in
>>> run_script
>>> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1207, in
>>> run_script
>>> File
>>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/EGG-IN
>>> FO/scripts/esgunpublish",
>>> line 221, in<module>
>>> main(sys.argv[1:])
>>> File
>>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/EGG-IN
>>> FO/scripts/esgunpublish",
>>> line 184, in main
>>> result = deleteDatasetList(datasetNames, Session, gatewayOp,
>>> thredds, las,
>>> deleteDset, deleteAll=deleteAll, republish=republish)
>>> File
>>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet
>>> /publish/unpublish.py",
>>> line 272, in deleteDatasetList
>>> result = reinitializeThredds()
>>> File
>>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet
>>> /publish/thredds.py",
>>> line 1030, in reinitializeThredds
>>> raise ESGPublishError("Error reinitializing the THREDDS Data Server:
>>> %s"%msg)
>>> esgcet.exceptions.ESGPublishError: Error reinitializing the THREDDS Data
>>> Server:
>>> HTTPError()
>>>
>>> Do you have any idea what is it?
>>>
>>> Thanks,
>>> Sergey
>>>
>>>
>>>
>>> On 02/23/2012 12:35 AM, Hans Vahlenkamp wrote:
>>>> Recall the discussion about how to determine the version number of the
>>>> installed ESGF software. I just realized that it is listed at the
>>>> bottom
>>>> of the home page:
>>>>
>>>> http://esgdata.gfdl.noaa.gov/esgf-node-manager/
>>>>
>>>> We have "v0.5.6 - bay_ridge release".
>>>>
>>>> Hans
>>>>
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>> _______________________________________________
>> GO-ESSP-TECH mailing list
>> GO-ESSP-TECH at ucar.edu
>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>
>
------------------------------
Message: 4
Date: Fri, 24 Feb 2012 16:33:50 +0100
From: Estanislao Gonzalez <gonzalez at dkrz.de>
Subject: Re: [Go-essp-tech] Error reinitializing THREDDS
To: Serguei Nikonov <serguei.nikonov at noaa.gov>
Cc: "Drach, Bob" <drach1 at llnl.gov>, "go-essp-tech at ucar.edu"
<go-essp-tech at ucar.edu>, Hans Vahlenkamp <hans.vahlenkamp at noaa.gov>
Message-ID: <4F47ADDE.2090006 at dkrz.de>
Content-Type: text/plain; charset=UTF-8; format=flowed
Hi,
yes I recall seeing them... but it's hard to reproduce, so I'm not sure
under which circumstances this happens... perhaps aborting the
publication leaves those empty files behind?
Thanks,
Estani
Am 24.02.2012 16:28, schrieb Serguei Nikonov:
> Hi Estani,
>
> have to admit that we regularly experience problem with too many open
> files error and lack of memory. We even increased yesterday memory
> allocation to 20G (from 16G) and open files limit to 8000. But this
> was another case, some datasets xmls happened to be empty (?!).
>
> Thanks,
> Sergey
>
> On 02/24/2012 08:25 AM, Estanislao Gonzalez wrote:
>> Hi Sergei, Hans,
>>
>> This also happens if your TDS crashed because of the already
>> mentioned problems
>> of memory or open files.
>>
>> Try restarting the node again. If it doesn't work immediately
>> thereafter, then
>> is indeed something different. But I've seen this happening and it's
>> kind of
>> frustrating to recheck everything for the n-th time, just to find out
>> that the
>> node was under a lot of stress...
>>
>> Thanks,
>> Estani
>>
>> Am 23.02.2012 20:10, schrieb Drach, Bob:
>>> Hi Sergey,
>>>
>>> Look in the TDS catalogInit.log. On our system it's in
>>> /esg/content/thredds/logs. Check for errors in the most recent
>>> initialization. If you can spot the dataset(s) that caused the
>>> initialization error, just move those catalogs aside then rerun the
>>> esgunpublish. If that succeeds, then fix the 'broken' catalogs/datasets
>>> and regenerate them.
>>>
>>> --Bob
>>>
>>> On 2/23/12 9:48 AM, "Serguei Nikonov"<serguei.nikonov at noaa.gov> wrote:
>>>
>>>> Hi Bob,
>>>>
>>>> I have constantly this error when try to publish/unpublish any data:
>>>>
>>>> ERROR 2012-02-23 12:36:21,046 Error reading url
>>>> https://localhost:443/thredds/admin/debug?catalogs/reinit: HTTPError()
>>>> Traceback (most recent call last):
>>>> File "/usr/local/cdat/bin/esgunpublish", line 5, in<module>
>>>> pkg_resources.run_script('esgcet==2.9.0', 'esgunpublish')
>>>> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 489, in
>>>> run_script
>>>> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1207, in
>>>> run_script
>>>> File
>>>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/EGG-IN
>>>>
>>>> FO/scripts/esgunpublish",
>>>> line 221, in<module>
>>>> main(sys.argv[1:])
>>>> File
>>>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/EGG-IN
>>>>
>>>> FO/scripts/esgunpublish",
>>>> line 184, in main
>>>> result = deleteDatasetList(datasetNames, Session, gatewayOp,
>>>> thredds, las,
>>>> deleteDset, deleteAll=deleteAll, republish=republish)
>>>> File
>>>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet
>>>>
>>>> /publish/unpublish.py",
>>>> line 272, in deleteDatasetList
>>>> result = reinitializeThredds()
>>>> File
>>>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet
>>>>
>>>> /publish/thredds.py",
>>>> line 1030, in reinitializeThredds
>>>> raise ESGPublishError("Error reinitializing the THREDDS Data Server:
>>>> %s"%msg)
>>>> esgcet.exceptions.ESGPublishError: Error reinitializing the THREDDS
>>>> Data
>>>> Server:
>>>> HTTPError()
>>>>
>>>> Do you have any idea what is it?
>>>>
>>>> Thanks,
>>>> Sergey
>>>>
>>>>
>>>>
>>>> On 02/23/2012 12:35 AM, Hans Vahlenkamp wrote:
>>>>> Recall the discussion about how to determine the version number of
>>>>> the
>>>>> installed ESGF software. I just realized that it is listed at the
>>>>> bottom
>>>>> of the home page:
>>>>>
>>>>> http://esgdata.gfdl.noaa.gov/esgf-node-manager/
>>>>>
>>>>> We have "v0.5.6 - bay_ridge release".
>>>>>
>>>>> Hans
>>>>>
>>>> _______________________________________________
>>>> GO-ESSP-TECH mailing list
>>>> GO-ESSP-TECH at ucar.edu
>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>
>>
>
--
Estanislao Gonzalez
Max-Planck-Institut f?r Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
Phone: +49 (40) 46 00 94-126
E-Mail: gonzalez at dkrz.de
------------------------------
Message: 5
Date: Fri, 24 Feb 2012 12:05:48 -0500
From: Serguei Nikonov <serguei.nikonov at noaa.gov>
Subject: Re: [Go-essp-tech] Error reinitializing THREDDS
To: Estanislao Gonzalez <gonzalez at dkrz.de>
Cc: "Drach, Bob" <drach1 at llnl.gov>, "go-essp-tech at ucar.edu"
<go-essp-tech at ucar.edu>, Hans Vahlenkamp <hans.vahlenkamp at noaa.gov>
Message-ID: <4F47C36C.9060005 at noaa.gov>
Content-Type: text/plain; charset=UTF-8; format=flowed
Those 2 datasets were published long ago.
On 02/24/2012 10:33 AM, Estanislao Gonzalez wrote:
> Hi,
>
> yes I recall seeing them... but it's hard to reproduce, so I'm not sure under
> which circumstances this happens... perhaps aborting the publication leaves
> those empty files behind?
>
> Thanks,
> Estani
>
>
> Am 24.02.2012 16:28, schrieb Serguei Nikonov:
>> Hi Estani,
>>
>> have to admit that we regularly experience problem with too many open files
>> error and lack of memory. We even increased yesterday memory allocation to 20G
>> (from 16G) and open files limit to 8000. But this was another case, some
>> datasets xmls happened to be empty (?!).
>>
>> Thanks,
>> Sergey
>>
>> On 02/24/2012 08:25 AM, Estanislao Gonzalez wrote:
>>> Hi Sergei, Hans,
>>>
>>> This also happens if your TDS crashed because of the already mentioned problems
>>> of memory or open files.
>>>
>>> Try restarting the node again. If it doesn't work immediately thereafter, then
>>> is indeed something different. But I've seen this happening and it's kind of
>>> frustrating to recheck everything for the n-th time, just to find out that the
>>> node was under a lot of stress...
>>>
>>> Thanks,
>>> Estani
>>>
>>> Am 23.02.2012 20:10, schrieb Drach, Bob:
>>>> Hi Sergey,
>>>>
>>>> Look in the TDS catalogInit.log. On our system it's in
>>>> /esg/content/thredds/logs. Check for errors in the most recent
>>>> initialization. If you can spot the dataset(s) that caused the
>>>> initialization error, just move those catalogs aside then rerun the
>>>> esgunpublish. If that succeeds, then fix the 'broken' catalogs/datasets
>>>> and regenerate them.
>>>>
>>>> --Bob
>>>>
>>>> On 2/23/12 9:48 AM, "Serguei Nikonov"<serguei.nikonov at noaa.gov> wrote:
>>>>
>>>>> Hi Bob,
>>>>>
>>>>> I have constantly this error when try to publish/unpublish any data:
>>>>>
>>>>> ERROR 2012-02-23 12:36:21,046 Error reading url
>>>>> https://localhost:443/thredds/admin/debug?catalogs/reinit: HTTPError()
>>>>> Traceback (most recent call last):
>>>>> File "/usr/local/cdat/bin/esgunpublish", line 5, in<module>
>>>>> pkg_resources.run_script('esgcet==2.9.0', 'esgunpublish')
>>>>> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 489, in
>>>>> run_script
>>>>> File "build/bdist.linux-x86_64/egg/pkg_resources.py", line 1207, in
>>>>> run_script
>>>>> File
>>>>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/EGG-IN
>>>>> FO/scripts/esgunpublish",
>>>>> line 221, in<module>
>>>>> main(sys.argv[1:])
>>>>> File
>>>>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/EGG-IN
>>>>> FO/scripts/esgunpublish",
>>>>> line 184, in main
>>>>> result = deleteDatasetList(datasetNames, Session, gatewayOp,
>>>>> thredds, las,
>>>>> deleteDset, deleteAll=deleteAll, republish=republish)
>>>>> File
>>>>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet
>>>>> /publish/unpublish.py",
>>>>> line 272, in deleteDatasetList
>>>>> result = reinitializeThredds()
>>>>> File
>>>>> "/usr/local/cdat/lib/python2.6/site-packages/esgcet-2.9.0-py2.6.egg/esgcet
>>>>> /publish/thredds.py",
>>>>> line 1030, in reinitializeThredds
>>>>> raise ESGPublishError("Error reinitializing the THREDDS Data Server:
>>>>> %s"%msg)
>>>>> esgcet.exceptions.ESGPublishError: Error reinitializing the THREDDS Data
>>>>> Server:
>>>>> HTTPError()
>>>>>
>>>>> Do you have any idea what is it?
>>>>>
>>>>> Thanks,
>>>>> Sergey
>>>>>
>>>>>
>>>>>
>>>>> On 02/23/2012 12:35 AM, Hans Vahlenkamp wrote:
>>>>>> Recall the discussion about how to determine the version number of the
>>>>>> installed ESGF software. I just realized that it is listed at the
>>>>>> bottom
>>>>>> of the home page:
>>>>>>
>>>>>> http://esgdata.gfdl.noaa.gov/esgf-node-manager/
>>>>>>
>>>>>> We have "v0.5.6 - bay_ridge release".
>>>>>>
>>>>>> Hans
>>>>>>
>>>>> _______________________________________________
>>>>> GO-ESSP-TECH mailing list
>>>>> GO-ESSP-TECH at ucar.edu
>>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>> _______________________________________________
>>>> GO-ESSP-TECH mailing list
>>>> GO-ESSP-TECH at ucar.edu
>>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>>>
>>>
>>
>
>
------------------------------
_______________________________________________
GO-ESSP-TECH mailing list
GO-ESSP-TECH at ucar.edu
http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
End of GO-ESSP-TECH Digest, Vol 36, Issue 63
********************************************
More information about the GO-ESSP-TECH
mailing list