[Go-essp-tech] Finding new datasets? Was RE: [esg-gateway-dev] Catalog Browsing broken at WDCC

[Kettleborough, Jamie]

Hello Estani,

Thanks for following this problem up.

This is related to the thread on goessp-tech on 'risks that science is being done with deprecated data'.  The reason the MOHC user (one of our scientists) is going to the catalogue is to try and find if any new datasets have been published - either completely new, or new versions of datasets he already has.  I don't know if there is a better way for a user (or even a replication system) to answer this question - I think it's the 'notification' problem isn't it?  I think the scientists basic question is: 'is there any new Amon tas data from any model, any ensemble member, for any piControl run, any historical type run, or any  rcp run?'.  My guess is that kind of question is very common.

Jamie 

> -----Original Message-----
> From: esg-gateway-dev-bounces at mailman.earthsystemgrid.org 
> [mailto:esg-gateway-dev-bounces at mailman.earthsystemgrid.org] 
> On Behalf Of Estanislao Gonzalez
> Sent: 14 March 2012 11:23
> To: stephen.pascoe at stfc.ac.uk
> Cc: esg-gateway-dev at earthsystemgrid.org
> Subject: Re: [esg-gateway-dev] Catalog Browsing broken at WDCC
> 
> Hi Stephen,
> 
> the thing is that I only have NCC replicas (~400) and those 
> are already under a different top level collection...
> According to the CMIP5 view we have ~3100 datasets.... hmmm 
> it's a little more than you have though... but I can't even 
> display the replicas... although... again, I do have +3000 
> replicas there thought they are marked as "retracted", 
> perhaps that's why they are not working...
> 
> Should I completely remove them? I was afraid I was going to 
> loose the metrics, but I guess I already did, since there 
> seem to be no metrics stored at all at the Gateway...
> 
> This was a request from someone at MOHC that is apparently 
> using the catalogs.
> I tried listing MRI/MIROC top level collection, this are much 
> more that what we have... I'll leave this running for a while 
> and see if something pops up... but I guess this page needs 
> pagination...
> 
> Thanks,
> Estani
> Am 14.03.2012 12:12, schrieb stephen.pascoe at stfc.ac.uk:
> > It's a while since I've tried the browser but I had noticed 
> responsiveness pretty low as the number of datasets in the 
> single "cmip5" container increased.  Maybe you have more 
> locally-published datasets since you've published replicas?  
> My guess is that it's a straight-forward scalability problem.
> >
> > You could try moving some datasets to separate top-level 
> collections through some SQL but I'd try it out on a test 
> Gateway first.
> >
> > Stephen.
> >
> > ---
> > Stephen Pascoe  +44 (0)1235 445980
> > Centre of Environmental Data Archival
> > STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot 
> OX11 0QX, 
> > UK
> >
> > -----Original Message-----
> > From: esg-gateway-dev-bounces at mailman.earthsystemgrid.org 
> > 
> [mailto:esg-gateway-dev-bounces at mailman.earthsystemgrid.org] 
> On Behalf 
> > Of Estanislao Gonzalez
> > Sent: 14 March 2012 11:08
> > To: esg-gateway-dev at earthsystemgrid.org
> > Subject: [esg-gateway-dev] Catalog Browsing broken at WDCC
> >
> > Hi,
> >
> > for some reason I don't know it's impossible to browse the  
> catalogs 
> > at our Gateway: http://ipcc-ar5.dkrz.de/project/CMIP5.html
> > I've increase the memory... but it does nothing... the browser just 
> > waits for ever, no exception is thrown no problem is being 
> displayed.
> > I notice I had ~10 postgres threads running at 30%. After 
> restarting 
> > the server and waiting for a while I had 2 threads running at 95%. 
> > I've restarted it once more and I have a single threads 
> running at 100%.
> >
> > I don't see this happening in other gateways... any Idea 
> what it might 
> > be? Any idea on how to solve/debug it?
> >
> > Thanks,
> > Estani
> >
> 
> 
> --
> Estanislao Gonzalez
> 
> Max-Planck-Institut für Meteorologie (MPI-M) Deutsches 
> Klimarechenzentrum (DKRZ) - German Climate Computing Centre 
> Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
> 
> Phone:   +49 (40) 46 00 94-126
> E-Mail:  gonzalez at dkrz.de
> 
> _______________________________________________
> esg-gateway-dev mailing list
> esg-gateway-dev at mailman.earthsystemgrid.org
> http://mailman.earthsystemgrid.org/mailman/listinfo/esg-gateway-dev
>