[Go-essp-tech] Progress and Problems with P2P

[Kettleborough, Jamie]

Hello Stephen,

I know we've talked about this before, and you thought it was hard to
get right, but now seems a good time to bring it up again.

>From a user view it would be good to get a more useful error code - I
don't know that 403 Forbidden is really the most useful for a user in
this case.  I realise if its an attribute service thing then for the
system it kind of looks like an 'authorization' failure, but for the
user I think its more of a 'system busy try again later' message (503?).

I agree there are higher priority things to fix. One is reducing the
frequency you see these messages, but if that looks hard is there
anything that can be done to make this error clearer?

Jamie

> -----Original Message-----
> From: go-essp-tech-bounces at ucar.edu 
> [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of 
> stephen.pascoe at stfc.ac.uk
> Sent: 02 February 2012 09:41
> To: jma at cola.iges.org; go-essp-tech at ucar.edu; 
> Luca.Cinquini at jpl.nasa.gov
> Subject: Re: [Go-essp-tech] Progress and Problems with P2P
> 
> Hi Jenifer,
> 
> These are really useful results, thanks.
> 
> > These are the data nodes that are not allowing me to complete the 
> > download (for the decadal1980 example mentioned above).
> > esg.cnrm-game-meteo.fr
> > vesg.ipsl.fr
> > dias-esg-nd.tkl.iis.u-tokyo.ac.jp
> > cmip-dn.badc.rl.ac.uk
> > bmbf-ipcc-ar5.dkrz.de
> 
> I think that it is no coincidence that these are the European 
> datanodes (plus Japan).  My belief is that the "ERROR 403: 
> Forbidden." error is really failure of the AttributeService 
> call to PCMDI.  All nodes have to contact PCMDI on every 
> request to check the user is a member of the CMIP5 Research group.
> 
> We need to cache these AttributeService responses on the 
> datanode side.  Luca, does esg-security do this now?  Our 
> production datanode has a fairly old version.  We are also 
> looked into using our Python ESGF-security stack as a proxy.
> 
> Also the BADC datanode is throttling TCP connections to a 
> maximum of 20 per incoming IP.  This has made a substantial 
> difference to quality of service as it prevents a couple of 
> users from swamping the system.  If you are an institution 
> behind NAT or a proxy it could be a problem: let me know and 
> I'll see what we can do.  Being throttled wouldn't produce 
> the 403 error, the connection would just hang.
> 
> Cheers,
> Stephen.
> 
> 
> ---
> Stephen Pascoe  +44 (0)1235 445980
> Centre of Environmental Data Archival
> STFC Rutherford Appleton Laboratory, Harwell Oxford, Didcot 
> OX11 0QX, UK
> 
> 
> -----Original Message-----
> From: go-essp-tech-bounces at ucar.edu 
> [mailto:go-essp-tech-bounces at ucar.edu] On Behalf Of Jennifer Adams
> Sent: 01 February 2012 16:08
> To: go-essp-tech at ucar.edu
> Subject: [Go-essp-tech] Progress and Problems with P2P
> 
> Hi, Everyone --
> I have been working with Luca and Gavin and Estani on testing 
> the P2P system. I'm happy to report that the many significant 
> parts of my workflow for downloading data can now be fully automated:
> 1. Search for available data sets that meet my desired 
> requirements (e.g. decadal1980/atmos/mon/Amon, all models, 
> all members, selected variables) 2. Compare search results to 
> a list of what I've already got 3. Build script to download 
> and run wget scripts for datasets I still need All this using 
> shell scripts and without touching a browser! 
> 
> There are still a few wrinkles, however. The worst of these 
> is that not all data nodes are authenticating certificates properly. 
> 
> These data nodes are working:
> bmbf-ipcc-ar5.dkrz.de
> cmip-dn.badc.rl.ac.uk
> dias-esg-nd.tkl.iis.u-tokyo.ac.jp
> esgdata.gfdl.noaa.gov
> esg.cnrm-game-meteo.fr
> norstore-trd-bio1.hpc.ntnu.no
> pcmdi11.llnl.gov
> vesg.ipsl.fr
> 
> These data nodes are not:
> dap.cccma.uvic.ca
> esg.nccs.nasa.gov
> bcccsm.cma.gov.cn
> tds.ucar.edu
> pcmdi9.llnl.gov
> esg-datanode.jpl.nasa.gov
> 
> P2P wget scripts to download data from the second set of data 
> nodes always fail completely (for me, a "pure client" user). 
> I know that Luca and Gavin and Estani are working to fix 
> these problems, but here's some encouragement to the data 
> node administrators to help them resolve this issue. Wgets 
> that rely on the quickly-expiring authorization tokens ought 
> to be deprecated as soon as possible -- they are the biggest, 
> most irritating source of errors in the whole system (my 
> opinion only).
> 
> The second problem is that the wget scripts don't often 
> succeed in getting all the files they are configured to grab. 
> Of the 64 scripts that I ran this morning, 55 were incomplete 
> (that's 86%). The errors in log files are all "ERROR 403: 
> Forbidden." I was running the 64 scripts at one time, but 
> each one was grabbing its list of files in order -- I am not 
> parallelizing the wgets in the way that some other users have 
> described. When I rerun the wget scripts from the incomplete 
> runs, some of them finish the job on the 2nd try, others will 
> take as many as 8 tries before all the files are in. 
> 
> These are the data nodes that are not allowing me to complete 
> the download (for the decadal1980 example mentioned above). 
> esg.cnrm-game-meteo.fr
> vesg.ipsl.fr
> dias-esg-nd.tkl.iis.u-tokyo.ac.jp
> cmip-dn.badc.rl.ac.uk
> bmbf-ipcc-ar5.dkrz.de
> 
> I suspect there are some throttling settings in place to 
> limit the number of wgets that a data node will allow at any 
> particular time. I think my use of the data nodes is 
> reasonable, and these throttles are set too high. The dreaded 
> "Forbidden" errors may be related to another problem, but are 
> still a nuisance no matter what the reason. YAO (Yet Another 
> Obstacle). Please put this on the high priority list of 
> things to fix, right behind the certificate authentication issue. 
> 
> Respectfully submitted,
> Jennifer
> 
> 
> 
> 
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
> --
> Scanned by iCritical.
> _______________________________________________
> GO-ESSP-TECH mailing list
> GO-ESSP-TECH at ucar.edu
> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech
>