[Go-essp-tech] Progress and Problems with P2P

[Jennifer Adams]

On Feb 2, 2012, at 4:41 AM, <stephen.pascoe at stfc.ac.uk> <stephen.pascoe at stfc.ac.uk> wrote:
>  My belief is that the "ERROR 403: Forbidden." error is really failure of the AttributeService call to PCMDI.  All nodes have to contact PCMDI on every request to check the user is a member of the CMIP5 Research group.
> 
> We need to cache these AttributeService responses on the datanode side.
This sounds like an excellent idea, especially since the result is static and isn't likely to change… 

> Luca, does esg-security do this now?  Our production datanode has a fairly old version.  We are also looked into using our Python ESGF-security stack as a proxy.
Of course, it alos implies a change to the data node software suite, which is can of worms. 

> 
> Also the BADC datanode is throttling TCP connections to a maximum of 20 per incoming IP. This has made a substantial difference to quality of service as it prevents a couple of users from swamping the system.  If you are an institution behind NAT or a proxy it could be a problem: let me know and I'll see what we can do.  Being throttled wouldn't produce the 403 error, the connection would just hang.
That makes sense, because downloads from BADC fail more than any of the others combined. But I wouldn't sell this as improving the 'quality of service', because the end result is still failed downloads. Throttling ought to be more dynamic. If the server is not busy, why limit me to 20 connections? And if you can identify abusing users, you can throttle them specifically, but leave it open for the rest of us that are just trying to meet a deadline. 

If I am being throttled, does it mean the wget would sit and wait until one of my 20 working wgets is done, then it would proceed? Because I am also getting failed downloads due with "Connection refused" messages from cmip-dn.badc.rl.ac.uk. Is that a different problem? 

--Jennifer