[Go-essp-tech] [esg-node-dev] Data Node 404 Error

[martin.juckes at stfc.ac.uk]

Thanks -- I agree a low number is better for shorter distances, but I'm only getting around 100kBytes/s from Australia and US, so I'd like to use more. Now we've got GridFTP working smoothly from DKRZ to BADC, the best solution is probably to try and get it working from other centres as well,

cheers,
Martin
________________________________________
From: Estanislao Gonzalez [gonzalez at dkrz.de]
Sent: 16 December 2011 17:03
To: Juckes, Martin (STFC,RAL,RALSP)
Cc: sebastien.denvil at ipsl.jussieu.fr; go-essp-tech at ucar.edu; esg-node-dev at lists.llnl.gov
Subject: Re: [Go-essp-tech] [esg-node-dev] Data Node 404 Error

... probably because the number of errors you'll get.

The way I do it is adding one thread at a time and checking what
happens, if it brings no improvement at all you wouldn't want to add
more (I'm sure that if the distance is near, 3~5 are enough, this might
not be true for Australia though).

In any case I'm sure you would just skip and log the file that was
skipped, so in the worst scenario you would realize the number of
required threads is lower. There is no way of getting this number from
outside AFAIK.

It's not a nice solution, but it's the only one I could think of...

Thanks,
Estani

Am 16.12.2011 17:36, schrieb martin.juckes at stfc.ac.uk:
> Hello Estani,
>
> I'm not doing any wget transfers from your server, but if others have, like you, limited the number of simultaneous requests I can make, is there any way I can find out what that limit is?
>
> Cheers,
> Martin
>
>>> -----Original Message-----
>>> From: go-essp-tech-bounces at ucar.edu [mailto:go-essp-tech-
>>> bounces at ucar.edu] On Behalf Of Estanislao Gonzalez
>>> Sent: 16 December 2011 16:33
>>> To: Sébastien Denvil
>>> Cc: go-essp-tech at ucar.edu; esg-node-dev at lists.llnl.gov
>>> Subject: Re: [Go-essp-tech] [esg-node-dev] Data Node 404 Error
>>>
>>> Hi Sebastein,
>>>
>> >from our side this is caused by too many connections to the server,
>>> but
>>> I can't speak for others. The resulting error "might" be a 404 but
>>> might
>>> also be a 403, depending probably on wht exactly was the last
>>> connection
>>> breaking it (not sure about that).
>>>
>>> Our solution was limiting the number of connections per ip, but since
>>> people like jumping hurdles and start accessing from multiple
>>> machines,
>>> we might have to start limiting whole subnet masks, which will be very
>>> unfair for some... If I could find a better option I would.
>>>
>>> This might not proof to be a perfect solution , since for whatever
>>> reason it might be, tools used for accesing our data node do not close
>>> connections properly and leave them Half-open. This "might" be solved
>> >from the server side when intentionally forcing the connection to be
>>> closed, probably by a GC task (I've also re-tuned the GC of tomcat
>>> hoping it will clean unused connections faster without bothering the
>>> huge TDS cache).
>>>
>>> As I said, this is just our particular case.
>>>
>>> Thanks,
>>> Estani
>>>
>>> Am 16.12.2011 17:16, schrieb Sébastien Denvil:
>>>>   Dear CMIP5 data node administrators,
>>>>
>>>> from time to time ; when downloading data we encounter "ERREUR 404:
>>>> Not Found".
>>>>
>>>> It happened on our node for a few files. We had to perform the
>>>> following sanity command to be sure:
>>>> chmod -R a+rX *
>>>>
>>>> Depending on your constraint the command may be more complex. For
>>>> example if the data are not readable by all users be sure that
>>> tomcat
>>>> belongs to the group able to read the data and be sure that the
>>>> corresponding group can read the data.
>>>>
>>>> It may seems absolutely obvious but the tomcat user needs to be able
>>>> to read the data. Sorry for such a trivial reminder but ... :-)
>>>>
>>>> Regards.
>>>> Sébastien
>>>>
>>>
>>> --
>>> Estanislao Gonzalez
>>>
>>> Max-Planck-Institut für Meteorologie (MPI-M)
>>> Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
>>> Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany
>>>
>>> Phone:   +49 (40) 46 00 94-126
>>> E-Mail:  gonzalez at dkrz.de
>>>
>>> _______________________________________________
>>> GO-ESSP-TECH mailing list
>>> GO-ESSP-TECH at ucar.edu
>>> http://mailman.ucar.edu/mailman/listinfo/go-essp-tech


--
Estanislao Gonzalez

Max-Planck-Institut für Meteorologie (MPI-M)
Deutsches Klimarechenzentrum (DKRZ) - German Climate Computing Centre
Room 108 - Bundesstrasse 45a, D-20146 Hamburg, Germany

Phone:   +49 (40) 46 00 94-126
E-Mail:  gonzalez at dkrz.de

-- 
Scanned by iCritical.