How fast can I actually crawl a website?

+4 votes
asked Dec 9, 2010 by iqlessthan70

I'm gonna crawl a website for some information. It's about 170 000+ pages. So, how many request can I make? I'm gonna extract til HTML and get some information. This is a already very popular site, so I don't think it would die if was just cruising fast over all pages... Only thing that makes me nervous is that I don't know if the ownser will block my IP or something if you do that? Is that normal? Should I just load 5 pages/min ? Then it will take forever... I want to get new data every 24 hour see.

Thanks for all response!

4 Answers

+1 vote
answered Dec 9, 2010 by tyler-eaves

A second or two delay after each request should be sufficient. Making your bot crawl as fast as possible may in fact get you banned. In my day job I manage the sites for a couple of newspapers and I see homegrown crawlers occasionally. Bad ones really can cause quite a lot of system code and result in a new addition to the IP blacklist. Don't be that guy.

+1 vote
answered Dec 9, 2010 by blueberryfields

As long as you're obeying their robots.txt instructions, you should probably be alright. The standard delay I've seen between requests is 2 seconds - that's fairly often the limit after which you might start having your traffic throttled, or ip blocked.

+5 votes
answered Dec 9, 2010 by ruel

It will take sometime, actually I suggest you use rotating proxies, and add multi-threading. 10 threads will do. This way, you can have 10 requests at the same time. Using proxies will be slow though, and add timeout of atleast 1.5 secs each request, it will slow you down, but lowers the risk of getting banned.

+2 votes
answered Dec 9, 2010 by jan-vladimir-mostert

I've created a webcrawler a couple years ago that crawled about 7GB a night from BBC's website (limited by bandwidth) and never got blocked, but adding a 1 second delay between requests is the decent thing to do.

Welcome to Q&A, where you can ask questions and receive answers from other members of the community.
Website Online Counter