I'm glad to see that many of you are well on your way with the crawler assignment! Some of you are producing very optimized crawlers, which is great...but be careful to be polite! If you're crawler is really optimized and you hit a particular server up with too fast a request rate you can cause some trouble...it can seem like a DoS attack. So far, this has caused some issues with UGCS and the Library!
Don't get me in too much trouble!
P.S. It's great to see so many blog posts go up in recent days. So far they have all been quite interesting, so the first round is setting a high bar for those of you later in the alphabet.
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment