Working example of a dumb spider:
ASPIDER is a dumb spider that just spiders a given
URL and attempts to download any file that matches a given mime type.
Other examples of dumb spiders being used:
Google or other search engines for content indexing. These spiders run all over the internet and attempt to "index" the entire internet's available content. They often obey and respect the
/robots.txt file. While Google spiders are highly advanced and can
probably do complex pattern matching, they still perform the same function on
each page and rank pages based on keywords.