What is a Web Spider?

Scrapers or data miners are also known as smart spiders. Ironically, these spiders' logic often breaks and are more difficult to maintain because they are based on website design and layout ( typically ). These spiders get the reputation of being the "duct tape" of the internet because there is often no real official protocol to collect the desired data. However, their existence and the fact that they do work are what some entire online industries are based on.

Instead of having the spider run all over every available link, a specific set of data is usually desired and the whole site need not be visited. This also requires some customization for site-specific modules or logic for parsing the desired data.