A-Spider is growing and is adding more features. Currently A-Spider v0.1a has the following features:
- A Command line interface
- Multi Threaded option, can configure the number of threads.
- Mime-type option which will download any file matching the given mime-type.
- File type extention option which will download any file extention matching the given file type..
- Handles & follows the following html tags: Anchor, Area, Frame, IFrame.
- Pattern matches email addresses by default.
- Options to download content from internal and or external sites based on the given start url.
- Options to follow links from internal and or external links based on the given start url.
- Proxy Support for spidering web sites via a given list of proxies.
- Automatically handles and follows meta-refresh tags like a browser would. (This functionality can be optionally turned off)
- Automatically handles web site cookies with an option to set different cookie policies if needed.
- Allows command line support for log4j configuration files.
- Download location options.
Last Modfied: Friday, 10-Jun-2005 15:03:04 UTC