Personal License
- Perpetual license (does not expire)
- 1 site, unlimited servers
- No distribution (hosted use only)
- Commercial use allowed
Developer License
- Perpetual license (does not expire)
- Unlimited projects
- Can distribute code and binary products
- Commercial use allowed
14 Day money-back guarantee
Full refund within 14 days of purchase date.
- Create an account
- Log-in
-
Released:
Feb 16, 2011
Last Update: Feb 14, 2011 - Language: PHP
For customized crawling and scraping services check out Crawley Cloud
PHP Web Crawler is a software that searches for links in the web. It stores the links and some extra data in a database and shows them as HTML output.
Features:
- The crawler can be run as multiple instances
- It can be run by a cron job.
- Crawl results are saved in a MySQL database. It generates the table "urls" to store the crawls.
- For each url it saves the url of source, the url of the destiny and the anchor text. - Validates the urls via a regular expression. It avoids the links to static data into the site. Including the unnecessary media files. Despite this I can't ensure that the crawler avoids all the media files. That be more complex to validate.
Here's a tutorial about PHP Web Crawler
There's also a Python Web Crawler available.
PHP Web Crawler
See the follow to get started with the PHP web crawler:
http://codescience.wordpress.com/2011/02/15/php-web-crawler/
Installation / UnPacking
PHP Web crawler can run in any directory. But if you want use the Web UI please set it in a directory that can be served by the apache web server.
Dependencies:
- Apache2
- php5
- php5-mysql
Warning: Ensure that apache have the permision to write the config.ini file. Else you can do ~$chmod 777 config.ini (all the permises) on a unix like system.
Related Projects
Recently I started a Huge project!. It's a Crawling / Scraping framework written in Python.
It's totally open source and was realead under the GPL v3 license.
The repository is at github
And there's also a project website
Checkout it for free!
-
From the blog
Freedom, beer and open-source
Open source cannot live on donations alone
Fighting Online Fraud
[Software] Pricing Redacted -
Open-Source Projects
Publishing guidelines
Project proposals
Software licensing
License generator -
Participating
Publish on Binpress
Our commission
Affiliate program -
About us
Contact us
Terms of service
Privacy policy
FAQ
© 2013 Binpress Inc., All rights reserved.
Questions & Comments