crawler4j - Open Source Web Crawler for Java - Google Project Hosting
Crawler4j is an open source Java crawler which provides a simple interface for crawling the Web. You can setup a multi-threaded web crawler in 5 minutes!
NOTE: version 3.0 is deprecated and should not be used. Please use the latest 3.3 version.
Sample Usage
You need to create a crawler class that extends WebCrawler . This class decides which URLs should be crawled and handles the downloaded page. The following is a sample implementation:
public class MyCrawler extends WebCrawler {
? ? ? ? private final static Pattern FILTERS = Pattern . compile ( ".*(\\.(css|js|bmp|gif|jpe?g"
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? + "|png|tiff?|mid|mp2|mp3|mp4"
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? + "|wav|avi|mov|mpeg|ram|m4v|pdf"
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? + "|rm|smil|wmv|swf|wma|zip|rar|gz))$" );
? ? ? ? /**
? ? ? ? ?* You should implement this function to specify whether
? ? ? ? ?* the given url should be crawled or not (based on your
? ? ? ? ?* crawling logic).
crawler4j - Open Source Web Crawler for Java - Google Project Hosting
更多文章、技術交流、商務合作、聯系博主
微信掃碼或搜索：z360901061
微信掃一掃加我為好友
QQ號聯系： 360901061
您的支持是博主寫作最大的動力，如果您喜歡我的文章，感覺我的文章對您有幫助，請用微信掃描下面二維碼支持博主2元、5元、10元、20元等您想捐的金額吧，狠狠點擊下面給點支持吧，站長非常感激您！手機微信長按不能支付解決辦法：請將微信支付二維碼保存到相冊，切換到微信，然后點擊微信右上角掃一掃功能，選擇支付二維碼完成支付。
【本文對您有幫助就好】元
