crawler4j
by yasserg
Open Source Web Crawler for Java
AI summary
Web crawler
A Java-based web crawler for extracting and processing web page content
- stars
- 4.6K
- forks
- 1.9K
- watching
- 306
- awesome lists
- 3
Similar projects
Found by comparing what the projects do, not just their names.
code4craft/webmagic11.5K
crawler
A framework for building scalable web crawlers in Java.
unclecode/crawl4ai18.5K
Web crawler
A web crawling tool designed to extract structured data from the web for use in AI applications
apify/crawlee16.1K
Web scraper
A tool for building reliable web scraping and browser automation pipelines in Node.js.
spatie/crawler2.6K
Crawler
A powerful web crawler written in PHP that can execute JavaScript and crawl multiple URLs concurrently.
Crawler
A distributed crawling framework that leverages Headless Chrome to scrape dynamic websites
Crawler framework
A scalable and versatile web crawling framework based on Apache Storm
xtuhcy/gecco2.5K
Web Crawler Framework
A lightweight web crawler framework that enables easy extraction of web page data using jQuery-like selectors and supports asynchronous requests and distributed crawling.
Crawler
A tool for automatically discovering and crawling web application endpoints and assets
Web Crawler
A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages.
Link crawler
A tool to crawl websites by exploring links recursively with support for JavaScript rendering.
Web Crawler
A versatile web crawler built with modern tools and concurrency to handle various crawl tasks
Web Crawler
A flexible web crawler that can be used to extract data from websites in a scalable and efficient manner
Crawler framework
A Java framework for building web-based crawlers with features like distributed crawling and proxy support.
Content scraper
Automates the process of generating knowledge files to create custom AI models from website content
Tweet client
A Java library providing access to the Twitter API for sending and retrieving tweets.
Featured in 3 awesome lists
Each link jumps to the spot where the list mentions crawler4j.
