webmagic
by code4craft
A scalable web crawler framework for Java.
AI summary
crawler
A framework for building scalable web crawlers in Java.
- stars
- 11.5K
- forks
- 4.2K
- watching
- 767
- awesome lists
- 4
Similar projects
Found by comparing what the projects do, not just their names.
Web crawler
A Java-based web crawler for extracting and processing web page content
unclecode/crawl4ai18.5K
Web crawler
A web crawling tool designed to extract structured data from the web for use in AI applications
apify/crawlee16.1K
Web scraper
A tool for building reliable web scraping and browser automation pipelines in Node.js.
Crawler
A distributed crawling framework that leverages Headless Chrome to scrape dynamic websites
Crawler framework
A Java framework for building web-based crawlers with features like distributed crawling and proxy support.
spatie/crawler2.6K
Crawler
A powerful web crawler written in PHP that can execute JavaScript and crawl multiple URLs concurrently.
xtuhcy/gecco2.5K
Web Crawler Framework
A lightweight web crawler framework that enables easy extraction of web page data using jQuery-like selectors and supports asynchronous requests and distributed crawling.
Web Crawler
A flexible web crawler that can be used to extract data from websites in a scalable and efficient manner
Crawler framework
A distributed crawler framework that simplifies the process of building crawlers using Spring Boot and Redis
sjdirect/abot2.3K
Web crawler
A C# web crawler framework built for speed and flexibility, allowing developers to easily crawl websites with customizable logic.
Content scraper
Automates the process of generating knowledge files to create custom AI models from website content
Web Crawler
A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages.
dyweb/scrala113
Web crawler framework
A web crawling framework written in Scala that allows users to define the start URL and parse response from it
spine/spine3.7K
Web framework
An MVC framework that provides structure and simplicity for building JavaScript web applications
Crawler
A tool for automatically discovering and crawling web application endpoints and assets
Featured in 4 awesome lists
Each link jumps to the spot where the list mentions webmagic.
