crawlee
by apify
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
AI summary
Web scraper
A tool for building reliable web scraping and browser automation pipelines in Node.js.
- stars
- 16.1K
- forks
- 699
- watching
- 105
- awesome list
- 1
Similar projects
Found by comparing what the projects do, not just their names.
spatie/crawler2.6K
Crawler
A powerful web crawler written in PHP that can execute JavaScript and crawl multiple URLs concurrently.
Web scraper
A NodeJS-based web crawler and spider that extracts data from websites.
Crawler
A distributed crawling framework that leverages Headless Chrome to scrape dynamic websites
unclecode/crawl4ai18.5K
Web crawler
A web crawling tool designed to extract structured data from the web for use in AI applications
code4craft/webmagic11.5K
crawler
A framework for building scalable web crawlers in Java.
crawler proxy
intercepts requests from web crawlers and proxies them to a prerendering service for rendering HTML
howie6879/ruia1.8K
Crawler framework
An async web scraping micro-framework built with asyncio and aiohttp to simplify URL crawling
Web scraper
A versatile web scraping module with two scrapers for static and dynamic content extraction.
Web scraper
A flexible web scraping framework for extracting data from websites with customizable selectors and pagination support.
crawler
A Python web crawling framework utilizing asyncio and aiohttp for efficient data extraction from websites.
Content scraper
Automates the process of generating knowledge files to create custom AI models from website content
Crawler
An OSINT bot that crawls pastebin sites to search for sensitive data leaks
Web crawler
A Java-based web crawler for extracting and processing web page content
Web Crawler
A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages.
gocolly/colly23.4K
Website scraper
A framework for extracting structured data from websites in a fast and elegant way
Featured in 1 awesome list
Each link jumps to the spot where the list mentions crawlee.
