abot

code4craft/webmagic11.5K

crawler

A framework for building scalable web crawlers in Java.

yujiosaka/headless-chrome-crawler5.5K

Crawler

A distributed crawling framework that leverages Headless Chrome to scrape dynamic websites

apify/crawlee16.1K

Web scraper

A tool for building reliable web scraping and browser automation pipelines in Node.js.

abpframework/abp13K

Enterprise app framework

A framework for building enterprise software applications with a focus on opinionated architecture and best practices.

bda-research/node-crawler6.7K

Web scraper

A NodeJS-based web crawler and spider that extracts data from websites.

aspnetboilerplate/aspnetboilerplate11.8K

Web framework

A general-purpose web application framework that automates common software development tasks and provides a modular, extensible architecture for building modern web applications.

spatie/crawler2.6K

Crawler

A powerful web crawler written in PHP that can execute JavaScript and crawl multiple URLs concurrently.

brendonboshell/supercrawler380

Web Crawler

A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages.

unclecode/crawl4ai18.5K

Web crawler

A web crawling tool designed to extract structured data from the web for use in AI applications

bflattened/bflat3.7K

Native C# compiler

A C# compiler and runtime system that compiles to native executables with the performance of CoreCLR GC and RyuJIT

xtuhcy/gecco2.5K

Web Crawler Framework

A lightweight web crawler framework that enables easy extraction of web page data using jQuery-like selectors and supports asynchronous requests and distributed crawling.

turnersoftware/infinitycrawler248

Crawler library

A web crawling library for .NET that allows customizable crawling and throttling of websites.

oatpp/oatpp8K

Web framework

A C++ web framework designed to build scalable and resource-efficient web applications.

dotnet/silk.net4.2K

Bindings library

A high-performance library providing C# bindings to various low-level APIs for multimedia, graphics, and compute applications.

hugoblox/hugo-blox-builder8.4K

Website builder

An all-in-one website builder that uses Hugo as the underlying static site generator and provides a drag-and-drop interface for creating custom websites.