abot
Web crawler
A C# web crawler framework built for speed and flexibility, allowing developers to easily crawl websites with customizable logic.
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
2k stars
199 watching
560 forks
Language: C#
last commit: 2 months ago
Linked from 2 awesome lists
abotabot-nugetc-sharpcrawlercross-platformcsharpcsharp-libraryjavascript-renderernetcorenetcore2netcore3netstanetstandard20netstandard21parsingpluggablespiderspidersunit-testingweb-crawler
brucedone/awesome-crawler
uhub/awesome-c-sharpRelated projects:
| Repository | Description | Stars |
|---|---|---|
 code4craft/webmagic | A scalable framework for building web crawlers in Java. | 11,432 |
 yujiosaka/headless-chrome-crawler | A distributed crawling framework that leverages Headless Chrome to scrape dynamic websites | 5,527 |
 apify/crawlee | A tool for building reliable web scraping and browser automation pipelines in Node.js. | 15,604 |
 abpframework/abp | A framework for building enterprise software applications with a focus on opinionated architecture and best practices. | 12,938 |
 bda-research/node-crawler | A NodeJS-based web crawler and spider that extracts data from websites. | 6,704 |
 aspnetboilerplate/aspnetboilerplate | A general-purpose web application framework that automates common software development tasks and provides a modular, extensible architecture for building modern web applications. | 11,823 |
 spatie/crawler | A powerful web crawler written in PHP that can execute JavaScript and crawl multiple URLs concurrently. | 2,537 |
 brendonboshell/supercrawler | A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages. | 378 |
 unclecode/crawl4ai | A tool for web crawling and data extraction, designed to work with large language models. | 16,180 |
 bflattened/bflat | A C# compiler and runtime system that compiles to native executables with the performance of CoreCLR GC and RyuJIT | 3,651 |
 xtuhcy/gecco | A lightweight web crawler framework that enables easy extraction of web page data using jQuery-like selectors and supports asynchronous requests and distributed crawling. | 2,502 |
 turnersoftware/infinitycrawler | A web crawling library for .NET that allows customizable crawling and throttling of websites. | 248 |
 oatpp/oatpp | A C++ web framework designed to build scalable and resource-efficient web applications | 7,910 |
 dotnet/silk.net | A high-performance library providing C# bindings to various low-level APIs for multimedia, graphics, and compute applications. | 4,170 |
 hugoblox/hugo-blox-builder | An all-in-one website builder that uses Hugo as the underlying static site generator and provides a drag-and-drop interface for creating custom websites. | 8,376 |