abot
Web crawler
A C# web crawler framework built for speed and flexibility, allowing developers to easily crawl websites with customizable logic.
Cross Platform C# web crawler framework built for speed and flexibility. Please star this project! +1.
2k stars
199 watching
560 forks
Language: C#
last commit: 6 months ago
Linked from 2 awesome lists
abotabot-nugetc-sharpcrawlercross-platformcsharpcsharp-libraryjavascript-renderernetcorenetcore2netcore3netstanetstandard20netstandard21parsingpluggablespiderspidersunit-testingweb-crawler
brucedone/awesome-crawler
uhub/awesome-c-sharpRelated projects:
| Repository | Description | Stars |
|---|---|---|
 code4craft/webmagic | A framework for building scalable web crawlers in Java. | 11,456 |
 yujiosaka/headless-chrome-crawler | A distributed crawling framework that leverages Headless Chrome to scrape dynamic websites | 5,534 |
 apify/crawlee | A tool for building reliable web scraping and browser automation pipelines in Node.js. | 16,081 |
 abpframework/abp | A framework for building enterprise software applications with a focus on opinionated architecture and best practices. | 13,038 |
 bda-research/node-crawler | A NodeJS-based web crawler and spider that extracts data from websites. | 6,718 |
 aspnetboilerplate/aspnetboilerplate | A general-purpose web application framework that automates common software development tasks and provides a modular, extensible architecture for building modern web applications. | 11,849 |
 spatie/crawler | A powerful web crawler written in PHP that can execute JavaScript and crawl multiple URLs concurrently. | 2,552 |
 brendonboshell/supercrawler | A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages. | 380 |
 unclecode/crawl4ai | A web crawling tool designed to extract structured data from the web for use in AI applications | 18,541 |
 bflattened/bflat | A C# compiler and runtime system that compiles to native executables with the performance of CoreCLR GC and RyuJIT | 3,676 |
 xtuhcy/gecco | A lightweight web crawler framework that enables easy extraction of web page data using jQuery-like selectors and supports asynchronous requests and distributed crawling. | 2,504 |
 turnersoftware/infinitycrawler | A web crawling library for .NET that allows customizable crawling and throttling of websites. | 248 |
 oatpp/oatpp | A C++ web framework designed to build scalable and resource-efficient web applications. | 7,958 |
 dotnet/silk.net | A high-performance library providing C# bindings to various low-level APIs for multimedia, graphics, and compute applications. | 4,224 |
 hugoblox/hugo-blox-builder | An all-in-one website builder that uses Hugo as the underlying static site generator and provides a drag-and-drop interface for creating custom websites. | 8,445 |