Awesome Lists

supercrawler

by brendonboshell

JavaScriptpushed almost 4 years ago

A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.

AI summary

Web Crawler

A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages.

stars
380
forks
62
watching
11
awesome list
1
View on GitHub

Embed the badge

Show how many awesome lists link to your project. The count updates automatically.

Awesome Lists badge
Markdown
[![Awesome Lists Badge](https://awesome.facts.dev/shield/brendonboshell/supercrawler/links.svg)](https://awesome.facts.dev/awesome/brendonboshell/supercrawler)
HTML
<a href="https://awesome.facts.dev/awesome/brendonboshell/supercrawler"><img src="https://awesome.facts.dev/shield/brendonboshell/supercrawler/links.svg" alt="Awesome Lists Badge" /></a>
Image URL
https://awesome.facts.dev/shield/brendonboshell/supercrawler/links.svg

Add a GitHub project

Missing a project or an awesome list? Paste its GitHub URL and we fetch it right away.