spider_man

Crawler library

A high-level web crawling and scraping framework for Elixir.

SpiderMan,a base-on Broadway fast high-level web crawling & scraping framework for Elixir.

GitHub

23 stars
4 watching
4 forks
Language: Elixir
last commit: over 2 years ago
Linked from 1 awesome list

crawlerdata-miningelixirerlangframeworkspider

Backlinks from these awesome lists:

Related projects:

RepositoryDescriptionStars
elixir-crawly/crawlyA framework for extracting structured data from websites994
fredwu/crawlerA high-performance web crawling and scraping solution with customizable settings and worker pooling.945
hu17889/go_spiderA modular, concurrent web crawler framework written in Go.1,827
postmodern/spidrA Ruby web crawling library that provides flexible and customizable methods to crawl websites809
matteoredaelli/ebotAn Erlang-based web crawler designed to be scalable and highly configurable330
chenjiandongx/github-spiderA Python-based web crawler for scraping Github user and repository data.264
elliotgao2/gainA Python web crawling framework utilizing asyncio and aiohttp for efficient data extraction from websites.2,037
spider-rs/spiderA tool for web data extraction and processing using Rust1,234
howie6879/ruiaAn async web scraping micro-framework built with asyncio and aiohttp to simplify URL crawling1,753
antchfx/antchA framework for building fast and efficient web crawlers and scrapers in Go.261
turnersoftware/infinitycrawlerA web crawling library for .NET that allows customizable crawling and throttling of websites.248
qinxuye/colaA high-level framework for building distributed data extractors from web pages1,501
dyweb/scralaA web crawling framework written in Scala that allows users to define the start URL and parse response from it113
gushonorato/mechanizeA web scraping and automation tool for Elixir.30
xianhu/pspiderA Python web crawler framework with support for multi-threading and proxy usage.1,828