StormCrawler is described as 'Open source SDK for building distributed web crawlers with Apache Storm. The project is under Apache license v2 and consists of a collection of reusable resources and components, written mostly in Java' and is an app. There are more than 10 alternatives to StormCrawler for a variety of platforms, including Windows, Linux, Mac, Web-based and BSD apps. The best StormCrawler alternative is Scrapy, which is both free and Open Source. Other great apps like StormCrawler are Lookyloo, Flyscrape, Crawlbase and @harshvz/crawler.
Scrapy is a free and open-source web-crawling framework written in Python. Originally designed for web scraping, it can also be used to extract data using APIs or as a general-purpose web crawler. It was developed and is maintained by Zyte formerly Scrapinghub, a web-scraping...
Lookyloo is a web interface that allows users to capture a website page and then display a tree of domains that call each other.
Switzerland
Flyscrape is a standalone and scriptable web scraper, combining the speed of Go with the flexibility of JavaScript. — Focus on data extraction rather than request juggling.
Germany
EU
Crawlbase, formerly ProxyCrawl, helps you stay anonymous while crawling the web, web crawling protection the way it should be.
A flexible web crawler and scraping tool using Playwright, supporting both BFS and DFS crawling strategies with screenshot capture and structured output. Installable via npm and usable both as a CLI and programmatically.
India
Scraperr is a self-hosted web application that allows users to scrape data from web pages by specifying elements via XPath. Users can submit URLs and the corresponding elements to be scraped, and the results will be displayed in a table.
United States
Open-source, extensible web crawler designed for large-scale, archival-quality web archiving, preserves digital artifacts, supports modular plugins, distributed crawling, detailed monitoring, scheduling, and exports data in standardized formats for preservation.
United States
Mixnode is a fast, flexible, massively scalable platform to extract and analyze data from the web.
United States
Apache Nutch is a highly extensible and scalable open source web crawler software project.
Kaddara is a platform designed for professionals who need fresh leads to run their business and whose business is affected by how competitors operate.
