Apps with 'Web Crawler' feature

Free software that downloads entire sites to local storage for offline use, maintaining the original link structure for seamless browsing.

Open source search engine crawling the web’s independent, non-commercial content, prioritizing small sites, supports white-label setups and low-cost operation.

Portia is an open source visual scraping tool, allows you to scrape websites without any programming knowledge required! Simply annotate pages you're interested in, and Portia will create a spider to extract data from similar pages.

Cloud-based platform for extracting data and automating website workflows, featuring headless browser support, advanced web crawling, reusable code acts and scalable storage.

Boost your site's SEO with SEO Tracer! Crawl fast, find broken links, analyze meta tags, and optimize. Free, private and secure.

Netpeak Spider is an SEO crawler for a day-to-day SEO audit, fast issue check, comprehensive analysis, and website scraping.

grab-site is a crawler for archiving websites to WARC files. It includes a dashboard for monitoring multiple crawls, and supports changing URL ignore patterns during the crawl.

Minexa.ai is a next-generation tool that makes web scraping faster and more affordable with an AI-powered solution no other alternative has. Unlike others that require constant tweaking, struggle under heavy loads, or charge extra for natural language processing, Minexa adapts...

API for website crawling, scraping, and structured data extraction. Converts web content to markdown or data for AI apps, automates multi-page analysis, and supports custom output formats.

Offers premium proxies, AI-powered web scraping tools, a global pool of over 102 million IPs, automated infrastructure, datasets, robust management, and custom APIs.

Cockroach Crawler is a governed web acquisition and extraction layer for AI agents. It supports bounded public-web crawling, searchable site maps, robots handling, JavaScript rendering, readable text and Markdown, JSON and JSONL, structured fields, CSS, XPath, restricted-regex...

The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.

Anakin.io is a web scraping and structured data extraction API built for developers and AI teams. It converts any website into clean Markdown or JSON with a single API call - handling JavaScript rendering, anti-bot bypass, proxy rotation, and authenticated scraping (behind login...

Zeno is a web crawler designed to operate wide crawls or to simply archive one web page. Zeno's key concepts are: portability, performance, simplicity. With an emphasis on performance.

Automated SEO solution offering site audits, on-page optimization, keyword and SERP tracking, backlink monitoring, competitor analysis, and technical issue detection.

This project is a java web spider (web crawler) with the ability to download (and resume) files. It is also highly customizable with regular expressions and download templates.

Pulno is a website audit tool which checks for SEO related issues and sends comprehensible tips to improve on-page SEO with page speed optimizer (CSS and image file optimization), meta tags and unique text analyzer.

A browser extension that uses AI to detect listings type data which can be easily scraped into CSV or Excel file, no coding required. Can automatically click next button to continue to the next page. The extension runs completely in user’s browser.

Infatica boasts a global portfolio of residential IPs - over 2,500,000 residential socks5 proxies sourced from real consumers across dozens of countries. Support via tickets, live chat, and phone, with 24-7 response for urgent technical issues.

Open-source, extensible crawler for large-scale web archiving, preserves digital artifacts, offers plugin support, distributed crawling, and standardized export formats.

Extract all webpage links, identify deceptive overlays, download images, generate QR codes, filter by source, and scrape links from files, clipboard, or web pages.

Extract information from web sites with a visual point-and-click toolkit. Turn websites into useful data. Automate data workflows on the web, process, and transform data at any scale.

Octoparse is a no-code web scraping tool. It provides both free pre-made scraper templates and custom scraping features with which people without coding knowledge can extract various web data with simple point-and-click.

Listly, a web extension, simplifies web scraping without coding. This helps you collect and export enormous volumes of data into either Excel or Google Sheets.

We crawl the web so you don't have to. Our crawlers download and structure millions of posts a day, we store and index the data so all you have to do is to define what part of the data you need.