Welcome to SmoothCrawler’s documentation!

OS	Building Status	Coverage Status
Linux/MacOS
Windows

SmoothCrawler is a Python framework for being faster and easier to build crawler (or be called web spider). The core concept of its implementation is SoC (Separation of Concerns).

General documentation:

• Introduction
  • About
  • Comparison
  • Quickly Demo
• Installation
  • Install by pip
  • Dependencies
  • Related Dependencies in usage with SmoothCrawler
• Quickly Start
  • Create a web crawler (in the same object)
  • Create a web crawler (separate to different objects)
  • Implement different crawler to face different scenario

API Reference:

• URL
  • URL
• Crawler
  • Framework Modules
  • Implementation Modules
• Factory
  • CrawlerFactory
  • AsyncCrawlerFactory
• Components
  • HTTP Sender
  • HTTP Response Parser
  • Data Processing Handler
  • Persistence