A full-featured web UI for Scrapyd cluster management

• :diamond_shape_with_a_dot_inside: Scrapyd Cluster Management

  • :100: All Scrapyd JSON API Supported
  • :ballot_box_with_check: Group, filter and select any number of nodes
  • Execute command on multinodes with just a few clicks

• :mag: Scrapy Log Analysis

  • :bar_chart: Stats collection
  • :chart_with_upwards_trend: Progress visualization
  • :bookmark_tabs: Logs categorization

• :battery: Enhancements

  • :package: Auto eggify your projects
  • ️ Integrated with :link: LogParser
  • :e-mail: Email notice
  • :iphone: Mobile UI
  • :closed_lock_with_key: Basic auth for web UI

:warning: Prerequisites

:exclamation:️ Make sure that :link: Scrapyd has been installed and started on all of your hosts.

:bangbang: Note that for remote access, you have to manually set 'bind_address = 0.0.0.0' in :link: the configuration file of Scrapyd and restart Scrapyd to make it visible externally.

:arrow_down: Install

• Use pip:

pip install scrapydweb

• Use git:

git clone https://github.com/my8100/scrapydweb.git
cd scrapydweb
python setup.py install

:arrow_forward: Start

1. Start ScrapydWeb via command scrapydweb . (a config file would be generated for customizing settings at the first startup.)
2. Visit http://127.0.0.1:5000 (It's recommended to use Google Chrome for a better experience.)

:globe_with_meridians: Browser Support

The latest version of Google Chrome, Firefox, and Safari.

$ git clone https://github.com/my8100/scrapydweb.git
$ cd scrapydweb

# To create isolated Python environments
$ pip install virtualenv
$ virtualenv venv/scrapydweb
# Or specify your Python interpreter: $ virtualenv -p /usr/local/bin/python3.7 venv/scrapydweb
$ source venv/scrapydweb/bin/activate

# Install dependent libraries
(scrapydweb) $ python setup.py install
(scrapydweb) $ pip install pytest
(scrapydweb) $ pip install coverage

# Make sure Scrapyd has been installed and started, then update the custom_settings item in tests/conftest.py
(scrapydweb) $ vi tests/conftest.py
(scrapydweb) $ curl http://127.0.0.1:6800

(scrapydweb) $ coverage run --source=scrapydweb -m pytest tests/test_a_factory.py -s -vv
(scrapydweb) $ coverage run --source=scrapydweb -m pytest tests -s -vv
(scrapydweb) $ coverage report
# To create an HTML report, check out htmlcov/index.html
(scrapydweb) $ coverage html

• Front End
  • :link: jQuery
  • :link: Vue.js
  • :link: Element
  • :link: ECharts
• Back End
  • :link: Flask
  • :link: Flask-Compress
  • :link: Requests