Scrapy project

Repositories

• scrapy

  Scrapy, a fast high-level web crawling & scraping framework for Python.

  python crawler framework scraping crawling
  Python 8,703 37,742 448 (36 issues need help) 303 Updated Jul 17, 2020

• scrapy.org

  The scrapy.org website

  html
  HTML 97 38 3 5 Updated Jul 17, 2020

• scrapy-bench

  A CLI for benchmarking Scrapy.

  python web-crawler scrapy command-line-tool benchmark-suite scrapy-bench
  Python MIT 14 25 6 0 Updated Jul 16, 2020

• queuelib

  Collection of persistent (disk-based) queues
  Python 45 193 2 5 Updated Jul 15, 2020

• quotesbot

  This is a sample Scrapy project for educational purposes
  Python MIT 620 917 0 6 Updated Jul 13, 2020

• itemloaders

  Library to populate items using XPath and CSS with a convenient API
  Python BSD-3-Clause 1 12 2 1 Updated Jul 2, 2020

• itemadapter

  Common interface for data container classes

  python metadata python3 scrapy python-dataclasses python-attrs
  Python BSD-3-Clause 1 5 2 1 Updated Jun 24, 2020

• parsel

  Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

  css python xml scraping selectors xpath lxml
  Python 92 555 20 (1 issue needs help) 10 Updated Jun 18, 2020

• w3lib

  Python library of web-related functions

  python
  Python 83 306 12 11 Updated May 13, 2020

• scrapy-itemloader Archived

  [Archived] Library to populate Scrapy items using XPath and CSS with a convenient API
  Python BSD-3-Clause 6 5 2 0 Updated May 5, 2020

• cssselect

  CSS Selectors for Python

  css python selectors
  Python 44 225 14 12 Updated Apr 28, 2020

• protego

  A pure-Python robots.txt parser with support for modern conventions.

  python robots-txt robots-parser
  DIGITAL Command Language BSD-3-Clause 10 13 0 0 Updated Apr 6, 2020

• scrapyd

  A service daemon to run Scrapy spiders
  Python 499 2,099 98 24 Updated Feb 20, 2020

• scrapyd-client

  Command line client for Scrapyd server
  Python 98 542 17 9 Updated Dec 6, 2019

• scrapely

  A pure-python HTML screen-scraping library
  HTML 248 1,691 24 5 Updated Nov 28, 2019

• loginform

  Fill HTML login forms automatically
  Python 67 232 9 2 Updated Oct 18, 2019

• booksbot

  Forked from stummjr/books_crawler

  A crawler for http://books.toscrape.com
  Python 622 27 0 1 Updated Mar 13, 2019

• scurl

  Performance-focused replacement for Python urllib

  python cython chromium gurl urlparse
  Python Apache-2.0 6 16 10 (1 issue needs help) 1 Updated Oct 2, 2018

• url-chromium

  url component from Chromium source code, forked from https://chromium.googlesource.com/chromium/src/url

  chromium gurl
  C++ 2 0 0 0 Updated Aug 7, 2018

• base-chromium

  base component forked from Chromium source https://chromium.googlesource.com/chromium/src/base/
  C++ 3 1 0 0 Updated Jul 31, 2018

• dirbot

  Scrapy project to scrape public web directories (educational) [DEPRECATED]
  Python 1,135 1,593 0 0 Updated Oct 27, 2017

• scrapy-bench-speedcenter

  Forked from Parth-Vader/scrapy-bench-speedcenter

  Codespeed for scrapy-bench
  Python 2 2 0 0 Updated Aug 28, 2017

• pypydispatcher

  A fork of http://pydispatcher.sourceforge.net/ with PyPy support
  Python 2 12 1 0 Updated Jul 3, 2017

• slybot
  60 221 5 0 Updated Apr 27, 2015

• gsoc2014-integration-tests

  GSoC2014 - Scrapy Integration tests project
  Shell 3 3 0 0 Updated Mar 18, 2014