Scrapy project

Repositories

• scrapy

  Scrapy, a fast high-level web crawling & scraping framework for Python.

  python crawler framework scraping crawling hacktoberfest
  Python 8,831 38,457 453 (34 issues need help) 296 Updated Oct 5, 2020

• parsel

  Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

  css python xml scraping selectors xpath lxml
  Python 89 574 23 (1 issue needs help) 9 Updated Oct 5, 2020

• cssselect

  CSS Selectors for Python

  css python selectors
  Python 44 227 14 13 Updated Oct 5, 2020

• scrapy.org

  The scrapy.org website

  html
  HTML 145 39 3 6 Updated Oct 4, 2020

• itemadapter

  Common interface for data container classes

  python metadata python3 scrapy hacktoberfest python-dataclasses python-attrs
  Python BSD-3-Clause 2 12 2 4 Updated Oct 1, 2020

• scrapy-bench

  A CLI for benchmarking Scrapy.

  python web-crawler scrapy command-line-tool benchmark-suite scrapy-bench
  Python MIT 15 26 6 1 Updated Sep 21, 2020

• itemloaders

  Library to populate items using XPath and CSS with a convenient API
  Python BSD-3-Clause 4 18 3 (1 issue needs help) 3 Updated Sep 16, 2020

• queuelib

  Collection of persistent (disk-based) queues
  Python 45 198 2 5 Updated Aug 26, 2020

• booksbot

  Forked from stummjr/books_crawler

  A crawler for http://books.toscrape.com
  Python 653 27 0 2 Updated Aug 8, 2020

• scrapyd

  A service daemon to run Scrapy spiders
  Python 511 2,136 102 24 Updated Jul 31, 2020

• quotesbot

  This is a sample Scrapy project for educational purposes
  Python MIT 635 942 0 6 Updated Jul 13, 2020

• w3lib

  Python library of web-related functions

  python
  Python 83 311 12 11 Updated May 14, 2020

• scrapy-itemloader Archived

  [Archived] Library to populate Scrapy items using XPath and CSS with a convenient API
  Python BSD-3-Clause 7 5 2 0 Updated May 5, 2020

• protego

  A pure-Python robots.txt parser with support for modern conventions.

  python robots-txt robots-parser
  DIGITAL Command Language BSD-3-Clause 10 14 0 0 Updated Apr 6, 2020

• scrapyd-client

  Command line client for Scrapyd server
  Python 103 553 19 9 Updated Dec 6, 2019

• scrapely

  A pure-python HTML screen-scraping library
  HTML 252 1,706 25 5 Updated Nov 28, 2019

• loginform

  Fill HTML login forms automatically
  Python 68 232 9 2 Updated Oct 18, 2019

• scurl

  Performance-focused replacement for Python urllib

  python cython chromium gurl urlparse
  Python Apache-2.0 6 16 10 (1 issue needs help) 1 Updated Oct 2, 2018

• url-chromium

  url component from Chromium source code, forked from https://chromium.googlesource.com/chromium/src/url

  chromium gurl
  C++ 2 0 0 0 Updated Aug 7, 2018

• base-chromium

  base component forked from Chromium source https://chromium.googlesource.com/chromium/src/base/
  C++ 3 1 0 0 Updated Aug 1, 2018

• dirbot

  Scrapy project to scrape public web directories (educational) [DEPRECATED]
  Python 1,133 1,600 0 0 Updated Oct 27, 2017

• scrapy-bench-speedcenter

  Forked from Parth-Vader/scrapy-bench-speedcenter

  Codespeed for scrapy-bench
  Python 2 2 0 0 Updated Aug 28, 2017

• pypydispatcher

  A fork of http://pydispatcher.sourceforge.net/ with PyPy support
  Python 3 12 1 0 Updated Jul 3, 2017

• slybot
  60 221 5 0 Updated Apr 27, 2015

• gsoc2014-integration-tests

  GSoC2014 - Scrapy Integration tests project
  Shell 3 3 0 0 Updated Mar 18, 2014