Scrapy project

Repositories

scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

python crawler framework scraping crawling

Python 8,718 37,845 451 (36 issues need help) 307 Updated Jul 29, 2020

scrapy-bench

A CLI for benchmarking Scrapy.

python web-crawler scrapy command-line-tool benchmark-suite scrapy-bench

Python MIT 14 26 6 1 Updated Jul 27, 2020

scrapy.org

The scrapy.org website

html

HTML 97 38 3 5 Updated Jul 17, 2020

queuelib

Collection of persistent (disk-based) queues

Python 45 193 2 5 Updated Jul 15, 2020

quotesbot

This is a sample Scrapy project for educational purposes

Python MIT 625 921 0 6 Updated Jul 13, 2020

itemloaders

Library to populate items using XPath and CSS with a convenient API

Python BSD-3-Clause 1 12 2 1 Updated Jul 2, 2020

itemadapter

Common interface for data container classes

python metadata python3 scrapy python-dataclasses python-attrs

Python BSD-3-Clause 1 5 2 1 Updated Jun 24, 2020

parsel

Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

css python xml scraping selectors xpath lxml

Python 91 559 20 (1 issue needs help) 10 Updated Jun 18, 2020

w3lib

Python library of web-related functions

python

Python 83 310 12 11 Updated May 14, 2020

scrapy-itemloader Archived

[Archived] Library to populate Scrapy items using XPath and CSS with a convenient API

Python BSD-3-Clause 6 5 2 0 Updated May 5, 2020

cssselect

CSS Selectors for Python

css python selectors

Python 44 226 14 12 Updated Apr 28, 2020

protego

A pure-Python robots.txt parser with support for modern conventions.

python robots-txt robots-parser

DIGITAL Command Language BSD-3-Clause 10 13 0 0 Updated Apr 6, 2020

scrapyd

A service daemon to run Scrapy spiders

Python 501 2,109 98 24 Updated Feb 20, 2020

scrapyd-client

Command line client for Scrapyd server

Python 98 545 17 9 Updated Dec 6, 2019

scrapely

A pure-python HTML screen-scraping library

HTML 248 1,692 24 5 Updated Nov 28, 2019

loginform

Fill HTML login forms automatically

Python 67 232 9 2 Updated Oct 18, 2019

booksbot

Forked from stummjr/books_crawler

A crawler for http://books.toscrape.com

Python 623 26 0 1 Updated Mar 13, 2019

scurl

Performance-focused replacement for Python urllib

python cython chromium gurl urlparse

Python Apache-2.0 6 16 10 (1 issue needs help) 1 Updated Oct 2, 2018

url-chromium

url component from Chromium source code, forked from https://chromium.googlesource.com/chromium/src/url

chromium gurl

C++ 2 0 0 0 Updated Aug 7, 2018

base-chromium

base component forked from Chromium source https://chromium.googlesource.com/chromium/src/base/

C++ 3 1 0 0 Updated Jul 31, 2018

dirbot

Scrapy project to scrape public web directories (educational) [DEPRECATED]

Python 1,136 1,594 0 0 Updated Oct 27, 2017

scrapy-bench-speedcenter

Forked from Parth-Vader/scrapy-bench-speedcenter

Codespeed for scrapy-bench

Python 2 2 0 0 Updated Aug 28, 2017

pypydispatcher

A fork of http://pydispatcher.sourceforge.net/ with PyPy support

Python 3 12 1 0 Updated Jul 3, 2017

slybot

60 221 5 0 Updated Apr 27, 2015

gsoc2014-integration-tests

GSoC2014 - Scrapy Integration tests project

Shell 3 3 0 0 Updated Mar 18, 2014