The Wayback Machine - https://web.archive.org/web/20201113063603/https://github.com/topics/crawler

#

crawler

Here are 4,503 public repositories matching this topic...

scrapy

scrapy / scrapy

Star

Open

Usage of HttpCompressionMiddleware needs to be reflected in Scrapy stats

4

GeorgeA92 commented Sep 19, 2020

Summary

Usage of HttpCompressionMiddleware needs to be relfected in Scrapy stats.

Motivation

In order to estimate scrapy memory usage efficiency and prevent.. memory leaks like this.
I will need to know:

number of request/response objects that can be active (can be achieved by using [trackref](https://docs.scrapy.org/en/latest/topi

Read more

enhancement good first issue

Open

Add a job for Python 3.9 to .travis.yml

1

Open

Test the upcoming pip dependency resolver in CI

9

Find more good first issues →

binux / pyspider

Star

A Powerful Spider(Web Crawler) System in Python.

Updated Oct 22, 2020
Python

iawia002 / annie

Star

👾 Fast, simple and clean video downloader

go crawler scraper downloader youtube video tumblr bilibili qq youku iqiyi

Updated Nov 8, 2020
Go

gocolly / colly

Star

Elegant Scraper and Crawler Framework for Golang

go golang crawler scraper framework spider scraping crawling

Updated Oct 13, 2020
Go

jhao104 / proxy_pool

Star

Python爬虫代理IP池(proxy pool)

redis flask crawler spider proxy crawl proxypool ssdb

Updated Nov 4, 2020
Python

codelucas / newspaper

Star

News, full-text, and article metadata extraction in Python 3. Advanced docs:

python crawler scraper news crawling news-aggregator

Updated Nov 12, 2020
Python

code4craft / webmagic

Star

A scalable web crawler framework for Java.

java crawler framework scraping

Updated Oct 27, 2020
Java

shengqiangzhang / examples-of-web-crawlers

Star

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

python crawler spider example selenium multithreading stock wechat taobao pyquery tmall fund agent-pool wechat-report

Updated May 15, 2020
Python

guyueyingmu / avbook

Star

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

crawler scraper laravel database spider magnet-link guzzlehttp magnet adult javbus javlibrary avmoo adult-video

Updated Oct 1, 2020
PHP

s0md3v / Photon

Star

Incredibly fast crawler designed for OSINT.

python crawler osint spider information-gathering

Updated Oct 28, 2020
Python

crawlab

crawlab-team / crawlab

Star

Open

不能使用非crawlab里面mongodb么?

6

52Lau commented Jun 30, 2020

不能使用非crawlab里面mongodb么?

Read more

good first issue

Open

可配置爬虫，在界面上对进行修改，提示保存成功，然而并没有

1

Open

docker安装的任务执行有问题

5

injetlee / Python

Star

Python脚本。模拟登录知乎，爬虫，操作excel，微信公众号，远程开机

python crawler excel wechat

Updated Oct 30, 2020
Python

bda-research / node-crawler

Star

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

nodejs javascript jquery crawler spider cheerio extract-data

Updated Oct 15, 2020
JavaScript

yujiosaka / headless-chrome-crawler

Star

Distributed crawler powered by Headless Chrome

jquery crawler chrome scraper promise scraping crawling chromium headless-chrome puppeteer

Updated Oct 26, 2020
JavaScript

chyroc / WechatSogou

Star

基于搜狗微信搜索的微信公众号爬虫接口

python crawler pypi dict scrapy wechat sogou wsapi

Updated Apr 23, 2020
Python

rmax / scrapy-redis

Star

Redis-based components for Scrapy.

redis crawler distributed scrapy

Updated Sep 21, 2020
Python

SpiderClub / haipproxy

Star

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

redis crawler spider scheduler distributed scrapy high-availability ipproxy

Updated Nov 1, 2019
Python

BruceDone / awesome-crawler

Star

A collection of awesome web crawler,spider in different languages

crawler scraper awesome spider web-crawler web-scraper node-crawler

Updated Oct 29, 2020

ferret

MontFerret / ferret

Star

Open

Add .nextElementSibling and .previousElementSibling

ziflex commented Sep 24, 2020

Add .nextElementSibling and .previousElementSibling properties to DOM Element that point to sibling elements.

nextElementSibling
previousElementSibling

Read more

area/drivers area/drivers/cdp area/drivers/http good first issue

Open

Don't stop script execution on non HTTP 200

Open

Detecting missing links and 404s

2

Find more good first issues →

symfony / dom-crawler

Star

The DomCrawler component eases DOM navigation for HTML and XML documents.

php crawler symfony component symfony-component dom-crawler

Updated Nov 10, 2020
PHP

gaojiuli / toapi

Star

Every web site provides APIs.

python html api flask json crawler web spider toapi

Updated Feb 29, 2020
Python

imWildCat / scylla

Star

Intelligent proxy pool for Humans™ (Maintainer needed)

python crawler scylla python3 proxy-pool

Updated Oct 3, 2020
Python

Arachni / arachni

Star

Web Application Security Scanner Framework

javascript ruby crawler security-audit modular hack dom analysis scanner detection hacking xss audit web-application penetration-testing sql-injection vulnerability-detection arachni scanners

Updated Jan 28, 2020
Ruby

dotnetcore / DotnetSpider

Star

DotnetSpider, a .NET Standard web crawling library. It is lightweight, efficient and fast high-level web crawling & scraping framework

crawler csharp cross-platform dotnetcore distributed

Updated Nov 1, 2020
C#

autoscraper

alirezamika / autoscraper

Star

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

python crawler machine-learning scraper automation ai scraping artificial-intelligence web-scraping scrape webscraping webautomation

Updated Nov 5, 2020
Python

constverum / ProxyBroker

Star

Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭

crawler privacy proxy proxy-server http-proxy socks proxies anonymity anonymous proxypool proxy-list proxy-checker

Updated Aug 20, 2020
Python

DropsDevopsOrg / ECommerceCrawlers

Star

实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评、携程、小米应用商店、安居客、途家民宿❤️

❤️

❤️。微信爬虫展示项目:

Updated Nov 4, 2020
Python

NikolaiT / GoogleScraper

Star

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

python search-engine crawler scraping search-engines search-engine-optimization

Updated Sep 3, 2020
HTML

jae-jae / QueryList

Star

🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

crawler scraper spider querylist

Updated Sep 27, 2020
PHP

xtuhcy / gecco

Star

Easy to use lightweight web crawler（易用的轻量化网络爬虫）

java crawler dynamic jsoup gecco fastjson

Updated Jul 23, 2020
Java

Improve this page

Add a description, image, and links to the crawler topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the crawler topic, visit your repo's landing page and select "manage topics."

You can’t perform that action at this time.