crawler

Some of the links like cryptography installation in file install.rst are outdated and require an update.
It will be nice to check for other parts of the documentation for the same issue.

Is your feature request related to a problem? Please describe.
Currently, there are services that secure website from automation tools like ferret. Some of them send 405 in response to the DOCUMENT function call that make a ferret script fail with an error even though a page is available (not the original page, but usually a page with the captcha).

Describe the solution you'd like
It

Oct	NOV	Jan
	25
2019	2020	2021

crawler

Here are 4,522 public repositories matching this topic...

scrapy / scrapy

binux / pyspider

iawia002 / annie

gocolly / colly

jhao104 / proxy_pool

codelucas / newspaper

code4craft / webmagic

shengqiangzhang / examples-of-web-crawlers

guyueyingmu / avbook

s0md3v / Photon

crawlab-team / crawlab

injetlee / Python

bda-research / node-crawler

yujiosaka / headless-chrome-crawler

chyroc / WechatSogou

rmax / scrapy-redis

SpiderClub / haipproxy

BruceDone / awesome-crawler

MontFerret / ferret

symfony / dom-crawler

imWildCat / scylla

gaojiuli / toapi

alirezamika / autoscraper

Arachni / arachni

dotnetcore / DotnetSpider

constverum / ProxyBroker

DropsDevopsOrg / ECommerceCrawlers

NikolaiT / GoogleScraper

jae-jae / QueryList

xtuhcy / gecco

Improve this page

Add this topic to your repo

Essential cookies

Always active

Analytics cookies