Skip to content

IrSent/proxyscrape

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

4 Commits
 
 
 
 
 
 
 
 

Repository files navigation

proxyscrape

Installation

homebrew (https://brew.sh):

$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
$ brew update
$ brew install python3

env:

$ pip3 install virtualenvwrapper
$ export WORKON_HOME=~/envs
$ mkdir -p $WORKON_HOME
$ source /usr/local/bin/virtualenvwrapper.sh
$ mkvirtualenv proxyscrape -p python3
$ cdvirtualenv

repo:

$ git clone https://github.com/IrSent/proxyscrape project/
$ cd project/
$ pip install -r requirements.txt

Usage

$ cd proxy
$ scrapy crawl proxy_spider

Results stored in proxydb sqlite file

About

testing the scrapy lib

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages