GitHub - weizhang1992/WebSearchEngine-CourseProject-Webcrawler: For Web Search Engine Course

Branches Tags

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
.metadata		.metadata
result_file		result_file
.project		.project
.pydevproject		.pydevproject
explain.txt		explain.txt
readme.txt		readme.txt
urlCrawler.py		urlCrawler.py
urlcheck.py		urlcheck.py
urlcheck.pyc		urlcheck.pyc
urlfilter.py		urlfilter.py
urlfilter.pyc		urlfilter.pyc
urlinitials.py		urlinitials.py
urlinitials.pyc		urlinitials.pyc
urllinks.py		urllinks.py
urllinks.pyc		urllinks.pyc
urlrobot.py		urlrobot.py
urlrobot.pyc		urlrobot.pyc
urlscore.py		urlscore.py
urlscore.pyc		urlscore.pyc

Repository files navigation

(a) go to urlCrawler.py, you will find the code at first fewer lines:
		thequery="software engineer"
		maxpage=500 
(b)change thequery and maxpage to what you want to set

(c)the folder name will be Webcrawler

(d)set you basepath, result file will store at basepath+"/Webcrawler/result_file/"+thequery+"/"  

(e)run python urlCrawler.py at Terminal

(f)check the log or result file

(g)the crawler will be slow at start, because score all links. But will be fast after that.