GitHub - CollComm/Tuan: Tuan Analysis

Branches Tags

Name		Name	Last commit message	Last commit date
Latest commit History 23 Commits
analysis		analysis
lib		lib
static		static
templates		templates
.gitignore		.gitignore
MonitorHandler.py		MonitorHandler.py
README		README
archiver.py		archiver.py
async_pipeline.py		async_pipeline.py
carrier.py		carrier.py
check_city_names.py		check_city_names.py
check_config.py		check_config.py
city_list		city_list
compress.py		compress.py
crawl_config		crawl_config
database.py		database.py
dianping_crawl.py		dianping_crawl.py
dianping_parse.py		dianping_parse.py
dida_crawl.py		dida_crawl.py
dida_parse.py		dida_parse.py
extract_shop_info.py		extract_shop_info.py
find_deal_num.py		find_deal_num.py
ftuan_crawl.py		ftuan_crawl.py
ftuan_parse.py		ftuan_parse.py
lashou_crawl.py		lashou_crawl.py
lashou_parse.py		lashou_parse.py
main.py		main.py
manzuo_crawl.py		manzuo_crawl.py
manzuo_parse.py		manzuo_parse.py
meituan_crawl.py		meituan_crawl.py
meituan_parse.py		meituan_parse.py
miner.py		miner.py
monitor_tuan.py		monitor_tuan.py
nuomi_crawl.py		nuomi_crawl.py
nuomi_parse.py		nuomi_parse.py
parse_basic_numeric_info.py		parse_basic_numeric_info.py
pipeline.py		pipeline.py
s3archiver.py		s3archiver.py
tuan_http.py		tuan_http.py
wowo_crawl.py		wowo_crawl.py
wowo_parse.py		wowo_parse.py
wuba_crawl.py		wuba_crawl.py
wuba_parse.py		wuba_parse.py

Repository files navigation

Note:
manzuo.com and ftuan.com do not have shop information.
58.com does not have shop info as well... although it has shop address, but it's pretty vague.

to crawl deal information from lashou.com, python pipeline.py -s lashou
for other websites, we can use python async_pipeline.py -s meituan

Run the data collection and analysis pipeline:
python main.py
To end the process, create a file under the current directory called "main_term_sig"