Skip to content

simonhlm/crawler

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

13 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

"""
This is a crawler written by python v3.6
"""
For the setup: 

Before you step up the program, you should ensure you are in python 3 ( or later) 
and has installed below packages
1, requests
2, beautifulsoup
3, Sqlalchmey 

and then, follow up below ones to setup it: 

1, Run the orm.py firstly to create the tables in sqlite. 
2, You can run the lkong_crawler.py to fetch the records
3, Run the test.py to check the records that you stored in the sqlite

# Basic introduce to the main files:

lkong_crawler.py : this is a specifiled crawler to lkong.net. Now it is in progress
orm.py: This is the ORM file & setup of tables.
craw.py : this is the class for the crawler - draft one . only implmenet the basic function. 

About

web crawler by python

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors

Languages