Skip to content

Solitude37/Crawler

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

8 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Crawler

Environment

  • Python 3.12.4
  • playwright 1.52.0
  • fastapi 0.115.12

Interface

  • 会话管理
    • POST /sessions 创建一个新的会话并导航到指定URL
    • DELETE /sessions/{session_id} 关闭指定ID的会话
  • 静态信息获取
    • GET /sessions/{session_id}/static/dom 获取页面的静态html信息
    • GET /sessions/{session_id}/static/interactive_elements 获取页面的可交互元素列表
  • 动态交互
    • POST /sessions/{session_id}/interact 对指定元素施加某个动作,返回DOM变化日志和事件日志

Directory Structure

.
│  main.py     # 项目入口 
│  README.md
│
├─api          # 接口定义
│  └─v1
│      │  interact.py
│      │  sessions.py
│      └─ static.py
│      
│
├─test         # 测试模块及日志结果
│  │  test_api.py
│  │
│  ├─task3-click_search_button
│  │
│  ├─task4-input_text
│  │
│  └─task5-press_enter
│
└─utils        # 工具代码
   │  compress.py
   │  Compression_v5.py
   │  dom_analyzer.py
   │  listeners.py
   │
   ├─scripts   # 脚本
   │      buildDomTree.js
   │      event_listener.js
   └─     mutation_observer.js

Run

# 启动API服务器
uvicorn main:app  # 默认运行在127.0.0.1:8000

# 运行测试代码
pytest test/test_api.py -v -s

About

Crawler api for LLM

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors