docs: README 改为中文并面向 AI 使用场景; 删除 CI 配置
This commit is contained in:
Metona Team
@@ -1,39 +0,0 @@
|
|||||||
name: CI
|
|
||||||
|
|
||||||
on:
|
|
||||||
push:
|
|
||||||
branches: [main, master]
|
|
||||||
pull_request:
|
|
||||||
branches: [main, master]
|
|
||||||
|
|
||||||
jobs:
|
|
||||||
test:
|
|
||||||
runs-on: debian-latest
|
|
||||||
strategy:
|
|
||||||
fail-fast: false
|
|
||||||
matrix:
|
|
||||||
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
|
|
||||||
|
|
||||||
steps:
|
|
||||||
- name: Checkout
|
|
||||||
uses: actions/checkout@v4
|
|
||||||
|
|
||||||
- name: Setup Python ${{ matrix.python-version }}
|
|
||||||
uses: actions/setup-python@v5
|
|
||||||
with:
|
|
||||||
python-version: ${{ matrix.python-version }}
|
|
||||||
|
|
||||||
- name: Install dependencies
|
|
||||||
run: |
|
|
||||||
python -m pip install --upgrade pip
|
|
||||||
pip install pytest requests beautifulsoup4
|
|
||||||
# tomli backport for Python 3.8-3.10 (tomllib is stdlib in 3.11+)
|
|
||||||
pip install tomli || true
|
|
||||||
|
|
||||||
- name: Run tests
|
|
||||||
run: python -m pytest -q
|
|
||||||
|
|
||||||
- name: Verify scripts run
|
|
||||||
run: |
|
|
||||||
python scripts/search.py --version
|
|
||||||
python scripts/fetch.py --version
|
|
||||||
@@ -1,88 +1,106 @@
|
|||||||
# SearXNG CLI Toolkit
|
# SearXNG CLI 技能包
|
||||||
|
|
||||||
[](https://git.metona.cn/MetonaTeam/searxng-use-cli/actions)
|
|
||||||
[](https://www.python.org/downloads/)
|
[](https://www.python.org/downloads/)
|
||||||
[](LICENSE)
|
[](LICENSE)
|
||||||
|
|
||||||
A privacy-respecting metasearch CLI toolkit that runs against your own SearXNG instance. Zero external dependencies for search (stdlib only), optional `requests` + `beautifulsoup4` for enhanced page fetching. Works with any AI agent (Hermes, Claude Code, Codex, OpenCode, Cursor, Trae) or directly from your terminal.
|
一个面向 AI Agent 的网络搜索与网页抓取技能包。AI 通过终端调用 CLI 脚本即可获得"搜索网络"和"抓取网页"两项核心能力，无需 API Key，路由通过用户自建的 SearXNG 实例完成。
|
||||||
|
|
||||||
## Features
|
**这不是给人用的工具，而是给 AI 用的技能。** 任何能执行终端命令的 AI Agent（Hermes、Claude Code、Codex、OpenCode、Cursor、Trae 等）均可直接调用。
|
||||||
|
|
||||||
**Search & results**
|
## 为什么需要这个技能
|
||||||
- Multi-instance failover with parallel probing
|
|
||||||
- Exponential-backoff retry on transient errors (429/5xx/connection)
|
|
||||||
- `--verify` health-check mode (reachability / JSON-API / latency / POST / engines / auth)
|
|
||||||
- Cross-engine result deduplication (default on; `--no-dedup` disables)
|
|
||||||
- Result sorting (`--sort-by {score,date,engine,none}`)
|
|
||||||
- Domain allowlist/blocklist (`--include-domain` / `--exclude-domain`)
|
|
||||||
- Batch mode (`--queries-file`)
|
|
||||||
|
|
||||||
**Output formats**
|
AI Agent 在以下场景中需要实时网络信息：
|
||||||
- JSON (default), brief, urls, CSV
|
- 查询最新新闻、文档、技术资料
|
||||||
- Enhanced Markdown conversion (GFM tables, code blocks, blockquotes, nested lists, definition lists)
|
- 验证事实、获取实时数据
|
||||||
- Structured JSON error output for machine-readable failure reporting
|
- 抓取网页内容进行阅读和分析
|
||||||
|
|
||||||
**Caching & config**
|
传统方案依赖付费搜索 API（Google、Bing）或专有爬虫服务。本技能通过用户自建的 SearXNG 实例（隐私元搜索引擎，聚合 70+ 搜索服务）实现零成本、零 API Key、隐私可控的网络搜索能力。
|
||||||
- SQLite result caching (`--cache-ttl`) with TTL management
|
|
||||||
- `searxng.toml` config file for defaults; `--config FILE` for explicit loading
|
|
||||||
- Instance resolution: `-i` → `SEARXNG_INSTANCE` env → config file
|
|
||||||
|
|
||||||
**Network & auth**
|
## AI 如何使用
|
||||||
- Proxy support (`--proxy`) for both search and fetch
|
|
||||||
- Auth via CLI flag, file, or env var (`--auth-bearer` / `--auth-basic` + `*-file` variants)
|
|
||||||
- Credentials-file permission warning (POSIX)
|
|
||||||
|
|
||||||
**Engineering**
|
### 搜索网络
|
||||||
- Shared `common.py` (unified retry/charset/auth/logging)
|
|
||||||
- Structured logging (`--verbose` / `--quiet`)
|
|
||||||
- 155 unit + integration tests with pytest
|
|
||||||
|
|
||||||
## Quick Start
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Prerequisites: Python 3.8+ (optional: pip install requests beautifulsoup4)
|
python scripts/search.py -q "搜索词" -i https://your-searxng-instance
|
||||||
|
|
||||||
# Search against YOUR instance (instance URL is required)
|
|
||||||
python scripts/search.py -q "python asyncio tutorial" -i https://my-searxng.example.com
|
|
||||||
|
|
||||||
# Multiple instances for failover (comma-separated)
|
|
||||||
python scripts/search.py -q "rust memory safety" \
|
|
||||||
-i https://a.example.com,https://b.example.com --format brief
|
|
||||||
|
|
||||||
# Search + auto-fetch top 3 result pages
|
|
||||||
python scripts/search.py -q "climate policy" -i https://my-searxng.example.com --fetch 3
|
|
||||||
|
|
||||||
# Fetch a web page
|
|
||||||
python scripts/fetch.py -u "https://example.com" --extract markdown
|
|
||||||
|
|
||||||
# Skip -i via env var
|
|
||||||
export SEARXNG_INSTANCE="https://my-searxng.example.com"
|
|
||||||
python scripts/search.py -q "python asyncio tutorial"
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Installation
|
AI 调用后，stdout 输出 JSON 格式搜索结果，AI 可解析并使用：
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"query": "搜索词",
|
||||||
|
"results": [
|
||||||
|
{"title": "...", "url": "https://...", "content": "摘要...", "engine": "google", "score": 1.0}
|
||||||
|
],
|
||||||
|
"suggestions": ["相关建议词"],
|
||||||
|
"answers": ["直接答案（如有）"]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 抓取网页
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python scripts/fetch.py -u "https://example.com" --extract markdown
|
||||||
|
```
|
||||||
|
|
||||||
|
AI 调用后，stdout 输出网页正文（text/html/markdown 三种格式），AI 可直接阅读分析。
|
||||||
|
|
||||||
|
### 输入输出约定（AI 必读）
|
||||||
|
|
||||||
|
| 通道 | 内容 | 说明 |
|
||||||
|
|------|------|------|
|
||||||
|
| stdout | 数据 | JSON/CSV/文本，AI 解析的唯一来源 |
|
||||||
|
| stderr | 日志 | 进度、警告、错误，AI 可忽略或用于调试 |
|
||||||
|
| exit 0 | 成功 | 有结果 |
|
||||||
|
| exit 1 | 失败 | 致命错误（所有实例不可用、参数错误等） |
|
||||||
|
| exit 2 | 空结果 | 搜索成功但无结果 |
|
||||||
|
|
||||||
|
**错误处理**：`--format json` 模式下，错误以 JSON 输出到 stdout（非 stderr），格式为 `{"error": "...", "exit_code": 1}`，AI 可程序化捕获。
|
||||||
|
|
||||||
|
## 部署
|
||||||
|
|
||||||
|
### 前置条件
|
||||||
|
|
||||||
|
- Python 3.8+
|
||||||
|
- 一个 SearXNG 实例 URL（自建或受信任的实例）
|
||||||
|
|
||||||
|
### 安装
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git clone https://git.metona.cn/MetonaTeam/searxng-use-cli.git
|
git clone https://git.metona.cn/MetonaTeam/searxng-use-cli.git
|
||||||
cd searxng-use-cli
|
```
|
||||||
|
|
||||||
# Zero deps — search.py runs on stdlib alone
|
无需安装依赖。`search.py` 仅使用 Python 标准库，开箱即用。
|
||||||
python scripts/search.py --version
|
|
||||||
|
|
||||||
# Optional: enhanced fetch quality
|
可选安装（提升 `fetch.py` 抓取质量）：
|
||||||
|
```bash
|
||||||
pip install requests beautifulsoup4
|
pip install requests beautifulsoup4
|
||||||
```
|
```
|
||||||
|
|
||||||
## Usage
|
### 配置实例
|
||||||
|
|
||||||
### search.py — Execute SearXNG Search
|
AI Agent 无需每次传入实例 URL，支持三种配置方式（优先级从高到低）：
|
||||||
|
|
||||||
|
1. **CLI 参数**：`-i https://your-instance`
|
||||||
|
2. **环境变量**：`export SEARXNG_INSTANCE="https://your-instance"`
|
||||||
|
3. **配置文件**：`./searxng.toml` 或 `~/.config/searxng-cli/searxng.toml`
|
||||||
|
|
||||||
|
```toml
|
||||||
|
# searxng.toml
|
||||||
|
[searxng]
|
||||||
|
instance = "https://your-searxng.example.com"
|
||||||
|
```
|
||||||
|
|
||||||
|
## 参数参考
|
||||||
|
|
||||||
|
### search.py — 网络搜索
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python scripts/search.py -q "your query" -i https://your-instance \
|
python scripts/search.py -q "查询词" -i https://your-instance \
|
||||||
[--format json|brief|urls|csv] \
|
[--format json|brief|urls|csv] \
|
||||||
[--engines google,bing,brave] \
|
[--engines google,bing,brave] \
|
||||||
[--time-range day|month|year|none] \
|
[--time-range day|month|year|none] \
|
||||||
[--language en] \
|
[--language zh-CN] \
|
||||||
[--sort-by score|date|engine|none] \
|
[--sort-by score|date|engine|none] \
|
||||||
[--no-dedup] \
|
[--no-dedup] \
|
||||||
[--max-results 10] \
|
[--max-results 10] \
|
||||||
@@ -97,94 +115,139 @@ python scripts/search.py -q "your query" -i https://your-instance \
|
|||||||
[--verbose|-v] [--quiet]
|
[--verbose|-v] [--quiet]
|
||||||
```
|
```
|
||||||
|
|
||||||
### fetch.py — Fetch & Extract Web Page Content
|
| 参数 | 说明 | 默认值 |
|
||||||
|
|------|------|--------|
|
||||||
|
| `-q / --query` | 搜索词（必填，除非用 `--verify`/`--queries-file`/`--clear-cache`/`--cache-stats`） | — |
|
||||||
|
| `-i / --instance` | SearXNG 实例 URL，逗号分隔实现故障转移 | — |
|
||||||
|
| `-f / --format` | 输出格式：json/brief/urls/csv | json |
|
||||||
|
| `--engines` | 搜索引擎列表 | google,bing,brave,duckduckgo,startpage,wikipedia,wikidata |
|
||||||
|
| `-t / --time-range` | 时间范围：day/month/year/none | year |
|
||||||
|
| `-s / --safesearch` | 安全搜索：0/1/2 | 0（关闭） |
|
||||||
|
| `-l / --language` | 语言代码 | — |
|
||||||
|
| `-p / --pageno` | 页码 | 1 |
|
||||||
|
| `--sort-by` | 排序：score/date/engine/none | score（降序） |
|
||||||
|
| `--no-dedup` | 禁用跨引擎去重 | 默认开启去重 |
|
||||||
|
| `--max-results` | 限制结果数（去重+排序后截取） | 不限 |
|
||||||
|
| `--fetch N` | 自动抓取前 N 个结果的网页正文 | 0（不抓取） |
|
||||||
|
| `--cache-ttl` | 缓存分钟数 | 0（不缓存） |
|
||||||
|
| `--queries-file` | 批量查询文件（每行一个查询） | — |
|
||||||
|
| `--include-domain` | 域名白名单 | — |
|
||||||
|
| `--exclude-domain` | 域名黑名单 | — |
|
||||||
|
| `--proxy` | 代理 URL | — |
|
||||||
|
| `--auth-bearer-file` | Bearer Token 文件 | — |
|
||||||
|
| `--auth-basic-file` | Basic Auth 文件 | — |
|
||||||
|
| `--verify` | 实例健康检查模式 | — |
|
||||||
|
| `--config` | 指定配置文件 | 自动发现 |
|
||||||
|
| `-v / --verbose` | 调试日志 | — |
|
||||||
|
| `--quiet` | 仅输出警告和错误 | — |
|
||||||
|
|
||||||
|
### fetch.py — 网页抓取
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python scripts/fetch.py -u https://example.com \
|
python scripts/fetch.py -u https://example.com \
|
||||||
--extract text|html|markdown \
|
--extract text|html|markdown \
|
||||||
[--encoding gbk] \
|
[--encoding gbk] \
|
||||||
[--max-size 5242880] \
|
[--max-size 5242880] \
|
||||||
|
[--timeout 15] \
|
||||||
|
[--retries 3] \
|
||||||
|
[--no-redirect] \
|
||||||
[--proxy http://corp:8080] \
|
[--proxy http://corp:8080] \
|
||||||
[--auth-bearer-file ~/.token]
|
[--auth-bearer-file ~/.token]
|
||||||
```
|
```
|
||||||
|
|
||||||
## Configuration
|
| 参数 | 说明 | 默认值 |
|
||||||
|
|------|------|--------|
|
||||||
|
| `-u / --url` | 目标 URL（必填） | — |
|
||||||
|
| `-e / --extract` | 提取模式：text/html/markdown | text |
|
||||||
|
| `--encoding` | 强制字符编码 | 自动检测 |
|
||||||
|
| `--max-size` | 最大字节数 | 不限 |
|
||||||
|
| `--timeout` | 超时秒数 | 15 |
|
||||||
|
| `--retries` | 重试次数 | 3 |
|
||||||
|
| `--no-redirect` | 不跟随重定向 | 跟随 |
|
||||||
|
| `--proxy` | 代理 URL | — |
|
||||||
|
| `--auth-bearer-file` | Bearer Token 文件 | — |
|
||||||
|
| `--auth-basic-file` | Basic Auth 文件 | — |
|
||||||
|
|
||||||
Instance URLs resolve in priority order:
|
## 能力清单
|
||||||
1. `-i / --instance` (comma-separated for failover)
|
|
||||||
2. `SEARXNG_INSTANCE` environment variable
|
|
||||||
3. Config file: `./searxng.toml` → `~/.config/searxng-cli/searxng.toml` → `./instances.txt` → `~/.config/searxng-cli/instances.txt`
|
|
||||||
|
|
||||||
Example `searxng.toml`:
|
**搜索**
|
||||||
```toml
|
- 多实例故障转移 + 并行探测
|
||||||
[searxng]
|
- 指数退避重试（429/5xx/连接错误）
|
||||||
instance = "https://my-searxng.example.com"
|
- 跨引擎去重（忽略 utm_*/gclid 等跟踪参数）
|
||||||
# or: instances = ["https://a.example.com", "https://b.example.com"]
|
- 结果排序（score/date/engine）
|
||||||
engines = "google,bing,brave,duckduckgo,startpage,wikipedia,wikidata"
|
- 域名白名单/黑名单
|
||||||
time_range = "year"
|
- 批量查询（`--queries-file`）
|
||||||
safesearch = 0
|
- 实例健康检查（`--verify`）
|
||||||
format = "json"
|
|
||||||
cache_ttl = 30
|
|
||||||
```
|
|
||||||
|
|
||||||
## Testing
|
**输出**
|
||||||
|
- JSON（默认，含完整元数据）
|
||||||
|
- brief（标题+URL+摘要）
|
||||||
|
- urls（纯 URL 列表）
|
||||||
|
- CSV（表格导出）
|
||||||
|
- 结构化 JSON 错误输出
|
||||||
|
|
||||||
|
**网页抓取**
|
||||||
|
- text：提取纯文本
|
||||||
|
- html：原始 HTML
|
||||||
|
- markdown：增强 Markdown 转换（GFM 表格、代码块、引用块、嵌套列表、定义列表）
|
||||||
|
|
||||||
|
**缓存**
|
||||||
|
- SQLite 缓存（`--cache-ttl`），相同查询在 TTL 内跳过网络
|
||||||
|
- `--clear-cache` / `--cache-stats` 管理缓存
|
||||||
|
|
||||||
|
**网络**
|
||||||
|
- 代理支持（`--proxy`）
|
||||||
|
- 认证（Bearer/Basic，支持文件和环境变量，避免 shell 历史泄露）
|
||||||
|
- 凭证文件权限警告（POSIX）
|
||||||
|
|
||||||
|
**工程**
|
||||||
|
- 共享 `common.py`（统一重试/字符集/认证/日志）
|
||||||
|
- 结构化日志（`--verbose` / `--quiet`）
|
||||||
|
- 155 个单元+集成测试
|
||||||
|
|
||||||
|
## 跨 Agent 兼容性
|
||||||
|
|
||||||
|
本技能与 Agent 无关——任何能执行终端命令的 AI 均可使用：
|
||||||
|
|
||||||
|
| Agent | 调用方式 |
|
||||||
|
|-------|---------|
|
||||||
|
| Hermes | `python scripts/search.py -q "..." -i https://your-instance` |
|
||||||
|
| Claude Code | 同上，通过终端工具调用 |
|
||||||
|
| Codex (OpenAI) | 同上 |
|
||||||
|
| OpenCode | 同上 |
|
||||||
|
| Cursor | 同上 |
|
||||||
|
| Trae | 同上 |
|
||||||
|
|
||||||
|
设计要点：
|
||||||
|
- 零外部依赖（`search.py` 仅标准库）
|
||||||
|
- 脚本自注入目录到 `sys.path`，可从任意工作目录运行
|
||||||
|
- stdout 纯数据，stderr 纯日志
|
||||||
|
- 无 Agent 专属 API 调用，纯 CLI，跨平台可移植
|
||||||
|
|
||||||
|
## 测试
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pip install pytest
|
pip install pytest
|
||||||
pytest -q
|
pytest -q
|
||||||
```
|
```
|
||||||
|
|
||||||
155 tests cover: cache operations, auth resolution, domain filtering, Markdown conversion, search logic, integration flows, and logging configuration.
|
155 个测试覆盖：缓存操作、认证解析、域名过滤、Markdown 转换、搜索逻辑、集成流程、日志配置。
|
||||||
|
|
||||||
## Project Structure
|
## 项目结构
|
||||||
|
|
||||||
```
|
```
|
||||||
├── scripts/
|
├── scripts/
|
||||||
│ ├── search.py # Search with multi-instance failover, cache, batch, domain filter
|
│ ├── search.py # 搜索（多实例故障转移/缓存/批量/域名过滤）
|
||||||
│ ├── fetch.py # Web page fetcher with text/markdown extraction
|
│ ├── fetch.py # 网页抓取（text/markdown 提取）
|
||||||
│ ├── common.py # Shared utilities (auth, retry, charset, logging)
|
│ ├── common.py # 共享工具（认证/重试/字符集/日志）
|
||||||
│ ├── cache.py # SQLite-backed result cache
|
│ ├── cache.py # SQLite 结果缓存
|
||||||
│ └── _config.py # Version + User-Agent constants
|
│ └── _config.py # 版本号 + User-Agent
|
||||||
├── tests/ # pytest unit + integration tests
|
├── tests/ # pytest 单元+集成测试
|
||||||
├── .gitea/workflows/ # Gitea Actions CI
|
├── SKILL.md # 完整技能文档（Agent 技能描述）
|
||||||
├── SKILL.md # Full skill documentation (agent-facing)
|
├── pytest.ini # 测试配置
|
||||||
├── pytest.ini # Test configuration
|
|
||||||
└── README.md
|
└── README.md
|
||||||
```
|
```
|
||||||
|
|
||||||
## Defaults
|
## 许可证
|
||||||
|
|
||||||
| Setting | Default | Flag |
|
|
||||||
|---------|---------|------|
|
|
||||||
| Instance | **required** | `-i` / `SEARXNG_INSTANCE` / config |
|
|
||||||
| Safe search | 0 (off) | `-s` |
|
|
||||||
| Time range | year | `-t` |
|
|
||||||
| Output format | json | `-f` |
|
|
||||||
| Engines | google,bing,brave,duckduckgo,startpage,wikipedia,wikidata | `--engines` |
|
|
||||||
| Sort | score descending | `--sort-by` |
|
|
||||||
| Dedup | on | `--no-dedup` |
|
|
||||||
|
|
||||||
## Cross-Agent Compatibility
|
|
||||||
|
|
||||||
These scripts are agent-agnostic — they work with any AI agent that can invoke terminal commands:
|
|
||||||
|
|
||||||
| Agent | How to invoke |
|
|
||||||
|-------|--------------|
|
|
||||||
| Hermes | `python scripts/search.py -q "..." -i https://your-instance` |
|
|
||||||
| Claude Code | Same — call via terminal tool |
|
|
||||||
| Codex (OpenAI) | Same — call via terminal tool |
|
|
||||||
| OpenCode | Same — call via terminal tool |
|
|
||||||
| Cursor | Same — call via terminal tool |
|
|
||||||
| Trae | Same — call via terminal tool |
|
|
||||||
| Standalone (human) | Run directly in any terminal |
|
|
||||||
|
|
||||||
Key design decisions for universal compatibility:
|
|
||||||
- Zero external dependencies (stdlib-only for `search.py`)
|
|
||||||
- Scripts self-inject their directory into `sys.path` — run from any working directory
|
|
||||||
- Stdout carries data (JSON/CSV/text), stderr carries progress/warnings
|
|
||||||
- Exit codes: 0=success, 1=fatal error, 2=no results/empty
|
|
||||||
- NO agent-specific API calls — purely CLI-based, portable across all agent platforms
|
|
||||||
|
|
||||||
## License
|
|
||||||
|
|
||||||
MIT
|
MIT
|
||||||
|
|||||||
Reference in New Issue
Block a user