diff --git a/.gitea/workflows/ci.yml b/.gitea/workflows/ci.yml deleted file mode 100644 index 7f2feb4..0000000 --- a/.gitea/workflows/ci.yml +++ /dev/null @@ -1,39 +0,0 @@ -name: CI - -on: - push: - branches: [main, master] - pull_request: - branches: [main, master] - -jobs: - test: - runs-on: debian-latest - strategy: - fail-fast: false - matrix: - python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"] - - steps: - - name: Checkout - uses: actions/checkout@v4 - - - name: Setup Python ${{ matrix.python-version }} - uses: actions/setup-python@v5 - with: - python-version: ${{ matrix.python-version }} - - - name: Install dependencies - run: | - python -m pip install --upgrade pip - pip install pytest requests beautifulsoup4 - # tomli backport for Python 3.8-3.10 (tomllib is stdlib in 3.11+) - pip install tomli || true - - - name: Run tests - run: python -m pytest -q - - - name: Verify scripts run - run: | - python scripts/search.py --version - python scripts/fetch.py --version diff --git a/README.md b/README.md index 15f3364..631821a 100644 --- a/README.md +++ b/README.md @@ -1,88 +1,106 @@ -# SearXNG CLI Toolkit +# SearXNG CLI 技能包 -[![CI](https://git.metona.cn/MetonaTeam/searxng-use-cli/actions/workflows/ci.yml/badge.svg)](https://git.metona.cn/MetonaTeam/searxng-use-cli/actions) [![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) -A privacy-respecting metasearch CLI toolkit that runs against your own SearXNG instance. Zero external dependencies for search (stdlib only), optional `requests` + `beautifulsoup4` for enhanced page fetching. Works with any AI agent (Hermes, Claude Code, Codex, OpenCode, Cursor, Trae) or directly from your terminal. +一个面向 AI Agent 的网络搜索与网页抓取技能包。AI 通过终端调用 CLI 脚本即可获得"搜索网络"和"抓取网页"两项核心能力,无需 API Key,路由通过用户自建的 SearXNG 实例完成。 -## Features +**这不是给人用的工具,而是给 AI 用的技能。** 任何能执行终端命令的 AI Agent(Hermes、Claude Code、Codex、OpenCode、Cursor、Trae 等)均可直接调用。 -**Search & results** -- Multi-instance failover with parallel probing -- Exponential-backoff retry on transient errors (429/5xx/connection) -- `--verify` health-check mode (reachability / JSON-API / latency / POST / engines / auth) -- Cross-engine result deduplication (default on; `--no-dedup` disables) -- Result sorting (`--sort-by {score,date,engine,none}`) -- Domain allowlist/blocklist (`--include-domain` / `--exclude-domain`) -- Batch mode (`--queries-file`) +## 为什么需要这个技能 -**Output formats** -- JSON (default), brief, urls, CSV -- Enhanced Markdown conversion (GFM tables, code blocks, blockquotes, nested lists, definition lists) -- Structured JSON error output for machine-readable failure reporting +AI Agent 在以下场景中需要实时网络信息: +- 查询最新新闻、文档、技术资料 +- 验证事实、获取实时数据 +- 抓取网页内容进行阅读和分析 -**Caching & config** -- SQLite result caching (`--cache-ttl`) with TTL management -- `searxng.toml` config file for defaults; `--config FILE` for explicit loading -- Instance resolution: `-i` → `SEARXNG_INSTANCE` env → config file +传统方案依赖付费搜索 API(Google、Bing)或专有爬虫服务。本技能通过用户自建的 SearXNG 实例(隐私元搜索引擎,聚合 70+ 搜索服务)实现零成本、零 API Key、隐私可控的网络搜索能力。 -**Network & auth** -- Proxy support (`--proxy`) for both search and fetch -- Auth via CLI flag, file, or env var (`--auth-bearer` / `--auth-basic` + `*-file` variants) -- Credentials-file permission warning (POSIX) +## AI 如何使用 -**Engineering** -- Shared `common.py` (unified retry/charset/auth/logging) -- Structured logging (`--verbose` / `--quiet`) -- 155 unit + integration tests with pytest - -## Quick Start +### 搜索网络 ```bash -# Prerequisites: Python 3.8+ (optional: pip install requests beautifulsoup4) - -# Search against YOUR instance (instance URL is required) -python scripts/search.py -q "python asyncio tutorial" -i https://my-searxng.example.com - -# Multiple instances for failover (comma-separated) -python scripts/search.py -q "rust memory safety" \ - -i https://a.example.com,https://b.example.com --format brief - -# Search + auto-fetch top 3 result pages -python scripts/search.py -q "climate policy" -i https://my-searxng.example.com --fetch 3 - -# Fetch a web page -python scripts/fetch.py -u "https://example.com" --extract markdown - -# Skip -i via env var -export SEARXNG_INSTANCE="https://my-searxng.example.com" -python scripts/search.py -q "python asyncio tutorial" +python scripts/search.py -q "搜索词" -i https://your-searxng-instance ``` -## Installation +AI 调用后,stdout 输出 JSON 格式搜索结果,AI 可解析并使用: + +```json +{ + "query": "搜索词", + "results": [ + {"title": "...", "url": "https://...", "content": "摘要...", "engine": "google", "score": 1.0} + ], + "suggestions": ["相关建议词"], + "answers": ["直接答案(如有)"] +} +``` + +### 抓取网页 + +```bash +python scripts/fetch.py -u "https://example.com" --extract markdown +``` + +AI 调用后,stdout 输出网页正文(text/html/markdown 三种格式),AI 可直接阅读分析。 + +### 输入输出约定(AI 必读) + +| 通道 | 内容 | 说明 | +|------|------|------| +| stdout | 数据 | JSON/CSV/文本,AI 解析的唯一来源 | +| stderr | 日志 | 进度、警告、错误,AI 可忽略或用于调试 | +| exit 0 | 成功 | 有结果 | +| exit 1 | 失败 | 致命错误(所有实例不可用、参数错误等) | +| exit 2 | 空结果 | 搜索成功但无结果 | + +**错误处理**:`--format json` 模式下,错误以 JSON 输出到 stdout(非 stderr),格式为 `{"error": "...", "exit_code": 1}`,AI 可程序化捕获。 + +## 部署 + +### 前置条件 + +- Python 3.8+ +- 一个 SearXNG 实例 URL(自建或受信任的实例) + +### 安装 ```bash git clone https://git.metona.cn/MetonaTeam/searxng-use-cli.git -cd searxng-use-cli +``` -# Zero deps — search.py runs on stdlib alone -python scripts/search.py --version +无需安装依赖。`search.py` 仅使用 Python 标准库,开箱即用。 -# Optional: enhanced fetch quality +可选安装(提升 `fetch.py` 抓取质量): +```bash pip install requests beautifulsoup4 ``` -## Usage +### 配置实例 -### search.py — Execute SearXNG Search +AI Agent 无需每次传入实例 URL,支持三种配置方式(优先级从高到低): + +1. **CLI 参数**:`-i https://your-instance` +2. **环境变量**:`export SEARXNG_INSTANCE="https://your-instance"` +3. **配置文件**:`./searxng.toml` 或 `~/.config/searxng-cli/searxng.toml` + +```toml +# searxng.toml +[searxng] +instance = "https://your-searxng.example.com" +``` + +## 参数参考 + +### search.py — 网络搜索 ```bash -python scripts/search.py -q "your query" -i https://your-instance \ +python scripts/search.py -q "查询词" -i https://your-instance \ [--format json|brief|urls|csv] \ [--engines google,bing,brave] \ [--time-range day|month|year|none] \ - [--language en] \ + [--language zh-CN] \ [--sort-by score|date|engine|none] \ [--no-dedup] \ [--max-results 10] \ @@ -97,94 +115,139 @@ python scripts/search.py -q "your query" -i https://your-instance \ [--verbose|-v] [--quiet] ``` -### fetch.py — Fetch & Extract Web Page Content +| 参数 | 说明 | 默认值 | +|------|------|--------| +| `-q / --query` | 搜索词(必填,除非用 `--verify`/`--queries-file`/`--clear-cache`/`--cache-stats`) | — | +| `-i / --instance` | SearXNG 实例 URL,逗号分隔实现故障转移 | — | +| `-f / --format` | 输出格式:json/brief/urls/csv | json | +| `--engines` | 搜索引擎列表 | google,bing,brave,duckduckgo,startpage,wikipedia,wikidata | +| `-t / --time-range` | 时间范围:day/month/year/none | year | +| `-s / --safesearch` | 安全搜索:0/1/2 | 0(关闭) | +| `-l / --language` | 语言代码 | — | +| `-p / --pageno` | 页码 | 1 | +| `--sort-by` | 排序:score/date/engine/none | score(降序) | +| `--no-dedup` | 禁用跨引擎去重 | 默认开启去重 | +| `--max-results` | 限制结果数(去重+排序后截取) | 不限 | +| `--fetch N` | 自动抓取前 N 个结果的网页正文 | 0(不抓取) | +| `--cache-ttl` | 缓存分钟数 | 0(不缓存) | +| `--queries-file` | 批量查询文件(每行一个查询) | — | +| `--include-domain` | 域名白名单 | — | +| `--exclude-domain` | 域名黑名单 | — | +| `--proxy` | 代理 URL | — | +| `--auth-bearer-file` | Bearer Token 文件 | — | +| `--auth-basic-file` | Basic Auth 文件 | — | +| `--verify` | 实例健康检查模式 | — | +| `--config` | 指定配置文件 | 自动发现 | +| `-v / --verbose` | 调试日志 | — | +| `--quiet` | 仅输出警告和错误 | — | + +### fetch.py — 网页抓取 ```bash python scripts/fetch.py -u https://example.com \ --extract text|html|markdown \ [--encoding gbk] \ [--max-size 5242880] \ + [--timeout 15] \ + [--retries 3] \ + [--no-redirect] \ [--proxy http://corp:8080] \ [--auth-bearer-file ~/.token] ``` -## Configuration +| 参数 | 说明 | 默认值 | +|------|------|--------| +| `-u / --url` | 目标 URL(必填) | — | +| `-e / --extract` | 提取模式:text/html/markdown | text | +| `--encoding` | 强制字符编码 | 自动检测 | +| `--max-size` | 最大字节数 | 不限 | +| `--timeout` | 超时秒数 | 15 | +| `--retries` | 重试次数 | 3 | +| `--no-redirect` | 不跟随重定向 | 跟随 | +| `--proxy` | 代理 URL | — | +| `--auth-bearer-file` | Bearer Token 文件 | — | +| `--auth-basic-file` | Basic Auth 文件 | — | -Instance URLs resolve in priority order: -1. `-i / --instance` (comma-separated for failover) -2. `SEARXNG_INSTANCE` environment variable -3. Config file: `./searxng.toml` → `~/.config/searxng-cli/searxng.toml` → `./instances.txt` → `~/.config/searxng-cli/instances.txt` +## 能力清单 -Example `searxng.toml`: -```toml -[searxng] -instance = "https://my-searxng.example.com" -# or: instances = ["https://a.example.com", "https://b.example.com"] -engines = "google,bing,brave,duckduckgo,startpage,wikipedia,wikidata" -time_range = "year" -safesearch = 0 -format = "json" -cache_ttl = 30 -``` +**搜索** +- 多实例故障转移 + 并行探测 +- 指数退避重试(429/5xx/连接错误) +- 跨引擎去重(忽略 utm_*/gclid 等跟踪参数) +- 结果排序(score/date/engine) +- 域名白名单/黑名单 +- 批量查询(`--queries-file`) +- 实例健康检查(`--verify`) -## Testing +**输出** +- JSON(默认,含完整元数据) +- brief(标题+URL+摘要) +- urls(纯 URL 列表) +- CSV(表格导出) +- 结构化 JSON 错误输出 + +**网页抓取** +- text:提取纯文本 +- html:原始 HTML +- markdown:增强 Markdown 转换(GFM 表格、代码块、引用块、嵌套列表、定义列表) + +**缓存** +- SQLite 缓存(`--cache-ttl`),相同查询在 TTL 内跳过网络 +- `--clear-cache` / `--cache-stats` 管理缓存 + +**网络** +- 代理支持(`--proxy`) +- 认证(Bearer/Basic,支持文件和环境变量,避免 shell 历史泄露) +- 凭证文件权限警告(POSIX) + +**工程** +- 共享 `common.py`(统一重试/字符集/认证/日志) +- 结构化日志(`--verbose` / `--quiet`) +- 155 个单元+集成测试 + +## 跨 Agent 兼容性 + +本技能与 Agent 无关——任何能执行终端命令的 AI 均可使用: + +| Agent | 调用方式 | +|-------|---------| +| Hermes | `python scripts/search.py -q "..." -i https://your-instance` | +| Claude Code | 同上,通过终端工具调用 | +| Codex (OpenAI) | 同上 | +| OpenCode | 同上 | +| Cursor | 同上 | +| Trae | 同上 | + +设计要点: +- 零外部依赖(`search.py` 仅标准库) +- 脚本自注入目录到 `sys.path`,可从任意工作目录运行 +- stdout 纯数据,stderr 纯日志 +- 无 Agent 专属 API 调用,纯 CLI,跨平台可移植 + +## 测试 ```bash pip install pytest pytest -q ``` -155 tests cover: cache operations, auth resolution, domain filtering, Markdown conversion, search logic, integration flows, and logging configuration. +155 个测试覆盖:缓存操作、认证解析、域名过滤、Markdown 转换、搜索逻辑、集成流程、日志配置。 -## Project Structure +## 项目结构 ``` ├── scripts/ -│ ├── search.py # Search with multi-instance failover, cache, batch, domain filter -│ ├── fetch.py # Web page fetcher with text/markdown extraction -│ ├── common.py # Shared utilities (auth, retry, charset, logging) -│ ├── cache.py # SQLite-backed result cache -│ └── _config.py # Version + User-Agent constants -├── tests/ # pytest unit + integration tests -├── .gitea/workflows/ # Gitea Actions CI -├── SKILL.md # Full skill documentation (agent-facing) -├── pytest.ini # Test configuration +│ ├── search.py # 搜索(多实例故障转移/缓存/批量/域名过滤) +│ ├── fetch.py # 网页抓取(text/markdown 提取) +│ ├── common.py # 共享工具(认证/重试/字符集/日志) +│ ├── cache.py # SQLite 结果缓存 +│ └── _config.py # 版本号 + User-Agent +├── tests/ # pytest 单元+集成测试 +├── SKILL.md # 完整技能文档(Agent 技能描述) +├── pytest.ini # 测试配置 └── README.md ``` -## Defaults - -| Setting | Default | Flag | -|---------|---------|------| -| Instance | **required** | `-i` / `SEARXNG_INSTANCE` / config | -| Safe search | 0 (off) | `-s` | -| Time range | year | `-t` | -| Output format | json | `-f` | -| Engines | google,bing,brave,duckduckgo,startpage,wikipedia,wikidata | `--engines` | -| Sort | score descending | `--sort-by` | -| Dedup | on | `--no-dedup` | - -## Cross-Agent Compatibility - -These scripts are agent-agnostic — they work with any AI agent that can invoke terminal commands: - -| Agent | How to invoke | -|-------|--------------| -| Hermes | `python scripts/search.py -q "..." -i https://your-instance` | -| Claude Code | Same — call via terminal tool | -| Codex (OpenAI) | Same — call via terminal tool | -| OpenCode | Same — call via terminal tool | -| Cursor | Same — call via terminal tool | -| Trae | Same — call via terminal tool | -| Standalone (human) | Run directly in any terminal | - -Key design decisions for universal compatibility: -- Zero external dependencies (stdlib-only for `search.py`) -- Scripts self-inject their directory into `sys.path` — run from any working directory -- Stdout carries data (JSON/CSV/text), stderr carries progress/warnings -- Exit codes: 0=success, 1=fatal error, 2=no results/empty -- NO agent-specific API calls — purely CLI-based, portable across all agent platforms - -## License +## 许可证 MIT