MetonaTeam/searxng-use-cli
2
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(v1.8.0): 稳定性修复 + AI Agent 体验增强

Browse Source 
稳定性修复:

- 修复 cache.py SQLite 连接泄漏（contextlib.closing 包装）

- 修复 fetch.py requests stream=True 连接泄漏（try/finally resp.close()）

- RETRYABLE_STATUS 新增 403，激活 UA fallback 切换逻辑

- --cache-stats 移至实例解析前，无需实例即可查询

- classify_error 从错误消息提取 HTTP 状态码，正确分类 E_AUTH/E_RATE_LIMIT

- --stream 与 --queries-file 互斥检查，违规报 E_INPUT

- batch 退出码语义统一（0=有结果 / 1=全部错误 / 2=全部空结果）

AI Agent 体验增强:

- 错误码体系完善：E_CONFIG/E_AUTH/E_NETWORK/E_RATE_LIMIT/E_PARSE/E_EMPTY/E_INPUT/E_INTERNAL

- recovery_hint 恢复提示字段，AI Agent 可程序化决策恢复策略

- stream 模式新增 error 事件类型（含 error_code + recovery_hint）

- 进度事件扩展：instance_try/instance_ok/instance_fail

- batch 模式统一 schema（status 字段区分 success/failed）

- JSON 输出含 schema_version 字段确保版本兼容

测试与文档:

- 测试覆盖：330 -> 352

- SKILL.md / README.md 同步更新
This commit is contained in:
thzxx
2026-08-01 19:02:44 +08:00
parent dea899143d
commit fb9b2af45f
12 changed files with 871 additions and 131 deletions
Download Patch File Download Diff File Expand all files Collapse all files
SKILL.md
+56 -22
View File
@@ -1,7 +1,7 @@
---
name: searxng-use-cli
description: Use when you need to search the web via your OWN SearXNG instance (no public-instance discovery). 3 CLI scripts + a shared common.py module — execute privacy-respecting searches against a user-supplied instance (with multi-instance failover, 5xx/429 retry, auto-fetch) or via SEARXNG_INSTANCE env / config file, fetch/extract readable text or markdown from web pages. Zero-config replacement for proprietary search APIs.
version: 1.7.0
version: 1.8.0
author: Metona Team
license: MIT
platforms: [linux, macos, windows]
@@ -188,23 +188,24 @@ In `--format json` mode, errors are emitted as structured JSON on stdout:
{
"error": "All 2 instances failed. Last error: connection refused",
"error_code": "E_NETWORK",
"recovery_hint": "Retry with backoff, or try a different SearXNG instance. Check network connectivity, proxy settings, and instance uptime.",
"exit_code": 1,
"query": "search term"
}
```
AI agents can use `error_code` to programmatically decide recovery strategy:
AI agents can use `error_code` to programmatically decide recovery strategy, and `recovery_hint` for a ready-to-use actionable suggestion:
| Code | Meaning | AI Recovery Strategy |
|------|---------|---------------------|
| `E_CONFIG` | Configuration error (no instance resolved) | Prompt user to set `-i` / `SEARXNG_INSTANCE` / config file |
| `E_AUTH` | Authentication failed (401/403) | Check token/credentials, prompt user to re-authenticate |
| `E_NETWORK` | Network error (connection refused, timeout, 5xx, all instances failed) | Retry with backoff, switch instance or proxy |
| `E_RATE_LIMIT` | Rate limited (429) | Wait and retry with reduced frequency |
| `E_PARSE` | Parse error (JSON/HTML parsing failed) | Check if instance supports JSON, try HTML fallback |
| `E_EMPTY` | Empty results (exit code 2) | Adjust query terms or time range |
| `E_INPUT` | Input error (bad parameters, file not found) | Fix CLI arguments or file paths |
| `E_INTERNAL` | Internal error (unexpected exception) | Report bug with full error message |
| Code | Meaning | recovery_hint (abridged) |
|------|---------|--------------------------|
| `E_CONFIG` | Configuration error (no instance resolved) | Provide `-i`/`SEARXNG_INSTANCE`/config file |
| `E_AUTH` | Authentication failed (401/403) | Verify credentials, check token expiry & permissions |
| `E_NETWORK` | Network error (connection refused, timeout, 5xx, all instances failed) | Retry with backoff, switch instance, check proxy |
| `E_RATE_LIMIT` | Rate limited (429) | Wait and retry, reduce frequency, distribute load |
| `E_PARSE` | Parse error (JSON/HTML parsing failed) | Try different instance, switch `--method` |
| `E_EMPTY` | Empty results (exit code 2) | Refine query, broaden `--time-range`, add `--categories` |
| `E_INPUT` | Input error (bad parameters, file not found) | Check query syntax, flag combinations, file paths |
| `E_INTERNAL` | Internal error (unexpected exception) | Re-run with `--verbose`, report bug |
### JSON Lines Streaming (`--stream`)
@@ -221,14 +222,19 @@ Output format (each line is a separate JSON object):
```
{"type": "result", "result": {"title": "...", "url": "...", "content": "..."}}
{"type": "result", "result": {"title": "...", "url": "...", "content": "..."}}
{"type": "done", "count": 2, "query": "large topic"}
{"type": "done", "schema_version": "1.0", "count": 2, "query": "large topic"}
```
- `type: "result"` — one per search result, emitted as soon as available
- `type: "done"` — terminal event with total count, always emitted last
- Exit code 0 on success, 2 on empty results (done event still emitted)
- `type: "done"` — terminal event with total count and `schema_version`, always emitted last
- `type: "error"` — emitted when the search fails, includes `error_code` and `recovery_hint`:
```
{"type": "error", "error": "All 2 instances failed. Last error: HTTP Error 403", "error_code": "E_AUTH", "recovery_hint": "Verify credentials...", "query": "..."}
```
- Exit code 0 on success, 1 on error, 2 on empty results (done event still emitted)
Only valid with `--format json` (the default).
Only valid with `--format json` (single query mode). Using `--stream` with
`--queries-file` or non-json formats raises `E_INPUT` immediately.
### Progress Events (`--progress`)
@@ -243,6 +249,9 @@ Event types (each on its own line, JSON Lines format on stderr):
```jsonl
{"event": "start", "query": "research topic", "instances": 2}
{"event": "instance_try", "url": "https://instance1.example.com", "attempt": 1}
{"event": "instance_ok", "url": "https://instance1.example.com", "latency": 0.342, "results": 10}
{"event": "instance_fail", "url": "https://instance2.example.com", "error": "HTTP 503", "error_code": "E_NETWORK"}
{"event": "cache_hit", "query": "research topic", "ttl": 30}
{"event": "cache_store", "query": "research topic", "ttl": 30}
{"event": "fetch_start", "count": 3}
@@ -264,10 +273,19 @@ logs are human-readable text.
### Output JSON Schema
The default `--format json` output shape (for reference, not enforced):
The default `--format json` output includes a `schema_version` field so AI
agents can detect breaking changes. Run `--dump-schema` to get the full JSON
Schema document programmatically:
```bash
python scripts/search.py --dump-schema
```
Single query output shape:
```json
{
"schema_version": "1.0",
"query": "search term",
"number_of_results": 10,
"results": [
@@ -294,13 +312,28 @@ The default `--format json` output shape (for reference, not enforced):
"text_length": 12345,
"truncated": false,
"final_url": "https://example.com/final",
"user_agent_used": "searxng-cli/1.7.0"
"user_agent_used": "searxng-cli/1.8.0"
}
],
"fetched_source": "json"
}
```
Batch mode (`--queries-file`) wraps results in a unified schema:
```json
{
"schema_version": "1.0",
"queries": [
{"query": "term1", "status": "ok", "results": {...}},
{"query": "term2", "status": "error", "error": "...", "error_code": "E_NETWORK"}
]
}
```
Each batch entry has a `status` field (`"ok"` or `"error"`). Successful entries
contain `results`; failed entries contain `error` and `error_code`.
Fields marked as optional may be absent. The `fetched` and `fetched_source`
fields only appear when `--fetch N` is used.
@@ -431,7 +464,7 @@ usage: search.py [-h] [--query QUERY] [--instance URL]
[--exclude-domain DOMAINS] [--queries-file FILE]
[--cache-ttl MINUTES] [--clear-cache] [--cache-stats]
[--sort-by {score,date,engine,none}] [--no-dedup]
[--config FILE] [--verbose] [--quiet] [--version]
[--config FILE] [--dump-schema] [--verbose] [--quiet] [--version]
```
**What it does:**
@@ -445,14 +478,14 @@ usage: search.py [-h] [--query QUERY] [--instance URL]
8. **Stable auto-fetch:** `--fetch 3` concurrently downloads top 3 result pages with retry, browser-UA fallback, CAPTCHA detection, and **`fetch.py`'s higher-quality text extractor** (the same engine `fetch.py` uses)
9. **Health-check mode:** `--verify` probes each instance (reachability / JSON-API support / latency / POST support / engine list / auth status) and prints a report, then exits without searching — use it to validate your instance list
10. **Result caching:** `--cache-ttl 30` stores results for 30 min; identical queries within the TTL skip the network entirely. Cache lives at `$SEARXNG_CACHE_DIR` or `~/.cache/searxng-cli/cache.db` (SQLite, WAL mode). `--clear-cache` / `--cache-stats` manage it without searching
11. **Batch mode:** `--queries-file FILE` reads one query per line (blank/`#` lines skipped) and runs them in sequence; output is a JSON array (or one block per query in brief/urls). A failed query is recorded but does not abort the batch
11. **Batch mode:** `--queries-file FILE` reads one query per line (blank/`#` lines skipped) and runs them in sequence; JSON output is `{"schema_version": "1.0", "queries": [{"query":..., "status": "ok"|"error", ...}]}` (or one block per query in brief/urls). A failed query is recorded but does not abort the batch. Exit codes: 0 if any query returned results, 1 if all errored, 2 if all empty
12. **Dedup + sort + domain filter:** After search (and cache), duplicate URLs are collapsed (default; `--no-dedup` disables), results are sorted (`--sort-by`; default: score descending), and then `--include-domain`/`--exclude-domain` filter by domain. Matching is case-insensitive and ignores a leading `www.`; when a domain is in both lists, exclude wins
13. **Proxy & auth:** `--proxy URL` routes both search and fetch through a proxy; `--auth-bearer` / `--auth-basic` (plus `*-file` variants, `searxng.toml` `auth_basic`/`auth_bearer` fields, and `SEARXNG_BEARER_TOKEN` / `SEARXNG_BASIC_AUTH` env vars) supply credentials. Priority: CLI flag > file > config file > env var
14. **Config defaults:** `searxng.toml` may pre-set most flags (engines, categories, language, safesearch, time_range, method, format, sort_by, timeout, max_retries, proxy, cache_ttl, fetch, fetch_timeout, fetch_retries, max_size, auth_basic, auth_bearer); explicit CLI flags always win
15. **Structured errors:** in `--format json` mode, failures print a JSON object `{"error": "...", "exit_code": N, "query": "..."}` to stdout so agents can parse them
15. **Structured errors:** in `--format json` mode, failures print a JSON object `{"error": "...", "error_code": "E_*", "recovery_hint": "...", "exit_code": N, "query": "..."}` to stdout so agents can parse them and decide recovery strategy
**Key options:**
- `--query "your search"` — **required unless** `--verify`, `--queries-file`, `--clear-cache`, or `--cache-stats` is used
- `--query "your search"` — **required unless** `--verify`, `--queries-file`, `--clear-cache`, `--cache-stats`, or `--dump-schema` is used
- `--instance https://searx.example.org` — **required unless** `SEARXNG_INSTANCE` env var or a config file supplies it; comma-separated list enables failover
- `--queries-file FILE` — read queries from a file (one per line; blank/`#` skipped) and run them in sequence; overrides `--query`
- `--engines google,duckduckgo` — restrict to specific search engines (whitespace around commas is auto-stripped; see default list above)
@@ -487,6 +520,7 @@ usage: search.py [-h] [--query QUERY] [--instance URL]
- `--cache-ttl MINUTES` — cache results for N minutes (default: `0` = disabled); identical queries within the TTL skip the network
- `--clear-cache` — delete all cached entries and exit (no search)
- `--cache-stats` — print cache statistics (entries, age, size, path) and exit
- `--dump-schema` — print the JSON Schema for `--format json` output and exit; lets AI agents programmatically discover field names and types without parsing prose docs
- `--auth-bearer TOKEN` — `Authorization: Bearer` header for private instances
- `--auth-bearer-file FILE` — read Bearer token from a file (first non-empty, non-`#` line); also honors `SEARXNG_BEARER_TOKEN` env var
- `--auth-basic USER:PASS` — `Authorization: Basic` header (auto base64-encoded)