feat(v2.4.0): 新增 E_BLOCKED 错误码——细分 fetch 403 反爬拦截

真实环境问题: 抓取被墙/反爬站点 (如 baike.baidu.com) 返回 403 时,
classify_error 统一归为 E_AUTH, AI Agent 会误判为凭证问题而做无效的
认证重试。被封锁不是认证失败。

改动:
- common.py: 新增 E_BLOCKED 错误码 + recovery_hint (提示换 URL/镜像/
  用 Wayback 兜底/--exclude-domain); 新增 classify_fetch_error() 与
  _extract_status_code(): fetch 场景 403→E_BLOCKED, 401→E_AUTH,
  其余委托 classify_error (搜索场景 403 仍为 E_AUTH, 不变)
- fetch.py main(): 错误路径改用 classify_fetch_error (替代 classify_error)
- search.py: fetch_page 与 fetch_top_results 线程错误路径同样切换
- 新增 11 个测试 (test_v240_blocked_code.py), 全量 572 测试通过
- 真实环境验证: baike 403 → E_BLOCKED, 正常站点不受影响
- 文档同步 (SKILL.md/README.md 错误码表, 版本号 2.4.0)
This commit is contained in:
2026-08-05 20:24:25 +08:00
parent 4df521dc9d
commit 471818074d
7 changed files with 177 additions and 9 deletions
+5 -1
View File
@@ -1,7 +1,7 @@
---
name: searxng-use-cli
description: Use when you need to search the web via your OWN SearXNG instance (no public-instance discovery). 3 CLI scripts + a shared common.py module — execute privacy-respecting searches against a user-supplied instance (with multi-instance failover, 5xx/429 retry, auto-fetch) or via SEARXNG_INSTANCE env / config file, fetch/extract readable text or markdown from web pages. Zero-config replacement for proprietary search APIs.
version: 2.3.0
version: 2.4.0
author: Metona Team
license: MIT
platforms: [linux, macos, windows]
@@ -250,16 +250,20 @@ In `--format json` mode, errors are emitted as structured JSON on stdout:
AI agents can use `error_code` to programmatically decide recovery strategy, and `recovery_hint` for a ready-to-use actionable suggestion:
> **v2.4.0:** fetch-scenario 403 (WAF/anti-bot blocking) is classified as `E_BLOCKED`, not `E_AUTH` — a blocked page is not a credentials problem, so agents should not waste attempts re-authenticating. `classify_fetch_error()` applies to page fetches (`fetch.py`, `search.py --fetch`); SearXNG instance 403 still maps to `E_AUTH` (correct for instance auth).
| Code | Meaning | recovery_hint (abridged) |
|------|---------|--------------------------|
| `E_CONFIG` | Configuration error (no instance resolved) | Provide `-i`/`SEARXNG_INSTANCE`/config file |
| `E_AUTH` | Authentication failed (401/403) | Verify credentials, check token expiry & permissions |
| `E_BLOCKED` | Fetch blocked by anti-bot/WAF (403, v2.4.0) | NOT a credentials problem — use Wayback fallback (default on), fetch a mirror/other URL, or drop the domain (`--exclude-domain`) |
| `E_NETWORK` | Network error (connection refused, timeout, 5xx, all instances failed) | Retry with backoff, switch instance, check proxy |
| `E_RATE_LIMIT` | Rate limited (429) | Wait and retry, reduce frequency, distribute load |
| `E_PARSE` | Parse error (JSON/HTML parsing failed) | Try different instance, switch `--method` |
| `E_EMPTY` | Empty results (exit code 2) | Refine query, broaden `--time-range`, add `--categories` |
| `E_INPUT` | Input error (bad parameters, file not found) | Check query syntax, flag combinations, file paths |
| `E_INTERNAL` | Internal error (unexpected exception) | Re-run with `--verbose`, report bug |
| `E_BLOCKED` | Fetch blocked by anti-bot/WAF (403, v2.4.0) | Not a credentials issue — Wayback fallback, mirror URL, or `--exclude-domain` |
| `E_UNSUPPORTED_MEDIA` | Unsupported binary media type (PDF/docx/xlsx parse failure or unsupported content type) | No auto hint — try a different URL, or install `pdftotext` for PDF parsing |
### JSON Lines Streaming (`--stream`)