feat(v2.4.0): 新增 E_BLOCKED 错误码——细分 fetch 403 反爬拦截

真实环境问题: 抓取被墙/反爬站点 (如 baike.baidu.com) 返回 403 时,
classify_error 统一归为 E_AUTH, AI Agent 会误判为凭证问题而做无效的
认证重试。被封锁不是认证失败。

改动:
- common.py: 新增 E_BLOCKED 错误码 + recovery_hint (提示换 URL/镜像/
  用 Wayback 兜底/--exclude-domain); 新增 classify_fetch_error() 与
  _extract_status_code(): fetch 场景 403→E_BLOCKED, 401→E_AUTH,
  其余委托 classify_error (搜索场景 403 仍为 E_AUTH, 不变)
- fetch.py main(): 错误路径改用 classify_fetch_error (替代 classify_error)
- search.py: fetch_page 与 fetch_top_results 线程错误路径同样切换
- 新增 11 个测试 (test_v240_blocked_code.py), 全量 572 测试通过
- 真实环境验证: baike 403 → E_BLOCKED, 正常站点不受影响
- 文档同步 (SKILL.md/README.md 错误码表, 版本号 2.4.0)
This commit is contained in:
2026-08-05 20:24:25 +08:00
parent 4df521dc9d
commit 471818074d
7 changed files with 177 additions and 9 deletions
+44
View File
@@ -624,6 +624,7 @@ def is_retryable_error(exc: BaseException) -> bool:
# 错误码常量(供 search.py / fetch.py 引用)
E_CONFIG = "E_CONFIG"
E_AUTH = "E_AUTH"
E_BLOCKED = "E_BLOCKED" # fetch 被站点反爬拦截(403),非凭证问题
E_NETWORK = "E_NETWORK"
E_RATE_LIMIT = "E_RATE_LIMIT"
E_PARSE = "E_PARSE"
@@ -639,6 +640,11 @@ RECOVERY_HINTS = {
E_AUTH: "Verify --auth-bearer/--auth-basic credentials or "
"SEARXNG_BEARER_TOKEN/SEARXNG_BASIC_AUTH env vars. Check token "
"expiry and instance access permissions.",
E_BLOCKED: "The site is blocking automated access (WAF / anti-bot / "
"geo-block) — this is NOT a credentials problem; no auth "
"change will help. Use the Wayback Machine fallback (on by "
"default; --no-fallback disables), fetch a mirror or a "
"different URL, or drop the domain with --exclude-domain.",
E_NETWORK: "Retry with backoff, or try a different SearXNG instance. "
"Check network connectivity, proxy settings, and instance uptime.",
E_RATE_LIMIT: "Wait before retrying (exponential backoff). Reduce query "
@@ -767,6 +773,44 @@ def classify_error(exc: BaseException) -> str:
return E_INTERNAL
def _extract_status_code(exc) -> int:
"""从异常及其 ``__cause__`` 链中提取 HTTP 状态码(无则返回 None)。
检查顺序:urllib ``.code`` → requests ``.response.status_code`` →
消息中的 ``HTTP NNN`` 模式(fetch_url 抛出的 RuntimeError 消息)。
"""
target = getattr(exc, "__cause__", None) or exc
status = getattr(target, "code", None)
if status is None:
resp = getattr(target, "response", None)
status = getattr(resp, "status_code", None)
if status is None:
m = re.search(r"HTTP (\d{3})", str(exc))
if m:
status = int(m.group(1))
return status
def classify_fetch_error(exc: BaseException, status_code: int = None) -> str:
"""fetch 场景的错误分类:403 → E_BLOCKED401 → E_AUTH,其余委托 classify_error。
网页抓取(fetch.py / search.py --fetch)遇到的 403 绝大多数是站点反爬
拦截(UA/JS 指纹、WAF、区域封锁),**不是凭证错误**。classify_error
把 401/403 统一归为 E_AUTH 是为 SearXNG 实例认证设计的——若 fetch 也
用它,AI Agent 会误判为"需要检查凭证"而去做无效的认证重试。本函数在
classify_error 基础上仅做 fetch 场景的细分。
``status_code`` 可选:调用方已从异常链提取时可直接传入,避免重复解析。
"""
if status_code is None:
status_code = _extract_status_code(exc)
if status_code == 403:
return E_BLOCKED
if status_code == 401:
return E_AUTH
return classify_error(exc)
# ----- Progress event emitter (for --progress flag) -----
#
# 当 --progress 启用时,search.py 会调用 emit_progress() 发射结构化事件到