feat(v2.4.0): 新增 E_BLOCKED 错误码——细分 fetch 403 反爬拦截

真实环境问题: 抓取被墙/反爬站点 (如 baike.baidu.com) 返回 403 时,
classify_error 统一归为 E_AUTH, AI Agent 会误判为凭证问题而做无效的
认证重试。被封锁不是认证失败。

改动:
- common.py: 新增 E_BLOCKED 错误码 + recovery_hint (提示换 URL/镜像/
  用 Wayback 兜底/--exclude-domain); 新增 classify_fetch_error() 与
  _extract_status_code(): fetch 场景 403→E_BLOCKED, 401→E_AUTH,
  其余委托 classify_error (搜索场景 403 仍为 E_AUTH, 不变)
- fetch.py main(): 错误路径改用 classify_fetch_error (替代 classify_error)
- search.py: fetch_page 与 fetch_top_results 线程错误路径同样切换
- 新增 11 个测试 (test_v240_blocked_code.py), 全量 572 测试通过
- 真实环境验证: baike 403 → E_BLOCKED, 正常站点不受影响
- 文档同步 (SKILL.md/README.md 错误码表, 版本号 2.4.0)

README.md

@@ -63,12 +63,14 @@ AI 调用后，stdout 输出网页正文（text/html/markdown 三种格式），
 |--------|------|----------------------|
 | `E_CONFIG` | 配置错误（无实例） | 设置 `-i`/`SEARXNG_INSTANCE`/配置文件 |
 | `E_AUTH` | 认证失败（401/403） | 检查凭证、token 过期与权限 |
+| `E_BLOCKED` | 抓取被站点反爬/WAF 拦截（403，v2.4.0） | 不是凭证问题——用 Wayback 兜底（默认开）、换镜像/其他 URL、或 `--exclude-domain` 排除该域名 |
 | `E_NETWORK` | 网络错误（连接失败、5xx） | 重试退避、切换实例、检查代理 |
 | `E_RATE_LIMIT` | 限流（429） | 等待重试、降低频率、分散负载 |
 | `E_PARSE` | 解析错误 | 切换实例、切换 `--method` |
 | `E_EMPTY` | 空结果（exit 2） | 调整查询词、扩大 `--time-range`/`--categories` |
 | `E_INPUT` | 输入错误（参数/文件） | 检查语法、标志组合、文件路径 |
 | `E_INTERNAL` | 内部错误 | 用 `--verbose` 重跑并报告 |
+| `E_BLOCKED` (v2.4.0) | 抓取被反爬/WAF 拦截（403） | 换 URL/镜像，或 `--exclude-domain`；Wayback 兜底默认开启 |
 | `E_UNSUPPORTED_MEDIA` (v2.2.0) | 不支持的二进制媒体类型（PDF/docx/xlsx 解析失败） | 换 URL，或安装 `pdftotext` 用于 PDF 解析 |
 
 ### 流式输出与进度事件（AI 高级用法）
@@ -266,6 +268,9 @@ python scripts/fetch.py -u https://example.com \
 
 ## 能力清单
 
+**v2.4.0 新功能**
+- 新增 `E_BLOCKED` 错误码：fetch 场景的 403（WAF/反爬拦截）从 `E_AUTH` 细分出来——被封锁不是凭证问题，AI Agent 不再误判为"需要检查认证"。新增 `classify_fetch_error()` / `_extract_status_code()`（common.py），应用于 `fetch.py` 与 `search.py --fetch`；SearXNG 实例认证的 403 仍映射 `E_AUTH`（搜索场景不变）
+
 **v2.3.0 新功能与修复**
 - `fetch.py --format json`：结构化 JSON 输出契约。成功 `{status, url, final_url, content_type, extract, truncated, text_length, user_agent}`；失败 `{status, error, error_code, status_code, url}`（对齐 search.py 错误码体系，含 `E_UNSUPPORTED_MEDIA`）。AI Agent 可程序化解析抓取结果，无需再解析裸文本
 - `--parallel-queries N`：批量模式并发执行（1-8 workers，输出保持文件顺序），受 AdaptiveThrottle 真实并发门控约束；并发模式下禁用 `--fetch`（嵌套并行抓取不安全）。queries 文件编码自动检测（UTF-8 → GBK 回退）
@@ -365,7 +370,7 @@ python scripts/fetch.py -u https://example.com \
 - JSON Lines 流式输出（`--stream`，含 `error` 事件类型）
 - 进度事件（`--progress`，含 `instance_try`/`instance_ok`/`instance_fail`，JSON Lines 到 stderr）
 - batch 模式统一 schema（`status` 字段区分成功/失败）
-- 561 个单元+集成测试
+- 572 个单元+集成测试
 
 ## 跨 Agent 兼容性
 
@@ -393,7 +398,7 @@ pip install pytest
 pytest -q
 ```
 
-561 个测试覆盖：缓存操作、认证解析、域名过滤、Markdown 转换、搜索逻辑、集成流程、日志配置、HTML 回退、自动抓取、健康检查、输出格式化、实例解析、并行搜索、CLI 端到端、错误码分类、流式输出、进度事件、配置文件认证、schema_version、recovery_hint、batch 统一 schema、--dump-schema、UTF-8 stdout 强制、Windows APPDATA 路径、v2.0.0 浏览器指纹头、WAF 反爬检测（v2.0.1 收窄误判 + title 精准检测）、Wayback Machine 兜底（v2.0.1 修复 HTTP 200 反爬页触发）、自适应限流（v2.1.1 结构化 error_code 检测 429）、v2.1.0 Wayback 共享逻辑（should_try_wayback/build_wayback_url）、被墙站点智能回退（v2.1.1 精确化 baidu 子域，pan/cloud 不再误伤）、--research 研究模式（v2.1.1 跨角度合并去重 + 中英文双语后缀）、--time-range week 支持（v2.1.1）、stdlib gzip/deflate 解压（v2.1.1）、实例引擎挂起检测（v2.1.1 _warn_unresponsive_engines）、v2.3.0 新功能（多页实时查询跟踪、GBK charset 检测与 --encoding、UA 单源、并发槽位门控、fetch JSON 契约、fetch_page title/latency、GBK queries 文件、--parallel-queries 并发批量）。
+572 个测试覆盖：缓存操作、认证解析、域名过滤、Markdown 转换、搜索逻辑、集成流程、日志配置、HTML 回退、自动抓取、健康检查、输出格式化、实例解析、并行搜索、CLI 端到端、错误码分类、流式输出、进度事件、配置文件认证、schema_version、recovery_hint、batch 统一 schema、--dump-schema、UTF-8 stdout 强制、Windows APPDATA 路径、v2.0.0 浏览器指纹头、WAF 反爬检测（v2.0.1 收窄误判 + title 精准检测）、Wayback Machine 兜底（v2.0.1 修复 HTTP 200 反爬页触发）、自适应限流（v2.1.1 结构化 error_code 检测 429）、v2.1.0 Wayback 共享逻辑（should_try_wayback/build_wayback_url）、被墙站点智能回退（v2.1.1 精确化 baidu 子域，pan/cloud 不再误伤）、--research 研究模式（v2.1.1 跨角度合并去重 + 中英文双语后缀）、--time-range week 支持（v2.1.1）、stdlib gzip/deflate 解压（v2.1.1）、实例引擎挂起检测（v2.1.1 _warn_unresponsive_engines）、v2.3.0 新功能（多页实时查询跟踪、GBK charset 检测与 --encoding、UA 单源、并发槽位门控、fetch JSON 契约、fetch_page title/latency、GBK queries 文件、--parallel-queries 并发批量）、v2.4.0 E_BLOCKED 错误码（fetch 403 反爬拦截细分，classify_fetch_error）。
 
 ## 项目结构
 

SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: searxng-use-cli
 description: Use when you need to search the web via your OWN SearXNG instance (no public-instance discovery). 3 CLI scripts + a shared common.py module — execute privacy-respecting searches against a user-supplied instance (with multi-instance failover, 5xx/429 retry, auto-fetch) or via SEARXNG_INSTANCE env / config file, fetch/extract readable text or markdown from web pages. Zero-config replacement for proprietary search APIs.
-version: 2.3.0
+version: 2.4.0
 author: Metona Team
 license: MIT
 platforms: [linux, macos, windows]
@@ -250,16 +250,20 @@ In `--format json` mode, errors are emitted as structured JSON on stdout:
 
 AI agents can use `error_code` to programmatically decide recovery strategy, and `recovery_hint` for a ready-to-use actionable suggestion:
 
+> **v2.4.0:** fetch-scenario 403 (WAF/anti-bot blocking) is classified as `E_BLOCKED`, not `E_AUTH` — a blocked page is not a credentials problem, so agents should not waste attempts re-authenticating. `classify_fetch_error()` applies to page fetches (`fetch.py`, `search.py --fetch`); SearXNG instance 403 still maps to `E_AUTH` (correct for instance auth).
+
 | Code | Meaning | recovery_hint (abridged) |
 |------|---------|--------------------------|
 | `E_CONFIG` | Configuration error (no instance resolved) | Provide `-i`/`SEARXNG_INSTANCE`/config file |
 | `E_AUTH` | Authentication failed (401/403) | Verify credentials, check token expiry & permissions |
+| `E_BLOCKED` | Fetch blocked by anti-bot/WAF (403, v2.4.0) | NOT a credentials problem — use Wayback fallback (default on), fetch a mirror/other URL, or drop the domain (`--exclude-domain`) |
 | `E_NETWORK` | Network error (connection refused, timeout, 5xx, all instances failed) | Retry with backoff, switch instance, check proxy |
 | `E_RATE_LIMIT` | Rate limited (429) | Wait and retry, reduce frequency, distribute load |
 | `E_PARSE` | Parse error (JSON/HTML parsing failed) | Try different instance, switch `--method` |
 | `E_EMPTY` | Empty results (exit code 2) | Refine query, broaden `--time-range`, add `--categories` |
 | `E_INPUT` | Input error (bad parameters, file not found) | Check query syntax, flag combinations, file paths |
 | `E_INTERNAL` | Internal error (unexpected exception) | Re-run with `--verbose`, report bug |
+| `E_BLOCKED` | Fetch blocked by anti-bot/WAF (403, v2.4.0) | Not a credentials issue — Wayback fallback, mirror URL, or `--exclude-domain` |
 | `E_UNSUPPORTED_MEDIA` | Unsupported binary media type (PDF/docx/xlsx parse failure or unsupported content type) | No auto hint — try a different URL, or install `pdftotext` for PDF parsing |
 
 ### JSON Lines Streaming (`--stream`)

scripts/_config.py

@@ -7,7 +7,7 @@ Retry settings and shared HTTP utilities now live in ``common.py`` so that
 both ``search.py`` and ``fetch.py`` share one consistent implementation.
 """
 
-VERSION = "2.3.0"
+VERSION = "2.4.0"
 SCHEMA_VERSION = "1.0"
 USER_AGENT = f"searxng-cli/{VERSION}"
 

scripts/common.py

@@ -624,6 +624,7 @@ def is_retryable_error(exc: BaseException) -> bool:
 # 错误码常量（供 search.py / fetch.py 引用）
 E_CONFIG = "E_CONFIG"
 E_AUTH = "E_AUTH"
+E_BLOCKED = "E_BLOCKED"  # fetch 被站点反爬拦截（403），非凭证问题
 E_NETWORK = "E_NETWORK"
 E_RATE_LIMIT = "E_RATE_LIMIT"
 E_PARSE = "E_PARSE"
@@ -639,6 +640,11 @@ RECOVERY_HINTS = {
     E_AUTH: "Verify --auth-bearer/--auth-basic credentials or "
             "SEARXNG_BEARER_TOKEN/SEARXNG_BASIC_AUTH env vars. Check token "
             "expiry and instance access permissions.",
+    E_BLOCKED: "The site is blocking automated access (WAF / anti-bot / "
+               "geo-block) — this is NOT a credentials problem; no auth "
+               "change will help. Use the Wayback Machine fallback (on by "
+               "default; --no-fallback disables), fetch a mirror or a "
+               "different URL, or drop the domain with --exclude-domain.",
     E_NETWORK: "Retry with backoff, or try a different SearXNG instance. "
                "Check network connectivity, proxy settings, and instance uptime.",
     E_RATE_LIMIT: "Wait before retrying (exponential backoff). Reduce query "
@@ -767,6 +773,44 @@ def classify_error(exc: BaseException) -> str:
     return E_INTERNAL
 
 
+def _extract_status_code(exc) -> int:
+    """从异常及其 ``__cause__`` 链中提取 HTTP 状态码（无则返回 None）。
+
+    检查顺序：urllib ``.code`` → requests ``.response.status_code`` →
+    消息中的 ``HTTP NNN`` 模式（fetch_url 抛出的 RuntimeError 消息）。
+    """
+    target = getattr(exc, "__cause__", None) or exc
+    status = getattr(target, "code", None)
+    if status is None:
+        resp = getattr(target, "response", None)
+        status = getattr(resp, "status_code", None)
+    if status is None:
+        m = re.search(r"HTTP (\d{3})", str(exc))
+        if m:
+            status = int(m.group(1))
+    return status
+
+
+def classify_fetch_error(exc: BaseException, status_code: int = None) -> str:
+    """fetch 场景的错误分类：403 → E_BLOCKED，401 → E_AUTH，其余委托 classify_error。
+
+    网页抓取（fetch.py / search.py --fetch）遇到的 403 绝大多数是站点反爬
+    拦截（UA/JS 指纹、WAF、区域封锁），**不是凭证错误**。classify_error
+    把 401/403 统一归为 E_AUTH 是为 SearXNG 实例认证设计的——若 fetch 也
+    用它，AI Agent 会误判为"需要检查凭证"而去做无效的认证重试。本函数在
+    classify_error 基础上仅做 fetch 场景的细分。
+
+    ``status_code`` 可选：调用方已从异常链提取时可直接传入，避免重复解析。
+    """
+    if status_code is None:
+        status_code = _extract_status_code(exc)
+    if status_code == 403:
+        return E_BLOCKED
+    if status_code == 401:
+        return E_AUTH
+    return classify_error(exc)
+
+
 # ----- Progress event emitter (for --progress flag) -----
 #
 # 当 --progress 启用时，search.py 会调用 emit_progress() 发射结构化事件到

scripts/fetch.py

@@ -54,7 +54,7 @@ from common import (
     build_auth_headers,
     build_browser_headers,
     build_wayback_url,
-    classify_error,
+    classify_fetch_error,
     compute_backoff_delay,
     detect_charset,
     force_utf8_stdout,
@@ -1346,7 +1346,7 @@ Examples:
         # v2.1.0: Wayback Machine 兜底
         # 触发条件：兜底启用 + (错误可恢复 OR 命中被墙站点)
         fatal_error = str(e) if str(e) else e.__class__.__name__
-        fatal_error_code = classify_error(e)
+        fatal_error_code = classify_fetch_error(e, status_code=status_code)
         fatal_status_code = status_code
         error_msg = fatal_error
         if fallback_enabled and (should_try_wayback(error_msg) or hard_blocked):

scripts/search.py

@@ -38,6 +38,7 @@ from common import (
     build_auth_headers,
     build_wayback_url,
     classify_error,
+    classify_fetch_error,
     compute_backoff_delay,
     detect_charset,
     emit_progress,
@@ -906,8 +907,10 @@ def fetch_page(url: str, timeout: int = 10, auth_headers: dict = None,
     except Exception as e:
         error_msg = str(e) if str(e) else e.__class__.__name__
         # v2.1.0：结构化错误码，让 AdaptiveThrottle 能用 error_code 检测 429
-        # 而非字符串匹配（"Too Many Requests" 不含 "429" 会漏判）
+        # 而非字符串匹配（"Too Many Requests" 不含 "429" 会漏判）。
-        error_code = classify_error(e)
+        # v2.4.0：抓取场景用 classify_fetch_error——403 反爬拦截细分为
+        # E_BLOCKED，避免误判为 E_AUTH（凭证问题）。
+        error_code = classify_fetch_error(e)
 
     # fetch_url 对 PDF/DOCX/XLSX 解析失败不抛异常，而是返回带 error_code
     # 的 FetchResult——此处必须显式检查，否则失败会被当作成功处理
@@ -1430,7 +1433,7 @@ def fetch_top_results(results: dict, count: int, timeout: int = 10,
             except Exception as e:
                 u = future_map[future]
                 fetched.append({"url": u, "status": "error", "error": str(e),
-                                "error_code": classify_error(e),
+                                "error_code": classify_fetch_error(e),
                                 "text": "", "text_length": 0, "truncated": False,
                                 "anti_bot_detected": False, "waf_type": None,
                                 "fallback_used": None,

tests/test_v240_blocked_code.py

@@ -0,0 +1,112 @@
+"""Tests for v2.4.0 E_BLOCKED error classification (fetch scenario).
+
+Real-world motivation: fetching baike.baidu.com etc. returns 403 because
+the site blocks automated access (WAF/anti-bot), not because credentials
+are wrong. classify_error maps 401/403 -> E_AUTH (correct for SearXNG
+instance auth), but for page fetches that misleads agents into checking
+credentials. classify_fetch_error narrows fetch-scenario 403 -> E_BLOCKED.
+
+Covers:
+  * classify_fetch_error 403/401/5xx/4xx + message-only fallback
+  * classify_error keeps 403 -> E_AUTH (search scenario untouched)
+  * fetch_page error_code uses the fetch-scenario classification
+"""
+import urllib.error
+from unittest.mock import patch
+
+import pytest
+
+from common import (
+    E_AUTH,
+    E_BLOCKED,
+    E_INPUT,
+    E_NETWORK,
+    classify_error,
+    classify_fetch_error,
+)
+import search as search_mod
+from search import fetch_page
+
+
+def _http_error(code):
+    try:
+        raise urllib.error.HTTPError(
+            "https://x.example.com", code, "Error", {}, None)
+    except urllib.error.HTTPError as e:
+        return e
+
+
+# ===== classify_fetch_error =====
+
+def test_fetch_403_is_blocked():
+    """Fetch 403 = anti-bot block, NOT auth."""
+    assert classify_fetch_error(_http_error(403)) == E_BLOCKED
+
+
+def test_fetch_401_is_auth():
+    """Fetch 401 stays E_AUTH (real credentials problem)."""
+    assert classify_fetch_error(_http_error(401)) == E_AUTH
+
+
+def test_fetch_5xx_delegates_to_network():
+    assert classify_fetch_error(_http_error(503)) == E_NETWORK
+
+
+def test_fetch_404_delegates_to_input():
+    """4xx other than 403/401 keeps classify_error's E_INPUT."""
+    assert classify_fetch_error(_http_error(404)) == E_INPUT
+
+
+def test_fetch_runtime_error_message_fallback():
+    """No __cause__ but message contains 'HTTP 403' -> E_BLOCKED."""
+    err = RuntimeError("HTTP 403 for https://x.example.com")
+    assert classify_fetch_error(err) == E_BLOCKED
+
+
+def test_fetch_explicit_status_code_shortcut():
+    """Caller-provided status_code avoids re-parsing the exception."""
+    assert classify_fetch_error(RuntimeError("no code in message"),
+                                status_code=403) == E_BLOCKED
+
+
+def test_fetch_generic_error_delegates():
+    """Connection-level exceptions delegate to classify_error -> E_NETWORK."""
+    err = urllib.error.URLError("connection refused")
+    assert classify_fetch_error(err) == E_NETWORK
+
+
+# ===== classify_error unchanged (search scenario) =====
+
+def test_classify_error_403_still_auth():
+    """SearXNG instance 403 remains E_AUTH — search scenario untouched."""
+    assert classify_error(_http_error(403)) == E_AUTH
+
+
+# ===== fetch_page integration =====
+
+def test_fetch_page_403_error_code_blocked():
+    """fetch_page reports 403 as E_BLOCKED (not E_AUTH)."""
+    with patch.object(search_mod, "fetch_url",
+                      side_effect=RuntimeError("HTTP 403 for url")):
+        r = fetch_page("https://blocked.example.com", fallback_enabled=False)
+    assert r["status"] == "error"
+    assert r["error_code"] == E_BLOCKED
+    assert "403" in r["error"]
+
+
+def test_fetch_page_404_error_code_input():
+    """fetch_page 404 stays E_INPUT (not blocked/auth)."""
+    with patch.object(search_mod, "fetch_url",
+                      side_effect=RuntimeError("HTTP 404 for url")):
+        r = fetch_page("https://missing.example.com", fallback_enabled=False)
+    assert r["status"] == "error"
+    assert r["error_code"] == E_INPUT
+
+
+def test_fetch_page_401_error_code_auth():
+    """fetch_page 401 stays E_AUTH (genuine credentials problem)."""
+    with patch.object(search_mod, "fetch_url",
+                      side_effect=RuntimeError("HTTP 401 for url")):
+        r = fetch_page("https://private.example.com", fallback_enabled=False)
+    assert r["status"] == "error"
+    assert r["error_code"] == E_AUTH