diff --git a/README.md b/README.md index be13603..6c4cd2a 100644 --- a/README.md +++ b/README.md @@ -63,12 +63,14 @@ AI 调用后，stdout 输出网页正文（text/html/markdown 三种格式）， |--------|------|----------------------| | `E_CONFIG` | 配置错误（无实例） | 设置 `-i`/`SEARXNG_INSTANCE`/配置文件 | | `E_AUTH` | 认证失败（401/403） | 检查凭证、token 过期与权限 | +| `E_BLOCKED` | 抓取被站点反爬/WAF 拦截（403，v2.4.0） | 不是凭证问题——用 Wayback 兜底（默认开）、换镜像/其他 URL、或 `--exclude-domain` 排除该域名 | | `E_NETWORK` | 网络错误（连接失败、5xx） | 重试退避、切换实例、检查代理 | | `E_RATE_LIMIT` | 限流（429） | 等待重试、降低频率、分散负载 | | `E_PARSE` | 解析错误 | 切换实例、切换 `--method` | | `E_EMPTY` | 空结果（exit 2） | 调整查询词、扩大 `--time-range`/`--categories` | | `E_INPUT` | 输入错误（参数/文件） | 检查语法、标志组合、文件路径 | | `E_INTERNAL` | 内部错误 | 用 `--verbose` 重跑并报告 | +| `E_BLOCKED` (v2.4.0) | 抓取被反爬/WAF 拦截（403） | 换 URL/镜像，或 `--exclude-domain`；Wayback 兜底默认开启 | | `E_UNSUPPORTED_MEDIA` (v2.2.0) | 不支持的二进制媒体类型（PDF/docx/xlsx 解析失败） | 换 URL，或安装 `pdftotext` 用于 PDF 解析 | ### 流式输出与进度事件（AI 高级用法） @@ -266,6 +268,9 @@ python scripts/fetch.py -u https://example.com \ ## 能力清单 +**v2.4.0 新功能** +- 新增 `E_BLOCKED` 错误码：fetch 场景的 403（WAF/反爬拦截）从 `E_AUTH` 细分出来——被封锁不是凭证问题，AI Agent 不再误判为"需要检查认证"。新增 `classify_fetch_error()` / `_extract_status_code()`（common.py），应用于 `fetch.py` 与 `search.py --fetch`；SearXNG 实例认证的 403 仍映射 `E_AUTH`（搜索场景不变） + **v2.3.0 新功能与修复** - `fetch.py --format json`：结构化 JSON 输出契约。成功 `{status, url, final_url, content_type, extract, truncated, text_length, user_agent}`；失败 `{status, error, error_code, status_code, url}`（对齐 search.py 错误码体系，含 `E_UNSUPPORTED_MEDIA`）。AI Agent 可程序化解析抓取结果，无需再解析裸文本 - `--parallel-queries N`：批量模式并发执行（1-8 workers，输出保持文件顺序），受 AdaptiveThrottle 真实并发门控约束；并发模式下禁用 `--fetch`（嵌套并行抓取不安全）。queries 文件编码自动检测（UTF-8 → GBK 回退） @@ -365,7 +370,7 @@ python scripts/fetch.py -u https://example.com \ - JSON Lines 流式输出（`--stream`，含 `error` 事件类型） - 进度事件（`--progress`，含 `instance_try`/`instance_ok`/`instance_fail`，JSON Lines 到 stderr） - batch 模式统一 schema（`status` 字段区分成功/失败） -- 561 个单元+集成测试 +- 572 个单元+集成测试 ## 跨 Agent 兼容性 @@ -393,7 +398,7 @@ pip install pytest pytest -q ``` -561 个测试覆盖：缓存操作、认证解析、域名过滤、Markdown 转换、搜索逻辑、集成流程、日志配置、HTML 回退、自动抓取、健康检查、输出格式化、实例解析、并行搜索、CLI 端到端、错误码分类、流式输出、进度事件、配置文件认证、schema_version、recovery_hint、batch 统一 schema、--dump-schema、UTF-8 stdout 强制、Windows APPDATA 路径、v2.0.0 浏览器指纹头、WAF 反爬检测（v2.0.1 收窄误判 + title 精准检测）、Wayback Machine 兜底（v2.0.1 修复 HTTP 200 反爬页触发）、自适应限流（v2.1.1 结构化 error_code 检测 429）、v2.1.0 Wayback 共享逻辑（should_try_wayback/build_wayback_url）、被墙站点智能回退（v2.1.1 精确化 baidu 子域，pan/cloud 不再误伤）、--research 研究模式（v2.1.1 跨角度合并去重 + 中英文双语后缀）、--time-range week 支持（v2.1.1）、stdlib gzip/deflate 解压（v2.1.1）、实例引擎挂起检测（v2.1.1 _warn_unresponsive_engines）、v2.3.0 新功能（多页实时查询跟踪、GBK charset 检测与 --encoding、UA 单源、并发槽位门控、fetch JSON 契约、fetch_page title/latency、GBK queries 文件、--parallel-queries 并发批量）。 +572 个测试覆盖：缓存操作、认证解析、域名过滤、Markdown 转换、搜索逻辑、集成流程、日志配置、HTML 回退、自动抓取、健康检查、输出格式化、实例解析、并行搜索、CLI 端到端、错误码分类、流式输出、进度事件、配置文件认证、schema_version、recovery_hint、batch 统一 schema、--dump-schema、UTF-8 stdout 强制、Windows APPDATA 路径、v2.0.0 浏览器指纹头、WAF 反爬检测（v2.0.1 收窄误判 + title 精准检测）、Wayback Machine 兜底（v2.0.1 修复 HTTP 200 反爬页触发）、自适应限流（v2.1.1 结构化 error_code 检测 429）、v2.1.0 Wayback 共享逻辑（should_try_wayback/build_wayback_url）、被墙站点智能回退（v2.1.1 精确化 baidu 子域，pan/cloud 不再误伤）、--research 研究模式（v2.1.1 跨角度合并去重 + 中英文双语后缀）、--time-range week 支持（v2.1.1）、stdlib gzip/deflate 解压（v2.1.1）、实例引擎挂起检测（v2.1.1 _warn_unresponsive_engines）、v2.3.0 新功能（多页实时查询跟踪、GBK charset 检测与 --encoding、UA 单源、并发槽位门控、fetch JSON 契约、fetch_page title/latency、GBK queries 文件、--parallel-queries 并发批量）、v2.4.0 E_BLOCKED 错误码（fetch 403 反爬拦截细分，classify_fetch_error）。 ## 项目结构 diff --git a/SKILL.md b/SKILL.md index 3bb0dd8..be75465 100644 --- a/SKILL.md +++ b/SKILL.md @@ -1,7 +1,7 @@ --- name: searxng-use-cli description: Use when you need to search the web via your OWN SearXNG instance (no public-instance discovery). 3 CLI scripts + a shared common.py module — execute privacy-respecting searches against a user-supplied instance (with multi-instance failover, 5xx/429 retry, auto-fetch) or via SEARXNG_INSTANCE env / config file, fetch/extract readable text or markdown from web pages. Zero-config replacement for proprietary search APIs. -version: 2.3.0 +version: 2.4.0 author: Metona Team license: MIT platforms: [linux, macos, windows] @@ -250,16 +250,20 @@ In `--format json` mode, errors are emitted as structured JSON on stdout: AI agents can use `error_code` to programmatically decide recovery strategy, and `recovery_hint` for a ready-to-use actionable suggestion: +> **v2.4.0:** fetch-scenario 403 (WAF/anti-bot blocking) is classified as `E_BLOCKED`, not `E_AUTH` — a blocked page is not a credentials problem, so agents should not waste attempts re-authenticating. `classify_fetch_error()` applies to page fetches (`fetch.py`, `search.py --fetch`); SearXNG instance 403 still maps to `E_AUTH` (correct for instance auth). + | Code | Meaning | recovery_hint (abridged) | |------|---------|--------------------------| | `E_CONFIG` | Configuration error (no instance resolved) | Provide `-i`/`SEARXNG_INSTANCE`/config file | | `E_AUTH` | Authentication failed (401/403) | Verify credentials, check token expiry & permissions | +| `E_BLOCKED` | Fetch blocked by anti-bot/WAF (403, v2.4.0) | NOT a credentials problem — use Wayback fallback (default on), fetch a mirror/other URL, or drop the domain (`--exclude-domain`) | | `E_NETWORK` | Network error (connection refused, timeout, 5xx, all instances failed) | Retry with backoff, switch instance, check proxy | | `E_RATE_LIMIT` | Rate limited (429) | Wait and retry, reduce frequency, distribute load | | `E_PARSE` | Parse error (JSON/HTML parsing failed) | Try different instance, switch `--method` | | `E_EMPTY` | Empty results (exit code 2) | Refine query, broaden `--time-range`, add `--categories` | | `E_INPUT` | Input error (bad parameters, file not found) | Check query syntax, flag combinations, file paths | | `E_INTERNAL` | Internal error (unexpected exception) | Re-run with `--verbose`, report bug | +| `E_BLOCKED` | Fetch blocked by anti-bot/WAF (403, v2.4.0) | Not a credentials issue — Wayback fallback, mirror URL, or `--exclude-domain` | | `E_UNSUPPORTED_MEDIA` | Unsupported binary media type (PDF/docx/xlsx parse failure or unsupported content type) | No auto hint — try a different URL, or install `pdftotext` for PDF parsing | ### JSON Lines Streaming (`--stream`) diff --git a/scripts/_config.py b/scripts/_config.py index 0a25ddf..ed62d46 100644 --- a/scripts/_config.py +++ b/scripts/_config.py @@ -7,7 +7,7 @@ Retry settings and shared HTTP utilities now live in ``common.py`` so that both ``search.py`` and ``fetch.py`` share one consistent implementation. """ -VERSION = "2.3.0" +VERSION = "2.4.0" SCHEMA_VERSION = "1.0" USER_AGENT = f"searxng-cli/{VERSION}" diff --git a/scripts/common.py b/scripts/common.py index 8c01eab..94119c4 100644 --- a/scripts/common.py +++ b/scripts/common.py @@ -624,6 +624,7 @@ def is_retryable_error(exc: BaseException) -> bool: # 错误码常量（供 search.py / fetch.py 引用） E_CONFIG = "E_CONFIG" E_AUTH = "E_AUTH" +E_BLOCKED = "E_BLOCKED" # fetch 被站点反爬拦截（403），非凭证问题 E_NETWORK = "E_NETWORK" E_RATE_LIMIT = "E_RATE_LIMIT" E_PARSE = "E_PARSE" @@ -639,6 +640,11 @@ RECOVERY_HINTS = { E_AUTH: "Verify --auth-bearer/--auth-basic credentials or " "SEARXNG_BEARER_TOKEN/SEARXNG_BASIC_AUTH env vars. Check token " "expiry and instance access permissions.", + E_BLOCKED: "The site is blocking automated access (WAF / anti-bot / " + "geo-block) — this is NOT a credentials problem; no auth " + "change will help. Use the Wayback Machine fallback (on by " + "default; --no-fallback disables), fetch a mirror or a " + "different URL, or drop the domain with --exclude-domain.", E_NETWORK: "Retry with backoff, or try a different SearXNG instance. " "Check network connectivity, proxy settings, and instance uptime.", E_RATE_LIMIT: "Wait before retrying (exponential backoff). Reduce query " @@ -767,6 +773,44 @@ def classify_error(exc: BaseException) -> str: return E_INTERNAL +def _extract_status_code(exc) -> int: + """从异常及其 ``__cause__`` 链中提取 HTTP 状态码（无则返回 None）。 + + 检查顺序：urllib ``.code`` → requests ``.response.status_code`` → + 消息中的 ``HTTP NNN`` 模式（fetch_url 抛出的 RuntimeError 消息）。 + """ + target = getattr(exc, "__cause__", None) or exc + status = getattr(target, "code", None) + if status is None: + resp = getattr(target, "response", None) + status = getattr(resp, "status_code", None) + if status is None: + m = re.search(r"HTTP (\d{3})", str(exc)) + if m: + status = int(m.group(1)) + return status + + +def classify_fetch_error(exc: BaseException, status_code: int = None) -> str: + """fetch 场景的错误分类：403 → E_BLOCKED，401 → E_AUTH，其余委托 classify_error。 + + 网页抓取（fetch.py / search.py --fetch）遇到的 403 绝大多数是站点反爬 + 拦截（UA/JS 指纹、WAF、区域封锁），**不是凭证错误**。classify_error + 把 401/403 统一归为 E_AUTH 是为 SearXNG 实例认证设计的——若 fetch 也 + 用它，AI Agent 会误判为"需要检查凭证"而去做无效的认证重试。本函数在 + classify_error 基础上仅做 fetch 场景的细分。 + + ``status_code`` 可选：调用方已从异常链提取时可直接传入，避免重复解析。 + """ + if status_code is None: + status_code = _extract_status_code(exc) + if status_code == 403: + return E_BLOCKED + if status_code == 401: + return E_AUTH + return classify_error(exc) + + # ----- Progress event emitter (for --progress flag) ----- # # 当 --progress 启用时，search.py 会调用 emit_progress() 发射结构化事件到 diff --git a/scripts/fetch.py b/scripts/fetch.py index 290feeb..96239d6 100644 --- a/scripts/fetch.py +++ b/scripts/fetch.py @@ -54,7 +54,7 @@ from common import ( build_auth_headers, build_browser_headers, build_wayback_url, - classify_error, + classify_fetch_error, compute_backoff_delay, detect_charset, force_utf8_stdout, @@ -1346,7 +1346,7 @@ Examples: # v2.1.0: Wayback Machine 兜底 # 触发条件：兜底启用 + (错误可恢复 OR 命中被墙站点) fatal_error = str(e) if str(e) else e.__class__.__name__ - fatal_error_code = classify_error(e) + fatal_error_code = classify_fetch_error(e, status_code=status_code) fatal_status_code = status_code error_msg = fatal_error if fallback_enabled and (should_try_wayback(error_msg) or hard_blocked): diff --git a/scripts/search.py b/scripts/search.py index ab27b16..8ccc921 100644 --- a/scripts/search.py +++ b/scripts/search.py @@ -38,6 +38,7 @@ from common import ( build_auth_headers, build_wayback_url, classify_error, + classify_fetch_error, compute_backoff_delay, detect_charset, emit_progress, @@ -906,8 +907,10 @@ def fetch_page(url: str, timeout: int = 10, auth_headers: dict = None, except Exception as e: error_msg = str(e) if str(e) else e.__class__.__name__ # v2.1.0：结构化错误码，让 AdaptiveThrottle 能用 error_code 检测 429 - # 而非字符串匹配（"Too Many Requests" 不含 "429" 会漏判） - error_code = classify_error(e) + # 而非字符串匹配（"Too Many Requests" 不含 "429" 会漏判）。 + # v2.4.0：抓取场景用 classify_fetch_error——403 反爬拦截细分为 + # E_BLOCKED，避免误判为 E_AUTH（凭证问题）。 + error_code = classify_fetch_error(e) # fetch_url 对 PDF/DOCX/XLSX 解析失败不抛异常，而是返回带 error_code # 的 FetchResult——此处必须显式检查，否则失败会被当作成功处理 @@ -1430,7 +1433,7 @@ def fetch_top_results(results: dict, count: int, timeout: int = 10, except Exception as e: u = future_map[future] fetched.append({"url": u, "status": "error", "error": str(e), - "error_code": classify_error(e), + "error_code": classify_fetch_error(e), "text": "", "text_length": 0, "truncated": False, "anti_bot_detected": False, "waf_type": None, "fallback_used": None, diff --git a/tests/test_v240_blocked_code.py b/tests/test_v240_blocked_code.py new file mode 100644 index 0000000..1485845 --- /dev/null +++ b/tests/test_v240_blocked_code.py @@ -0,0 +1,112 @@ +"""Tests for v2.4.0 E_BLOCKED error classification (fetch scenario). + +Real-world motivation: fetching baike.baidu.com etc. returns 403 because +the site blocks automated access (WAF/anti-bot), not because credentials +are wrong. classify_error maps 401/403 -> E_AUTH (correct for SearXNG +instance auth), but for page fetches that misleads agents into checking +credentials. classify_fetch_error narrows fetch-scenario 403 -> E_BLOCKED. + +Covers: + * classify_fetch_error 403/401/5xx/4xx + message-only fallback + * classify_error keeps 403 -> E_AUTH (search scenario untouched) + * fetch_page error_code uses the fetch-scenario classification +""" +import urllib.error +from unittest.mock import patch + +import pytest + +from common import ( + E_AUTH, + E_BLOCKED, + E_INPUT, + E_NETWORK, + classify_error, + classify_fetch_error, +) +import search as search_mod +from search import fetch_page + + +def _http_error(code): + try: + raise urllib.error.HTTPError( + "https://x.example.com", code, "Error", {}, None) + except urllib.error.HTTPError as e: + return e + + +# ===== classify_fetch_error ===== + +def test_fetch_403_is_blocked(): + """Fetch 403 = anti-bot block, NOT auth.""" + assert classify_fetch_error(_http_error(403)) == E_BLOCKED + + +def test_fetch_401_is_auth(): + """Fetch 401 stays E_AUTH (real credentials problem).""" + assert classify_fetch_error(_http_error(401)) == E_AUTH + + +def test_fetch_5xx_delegates_to_network(): + assert classify_fetch_error(_http_error(503)) == E_NETWORK + + +def test_fetch_404_delegates_to_input(): + """4xx other than 403/401 keeps classify_error's E_INPUT.""" + assert classify_fetch_error(_http_error(404)) == E_INPUT + + +def test_fetch_runtime_error_message_fallback(): + """No __cause__ but message contains 'HTTP 403' -> E_BLOCKED.""" + err = RuntimeError("HTTP 403 for https://x.example.com") + assert classify_fetch_error(err) == E_BLOCKED + + +def test_fetch_explicit_status_code_shortcut(): + """Caller-provided status_code avoids re-parsing the exception.""" + assert classify_fetch_error(RuntimeError("no code in message"), + status_code=403) == E_BLOCKED + + +def test_fetch_generic_error_delegates(): + """Connection-level exceptions delegate to classify_error -> E_NETWORK.""" + err = urllib.error.URLError("connection refused") + assert classify_fetch_error(err) == E_NETWORK + + +# ===== classify_error unchanged (search scenario) ===== + +def test_classify_error_403_still_auth(): + """SearXNG instance 403 remains E_AUTH — search scenario untouched.""" + assert classify_error(_http_error(403)) == E_AUTH + + +# ===== fetch_page integration ===== + +def test_fetch_page_403_error_code_blocked(): + """fetch_page reports 403 as E_BLOCKED (not E_AUTH).""" + with patch.object(search_mod, "fetch_url", + side_effect=RuntimeError("HTTP 403 for url")): + r = fetch_page("https://blocked.example.com", fallback_enabled=False) + assert r["status"] == "error" + assert r["error_code"] == E_BLOCKED + assert "403" in r["error"] + + +def test_fetch_page_404_error_code_input(): + """fetch_page 404 stays E_INPUT (not blocked/auth).""" + with patch.object(search_mod, "fetch_url", + side_effect=RuntimeError("HTTP 404 for url")): + r = fetch_page("https://missing.example.com", fallback_enabled=False) + assert r["status"] == "error" + assert r["error_code"] == E_INPUT + + +def test_fetch_page_401_error_code_auth(): + """fetch_page 401 stays E_AUTH (genuine credentials problem).""" + with patch.object(search_mod, "fetch_url", + side_effect=RuntimeError("HTTP 401 for url")): + r = fetch_page("https://private.example.com", fallback_enabled=False) + assert r["status"] == "error" + assert r["error_code"] == E_AUTH