feat(v2.5.0): Sec-Ch-Ua 头修复 + 结构化提取 + token 预算 + 正文去重 + 连接复用
正确性修复: - 修复 Sec-Ch-Ua 构造 bug: 原实现产出 ""Not_A Brand";v="99"" 双重引号 畸形头(Chrome/Edge 两路径), 严格校验的 WAF 会忽略; 改为品牌数组拼接 - search 403 快速失败: 实例级 403 不再退避重试(~10.5s 空等), 立即 failover - AdaptiveThrottle: --throttle-failure-threshold 0 现为真正禁用语义 - number_of_results 缺失时用 len(results) 兜底(JSON/HTML 路径契约对齐) - 版本对齐: pyproject.toml 与 _config.py 同步 2.5.0 AI 代理体验: - fetch.py --extract json: 结构化骨架(title/meta/headings/links/images) - fetch.py --max-chars N: 提取后语义级截断(区别于 --max-size 字节截断) - search --fetch-total-chars N: --fetch 全局字符预算, 耗尽后 status=skipped - --dedup-fetched-content: 抓取正文 SimHash 去重, status=duplicate - --progress 新增 angle_start/ok/fail + fetch_skip/fetch_duplicate 事件 - fetch.py 补齐 --log-format json + --dump-schema - CSV 媒体列自适应(images/videos 类别自动追加媒体字段列) - --dry-run 批量模式打印实际查询列表 - search 连接复用: requests 可用时走模块级 Session(连接池) 工程治理: - 新增 scripts/release_check.py 发布一致性检查(版本/错误码表漂移) - 新增 tests/test_v250_features.py 46+4 个回归测试(全量 622 通过) - tests/conftest.py: autouse fixture 强制 stdlib 路径(本机有 requests 时 既有 urllib mock 测试不失效)
This commit is contained in:
@@ -268,6 +268,22 @@ python scripts/fetch.py -u https://example.com \
|
||||
|
||||
## 能力清单
|
||||
|
||||
**v2.5.0 新功能与修复**
|
||||
- 修复核心反爬头 `Sec-Ch-Ua` 构造 bug：原实现产出 `""Not_A Brand";v="99""` 畸形头（双重引号），Chrome/Edge 两条路径都存在，严格校验的 WAF 会忽略；现改为品牌数组统一拼接，产出 `"Not_A Brand";v="99", "Chromium";v="140", "Google Chrome";v="140"` 合法格式（真实浏览器已验证）
|
||||
- `fetch.py --extract json`：结构化骨架输出 `{title, meta_description, headings[], links[], images[]}`——AI 只看骨架即可判断页面价值，无需拉全文（隐含 `--format json`；stdlib/bs4 双路径输出同构）
|
||||
- `fetch.py --max-chars N`：提取后语义级字符截断（区别于 `--max-size` 的原始字节截断），`truncated` 字段如实标记
|
||||
- `search.py --fetch-total-chars N`：`--fetch` 全局字符预算，按结果顺序自上而下分配（剩余额度留给下一页），耗尽后剩余 URL 标记 `status="skipped"`（不发请求）——token 受限 agent 的上下文预算控制
|
||||
- `--dedup-fetched-content`：抓取正文相似度去重（SimHash，复用 `--similarity-threshold`），镜像/转载页面标记 `status="duplicate"`（text 清空），避免 AI 重复阅读同一内容
|
||||
- `--progress` 新增 research 角度级事件：`angle_start`/`angle_ok`/`angle_fail`（含 `index`/`total`）；fetch 链路新增 `fetch_skip`/`fetch_duplicate` 事件
|
||||
- `fetch.py` 补齐 `--log-format json` 与 `--dump-schema`（与 search.py 对齐，结构化 JSON 契约可程序化发现）
|
||||
- CSV 输出媒体列自适应：images/videos 类别结果自动追加 `img_src`/`thumbnail_src`/`resolution`/`iframe_src`/`source` 列（纯文本结果不变，向后兼容）
|
||||
- `--dry-run` 批量模式打印实际查询列表（`queries` 字段，读本地文件不发 HTTP）
|
||||
- search 连接复用：requests 可用时走模块级 Session（连接池 + TLS 会话恢复），`--pages`/批量/研究模式多请求场景显著降开销；stdlib 零依赖路径不变
|
||||
- search 403 快速失败：实例级 403 不再退避重试（原 ~10.5s 空等），立即 failover；fetch 的 403 UA 轮换语义不受影响
|
||||
- `AdaptiveThrottle` 修复 `--throttle-failure-threshold 0` 语义：现为真正的"禁用自适应退避"（原实现 0 导致首次失败即退避）
|
||||
- 新增 `scripts/release_check.py` 发布一致性检查：版本号（pyproject/_config/文档）与错误码表（common.py/README/SKILL）漂移检测
|
||||
- 版本对齐：pyproject.toml 与 _config.py 同步为 2.5.0（此前 v2.4.0 发布时 pyproject 停在 2.3.0）
|
||||
|
||||
**v2.4.0 新功能**
|
||||
- 新增 `E_BLOCKED` 错误码：fetch 场景的 403（WAF/反爬拦截）从 `E_AUTH` 细分出来——被封锁不是凭证问题，AI Agent 不再误判为"需要检查认证"。新增 `classify_fetch_error()` / `_extract_status_code()`（common.py），应用于 `fetch.py` 与 `search.py --fetch`；SearXNG 实例认证的 403 仍映射 `E_AUTH`（搜索场景不变）
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
---
|
||||
name: searxng-use-cli
|
||||
description: Use when you need to search the web via your OWN SearXNG instance (no public-instance discovery). 3 CLI scripts + a shared common.py module — execute privacy-respecting searches against a user-supplied instance (with multi-instance failover, 5xx/429 retry, auto-fetch) or via SEARXNG_INSTANCE env / config file, fetch/extract readable text or markdown from web pages. Zero-config replacement for proprietary search APIs.
|
||||
version: 2.4.0
|
||||
version: 2.5.0
|
||||
author: Metona Team
|
||||
license: MIT
|
||||
platforms: [linux, macos, windows]
|
||||
@@ -37,6 +37,8 @@ SearXNG is a privacy-respecting metasearch engine that aggregates results from 7
|
||||
|
||||
**Key capabilities:**
|
||||
|
||||
> **v2.5.0 highlights:** structured page skeleton output (`fetch.py --extract json` — title/meta/headings/links/images, no full body needed); token budget control (`search.py --fetch-total-chars N`, `fetch.py --max-chars N`); fetched-content dedup (`--dedup-fetched-content`, mirrors/republished pages marked `status="duplicate"`); fixed `Sec-Ch-Ua` header construction (was malformed in Chrome/Edge paths); research-angle progress events (`angle_start/ok/fail`); `fetch.py` now has `--log-format json` + `--dump-schema`; CSV auto-appends media columns for images/videos categories; search reuses a requests Session when available (connection pooling). Full changelog in README.md.
|
||||
|
||||
**Search & results**
|
||||
- Multi-instance failover with parallel probing (faster failover, deterministic output order)
|
||||
- Exponential-backoff retry on transient errors (403/429/5xx/connection) via shared retry-policy components in `common.py` (backoff/retryable-status/Retry-After), with the retry loop in each script
|
||||
@@ -544,11 +546,13 @@ usage: fetch.py [-h] --url URL [--extract {text,html,markdown}]
|
||||
7. **PDF/document parsing** (regardless of `--extract` mode): PDF via `pdftotext` subprocess, `.docx`/`.xlsx` via stdlib `zipfile`. Unsupported binary types return `E_UNSUPPORTED_MEDIA`
|
||||
|
||||
**Key options:**
|
||||
- `--url https://...` — required
|
||||
- `--url https://...` — required (unless `--dump-schema`)
|
||||
- `--format {text,json}` — output format (default: `text`; `json` = structured contract, v2.3.0). Success: `{status: ok, url, final_url, content_type, extract, truncated, text_length, user_agent}`; failure: `{status: error, url, error, error_code, status_code}`. stdout is pure JSON; logs stay on stderr; exit 0 on success / 1 on failure
|
||||
- `--extract text` — clean readable text (default); PDF/docx/xlsx parsed automatically regardless of extract mode (see "What it does" #7)
|
||||
- `--extract html` — raw HTML
|
||||
- `--extract markdown` — Markdown conversion (tree-based; handles nested tags, GFM tables, fenced code blocks, blockquotes, inline code, ordered/unordered/nested lists, definition lists, images, emphasis)
|
||||
- `--extract json` — v2.5.0 structured skeleton + text. Success payload adds `{title, meta_description, headings[], links[], images[]}` so agents can judge a page without reading the full body. Implies `--format json`
|
||||
- `--max-chars N` — v2.5.0 truncate extracted text to N chars (semantic cut after extraction; sets `truncated=true`). Distinct from `--max-size` which caps raw response bytes
|
||||
- `--encoding gbk` — force charset for non-UTF-8 pages (else auto-detected from HTTP header / HTML meta / UTF-8 fallback)
|
||||
- `--timeout 15` — request timeout in seconds (split into connect/read tuple internally)
|
||||
- `--retries 3` — max retries on transient errors (429/5xx/connection); falls back through the 15-UA pool when blocked
|
||||
|
||||
+1
-1
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "searxng-use-cli"
|
||||
version = "2.3.0"
|
||||
version = "2.5.0"
|
||||
description = "Privacy-respecting SearXNG search + web-fetch CLI toolkit for AI agents"
|
||||
readme = "README.md"
|
||||
license = { text = "MIT" }
|
||||
|
||||
+1
-1
@@ -7,7 +7,7 @@ Retry settings and shared HTTP utilities now live in ``common.py`` so that
|
||||
both ``search.py`` and ``fetch.py`` share one consistent implementation.
|
||||
"""
|
||||
|
||||
VERSION = "2.4.0"
|
||||
VERSION = "2.5.0"
|
||||
SCHEMA_VERSION = "1.0"
|
||||
USER_AGENT = f"searxng-cli/{VERSION}"
|
||||
|
||||
|
||||
+38
-5
@@ -329,14 +329,25 @@ def build_browser_headers(user_agent: str, referer: str = None,
|
||||
|
||||
# Sec-Ch-Ua 系列仅 Chrome/Edge 发送，Firefox 不发
|
||||
if not is_firefox:
|
||||
# 从 UA 提取主版本号，构造 Sec-Ch-Ua
|
||||
# 从 UA 提取主版本号，构造 Sec-Ch-Ua。
|
||||
# v2.5.0 修复：原实现把已含引号的 not_a_brand 再包进 f-string 引号，
|
||||
# 生成 ""Not_A Brand";v="99"" 的畸形头——Chrome/Edge 两条路径都中招，
|
||||
# 严格校验 Sec-CH-UA 的 WAF（Cloudflare 等会校验与 UA 一致性）会直接
|
||||
# 忽略或判定不一致。现改为品牌数组统一拼接，产出合法格式：
|
||||
# Chrome: "Not_A Brand";v="99", "Chromium";v="138", "Google Chrome";v="138"
|
||||
# Edge: 上述 + , "Microsoft Edge";v="138"
|
||||
m = re.search(r"Chrome/(\d+)", user_agent)
|
||||
ver = m.group(1) if m else "131"
|
||||
not_a_brand = '"Not_A Brand";v="99"' if ver != "99" else '"Not/A)Brand";v="99"'
|
||||
headers["Sec-Ch-Ua"] = f'"{not_a_brand}", "Chromium";v="{ver}", "Google Chrome";v="{ver}"'
|
||||
not_a_brand = "Not_A Brand" if ver != "99" else "Not/A)Brand"
|
||||
brands = [
|
||||
f'"{not_a_brand}";v="99"',
|
||||
f'"Chromium";v="{ver}"',
|
||||
f'"Google Chrome";v="{ver}"',
|
||||
]
|
||||
if is_edge:
|
||||
# Edge 的品牌标识
|
||||
headers["Sec-Ch-Ua"] = headers["Sec-Ch-Ua"].rstrip('"') + f'", "Microsoft Edge";v="{ver}"'
|
||||
# Edge 额外声明 Microsoft Edge 品牌
|
||||
brands.append(f'"Microsoft Edge";v="{ver}"')
|
||||
headers["Sec-Ch-Ua"] = ", ".join(brands)
|
||||
headers["Sec-Ch-Ua-Mobile"] = '"?1"' if "Mobile" in user_agent else '"?0"'
|
||||
# 平台标识
|
||||
if "Windows" in user_agent:
|
||||
@@ -1054,3 +1065,25 @@ def is_similar(result_a: dict, result_b: dict, threshold: float = 0.85) -> bool:
|
||||
max_distance = max(0, min(3, int(round((1.0 - threshold) / 0.05))))
|
||||
return _hamming_distance(hash_a, hash_b) <= max_distance
|
||||
|
||||
|
||||
def texts_are_similar(text_a: str, text_b: str, threshold: float = 0.85) -> bool:
|
||||
"""判断两段正文是否近似重复（v2.5.0，供抓取正文去重用）。
|
||||
|
||||
镜像/转载站点的正文与原文高度相似但 URL 不同——URL 级去重无法合并，
|
||||
需要按正文指纹判定。复用与 :func:`is_similar` 相同的
|
||||
SimHash + 汉明距离逻辑，但针对**正文**而非标题：
|
||||
|
||||
* 归一化（小写/去标点/CJK 单字分词）后取前 1000 字符做指纹窗口——
|
||||
镜像页前 1000 字符通常一致，长文全量计算只会稀释指纹且增加开销
|
||||
* 空文本（len<窗口）返回 False，避免短错误文本误判为重复
|
||||
* threshold → 汉明距离阈值映射与 is_similar 一致（0.85→3 ... 1.0→0）
|
||||
"""
|
||||
norm_a = _normalize_title(text_a)[:1000]
|
||||
norm_b = _normalize_title(text_b)[:1000]
|
||||
if len(norm_a) < 5 or len(norm_b) < 5:
|
||||
return False
|
||||
hash_a = _simhash(norm_a)
|
||||
hash_b = _simhash(norm_b)
|
||||
max_distance = max(0, min(3, int(round((1.0 - threshold) / 0.05))))
|
||||
return _hamming_distance(hash_a, hash_b) <= max_distance
|
||||
|
||||
|
||||
+252
-7
@@ -600,6 +600,145 @@ def html_to_markdown(html_content: str) -> str:
|
||||
return converter.get_markdown()
|
||||
|
||||
|
||||
# ----- Structured extraction (--extract json, v2.5.0) -----
|
||||
# 输出页面的结构化骨架（title/meta/headings/links/images），AI Agent 可以
|
||||
# 只看骨架就判断页面是否有用，无需拉全文——显著节省 token。
|
||||
|
||||
# 元素数量上限：防止超大页面（5MB HTML 有数千链接）撑爆输出
|
||||
_MAX_HEADINGS = 50
|
||||
_MAX_LINKS = 100
|
||||
_MAX_IMAGES = 50
|
||||
|
||||
|
||||
class _StructureExtractor(HTMLParser):
|
||||
"""stdlib 结构化提取器（bs4 不可用时的回退）。
|
||||
|
||||
与 bs4 路径输出同一 shape：title / meta_description / headings / links /
|
||||
images。heading 文本带层级前缀（"h2: 内容"），便于 AI 直接理解结构。
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.title = ""
|
||||
self.meta_description = ""
|
||||
self.headings = []
|
||||
self.links = []
|
||||
self.images = []
|
||||
self._in_title = False
|
||||
self._current_heading = None
|
||||
self._heading_text = []
|
||||
|
||||
def handle_starttag(self, tag, attrs):
|
||||
tag = tag.lower()
|
||||
attrs_dict = dict(attrs)
|
||||
if tag == "title":
|
||||
self._in_title = True
|
||||
elif tag == "meta":
|
||||
name = (attrs_dict.get("name") or "").lower()
|
||||
if name == "description" and attrs_dict.get("content"):
|
||||
self.meta_description = attrs_dict["content"].strip()[:500]
|
||||
elif tag in ("h1", "h2", "h3", "h4", "h5", "h6"):
|
||||
self._current_heading = tag
|
||||
self._heading_text = []
|
||||
elif tag == "a" and attrs_dict.get("href"):
|
||||
href = attrs_dict["href"].strip()
|
||||
if href and href not in self.links and len(self.links) < _MAX_LINKS:
|
||||
self.links.append(href)
|
||||
elif tag == "img":
|
||||
# data-src 兜底：懒加载站点的 src 常为空
|
||||
src = attrs_dict.get("src") or attrs_dict.get("data-src") or ""
|
||||
if src and src not in self.images and len(self.images) < _MAX_IMAGES:
|
||||
self.images.append(src)
|
||||
|
||||
def handle_endtag(self, tag):
|
||||
tag = tag.lower()
|
||||
if tag == "title":
|
||||
self._in_title = False
|
||||
elif tag in ("h1", "h2", "h3", "h4", "h5", "h6") and self._current_heading:
|
||||
text = " ".join(self._heading_text).strip()
|
||||
if text and len(self.headings) < _MAX_HEADINGS:
|
||||
self.headings.append(f"{self._current_heading}: {text}"[:200])
|
||||
self._current_heading = None
|
||||
|
||||
def handle_data(self, data):
|
||||
if self._in_title:
|
||||
self.title += data
|
||||
elif self._current_heading:
|
||||
self._heading_text.append(data)
|
||||
|
||||
|
||||
def _extract_structure_stdlib(html_content: str) -> dict:
|
||||
"""stdlib HTMLParser 路径的结构化提取。"""
|
||||
ex = _StructureExtractor()
|
||||
try:
|
||||
ex.feed(html_content)
|
||||
except Exception:
|
||||
pass # 解析失败时返回部分结果，绝不因提取失败而丢弃页面
|
||||
return {
|
||||
"title": ex.title.strip()[:300],
|
||||
"meta_description": ex.meta_description[:500],
|
||||
"headings": ex.headings,
|
||||
"links": ex.links,
|
||||
"images": ex.images,
|
||||
}
|
||||
|
||||
|
||||
def _extract_structure_bs4(html_content: str) -> dict:
|
||||
"""bs4 路径的结构化提取（质量更高，优先使用）。"""
|
||||
soup = _BeautifulSoup(html_content, "html.parser")
|
||||
|
||||
title = soup.title.get_text(strip=True)[:300] if soup.title else ""
|
||||
|
||||
meta_description = ""
|
||||
m = soup.find("meta", attrs={"name": lambda v: v and v.lower() == "description"})
|
||||
if m and m.get("content"):
|
||||
meta_description = m["content"].strip()[:500]
|
||||
|
||||
headings = []
|
||||
for tag in soup.find_all(["h1", "h2", "h3", "h4", "h5", "h6"]):
|
||||
t = tag.get_text(" ", strip=True)
|
||||
if t:
|
||||
headings.append(f"{tag.name}: {t}"[:200])
|
||||
if len(headings) >= _MAX_HEADINGS:
|
||||
break
|
||||
|
||||
links, images = [], []
|
||||
for a in soup.find_all("a", href=True):
|
||||
href = a["href"].strip()
|
||||
if href and href not in links:
|
||||
links.append(href)
|
||||
if len(links) >= _MAX_LINKS:
|
||||
break
|
||||
for img in soup.find_all("img"):
|
||||
src = img.get("src") or img.get("data-src") or ""
|
||||
if src and src not in images:
|
||||
images.append(src)
|
||||
if len(images) >= _MAX_IMAGES:
|
||||
break
|
||||
|
||||
return {
|
||||
"title": title,
|
||||
"meta_description": meta_description,
|
||||
"headings": headings,
|
||||
"links": links,
|
||||
"images": images,
|
||||
}
|
||||
|
||||
|
||||
def extract_structure(html_content: str) -> dict:
|
||||
"""从 HTML 提取结构化骨架（--extract json 用）。
|
||||
|
||||
bs4 优先；stdlib HTMLParser 回退（两路径输出同一 shape）。
|
||||
非 HTML 内容返回空结构（调用方判断 ``is_html`` 后再调用）。
|
||||
"""
|
||||
if _HAS_BS4:
|
||||
try:
|
||||
return _extract_structure_bs4(html_content)
|
||||
except Exception:
|
||||
pass
|
||||
return _extract_structure_stdlib(html_content)
|
||||
|
||||
|
||||
# ----- HTTP Fetch -----
|
||||
# RETRY_BACKOFF_BASE, FALLBACK_UAS, detect_charset and is_retryable_error are
|
||||
# imported from common.py (shared with search.py for a consistent retry policy).
|
||||
@@ -1136,11 +1275,59 @@ def fetch_url(url: str, timeout=15, user_agent: str = None,
|
||||
|
||||
# ----- Main -----
|
||||
|
||||
def _get_output_schema() -> dict:
|
||||
"""返回 fetch.py ``--format json`` 输出的 JSON Schema（v2.5.0）。
|
||||
|
||||
AI Agent 可程序化发现字段名与类型，无需解析散文文档。
|
||||
成功 shape 的 ``extract`` 字段枚举四种模式；``--extract json`` 时
|
||||
额外包含结构化骨架字段（title/meta_description/headings/links/images）。
|
||||
"""
|
||||
return {
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": "SearXNG CLI Fetch Result",
|
||||
"description": "Output schema for 'python fetch.py --format json'. "
|
||||
"With --extract json (v2.5.0) the success payload "
|
||||
"includes structured skeleton fields "
|
||||
"(title/meta_description/headings/links/images) "
|
||||
"alongside the extracted text.",
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"status": {"type": "string", "enum": ["ok", "error"]},
|
||||
"url": {"type": "string", "format": "uri"},
|
||||
"final_url": {"type": "string",
|
||||
"description": "URL after redirects / Wayback."},
|
||||
"content_type": {"type": "string"},
|
||||
"extract": {"type": "string",
|
||||
"enum": ["text", "html", "markdown", "json"]},
|
||||
"truncated": {"type": "boolean",
|
||||
"description": "True when --max-size cut raw bytes or "
|
||||
"--max-chars cut extracted text."},
|
||||
"text_length": {"type": "integer"},
|
||||
"user_agent": {"type": "string"},
|
||||
"title": {"type": "string",
|
||||
"description": "Page <title> (extract=json only)."},
|
||||
"meta_description": {"type": "string",
|
||||
"description": "Meta description (extract=json only)."},
|
||||
"headings": {"type": "array", "items": {"type": "string"},
|
||||
"description": "h1-h6 headings, prefixed with tag "
|
||||
"(extract=json only)."},
|
||||
"links": {"type": "array", "items": {"type": "string"},
|
||||
"description": "Up to 100 unique hrefs (extract=json only)."},
|
||||
"images": {"type": "array", "items": {"type": "string"},
|
||||
"description": "Up to 50 image srcs (extract=json only)."},
|
||||
"error": {"type": "string"},
|
||||
"error_code": {"type": "string"},
|
||||
"status_code": {"type": "integer"},
|
||||
},
|
||||
"required": ["status", "url"],
|
||||
}
|
||||
|
||||
def _emit_fetch_result(args, output: str, url: str, final_url: str,
|
||||
content_type: str, truncated: bool,
|
||||
user_agent: str = None,
|
||||
error: str = None, error_code: str = None,
|
||||
status_code: int = None) -> None:
|
||||
status_code: int = None,
|
||||
structure: dict = None) -> None:
|
||||
"""输出抓取结果到 stdout / --output 文件。
|
||||
|
||||
v2.3.0: ``--format json`` 提供结构化 JSON 契约，AI Agent 可程序化
|
||||
@@ -1155,6 +1342,10 @@ def _emit_fetch_result(args, output: str, url: str, final_url: str,
|
||||
失败 shape::
|
||||
|
||||
{"status": "error", "url", "error", "error_code", "status_code"}
|
||||
|
||||
v2.5.0: ``structure`` 参数携带 ``--extract json`` 的结构化骨架字段
|
||||
（title/meta_description/headings/links/images），合并进 json 成功
|
||||
payload。text 模式忽略该参数。
|
||||
"""
|
||||
if args.format == "json":
|
||||
if error:
|
||||
@@ -1174,6 +1365,8 @@ def _emit_fetch_result(args, output: str, url: str, final_url: str,
|
||||
"text_length": len(output),
|
||||
"user_agent": user_agent,
|
||||
}
|
||||
if structure:
|
||||
payload.update(structure)
|
||||
text = json.dumps(payload, indent=2, ensure_ascii=False)
|
||||
else:
|
||||
text = output
|
||||
@@ -1200,9 +1393,16 @@ Examples:
|
||||
%(prog)s -u https://example.com --format json structured JSON output
|
||||
""",
|
||||
)
|
||||
parser.add_argument("--url", "-u", required=True, help="URL to fetch")
|
||||
parser.add_argument("--extract", "-e", choices=["text", "html", "markdown"],
|
||||
default="text", help="Extraction mode (default: text)")
|
||||
parser.add_argument("--url", "-u", required=False,
|
||||
help="URL to fetch (required unless --dump-schema)")
|
||||
parser.add_argument("--extract", "-e",
|
||||
choices=["text", "html", "markdown", "json"],
|
||||
default="text",
|
||||
help="Extraction mode (default: text). 'json' (v2.5.0) "
|
||||
"outputs a structured JSON object with the page's "
|
||||
"text plus its skeleton: title, meta_description, "
|
||||
"headings, links, images — lets agents judge a page "
|
||||
"without reading the full body. Implies --format json.")
|
||||
parser.add_argument("--format", "-f", choices=["text", "json"], default="text",
|
||||
help="Output format (default: text). 'json' emits a structured "
|
||||
"JSON object {status, url, final_url, content_type, extract, "
|
||||
@@ -1215,6 +1415,11 @@ Examples:
|
||||
help="Max retries on transient errors (default: 3)")
|
||||
parser.add_argument("--max-size", type=int, default=None, metavar="BYTES",
|
||||
help="Max page size in bytes (default: unlimited). Set to 5242880 for 5MB cap.")
|
||||
parser.add_argument("--max-chars", type=int, default=None, metavar="N",
|
||||
help="v2.5.0: truncate EXTRACTED text to N chars "
|
||||
"(semantic cut, after extraction; sets truncated=true "
|
||||
"in json output). Distinct from --max-size which caps "
|
||||
"raw response bytes. 0 = no limit (default).")
|
||||
parser.add_argument("--user-agent", default=None,
|
||||
help="Custom User-Agent header")
|
||||
parser.add_argument("--encoding", default=None,
|
||||
@@ -1251,16 +1456,43 @@ Examples:
|
||||
parser.add_argument("--quiet", action="store_true", default=False,
|
||||
help="Quiet output: suppress progress messages on stderr; "
|
||||
"only warnings and errors are shown")
|
||||
parser.add_argument("--log-format", choices=["text", "json"], default="text",
|
||||
help="v2.5.0: log output format (default: text). 'json' "
|
||||
"emits one JSON object per line for programmatic "
|
||||
"parsing by AI Agents.")
|
||||
parser.add_argument("--dump-schema", action="store_true",
|
||||
help="v2.5.0: print the JSON Schema for --format json "
|
||||
"output to stdout and exit. Lets AI agents discover "
|
||||
"field names and types programmatically.")
|
||||
parser.add_argument("--version", action="version", version=f"searxng-cli v{VERSION}")
|
||||
|
||||
args = parser.parse_args()
|
||||
setup_logging(verbose=args.verbose, quiet=args.quiet)
|
||||
setup_logging(verbose=args.verbose, quiet=args.quiet,
|
||||
log_format=args.log_format)
|
||||
force_utf8_stdout() # Windows: prevent GBK crash on non-ASCII chars
|
||||
|
||||
# --dump-schema：输出 JSON Schema 到 stdout 并退出，AI Agent 可程序化
|
||||
# 发现字段（无需 --url）。必须在 URL 校验之前处理。
|
||||
if args.dump_schema:
|
||||
print(json.dumps(_get_output_schema(), indent=2, ensure_ascii=False))
|
||||
sys.exit(0)
|
||||
|
||||
# --url 手动校验（required=False 是为了让 --dump-schema 单独可用）
|
||||
if not args.url:
|
||||
logger.error("Error: --url is required")
|
||||
sys.exit(1)
|
||||
|
||||
if not args.url.startswith(("http://", "https://")):
|
||||
logger.error("Error: URL must start with http:// or https://")
|
||||
sys.exit(1)
|
||||
|
||||
# --extract json 隐含 --format json（结构化输出必须走 json 通道）。
|
||||
# 用户显式传 --format text 时以 json 为准并提示，避免静默产出无意义文本。
|
||||
if args.extract == "json" and args.format != "json":
|
||||
logger.info("--extract json implies --format json; ignoring --format "
|
||||
f"{args.format}")
|
||||
args.format = "json"
|
||||
|
||||
# Apply proxy via env vars so both urllib and requests honor it.
|
||||
if args.proxy:
|
||||
apply_proxy(args.proxy)
|
||||
@@ -1298,6 +1530,7 @@ Examples:
|
||||
fatal_error = None
|
||||
fatal_error_code = None
|
||||
fatal_status_code = None
|
||||
structure = None # v2.5.0: --extract json 的结构化骨架
|
||||
|
||||
try:
|
||||
result = fetch_url(
|
||||
@@ -1395,20 +1628,32 @@ Examples:
|
||||
content.strip().startswith("<!") or
|
||||
content.strip().startswith("<htm"))
|
||||
|
||||
if args.extract == "html":
|
||||
if args.extract == "json":
|
||||
# v2.5.0：结构化骨架 + 正文。骨架字段由 _emit_fetch_result 合并进
|
||||
# json payload；output 保持为正文文本（供 text_length / --max-chars）。
|
||||
structure = extract_structure(content) if is_html else {}
|
||||
output = extract_text(content) if is_html else content
|
||||
elif args.extract == "html":
|
||||
output = content
|
||||
elif args.extract == "markdown":
|
||||
output = html_to_markdown(content) if is_html else content
|
||||
else: # text
|
||||
output = extract_text(content) if is_html else content
|
||||
|
||||
# v2.5.0: --max-chars 语义级截断（提取后按字符截断，区别于 --max-size
|
||||
# 的原始字节截断）。截断后标记 truncated=true（json 契约可感知）。
|
||||
if args.max_chars and args.max_chars > 0 and output and len(output) > args.max_chars:
|
||||
output = output[:args.max_chars]
|
||||
truncated = True
|
||||
logger.info(f"Truncated extracted text to {args.max_chars} chars")
|
||||
|
||||
# Quality check (threshold: 500 chars)
|
||||
if args.extract == "text" and len(output.strip()) < 500 and is_html:
|
||||
logger.warning(f"Warning: extracted text is very short ({len(output.strip())} chars). "
|
||||
"The page may be JS-heavy or use anti-bot protection.")
|
||||
|
||||
_emit_fetch_result(args, output, args.url, final_url, content_type,
|
||||
truncated, user_agent)
|
||||
truncated, user_agent, structure=structure)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
#!/usr/bin/env python3
|
||||
"""发布前一致性检查（v2.5.0）。
|
||||
|
||||
检查版本号与错误码表在多个文件间是否漂移——历史上 pyproject.toml 与
|
||||
``_config.VERSION`` 曾不同步（v2.4.0 发布时前者停在 2.3.0）。发布前运行
|
||||
本脚本即可一次性发现：
|
||||
|
||||
1. 版本一致性：pyproject.toml <-> _config.py（同一来源），并警告
|
||||
README.md / SKILL.md 中提到的版本号是否与最新版本一致（文档是
|
||||
手工维护的，仅警告不阻断）
|
||||
2. 错误码一致性：common.py 的 ``E_*`` 常量 vs 各脚本实际使用 vs
|
||||
RECOVERY_HINTS 覆盖 vs README/SKILL 错误码表中提及的错误码
|
||||
|
||||
用法（项目根目录）::
|
||||
|
||||
python scripts/release_check.py # 完整检查
|
||||
python scripts/release_check.py --strict # 文档警告也视为失败
|
||||
|
||||
零依赖（stdlib），退出码 0=通过 / 1=有阻断问题。
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
SCRIPTS_DIR = ROOT / "scripts"
|
||||
|
||||
# 文档中可能提及版本号的锚点模式（供提示）
|
||||
_VERSION_HINT_RE = re.compile(r"\b\d+\.\d+\.\d+\b")
|
||||
|
||||
# 错误码表（README.md 与 SKILL.md 均有）应包含的 E_* 代码，从 common.py
|
||||
# 动态提取，与源码单一来源保持一致。捕获组只取标识符本身（不含 = "）。
|
||||
_ERROR_CODE_RE = re.compile(r"^\s*(E_[A-Z_]+)\s*=\s*[\"']", re.MULTILINE)
|
||||
|
||||
|
||||
def _read(path: Path) -> str:
|
||||
try:
|
||||
return path.read_text(encoding="utf-8")
|
||||
except OSError as e:
|
||||
print(f" [WARN] 无法读取 {path}: {e}")
|
||||
return ""
|
||||
|
||||
|
||||
def check_version(strict: bool) -> list:
|
||||
"""检查版本号一致性。返回错误消息列表（空 = 通过）。"""
|
||||
errors = []
|
||||
|
||||
# 1. pyproject.toml 的 version 字段
|
||||
pyproject = ROOT / "pyproject.toml"
|
||||
pp = _read(pyproject)
|
||||
m = re.search(r"^version\s*=\s*[\"']([\d.]+)[\"']", pp, re.MULTILINE)
|
||||
pp_version = m.group(1) if m else None
|
||||
|
||||
# 2. _config.py 的 VERSION（权威来源）
|
||||
import importlib.util
|
||||
spec = importlib.util.spec_from_file_location("_config", SCRIPTS_DIR / "_config.py")
|
||||
cfg = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(cfg)
|
||||
cfg_version = cfg.VERSION
|
||||
|
||||
if pp_version and pp_version != cfg_version:
|
||||
errors.append(
|
||||
f"版本漂移: pyproject.toml={pp_version} 但 _config.VERSION={cfg_version}。"
|
||||
f"请同步 pyproject.toml 或改为从 _config 动态读取。"
|
||||
)
|
||||
elif not pp_version:
|
||||
errors.append("pyproject.toml 缺少 version 字段。")
|
||||
|
||||
# 3. 文档版本提示（仅警告）
|
||||
if cfg_version:
|
||||
for doc_name in ("README.md", "SKILL.md"):
|
||||
doc = _read(ROOT / doc_name)
|
||||
if not doc:
|
||||
continue
|
||||
mentioned = set(_VERSION_HINT_RE.findall(doc))
|
||||
# 文档可能同时提到旧版本历史（changelog 里 v2.3.0 等），
|
||||
# 只要"存在"当前版本号即视为已更新
|
||||
if cfg_version not in mentioned:
|
||||
msg = (f"文档 {doc_name} 未提及当前版本 {cfg_version} "
|
||||
f"(提到: {sorted(mentioned)[:6]}...)。请更新文档。")
|
||||
if strict:
|
||||
errors.append(msg)
|
||||
else:
|
||||
print(f" [WARN] {msg}")
|
||||
return errors
|
||||
|
||||
|
||||
def check_error_codes(strict: bool) -> list:
|
||||
"""检查错误码表一致性。返回错误消息列表（空 = 通过）。"""
|
||||
errors = []
|
||||
|
||||
common_src = _read(SCRIPTS_DIR / "common.py")
|
||||
codes = _ERROR_CODE_RE.findall(common_src)
|
||||
if not codes:
|
||||
errors.append("common.py 中未找到 E_* 常量定义。")
|
||||
return errors
|
||||
codes = sorted(codes)
|
||||
|
||||
# 1. 每个 E_* 是否有 recovery_hint
|
||||
hint_keys = set(re.findall(r'^\s*"?(E_[A-Z_]+)"?\s*:\s*[\(\["\']',
|
||||
common_src, re.MULTILINE))
|
||||
missing_hints = [c for c in codes if c not in hint_keys]
|
||||
if missing_hints:
|
||||
errors.append(f"缺少 RECOVERY_HINTS 的错误码: {missing_hints}")
|
||||
|
||||
# 2. README/SKILL 错误码表是否覆盖所有 E_*（文档漂移提示）
|
||||
for doc_name in ("README.md", "SKILL.md"):
|
||||
doc = _read(ROOT / doc_name)
|
||||
if not doc:
|
||||
continue
|
||||
missing_doc = [c for c in codes if c not in doc]
|
||||
if missing_doc:
|
||||
msg = (f"文档 {doc_name} 错误码表缺少: {missing_doc}。"
|
||||
f"源码已新增这些错误码，请更新文档表格。")
|
||||
if strict:
|
||||
errors.append(msg)
|
||||
else:
|
||||
print(f" [WARN] {msg}")
|
||||
return errors
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description="searxng-cli 发布前一致性检查")
|
||||
ap.add_argument("--strict", action="store_true",
|
||||
help="文档警告也视为失败（默认仅提示）")
|
||||
args = ap.parse_args()
|
||||
|
||||
errors = []
|
||||
errors += check_version(args.strict)
|
||||
errors += check_error_codes(args.strict)
|
||||
|
||||
if errors:
|
||||
print("✗ 检查未通过:")
|
||||
for e in errors:
|
||||
print(f" - {e}")
|
||||
return 1
|
||||
print("✓ 版本号与错误码一致性检查全部通过")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+338
-22
@@ -45,6 +45,7 @@ from common import (
|
||||
force_utf8_stdout,
|
||||
is_hard_blocked_domain,
|
||||
is_similar,
|
||||
texts_are_similar,
|
||||
parse_retry_after,
|
||||
resolve_auth_basic,
|
||||
resolve_auth_bearer,
|
||||
@@ -488,13 +489,19 @@ def _retry_with_backoff(fn, max_retries: int = MAX_RETRIES, base_delay: float =
|
||||
v2.1.0 修复：复用 common.compute_backoff_delay（带 60s 封顶），
|
||||
避免高重试次数（如 --retry 10）时 1.5*2^10=1536s 卡死进程。
|
||||
同时遵守 Retry-After header（429/503），与 fetch.py 保持一致。
|
||||
|
||||
v2.5.0：403 不再重试。RETRYABLE_STATUS 含 403 是为 fetch.py 的 UA
|
||||
轮换设计的（每次重试换新 UA），但本函数仅供 search 使用——search 不
|
||||
换 UA，实例级 403 是认证/封禁问题，重试只会空等 ~10.5s 后才 failover。
|
||||
现在 403 立即 raise，由 search_multi 直接切换下一实例，classify_error
|
||||
仍正确归为 E_AUTH（实例认证失败语义不变）。
|
||||
"""
|
||||
last_error = None
|
||||
for attempt in range(max_retries + 1):
|
||||
try:
|
||||
return fn()
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code in RETRYABLE_STATUS: # 403 (UA block) + 429 + 5xx
|
||||
if e.code in RETRYABLE_STATUS and e.code != 403: # 429 + 5xx
|
||||
last_error = e
|
||||
if attempt < max_retries:
|
||||
# 429/503：遵守 Retry-After header，避免触发更严厉限流
|
||||
@@ -523,23 +530,115 @@ def _retry_with_backoff(fn, max_retries: int = MAX_RETRIES, base_delay: float =
|
||||
|
||||
# ----- Search execution -----
|
||||
|
||||
# v2.5.0 连接复用：requests 可用时复用模块级 Session（连接池 + TLS 会话
|
||||
# 恢复），--pages/批量/研究模式的多请求场景显著减少 TLS 握手开销。
|
||||
# stdlib 路径保持零依赖可用（每次新建连接，功能等价）——与 fetch.py 的
|
||||
# "requests 路径受益，stdlib 路径功能完整" 设计一致。
|
||||
_HAS_REQUESTS = False
|
||||
try:
|
||||
import requests as _requests # type: ignore
|
||||
_HAS_REQUESTS = True
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
_session = None
|
||||
|
||||
|
||||
def _get_session():
|
||||
"""获取（惰性创建）模块级 requests.Session。
|
||||
|
||||
连接池：每主机最多 10 连接，总最多 20 连接。max_retries=0 让 requests
|
||||
不做自己的重试——重试统一由 _retry_with_backoff 控制，避免双重退避。
|
||||
"""
|
||||
global _session
|
||||
if _session is None and _HAS_REQUESTS:
|
||||
_session = _requests.Session()
|
||||
adapter = _requests.adapters.HTTPAdapter(
|
||||
pool_connections=10, pool_maxsize=10, max_retries=0,
|
||||
)
|
||||
_session.mount("http://", adapter)
|
||||
_session.mount("https://", adapter)
|
||||
return _session
|
||||
|
||||
|
||||
def _reset_session() -> None:
|
||||
"""关闭并重置模块级 Session。测试用。"""
|
||||
global _session
|
||||
if _session is not None:
|
||||
try:
|
||||
_session.close()
|
||||
except Exception:
|
||||
pass
|
||||
_session = None
|
||||
|
||||
|
||||
def _finalize_json_result(data):
|
||||
"""兜底填充 SearXNG JSON 响应的可选契约字段。
|
||||
|
||||
部分实例/版本的 JSON 输出缺失 ``number_of_results``（服务端
|
||||
``search.result_number()`` 的估算值，见 --dump-schema 描述）。缺失时
|
||||
用实际结果数近似填充，保证 JSON 路径与 HTML fallback 路径
|
||||
（parse_html_results 必然填充）的输出契约一致——AI Agent 依赖该字段
|
||||
判断结果规模与是否需要翻页。
|
||||
"""
|
||||
if isinstance(data, dict) and "number_of_results" not in data:
|
||||
data["number_of_results"] = len(data.get("results", []))
|
||||
return data
|
||||
|
||||
|
||||
def search_json(instance: str, params: dict, method: str = "GET",
|
||||
timeout: int = 15, auth_headers: dict = None) -> dict:
|
||||
"""Execute search via JSON API. Returns None if JSON unsupported."""
|
||||
"""Execute search via JSON API. Returns None if JSON unsupported.
|
||||
|
||||
v2.5.0: requests 可用时走模块级 Session（连接池复用）；否则 stdlib。
|
||||
两个后端发出完全相同的 URL（query_string 拼好后原样使用），解码逻辑
|
||||
一致（resp.content 手动 decode，不用 resp.text 的自动解码），保证
|
||||
行为可预测、跨后端可复现。
|
||||
"""
|
||||
query_string = urllib.parse.urlencode(params)
|
||||
headers = _merge_headers({"User-Agent": USER_AGENT}, auth_headers)
|
||||
|
||||
if _HAS_REQUESTS:
|
||||
session = _get_session()
|
||||
try:
|
||||
if method.upper() == "POST":
|
||||
resp = session.post(f"{instance}/search", data=query_string,
|
||||
headers=headers, timeout=timeout)
|
||||
else:
|
||||
resp = session.get(f"{instance}/search?{query_string}",
|
||||
headers=headers, timeout=timeout)
|
||||
if resp.status_code == 404:
|
||||
# 404 = JSON endpoint truly absent → fall back to HTML scraping
|
||||
return None
|
||||
if resp.status_code in (401, 403):
|
||||
# auth / IP issue → raise so it isn't silently masked by an
|
||||
# HTML fallback that would just 403 again.
|
||||
resp.raise_for_status()
|
||||
raw = resp.content.decode("utf-8")
|
||||
if raw.strip().startswith("{") or raw.strip().startswith("["):
|
||||
return _finalize_json_result(json.loads(raw))
|
||||
# Got HTML — JSON unsupported
|
||||
return None
|
||||
except _requests.exceptions.HTTPError as e:
|
||||
if getattr(e, "response", None) is not None and \
|
||||
e.response.status_code == 404:
|
||||
return None
|
||||
raise
|
||||
|
||||
# stdlib path（零依赖可用）
|
||||
if method.upper() == "POST":
|
||||
data = query_string.encode("utf-8")
|
||||
req = urllib.request.Request(f"{instance}/search", data=data, headers=headers, method="POST")
|
||||
req = urllib.request.Request(f"{instance}/search", data=data,
|
||||
headers=headers, method="POST")
|
||||
else:
|
||||
req = urllib.request.Request(f"{instance}/search?{query_string}", headers=headers)
|
||||
req = urllib.request.Request(f"{instance}/search?{query_string}",
|
||||
headers=headers)
|
||||
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
raw = resp.read().decode("utf-8")
|
||||
if raw.strip().startswith("{") or raw.strip().startswith("["):
|
||||
return json.loads(raw)
|
||||
return _finalize_json_result(json.loads(raw))
|
||||
# Got HTML — JSON unsupported
|
||||
return None
|
||||
except urllib.error.HTTPError as e:
|
||||
@@ -559,17 +658,30 @@ def search_html(instance: str, params: dict, timeout: int = 15,
|
||||
v2.2.2：解码改用 common.detect_charset（此前硬编码 utf-8，GBK/Shift-JIS
|
||||
等非 UTF-8 实例的页面会整体乱码）。``encoding`` 为显式覆盖（来自
|
||||
``--language`` 无关的 CLI ``--encoding``），优先级最高。
|
||||
v2.5.0：requests 可用时走模块级 Session（与 search_json 一致）。
|
||||
"""
|
||||
html_params = {k: v for k, v in params.items() if k != "format"}
|
||||
query_string = urllib.parse.urlencode(html_params)
|
||||
url = f"{instance}/search?{query_string}"
|
||||
|
||||
headers = _merge_headers({"User-Agent": USER_AGENT}, auth_headers)
|
||||
|
||||
if _HAS_REQUESTS:
|
||||
try:
|
||||
resp = _get_session().get(url, headers=headers, timeout=timeout)
|
||||
raw = resp.content
|
||||
content_type = resp.headers.get("Content-Type", "")
|
||||
except Exception as e:
|
||||
raise RuntimeError(f"HTML search failed for {instance}: {e}")
|
||||
else:
|
||||
req = urllib.request.Request(url, headers=headers)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
raw = resp.read()
|
||||
content_type = resp.headers.get("Content-Type", "")
|
||||
except Exception as e:
|
||||
raise RuntimeError(f"HTML search failed for {instance}: {e}")
|
||||
|
||||
if encoding:
|
||||
try:
|
||||
html = raw.decode(encoding)
|
||||
@@ -582,8 +694,6 @@ def search_html(instance: str, params: dict, timeout: int = 15,
|
||||
except (UnicodeDecodeError, LookupError):
|
||||
html = raw.decode("utf-8", errors="replace")
|
||||
return parse_html_results(html, query=params.get("q", ""))
|
||||
except Exception as e:
|
||||
raise RuntimeError(f"HTML search failed for {instance}: {e}")
|
||||
|
||||
|
||||
def search_single(instance: str, params: dict, method: str = "GET",
|
||||
@@ -1272,8 +1382,13 @@ class AdaptiveThrottle:
|
||||
"rate limit" in error_msg.lower())
|
||||
if is_rate_limit:
|
||||
self._global_pause_until = time.monotonic() + self._pause_seconds
|
||||
# 连续失败达阈值 → 退避 + 降并发
|
||||
if self._consecutive_failures >= self._failure_threshold:
|
||||
# 连续失败达阈值 → 退避 + 降并发。
|
||||
# v2.5.0：threshold <= 0 表示"禁用自适应退避"——只统计失败次数
|
||||
# （供 stats()/报告展示），不再触发翻倍延迟与降并发。原实现
|
||||
# threshold=0 时 ``0 >= 0`` 恒真，首次失败即退避，与 CLI 帮助
|
||||
# 声称的 "0 disables throttling" 相反。
|
||||
if self._failure_threshold > 0 and \
|
||||
self._consecutive_failures >= self._failure_threshold:
|
||||
self._consecutive_failures = 0
|
||||
self._delay = min(self._delay * 2, self._max_delay)
|
||||
self._concurrency = max(1, self._concurrency // 2)
|
||||
@@ -1334,7 +1449,8 @@ def fetch_top_results(results: dict, count: int, timeout: int = 10,
|
||||
request_delay: float = 0.3,
|
||||
referer: str = None,
|
||||
fallback_enabled: bool = True,
|
||||
throttle: "AdaptiveThrottle" = None) -> list:
|
||||
throttle: "AdaptiveThrottle" = None,
|
||||
total_chars: int = 0) -> list:
|
||||
"""Fetch full text of top N result pages concurrently.
|
||||
|
||||
Features:
|
||||
@@ -1345,10 +1461,17 @@ def fetch_top_results(results: dict, count: int, timeout: int = 10,
|
||||
- Falls back to browser User-Agent if blocked (in fetch_url)
|
||||
- Small delay between requests to avoid rate limits
|
||||
|
||||
v2.5.0 全局字符预算（``total_chars``）：
|
||||
按原始结果顺序从顶部开始分配字符额度——前一页实际消费后剩余的预算
|
||||
留给下一页（顺序填满，保证最重要的结果拿到完整正文）。预算耗尽后
|
||||
剩余 URL 返回 ``status="skipped"``（不发请求），让 AI Agent 明确知道
|
||||
是预算跳过而非抓取失败。默认 0 = 不限制。
|
||||
|
||||
Args:
|
||||
referer: Referer URL（v2.0.0，通常设为 SearXNG 实例 URL）
|
||||
fallback_enabled: 是否启用 Wayback 兜底（默认 True）
|
||||
throttle: 外部传入的 AdaptiveThrottle 实例（可选；不传则内部创建）
|
||||
total_chars: 全局字符预算（v2.5.0；0 = 不限）
|
||||
"""
|
||||
urls = []
|
||||
seen = set()
|
||||
@@ -1370,16 +1493,57 @@ def fetch_top_results(results: dict, count: int, timeout: int = 10,
|
||||
logger.info(f"\nFetching {len(urls)} result pages "
|
||||
f"(timeout={timeout}s, retries={max_retries}, "
|
||||
f"delay={throttle.delay}s, concurrency={throttle.concurrency})...")
|
||||
if total_chars and total_chars > 0:
|
||||
logger.info(f"Char budget: {total_chars:,} total (--fetch-total-chars)")
|
||||
fetched = []
|
||||
ok_count = [0]
|
||||
err_count = [0]
|
||||
skipped_count = [0]
|
||||
anti_bot_count = [0]
|
||||
fallback_count = [0]
|
||||
|
||||
# v2.5.0 全局预算的线程安全计数器：_budget_remaining[0] 是剩余可用字符。
|
||||
# _budget_enabled[0] 区分"未启用预算"（total_chars<=0，无限）与"已耗尽"
|
||||
# （remaining=0）——前者放行所有 URL，后者跳过。_consume_budget 只在
|
||||
# 抓取成功后按实际 text_length 扣减，未用掉的额度自动留给下一页。
|
||||
_budget_enabled = [total_chars is not None and total_chars > 0]
|
||||
_budget_remaining = [total_chars if _budget_enabled[0] else 0]
|
||||
_budget_lock = threading.Lock()
|
||||
|
||||
def _take_budget() -> "tuple":
|
||||
"""分配本 URL 的字符上限。返回 (cap, allowed)。
|
||||
|
||||
cap 为 None 表示预算未启用（无限）。allowed=False 表示预算已耗尽，
|
||||
调用方应跳过本 URL（status="skipped"，不发请求）。
|
||||
"""
|
||||
if not _budget_enabled[0]:
|
||||
return None, True
|
||||
with _budget_lock:
|
||||
remaining = _budget_remaining[0]
|
||||
if remaining <= 0:
|
||||
return 0, False
|
||||
return remaining, True
|
||||
|
||||
def _consume_budget(used: int) -> None:
|
||||
"""抓取成功后按实际消费的字符数扣减预算。"""
|
||||
with _budget_lock:
|
||||
_budget_remaining[0] = max(0, _budget_remaining[0] - used)
|
||||
|
||||
def _fetch_one(u: str) -> dict:
|
||||
"""Fetch one URL with adaptive throttling to avoid rate limiting."""
|
||||
# 全局暂停检查（429 触发）
|
||||
throttle.wait_if_paused()
|
||||
# v2.5.0 预算检查放在取槽位之前：预算耗尽时连并发槽位都不占用
|
||||
cap, allowed = _take_budget()
|
||||
if not allowed:
|
||||
skipped_count[0] += 1
|
||||
logger.info(f" [BUDGET] char budget exhausted, skipping {u[:55]}")
|
||||
return {"url": u, "status": "skipped",
|
||||
"error": "char budget exhausted (--fetch-total-chars)",
|
||||
"text": "", "text_length": 0, "truncated": False,
|
||||
"anti_bot_detected": False, "waf_type": None,
|
||||
"fallback_used": None,
|
||||
"title": None, "latency": None}
|
||||
# v2.2.2：真实并发门控。退避降并发后，超出当前并发目标的新请求
|
||||
# 在这里被拒绝（不占用 fetch_page），实现持久降并发而非名义降并发。
|
||||
if not throttle.acquire_slot():
|
||||
@@ -1403,6 +1567,15 @@ def fetch_top_results(results: dict, count: int, timeout: int = 10,
|
||||
if result["status"] == "ok":
|
||||
ok_count[0] += 1
|
||||
throttle.report_success()
|
||||
# v2.5.0 字符预算：按分配到的上限截断正文（提取后语义截断），
|
||||
# 再按实际消费扣减预算。cap 为 None 表示未启用预算。
|
||||
if cap:
|
||||
text = result.get("text", "")
|
||||
if len(text) > cap:
|
||||
result["text"] = text[:cap]
|
||||
result["text_length"] = cap
|
||||
result["truncated"] = True
|
||||
_consume_budget(result["text_length"])
|
||||
trunc = ", TRUNCATED" if result.get("truncated") else ""
|
||||
ua_note = ""
|
||||
if result.get("user_agent_used") != USER_AGENT:
|
||||
@@ -1445,10 +1618,50 @@ def fetch_top_results(results: dict, count: int, timeout: int = 10,
|
||||
fetched.sort(key=lambda f: url_order.get(f["url"], 999))
|
||||
|
||||
logger.info(f"Fetched: {ok_count[0]} ok, {err_count[0]} errors"
|
||||
f" (anti-bot: {anti_bot_count[0]}, wayback: {fallback_count[0]})")
|
||||
f" (anti-bot: {anti_bot_count[0]}, wayback: {fallback_count[0]}, "
|
||||
f"skipped: {skipped_count[0]})")
|
||||
return fetched
|
||||
|
||||
|
||||
def deduplicate_fetched_content(fetched: list, threshold: float = 0.85) -> int:
|
||||
"""按正文相似度去重已抓取页面（v2.5.0 --dedup-fetched-content）。
|
||||
|
||||
镜像/转载站点的正文与原文高度相似但 URL 不同——URL 级去重
|
||||
（deduplicate_results）无法合并。这里对每个成功页面（status="ok"）
|
||||
的正文计算 SimHash 指纹，与已保留的页面两两比较，后出现的近似重复
|
||||
条目标记为 ``status="duplicate"``（text 清空，保留 url/title/error 供
|
||||
AI 查看原因），避免 AI 阅读同一内容的多个副本浪费 token。
|
||||
|
||||
只对 ok 条目参与去重；error/skipped/duplicate 条目原样保留。
|
||||
``fetched`` 保持原始结果顺序（首个出现的保留，通常 score 最高）。
|
||||
返回标记为 duplicate 的条目数。
|
||||
"""
|
||||
kept = []
|
||||
dup_count = 0
|
||||
for f in fetched:
|
||||
if f.get("status") != "ok":
|
||||
kept.append(f)
|
||||
continue
|
||||
text = f.get("text", "")
|
||||
is_dup = False
|
||||
for k in kept:
|
||||
if k.get("status") != "ok":
|
||||
continue
|
||||
if texts_are_similar(text, k.get("text", ""), threshold=threshold):
|
||||
is_dup = True
|
||||
break
|
||||
if is_dup:
|
||||
dup_count += 1
|
||||
f["status"] = "duplicate"
|
||||
f["text"] = ""
|
||||
f["text_length"] = 0
|
||||
f["error"] = f"duplicate content (similar to {k.get('url', '')})"
|
||||
kept.append(f)
|
||||
else:
|
||||
kept.append(f)
|
||||
return dup_count
|
||||
|
||||
|
||||
def _emit_fetch_report(fetched: list, throttle: "AdaptiveThrottle",
|
||||
fmt: str = "text") -> None:
|
||||
"""输出结构化抓取报告到 stderr。
|
||||
@@ -1491,10 +1704,14 @@ def _emit_fetch_report_text(fetched: list, throttle: "AdaptiveThrottle") -> None
|
||||
total = len(fetched)
|
||||
ok = sum(1 for f in fetched if f.get("status") == "ok")
|
||||
err = total - ok
|
||||
skipped = sum(1 for f in fetched if f.get("status") == "skipped")
|
||||
dup = sum(1 for f in fetched if f.get("status") == "duplicate")
|
||||
err = err - skipped - dup # skipped/duplicate 不是错误，单独统计
|
||||
anti_bot = sum(1 for f in fetched if f.get("anti_bot_detected"))
|
||||
wayback = sum(1 for f in fetched if f.get("fallback_used") == "wayback")
|
||||
lines.append("-" * len(header))
|
||||
lines.append(f"Total: {total} | OK: {ok} | Error: {err} | "
|
||||
f"Skipped: {skipped} | Duplicate: {dup} | "
|
||||
f"Anti-bot blocked: {anti_bot} | Wayback recovered: {wayback}")
|
||||
|
||||
# 自适应限流状态
|
||||
@@ -1507,7 +1724,8 @@ def _emit_fetch_report_text(fetched: list, throttle: "AdaptiveThrottle") -> None
|
||||
|
||||
# JSON 摘要（一行，便于 Agent 解析）
|
||||
summary = {
|
||||
"total": total, "ok": ok, "error": err,
|
||||
"total": total, "ok": ok, "error": err, "skipped": skipped,
|
||||
"duplicate": dup,
|
||||
"anti_bot_blocked": anti_bot, "wayback_recovered": wayback,
|
||||
"throttle": s,
|
||||
}
|
||||
@@ -1551,6 +1769,9 @@ def _emit_fetch_report_json(fetched: list, throttle: "AdaptiveThrottle") -> None
|
||||
total = len(fetched)
|
||||
ok = sum(1 for f in fetched if f.get("status") == "ok")
|
||||
err = total - ok
|
||||
skipped = sum(1 for f in fetched if f.get("status") == "skipped")
|
||||
dup = sum(1 for f in fetched if f.get("status") == "duplicate")
|
||||
err = err - skipped - dup # skipped/duplicate 不是错误，单独统计
|
||||
anti_bot = sum(1 for f in fetched if f.get("anti_bot_detected"))
|
||||
wayback = sum(1 for f in fetched if f.get("fallback_used") == "wayback")
|
||||
|
||||
@@ -1559,6 +1780,8 @@ def _emit_fetch_report_json(fetched: list, throttle: "AdaptiveThrottle") -> None
|
||||
"total": total,
|
||||
"ok": ok,
|
||||
"error": err,
|
||||
"skipped": skipped,
|
||||
"duplicate": dup,
|
||||
"anti_bot_blocked": anti_bot,
|
||||
"wayback_recovered": wayback,
|
||||
"throttle": throttle.stats(),
|
||||
@@ -1782,6 +2005,28 @@ def format_urls(results: dict) -> str:
|
||||
return "\n".join(r.get("url", "") for r in results.get("results", []) if r.get("url"))
|
||||
|
||||
|
||||
# ----- Media-category CSV columns (v2.5.0) -----
|
||||
# SearXNG 的 images 类别结果（template="images.html"）带 img_src/thumbnail_src/
|
||||
# resolution/source，videos 类别（template="videos.html"）带 iframe_src/
|
||||
# thumbnail_src。--format json 已透传这些字段（原始 dict 原样序列化），但
|
||||
# CSV 固定列会丢弃。以下辅助让 CSV 在结果含媒体字段时自动追加对应列，
|
||||
# 表格分析（图片缩略图链接、视频内嵌 URL）直接可用。
|
||||
_MEDIA_COLUMNS = ["img_src", "thumbnail_src", "resolution", "iframe_src", "source"]
|
||||
|
||||
|
||||
def _detect_media_columns(results_iter) -> list:
|
||||
"""返回结果集中实际存在的媒体列子集（保持 _MEDIA_COLUMNS 顺序）。
|
||||
|
||||
仅当至少一个结果包含非空值时才列出该列，避免 CSV 出现整列空白。
|
||||
"""
|
||||
present = set()
|
||||
for r in results_iter:
|
||||
for col in _MEDIA_COLUMNS:
|
||||
if r.get(col):
|
||||
present.add(col)
|
||||
return [c for c in _MEDIA_COLUMNS if c in present]
|
||||
|
||||
|
||||
def _format_results(results: dict, args) -> str:
|
||||
"""Format a results dict into the output string selected by args.format.
|
||||
|
||||
@@ -1796,9 +2041,11 @@ def _format_results(results: dict, args) -> str:
|
||||
if args.format == "csv":
|
||||
out = io.StringIO()
|
||||
writer = csv.writer(out, lineterminator="\n")
|
||||
rs = results.get("results", [])
|
||||
media_cols = _detect_media_columns(rs)
|
||||
writer.writerow(["title", "url", "engine", "score",
|
||||
"published_date", "content"])
|
||||
for r in results.get("results", []):
|
||||
"published_date", "content"] + media_cols)
|
||||
for r in rs:
|
||||
writer.writerow([
|
||||
r.get("title", ""),
|
||||
r.get("url", ""),
|
||||
@@ -1806,7 +2053,7 @@ def _format_results(results: dict, args) -> str:
|
||||
r.get("score", "") if r.get("score") is not None else "",
|
||||
r.get("published_date", ""),
|
||||
r.get("content", ""),
|
||||
])
|
||||
] + [r.get(c, "") for c in media_cols])
|
||||
return out.getvalue().rstrip()
|
||||
# brief (also the safe fallthrough)
|
||||
output = format_brief(results, snippet_len=args.snippet_len)
|
||||
@@ -1818,6 +2065,10 @@ def _format_results(results: dict, args) -> str:
|
||||
output += f"\n--- {f['url']} ---\n"
|
||||
if f["status"] == "ok":
|
||||
output += f"{f['text']}\n"
|
||||
elif f["status"] == "skipped":
|
||||
output += f"[SKIPPED: {f.get('error', 'char budget exhausted')}]\n"
|
||||
elif f["status"] == "duplicate":
|
||||
output += f"[DUPLICATE: {f.get('error', 'similar content')}]\n"
|
||||
else:
|
||||
output += f"[ERROR: {f.get('error', 'unknown')}]\n"
|
||||
return output
|
||||
@@ -2045,13 +2296,28 @@ def _run_single_query(query: str, args, instance_urls: list,
|
||||
referer=referer,
|
||||
fallback_enabled=not getattr(args, "no_fallback", False),
|
||||
throttle=fetch_throttle,
|
||||
total_chars=getattr(args, "fetch_total_chars", 0),
|
||||
)
|
||||
# Emit fetch_ok / fetch_fail events
|
||||
# v2.5.0: 正文相似度去重（--dedup-fetched-content），复用
|
||||
# --similarity-threshold 阈值。镜像/转载页标记为 status="duplicate"。
|
||||
if getattr(args, "dedup_fetched_content", False):
|
||||
dup_n = deduplicate_fetched_content(fetched,
|
||||
threshold=args.similarity_threshold)
|
||||
if dup_n:
|
||||
logger.info(f"Fetched-content dedup: {dup_n} duplicate page(s) "
|
||||
f"dropped (threshold={args.similarity_threshold})")
|
||||
# Emit fetch_ok / fetch_fail / fetch_duplicate / fetch_skip events
|
||||
for f in fetched:
|
||||
if f.get("status") == "ok":
|
||||
emit_progress("fetch_ok", url=f.get("url", ""),
|
||||
chars=f.get("text_length", 0),
|
||||
fallback=f.get("fallback_used"))
|
||||
elif f.get("status") == "duplicate":
|
||||
emit_progress("fetch_duplicate", url=f.get("url", ""),
|
||||
error=f.get("error", "duplicate content"))
|
||||
elif f.get("status") == "skipped":
|
||||
emit_progress("fetch_skip", url=f.get("url", ""),
|
||||
error=f.get("error", "budget exhausted"))
|
||||
else:
|
||||
emit_progress("fetch_fail", url=f.get("url", ""),
|
||||
error=f.get("error", "unknown"),
|
||||
@@ -2110,7 +2376,15 @@ def _get_output_schema():
|
||||
"url": {"type": "string"},
|
||||
"final_url": {"type": ["string", "null"],
|
||||
"description": "URL after redirects / Wayback."},
|
||||
"status": {"type": "string", "enum": ["ok", "error"]},
|
||||
"status": {"type": "string",
|
||||
"enum": ["ok", "error", "skipped", "duplicate"],
|
||||
"description": "'skipped' (v2.5.0) means the page was "
|
||||
"not fetched because the "
|
||||
"--fetch-total-chars budget was "
|
||||
"exhausted. 'duplicate' (v2.5.0) means "
|
||||
"the body was near-identical to an "
|
||||
"earlier page and was dropped by "
|
||||
"--dedup-fetched-content."},
|
||||
"content_type": {"type": ["string", "null"]},
|
||||
"text": {"type": "string", "description": "Extracted page text."},
|
||||
"text_length": {"type": "integer"},
|
||||
@@ -2371,6 +2645,7 @@ def _save_config(args, path: str) -> None:
|
||||
"fetch": "fetch",
|
||||
"fetch_timeout": "fetch_timeout",
|
||||
"fetch_retries": "fetch_retries",
|
||||
"fetch_total_chars": "fetch_total_chars",
|
||||
"throttle_failure_threshold": "throttle_failure_threshold",
|
||||
"throttle_pause_seconds": "throttle_pause_seconds",
|
||||
"throttle_max_delay": "throttle_max_delay",
|
||||
@@ -2421,6 +2696,13 @@ def _dry_run_preview(args, instance_urls: list, auth_headers: dict) -> None:
|
||||
elif args.queries_file:
|
||||
preview["action"] = "batch"
|
||||
preview["queries_file"] = args.queries_file
|
||||
# v2.5.0: dry-run 批量模式打印实际查询列表（读本地文件，不发 HTTP）。
|
||||
# 读取失败时降级为仅文件名，不阻塞预览。
|
||||
try:
|
||||
qs = _read_queries_file(args.queries_file)
|
||||
preview["queries"] = qs
|
||||
except RuntimeError as e:
|
||||
preview["queries_error"] = str(e)
|
||||
elif args.verify:
|
||||
preview["action"] = "verify"
|
||||
else:
|
||||
@@ -2539,19 +2821,27 @@ def _handle_research(args, instance_urls: list, auth_headers: dict,
|
||||
error_count = 0
|
||||
for i, (angle, q) in enumerate(research_queries, 1):
|
||||
logger.info(f"\n[{i}/{len(research_queries)}] [{angle}] {q}")
|
||||
# v2.5.0: 角度级进度事件——_run_single_query 内部的事件无法区分
|
||||
# "当前第几个角度"，加 angle_* 事件让 AI Agent 能跟踪多角度进度
|
||||
emit_progress("angle_start", angle=angle, query=q,
|
||||
index=i, total=len(research_queries))
|
||||
results, err, err_code = _run_single_query(q, args, instance_urls,
|
||||
auth_headers, ttl_seconds)
|
||||
if err:
|
||||
error_count += 1
|
||||
logger.error(f" [ERROR] {err}")
|
||||
emit_progress("angle_fail", angle=angle, query=q,
|
||||
error=err, error_code=err_code)
|
||||
entry = {"query": q, "angle": angle, "status": "error",
|
||||
"error": err}
|
||||
if err_code:
|
||||
entry["error_code"] = err_code
|
||||
batch.append(entry)
|
||||
else:
|
||||
if len(results.get("results", [])) > 0:
|
||||
angle_count = len(results.get("results", []))
|
||||
if angle_count > 0:
|
||||
any_with_results = True
|
||||
emit_progress("angle_ok", angle=angle, query=q, results=angle_count)
|
||||
batch.append({"query": q, "angle": angle, "status": "ok",
|
||||
"results": results})
|
||||
|
||||
@@ -2585,8 +2875,12 @@ def _handle_research(args, instance_urls: list, auth_headers: dict,
|
||||
elif args.format == "csv":
|
||||
out = io.StringIO()
|
||||
writer = csv.writer(out, lineterminator="\n")
|
||||
# v2.5.0: 媒体列检测——研究模式内任意角度含图片/视频字段则追加
|
||||
media_cols = _detect_media_columns(
|
||||
br["results"].get("results", [])
|
||||
for br in batch if "results" in br)
|
||||
writer.writerow(["angle", "query", "title", "url", "engine",
|
||||
"score", "published_date", "content"])
|
||||
"score", "published_date", "content"] + media_cols)
|
||||
for br in batch:
|
||||
q = br["query"]
|
||||
angle = br.get("angle", "")
|
||||
@@ -2600,7 +2894,7 @@ def _handle_research(args, instance_urls: list, auth_headers: dict,
|
||||
r.get("score", "") if r.get("score") is not None else "",
|
||||
r.get("published_date", ""),
|
||||
r.get("content", ""),
|
||||
])
|
||||
] + [r.get(c, "") for c in media_cols])
|
||||
else:
|
||||
writer.writerow([angle, q, "", "", "", "", "",
|
||||
f"[ERROR: {br['error']}]"])
|
||||
@@ -2772,8 +3066,12 @@ def _handle_batch(args, instance_urls: list, auth_headers: dict,
|
||||
elif args.format == "csv":
|
||||
out = io.StringIO()
|
||||
writer = csv.writer(out, lineterminator="\n")
|
||||
# v2.5.0: 媒体列检测——批量内任意结果含图片/视频字段则追加对应列
|
||||
media_cols = _detect_media_columns(
|
||||
br["results"].get("results", [])
|
||||
for br in batch if "results" in br)
|
||||
writer.writerow(["query", "title", "url", "engine", "score",
|
||||
"published_date", "content"])
|
||||
"published_date", "content"] + media_cols)
|
||||
for br in batch:
|
||||
q = br["query"]
|
||||
if "results" in br:
|
||||
@@ -2786,7 +3084,7 @@ def _handle_batch(args, instance_urls: list, auth_headers: dict,
|
||||
r.get("score", "") if r.get("score") is not None else "",
|
||||
r.get("published_date", ""),
|
||||
r.get("content", ""),
|
||||
])
|
||||
] + [r.get(c, "") for c in media_cols])
|
||||
else:
|
||||
writer.writerow([q, "", "", "", "", "",
|
||||
f"[ERROR: {br['error']}]"])
|
||||
@@ -2992,6 +3290,14 @@ Use --config FILE to load a non-default config file (overrides the auto-discover
|
||||
metavar="FLOAT",
|
||||
help="相似度去重阈值（默认: 0.85）。越高越严格，1.0 要求标题几乎完全相同。"
|
||||
"仅当 --similarity-dedup 启用时生效。")
|
||||
parser.add_argument("--dedup-fetched-content", action="store_true",
|
||||
help="v2.5.0: deduplicate fetched page content by "
|
||||
"similarity (SimHash). Mirror/republished pages "
|
||||
"with near-identical bodies are marked "
|
||||
"status='duplicate' (text cleared) — keeps AI "
|
||||
"from reading the same content multiple times. "
|
||||
"Uses --similarity-threshold (default 0.85). "
|
||||
"Only applies with --fetch.")
|
||||
parser.add_argument("--format", "-f", choices=["json", "brief", "urls", "csv"],
|
||||
default=config.get("format", "json"),
|
||||
help="Output format (default: json). 'csv' exports "
|
||||
@@ -3004,6 +3310,16 @@ Use --config FILE to load a non-default config file (overrides the auto-discover
|
||||
help="Timeout per page fetch in seconds (default: 10)")
|
||||
parser.add_argument("--fetch-retries", type=int, default=_cfg_int(config, "fetch_retries", 3),
|
||||
help="Max retries per page fetch (default: 3)")
|
||||
parser.add_argument("--fetch-total-chars", type=int,
|
||||
default=_cfg_int(config, "fetch_total_chars", 0),
|
||||
metavar="N",
|
||||
help="v2.5.0: global character budget for --fetch. "
|
||||
"Chars are allocated top-down in result order — "
|
||||
"unused budget rolls over to the next page. Once "
|
||||
"exhausted, remaining URLs are marked status="
|
||||
"'skipped' (no request sent). Lets token-limited "
|
||||
"agents cap total fetched content. 0 = unlimited "
|
||||
"(default).")
|
||||
parser.add_argument("--fetch-report", dest="fetch_report",
|
||||
nargs="?", const="text", default=False, metavar="FORMAT",
|
||||
help="When used with --fetch, emit a structured fetch report to stderr "
|
||||
|
||||
@@ -16,6 +16,22 @@ if str(SCRIPTS_DIR) not in sys.path:
|
||||
import pytest
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def force_stdlib_http_path(monkeypatch):
|
||||
"""强制所有测试走 stdlib HTTP 路径（v2.5.0）。
|
||||
|
||||
search.py 的 search_json/search_html 自 v2.5.0 起在 requests 可用时
|
||||
走模块级 Session（连接池复用），但既有测试 mock 的是
|
||||
``urllib.request.urlopen``——本机装了 requests 时这些 mock 会失效。
|
||||
统一强制 stdlib 路径，让既有测试原样工作且互不干扰；requests 路径
|
||||
的回归由端到端/专项测试覆盖。
|
||||
"""
|
||||
import search as _search
|
||||
import fetch as _fetch
|
||||
monkeypatch.setattr(_search, "_HAS_REQUESTS", False)
|
||||
monkeypatch.setattr(_fetch, "_HAS_REQUESTS", False)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def isolated_cache(monkeypatch, tmp_path):
|
||||
"""Redirect the SQLite cache to a per-test temp directory.
|
||||
|
||||
@@ -0,0 +1,918 @@
|
||||
"""Tests for v2.5.0 changes.
|
||||
|
||||
Covers:
|
||||
* Sec-Ch-Ua header construction fix — Chrome/Edge headers must be valid
|
||||
(no double-quoted first brand, no rstrip-clipped version quotes)
|
||||
* AdaptiveThrottle ``--throttle-failure-threshold 0`` = disabled semantics
|
||||
* search 403 fast-fail — 403 no longer backoff-retried (immediate
|
||||
failover); 429/5xx still retried
|
||||
* fetch.py ``--extract json`` structured skeleton output (bs4 + stdlib
|
||||
parity, caps, CLI integration, implies --format json)
|
||||
* fetch.py ``--max-chars N`` semantic truncation (CLI, truncated flag)
|
||||
* fetch.py ``--log-format json`` + ``--dump-schema`` (CLI)
|
||||
* ``--fetch-total-chars N`` global budget (allocation, truncation,
|
||||
skipped status, disabled-by-default)
|
||||
* research-mode angle progress events (angle_start/ok/fail)
|
||||
* CSV media columns for images/videos categories (plus backward compat)
|
||||
* ``--dedup-fetched-content`` SimHash body dedup (mirror/footer-noise/
|
||||
distinct/error-preserved/threshold)
|
||||
* ``--dry-run`` batch mode prints the actual query list
|
||||
* search requests-backed HTTP path (Session reuse, 404/403 handling)
|
||||
* release_check.py version-drift + missing-hint detection
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.error
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
import common as common_mod
|
||||
import _config
|
||||
import search as search_mod
|
||||
import fetch as fetch_mod
|
||||
from common import build_browser_headers
|
||||
from fetch import FetchResult, _emit_fetch_result, extract_structure
|
||||
from search import (
|
||||
AdaptiveThrottle,
|
||||
_retry_with_backoff,
|
||||
deduplicate_fetched_content,
|
||||
fetch_top_results,
|
||||
)
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent
|
||||
sys.path.insert(0, str(PROJECT_ROOT / "scripts"))
|
||||
|
||||
|
||||
def _make_args(**overrides):
|
||||
"""args object matching the argparse.Namespace shape main() produces.
|
||||
|
||||
Extends the v2.3.0 baseline with v2.5.0 fields (fetch_total_chars,
|
||||
dedup_fetched_content) and the fields the research/batch/dry-run paths
|
||||
read (research, research_angles, stream, dry_run, parallel_queries...).
|
||||
"""
|
||||
base = dict(
|
||||
query="test", format="json", method="GET", timeout=15, retry=0,
|
||||
serial=False, no_dedup=False, sort_by="score", max_results=None,
|
||||
include_domain=None, exclude_domain=None, fetch=0, fetch_timeout=10,
|
||||
fetch_retries=3, fetch_total_chars=0, dedup_fetched_content=False,
|
||||
max_size=None, cache_ttl=0, snippet_len=0,
|
||||
categories=None, language=None, pageno=1, pages=1,
|
||||
time_range="year", safesearch=0, engines="google,bing",
|
||||
referer=None, no_fallback=False, fetch_report=False,
|
||||
request_delay=0.3, throttle_failure_threshold=3,
|
||||
throttle_pause_seconds=30, throttle_max_delay=10,
|
||||
similarity_dedup=False, similarity_threshold=0.85,
|
||||
encoding=None, output=None, parallel_queries=0,
|
||||
research=None, research_angles=None, stream=False,
|
||||
progress=False, dry_run=False, verify=False, fail_fast=False,
|
||||
queries_file=None, save_config=None, dump_schema=False,
|
||||
cache_max_size=0, clear_cache=False, cache_stats=False,
|
||||
proxy=None, log_format="text", config=None, instance=None,
|
||||
auth_bearer=None, auth_basic=None, auth_bearer_file=None,
|
||||
auth_basic_file=None, verbose=False, quiet=False,
|
||||
include_domains=None, snippet_len_brief=0,
|
||||
)
|
||||
base.update(overrides)
|
||||
return SimpleNamespace(**base)
|
||||
|
||||
|
||||
# ===== Sec-Ch-Ua header construction fix (v2.5.0) =====
|
||||
|
||||
def _chrome_ua():
|
||||
return next(ua for ua in _config.UA_POOL
|
||||
if "Chrome/" in ua and "Edg/" not in ua and "Firefox/" not in ua)
|
||||
|
||||
|
||||
def _edge_ua():
|
||||
return next(ua for ua in _config.UA_POOL if "Edg/" in ua)
|
||||
|
||||
|
||||
def test_sec_ch_ua_chrome_first_brand_has_single_quotes():
|
||||
"""Chrome Sec-Ch-Ua must not have double-quoted first brand.
|
||||
|
||||
Regression (v2.4.0): the first brand variable already contained quotes
|
||||
and was wrapped in the f-string again, producing
|
||||
``""Not_A Brand";v="99""`` — a malformed header strict WAFs ignore.
|
||||
"""
|
||||
h = build_browser_headers(_chrome_ua())
|
||||
value = h["Sec-Ch-Ua"]
|
||||
assert '"Not_A Brand";v="99"' in value
|
||||
assert '""Not_A Brand' not in value # no doubled opening quote
|
||||
|
||||
|
||||
def test_sec_ch_ua_chrome_brands_complete():
|
||||
"""Chrome header carries Not_A Brand + Chromium + Google Chrome."""
|
||||
value = build_browser_headers(_chrome_ua())["Sec-Ch-Ua"]
|
||||
assert '"Not_A Brand";v="99"' in value
|
||||
assert '"Chromium";v="' in value
|
||||
assert '"Google Chrome";v="' in value
|
||||
|
||||
|
||||
def test_sec_ch_ua_edge_includes_edge_brand_valid():
|
||||
"""Edge Sec-Ch-Ua includes Microsoft Edge AND keeps valid syntax.
|
||||
|
||||
Regression (v2.4.0): rstrip('"') clipped the previous brand's version
|
||||
quotes, producing ``"Google Chrome";v="138, "Microsoft Edge";v="138"`` —
|
||||
detectable as a `"138, ` (version value directly followed by comma).
|
||||
"""
|
||||
value = build_browser_headers(_edge_ua())["Sec-Ch-Ua"]
|
||||
assert '"Microsoft Edge";v="' in value
|
||||
# Each brand is a self-contained quoted pair — no dangling unclosed quote
|
||||
assert value.count('"') % 2 == 0
|
||||
assert '"138, ' not in value # no clipped version value
|
||||
|
||||
|
||||
def test_sec_ch_ua_version_matches_ua():
|
||||
"""Brand version must equal the UA's Chrome major version."""
|
||||
ua = _chrome_ua()
|
||||
import re
|
||||
ver = re.search(r"Chrome/(\d+)", ua).group(1)
|
||||
value = build_browser_headers(ua)["Sec-Ch-Ua"]
|
||||
assert f'"Chromium";v="{ver}"' in value
|
||||
assert f'"Google Chrome";v="{ver}"' in value
|
||||
|
||||
|
||||
# ===== AdaptiveThrottle threshold=0 = disabled semantics (v2.5.0) =====
|
||||
|
||||
def test_throttle_zero_threshold_never_escalates():
|
||||
"""threshold=0 disables adaptive backoff entirely (delay/concurrency)."""
|
||||
t = AdaptiveThrottle(0.3, 5, failure_threshold=0)
|
||||
for _ in range(10):
|
||||
t.report_failure()
|
||||
assert t.delay == 0.3 # never doubled
|
||||
assert t.concurrency == 5 # never halved
|
||||
|
||||
|
||||
def test_throttle_zero_threshold_still_counts_failures():
|
||||
"""threshold=0 still tracks the failure counter for stats/reports."""
|
||||
t = AdaptiveThrottle(0.3, 5, failure_threshold=0)
|
||||
t.report_failure()
|
||||
t.report_failure()
|
||||
assert t.stats()["consecutive_failures"] == 2
|
||||
|
||||
|
||||
def test_throttle_zero_threshold_429_pause_still_works():
|
||||
"""threshold=0 disables backoff but NOT the 429 global pause."""
|
||||
t = AdaptiveThrottle(0.3, 5, failure_threshold=0, pause_seconds=30)
|
||||
t.report_failure("HTTP 429 Too Many Requests")
|
||||
assert t.stats()["global_paused"] is True
|
||||
|
||||
|
||||
# ===== search 403 fast-fail (v2.5.0) =====
|
||||
|
||||
def _http_error(code, headers=None):
|
||||
return urllib.error.HTTPError("https://inst.example.com", code,
|
||||
"reason", headers or {}, None)
|
||||
|
||||
|
||||
def test_retry_with_backoff_403_raises_immediately():
|
||||
"""403 is NOT in the search retry set — raises without sleeping.
|
||||
|
||||
Regression (v2.4.0): 403 was retried ~3x with backoff (~10.5s wasted)
|
||||
even though search never switches UAs, so instance-level 403s (auth/
|
||||
ban) only failed over after the backoff budget was exhausted.
|
||||
"""
|
||||
err = _http_error(403)
|
||||
|
||||
def raiser():
|
||||
raise err
|
||||
|
||||
with patch("search.time.sleep") as mock_sleep:
|
||||
with pytest.raises(urllib.error.HTTPError) as exc_info:
|
||||
_retry_with_backoff(raiser, max_retries=3)
|
||||
assert exc_info.value.code == 403
|
||||
mock_sleep.assert_not_called()
|
||||
|
||||
|
||||
def test_retry_with_backoff_429_still_retries():
|
||||
"""429 remains retryable (backoff honored) — behavior unchanged."""
|
||||
calls = [0]
|
||||
err = _http_error(429, {"Retry-After": "0"})
|
||||
|
||||
def flaky():
|
||||
calls[0] += 1
|
||||
if calls[0] == 1:
|
||||
raise err
|
||||
return "ok"
|
||||
|
||||
with patch("search.time.sleep"):
|
||||
assert _retry_with_backoff(flaky, max_retries=3) == "ok"
|
||||
assert calls[0] == 2
|
||||
|
||||
|
||||
def test_retry_with_backoff_500_still_retries():
|
||||
"""5xx remains retryable — behavior unchanged."""
|
||||
calls = [0]
|
||||
err = _http_error(503)
|
||||
|
||||
def flaky():
|
||||
calls[0] += 1
|
||||
if calls[0] == 1:
|
||||
raise err
|
||||
return "ok"
|
||||
|
||||
with patch("search.time.sleep"):
|
||||
assert _retry_with_backoff(flaky, max_retries=3) == "ok"
|
||||
assert calls[0] == 2
|
||||
|
||||
|
||||
def test_search_multi_403_fails_over_fast():
|
||||
"""Serial failover: instance1 403 → instance2 succeeds, no sleeps."""
|
||||
err = _http_error(403)
|
||||
ok = {"results": [{"url": "https://ok.com", "title": "OK"}]}
|
||||
with patch.object(search_mod, "search_single",
|
||||
side_effect=[err, ok]) as mock_ss:
|
||||
with patch("search.time.sleep") as mock_sleep:
|
||||
r = search_mod.search_multi(
|
||||
["https://a.example.com", "https://b.example.com"],
|
||||
{"q": "t"}, parallel=False)
|
||||
assert r["results"][0]["url"] == "https://ok.com"
|
||||
assert mock_ss.call_count == 2
|
||||
mock_sleep.assert_not_called() # 403 fast-fail: no backoff wait
|
||||
|
||||
|
||||
# ===== fetch.py --extract json structured skeleton (v2.5.0) =====
|
||||
|
||||
def _sample_html():
|
||||
return (
|
||||
"<html><head><title>Test Page Title</title>"
|
||||
'<meta name="description" content="A meta description">'
|
||||
"</head><body>"
|
||||
"<h1>Main Heading</h1><p>Intro.</p>"
|
||||
"<h2>Sub Section</h2><p>More.</p>"
|
||||
'<a href="/link1">L1</a><a href="/link2">L2</a>'
|
||||
'<img src="/img1.png"><img data-src="/img2.jpg">'
|
||||
"</body></html>"
|
||||
)
|
||||
|
||||
|
||||
def test_extract_structure_bs4_full():
|
||||
"""bs4 path extracts title/meta/headings/links/images."""
|
||||
s = extract_structure(_sample_html())
|
||||
assert s["title"] == "Test Page Title"
|
||||
assert s["meta_description"] == "A meta description"
|
||||
assert s["headings"] == ["h1: Main Heading", "h2: Sub Section"]
|
||||
assert s["links"] == ["/link1", "/link2"]
|
||||
assert "/img1.png" in s["images"] and "/img2.jpg" in s["images"]
|
||||
|
||||
|
||||
def test_extract_structure_stdlib_fallback_parity():
|
||||
"""stdlib HTMLParser path outputs the same shape as bs4."""
|
||||
fetch_mod._HAS_BS4 = False
|
||||
try:
|
||||
s = extract_structure(_sample_html())
|
||||
finally:
|
||||
fetch_mod._HAS_BS4 = True
|
||||
assert s["title"] == "Test Page Title"
|
||||
assert s["headings"] == ["h1: Main Heading", "h2: Sub Section"]
|
||||
assert s["links"] == ["/link1", "/link2"]
|
||||
assert s["images"] == ["/img1.png", "/img2.jpg"]
|
||||
|
||||
|
||||
def test_extract_structure_limits_caps():
|
||||
"""headings/links/images are capped to avoid huge outputs."""
|
||||
html = "<html><body>"
|
||||
html += "".join(f"<h1>Heading {i}</h1>" for i in range(80))
|
||||
html += "".join(f'<a href="/l{i}">L</a>' for i in range(200))
|
||||
html += "".join(f'<img src="/i{i}">' for i in range(80))
|
||||
html += "</body></html>"
|
||||
s = extract_structure(html)
|
||||
assert len(s["headings"]) == 50
|
||||
assert len(s["links"]) == 100
|
||||
assert len(s["images"]) == 50
|
||||
|
||||
|
||||
def test_extract_structure_empty():
|
||||
assert extract_structure("") == {
|
||||
"title": "", "meta_description": "", "headings": [], "links": [],
|
||||
"images": [],
|
||||
}
|
||||
|
||||
|
||||
def test_emit_fetch_result_json_includes_structure(capsys):
|
||||
"""--extract json structure merges into the json success payload."""
|
||||
structure = {"title": "T", "meta_description": "M", "headings": ["h1: H"],
|
||||
"links": ["/a"], "images": []}
|
||||
args = SimpleNamespace(url="https://x.example.com", extract="json",
|
||||
format="json", output=None)
|
||||
_emit_fetch_result(args, "body text", "https://x.example.com",
|
||||
"https://x.example.com/", "text/html", False,
|
||||
user_agent="UA", structure=structure)
|
||||
out, _ = capsys.readouterr()
|
||||
data = json.loads(out)
|
||||
assert data["title"] == "T"
|
||||
assert data["headings"] == ["h1: H"]
|
||||
assert data["text_length"] == len("body text")
|
||||
|
||||
|
||||
# ===== fetch.py CLI: --extract json / --max-chars / --dump-schema =====
|
||||
|
||||
def _run_fetch_main(args_list, capsys, fetch_result):
|
||||
"""Run fetch.main() with patched fetch_url + argv.
|
||||
|
||||
Returns (stdout, stderr) via capsys — callers must use the returned
|
||||
tuple, NOT call capsys.readouterr() again (it is consumed here).
|
||||
"""
|
||||
_fresh_logging() # rebind handlers to this test's capsys stderr
|
||||
with patch.object(fetch_mod, "fetch_url", return_value=fetch_result):
|
||||
with patch.object(sys, "argv", ["fetch.py"] + args_list):
|
||||
fetch_mod.main()
|
||||
return capsys.readouterr()
|
||||
|
||||
|
||||
def _fresh_logging():
|
||||
"""Clear the module-level 'searxng' logger handlers.
|
||||
|
||||
setup_logging() caches a StreamHandler bound to the stderr of whichever
|
||||
test first called fetch/main().main() — a later test's capsys captures a
|
||||
different stderr, so handler output silently goes to the stale stream.
|
||||
Removing handlers forces setup_logging to re-create one for the current
|
||||
capsys context.
|
||||
"""
|
||||
import logging as _logging
|
||||
root = _logging.getLogger("searxng")
|
||||
for h in list(root.handlers):
|
||||
root.removeHandler(h)
|
||||
|
||||
|
||||
def _fetch_result(content, content_type="text/html"):
|
||||
return FetchResult(content=content, content_type=content_type,
|
||||
final_url="https://x.example.com", truncated=False,
|
||||
user_agent="TestUA")
|
||||
|
||||
|
||||
def test_fetch_cli_extract_json(capsys):
|
||||
"""fetch.py -u URL --extract json emits structured JSON with skeleton."""
|
||||
html = _sample_html()
|
||||
out, _ = _run_fetch_main(["-u", "https://x.example.com", "--extract", "json"],
|
||||
capsys, _fetch_result(html))
|
||||
data = json.loads(out)
|
||||
assert data["status"] == "ok"
|
||||
assert data["title"] == "Test Page Title"
|
||||
assert data["headings"] == ["h1: Main Heading", "h2: Sub Section"]
|
||||
assert data["extract"] == "json"
|
||||
|
||||
|
||||
def test_fetch_cli_extract_json_implies_format_json(capsys):
|
||||
"""--extract json with --format text still emits JSON (not raw text)."""
|
||||
html = _sample_html()
|
||||
out, _ = _run_fetch_main(["-u", "https://x.example.com", "--extract", "json",
|
||||
"--format", "text"], capsys, _fetch_result(html))
|
||||
data = json.loads(out) # would fail if raw text leaked to stdout
|
||||
assert data["status"] == "ok"
|
||||
|
||||
|
||||
def test_fetch_cli_max_chars_truncates(capsys):
|
||||
"""--max-chars N truncates extracted text and sets truncated=true."""
|
||||
long_html = "<html><body><article>" + ("lorem ipsum dolor " * 100) + \
|
||||
"</article></body></html>"
|
||||
out, _ = _run_fetch_main(["-u", "https://x.example.com", "--format", "json",
|
||||
"--max-chars", "30"], capsys,
|
||||
_fetch_result(long_html))
|
||||
data = json.loads(out)
|
||||
assert data["truncated"] is True
|
||||
assert data["text_length"] == 30
|
||||
|
||||
|
||||
def test_fetch_cli_max_chars_no_truncation_within_limit(capsys):
|
||||
"""--max-chars larger than the text leaves it untouched."""
|
||||
out, _ = _run_fetch_main(["-u", "https://x.example.com", "--format", "json",
|
||||
"--max-chars", "5000"], capsys,
|
||||
_fetch_result(_sample_html()))
|
||||
data = json.loads(out)
|
||||
assert data["truncated"] is False
|
||||
|
||||
|
||||
def test_fetch_cli_dump_schema(capsys):
|
||||
"""fetch.py --dump-schema exits 0 and prints a JSON schema (no --url)."""
|
||||
_fresh_logging() # avoid stale-handler pollution across capsys tests
|
||||
with pytest.raises(SystemExit) as exc_info:
|
||||
with patch.object(sys, "argv", ["fetch.py", "--dump-schema"]):
|
||||
fetch_mod.main()
|
||||
assert exc_info.value.code == 0
|
||||
out, _ = capsys.readouterr()
|
||||
schema = json.loads(out)
|
||||
assert "properties" in schema
|
||||
assert schema["properties"]["status"]["enum"] == ["ok", "error"]
|
||||
# v2.5.0: extract enum includes 'json'
|
||||
assert "json" in schema["properties"]["extract"]["enum"]
|
||||
|
||||
|
||||
def test_fetch_cli_log_format_json(capsys):
|
||||
"""fetch.py --log-format json emits JSON log lines on stderr.
|
||||
|
||||
stdout stays pure data (extracted text by default); the JSON-formatted
|
||||
logs go to stderr as one JSON object per line.
|
||||
"""
|
||||
html = _sample_html()
|
||||
out, err = _run_fetch_main(["-u", "https://x.example.com",
|
||||
"--log-format", "json"], capsys,
|
||||
_fetch_result(html))
|
||||
# stdout is the extracted text (data), not JSON
|
||||
assert "Main Heading" in out
|
||||
# stderr carries JSON log lines
|
||||
assert any(line.lstrip().startswith("{") for line in err.splitlines())
|
||||
|
||||
|
||||
# ===== --fetch-total-chars global budget (v2.5.0) =====
|
||||
|
||||
def _make_search_results(n=3):
|
||||
return {"results": [{"url": f"https://example{i}.com", "title": f"T{i}"}
|
||||
for i in range(n)]}
|
||||
|
||||
|
||||
def _make_ok_page(url, size=500):
|
||||
return {"url": url, "status": "ok", "text": "x" * size,
|
||||
"text_length": size, "truncated": False,
|
||||
"anti_bot_detected": False, "waf_type": None,
|
||||
"fallback_used": None, "title": "T", "latency": 0.1,
|
||||
"user_agent_used": "TestUA", "final_url": url}
|
||||
|
||||
|
||||
def test_fetch_top_results_budget_allocation_and_skip():
|
||||
"""Budget 800 with 3×500-char pages: 500 + 300(truncated) + skipped."""
|
||||
calls = []
|
||||
|
||||
def _fake_fetch(url, **kwargs):
|
||||
calls.append(url)
|
||||
return _make_ok_page(url)
|
||||
|
||||
with patch.object(search_mod, "fetch_page", side_effect=_fake_fetch):
|
||||
out = fetch_top_results(_make_search_results(3), 3, total_chars=800,
|
||||
request_delay=0)
|
||||
assert len(calls) == 2 # third page skipped without a request
|
||||
statuses = {f["url"]: f["status"] for f in out}
|
||||
assert statuses["https://example0.com"] == "ok"
|
||||
assert statuses["https://example1.com"] == "ok"
|
||||
assert statuses["https://example2.com"] == "skipped"
|
||||
ok_items = [f for f in out if f["status"] == "ok"]
|
||||
assert ok_items[0]["text_length"] == 500
|
||||
assert ok_items[1]["text_length"] == 300 # remaining budget applied
|
||||
assert ok_items[1]["truncated"] is True
|
||||
skip = next(f for f in out if f["status"] == "skipped")
|
||||
assert "budget" in skip["error"].lower()
|
||||
|
||||
|
||||
def test_fetch_top_results_budget_disabled_by_default():
|
||||
"""total_chars=0 (or None) → full fetch, no truncation, no skipping."""
|
||||
for budget in (0, None):
|
||||
calls = []
|
||||
|
||||
def _fake_fetch(url, **kwargs):
|
||||
calls.append(url)
|
||||
return _make_ok_page(url)
|
||||
|
||||
with patch.object(search_mod, "fetch_page", side_effect=_fake_fetch):
|
||||
out = fetch_top_results(_make_search_results(3), 3,
|
||||
total_chars=budget, request_delay=0)
|
||||
assert len(calls) == 3
|
||||
assert all(f["status"] == "ok" for f in out)
|
||||
assert all(f["text_length"] == 500 for f in out)
|
||||
|
||||
|
||||
def test_fetch_top_results_budget_small_caps_first_only():
|
||||
"""Budget 100: first page truncated to 100, remaining two skipped."""
|
||||
calls = []
|
||||
|
||||
def _fake_fetch(url, **kwargs):
|
||||
calls.append(url)
|
||||
return _make_ok_page(url)
|
||||
|
||||
with patch.object(search_mod, "fetch_page", side_effect=_fake_fetch):
|
||||
out = fetch_top_results(_make_search_results(3), 3, total_chars=100,
|
||||
request_delay=0)
|
||||
assert len(calls) == 1
|
||||
assert out[0]["status"] == "ok" and out[0]["text_length"] == 100
|
||||
assert sum(1 for f in out if f["status"] == "skipped") == 2
|
||||
|
||||
|
||||
def test_fetch_top_results_budget_rolls_over_unused():
|
||||
"""Unused budget on a short page rolls over to the next."""
|
||||
def _fake_fetch(url, **kwargs):
|
||||
if url == "https://example0.com":
|
||||
return _make_ok_page(url, size=200) # consumes only 200 of 500
|
||||
return _make_ok_page(url, size=500)
|
||||
|
||||
with patch.object(search_mod, "fetch_page", side_effect=_fake_fetch):
|
||||
out = fetch_top_results(_make_search_results(3), 3, total_chars=500,
|
||||
request_delay=0)
|
||||
assert out[0]["status"] == "ok" and out[0]["text_length"] == 200
|
||||
assert out[1]["status"] == "ok" and out[1]["text_length"] == 300 # rollover
|
||||
assert out[2]["status"] == "skipped"
|
||||
|
||||
|
||||
# ===== research-mode angle progress events (v2.5.0) =====
|
||||
|
||||
def test_research_emits_angle_progress_events(monkeypatch, capsys):
|
||||
"""angle_start/angle_ok events carry angle + index/total on stderr."""
|
||||
search_mod.set_progress_enabled(True)
|
||||
try:
|
||||
def fake_multi(urls, params, **kw):
|
||||
return {"results": [{"url": "https://r.com/1", "title": "R"}]}
|
||||
|
||||
args = _make_args(research="some topic", format="json")
|
||||
with patch.object(search_mod, "search_multi", side_effect=fake_multi):
|
||||
with pytest.raises(SystemExit) as exc_info:
|
||||
search_mod._handle_research(
|
||||
args, ["https://x.example.com"], {}, 0)
|
||||
assert exc_info.value.code == 0
|
||||
out, err = capsys.readouterr()
|
||||
assert '"angle_start"' in err
|
||||
assert '"angle_ok"' in err
|
||||
assert '"index"' in err and '"total"' in err
|
||||
assert json.loads(out)["research_topic"] == "some topic"
|
||||
finally:
|
||||
search_mod.set_progress_enabled(False)
|
||||
|
||||
|
||||
def test_research_emits_angle_fail_event(monkeypatch, capsys):
|
||||
"""A failing angle emits angle_fail with error info."""
|
||||
search_mod.set_progress_enabled(True)
|
||||
try:
|
||||
def fake_multi(urls, params, **kw):
|
||||
raise RuntimeError("network down")
|
||||
|
||||
args = _make_args(research="topic", format="json")
|
||||
with patch.object(search_mod, "search_multi", side_effect=fake_multi):
|
||||
with pytest.raises(SystemExit) as exc_info:
|
||||
search_mod._handle_research(
|
||||
args, ["https://x.example.com"], {}, 0)
|
||||
assert exc_info.value.code == 1 # all angles errored
|
||||
_, err = capsys.readouterr()
|
||||
assert '"angle_fail"' in err
|
||||
assert "network down" in err
|
||||
finally:
|
||||
search_mod.set_progress_enabled(False)
|
||||
|
||||
|
||||
# ===== CSV media columns for images/videos (v2.5.0) =====
|
||||
|
||||
def _csv_args():
|
||||
return SimpleNamespace(format="csv", fetch=0, snippet_len=0)
|
||||
|
||||
|
||||
def test_csv_appends_media_columns_when_present():
|
||||
"""images/videos results get img_src/thumbnail_src/resolution/iframe_src."""
|
||||
from search import _format_results, _detect_media_columns
|
||||
results = {"results": [
|
||||
{"title": "img1", "url": "https://x/i1", "engine": "google",
|
||||
"score": 1.0, "published_date": "", "content": "s",
|
||||
"template": "images.html", "img_src": "https://x/1.jpg",
|
||||
"thumbnail_src": "https://x/1t.jpg", "resolution": "800x600"},
|
||||
{"title": "vid1", "url": "https://x/v1", "engine": "bing",
|
||||
"score": 0.5, "published_date": "", "content": "s",
|
||||
"template": "videos.html", "iframe_src": "https://x/v1.html"},
|
||||
]}
|
||||
csv_text = _format_results(results, _csv_args())
|
||||
header = csv_text.splitlines()[0]
|
||||
assert "img_src" in header and "thumbnail_src" in header
|
||||
assert "resolution" in header and "iframe_src" in header
|
||||
rows = csv_text.splitlines()[1:]
|
||||
assert rows[0].startswith("img1,https://x/i1,google,1.0")
|
||||
assert rows[0].endswith("800x600,") # img_src + thumb + res present
|
||||
assert "https://x/v1.html" in rows[1]
|
||||
|
||||
|
||||
def test_csv_no_media_columns_for_general_results():
|
||||
"""Plain general results keep the original 6 columns (backward compat)."""
|
||||
from search import _format_results
|
||||
results = {"results": [
|
||||
{"title": "t", "url": "https://x/t", "engine": "google",
|
||||
"score": 1.0, "published_date": "", "content": "c"},
|
||||
]}
|
||||
csv_text = _format_results(results, _csv_args())
|
||||
header = csv_text.splitlines()[0]
|
||||
assert header == "title,url,engine,score,published_date,content"
|
||||
assert "img_src" not in header
|
||||
|
||||
|
||||
def test_detect_media_columns_empty():
|
||||
from search import _detect_media_columns
|
||||
assert _detect_media_columns([{"url": "https://x/1"},
|
||||
{"url": "https://x/2"}]) == []
|
||||
assert _detect_media_columns([{"img_src": ""}]) == []
|
||||
|
||||
|
||||
# ===== --dedup-fetched-content SimHash body dedup (v2.5.0) =====
|
||||
|
||||
def _long_body():
|
||||
return ("This is a long article about machine learning and neural "
|
||||
"networks. It covers supervised and unsupervised learning "
|
||||
"approaches in detail. ") * 20
|
||||
|
||||
|
||||
def test_deduplicate_fetched_content_identical():
|
||||
"""Exact duplicate body → status=duplicate, text cleared."""
|
||||
body = _long_body()
|
||||
fetched = [
|
||||
{"url": "https://orig.example/1", "status": "ok", "text": body,
|
||||
"text_length": len(body)},
|
||||
{"url": "https://mirror.example/2", "status": "ok", "text": body,
|
||||
"text_length": len(body)},
|
||||
]
|
||||
n = deduplicate_fetched_content(fetched)
|
||||
assert n == 1
|
||||
assert fetched[0]["status"] == "ok"
|
||||
assert fetched[1]["status"] == "duplicate"
|
||||
assert fetched[1]["text"] == "" and fetched[1]["text_length"] == 0
|
||||
assert "similar" in fetched[1]["error"]
|
||||
|
||||
|
||||
def test_deduplicate_fetched_content_footer_noise():
|
||||
"""Original + site footer noise is still a duplicate (first-1000 window)."""
|
||||
body = _long_body()
|
||||
with_footer = body + ("\n\nCopyright 2026 Example Network. All rights "
|
||||
"reserved. Privacy Policy. Terms of Service.")
|
||||
fetched = [
|
||||
{"url": "https://orig.example/1", "status": "ok", "text": body,
|
||||
"text_length": len(body)},
|
||||
{"url": "https://repub.example/2", "status": "ok",
|
||||
"text": with_footer, "text_length": len(with_footer)},
|
||||
]
|
||||
assert deduplicate_fetched_content(fetched) == 1
|
||||
assert fetched[1]["status"] == "duplicate"
|
||||
|
||||
|
||||
def test_deduplicate_fetched_content_distinct_kept():
|
||||
"""Different bodies are kept (no false positive)."""
|
||||
a = ("Completely different content about cooking recipes. ") * 30
|
||||
b = ("Sports news covering the latest football match results. ") * 30
|
||||
fetched = [
|
||||
{"url": "https://a.example", "status": "ok", "text": a,
|
||||
"text_length": len(a)},
|
||||
{"url": "https://b.example", "status": "ok", "text": b,
|
||||
"text_length": len(b)},
|
||||
]
|
||||
assert deduplicate_fetched_content(fetched) == 0
|
||||
assert fetched[0]["status"] == "ok" and fetched[1]["status"] == "ok"
|
||||
|
||||
|
||||
def test_deduplicate_fetched_content_error_preserved():
|
||||
"""error/skipped entries pass through untouched."""
|
||||
body = _long_body()
|
||||
fetched = [
|
||||
{"url": "https://ok.example", "status": "ok", "text": body,
|
||||
"text_length": len(body)},
|
||||
{"url": "https://err.example", "status": "error", "text": "",
|
||||
"text_length": 0},
|
||||
{"url": "https://skip.example", "status": "skipped", "text": "",
|
||||
"text_length": 0},
|
||||
]
|
||||
assert deduplicate_fetched_content(fetched) == 0
|
||||
assert fetched[1]["status"] == "error" and fetched[2]["status"] == "skipped"
|
||||
|
||||
|
||||
def test_deduplicate_fetched_content_tighter_threshold():
|
||||
"""Higher threshold keeps near-identical-but-not-exact bodies.
|
||||
|
||||
Uses two variants whose SimHash fingerprints differ by exactly 2 bits
|
||||
in the 1000-char window: 0.85 maps to hamming<=3 (duplicate), 0.95 maps
|
||||
to hamming<=1 (kept).
|
||||
"""
|
||||
body = _long_body()
|
||||
v_full = body.replace("machine learning", "deep learning") # dist 6
|
||||
v_light = body.replace("machine learning", "deep learning", 1) # dist 4
|
||||
a = {"url": "https://a.example", "status": "ok", "text": v_full,
|
||||
"text_length": len(v_full)}
|
||||
b = {"url": "https://b.example", "status": "ok", "text": v_light,
|
||||
"text_length": len(v_light)}
|
||||
# lenient (0.85) → duplicate; strict (0.95) → kept
|
||||
assert deduplicate_fetched_content([dict(a), dict(b)]) == 1
|
||||
assert deduplicate_fetched_content([dict(a), dict(b)], threshold=0.95) == 0
|
||||
|
||||
|
||||
# ===== --dry-run batch lists queries (v2.5.0) =====
|
||||
|
||||
def test_dry_run_batch_includes_queries(tmp_path, capsys):
|
||||
"""--dry-run --queries-file prints the actual query list (no HTTP)."""
|
||||
qf = tmp_path / "queries.txt"
|
||||
qf.write_text("alpha\n# comment\nbeta\n\n", encoding="utf-8")
|
||||
# query must be None so the batch branch is selected (preview priority:
|
||||
# query > research > queries_file)
|
||||
args = _make_args(dry_run=True, query=None, queries_file=str(qf),
|
||||
format="json")
|
||||
with pytest.raises(SystemExit) as exc_info:
|
||||
search_mod._dry_run_preview(args, ["https://x.example.com"], {})
|
||||
assert exc_info.value.code == 0
|
||||
out, _ = capsys.readouterr()
|
||||
data = json.loads(out)
|
||||
assert data["action"] == "batch"
|
||||
assert data["queries"] == ["alpha", "beta"]
|
||||
|
||||
|
||||
def test_dry_run_batch_missing_file_reports_error(tmp_path, capsys):
|
||||
"""Unreadable queries file degrades gracefully in dry-run."""
|
||||
args = _make_args(dry_run=True, query=None,
|
||||
queries_file=str(tmp_path / "missing.txt"),
|
||||
format="json")
|
||||
with pytest.raises(SystemExit):
|
||||
search_mod._dry_run_preview(args, ["https://x.example.com"], {})
|
||||
out, _ = capsys.readouterr()
|
||||
data = json.loads(out)
|
||||
assert data["queries_error"]
|
||||
|
||||
|
||||
# ===== search requests-backed HTTP path (v2.5.0) =====
|
||||
|
||||
def _enable_requests_path(monkeypatch):
|
||||
"""Temporarily enable the requests backend (conftest forces stdlib)."""
|
||||
monkeypatch.setattr(search_mod, "_HAS_REQUESTS", True)
|
||||
monkeypatch.setattr(search_mod, "_session", None)
|
||||
|
||||
|
||||
def test_search_json_requests_path_success(monkeypatch):
|
||||
"""requests backend parses a JSON response via session.get."""
|
||||
_enable_requests_path(monkeypatch)
|
||||
resp = MagicMock()
|
||||
resp.status_code = 200
|
||||
resp.content = json.dumps({"results": [{"title": "R", "url": "https://u"}]}
|
||||
).encode("utf-8")
|
||||
resp.headers = {"Content-Type": "application/json"}
|
||||
session = MagicMock()
|
||||
session.get.return_value = resp
|
||||
monkeypatch.setattr(search_mod, "_get_session", lambda: session)
|
||||
|
||||
r = search_mod.search_json("https://s.example.com",
|
||||
{"q": "t", "format": "json"})
|
||||
assert r["results"][0]["title"] == "R"
|
||||
session.get.assert_called_once()
|
||||
|
||||
|
||||
def test_search_json_requests_path_404_returns_none(monkeypatch):
|
||||
"""requests backend: 404 → None (fall back to HTML scraping)."""
|
||||
_enable_requests_path(monkeypatch)
|
||||
resp = MagicMock()
|
||||
resp.status_code = 404
|
||||
resp.content = b"not found"
|
||||
resp.headers = {}
|
||||
session = MagicMock()
|
||||
session.get.return_value = resp
|
||||
monkeypatch.setattr(search_mod, "_get_session", lambda: session)
|
||||
|
||||
assert search_mod.search_json("https://s.example.com",
|
||||
{"q": "t", "format": "json"}) is None
|
||||
|
||||
|
||||
def test_search_json_requests_path_403_raises(monkeypatch):
|
||||
"""requests backend: 403 raises (auth/ban — not silently masked)."""
|
||||
import requests as requests_mod
|
||||
_enable_requests_path(monkeypatch)
|
||||
resp = MagicMock()
|
||||
resp.status_code = 403
|
||||
resp.content = b"forbidden"
|
||||
resp.headers = {}
|
||||
resp.raise_for_status.side_effect = requests_mod.exceptions.HTTPError(
|
||||
response=resp)
|
||||
session = MagicMock()
|
||||
session.get.return_value = resp
|
||||
monkeypatch.setattr(search_mod, "_get_session", lambda: session)
|
||||
|
||||
with pytest.raises(requests_mod.exceptions.HTTPError):
|
||||
search_mod.search_json("https://s.example.com",
|
||||
{"q": "t", "format": "json"})
|
||||
|
||||
|
||||
def test_search_html_requests_path_decodes(monkeypatch):
|
||||
"""requests backend HTML fallback decodes + parses results."""
|
||||
_enable_requests_path(monkeypatch)
|
||||
html = ('<html><body><article class="result"><h3>'
|
||||
'<a href="https://x.com">Title</a></h3>'
|
||||
'<p class="content">snip</p></article></body></html>').encode("utf-8")
|
||||
resp = MagicMock()
|
||||
resp.status_code = 200
|
||||
resp.content = html
|
||||
resp.headers = {"Content-Type": "text/html; charset=utf-8"}
|
||||
session = MagicMock()
|
||||
session.get.return_value = resp
|
||||
monkeypatch.setattr(search_mod, "_get_session", lambda: session)
|
||||
|
||||
r = search_mod.search_html("https://s.example.com", {"q": "t"})
|
||||
assert r["results"][0]["title"] == "Title"
|
||||
|
||||
|
||||
def test_search_session_singleton_reused(monkeypatch):
|
||||
"""_get_session returns the same Session instance across calls."""
|
||||
_enable_requests_path(monkeypatch)
|
||||
s1 = search_mod._get_session()
|
||||
s2 = search_mod._get_session()
|
||||
assert s1 is s2
|
||||
assert isinstance(s1, MagicMock) is False # real Session on requests env
|
||||
search_mod._reset_session()
|
||||
s3 = search_mod._get_session()
|
||||
assert s3 is not s1 # reset creates a fresh one
|
||||
|
||||
|
||||
# ===== number_of_results fallback (v2.5.0 fix) =====
|
||||
|
||||
def test_search_json_fills_missing_number_of_results(monkeypatch):
|
||||
"""Instance JSON missing number_of_results → filled with len(results).
|
||||
|
||||
Some instances/versions omit the field (verified against a real
|
||||
instance: `number_of_results` absent, 20 results present). The HTML
|
||||
fallback path always fills it; JSON must too, or the output contract
|
||||
differs between paths.
|
||||
"""
|
||||
raw = json.dumps({"query": "t",
|
||||
"results": [{"title": "A", "url": "https://u"}]}).encode()
|
||||
resp = MagicMock()
|
||||
resp.read.return_value = raw
|
||||
resp.__enter__.return_value = resp
|
||||
with patch("urllib.request.urlopen", return_value=resp):
|
||||
r = search_mod.search_json("https://s.example.com",
|
||||
{"q": "t", "format": "json"})
|
||||
assert r["number_of_results"] == 1
|
||||
|
||||
|
||||
def test_search_json_preserves_existing_number_of_results(monkeypatch):
|
||||
"""Instance-provided number_of_results is never overwritten."""
|
||||
raw = json.dumps({"query": "t", "results": [{"title": "A"}],
|
||||
"number_of_results": 999}).encode()
|
||||
resp = MagicMock()
|
||||
resp.read.return_value = raw
|
||||
resp.__enter__.return_value = resp
|
||||
with patch("urllib.request.urlopen", return_value=resp):
|
||||
r = search_mod.search_json("https://s.example.com",
|
||||
{"q": "t", "format": "json"})
|
||||
assert r["number_of_results"] == 999
|
||||
|
||||
|
||||
def test_search_json_requests_path_fills_number_of_results(monkeypatch):
|
||||
"""requests backend: same fallback applies."""
|
||||
_enable_requests_path(monkeypatch)
|
||||
resp = MagicMock()
|
||||
resp.status_code = 200
|
||||
resp.content = json.dumps({"query": "t", "results": [{"title": "A"},
|
||||
{"title": "B"}]}
|
||||
).encode("utf-8")
|
||||
resp.headers = {"Content-Type": "application/json"}
|
||||
session = MagicMock()
|
||||
session.get.return_value = resp
|
||||
monkeypatch.setattr(search_mod, "_get_session", lambda: session)
|
||||
|
||||
r = search_mod.search_json("https://s.example.com",
|
||||
{"q": "t", "format": "json"})
|
||||
assert r["number_of_results"] == 2
|
||||
|
||||
|
||||
def test_search_json_non_dict_response_untouched(monkeypatch):
|
||||
"""List responses (rare) are passed through without dict access."""
|
||||
raw = json.dumps([{"x": 1}]).encode()
|
||||
resp = MagicMock()
|
||||
resp.read.return_value = raw
|
||||
resp.__enter__.return_value = resp
|
||||
with patch("urllib.request.urlopen", return_value=resp):
|
||||
r = search_mod.search_json("https://s.example.com",
|
||||
{"q": "t", "format": "json"})
|
||||
assert isinstance(r, list) and r[0]["x"] == 1
|
||||
|
||||
|
||||
# ===== release_check.py (v2.5.0) =====
|
||||
|
||||
def test_release_check_ok_on_consistent_state(tmp_path, monkeypatch):
|
||||
"""Matching versions + complete hints → no errors."""
|
||||
import release_check
|
||||
scripts = tmp_path / "scripts"
|
||||
scripts.mkdir()
|
||||
(scripts / "_config.py").write_text('VERSION = "1.2.3"\n', encoding="utf-8")
|
||||
(scripts / "common.py").write_text(
|
||||
'E_TEST = "E_TEST"\n\nRECOVERY_HINTS = {\n "E_TEST": "hint",\n}\n',
|
||||
encoding="utf-8")
|
||||
(tmp_path / "pyproject.toml").write_text('version = "1.2.3"\n',
|
||||
encoding="utf-8")
|
||||
monkeypatch.setattr(release_check, "ROOT", tmp_path)
|
||||
monkeypatch.setattr(release_check, "SCRIPTS_DIR", scripts)
|
||||
assert release_check.check_version(strict=False) == []
|
||||
assert release_check.check_error_codes(strict=False) == []
|
||||
|
||||
|
||||
def test_release_check_detects_version_drift(tmp_path, monkeypatch):
|
||||
"""pyproject.toml vs _config.VERSION mismatch is reported."""
|
||||
import release_check
|
||||
scripts = tmp_path / "scripts"
|
||||
scripts.mkdir()
|
||||
(scripts / "_config.py").write_text('VERSION = "2.0.0"\n', encoding="utf-8")
|
||||
(tmp_path / "pyproject.toml").write_text('version = "1.0.0"\n',
|
||||
encoding="utf-8")
|
||||
monkeypatch.setattr(release_check, "ROOT", tmp_path)
|
||||
monkeypatch.setattr(release_check, "SCRIPTS_DIR", scripts)
|
||||
errors = release_check.check_version(strict=False)
|
||||
assert any("pyproject.toml" in e and "1.0.0" in e for e in errors)
|
||||
|
||||
|
||||
def test_release_check_detects_missing_hint(tmp_path, monkeypatch):
|
||||
"""An E_* constant without a RECOVERY_HINTS entry is reported."""
|
||||
import release_check
|
||||
scripts = tmp_path / "scripts"
|
||||
scripts.mkdir()
|
||||
(scripts / "common.py").write_text('E_ORPHAN = "E_ORPHAN"\n',
|
||||
encoding="utf-8")
|
||||
monkeypatch.setattr(release_check, "ROOT", tmp_path)
|
||||
monkeypatch.setattr(release_check, "SCRIPTS_DIR", scripts)
|
||||
errors = release_check.check_error_codes(strict=False)
|
||||
assert any("E_ORPHAN" in e and "RECOVERY_HINTS" in e for e in errors)
|
||||
Reference in New Issue
Block a user