feat(v2.3.0): fetch 结构化 JSON 契约 + 并发批量 + 真实并发门控

迭代 1 — 正确性修复: - 修复 --pages N 多页聚合的 unresponsive-engine 警告误判: 原用循环末次 cached 变量判断, 缓存命中时警告被错误跳过/误触发; 改用独立 performed_live_query 标记 - UA 池单一来源: 删除 common.py 手工副本 _FALLBACK_UAS_BUILTIN, FALLBACK_UAS 直接引用 _config.UA_POOL, 消除双份漂移 - search_html 解码修复: 硬编码 utf-8 改为 detect_charset(header/meta 自动检测), 新增 --encoding 强制覆盖, 贯穿 search_multi 全链 - AdaptiveThrottle 真实并发门控: acquire_slot()/release_slot() 槽位机制, 退避降并发后新请求被快速拒绝(E_RATE_LIMIT), 实现持久降并发而非名义降并发迭代 2 — fetch JSON 契约 + 批量并发: - fetch.py --format json: 成功 {status,url,final_url,content_type,extract, truncated,text_length,user_agent}; 失败 {status,error,error_code, status_code,url}, 对齐 search.py 错误码体系 - fetch_page 采集 title + latency, 填充 --fetch-report json 空字段 - --queries-file --parallel-queries N (1-8): 并发批量, 输出保序, 受 AdaptiveThrottle 门控; 并发模式禁用 --fetch(嵌套并行不安全) - queries 文件编码自动检测 (UTF-8 → GBK 回退) 迭代 3 — 工程化: - 新增 pyproject.toml (searxng-search/searxng-fetch 入口点) - 收敛 20+ 处函数内冗余导入 - --dump-schema 扩展: fetched.items 补全 15 字段, 新增 defs.batch/research - 新增 17 个测试 (tests/test_v230_features.py), 全量 561 测试通过 - 文档同步 (SKILL.md/README.md, 版本号 2.3.0)
2026-08-05 20:14:07 +08:00
parent 6cedba9042
commit 4df521dc9d
8 changed files with 1006 additions and 226 deletions
@@ -206,6 +206,7 @@ python scripts/search.py -q "查询词" -i https://your-instance \
 | `--request-delay` | v2.0.0 抓取请求间隔秒数（自适应限流可能增大） | 0.3 |
 | `--cache-ttl` | 缓存分钟数 | 0（不缓存） |
 | `--queries-file` | 批量查询文件（每行一个查询） | — |
 | `--parallel-queries` | v2.3.0 并发批量查询 worker 数（1-8，输出保序，受自适应限流门控；并发时禁用 --fetch） | 0（串行） |
 | `--research` | v2.1.0 研究模式：给定主题自动扩展 5 个多角度查询 | — |
 | `--include-domain` | 域名白名单 | — |
 | `--exclude-domain` | 域名黑名单 | — |
@@ -236,6 +237,7 @@ python scripts/search.py -q "查询词" -i https://your-instance \
 ```bash
 python scripts/fetch.py -u https://example.com \
  --extract text|html|markdown \
  --format text|json \  # v2.3.0: json = 结构化 JSON 契约
  [--encoding gbk] \
  [--max-size 5242880] \
  [--timeout 15] \
@@ -249,6 +251,7 @@ python scripts/fetch.py -u https://example.com \
 | 参数 | 说明 | 默认值 |
 |------|------|--------|
 | `-u / --url` | 目标 URL（必填） | — |
 | `-f / --format` | v2.3.0 输出格式：text/json（json = 结构化 JSON 契约，成功 `{status,url,final_url,content_type,extract,truncated,text_length,user_agent}`，失败 `{status,error,error_code,status_code,url}`） | text |
 | `-e / --extract` | 提取模式：text/html/markdown | text |
 | `--encoding` | 强制字符编码 | 自动检测 |
 | `--max-size` | 最大字节数 | 不限 |
@@ -263,6 +266,17 @@ python scripts/fetch.py -u https://example.com \
 ## 能力清单
 **v2.3.0 新功能与修复**
 - `fetch.py --format json`：结构化 JSON 输出契约。成功 `{status, url, final_url, content_type, extract, truncated, text_length, user_agent}`；失败 `{status, error, error_code, status_code, url}`（对齐 search.py 错误码体系，含 `E_UNSUPPORTED_MEDIA`）。AI Agent 可程序化解析抓取结果，无需再解析裸文本
 - `--parallel-queries N`：批量模式并发执行（1-8 workers，输出保持文件顺序），受 AdaptiveThrottle 真实并发门控约束；并发模式下禁用 `--fetch`（嵌套并行抓取不安全）。queries 文件编码自动检测（UTF-8 → GBK 回退）
 - 真实并发门控：`AdaptiveThrottle.acquire_slot()/release_slot()`——退避降并发后新请求被快速拒绝（返回 `E_RATE_LIMIT`），实现持久降并发（原实现仅名义降并发，线程池规模固定）
 - `--fetch-report json` 字段补齐：`fetch_page` 采集 `title`（页面标题）与 `latency`（耗时秒数），此前恒为 None
 - `--dump-schema` 扩展：`fetched.items` 字段补全（final_url/status/error_code/waf_type/fallback_used/title/latency 等），新增 `defs.batch`/`defs.research` 描述批量与研究模式输出 shape
 - 修复多页聚合（`--pages N`）的 unresponsive-engine 警告误判：原实现用循环末次 `cached` 变量判断，缓存命中时警告被错误跳过/误触发；现用独立 `performed_live_query` 标记
 - UA 池单一来源：删除 common.py 的手工副本 `_FALLBACK_UAS_BUILTIN`，`FALLBACK_UAS` 直接引用 `_config.UA_POOL`（消除双份漂移风险）
 - search_html 解码修复：改用 `detect_charset`（header/meta 自动检测），新增 `--encoding` 强制覆盖（非 UTF-8 SearXNG 实例不再乱码）
 - 新增 `pyproject.toml`（可选打包，`searxng-search`/`searxng-fetch` 入口点）；函数内冗余导入收敛
 **v2.2.1 修复**
 - Brotli 乱码修复：`build_browser_headers()` 智能声明 `Accept-Encoding`——仅当本机安装了 brotli/brotlicffi 包时才声明 `br`，避免服务器返回 Brotli 压缩字节而 requests 无法自动解压导致全页乱码
 - `fetch.py` 双路径 br 解压：requests 路径和 stdlib urllib 路径都添加了 Brotli 手动解压逻辑（作为双保险，应对代理/CDN 强制返回 br 的边缘情况）
@@ -351,7 +365,7 @@ python scripts/fetch.py -u https://example.com \
 - JSON Lines 流式输出（`--stream`，含 `error` 事件类型）
 - 进度事件（`--progress`，含 `instance_try`/`instance_ok`/`instance_fail`，JSON Lines 到 stderr）
 - batch 模式统一 schema（`status` 字段区分成功/失败）
- 544 个单元+集成测试
+- 561 个单元+集成测试
 ## 跨 Agent 兼容性
@@ -379,7 +393,7 @@ pip install pytest
 pytest -q
 ```
-539 个测试覆盖：缓存操作、认证解析、域名过滤、Markdown 转换、搜索逻辑、集成流程、日志配置、HTML 回退、自动抓取、健康检查、输出格式化、实例解析、并行搜索、CLI 端到端、错误码分类、流式输出、进度事件、配置文件认证、schema_version、recovery_hint、batch 统一 schema、--dump-schema、UTF-8 stdout 强制、Windows APPDATA 路径、v2.0.0 浏览器指纹头、WAF 反爬检测（v2.0.1 收窄误判 + title 精准检测）、Wayback Machine 兜底（v2.0.1 修复 HTTP 200 反爬页触发）、自适应限流（v2.1.1 结构化 error_code 检测 429）、v2.1.0 Wayback 共享逻辑（should_try_wayback/build_wayback_url）、被墙站点智能回退（v2.1.1 精确化 baidu 子域，pan/cloud 不再误伤）、--research 研究模式（v2.1.1 跨角度合并去重 + 中英文双语后缀）、--time-range week 支持（v2.1.1）、stdlib gzip/deflate 解压（v2.1.1）、实例引擎挂起检测（v2.1.1 _warn_unresponsive_engines）。
+561 个测试覆盖：缓存操作、认证解析、域名过滤、Markdown 转换、搜索逻辑、集成流程、日志配置、HTML 回退、自动抓取、健康检查、输出格式化、实例解析、并行搜索、CLI 端到端、错误码分类、流式输出、进度事件、配置文件认证、schema_version、recovery_hint、batch 统一 schema、--dump-schema、UTF-8 stdout 强制、Windows APPDATA 路径、v2.0.0 浏览器指纹头、WAF 反爬检测（v2.0.1 收窄误判 + title 精准检测）、Wayback Machine 兜底（v2.0.1 修复 HTTP 200 反爬页触发）、自适应限流（v2.1.1 结构化 error_code 检测 429）、v2.1.0 Wayback 共享逻辑（should_try_wayback/build_wayback_url）、被墙站点智能回退（v2.1.1 精确化 baidu 子域，pan/cloud 不再误伤）、--research 研究模式（v2.1.1 跨角度合并去重 + 中英文双语后缀）、--time-range week 支持（v2.1.1）、stdlib gzip/deflate 解压（v2.1.1）、实例引擎挂起检测（v2.1.1 _warn_unresponsive_engines）、v2.3.0 新功能（多页实时查询跟踪、GBK charset 检测与 --encoding、UA 单源、并发槽位门控、fetch JSON 契约、fetch_page title/latency、GBK queries 文件、--parallel-queries 并发批量）。
 ## 项目结构
@@ -1,7 +1,7 @@
 ---
 name: searxng-use-cli
 description: Use when you need to search the web via your OWN SearXNG instance (no public-instance discovery). 3 CLI scripts + a shared common.py module — execute privacy-respecting searches against a user-supplied instance (with multi-instance failover, 5xx/429 retry, auto-fetch) or via SEARXNG_INSTANCE env / config file, fetch/extract readable text or markdown from web pages. Zero-config replacement for proprietary search APIs.
-version: 2.2.1
+version: 2.3.0
 author: Metona Team
 license: MIT
 platforms: [linux, macos, windows]
@@ -45,8 +45,8 @@ SearXNG is a privacy-respecting metasearch engine that aggregates results from 7
 - Similarity deduplication (`--similarity-dedup`; off by default) — title-based SimHash + Jaccard, auto-skipped when results > 500
 - Result sorting (`--sort-by {score,date,engine,none}`; default: score descending) — applied after dedup, before `--max-results`
 - Domain allowlist/blocklist (`--include-domain` / `--exclude-domain`) — case-insensitive, ignores leading `www.`, exclude wins on conflict
 - Batch mode (`--queries-file`) — run multiple queries from a file in sequence, combined output; v2.3.0 adds `--parallel-queries N` (1-8 concurrent, output order preserved, gated by AdaptiveThrottle)
 - Pagination aggregation (`--pages N`) — fetch N pages in one run, merge with cross-page dedup
 - Batch mode (`--queries-file`) — run multiple queries from a file in sequence, combined output
 **Output formats**
 - JSON (default, rich metadata), brief (title+URL+snippet), urls (plain list), CSV (spreadsheet-ready)
@@ -54,6 +54,7 @@ SearXNG is a privacy-respecting metasearch engine that aggregates results from 7
 - Structured JSON error output (in `--format json` mode) with `error_code` field for machine-readable failure reporting
 - JSON Lines streaming (`--stream`) — each result emitted as a separate JSON line to stdout, enabling incremental processing
 - Progress events (`--progress`) — structured JSON Lines events to stderr for real-time execution tracking, includes `request_id` for correlation
 - `fetch.py --format json` (v2.3.0) — structured JSON contract for page fetches: success `{status, url, final_url, content_type, extract, truncated, text_length, user_agent}`, failure `{status, error, error_code, status_code, url}` (aligns with search.py's error-code system)
 **Caching & config**
 - SQLite result caching (`--cache-ttl`) with size cap + LRU eviction (`--cache-max-size`, default 100MB) — identical queries within a TTL skip the network entirely; `--clear-cache` / `--cache-stats` manage it
@@ -76,14 +77,14 @@ SearXNG is a privacy-respecting metasearch engine that aggregates results from 7
 - **WAF fingerprint library** — `_detect_anti_bot()` (in `search.py`, used by `--fetch` flow) identifies Cloudflare / Imperva / PerimeterX / DataDome / Akamai / Anubis / generic challenges via `<title>` tag matching + full-document scan. Returns `waf_type` for AI-agent decisioning
 - **Wayback Machine fallback** — 404/403/timeout AND anti-bot-blocked pages automatically retry via `https://web.archive.org/web/2/<url>` (latest snapshot). Default ON; `--no-fallback` disables. Wayback timeout = `min(--timeout, 10s)`; Wayback retries = `min(--retries, 2)`
 - **Hard-blocked domain fallback** — `is_hard_blocked_domain()` (in `common.py`) detects known strong-anti-bot sites (www.baidu.com, baike.baidu.com, zhihu.com, weibo.com, mp.weixin.qq.com, douban.com, etc.) that almost always return 403. These trigger Wayback immediately on failure. List maintained in `HARD_BLOCKED_DOMAINS`. Note: baidu uses exact subdomain matching (www/baike/zhidao/tieba/wenku) to avoid over-blocking pan.baidu.com; zhihu/weibo/douban use subdomain wildcard matching
- **Adaptive throttling** — `AdaptiveThrottle` (in `search.py`) state machine: consecutive failures → double delay + halve concurrency; consecutive successes → gradual recovery; 429 → global pause. Thread-safe. Parameters configurable via `--throttle-failure-threshold` / `--throttle-pause-seconds` / `--throttle-max-delay`
+- **Adaptive throttling** — `AdaptiveThrottle` (in `search.py`) state machine: consecutive failures → double delay + halve concurrency; consecutive successes → gradual recovery; 429 → global pause. Thread-safe. Parameters configurable via `--throttle-failure-threshold` / `--throttle-pause-seconds` / `--throttle-max-delay`. v2.3.0: real concurrency gating via `acquire_slot()`/`release_slot()` — after backoff reduces concurrency, new requests are rejected until in-flight drops below the target (persistent throttling, not just nominal)
 - **readability-lite extraction** — when `<article>`/`<main>`/`role="main"`/content-class `<div>` are all missing and only `<body>` remains, picks the highest text-density `<div>`/`<section>`/`<article>` node (scored by text density + `<p>` count weighting). Text-density threshold is language-aware — CJK content uses 100 chars, other languages use 200 chars
 - **PDF/document parsing** — `fetch.py` parses PDF (via `pdftotext` subprocess) and `.docx`/`.xlsx` (via stdlib `zipfile`). Unsupported binary types return `E_UNSUPPORTED_MEDIA`
 - **`--fetch-report [json]`** — structured per-URL report to stderr after `--fetch N`: status, WAF type, fallback used, char count, plus throttle stats. `--fetch-report json` outputs a full JSON report (items array + summary)
 - **`--referer`** — set Referer header for fetch requests (defaults to the instance URL when fetching result pages)
 - **`--request-delay`** — configurable delay between fetch requests (default 0.3s; adaptive throttling may increase this on failures)
- **`search.py --fetch` output fields** (in `fetched` array): `anti_bot_detected` (bool), `waf_type` (str|null), `fallback_used` (str|null), `error_code` (str|null — structured error code on fetch failure)
+- **`search.py --fetch` output fields** (in `fetched` array): `anti_bot_detected` (bool), `waf_type` (str|null), `fallback_used` (str|null), `error_code` (str|null — structured error code on fetch failure); v2.3.0 also `title` (str|null) and `latency` (float|null seconds) — consumed by `--fetch-report json`
- **`fetch.py` FetchResult fields** (standalone script output): `content` (str), `content_type` (str), `final_url` (str), `truncated` (bool), `user_agent` (str), `error_code` (str|null), `error_message` (str|null)
+- **`fetch.py` FetchResult fields** (standalone script output): `content` (str), `content_type` (str), `final_url` (str), `truncated` (bool), `user_agent` (str), `error_code` (str|null), `error_message` (str|null); with `--format json` (v2.3.0) these map to a structured JSON contract
 **Research mode**
 - **`--research <topic>`** — given a research topic, auto-expands into 5 multi-angle queries (overview/profile/background/works/review) and runs them in sequence. Results include `research_topic` and `research_queries` metadata so AI agents can structure their final report by angle
@@ -432,7 +433,7 @@ usage: search.py [-h] [--query QUERY] [--instance URL]
                 [--auth-basic USER:PASS] [--auth-basic-file FILE]
                 [--proxy URL] [--include-domain DOMAINS]
                 [--exclude-domain DOMAINS] [--queries-file FILE]
-                 [--research TOPIC] [--research-angles ANGLES]
+                 [--parallel-queries N] [--research TOPIC] [--research-angles ANGLES]
                 [--cache-ttl MINUTES] [--cache-max-size MB] [--clear-cache]
                 [--cache-stats] [--sort-by {score,date,engine,none}] [--no-dedup]
                 [--similarity-dedup] [--similarity-threshold FLOAT]
@@ -453,7 +454,7 @@ usage: search.py [-h] [--query QUERY] [--instance URL]
 8. **Auto-fetch:** `--fetch 3` concurrently downloads top 3 result pages with retry, full browser fingerprint headers, 15-UA deterministic per-domain pool, Retry-After compliance, WAF fingerprint detection, Wayback Machine fallback for 404/403/timeout, adaptive throttling, and `fetch.py`'s readability-lite extractor. `--fetch-report` prints a structured per-URL report to stderr
 9. **Health-check mode:** `--verify` probes each instance (reachability / JSON-API support / latency / POST support / engine list / auth status) and prints a report, then exits without searching — use it to validate your instance list
 10. **Result caching:** `--cache-ttl 30` stores results for 30 min; identical queries within the TTL skip the network entirely. Cache lives at `$SEARXNG_CACHE_DIR` or `~/.cache/searxng-cli/cache.db` (SQLite, WAL mode, LRU eviction). `--clear-cache` / `--cache-stats` manage it without searching
-11. **Batch mode:** `--queries-file FILE` reads one query per line (blank/`#` lines skipped) and runs them in sequence; JSON output is `{"schema_version": "1.0", "queries": [{"query":..., "status": "ok"|"error", ...}]}` (or one block per query in brief/urls). A failed query is recorded but does not abort the batch. Exit codes: 0 if any query returned results, 1 if all errored, 2 if all empty
+11. **Batch mode:** `--queries-file FILE` reads one query per line (blank/`#` lines skipped) and runs them in sequence; JSON output is `{"schema_version": "1.0", "queries": [{"query":..., "status": "ok"|"error", ...}]}` (or one block per query in brief/urls). A failed query is recorded but does not abort the batch. v2.3.0: `--parallel-queries N` (1-8) runs queries concurrently with output order preserved; concurrent mode disables `--fetch` (nested parallel fetch is unsafe). Queries-file encoding is auto-detected (UTF-8, GBK fallback — v2.3.0). Exit codes: 0 if any query returned results, 1 if all errored, 2 if all empty
 12. **Dedup + sort + domain filter:** After search (and cache), duplicate URLs are collapsed (default; `--no-dedup` disables), results are sorted (`--sort-by`; default: score descending), and then `--include-domain`/`--exclude-domain` filter by domain. Matching is case-insensitive and ignores a leading `www.`; when a domain is in both lists, exclude wins
 13. **Proxy & auth:** `--proxy URL` routes both search and fetch through a proxy; `--auth-bearer` / `--auth-basic` (plus `*-file` variants, `searxng.toml` `auth_basic`/`auth_bearer` fields, and `SEARXNG_BEARER_TOKEN` / `SEARXNG_BASIC_AUTH` env vars) supply credentials. Priority: CLI flag > file > config file > env var
 14. **Config defaults:** `searxng.toml` may pre-set most flags; explicit CLI flags always win
@@ -520,6 +521,7 @@ usage: search.py [-h] [--query QUERY] [--instance URL]
 ```
 usage: fetch.py [-h] --url URL [--extract {text,html,markdown}]
                [--format {text,json}]
                [--timeout SEC] [--retries N] [--max-size BYTES]
                [--user-agent STR] [--encoding CHARSET]
                [--no-redirect] [--no-fallback] [--referer URL] [--proxy URL]
@@ -539,6 +541,7 @@ usage: fetch.py [-h] --url URL [--extract {text,html,markdown}]
 **Key options:**
 - `--url https://...` — required
 - `--format {text,json}` — output format (default: `text`; `json` = structured contract, v2.3.0). Success: `{status: ok, url, final_url, content_type, extract, truncated, text_length, user_agent}`; failure: `{status: error, url, error, error_code, status_code}`. stdout is pure JSON; logs stay on stderr; exit 0 on success / 1 on failure
 - `--extract text` — clean readable text (default); PDF/docx/xlsx parsed automatically regardless of extract mode (see "What it does" #7)
 - `--extract html` — raw HTML
 - `--extract markdown` — Markdown conversion (tree-based; handles nested tags, GFM tables, fenced code blocks, blockquotes, inline code, ordered/unordered/nested lists, definition lists, images, emphasis)
@@ -565,4 +568,4 @@ usage: fetch.py [-h] --url URL [--extract {text,html,markdown}]
 4. Collapse whitespace, output clean UTF-8
 5. Warn if extracted text < 500 chars (likely JS-heavy or bot-blocked)
-**Completion criterion:** Outputs page content. Non-zero exit on HTTP failure. Stderr carries warnings for low-confidence extraction.
+**Completion criterion:** Outputs page content (or the structured JSON object with `--format json`). Non-zero exit on HTTP failure. Stderr carries warnings for low-confidence extraction.
@@ -0,0 +1,42 @@
 [build-system]
 requires = ["setuptools>=61.0"]
 build-backend = "setuptools.build_meta"
 [project]
 name = "searxng-use-cli"
 version = "2.3.0"
 description = "Privacy-respecting SearXNG search + web-fetch CLI toolkit for AI agents"
 readme = "README.md"
 license = { text = "MIT" }
 requires-python = ">=3.8"
 keywords = ["searxng", "search", "web-scraping", "ai-agent", "cli", "privacy"]
 classifiers = [
    "Environment :: Console",
    "Intended Audience :: Developers",
    "License :: OSI Approved :: MIT License",
    "Operating System :: OS Independent",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.8",
    "Programming Language :: Python :: 3.9",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "Topic :: Internet :: WWW/HTTP :: Indexing/Search",
    "Topic :: Software Development :: Libraries :: Python Modules",
 ]
 dependencies = []
 [project.optional-dependencies]
 fetch = ["requests", "beautifulsoup4"]
 toml = ["tomli; python_version < '3.11'"]
 all = ["requests", "beautifulsoup4", "tomli; python_version < '3.11'"]
 test = ["pytest", "requests", "beautifulsoup4"]
 [project.scripts]
 searxng-search = "scripts.search:main"
 searxng-fetch = "scripts.fetch:main"
 [tool.setuptools]
 packages = ["scripts"]
 [tool.setuptools.package-data]
 scripts = ["*.py"]
@@ -7,7 +7,7 @@ Retry settings and shared HTTP utilities now live in ``common.py`` so that
 both ``search.py`` and ``fetch.py`` share one consistent implementation.
 """
-VERSION = "2.2.1"
+VERSION = "2.3.0"
 SCHEMA_VERSION = "1.0"
 USER_AGENT = f"searxng-cli/{VERSION}"
@@ -18,13 +18,20 @@ as retryable and connection errors as transient, eliminating the previous
 inconsistency where ``search.py`` ignored 5xx.
 """
 import base64
 import datetime
 import hashlib
 import io
 import json
 import logging
 import os
 import random
 import re
 import sys
 import threading
 import urllib.error
 import urllib.parse
 from pathlib import Path
 # Root logger for the searxng-cli package. All modules create child loggers
 # via ``logging.getLogger("searxng.<module>")`` so a single setup_logging()
@@ -105,7 +112,6 @@ class _JsonFormatter(logging.Formatter):
    """
    def format(self, record):
        import json as _json
        entry = {
            "ts": _datetime_iso(record),
            "level": record.levelname,
@@ -117,13 +123,12 @@ class _JsonFormatter(logging.Formatter):
            entry["request_id"] = rid
        if record.exc_info and record.exc_info[1]:
            entry["exception"] = type(record.exc_info[1]).__name__
-        return _json.dumps(entry, ensure_ascii=False)
+        return json.dumps(entry, ensure_ascii=False)
 def _datetime_iso(record):
    """格式化日志时间戳为 ISO 8601 字符串。"""
-    import datetime as _dt
+    return datetime.datetime.fromtimestamp(record.created).isoformat(timespec="milliseconds")
    return _dt.datetime.fromtimestamp(record.created).isoformat(timespec="milliseconds")
 def force_utf8_stdout() -> None:
@@ -191,57 +196,17 @@ RETRYABLE_STATUS = frozenset({403, 429, 502, 503, 504})
 # Browser-like UA strings for fallback when the searxng-cli UA is blocked.
 #
-# v2.2.0：UA 池迁移至 _config.py 的 UA_POOL（SSOT），此处通过导入引用。
+# v2.2.2：单一来源（SSOT）。UA 池唯一权威定义在 _config.UA_POOL，
-# 若 _config.py 不可导入（如 common.py 被单独分发），回退到下方内置副本
+# 此处直接导入——删除原有的内置副本 _FALLBACK_UAS_BUILTIN。
-# _FALLBACK_UAS_BUILTIN，保证模块始终可用。两份列表需保持同步，
+# v2.2.0 曾保留一份手工同步副本，两份列表漂移会导致跨脚本 UA 行为不一致
-# _config.UA_POOL 为唯一权威来源。
+# （fetch.py 用 FALLBACK_UAS 轮换、search.py 的 --dry-run 报告引用同一池），
 # 且注释要求"保持同步"无任何机制保证。直接引用同一对象后，改一处即全局生效。
 #
-# v2.0.0 起覆盖 Chrome/Edge/Firefox × Windows/macOS/Linux；v2.2.0 升级到
+# 维护原则（见 _config.UA_POOL 注释）：
-# 2026 年版本（Chrome 138-140 / Edge 138 / Firefox 140 / Safari 18）。
+#   1. 版本号保持为当前年份的主流浏览器版本
-# 顺序固定以便 get_ua_for_domain() 的 hash 选择可复现。
+#   2. 顺序固定——get_ua_for_domain() 用 SHA-256 哈希选索引
-_FALLBACK_UAS_BUILTIN = [
+#   3. 至少覆盖 Chrome/Edge/Firefox × Windows/macOS/Linux
    # Chrome 140 — Windows / macOS / Linux
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36",
    # Chrome 139 — Windows / macOS / Linux
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36",
    # Chrome 138 — Windows / macOS / Linux
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
    # Edge 138 — Windows / macOS
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 Edg/138.0.0.0",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 Edg/138.0.0.0",
    # Firefox 140 — Windows / macOS / Linux
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:140.0) "
    "Gecko/20100101 Firefox/140.0",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:140.0) "
    "Gecko/20100101 Firefox/140.0",
    "Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0",
    # Safari 18 — macOS（WebKit 指纹，应对 Chromium 针对性拦截）
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 "
    "(KHTML, like Gecko) Version/18.0 Safari/605.1.15",
 ]
 # 从 _config.py 导入权威 UA_POOL；导入失败时回退到内置副本，保证向后兼容。
 try:
 from _config import UA_POOL as FALLBACK_UAS
 except ImportError:
    FALLBACK_UAS = _FALLBACK_UAS_BUILTIN
 def _ua_index_for_domain(domain: str, pool_size: int) -> int:
@@ -251,7 +216,6 @@ def _ua_index_for_domain(domain: str, pool_size: int) -> int:
    随机化（PYTHONHASHSEED），跨进程不可复现。SHA-256 保证同一域名
    永远映射到同一索引，跨进程一致——这对调试和日志分析至关重要。
    """
    import hashlib
    h = hashlib.sha256(domain.encode("utf-8")).digest()
    # 取前 8 字节作为无符号整数，避免负数和短字符串的分布不均
    return int.from_bytes(h[:8], "big") % pool_size
@@ -288,9 +252,8 @@ def get_ua_for_domain(url: str, user_agent: str = None) -> str:
    if user_agent:
        return user_agent
    import urllib.parse as _up
    try:
-        domain = _up.urlparse(url).netloc.lower()
+        domain = urllib.parse.urlparse(url).netloc.lower()
        if not domain:
            return FALLBACK_UAS[0]
    except Exception:
@@ -367,7 +330,6 @@ def build_browser_headers(user_agent: str, referer: str = None,
    # Sec-Ch-Ua 系列仅 Chrome/Edge 发送，Firefox 不发
    if not is_firefox:
        # 从 UA 提取主版本号，构造 Sec-Ch-Ua
        import re
        m = re.search(r"Chrome/(\d+)", user_agent)
        ver = m.group(1) if m else "131"
        not_a_brand = '"Not_A Brand";v="99"' if ver != "99" else '"Not/A)Brand";v="99"'
@@ -445,7 +407,6 @@ def compute_backoff_delay(attempt: int, base: float = RETRY_BACKOFF_BASE,
    ``base * 2^attempt + jitter``，但不超过 ``cap``。
    v2.0.0 新增封顶：原公式无上限，N=10 时达 1536s 会卡死进程。
    """
    import random
    delay = base * (2 ** attempt) + random.uniform(0, 1)
    return min(delay, cap)
@@ -459,8 +420,6 @@ def build_auth_headers(bearer_token: str = None, basic_auth: str = None) -> dict
    If both are provided, Bearer takes precedence (more common for APIs).
    Returns a dict to merge into request headers, or an empty dict.
    """
    import base64
    headers = {}
    if bearer_token:
        headers["Authorization"] = f"Bearer {bearer_token}"
@@ -476,7 +435,6 @@ def _warn_file_perms(path: str) -> None:
    On Windows the Unix permission bits in ``st_mode`` do not reflect the
    actual ACL, so the check is skipped to avoid false alarms.
    """
    import os
    if os.name != "posix":
        return
    log = logging.getLogger("searxng.common")
@@ -513,7 +471,6 @@ def resolve_auth_basic(cli_value: str = None, file_path: str = None,
    if file_path:
        try:
            from pathlib import Path
            text = Path(file_path).read_text(encoding="utf-8")
            _warn_file_perms(file_path)
            for line in text.splitlines():
@@ -527,7 +484,6 @@ def resolve_auth_basic(cli_value: str = None, file_path: str = None,
    if config_value:
        return config_value
    import os
    return os.environ.get(env_var)
@@ -550,7 +506,6 @@ def resolve_auth_bearer(cli_value: str = None, file_path: str = None,
    if file_path:
        try:
            from pathlib import Path
            text = Path(file_path).read_text(encoding="utf-8")
            _warn_file_perms(file_path)
            for line in text.splitlines():
@@ -564,7 +519,6 @@ def resolve_auth_bearer(cli_value: str = None, file_path: str = None,
    if config_value:
        return config_value
    import os
    return os.environ.get(env_var)
@@ -583,7 +537,6 @@ def apply_proxy(proxy_url: str) -> None:
    Pass an empty string to clear the proxy env vars (rarely needed; the
    default unset state already means "no proxy").
    """
    import os
    if not proxy_url:
        return
    os.environ["HTTP_PROXY"] = proxy_url
@@ -597,8 +550,6 @@ def detect_charset(raw: bytes, content_type: str) -> str:
    Falls back to UTF-8 (with replacement) if nothing reliable is found.
    """
    import re
    # 1. HTTP header
    if "charset=" in content_type:
        charset = content_type.split("charset=")[-1].split(";")[0].strip()
@@ -704,7 +655,7 @@ RECOVERY_HINTS = {
 }
-def _classify_by_type_and_status(exc, _json):
+def _classify_by_type_and_status(exc) -> str:
    """根据异常类型和 HTTP 状态码分类，返回错误码或 None。
    检查顺序：HTTP 状态码 → 文件错误 → 连接错误 → 解析错误。
@@ -743,7 +694,7 @@ def _classify_by_type_and_status(exc, _json):
        return E_NETWORK
    # 解析错误
-    if isinstance(exc, (ValueError, _json.JSONDecodeError)):
+    if isinstance(exc, (ValueError, json.JSONDecodeError)):
        return E_PARSE
    return None
@@ -764,8 +715,6 @@ def classify_error(exc: BaseException) -> str:
         兼容旧路径——search_multi 把 last_error 拼进消息）
      7. 其他 → E_INTERNAL
    """
    import json as _json
    # 优先检查异常链 __cause__：raise X from Y 时，Y 指向真实底层异常。
    # 旧逻辑只能从外层 RuntimeError 的消息字符串推断（脆弱，依赖 "403"/"auth"
    # 等关键字匹配），新逻辑直接从 __cause__ 的 .code/.status_code 或异常
@@ -773,12 +722,12 @@ def classify_error(exc: BaseException) -> str:
    # raise-from 模式，深层链罕见且递归有循环风险。
    cause = getattr(exc, "__cause__", None)
    if cause is not None and cause is not exc:
-        code = _classify_by_type_and_status(cause, _json)
+        code = _classify_by_type_and_status(cause)
        if code is not None:
            return code
    # 检查 exc 本身的类型和状态码
-    code = _classify_by_type_and_status(exc, _json)
+    code = _classify_by_type_and_status(exc)
    if code is not None:
        return code
@@ -795,8 +744,7 @@ def classify_error(exc: BaseException) -> str:
            return E_RATE_LIMIT
        # 从 "http error NNN" / "http NNN" 模式中提取状态码，
        # 正确分类 "All instances failed. Last error: HTTP Error 403" 等
-        import re as _re
+        status_match = re.search(r'http(?: error)? (\d{3})', msg)
        status_match = _re.search(r'http(?: error)? (\d{3})', msg)
        if status_match:
            status = int(status_match.group(1))
            if status == 429:
@@ -855,13 +803,12 @@ def emit_progress(event: str, **kwargs) -> None:
    """
    if not _progress_enabled:
        return
    import json as _json
    payload = {"event": event}
    rid = getattr(_LOG, "_request_id", None)
    if rid:
        payload["request_id"] = rid
    payload.update(kwargs)
-    print(_json.dumps(payload, ensure_ascii=False), file=sys.stderr, flush=True)
+    print(json.dumps(payload, ensure_ascii=False), file=sys.stderr, flush=True)
 # ----- Wayback Machine 兜底（v2.1.0 共享逻辑）-----
@@ -945,8 +892,7 @@ def is_hard_blocked_domain(url: str) -> bool:
        return False
    # 提取域名
    try:
-        from urllib.parse import urlparse
+        host = urllib.parse.urlparse(url).hostname or ""
        host = urlparse(url).hostname or ""
    except Exception:
        host = ""
    if not host:
@@ -1051,9 +997,8 @@ def is_similar(result_a: dict, result_b: dict, threshold: float = 0.85) -> bool:
    title_b = _normalize_title(result_b.get("title", ""))
    # 标题太短时 SimHash 不稳定，改用 Jaccard
    if len(title_a) < 5 or len(title_b) < 5:
-        import urllib.parse as _up
+        domain_a = urllib.parse.urlparse(result_a.get("url", "")).netloc.lower()
-        domain_a = _up.urlparse(result_a.get("url", "")).netloc.lower()
+        domain_b = urllib.parse.urlparse(result_b.get("url", "")).netloc.lower()
        domain_b = _up.urlparse(result_b.get("url", "")).netloc.lower()
        set_a = set(title_a.split()) | {domain_a}
        set_b = set(title_b.split()) | {domain_b}
        return _jaccard_similarity(set_a, set_b) >= threshold
@@ -11,6 +11,7 @@ for improved extraction quality (optional, falls back to stdlib).
 import argparse
 import gzip
 import io
 import json
 import logging
 import random
 import re
@@ -53,6 +54,7 @@ from common import (
    build_auth_headers,
    build_browser_headers,
    build_wayback_url,
    classify_error,
    compute_backoff_delay,
    detect_charset,
    force_utf8_stdout,
@@ -1134,6 +1136,55 @@ def fetch_url(url: str, timeout=15, user_agent: str = None,
 # ----- Main -----
 def _emit_fetch_result(args, output: str, url: str, final_url: str,
                       content_type: str, truncated: bool,
                       user_agent: str = None,
                       error: str = None, error_code: str = None,
                       status_code: int = None) -> None:
    """输出抓取结果到 stdout / --output 文件。
    v2.3.0: ``--format json`` 提供结构化 JSON 契约，AI Agent 可程序化
    解析（成功与失败统一为 {status, url, ...}）。``--format text``（默认）
    保持 v2.2.x 行为：成功输出正文，失败输出空 + stderr 日志。
    成功 shape::
        {"status": "ok", "url", "final_url", "content_type",
         "extract", "truncated", "text_length", "user_agent"}
    失败 shape::
        {"status": "error", "url", "error", "error_code", "status_code"}
    """
    if args.format == "json":
        if error:
            payload = {"status": "error", "url": url, "error": error}
            if error_code:
                payload["error_code"] = error_code
            if status_code is not None:
                payload["status_code"] = status_code
        else:
            payload = {
                "status": "ok",
                "url": url,
                "final_url": final_url,
                "content_type": content_type,
                "extract": args.extract,
                "truncated": truncated,
                "text_length": len(output),
                "user_agent": user_agent,
            }
        text = json.dumps(payload, indent=2, ensure_ascii=False)
    else:
        text = output
    if args.output:
        with open(args.output, "w", encoding="utf-8") as f:
            f.write(text)
        logger.info(f"Saved {len(text)} chars to {args.output}")
    else:
        print(text)
 def main():
    parser = argparse.ArgumentParser(
        description="Fetch a web page and extract readable content",
@@ -1146,11 +1197,18 @@ Examples:
  %(prog)s -u https://example.com -e markdown      markdown conversion
  %(prog)s -u https://example.com -o page.txt      save to file
  %(prog)s -u https://example.cn -e text --encoding gbk   force charset
  %(prog)s -u https://example.com --format json    structured JSON output
        """,
    )
    parser.add_argument("--url", "-u", required=True, help="URL to fetch")
    parser.add_argument("--extract", "-e", choices=["text", "html", "markdown"],
                        default="text", help="Extraction mode (default: text)")
    parser.add_argument("--format", "-f", choices=["text", "json"], default="text",
                        help="Output format (default: text). 'json' emits a structured "
                             "JSON object {status, url, final_url, content_type, extract, "
                             "truncated, text_length, user_agent} on success, or "
                             "{status: error, error, error_code, status_code, url} on "
                             "failure — machine-readable for agents. v2.3.0.")
    parser.add_argument("--timeout", "-t", type=int, default=15,
                        help="Request timeout in seconds (default: 15)")
    parser.add_argument("--retries", type=int, default=3,
@@ -1229,6 +1287,18 @@ Examples:
        logger.info(f"Hard-blocked domain detected — Wayback fallback "
                    f"will be prioritized if main fetch fails")
    # v2.3.0: 状态收集变量。所有失败路径设置 fatal_* 后落到统一输出
    # （_emit_fetch_result），json 模式输出结构化错误到 stdout，text 模式
    # 保持 v2.2.x 行为（stdout 空 + stderr 日志 + exit 1）。
    content = None
    final_url = args.url
    content_type = ""
    truncated = False
    user_agent = None
    fatal_error = None
    fatal_error_code = None
    fatal_status_code = None
    try:
        result = fetch_url(
            args.url, timeout=args.timeout, user_agent=args.user_agent,
@@ -1237,17 +1307,17 @@ Examples:
            allow_redirects=not args.no_redirect,
            referer=args.referer,
        )
-        content, content_type, final_url = (
+        content = result.content
-            result.content, result.content_type, result.final_url,
+        content_type = result.content_type or ""
-        )
+        final_url = result.final_url
        truncated = result.truncated
        user_agent = result.user_agent
        # 文档解析失败（PDF/DOCX/XLSX 等）时 fetch_url 不抛异常，
        # 而是返回带 error_code 的 FetchResult——必须显式检查，
        # 否则失败会被静默吞掉（空输出 + exit 0）。
        if result.error_code:
-            logger.error(
+            fatal_error = result.error_message or result.error_code
-                f"Error: {result.error_message or result.error_code} "
+            fatal_error_code = result.error_code
                f"(error_code={result.error_code}, url={args.url})")
            sys.exit(1)
    except Exception as e:
        # 诊断信息增强：从 __cause__ 链中提取 HTTP 状态码、原始异常类型，
        # 让 AI Agent 能程序化判断失败原因（404 vs 403 vs DNS 失败等），
@@ -1275,7 +1345,10 @@ Examples:
        # v2.1.0: Wayback Machine 兜底
        # 触发条件：兜底启用 + (错误可恢复 OR 命中被墙站点)
-        error_msg = str(e) if str(e) else e.__class__.__name__
+        fatal_error = str(e) if str(e) else e.__class__.__name__
        fatal_error_code = classify_error(e)
        fatal_status_code = status_code
        error_msg = fatal_error
        if fallback_enabled and (should_try_wayback(error_msg) or hard_blocked):
            wayback_url = build_wayback_url(args.url)
            wb_timeout = min(args.timeout, 10)  # Wayback 独立超时，不阻塞
@@ -1289,16 +1362,30 @@ Examples:
                    max_size=args.max_size,
                    allow_redirects=True,
                )
-                content, content_type, final_url = (
+                content = wb_result.content
-                    wb_result.content, wb_result.content_type,
+                content_type = wb_result.content_type or ""
-                    wb_result.final_url,
+                final_url = wb_result.final_url
-                )
+                truncated = wb_result.truncated
                user_agent = wb_result.user_agent
                if wb_result.error_code:
                    fatal_error = (wb_result.error_message or wb_result.error_code)
                    fatal_error_code = wb_result.error_code
                    fatal_status_code = None
                else:
                    fatal_error = None
                    fatal_error_code = None
                    fatal_status_code = None
                logger.info(f"[FALLBACK] Wayback recovery successful "
                            f"({len(content)} chars)")
            except Exception as wb_e:
                logger.error(f"[FALLBACK] Wayback also failed: {wb_e}")
-                sys.exit(1)
+                fatal_error = f"{fatal_error} ; Wayback also failed: {wb_e}"
-        else:
+
    if fatal_error:
        _emit_fetch_result(args, "", args.url, final_url, content_type,
                           truncated, user_agent, error=fatal_error,
                           error_code=fatal_error_code,
                           status_code=fatal_status_code)
        sys.exit(1)
    if final_url != args.url:
@@ -1320,12 +1407,8 @@ Examples:
        logger.warning(f"Warning: extracted text is very short ({len(output.strip())} chars). "
                       "The page may be JS-heavy or use anti-bot protection.")
-    if args.output:
+    _emit_fetch_result(args, output, args.url, final_url, content_type,
-        with open(args.output, "w", encoding="utf-8") as f:
+                       truncated, user_agent)
            f.write(output)
        logger.info(f"Saved {len(output)} chars to {args.output}")
    else:
        print(output)
 if __name__ == "__main__":
@@ -9,6 +9,8 @@ Instance URLs are REQUIRED (see --instance / SEARXNG_INSTANCE / config file).
 """
 import argparse
 import csv
 import io
 import json
 import logging
 import os
@@ -37,6 +39,7 @@ from common import (
    build_wayback_url,
    classify_error,
    compute_backoff_delay,
    detect_charset,
    emit_progress,
    force_utf8_stdout,
    is_hard_blocked_domain,
@@ -549,8 +552,13 @@ def search_json(instance: str, params: dict, method: str = "GET",
 def search_html(instance: str, params: dict, timeout: int = 15,
-                auth_headers: dict = None) -> dict:
+                auth_headers: dict = None, encoding: str = None) -> dict:
-    """Execute search via HTML scraping fallback."""
+    """Execute search via HTML scraping fallback.
    v2.2.2：解码改用 common.detect_charset（此前硬编码 utf-8，GBK/Shift-JIS
    等非 UTF-8 实例的页面会整体乱码）。``encoding`` 为显式覆盖（来自
    ``--language`` 无关的 CLI ``--encoding``），优先级最高。
    """
    html_params = {k: v for k, v in params.items() if k != "format"}
    query_string = urllib.parse.urlencode(html_params)
    url = f"{instance}/search?{query_string}"
@@ -559,26 +567,41 @@ def search_html(instance: str, params: dict, timeout: int = 15,
    req = urllib.request.Request(url, headers=headers)
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
-            html = resp.read().decode("utf-8")
+            raw = resp.read()
            content_type = resp.headers.get("Content-Type", "")
            if encoding:
                try:
                    html = raw.decode(encoding)
                except (UnicodeDecodeError, LookupError):
                    html = raw.decode("utf-8", errors="replace")
            else:
                charset = detect_charset(raw, content_type)
                try:
                    html = raw.decode(charset)
                except (UnicodeDecodeError, LookupError):
                    html = raw.decode("utf-8", errors="replace")
            return parse_html_results(html, query=params.get("q", ""))
    except Exception as e:
        raise RuntimeError(f"HTML search failed for {instance}: {e}")
 def search_single(instance: str, params: dict, method: str = "GET",
-                  timeout: int = 15, auth_headers: dict = None) -> dict:
+                  timeout: int = 15, auth_headers: dict = None,
                  encoding: str = None) -> dict:
    """Execute one search attempt, preferring JSON with HTML fallback."""
    result = search_json(instance, params, method=method, timeout=timeout,
                         auth_headers=auth_headers)
    if result is not None:
        return result
    logger.warning(f"Warning: {instance} does not support format=json, falling back to HTML parsing")
-    return search_html(instance, params, timeout=timeout, auth_headers=auth_headers)
+    return search_html(instance, params, timeout=timeout, auth_headers=auth_headers,
                       encoding=encoding)
 def search_multi(instance_urls: list, params: dict, method: str = "GET",
                 timeout: int = 15, retry_per: int = None,
-                 auth_headers: dict = None, parallel: bool = True) -> dict:
+                 auth_headers: dict = None, parallel: bool = True,
                 encoding: str = None) -> dict:
    """Search across multiple instances, failing over on error.
    With parallel=True (default, multi-instance only): every instance is
@@ -589,6 +612,9 @@ def search_multi(instance_urls: list, params: dict, method: str = "GET",
    With parallel=False (or a single instance): strictly sequential, one
    request at a time, trying the next instance only after the current fails.
    ``encoding`` (v2.2.2) is forwarded to the HTML-fallback path for
    non-UTF-8 instances; JSON responses are always UTF-8.
    """
    if retry_per is None:
        retry_per = MAX_RETRIES
@@ -604,7 +630,8 @@ def search_multi(instance_urls: list, params: dict, method: str = "GET",
            try:
                def _do():
                    return search_single(instance, params, method=method,
-                                         timeout=timeout, auth_headers=auth_headers)
+                                         timeout=timeout, auth_headers=auth_headers,
                                         encoding=encoding)
                result = _retry_with_backoff(_do, max_retries=retry_per)
                emit_progress("instance_ok", url=instance,
                              latency=round(time.time() - start, 3),
@@ -627,7 +654,8 @@ def search_multi(instance_urls: list, params: dict, method: str = "GET",
        start = time.time()
        def _do():
            return search_single(instance, params, method=method,
-                                 timeout=timeout, auth_headers=auth_headers)
+                                 timeout=timeout, auth_headers=auth_headers,
                                 encoding=encoding)
        try:
            result = _retry_with_backoff(_do, max_retries=retry_per)
            emit_progress("instance_ok", url=instance,
@@ -864,6 +892,8 @@ def fetch_page(url: str, timeout: int = 10, auth_headers: dict = None,
      * ``fallback_used`` (str|None): 兜底方式（"wayback"），仅当走兜底时有值
    """
    # 主抓取
    # v2.3.0: 计时整条链路（主抓取 + Wayback 兜底），填充 latency 字段。
    _start_time = time.monotonic()
    result = None
    error_msg = None
    error_code = None
@@ -946,6 +976,9 @@ def fetch_page(url: str, timeout: int = 10, auth_headers: dict = None,
            "text": "", "text_length": 0, "truncated": False,
            "anti_bot_detected": False, "waf_type": None,
            "fallback_used": None,
            # v2.3.0: title/latency 字段（--fetch-report json 消费）
            "title": None,
            "latency": round(time.monotonic() - _start_time, 3),
        }
    # 反爬仍被检测到（Wayback 也无能为力或兜底被禁用）
@@ -957,6 +990,8 @@ def fetch_page(url: str, timeout: int = 10, auth_headers: dict = None,
            "text": "", "text_length": 0, "truncated": False,
            "anti_bot_detected": True, "waf_type": waf_type,
            "fallback_used": fallback_used,
            "title": _extract_title(content) if is_html else None,
            "latency": round(time.monotonic() - _start_time, 3),
        }
    text = extract_text(content) if is_html else content
@@ -973,6 +1008,8 @@ def fetch_page(url: str, timeout: int = 10, auth_headers: dict = None,
        "anti_bot_detected": False,
        "waf_type": None,
        "fallback_used": fallback_used,
        "title": _extract_title(content) if is_html else None,
        "latency": round(time.monotonic() - _start_time, 3),
    }
@@ -1105,6 +1142,19 @@ _TITLE_ANTI_BOT_SIGNATURES = {
 _TITLE_RE = re.compile(r"<title[^>]*>(.*?)</title>", re.IGNORECASE | re.DOTALL)
 def _extract_title(content: str) -> str:
    """从 HTML 内容提取 <title> 文本（v2.3.0，供 --fetch-report json 使用）。
    截断到 200 字符，防止异常页面标题撑爆报告。非 HTML 内容返回空串。
    """
    if not content:
        return ""
    m = _TITLE_RE.search(content)
    if m:
        return m.group(1).strip()[:200]
    return ""
 def _detect_anti_bot(content: str) -> str:
    """检测反爬页面，返回 WAF 类型或 None。
@@ -1173,6 +1223,11 @@ class AdaptiveThrottle:
        self._failure_threshold = failure_threshold
        self._pause_seconds = pause_seconds
        self._max_delay = max_delay
        # v2.2.2：真实并发门控。计数信号量约束"瞬时在飞请求峰值"，
        # _in_flight 结合当前 concurrency 判断是否应放行新请求——退避降
        # 并发后，新请求会被快速拒绝（限流语义），而不是名义降并发。
        self._semaphore = threading.BoundedSemaphore(initial_concurrency)
        self._in_flight = 0
    @property
    def delay(self) -> float:
@@ -1228,6 +1283,36 @@ class AdaptiveThrottle:
            logger.info(f"  [THROTTLE] Global pause: waiting {remaining:.1f}s (429)")
            time.sleep(remaining)
    def acquire_slot(self, timeout: float = 0.05) -> bool:
        """获取一个并发执行槽位（v2.2.2 真实并发门控）。
        线程池规模（ThreadPoolExecutor max_workers）创建时一次性固定，
        无法随退避动态缩小。槽位机制在"请求真正发出前"做门控：
        信号量约束瞬时峰值不超过初始并发；``_in_flight`` 结合当前
        ``concurrency`` 判断——退避降并发后，即使信号量有空位，只要在飞
        请求数已 >= 当前并发目标，新请求也会被快速拒绝（返回 False），
        由调用方跳过本次抓取，实现持久降并发。
        返回 True 表示拿到槽位，调用方必须在 finally 中 release_slot()。
        """
        if not self._semaphore.acquire(timeout=timeout):
            return False
        with self._lock:
            if self._in_flight >= self._concurrency:
                self._semaphore.release()
                return False
            self._in_flight += 1
            return True
    def release_slot(self) -> None:
        """释放并发槽位。必须与 acquire_slot 成对使用。"""
        with self._lock:
            self._in_flight = max(0, self._in_flight - 1)
        try:
            self._semaphore.release()
        except ValueError:
            pass
    def stats(self) -> dict:
        """返回当前状态快照，供 --fetch-report 使用。"""
        with self._lock:
@@ -1292,6 +1377,19 @@ def fetch_top_results(results: dict, count: int, timeout: int = 10,
        """Fetch one URL with adaptive throttling to avoid rate limiting."""
        # 全局暂停检查（429 触发）
        throttle.wait_if_paused()
        # v2.2.2：真实并发门控。退避降并发后，超出当前并发目标的新请求
        # 在这里被拒绝（不占用 fetch_page），实现持久降并发而非名义降并发。
        if not throttle.acquire_slot():
            logger.info(f"  [THROTTLE] concurrency cap ({throttle.concurrency}) "
                        f"reached, skipping {u[:55]}")
            return {"url": u, "status": "error",
                    "error": "Throttled: concurrency limit reached",
                    "error_code": E_RATE_LIMIT,
                    "text": "", "text_length": 0, "truncated": False,
                    "anti_bot_detected": False, "waf_type": None,
                    "fallback_used": None,
                    "title": None, "latency": None}
        try:
            # 自适应延迟
            d = throttle.delay
            if d > 0:
@@ -1320,6 +1418,8 @@ def fetch_top_results(results: dict, count: int, timeout: int = 10,
                    anti_bot_count[0] += 1
                logger.error(f"  [ERR] {u[:55]} ({result.get('error', 'unknown')})")
            return result
        finally:
            throttle.release_slot()
    with ThreadPoolExecutor(max_workers=min(throttle.concurrency, len(urls))) as ex:
        future_map = {ex.submit(_fetch_one, u): u for u in urls}
@@ -1333,7 +1433,8 @@ def fetch_top_results(results: dict, count: int, timeout: int = 10,
                                "error_code": classify_error(e),
                                "text": "", "text_length": 0, "truncated": False,
                                "anti_bot_detected": False, "waf_type": None,
-                                "fallback_used": None})
+                                "fallback_used": None,
                                "title": None, "latency": None})
                logger.error(f"  [ERR] {u[:55]} (thread error: {e})")
    # Reorder to match original result order
@@ -1365,8 +1466,7 @@ def _emit_fetch_report(fetched: list, throttle: "AdaptiveThrottle",
 def _emit_fetch_report_text(fetched: list, throttle: "AdaptiveThrottle") -> None:
    """文本表格 + JSON 摘要行（v2.0.0 原始行为，向后兼容）。"""
-    import sys as _sys
+    out = sys.stderr
    out = _sys.stderr
    lines = []
    lines.append("\n" + "=" * 72)
    lines.append("FETCH REPORT (v2.0.0)")
@@ -1403,13 +1503,12 @@ def _emit_fetch_report_text(fetched: list, throttle: "AdaptiveThrottle") -> None
                 f"consec_ok={s['consecutive_successes']}")
    # JSON 摘要（一行，便于 Agent 解析）
    import json as _json
    summary = {
        "total": total, "ok": ok, "error": err,
        "anti_bot_blocked": anti_bot, "wayback_recovered": wayback,
        "throttle": s,
    }
-    lines.append("JSON: " + _json.dumps(summary, ensure_ascii=False))
+    lines.append("JSON: " + json.dumps(summary, ensure_ascii=False))
    lines.append("=" * 72 + "\n")
    print("\n".join(lines), file=out)
@@ -1418,11 +1517,9 @@ def _emit_fetch_report_json(fetched: list, throttle: "AdaptiveThrottle") -> None
    """完整 JSON 报告（items 数组 + summary），输出到 stderr。
    每个 URL 一个对象，包含 url / status / title / content_length /
-    error / error_code / latency / fetched_at 等字段。fetch_page 未采集
+    error / error_code / latency / fetched_at 等字段。v2.3.0 起
-    的字段（title / latency / fetched_at）为 None，便于 Agent 统一解析。
+    ``title`` 与 ``latency`` 由 fetch_page 采集填充（此前恒为 None）。
    """
    import json as _json
    import sys as _sys
    from datetime import datetime, timezone
    fetched_at = datetime.now(timezone.utc).isoformat()
@@ -1434,12 +1531,12 @@ def _emit_fetch_report_json(fetched: list, throttle: "AdaptiveThrottle") -> None
            "url": f.get("url", ""),
            "final_url": f.get("final_url"),
            "status": f.get("status", ""),
-            "title": f.get("title"),            # fetch_page 未提取，None
+            "title": f.get("title"),            # v2.3.0: fetch_page 已采集
            "content_length": f.get("text_length", 0),
            "content_type": f.get("content_type"),
            "error": f.get("error"),
            "error_code": f.get("error_code"),
-            "latency": f.get("latency"),         # fetch_page 未计时，None
+            "latency": f.get("latency"),         # v2.3.0: fetch_page 已计时
            "truncated": f.get("truncated", False),
            "fetched_at": f.get("fetched_at") or fetched_at,
            "waf_type": f.get("waf_type"),
@@ -1464,7 +1561,7 @@ def _emit_fetch_report_json(fetched: list, throttle: "AdaptiveThrottle") -> None
        "throttle": throttle.stats(),
        "items": items,
    }
-    print(_json.dumps(report, ensure_ascii=False), file=_sys.stderr)
+    print(json.dumps(report, ensure_ascii=False), file=sys.stderr)
 # ----- Output formatting -----
@@ -1694,10 +1791,8 @@ def _format_results(results: dict, args) -> str:
    if args.format == "urls":
        return format_urls(results)
    if args.format == "csv":
        import csv as csv_mod
        import io
        out = io.StringIO()
-        writer = csv_mod.writer(out, lineterminator="\n")
+        writer = csv.writer(out, lineterminator="\n")
        writer.writerow(["title", "url", "engine", "score",
                         "published_date", "content"])
        for r in results.get("results", []):
@@ -1803,6 +1898,13 @@ def _run_single_query(query: str, args, instance_urls: list,
    emit_progress("start", query=query, instances=len(instance_urls))
    # v2.2.2：区分"实时查询"与"缓存命中"。原实现用循环末次赋值的
    # ``cached`` 变量判断是否实时查询——多页路径下该变量保存的是最后一页
    # 的状态，导致：最后一页命中缓存但前页实时查询时，unresponsive_engines
    # 警告被错误跳过；反之仅最后一页未命中时误触发。用独立布尔标记精确
    # 跟踪"本次运行是否发起了至少一次实时查询"。
    performed_live_query = False
    # v2.2.0：--pages N 多页聚合。循环 pageno=1..N，每页独立缓存
    # （cache key 含 pageno），合并后统一 dedup/sort/max-results。
    if pages_to_fetch == 1:
@@ -1821,11 +1923,13 @@ def _run_single_query(query: str, args, instance_urls: list,
                    retry_per=args.retry,
                    auth_headers=auth_headers,
                    parallel=not args.serial,
                    encoding=getattr(args, "encoding", None),
                )
            except Exception as e:
                err_code = classify_error(e)
                emit_progress("error", error=str(e), error_code=err_code, query=query)
                return None, str(e), err_code
            performed_live_query = True
            if ttl_seconds > 0:
                cache_module.put(params, results, ttl_seconds)
                logger.info(f"[cache stored] q={query!r} TTL={args.cache_ttl}min")
@@ -1855,6 +1959,7 @@ def _run_single_query(query: str, args, instance_urls: list,
                        retry_per=args.retry,
                        auth_headers=auth_headers,
                        parallel=not args.serial,
                        encoding=getattr(args, "encoding", None),
                    )
                except Exception as e:
                    page_errors += 1
@@ -1862,6 +1967,7 @@ def _run_single_query(query: str, args, instance_urls: list,
                    emit_progress("page_fail", query=query, page=page_no,
                                  error=str(e), error_code=classify_error(e))
                    continue
                performed_live_query = True
                if ttl_seconds > 0:
                    cache_module.put(page_params, page_results, ttl_seconds)
                    emit_progress("cache_store", query=query, ttl=args.cache_ttl, page=page_no)
@@ -1886,7 +1992,7 @@ def _run_single_query(query: str, args, instance_urls: list,
    # v2.1.1：检测实例侧引擎挂起/限流（仅在实时查询后提示，cache hit 时
    # unresponsive_engines 信息可能已过期）
-    if cached is None:
+    if performed_live_query:
        _warn_unresponsive_engines(results, query,
                                   result_count=len(results.get("results", [])))
@@ -1975,14 +2081,54 @@ def _get_output_schema():
    Used by ``--dump-schema`` so AI agents can programmatically discover the
    output structure without parsing prose documentation.
    v2.3.0: fetched.items 字段补全（与 fetch_page 实际输出对齐——
    final_url/status/error_code/waf_type/fallback_used/title/latency 等），
    并新增 ``batch`` 与 ``research`` 两个属性描述对应模式的输出 shape。
    顶层 ``properties`` 仍以单查询为主，batch/research 为单独子树。
    """
-    return {
+    # 单查询结果条目（results[] 的元素）——batch/research 复用
-        "$schema": "https://json-schema.org/draft/2020-12/schema",
+    result_item = {
-        "title": "SearXNG CLI Search Result",
+        "type": "object",
-        "schema_version": SCHEMA_VERSION,
+        "properties": {
-        "description": "Output schema for 'python search.py --format json' (single query). "
+            "title": {"type": "string"},
-                       "Batch mode (--queries-file) wraps results in "
+            "url": {"type": "string", "format": "uri"},
-                       '{"schema_version, queries:[]}.',
+            "engine": {"type": "string", "description": "Source engine name."},
            "score": {"type": ["number", "null"]},
            "published_date": {"type": ["string", "null"]},
            "content": {"type": "string", "description": "Snippet/summary text."},
        },
        "required": ["title", "url"],
    }
    # --fetch N 时的 fetched[] 元素（与 search.fetch_page 返回对齐）
    fetched_item = {
        "type": "object",
        "properties": {
            "url": {"type": "string"},
            "final_url": {"type": ["string", "null"],
                          "description": "URL after redirects / Wayback."},
            "status": {"type": "string", "enum": ["ok", "error"]},
            "content_type": {"type": ["string", "null"]},
            "text": {"type": "string", "description": "Extracted page text."},
            "text_length": {"type": "integer"},
            "truncated": {"type": "boolean"},
            "title": {"type": ["string", "null"],
                      "description": "Page <title>, when available."},
            "latency": {"type": ["number", "null"],
                        "description": "Fetch latency in seconds."},
            "user_agent_used": {"type": ["string", "null"]},
            "error": {"type": ["string", "null"]},
            "error_code": {"type": ["string", "null"]},
            "waf_type": {"type": ["string", "null"]},
            "fallback_used": {"type": ["string", "null"],
                              "description": "'wayback' when the Wayback "
                                             "Machine recovered the page."},
            "anti_bot_detected": {"type": "boolean"},
        },
        "required": ["url", "status"],
    }
    # 单查询输出
    single_query = {
        "type": "object",
        "properties": {
            "schema_version": {
@@ -1999,18 +2145,7 @@ def _get_output_schema():
            "results": {
                "type": "array",
                "description": "Search result items, ordered by relevance (score desc).",
-                "items": {
+                "items": result_item,
                    "type": "object",
                    "properties": {
                        "title": {"type": "string"},
                        "url": {"type": "string", "format": "uri"},
                        "engine": {"type": "string", "description": "Source engine name."},
                        "score": {"type": ["number", "null"]},
                        "published_date": {"type": ["string", "null"]},
                        "content": {"type": "string", "description": "Snippet/summary text."},
                    },
                    "required": ["title", "url"],
                },
            },
            "unresponsive_engines": {
                "type": "array",
@@ -2022,19 +2157,16 @@ def _get_output_schema():
                "items": {"type": "string"},
                "description": "Related query suggestions from the instance.",
            },
            "answers": {
                "type": "array",
                "items": {"type": "string"},
                "description": "Direct answers from the instance.",
            },
            "fetched": {
                "type": "array",
                "description": "Present only when --fetch N is used. Page content "
                               "for the top N results.",
-                "items": {
+                "items": fetched_item,
                    "type": "object",
                    "properties": {
                        "url": {"type": "string"},
                        "text": {"type": "string"},
                        "text_length": {"type": "integer"},
                        "error": {"type": "string"},
                    },
                },
            },
            "fetched_source": {
                "type": "string",
@@ -2042,9 +2174,100 @@ def _get_output_schema():
                "description": "Present only when --fetch is used. Indicates whether "
                               "search results came from JSON API or HTML fallback.",
            },
            "pages_fetched": {
                "type": ["integer", "null"],
                "description": "Present only when --pages N > 1. Pages that "
                               "succeeded (before cross-page merge).",
            },
        },
        "required": ["query", "results"],
    }
    # --queries-file 批量输出
    batch_schema = {
        "type": "object",
        "description": "Batch mode (--queries-file) output shape.",
        "properties": {
            "schema_version": {"type": "string", "const": SCHEMA_VERSION},
            "queries": {
                "type": "array",
                "description": "One entry per query, in file order. "
                               "Entry: {query, status: ok|error, results} on "
                               "success or {query, status: error, error, "
                               "error_code} on failure.",
                "items": {"type": "object",
                          "required": ["query", "status"],
                          "properties": {
                              "query": {"type": "string"},
                              "status": {"type": "string", "enum": ["ok", "error"]},
                              "results": single_query,
                              "error": {"type": "string"},
                              "error_code": {"type": "string"},
                          }},
            },
        },
        "required": ["schema_version", "queries"],
    }
    # --research 研究模式输出
    research_schema = {
        "type": "object",
        "description": "Research mode (--research TOPIC) output shape.",
        "properties": {
            "schema_version": {"type": "string", "const": SCHEMA_VERSION},
            "research_topic": {"type": "string"},
            "research_queries": {
                "type": "array",
                "description": "Expanded per-angle queries (deterministic rules).",
                "items": {"type": "object",
                          "properties": {
                              "angle": {"type": "string"},
                              "query": {"type": "string"},
                          }},
            },
            "queries": {"type": "array",
                        "description": "Per-angle results. Entry: {query, angle, "
                                       "status: ok|error, results|error, error_code}.",
                        "items": {"type": "object",
                                  "properties": {
                                      "query": {"type": "string"},
                                      "angle": {"type": "string"},
                                      "status": {"type": "string",
                                                 "enum": ["ok", "error"]},
                                      "results": single_query,
                                      "error": {"type": "string"},
                                      "error_code": {"type": "string"},
                                  }}},
            "merged_results": {
                "type": "object",
                "description": "Cross-angle merged + deduplicated result set "
                               "(same shape as single-query 'results').",
                "properties": {
                    "query": {"type": "string"},
                    "results": {"type": "array", "items": result_item},
                },
            },
        },
        "required": ["schema_version", "research_topic", "queries",
                     "merged_results"],
    }
    return {
        "$schema": "https://json-schema.org/draft/2020-12/schema",
        "title": "SearXNG CLI Search Result",
        "schema_version": SCHEMA_VERSION,
        "description": "Output schema for 'python search.py --format json'. "
                       "Top-level 'properties' describe single-query output; "
                       "the 'batch' and 'research' subtrees describe "
                       "--queries-file and --research output respectively. "
                       "v2.3.0: fetched.items fields now match fetch_page "
                       "output (title/latency/waf_type/error_code etc.).",
        "type": "object",
        "properties": single_query["properties"],
        "required": single_query["required"],
        "defs": {
            "single_query": single_query,
            "batch": batch_schema,
            "research": research_schema,
        },
    }
 def _emit_error(message: str, args, query: str = None, exit_code: int = 1,
@@ -2087,13 +2310,30 @@ def _emit_error(message: str, args, query: str = None, exit_code: int = 1,
 def _read_queries_file(path: str) -> list:
    """Read queries from a file: one per line, skip blanks and ``#`` comments.
    v2.3.0: 编码自动检测。先按 UTF-8 读取；若解码失败（Windows 下 GBK 等
    非 UTF-8 文件常见），回退 GBK，再失败回退 utf-8 errors=replace——
    绝不因编码问题让批量任务整体失败。
    Raises :class:`RuntimeError` if the file cannot be read, so the caller
    can route it through :func:`_emit_error`.
    """
    raw = None
    try:
-        text = Path(path).read_text(encoding="utf-8")
+        raw = Path(path).read_bytes()
    except OSError as e:
        raise RuntimeError(f"cannot read queries file '{path}': {e}")
    text = None
    for enc in ("utf-8", "gbk"):
        try:
            text = raw.decode(enc)
            break
        except (UnicodeDecodeError, LookupError):
            continue
    if text is None:
        # 最后兜底：UTF-8 + 替换符，保证任务可继续
        text = raw.decode("utf-8", errors="replace")
    queries = []
    for line in text.splitlines():
        line = line.strip()
@@ -2260,7 +2500,7 @@ def expand_research_queries(topic: str, custom_angles: list = None) -> list:
 # ----- Mode handlers (v2.2.0: 从 main() 提取，降低单函数复杂度) -----
 # main() 只负责参数解析和分发，四条执行路径各自独立函数，便于维护和测试。
-# 纯提取重构，行为与 v2.1.1 完全一致，539 测试兜底验证。
+# 纯提取重构，行为与 v2.1.1 完全一致，544 测试兜底验证。
 def _handle_verify(args, instance_urls: list, auth_headers: dict) -> None:
@@ -2340,10 +2580,8 @@ def _handle_research(args, instance_urls: list, auth_headers: dict,
            "merged_results": merged,
        }, indent=2, ensure_ascii=False)
    elif args.format == "csv":
        import csv as csv_mod
        import io
        out = io.StringIO()
-        writer = csv_mod.writer(out, lineterminator="\n")
+        writer = csv.writer(out, lineterminator="\n")
        writer.writerow(["angle", "query", "title", "url", "engine",
                         "score", "published_date", "content"])
        for br in batch:
@@ -2412,11 +2650,33 @@ def _handle_research(args, instance_urls: list, auth_headers: dict,
    sys.exit(0)
 def _run_single_query_wrapper(query: str, args, instance_urls: list,
                              auth_headers: dict, ttl_seconds: int):
    """并发批量用的 _run_single_query 包装（v2.3.0）。
    移除不可跨线程共享的参数：
      * ``args.fetch`` 置 0 —— --fetch 的 ThreadPoolExecutor 在查询线程内
        创建，worker 再经 ThreadPoolExecutor 二次并发会超过线程安全上限；
        并发批量模式用 ``--queries-file`` 不适合内嵌抓取。
      * ``args.output`` 置 None —— 输出写入统一交给 _handle_batch。
    其余参数原样透传（行为与串行路径一致）。
    """
    import copy as _copy
    qargs = _copy.copy(args)
    qargs.fetch = 0
    qargs.output = None
    return _run_single_query(query, qargs, instance_urls, auth_headers,
                             ttl_seconds)
 def _handle_batch(args, instance_urls: list, auth_headers: dict,
                  ttl_seconds: int) -> None:
-    """批量模式：从文件读取多个查询，串行执行，输出合并结果。
+    """批量模式：从文件读取多个查询，串行或并发执行，输出合并结果。
    v2.2.0 从 main() 提取为独立函数（纯重构，行为不变）。
    v2.3.0 新增 --parallel-queries N：并发执行（受 AdaptiveThrottle
    约束），输出保持文件顺序；并发模式下 --fetch 被禁用（见
    _run_single_query_wrapper），日志压缩为每查询一行。
    """
    try:
        queries = _read_queries_file(args.queries_file)
@@ -2426,6 +2686,63 @@ def _handle_batch(args, instance_urls: list, auth_headers: dict,
        _emit_error(f"no queries found in '{args.queries_file}'", args,
                    error_code=E_INPUT)
    parallel = getattr(args, "parallel_queries", 0) or 0
    if parallel < 1:
        parallel = 1
    if parallel > 8:
        parallel = 8
    if parallel > 1:
        logger.info(f"Running {len(queries)} queries from {args.queries_file} "
                    f"in parallel ({parallel} workers)...")
        batch = [None] * len(queries)
        any_with_results = [False]
        error_count = [0]
        def _run(idx_q):
            idx, q = idx_q
            logger.info(f"[{idx+1}/{len(queries)}] {q}")
            results, err, err_code = _run_single_query_wrapper(
                q, args, instance_urls, auth_headers, ttl_seconds)
            return idx, q, results, err, err_code
        throttle = AdaptiveThrottle(0.0, parallel)
        def _job(idx_q):
            # 全局暂停（429）+ 并发槽位门控（退避降并发时快速拒绝）
            throttle.wait_if_paused()
            if not throttle.acquire_slot():
                logger.info(f"  [THROTTLE] concurrency cap ({throttle.concurrency}) "
                            f"reached, skipping query {idx_q[0]+1}")
                return idx_q[0], idx_q[1], None, "Throttled: concurrency limit", E_RATE_LIMIT
            try:
                return _run(idx_q)
            finally:
                throttle.release_slot()
        with ThreadPoolExecutor(max_workers=parallel) as ex:
            futures = [ex.submit(_job, item) for item in enumerate(queries)]
            for fut in as_completed(futures):
                idx, q, results, err, err_code = fut.result()
                if err:
                    error_count[0] += 1
                    entry = {"query": q, "status": "error", "error": err}
                    if err_code:
                        entry["error_code"] = err_code
                    batch[idx] = entry
                else:
                    if len(results.get("results", [])) > 0:
                        any_with_results[0] = True
                    batch[idx] = {"query": q, "status": "ok", "results": results}
        # 日志：并发模式下错误信息以单行输出（串行模式按序打印多行）
        if error_count[0]:
            logger.warning(f"Parallel batch: {error_count[0]} errors, "
                           f"{len(queries) - error_count[0]} ok")
        error_count = error_count[0]
        any_with_results = any_with_results[0]
    else:
        logger.info(f"Running {len(queries)} queries from {args.queries_file}...")
        batch = []
        any_with_results = False
@@ -2450,10 +2767,8 @@ def _handle_batch(args, instance_urls: list, auth_headers: dict,
        output = json.dumps({"schema_version": SCHEMA_VERSION, "queries": batch},
                            indent=2, ensure_ascii=False)
    elif args.format == "csv":
        import csv as csv_mod
        import io
        out = io.StringIO()
-        writer = csv_mod.writer(out, lineterminator="\n")
+        writer = csv.writer(out, lineterminator="\n")
        writer.writerow(["query", "title", "url", "engine", "score",
                         "published_date", "content"])
        for br in batch:
@@ -2627,6 +2942,12 @@ Use --config FILE to load a non-default config file (overrides the auto-discover
                        help="Comma-separated categories (e.g. general,images,news)")
    parser.add_argument("--language", "-l", default=config.get("language"),
                        help="Language code (e.g. en, zh-CN, de)")
    parser.add_argument("--encoding", default=config.get("encoding"),
                        help="Force charset for HTML-fallback decoding "
                             "(e.g. gbk, shift_jis). v2.2.2: auto-detected "
                             "from the HTTP header / HTML meta when omitted; "
                             "this flag overrides auto-detection for "
                             "misconfigured instances.")
    parser.add_argument("--pageno", "-p", type=int, default=1,
                        help="Page number (default: 1)")
    parser.add_argument("--pages", type=int, default=1,
@@ -2769,6 +3090,13 @@ Use --config FILE to load a non-default config file (overrides the auto-discover
                             "starting with '#' are skipped) and run them in sequence. "
                             "Results are emitted as a JSON array (or one brief block per query). "
                             "Overrides --query when set.")
    parser.add_argument("--parallel-queries", type=int, default=0, metavar="N",
                        help="v2.3.0: run batch queries concurrently with N workers "
                             "(1-8, capped at 8; default 0 = sequential). Output order "
                             "is preserved. Concurrency is gated by AdaptiveThrottle — "
                             "on repeated failures workers are throttled, not spammed. "
                             "When enabled, --fetch is disabled (nested parallel fetch "
                             "is unsafe) and per-query logs are compressed.")
    parser.add_argument("--research", default=None, metavar="TOPIC",
                        help="v2.1.0 Research mode: given a topic, auto-expand into 5 "
                             "multi-angle queries (overview/profile/background/works/review) "
@@ -2803,8 +3131,7 @@ Use --config FILE to load a non-default config file (overrides the auto-discover
    # v2.2.0：生成 request_id 并重新配置 logging（带 log_format + request_id）
    # request_id 贯穿所有日志、进度事件和错误输出，便于 batch 模式追溯。
-    import os as _os
+    request_id = os.urandom(4).hex()
    request_id = _os.urandom(4).hex()
    setup_logging(verbose=args.verbose, quiet=args.quiet,
                  log_format=getattr(args, "log_format", "text"),
                  request_id=request_id)
@@ -0,0 +1,366 @@
 """Tests for v2.3.0 changes.
 Covers:
  * multi-page (--pages N) live-query tracking — unresponsive-engine
    warnings fire when ANY page did a live query (regression for the
    ``cached``-variable pollution fix)
  * search_html charset auto-detection + --encoding override
  * UA pool single-source (common.FALLBACK_UAS is _config.UA_POOL)
  * AdaptiveThrottle acquire_slot/release_slot real concurrency gating
    (incl. persistent throttling after backoff reduces concurrency)
  * fetch_page title/latency fields (consumed by --fetch-report json)
  * fetch.py _emit_fetch_result JSON output shapes (success + error)
  * _read_queries_file GBK fallback
  * --parallel-queries concurrent batch (output order preserved)
 """
 import json
 import logging
 import sys
 import time
 import urllib.error
 from types import SimpleNamespace
 from unittest.mock import MagicMock, patch
 import pytest
 import common as common_mod
 import _config
 from fetch import FetchResult, _emit_fetch_result
 import search as search_mod
 from search import (
    AdaptiveThrottle,
    _read_queries_file,
    _run_single_query,
    _build_params,
    fetch_page,
    fetch_top_results,
    search_html,
 )
 import cache as cache_module
 def _make_args(**overrides):
    """args object matching the argparse.Namespace shape main() produces."""
    base = dict(
        query="test", format="json", method="GET", timeout=15, retry=0,
        serial=False, no_dedup=False, sort_by="none", max_results=None,
        include_domain=None, exclude_domain=None, fetch=0, fetch_timeout=10,
        fetch_retries=3, max_size=None, cache_ttl=0, snippet_len=0,
        categories=None, language=None, pageno=1, pages=1, time_range="year",
        safesearch=0, engines="google,bing",
        referer=None, no_fallback=False, fetch_report=False,
        request_delay=0.3, throttle_failure_threshold=3,
        throttle_pause_seconds=30, throttle_max_delay=10,
        similarity_dedup=False, similarity_threshold=0.85,
        encoding=None, output=None, parallel_queries=0,
    )
    base.update(overrides)
    return SimpleNamespace(**base)
 # ===== Multi-page live-query tracking (cached-var pollution fix) =====
 def test_multi_page_live_query_warns_unresponsive(isolated_cache, caplog):
    """pages=2: page1 live + page2 cached hit -> warning still fires.
    Regression: the old code used the last loop iteration's ``cached``
    variable, so a cache hit on page 2 suppressed the unresponsive-engine
    warning even though page 1 hit the network live.
    """
    args = _make_args(pages=2, cache_ttl=30)
    # Pre-fill page 2 cache so page 2 is a cache hit.
    page2_params = dict(_build_params("test", args))
    page2_params["pageno"] = "2"
    cache_module.put(page2_params, {"results": []}, 300)
    live = {"results": [{"url": "https://a.com/1", "title": "A"}],
            "unresponsive_engines": [["brave", "Suspended: too many requests"]]}
    def fake_multi(urls, params, **kw):
        assert params.get("pageno") is None  # only page 1 hits the network
        return live
    with patch.object(search_mod, "search_multi", side_effect=fake_multi):
        with caplog.at_level(logging.WARNING, logger="searxng.search"):
            results, err, _ = _run_single_query(
                "test", args, ["https://x.example.com"], {}, 300)
    assert err is None
    assert any("unresponsive" in r.getMessage()
               for r in caplog.records), "warning should fire after a live query"
    assert results["pages_fetched"] == 2
 def test_multi_page_all_cache_hit_no_warning(isolated_cache, caplog):
    """pages=2: both pages cached -> no network, no warning."""
    args = _make_args(pages=2, cache_ttl=30)
    cached = {"results": [{"url": "https://c.com/1", "title": "C"}]}
    for page_no in (1, 2):
        p = dict(_build_params("test", args))
        if page_no != 1:
            p["pageno"] = str(page_no)
        cache_module.put(p, cached, 300)
    with patch.object(search_mod, "search_multi",
                      side_effect=AssertionError("must not hit network")):
        with caplog.at_level(logging.WARNING, logger="searxng.search"):
            results, err, _ = _run_single_query(
                "test", args, ["https://x.example.com"], {}, 300)
    assert err is None
    assert not any("unresponsive" in r.getMessage()
                   for r in caplog.records)
 # ===== search_html charset detection =====
 def test_search_html_detects_gbk_charset():
    """GBK HTML detected via <meta charset> (header has no charset)."""
    html = ('<html><head><meta charset="gbk"></head><body>'
            '<article class="result"><h3><a href="https://x.com">中文标题</a>'
            '</h3></article></body></html>')
    raw = html.encode("gbk")
    resp = MagicMock()
    resp.read.return_value = raw
    resp.headers = {"Content-Type": "text/html"}  # no charset in header
    resp.__enter__.return_value = resp
    with patch("urllib.request.urlopen", return_value=resp):
        r = search_html("https://s.example.com", {"q": "test"})
    assert r["results"][0]["title"] == "中文标题"
 def test_search_html_encoding_override():
    """--encoding gbk forces the charset even with wrong header."""
    html = '<article class="result"><h3><a href="https://x.com">标题</a></h3></article>'
    raw = html.encode("gbk")
    resp = MagicMock()
    resp.read.return_value = raw
    resp.headers = {"Content-Type": "text/html; charset=utf-8"}  # lies
    resp.__enter__.return_value = resp
    with patch("urllib.request.urlopen", return_value=resp):
        r = search_html("https://s.example.com", {"q": "test"}, encoding="gbk")
    assert r["results"][0]["title"] == "标题"
 # ===== UA pool single source =====
 def test_ua_pool_is_single_source():
    """common.FALLBACK_UAS must be the _config.UA_POOL object (no duplicate)."""
    assert common_mod.FALLBACK_UAS is _config.UA_POOL
    assert len(_config.UA_POOL) >= 12
 # ===== AdaptiveThrottle slot gating =====
 def test_throttle_acquire_release_slot_basic():
    """acquire_slot respects the concurrency cap; release frees it."""
    t = AdaptiveThrottle(0.0, 1)
    assert t.acquire_slot() is True
    # Second acquisition must fail while one slot is in flight.
    assert t.acquire_slot(timeout=0.05) is False
    t.release_slot()
    assert t.acquire_slot(timeout=0.05) is True
    t.release_slot()
 def test_throttle_slot_gates_on_reduced_concurrency():
    """After backoff halves concurrency, in-flight requests gate new ones.
    Regression: ThreadPoolExecutor size is fixed at creation; the semaphore
    + in-flight check must persist the reduced concurrency. With
    initial=2, two slots in flight, then backoff -> concurrency=1: a new
    acquisition must be rejected even though a semaphore slot is free.
    """
    t = AdaptiveThrottle(0.0, 2)
    assert t.acquire_slot() is True
    assert t.acquire_slot() is True
    # 3 consecutive failures -> concurrency 2 -> 1
    for _ in range(3):
        t.report_failure()
    assert t.concurrency == 1
    # Release one in-flight slot: semaphore has room, but in_flight (1)
    # now exceeds the reduced concurrency target (1) -> reject.
    t.release_slot()
    assert t.acquire_slot(timeout=0.05) is False
    t.release_slot()  # cleanup the remaining acquired slot
 def test_fetch_top_results_reports_throttled_skips():
    """fetch_top_results marks requests gated by the concurrency cap.
    Semantics covered deterministically by the acquire_slot unit tests;
    this integration test asserts the end-to-end shape: a URL skipped by
    the gate appears with status=error + error_code=E_RATE_LIMIT. We
    simulate the gate directly by stubbing acquire_slot.
    """
    throttle = AdaptiveThrottle(0.0, 5)
    real_acquire = throttle.acquire_slot
    def _stub_acquire(timeout=0.05):
        # First call wins the slot; every subsequent call is gated.
        return False
    results = {"results": [{"url": "https://a.com"},
                           {"url": "https://b.com"}]}
    def _fake_fetch(url, **kwargs):
        return {"url": url, "status": "ok", "text": "x", "text_length": 1,
                "truncated": False}
    with patch.object(search_mod, "fetch_page", side_effect=_fake_fetch):
        with patch.object(throttle, "acquire_slot", side_effect=_stub_acquire):
            out = fetch_top_results(results, 2, request_delay=0.0,
                                    throttle=throttle)
    assert len(out) == 2
    for f in out:
        assert f["status"] == "error"
        assert f["error_code"] == "E_RATE_LIMIT"
        assert "Throttled" in f["error"]
 # ===== fetch_page title/latency =====
 def _mock_fetch_result(content, content_type="text/html",
                       final_url="https://example.com", ua="TestUA"):
    return FetchResult(content=content, content_type=content_type,
                       final_url=final_url, truncated=False, user_agent=ua)
 def test_fetch_page_collects_title_and_latency():
    html = "<html><head><title>Page Title</title></head>" \
           "<body><article>Hello world</article></body></html>"
    with patch.object(search_mod, "fetch_url",
                      return_value=_mock_fetch_result(html)):
        r = fetch_page("https://example.com", fallback_enabled=False)
    assert r["status"] == "ok"
    assert r["title"] == "Page Title"
    assert r["latency"] is not None and r["latency"] >= 0
 def test_fetch_page_title_none_for_non_html():
    with patch.object(search_mod, "fetch_url",
                      return_value=_mock_fetch_result(
                          "plain", content_type="text/plain")):
        r = fetch_page("https://example.com/x.txt", fallback_enabled=False)
    assert r["status"] == "ok"
    assert r["title"] is None
    assert r["latency"] is not None
 # ===== fetch.py --format json output =====
 def _json_args(**overrides):
    base = dict(url="https://x.example.com", extract="text", format="json",
                output=None)
    base.update(overrides)
    return SimpleNamespace(**base)
 def test_emit_fetch_result_json_success(capsys):
    _emit_fetch_result(_json_args(), "hello world", "https://x.example.com",
                       "https://x.example.com/", "text/html", False,
                       user_agent="TestUA")
    out, _ = capsys.readouterr()
    data = json.loads(out)
    assert data["status"] == "ok"
    assert data["url"] == "https://x.example.com"
    assert data["final_url"] == "https://x.example.com/"
    assert data["extract"] == "text"
    assert data["text_length"] == 11
    assert data["truncated"] is False
    assert data["user_agent"] == "TestUA"
    assert "content_type" in data
 def test_emit_fetch_result_json_error(capsys):
    _emit_fetch_result(_json_args(), "", "https://x.example.com",
                       "https://x.example.com", "", False,
                       error="HTTP 404", error_code="E_NETWORK",
                       status_code=404)
    out, _ = capsys.readouterr()
    data = json.loads(out)
    assert data["status"] == "error"
    assert data["error_code"] == "E_NETWORK"
    assert data["status_code"] == 404
    assert data["url"] == "https://x.example.com"
 def test_emit_fetch_result_text_mode_unchanged(capsys):
    """--format text keeps raw content on stdout (v2.2.x behavior)."""
    _emit_fetch_result(_json_args(format="text"), "raw content",
                       "https://x.example.com", "https://x.example.com",
                       "text/html", False)
    out, _ = capsys.readouterr()
    assert out == "raw content\n"
 # ===== _read_queries_file encoding fallback =====
 def test_read_queries_file_gbk_fallback(tmp_path):
    """GBK-encoded queries file decodes via the utf-8 -> gbk fallback."""
    p = tmp_path / "queries.txt"
    p.write_bytes("中文查询一\n中文查询二\n".encode("gbk"))
    qs = _read_queries_file(str(p))
    assert qs == ["中文查询一", "中文查询二"]
 def test_read_queries_file_utf8_preferred(tmp_path):
    p = tmp_path / "queries.txt"
    p.write_text("q1\n# comment\nq2\n", encoding="utf-8")
    assert _read_queries_file(str(p)) == ["q1", "q2"]
 # ===== --parallel-queries concurrent batch =====
 def _batch_argv(queries_file_path, parallel):
    return ["search.py", "-i", "https://x.example.com",
            "--queries-file", str(queries_file_path),
            "--parallel-queries", str(parallel),
            "--format", "json", "--retry", "0"]
 def test_parallel_batch_preserves_order(tmp_path, capsys):
    """Concurrent batch output keeps the file order (deterministic)."""
    qf = tmp_path / "queries.txt"
    qf.write_text("q1\nq2\nq3\nq4\n", encoding="utf-8")
    def fake_multi(urls, params, **kw):
        q = params["q"]
        time.sleep(0.01 * int(q[-1]))  # q4 slowest, q1 fastest
        return {"results": [{"url": f"https://{q}.com", "title": q}]}
    with patch.object(search_mod, "search_multi", side_effect=fake_multi):
        with pytest.raises(SystemExit) as exc_info:
            with patch.object(sys, "argv", _batch_argv(qf, 2)):
                with patch.object(search_mod, "setup_logging"):
                    search_mod.main()
    assert exc_info.value.code == 0
    out, _ = capsys.readouterr()
    data = json.loads(out)
    queries = [e["query"] for e in data["queries"]]
    assert queries == ["q1", "q2", "q3", "q4"]  # order preserved
    assert all(e["status"] == "ok" for e in data["queries"])
 def test_parallel_batch_records_errors(tmp_path, capsys):
    """Concurrent batch keeps per-query error entries without aborting."""
    qf = tmp_path / "queries.txt"
    qf.write_text("ok1\nbad1\nok2\n", encoding="utf-8")
    err = urllib.error.URLError("connection refused")
    def fake_multi(urls, params, **kw):
        if params["q"].startswith("bad"):
            raise err
        return {"results": [{"url": f"https://{params['q']}.com"}]}
    with patch.object(search_mod, "search_multi", side_effect=fake_multi):
        with pytest.raises(SystemExit) as exc_info:
            with patch.object(sys, "argv", _batch_argv(qf, 3)):
                with patch.object(search_mod, "setup_logging"):
                    search_mod.main()
    assert exc_info.value.code == 0  # partial success
    out, _ = capsys.readouterr()
    data = json.loads(out)
    statuses = {e["query"]: e for e in data["queries"]}
    assert statuses["bad1"]["status"] == "error"
    assert statuses["bad1"]["error_code"] == "E_NETWORK"
    assert statuses["ok1"]["status"] == "ok"
    assert statuses["ok2"]["status"] == "ok"