MetonaTeam/searxng-use-cli
2
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(v2.3.0): fetch 结构化 JSON 契约 + 并发批量 + 真实并发门控

Browse Source 
迭代 1 — 正确性修复:
- 修复 --pages N 多页聚合的 unresponsive-engine 警告误判: 原用循环末次
  cached 变量判断, 缓存命中时警告被错误跳过/误触发; 改用独立
  performed_live_query 标记
- UA 池单一来源: 删除 common.py 手工副本 _FALLBACK_UAS_BUILTIN,
  FALLBACK_UAS 直接引用 _config.UA_POOL, 消除双份漂移
- search_html 解码修复: 硬编码 utf-8 改为 detect_charset(header/meta
  自动检测), 新增 --encoding 强制覆盖, 贯穿 search_multi 全链
- AdaptiveThrottle 真实并发门控: acquire_slot()/release_slot() 槽位机制,
  退避降并发后新请求被快速拒绝(E_RATE_LIMIT), 实现持久降并发而非名义降并发

迭代 2 — fetch JSON 契约 + 批量并发:
- fetch.py --format json: 成功 {status,url,final_url,content_type,extract,
  truncated,text_length,user_agent}; 失败 {status,error,error_code,
  status_code,url}, 对齐 search.py 错误码体系
- fetch_page 采集 title + latency, 填充 --fetch-report json 空字段
- --queries-file --parallel-queries N (1-8): 并发批量, 输出保序, 受
  AdaptiveThrottle 门控; 并发模式禁用 --fetch(嵌套并行不安全)
- queries 文件编码自动检测 (UTF-8 → GBK 回退)

迭代 3 — 工程化:
- 新增 pyproject.toml (searxng-search/searxng-fetch 入口点)
- 收敛 20+ 处函数内冗余导入
- --dump-schema 扩展: fetched.items 补全 15 字段, 新增 defs.batch/research
- 新增 17 个测试 (tests/test_v230_features.py), 全量 561 测试通过
- 文档同步 (SKILL.md/README.md, 版本号 2.3.0)
This commit is contained in:
thzxx
2026-08-05 20:14:07 +08:00
parent 6cedba9042
commit 4df521dc9d
8 changed files with 1006 additions and 226 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/_config.py
+1 -1
View File
@@ -7,7 +7,7 @@ Retry settings and shared HTTP utilities now live in ``common.py`` so that
both ``search.py`` and ``fetch.py`` share one consistent implementation.
"""
VERSION = "2.2.1"
VERSION = "2.3.0"
SCHEMA_VERSION = "1.0"
USER_AGENT = f"searxng-cli/{VERSION}"
scripts/common.py
+29 -84
View File
@@ -18,13 +18,20 @@ as retryable and connection errors as transient, eliminating the previous
inconsistency where ``search.py`` ignored 5xx.
"""
import base64
import datetime
import hashlib
import io
import json
import logging
import os
import random
import re
import sys
import threading
import urllib.error
import urllib.parse
from pathlib import Path
# Root logger for the searxng-cli package. All modules create child loggers
# via ``logging.getLogger("searxng.<module>")`` so a single setup_logging()
@@ -105,7 +112,6 @@ class _JsonFormatter(logging.Formatter):
"""
def format(self, record):
import json as _json
entry = {
"ts": _datetime_iso(record),
"level": record.levelname,
@@ -117,13 +123,12 @@ class _JsonFormatter(logging.Formatter):
entry["request_id"] = rid
if record.exc_info and record.exc_info[1]:
entry["exception"] = type(record.exc_info[1]).__name__
return _json.dumps(entry, ensure_ascii=False)
return json.dumps(entry, ensure_ascii=False)
def _datetime_iso(record):
"""格式化日志时间戳为 ISO 8601 字符串。"""
import datetime as _dt
return _dt.datetime.fromtimestamp(record.created).isoformat(timespec="milliseconds")
return datetime.datetime.fromtimestamp(record.created).isoformat(timespec="milliseconds")
def force_utf8_stdout() -> None:
@@ -191,57 +196,17 @@ RETRYABLE_STATUS = frozenset({403, 429, 502, 503, 504})
# Browser-like UA strings for fallback when the searxng-cli UA is blocked.
#
# v2.2.0：UA 池迁移至 _config.py 的 UA_POOL（SSOT），此处通过导入引用。
# 若 _config.py 不可导入（如 common.py 被单独分发），回退到下方内置副本
# _FALLBACK_UAS_BUILTIN，保证模块始终可用。两份列表需保持同步，
# _config.UA_POOL 为唯一权威来源。
# v2.2.2：单一来源（SSOT）。UA 池唯一权威定义在 _config.UA_POOL，
# 此处直接导入——删除原有的内置副本 _FALLBACK_UAS_BUILTIN。
# v2.2.0 曾保留一份手工同步副本，两份列表漂移会导致跨脚本 UA 行为不一致
# （fetch.py 用 FALLBACK_UAS 轮换、search.py 的 --dry-run 报告引用同一池），
# 且注释要求"保持同步"无任何机制保证。直接引用同一对象后，改一处即全局生效。
#
# v2.0.0 起覆盖 Chrome/Edge/Firefox × Windows/macOS/Linux；v2.2.0 升级到
# 2026 年版本（Chrome 138-140 / Edge 138 / Firefox 140 / Safari 18）。
# 顺序固定以便 get_ua_for_domain() 的 hash 选择可复现。
_FALLBACK_UAS_BUILTIN = [
# Chrome 140 — Windows / macOS / Linux
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36",
# Chrome 139 — Windows / macOS / Linux
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36",
# Chrome 138 — Windows / macOS / Linux
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
# Edge 138 — Windows / macOS
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 Edg/138.0.0.0",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 Edg/138.0.0.0",
# Firefox 140 — Windows / macOS / Linux
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:140.0) "
"Gecko/20100101 Firefox/140.0",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:140.0) "
"Gecko/20100101 Firefox/140.0",
"Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0",
# Safari 18 — macOS（WebKit 指纹，应对 Chromium 针对性拦截）
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 "
"(KHTML, like Gecko) Version/18.0 Safari/605.1.15",
]
# 从 _config.py 导入权威 UA_POOL；导入失败时回退到内置副本，保证向后兼容。
try:
from _config import UA_POOL as FALLBACK_UAS
except ImportError:
FALLBACK_UAS = _FALLBACK_UAS_BUILTIN
# 维护原则（见 _config.UA_POOL 注释）：
# 1. 版本号保持为当前年份的主流浏览器版本
# 2. 顺序固定——get_ua_for_domain() 用 SHA-256 哈希选索引
# 3. 至少覆盖 Chrome/Edge/Firefox × Windows/macOS/Linux
from _config import UA_POOL as FALLBACK_UAS
def _ua_index_for_domain(domain: str, pool_size: int) -> int:
@@ -251,7 +216,6 @@ def _ua_index_for_domain(domain: str, pool_size: int) -> int:
随机化（PYTHONHASHSEED），跨进程不可复现。SHA-256 保证同一域名
永远映射到同一索引，跨进程一致——这对调试和日志分析至关重要。
"""
import hashlib
h = hashlib.sha256(domain.encode("utf-8")).digest()
# 取前 8 字节作为无符号整数，避免负数和短字符串的分布不均
return int.from_bytes(h[:8], "big") % pool_size
@@ -288,9 +252,8 @@ def get_ua_for_domain(url: str, user_agent: str = None) -> str:
if user_agent:
return user_agent
import urllib.parse as _up
try:
domain = _up.urlparse(url).netloc.lower()
domain = urllib.parse.urlparse(url).netloc.lower()
if not domain:
return FALLBACK_UAS[0]
except Exception:
@@ -367,7 +330,6 @@ def build_browser_headers(user_agent: str, referer: str = None,
# Sec-Ch-Ua 系列仅 Chrome/Edge 发送，Firefox 不发
if not is_firefox:
# 从 UA 提取主版本号，构造 Sec-Ch-Ua
import re
m = re.search(r"Chrome/(\d+)", user_agent)
ver = m.group(1) if m else "131"
not_a_brand = '"Not_A Brand";v="99"' if ver != "99" else '"Not/A)Brand";v="99"'
@@ -445,7 +407,6 @@ def compute_backoff_delay(attempt: int, base: float = RETRY_BACKOFF_BASE,
``base * 2^attempt + jitter``，但不超过 ``cap``。
v2.0.0 新增封顶：原公式无上限，N=10 时达 1536s 会卡死进程。
"""
import random
delay = base * (2 ** attempt) + random.uniform(0, 1)
return min(delay, cap)
@@ -459,8 +420,6 @@ def build_auth_headers(bearer_token: str = None, basic_auth: str = None) -> dict
If both are provided, Bearer takes precedence (more common for APIs).
Returns a dict to merge into request headers, or an empty dict.
"""
import base64
headers = {}
if bearer_token:
headers["Authorization"] = f"Bearer {bearer_token}"
@@ -476,7 +435,6 @@ def _warn_file_perms(path: str) -> None:
On Windows the Unix permission bits in ``st_mode`` do not reflect the
actual ACL, so the check is skipped to avoid false alarms.
"""
import os
if os.name != "posix":
return
log = logging.getLogger("searxng.common")
@@ -513,7 +471,6 @@ def resolve_auth_basic(cli_value: str = None, file_path: str = None,
if file_path:
try:
from pathlib import Path
text = Path(file_path).read_text(encoding="utf-8")
_warn_file_perms(file_path)
for line in text.splitlines():
@@ -527,7 +484,6 @@ def resolve_auth_basic(cli_value: str = None, file_path: str = None,
if config_value:
return config_value
import os
return os.environ.get(env_var)
@@ -550,7 +506,6 @@ def resolve_auth_bearer(cli_value: str = None, file_path: str = None,
if file_path:
try:
from pathlib import Path
text = Path(file_path).read_text(encoding="utf-8")
_warn_file_perms(file_path)
for line in text.splitlines():
@@ -564,7 +519,6 @@ def resolve_auth_bearer(cli_value: str = None, file_path: str = None,
if config_value:
return config_value
import os
return os.environ.get(env_var)
@@ -583,7 +537,6 @@ def apply_proxy(proxy_url: str) -> None:
Pass an empty string to clear the proxy env vars (rarely needed; the
default unset state already means "no proxy").
"""
import os
if not proxy_url:
return
os.environ["HTTP_PROXY"] = proxy_url
@@ -597,8 +550,6 @@ def detect_charset(raw: bytes, content_type: str) -> str:
Falls back to UTF-8 (with replacement) if nothing reliable is found.
"""
import re
# 1. HTTP header
if "charset=" in content_type:
charset = content_type.split("charset=")[-1].split(";")[0].strip()
@@ -704,7 +655,7 @@ RECOVERY_HINTS = {
}
def _classify_by_type_and_status(exc, _json):
def _classify_by_type_and_status(exc) -> str:
"""根据异常类型和 HTTP 状态码分类，返回错误码或 None。
检查顺序：HTTP 状态码 → 文件错误 → 连接错误 → 解析错误。
@@ -743,7 +694,7 @@ def _classify_by_type_and_status(exc, _json):
return E_NETWORK
# 解析错误
if isinstance(exc, (ValueError, _json.JSONDecodeError)):
if isinstance(exc, (ValueError, json.JSONDecodeError)):
return E_PARSE
return None
@@ -764,8 +715,6 @@ def classify_error(exc: BaseException) -> str:
兼容旧路径——search_multi 把 last_error 拼进消息）
7. 其他 → E_INTERNAL
"""
import json as _json
# 优先检查异常链 __cause__：raise X from Y 时，Y 指向真实底层异常。
# 旧逻辑只能从外层 RuntimeError 的消息字符串推断（脆弱，依赖 "403"/"auth"
# 等关键字匹配），新逻辑直接从 __cause__ 的 .code/.status_code 或异常
@@ -773,12 +722,12 @@ def classify_error(exc: BaseException) -> str:
# raise-from 模式，深层链罕见且递归有循环风险。
cause = getattr(exc, "__cause__", None)
if cause is not None and cause is not exc:
code = _classify_by_type_and_status(cause, _json)
code = _classify_by_type_and_status(cause)
if code is not None:
return code
# 检查 exc 本身的类型和状态码
code = _classify_by_type_and_status(exc, _json)
code = _classify_by_type_and_status(exc)
if code is not None:
return code
@@ -795,8 +744,7 @@ def classify_error(exc: BaseException) -> str:
return E_RATE_LIMIT
# 从 "http error NNN" / "http NNN" 模式中提取状态码，
# 正确分类 "All instances failed. Last error: HTTP Error 403" 等
import re as _re
status_match = _re.search(r'http(?: error)? (\d{3})', msg)
status_match = re.search(r'http(?: error)? (\d{3})', msg)
if status_match:
status = int(status_match.group(1))
if status == 429:
@@ -855,13 +803,12 @@ def emit_progress(event: str, **kwargs) -> None:
"""
if not _progress_enabled:
return
import json as _json
payload = {"event": event}
rid = getattr(_LOG, "_request_id", None)
if rid:
payload["request_id"] = rid
payload.update(kwargs)
print(_json.dumps(payload, ensure_ascii=False), file=sys.stderr, flush=True)
print(json.dumps(payload, ensure_ascii=False), file=sys.stderr, flush=True)
# ----- Wayback Machine 兜底（v2.1.0 共享逻辑）-----
@@ -945,8 +892,7 @@ def is_hard_blocked_domain(url: str) -> bool:
return False
# 提取域名
try:
from urllib.parse import urlparse
host = urlparse(url).hostname or ""
host = urllib.parse.urlparse(url).hostname or ""
except Exception:
host = ""
if not host:
@@ -1051,9 +997,8 @@ def is_similar(result_a: dict, result_b: dict, threshold: float = 0.85) -> bool:
title_b = _normalize_title(result_b.get("title", ""))
# 标题太短时 SimHash 不稳定，改用 Jaccard
if len(title_a) < 5 or len(title_b) < 5:
import urllib.parse as _up
domain_a = _up.urlparse(result_a.get("url", "")).netloc.lower()
domain_b = _up.urlparse(result_b.get("url", "")).netloc.lower()
domain_a = urllib.parse.urlparse(result_a.get("url", "")).netloc.lower()
domain_b = urllib.parse.urlparse(result_b.get("url", "")).netloc.lower()
set_a = set(title_a.split()) | {domain_a}
set_b = set(title_b.split()) | {domain_b}
return _jaccard_similarity(set_a, set_b) >= threshold
scripts/fetch.py
+104 -21
View File
@@ -11,6 +11,7 @@ for improved extraction quality (optional, falls back to stdlib).
import argparse
import gzip
import io
import json
import logging
import random
import re
@@ -53,6 +54,7 @@ from common import (
build_auth_headers,
build_browser_headers,
build_wayback_url,
classify_error,
compute_backoff_delay,
detect_charset,
force_utf8_stdout,
@@ -1134,6 +1136,55 @@ def fetch_url(url: str, timeout=15, user_agent: str = None,
# ----- Main -----
def _emit_fetch_result(args, output: str, url: str, final_url: str,
content_type: str, truncated: bool,
user_agent: str = None,
error: str = None, error_code: str = None,
status_code: int = None) -> None:
"""输出抓取结果到 stdout / --output 文件。
v2.3.0: ``--format json`` 提供结构化 JSON 契约，AI Agent 可程序化
解析（成功与失败统一为 {status, url, ...}）。``--format text``（默认）
保持 v2.2.x 行为：成功输出正文，失败输出空 + stderr 日志。
成功 shape::
{"status": "ok", "url", "final_url", "content_type",
"extract", "truncated", "text_length", "user_agent"}
失败 shape::
{"status": "error", "url", "error", "error_code", "status_code"}
"""
if args.format == "json":
if error:
payload = {"status": "error", "url": url, "error": error}
if error_code:
payload["error_code"] = error_code
if status_code is not None:
payload["status_code"] = status_code
else:
payload = {
"status": "ok",
"url": url,
"final_url": final_url,
"content_type": content_type,
"extract": args.extract,
"truncated": truncated,
"text_length": len(output),
"user_agent": user_agent,
}
text = json.dumps(payload, indent=2, ensure_ascii=False)
else:
text = output
if args.output:
with open(args.output, "w", encoding="utf-8") as f:
f.write(text)
logger.info(f"Saved {len(text)} chars to {args.output}")
else:
print(text)
def main():
parser = argparse.ArgumentParser(
description="Fetch a web page and extract readable content",
@@ -1146,11 +1197,18 @@ Examples:
%(prog)s -u https://example.com -e markdown markdown conversion
%(prog)s -u https://example.com -o page.txt save to file
%(prog)s -u https://example.cn -e text --encoding gbk force charset
%(prog)s -u https://example.com --format json structured JSON output
""",
)
parser.add_argument("--url", "-u", required=True, help="URL to fetch")
parser.add_argument("--extract", "-e", choices=["text", "html", "markdown"],
default="text", help="Extraction mode (default: text)")
parser.add_argument("--format", "-f", choices=["text", "json"], default="text",
help="Output format (default: text). 'json' emits a structured "
"JSON object {status, url, final_url, content_type, extract, "
"truncated, text_length, user_agent} on success, or "
"{status: error, error, error_code, status_code, url} on "
"failure — machine-readable for agents. v2.3.0.")
parser.add_argument("--timeout", "-t", type=int, default=15,
help="Request timeout in seconds (default: 15)")
parser.add_argument("--retries", type=int, default=3,
@@ -1229,6 +1287,18 @@ Examples:
logger.info(f"Hard-blocked domain detected — Wayback fallback "
f"will be prioritized if main fetch fails")
# v2.3.0: 状态收集变量。所有失败路径设置 fatal_* 后落到统一输出
# （_emit_fetch_result），json 模式输出结构化错误到 stdout，text 模式
# 保持 v2.2.x 行为（stdout 空 + stderr 日志 + exit 1）。
content = None
final_url = args.url
content_type = ""
truncated = False
user_agent = None
fatal_error = None
fatal_error_code = None
fatal_status_code = None
try:
result = fetch_url(
args.url, timeout=args.timeout, user_agent=args.user_agent,
@@ -1237,17 +1307,17 @@ Examples:
allow_redirects=not args.no_redirect,
referer=args.referer,
)
content, content_type, final_url = (
result.content, result.content_type, result.final_url,
)
content = result.content
content_type = result.content_type or ""
final_url = result.final_url
truncated = result.truncated
user_agent = result.user_agent
# 文档解析失败（PDF/DOCX/XLSX 等）时 fetch_url 不抛异常，
# 而是返回带 error_code 的 FetchResult——必须显式检查，
# 否则失败会被静默吞掉（空输出 + exit 0）。
if result.error_code:
logger.error(
f"Error: {result.error_message or result.error_code} "
f"(error_code={result.error_code}, url={args.url})")
sys.exit(1)
fatal_error = result.error_message or result.error_code
fatal_error_code = result.error_code
except Exception as e:
# 诊断信息增强：从 __cause__ 链中提取 HTTP 状态码、原始异常类型，
# 让 AI Agent 能程序化判断失败原因（404 vs 403 vs DNS 失败等），
@@ -1275,7 +1345,10 @@ Examples:
# v2.1.0: Wayback Machine 兜底
# 触发条件：兜底启用 + (错误可恢复 OR 命中被墙站点)
error_msg = str(e) if str(e) else e.__class__.__name__
fatal_error = str(e) if str(e) else e.__class__.__name__
fatal_error_code = classify_error(e)
fatal_status_code = status_code
error_msg = fatal_error
if fallback_enabled and (should_try_wayback(error_msg) or hard_blocked):
wayback_url = build_wayback_url(args.url)
wb_timeout = min(args.timeout, 10) # Wayback 独立超时，不阻塞
@@ -1289,17 +1362,31 @@ Examples:
max_size=args.max_size,
allow_redirects=True,
)
content, content_type, final_url = (
wb_result.content, wb_result.content_type,
wb_result.final_url,
)
content = wb_result.content
content_type = wb_result.content_type or ""
final_url = wb_result.final_url
truncated = wb_result.truncated
user_agent = wb_result.user_agent
if wb_result.error_code:
fatal_error = (wb_result.error_message or wb_result.error_code)
fatal_error_code = wb_result.error_code
fatal_status_code = None
else:
fatal_error = None
fatal_error_code = None
fatal_status_code = None
logger.info(f"[FALLBACK] Wayback recovery successful "
f"({len(content)} chars)")
except Exception as wb_e:
logger.error(f"[FALLBACK] Wayback also failed: {wb_e}")
sys.exit(1)
else:
sys.exit(1)
fatal_error = f"{fatal_error} ; Wayback also failed: {wb_e}"
if fatal_error:
_emit_fetch_result(args, "", args.url, final_url, content_type,
truncated, user_agent, error=fatal_error,
error_code=fatal_error_code,
status_code=fatal_status_code)
sys.exit(1)
if final_url != args.url:
logger.info(f"Redirected to: {final_url}")
@@ -1320,12 +1407,8 @@ Examples:
logger.warning(f"Warning: extracted text is very short ({len(output.strip())} chars). "
"The page may be JS-heavy or use anti-bot protection.")
if args.output:
with open(args.output, "w", encoding="utf-8") as f:
f.write(output)
logger.info(f"Saved {len(output)} chars to {args.output}")
else:
print(output)
_emit_fetch_result(args, output, args.url, final_url, content_type,
truncated, user_agent)
if __name__ == "__main__":
scripts/search.py
+437 -110
View File
@@ -9,6 +9,8 @@ Instance URLs are REQUIRED (see --instance / SEARXNG_INSTANCE / config file).
"""
import argparse
import csv
import io
import json
import logging
import os
@@ -37,6 +39,7 @@ from common import (
build_wayback_url,
classify_error,
compute_backoff_delay,
detect_charset,
emit_progress,
force_utf8_stdout,
is_hard_blocked_domain,
@@ -549,8 +552,13 @@ def search_json(instance: str, params: dict, method: str = "GET",
def search_html(instance: str, params: dict, timeout: int = 15,
auth_headers: dict = None) -> dict:
"""Execute search via HTML scraping fallback."""
auth_headers: dict = None, encoding: str = None) -> dict:
"""Execute search via HTML scraping fallback.
v2.2.2：解码改用 common.detect_charset（此前硬编码 utf-8，GBK/Shift-JIS
等非 UTF-8 实例的页面会整体乱码）。``encoding`` 为显式覆盖（来自
``--language`` 无关的 CLI ``--encoding``），优先级最高。
"""
html_params = {k: v for k, v in params.items() if k != "format"}
query_string = urllib.parse.urlencode(html_params)
url = f"{instance}/search?{query_string}"
@@ -559,26 +567,41 @@ def search_html(instance: str, params: dict, timeout: int = 15,
req = urllib.request.Request(url, headers=headers)
try:
with urllib.request.urlopen(req, timeout=timeout) as resp:
html = resp.read().decode("utf-8")
raw = resp.read()
content_type = resp.headers.get("Content-Type", "")
if encoding:
try:
html = raw.decode(encoding)
except (UnicodeDecodeError, LookupError):
html = raw.decode("utf-8", errors="replace")
else:
charset = detect_charset(raw, content_type)
try:
html = raw.decode(charset)
except (UnicodeDecodeError, LookupError):
html = raw.decode("utf-8", errors="replace")
return parse_html_results(html, query=params.get("q", ""))
except Exception as e:
raise RuntimeError(f"HTML search failed for {instance}: {e}")
def search_single(instance: str, params: dict, method: str = "GET",
timeout: int = 15, auth_headers: dict = None) -> dict:
timeout: int = 15, auth_headers: dict = None,
encoding: str = None) -> dict:
"""Execute one search attempt, preferring JSON with HTML fallback."""
result = search_json(instance, params, method=method, timeout=timeout,
auth_headers=auth_headers)
if result is not None:
return result
logger.warning(f"Warning: {instance} does not support format=json, falling back to HTML parsing")
return search_html(instance, params, timeout=timeout, auth_headers=auth_headers)
return search_html(instance, params, timeout=timeout, auth_headers=auth_headers,
encoding=encoding)
def search_multi(instance_urls: list, params: dict, method: str = "GET",
timeout: int = 15, retry_per: int = None,
auth_headers: dict = None, parallel: bool = True) -> dict:
auth_headers: dict = None, parallel: bool = True,
encoding: str = None) -> dict:
"""Search across multiple instances, failing over on error.
With parallel=True (default, multi-instance only): every instance is
@@ -589,6 +612,9 @@ def search_multi(instance_urls: list, params: dict, method: str = "GET",
With parallel=False (or a single instance): strictly sequential, one
request at a time, trying the next instance only after the current fails.
``encoding`` (v2.2.2) is forwarded to the HTML-fallback path for
non-UTF-8 instances; JSON responses are always UTF-8.
"""
if retry_per is None:
retry_per = MAX_RETRIES
@@ -604,7 +630,8 @@ def search_multi(instance_urls: list, params: dict, method: str = "GET",
try:
def _do():
return search_single(instance, params, method=method,
timeout=timeout, auth_headers=auth_headers)
timeout=timeout, auth_headers=auth_headers,
encoding=encoding)
result = _retry_with_backoff(_do, max_retries=retry_per)
emit_progress("instance_ok", url=instance,
latency=round(time.time() - start, 3),
@@ -627,7 +654,8 @@ def search_multi(instance_urls: list, params: dict, method: str = "GET",
start = time.time()
def _do():
return search_single(instance, params, method=method,
timeout=timeout, auth_headers=auth_headers)
timeout=timeout, auth_headers=auth_headers,
encoding=encoding)
try:
result = _retry_with_backoff(_do, max_retries=retry_per)
emit_progress("instance_ok", url=instance,
@@ -864,6 +892,8 @@ def fetch_page(url: str, timeout: int = 10, auth_headers: dict = None,
* ``fallback_used`` (str|None): 兜底方式（"wayback"），仅当走兜底时有值
"""
# 主抓取
# v2.3.0: 计时整条链路（主抓取 + Wayback 兜底），填充 latency 字段。
_start_time = time.monotonic()
result = None
error_msg = None
error_code = None
@@ -946,6 +976,9 @@ def fetch_page(url: str, timeout: int = 10, auth_headers: dict = None,
"text": "", "text_length": 0, "truncated": False,
"anti_bot_detected": False, "waf_type": None,
"fallback_used": None,
# v2.3.0: title/latency 字段（--fetch-report json 消费）
"title": None,
"latency": round(time.monotonic() - _start_time, 3),
}
# 反爬仍被检测到（Wayback 也无能为力或兜底被禁用）
@@ -957,6 +990,8 @@ def fetch_page(url: str, timeout: int = 10, auth_headers: dict = None,
"text": "", "text_length": 0, "truncated": False,
"anti_bot_detected": True, "waf_type": waf_type,
"fallback_used": fallback_used,
"title": _extract_title(content) if is_html else None,
"latency": round(time.monotonic() - _start_time, 3),
}
text = extract_text(content) if is_html else content
@@ -973,6 +1008,8 @@ def fetch_page(url: str, timeout: int = 10, auth_headers: dict = None,
"anti_bot_detected": False,
"waf_type": None,
"fallback_used": fallback_used,
"title": _extract_title(content) if is_html else None,
"latency": round(time.monotonic() - _start_time, 3),
}
@@ -1105,6 +1142,19 @@ _TITLE_ANTI_BOT_SIGNATURES = {
_TITLE_RE = re.compile(r"<title[^>]*>(.*?)</title>", re.IGNORECASE | re.DOTALL)
def _extract_title(content: str) -> str:
"""从 HTML 内容提取 <title> 文本（v2.3.0，供 --fetch-report json 使用）。
截断到 200 字符，防止异常页面标题撑爆报告。非 HTML 内容返回空串。
"""
if not content:
return ""
m = _TITLE_RE.search(content)
if m:
return m.group(1).strip()[:200]
return ""
def _detect_anti_bot(content: str) -> str:
"""检测反爬页面，返回 WAF 类型或 None。
@@ -1173,6 +1223,11 @@ class AdaptiveThrottle:
self._failure_threshold = failure_threshold
self._pause_seconds = pause_seconds
self._max_delay = max_delay
# v2.2.2：真实并发门控。计数信号量约束"瞬时在飞请求峰值"，
# _in_flight 结合当前 concurrency 判断是否应放行新请求——退避降
# 并发后，新请求会被快速拒绝（限流语义），而不是名义降并发。
self._semaphore = threading.BoundedSemaphore(initial_concurrency)
self._in_flight = 0
@property
def delay(self) -> float:
@@ -1228,6 +1283,36 @@ class AdaptiveThrottle:
logger.info(f" [THROTTLE] Global pause: waiting {remaining:.1f}s (429)")
time.sleep(remaining)
def acquire_slot(self, timeout: float = 0.05) -> bool:
"""获取一个并发执行槽位（v2.2.2 真实并发门控）。
线程池规模（ThreadPoolExecutor max_workers）创建时一次性固定，
无法随退避动态缩小。槽位机制在"请求真正发出前"做门控：
信号量约束瞬时峰值不超过初始并发；``_in_flight`` 结合当前
``concurrency`` 判断——退避降并发后，即使信号量有空位，只要在飞
请求数已 >= 当前并发目标，新请求也会被快速拒绝（返回 False），
由调用方跳过本次抓取，实现持久降并发。
返回 True 表示拿到槽位，调用方必须在 finally 中 release_slot()。
"""
if not self._semaphore.acquire(timeout=timeout):
return False
with self._lock:
if self._in_flight >= self._concurrency:
self._semaphore.release()
return False
self._in_flight += 1
return True
def release_slot(self) -> None:
"""释放并发槽位。必须与 acquire_slot 成对使用。"""
with self._lock:
self._in_flight = max(0, self._in_flight - 1)
try:
self._semaphore.release()
except ValueError:
pass
def stats(self) -> dict:
"""返回当前状态快照，供 --fetch-report 使用。"""
with self._lock:
@@ -1292,34 +1377,49 @@ def fetch_top_results(results: dict, count: int, timeout: int = 10,
"""Fetch one URL with adaptive throttling to avoid rate limiting."""
# 全局暂停检查（429 触发）
throttle.wait_if_paused()
# 自适应延迟
d = throttle.delay
if d > 0:
time.sleep(d * random.uniform(0.5, 1.5))
result = fetch_page(u, timeout=timeout, auth_headers=auth_headers,
max_retries=max_retries, max_size=max_size,
referer=referer, fallback_enabled=fallback_enabled)
if result["status"] == "ok":
ok_count[0] += 1
throttle.report_success()
trunc = ", TRUNCATED" if result.get("truncated") else ""
ua_note = ""
if result.get("user_agent_used") != USER_AGENT:
ua_note = " [fallback UA]"
fb_note = " [wayback]" if result.get("fallback_used") else ""
if fb_note:
fallback_count[0] += 1
logger.info(f" [OK] {u[:55]} ({result['text_length']:,} chars"
f"{trunc}{ua_note}{fb_note})")
else:
err_count[0] += 1
throttle.report_failure(result.get("error", ""),
error_code=result.get("error_code"))
# 统计反爬拦截
if result.get("anti_bot_detected"):
anti_bot_count[0] += 1
logger.error(f" [ERR] {u[:55]} ({result.get('error', 'unknown')})")
return result
# v2.2.2：真实并发门控。退避降并发后，超出当前并发目标的新请求
# 在这里被拒绝（不占用 fetch_page），实现持久降并发而非名义降并发。
if not throttle.acquire_slot():
logger.info(f" [THROTTLE] concurrency cap ({throttle.concurrency}) "
f"reached, skipping {u[:55]}")
return {"url": u, "status": "error",
"error": "Throttled: concurrency limit reached",
"error_code": E_RATE_LIMIT,
"text": "", "text_length": 0, "truncated": False,
"anti_bot_detected": False, "waf_type": None,
"fallback_used": None,
"title": None, "latency": None}
try:
# 自适应延迟
d = throttle.delay
if d > 0:
time.sleep(d * random.uniform(0.5, 1.5))
result = fetch_page(u, timeout=timeout, auth_headers=auth_headers,
max_retries=max_retries, max_size=max_size,
referer=referer, fallback_enabled=fallback_enabled)
if result["status"] == "ok":
ok_count[0] += 1
throttle.report_success()
trunc = ", TRUNCATED" if result.get("truncated") else ""
ua_note = ""
if result.get("user_agent_used") != USER_AGENT:
ua_note = " [fallback UA]"
fb_note = " [wayback]" if result.get("fallback_used") else ""
if fb_note:
fallback_count[0] += 1
logger.info(f" [OK] {u[:55]} ({result['text_length']:,} chars"
f"{trunc}{ua_note}{fb_note})")
else:
err_count[0] += 1
throttle.report_failure(result.get("error", ""),
error_code=result.get("error_code"))
# 统计反爬拦截
if result.get("anti_bot_detected"):
anti_bot_count[0] += 1
logger.error(f" [ERR] {u[:55]} ({result.get('error', 'unknown')})")
return result
finally:
throttle.release_slot()
with ThreadPoolExecutor(max_workers=min(throttle.concurrency, len(urls))) as ex:
future_map = {ex.submit(_fetch_one, u): u for u in urls}
@@ -1333,7 +1433,8 @@ def fetch_top_results(results: dict, count: int, timeout: int = 10,
"error_code": classify_error(e),
"text": "", "text_length": 0, "truncated": False,
"anti_bot_detected": False, "waf_type": None,
"fallback_used": None})
"fallback_used": None,
"title": None, "latency": None})
logger.error(f" [ERR] {u[:55]} (thread error: {e})")
# Reorder to match original result order
@@ -1365,8 +1466,7 @@ def _emit_fetch_report(fetched: list, throttle: "AdaptiveThrottle",
def _emit_fetch_report_text(fetched: list, throttle: "AdaptiveThrottle") -> None:
"""文本表格 + JSON 摘要行（v2.0.0 原始行为，向后兼容）。"""
import sys as _sys
out = _sys.stderr
out = sys.stderr
lines = []
lines.append("\n" + "=" * 72)
lines.append("FETCH REPORT (v2.0.0)")
@@ -1403,13 +1503,12 @@ def _emit_fetch_report_text(fetched: list, throttle: "AdaptiveThrottle") -> None
f"consec_ok={s['consecutive_successes']}")
# JSON 摘要（一行，便于 Agent 解析）
import json as _json
summary = {
"total": total, "ok": ok, "error": err,
"anti_bot_blocked": anti_bot, "wayback_recovered": wayback,
"throttle": s,
}
lines.append("JSON: " + _json.dumps(summary, ensure_ascii=False))
lines.append("JSON: " + json.dumps(summary, ensure_ascii=False))
lines.append("=" * 72 + "\n")
print("\n".join(lines), file=out)
@@ -1418,11 +1517,9 @@ def _emit_fetch_report_json(fetched: list, throttle: "AdaptiveThrottle") -> None
"""完整 JSON 报告（items 数组 + summary），输出到 stderr。
每个 URL 一个对象，包含 url / status / title / content_length /
error / error_code / latency / fetched_at 等字段。fetch_page 未采集
的字段（title / latency / fetched_at）为 None，便于 Agent 统一解析。
error / error_code / latency / fetched_at 等字段。v2.3.0 起
``title`` 与 ``latency`` 由 fetch_page 采集填充（此前恒为 None）。
"""
import json as _json
import sys as _sys
from datetime import datetime, timezone
fetched_at = datetime.now(timezone.utc).isoformat()
@@ -1434,12 +1531,12 @@ def _emit_fetch_report_json(fetched: list, throttle: "AdaptiveThrottle") -> None
"url": f.get("url", ""),
"final_url": f.get("final_url"),
"status": f.get("status", ""),
"title": f.get("title"), # fetch_page 未提取，None
"title": f.get("title"), # v2.3.0: fetch_page 已采集
"content_length": f.get("text_length", 0),
"content_type": f.get("content_type"),
"error": f.get("error"),
"error_code": f.get("error_code"),
"latency": f.get("latency"), # fetch_page 未计时，None
"latency": f.get("latency"), # v2.3.0: fetch_page 已计时
"truncated": f.get("truncated", False),
"fetched_at": f.get("fetched_at") or fetched_at,
"waf_type": f.get("waf_type"),
@@ -1464,7 +1561,7 @@ def _emit_fetch_report_json(fetched: list, throttle: "AdaptiveThrottle") -> None
"throttle": throttle.stats(),
"items": items,
}
print(_json.dumps(report, ensure_ascii=False), file=_sys.stderr)
print(json.dumps(report, ensure_ascii=False), file=sys.stderr)
# ----- Output formatting -----
@@ -1694,10 +1791,8 @@ def _format_results(results: dict, args) -> str:
if args.format == "urls":
return format_urls(results)
if args.format == "csv":
import csv as csv_mod
import io
out = io.StringIO()
writer = csv_mod.writer(out, lineterminator="\n")
writer = csv.writer(out, lineterminator="\n")
writer.writerow(["title", "url", "engine", "score",
"published_date", "content"])
for r in results.get("results", []):
@@ -1803,6 +1898,13 @@ def _run_single_query(query: str, args, instance_urls: list,
emit_progress("start", query=query, instances=len(instance_urls))
# v2.2.2：区分"实时查询"与"缓存命中"。原实现用循环末次赋值的
# ``cached`` 变量判断是否实时查询——多页路径下该变量保存的是最后一页
# 的状态，导致：最后一页命中缓存但前页实时查询时，unresponsive_engines
# 警告被错误跳过；反之仅最后一页未命中时误触发。用独立布尔标记精确
# 跟踪"本次运行是否发起了至少一次实时查询"。
performed_live_query = False
# v2.2.0：--pages N 多页聚合。循环 pageno=1..N，每页独立缓存
# （cache key 含 pageno），合并后统一 dedup/sort/max-results。
if pages_to_fetch == 1:
@@ -1821,11 +1923,13 @@ def _run_single_query(query: str, args, instance_urls: list,
retry_per=args.retry,
auth_headers=auth_headers,
parallel=not args.serial,
encoding=getattr(args, "encoding", None),
)
except Exception as e:
err_code = classify_error(e)
emit_progress("error", error=str(e), error_code=err_code, query=query)
return None, str(e), err_code
performed_live_query = True
if ttl_seconds > 0:
cache_module.put(params, results, ttl_seconds)
logger.info(f"[cache stored] q={query!r} TTL={args.cache_ttl}min")
@@ -1855,6 +1959,7 @@ def _run_single_query(query: str, args, instance_urls: list,
retry_per=args.retry,
auth_headers=auth_headers,
parallel=not args.serial,
encoding=getattr(args, "encoding", None),
)
except Exception as e:
page_errors += 1
@@ -1862,6 +1967,7 @@ def _run_single_query(query: str, args, instance_urls: list,
emit_progress("page_fail", query=query, page=page_no,
error=str(e), error_code=classify_error(e))
continue
performed_live_query = True
if ttl_seconds > 0:
cache_module.put(page_params, page_results, ttl_seconds)
emit_progress("cache_store", query=query, ttl=args.cache_ttl, page=page_no)
@@ -1886,7 +1992,7 @@ def _run_single_query(query: str, args, instance_urls: list,
# v2.1.1：检测实例侧引擎挂起/限流（仅在实时查询后提示，cache hit 时
# unresponsive_engines 信息可能已过期）
if cached is None:
if performed_live_query:
_warn_unresponsive_engines(results, query,
result_count=len(results.get("results", [])))
@@ -1975,14 +2081,54 @@ def _get_output_schema():
Used by ``--dump-schema`` so AI agents can programmatically discover the
output structure without parsing prose documentation.
v2.3.0: fetched.items 字段补全（与 fetch_page 实际输出对齐——
final_url/status/error_code/waf_type/fallback_used/title/latency 等），
并新增 ``batch`` 与 ``research`` 两个属性描述对应模式的输出 shape。
顶层 ``properties`` 仍以单查询为主，batch/research 为单独子树。
"""
return {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "SearXNG CLI Search Result",
"schema_version": SCHEMA_VERSION,
"description": "Output schema for 'python search.py --format json' (single query). "
"Batch mode (--queries-file) wraps results in "
'{"schema_version, queries:[]}.',
# 单查询结果条目（results[] 的元素）——batch/research 复用
result_item = {
"type": "object",
"properties": {
"title": {"type": "string"},
"url": {"type": "string", "format": "uri"},
"engine": {"type": "string", "description": "Source engine name."},
"score": {"type": ["number", "null"]},
"published_date": {"type": ["string", "null"]},
"content": {"type": "string", "description": "Snippet/summary text."},
},
"required": ["title", "url"],
}
# --fetch N 时的 fetched[] 元素（与 search.fetch_page 返回对齐）
fetched_item = {
"type": "object",
"properties": {
"url": {"type": "string"},
"final_url": {"type": ["string", "null"],
"description": "URL after redirects / Wayback."},
"status": {"type": "string", "enum": ["ok", "error"]},
"content_type": {"type": ["string", "null"]},
"text": {"type": "string", "description": "Extracted page text."},
"text_length": {"type": "integer"},
"truncated": {"type": "boolean"},
"title": {"type": ["string", "null"],
"description": "Page <title>, when available."},
"latency": {"type": ["number", "null"],
"description": "Fetch latency in seconds."},
"user_agent_used": {"type": ["string", "null"]},
"error": {"type": ["string", "null"]},
"error_code": {"type": ["string", "null"]},
"waf_type": {"type": ["string", "null"]},
"fallback_used": {"type": ["string", "null"],
"description": "'wayback' when the Wayback "
"Machine recovered the page."},
"anti_bot_detected": {"type": "boolean"},
},
"required": ["url", "status"],
}
# 单查询输出
single_query = {
"type": "object",
"properties": {
"schema_version": {
@@ -1999,18 +2145,7 @@ def _get_output_schema():
"results": {
"type": "array",
"description": "Search result items, ordered by relevance (score desc).",
"items": {
"type": "object",
"properties": {
"title": {"type": "string"},
"url": {"type": "string", "format": "uri"},
"engine": {"type": "string", "description": "Source engine name."},
"score": {"type": ["number", "null"]},
"published_date": {"type": ["string", "null"]},
"content": {"type": "string", "description": "Snippet/summary text."},
},
"required": ["title", "url"],
},
"items": result_item,
},
"unresponsive_engines": {
"type": "array",
@@ -2022,19 +2157,16 @@ def _get_output_schema():
"items": {"type": "string"},
"description": "Related query suggestions from the instance.",
},
"answers": {
"type": "array",
"items": {"type": "string"},
"description": "Direct answers from the instance.",
},
"fetched": {
"type": "array",
"description": "Present only when --fetch N is used. Page content "
"for the top N results.",
"items": {
"type": "object",
"properties": {
"url": {"type": "string"},
"text": {"type": "string"},
"text_length": {"type": "integer"},
"error": {"type": "string"},
},
},
"items": fetched_item,
},
"fetched_source": {
"type": "string",
@@ -2042,9 +2174,100 @@ def _get_output_schema():
"description": "Present only when --fetch is used. Indicates whether "
"search results came from JSON API or HTML fallback.",
},
"pages_fetched": {
"type": ["integer", "null"],
"description": "Present only when --pages N > 1. Pages that "
"succeeded (before cross-page merge).",
},
},
"required": ["query", "results"],
}
# --queries-file 批量输出
batch_schema = {
"type": "object",
"description": "Batch mode (--queries-file) output shape.",
"properties": {
"schema_version": {"type": "string", "const": SCHEMA_VERSION},
"queries": {
"type": "array",
"description": "One entry per query, in file order. "
"Entry: {query, status: ok|error, results} on "
"success or {query, status: error, error, "
"error_code} on failure.",
"items": {"type": "object",
"required": ["query", "status"],
"properties": {
"query": {"type": "string"},
"status": {"type": "string", "enum": ["ok", "error"]},
"results": single_query,
"error": {"type": "string"},
"error_code": {"type": "string"},
}},
},
},
"required": ["schema_version", "queries"],
}
# --research 研究模式输出
research_schema = {
"type": "object",
"description": "Research mode (--research TOPIC) output shape.",
"properties": {
"schema_version": {"type": "string", "const": SCHEMA_VERSION},
"research_topic": {"type": "string"},
"research_queries": {
"type": "array",
"description": "Expanded per-angle queries (deterministic rules).",
"items": {"type": "object",
"properties": {
"angle": {"type": "string"},
"query": {"type": "string"},
}},
},
"queries": {"type": "array",
"description": "Per-angle results. Entry: {query, angle, "
"status: ok|error, results|error, error_code}.",
"items": {"type": "object",
"properties": {
"query": {"type": "string"},
"angle": {"type": "string"},
"status": {"type": "string",
"enum": ["ok", "error"]},
"results": single_query,
"error": {"type": "string"},
"error_code": {"type": "string"},
}}},
"merged_results": {
"type": "object",
"description": "Cross-angle merged + deduplicated result set "
"(same shape as single-query 'results').",
"properties": {
"query": {"type": "string"},
"results": {"type": "array", "items": result_item},
},
},
},
"required": ["schema_version", "research_topic", "queries",
"merged_results"],
}
return {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"title": "SearXNG CLI Search Result",
"schema_version": SCHEMA_VERSION,
"description": "Output schema for 'python search.py --format json'. "
"Top-level 'properties' describe single-query output; "
"the 'batch' and 'research' subtrees describe "
"--queries-file and --research output respectively. "
"v2.3.0: fetched.items fields now match fetch_page "
"output (title/latency/waf_type/error_code etc.).",
"type": "object",
"properties": single_query["properties"],
"required": single_query["required"],
"defs": {
"single_query": single_query,
"batch": batch_schema,
"research": research_schema,
},
}
def _emit_error(message: str, args, query: str = None, exit_code: int = 1,
@@ -2087,13 +2310,30 @@ def _emit_error(message: str, args, query: str = None, exit_code: int = 1,
def _read_queries_file(path: str) -> list:
"""Read queries from a file: one per line, skip blanks and ``#`` comments.
v2.3.0: 编码自动检测。先按 UTF-8 读取；若解码失败（Windows 下 GBK 等
非 UTF-8 文件常见），回退 GBK，再失败回退 utf-8 errors=replace——
绝不因编码问题让批量任务整体失败。
Raises :class:`RuntimeError` if the file cannot be read, so the caller
can route it through :func:`_emit_error`.
"""
raw = None
try:
text = Path(path).read_text(encoding="utf-8")
raw = Path(path).read_bytes()
except OSError as e:
raise RuntimeError(f"cannot read queries file '{path}': {e}")
text = None
for enc in ("utf-8", "gbk"):
try:
text = raw.decode(enc)
break
except (UnicodeDecodeError, LookupError):
continue
if text is None:
# 最后兜底：UTF-8 + 替换符，保证任务可继续
text = raw.decode("utf-8", errors="replace")
queries = []
for line in text.splitlines():
line = line.strip()
@@ -2260,7 +2500,7 @@ def expand_research_queries(topic: str, custom_angles: list = None) -> list:
# ----- Mode handlers (v2.2.0: 从 main() 提取，降低单函数复杂度) -----
# main() 只负责参数解析和分发，四条执行路径各自独立函数，便于维护和测试。
# 纯提取重构，行为与 v2.1.1 完全一致，539 测试兜底验证。
# 纯提取重构，行为与 v2.1.1 完全一致，544 测试兜底验证。
def _handle_verify(args, instance_urls: list, auth_headers: dict) -> None:
@@ -2340,10 +2580,8 @@ def _handle_research(args, instance_urls: list, auth_headers: dict,
"merged_results": merged,
}, indent=2, ensure_ascii=False)
elif args.format == "csv":
import csv as csv_mod
import io
out = io.StringIO()
writer = csv_mod.writer(out, lineterminator="\n")
writer = csv.writer(out, lineterminator="\n")
writer.writerow(["angle", "query", "title", "url", "engine",
"score", "published_date", "content"])
for br in batch:
@@ -2412,11 +2650,33 @@ def _handle_research(args, instance_urls: list, auth_headers: dict,
sys.exit(0)
def _run_single_query_wrapper(query: str, args, instance_urls: list,
auth_headers: dict, ttl_seconds: int):
"""并发批量用的 _run_single_query 包装（v2.3.0）。
移除不可跨线程共享的参数：
* ``args.fetch`` 置 0 —— --fetch 的 ThreadPoolExecutor 在查询线程内
创建，worker 再经 ThreadPoolExecutor 二次并发会超过线程安全上限；
并发批量模式用 ``--queries-file`` 不适合内嵌抓取。
* ``args.output`` 置 None —— 输出写入统一交给 _handle_batch。
其余参数原样透传（行为与串行路径一致）。
"""
import copy as _copy
qargs = _copy.copy(args)
qargs.fetch = 0
qargs.output = None
return _run_single_query(query, qargs, instance_urls, auth_headers,
ttl_seconds)
def _handle_batch(args, instance_urls: list, auth_headers: dict,
ttl_seconds: int) -> None:
"""批量模式：从文件读取多个查询，串行执行，输出合并结果。
"""批量模式：从文件读取多个查询，串行或并发执行，输出合并结果。
v2.2.0 从 main() 提取为独立函数（纯重构，行为不变）。
v2.3.0 新增 --parallel-queries N：并发执行（受 AdaptiveThrottle
约束），输出保持文件顺序；并发模式下 --fetch 被禁用（见
_run_single_query_wrapper），日志压缩为每查询一行。
"""
try:
queries = _read_queries_file(args.queries_file)
@@ -2426,34 +2686,89 @@ def _handle_batch(args, instance_urls: list, auth_headers: dict,
_emit_error(f"no queries found in '{args.queries_file}'", args,
error_code=E_INPUT)
logger.info(f"Running {len(queries)} queries from {args.queries_file}...")
batch = []
any_with_results = False
error_count = 0
for i, q in enumerate(queries, 1):
logger.info(f"\n[{i}/{len(queries)}] {q}")
results, err, err_code = _run_single_query(q, args, instance_urls,
auth_headers, ttl_seconds)
if err:
error_count += 1
logger.error(f" [ERROR] {err}")
entry = {"query": q, "status": "error", "error": err}
if err_code:
entry["error_code"] = err_code
batch.append(entry)
else:
if len(results.get("results", [])) > 0:
any_with_results = True
batch.append({"query": q, "status": "ok", "results": results})
parallel = getattr(args, "parallel_queries", 0) or 0
if parallel < 1:
parallel = 1
if parallel > 8:
parallel = 8
if parallel > 1:
logger.info(f"Running {len(queries)} queries from {args.queries_file} "
f"in parallel ({parallel} workers)...")
batch = [None] * len(queries)
any_with_results = [False]
error_count = [0]
def _run(idx_q):
idx, q = idx_q
logger.info(f"[{idx+1}/{len(queries)}] {q}")
results, err, err_code = _run_single_query_wrapper(
q, args, instance_urls, auth_headers, ttl_seconds)
return idx, q, results, err, err_code
throttle = AdaptiveThrottle(0.0, parallel)
def _job(idx_q):
# 全局暂停（429）+ 并发槽位门控（退避降并发时快速拒绝）
throttle.wait_if_paused()
if not throttle.acquire_slot():
logger.info(f" [THROTTLE] concurrency cap ({throttle.concurrency}) "
f"reached, skipping query {idx_q[0]+1}")
return idx_q[0], idx_q[1], None, "Throttled: concurrency limit", E_RATE_LIMIT
try:
return _run(idx_q)
finally:
throttle.release_slot()
with ThreadPoolExecutor(max_workers=parallel) as ex:
futures = [ex.submit(_job, item) for item in enumerate(queries)]
for fut in as_completed(futures):
idx, q, results, err, err_code = fut.result()
if err:
error_count[0] += 1
entry = {"query": q, "status": "error", "error": err}
if err_code:
entry["error_code"] = err_code
batch[idx] = entry
else:
if len(results.get("results", [])) > 0:
any_with_results[0] = True
batch[idx] = {"query": q, "status": "ok", "results": results}
# 日志：并发模式下错误信息以单行输出（串行模式按序打印多行）
if error_count[0]:
logger.warning(f"Parallel batch: {error_count[0]} errors, "
f"{len(queries) - error_count[0]} ok")
error_count = error_count[0]
any_with_results = any_with_results[0]
else:
logger.info(f"Running {len(queries)} queries from {args.queries_file}...")
batch = []
any_with_results = False
error_count = 0
for i, q in enumerate(queries, 1):
logger.info(f"\n[{i}/{len(queries)}] {q}")
results, err, err_code = _run_single_query(q, args, instance_urls,
auth_headers, ttl_seconds)
if err:
error_count += 1
logger.error(f" [ERROR] {err}")
entry = {"query": q, "status": "error", "error": err}
if err_code:
entry["error_code"] = err_code
batch.append(entry)
else:
if len(results.get("results", [])) > 0:
any_with_results = True
batch.append({"query": q, "status": "ok", "results": results})
if args.format == "json":
output = json.dumps({"schema_version": SCHEMA_VERSION, "queries": batch},
indent=2, ensure_ascii=False)
elif args.format == "csv":
import csv as csv_mod
import io
out = io.StringIO()
writer = csv_mod.writer(out, lineterminator="\n")
writer = csv.writer(out, lineterminator="\n")
writer.writerow(["query", "title", "url", "engine", "score",
"published_date", "content"])
for br in batch:
@@ -2627,6 +2942,12 @@ Use --config FILE to load a non-default config file (overrides the auto-discover
help="Comma-separated categories (e.g. general,images,news)")
parser.add_argument("--language", "-l", default=config.get("language"),
help="Language code (e.g. en, zh-CN, de)")
parser.add_argument("--encoding", default=config.get("encoding"),
help="Force charset for HTML-fallback decoding "
"(e.g. gbk, shift_jis). v2.2.2: auto-detected "
"from the HTTP header / HTML meta when omitted; "
"this flag overrides auto-detection for "
"misconfigured instances.")
parser.add_argument("--pageno", "-p", type=int, default=1,
help="Page number (default: 1)")
parser.add_argument("--pages", type=int, default=1,
@@ -2769,6 +3090,13 @@ Use --config FILE to load a non-default config file (overrides the auto-discover
"starting with '#' are skipped) and run them in sequence. "
"Results are emitted as a JSON array (or one brief block per query). "
"Overrides --query when set.")
parser.add_argument("--parallel-queries", type=int, default=0, metavar="N",
help="v2.3.0: run batch queries concurrently with N workers "
"(1-8, capped at 8; default 0 = sequential). Output order "
"is preserved. Concurrency is gated by AdaptiveThrottle — "
"on repeated failures workers are throttled, not spammed. "
"When enabled, --fetch is disabled (nested parallel fetch "
"is unsafe) and per-query logs are compressed.")
parser.add_argument("--research", default=None, metavar="TOPIC",
help="v2.1.0 Research mode: given a topic, auto-expand into 5 "
"multi-angle queries (overview/profile/background/works/review) "
@@ -2803,8 +3131,7 @@ Use --config FILE to load a non-default config file (overrides the auto-discover
# v2.2.0：生成 request_id 并重新配置 logging（带 log_format + request_id）
# request_id 贯穿所有日志、进度事件和错误输出，便于 batch 模式追溯。
import os as _os
request_id = _os.urandom(4).hex()
request_id = os.urandom(4).hex()
setup_logging(verbose=args.verbose, quiet=args.quiet,
log_format=getattr(args, "log_format", "text"),
request_id=request_id)