MetonaTeam/searxng-use-cli
2
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(v2.3.0): fetch 结构化 JSON 契约 + 并发批量 + 真实并发门控

Browse Source 
迭代 1 — 正确性修复:
- 修复 --pages N 多页聚合的 unresponsive-engine 警告误判: 原用循环末次
  cached 变量判断, 缓存命中时警告被错误跳过/误触发; 改用独立
  performed_live_query 标记
- UA 池单一来源: 删除 common.py 手工副本 _FALLBACK_UAS_BUILTIN,
  FALLBACK_UAS 直接引用 _config.UA_POOL, 消除双份漂移
- search_html 解码修复: 硬编码 utf-8 改为 detect_charset(header/meta
  自动检测), 新增 --encoding 强制覆盖, 贯穿 search_multi 全链
- AdaptiveThrottle 真实并发门控: acquire_slot()/release_slot() 槽位机制,
  退避降并发后新请求被快速拒绝(E_RATE_LIMIT), 实现持久降并发而非名义降并发

迭代 2 — fetch JSON 契约 + 批量并发:
- fetch.py --format json: 成功 {status,url,final_url,content_type,extract,
  truncated,text_length,user_agent}; 失败 {status,error,error_code,
  status_code,url}, 对齐 search.py 错误码体系
- fetch_page 采集 title + latency, 填充 --fetch-report json 空字段
- --queries-file --parallel-queries N (1-8): 并发批量, 输出保序, 受
  AdaptiveThrottle 门控; 并发模式禁用 --fetch(嵌套并行不安全)
- queries 文件编码自动检测 (UTF-8 → GBK 回退)

迭代 3 — 工程化:
- 新增 pyproject.toml (searxng-search/searxng-fetch 入口点)
- 收敛 20+ 处函数内冗余导入
- --dump-schema 扩展: fetched.items 补全 15 字段, 新增 defs.batch/research
- 新增 17 个测试 (tests/test_v230_features.py), 全量 561 测试通过
- 文档同步 (SKILL.md/README.md, 版本号 2.3.0)
This commit is contained in:
thzxx
2026-08-05 20:14:07 +08:00
parent 6cedba9042
commit 4df521dc9d
8 changed files with 1006 additions and 226 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/fetch.py
+104 -21
View File
@@ -11,6 +11,7 @@ for improved extraction quality (optional, falls back to stdlib).
import argparse
import gzip
import io
import json
import logging
import random
import re
@@ -53,6 +54,7 @@ from common import (
build_auth_headers,
build_browser_headers,
build_wayback_url,
classify_error,
compute_backoff_delay,
detect_charset,
force_utf8_stdout,
@@ -1134,6 +1136,55 @@ def fetch_url(url: str, timeout=15, user_agent: str = None,
# ----- Main -----
def _emit_fetch_result(args, output: str, url: str, final_url: str,
content_type: str, truncated: bool,
user_agent: str = None,
error: str = None, error_code: str = None,
status_code: int = None) -> None:
"""输出抓取结果到 stdout / --output 文件。
v2.3.0: ``--format json`` 提供结构化 JSON 契约，AI Agent 可程序化
解析（成功与失败统一为 {status, url, ...}）。``--format text``（默认）
保持 v2.2.x 行为：成功输出正文，失败输出空 + stderr 日志。
成功 shape::
{"status": "ok", "url", "final_url", "content_type",
"extract", "truncated", "text_length", "user_agent"}
失败 shape::
{"status": "error", "url", "error", "error_code", "status_code"}
"""
if args.format == "json":
if error:
payload = {"status": "error", "url": url, "error": error}
if error_code:
payload["error_code"] = error_code
if status_code is not None:
payload["status_code"] = status_code
else:
payload = {
"status": "ok",
"url": url,
"final_url": final_url,
"content_type": content_type,
"extract": args.extract,
"truncated": truncated,
"text_length": len(output),
"user_agent": user_agent,
}
text = json.dumps(payload, indent=2, ensure_ascii=False)
else:
text = output
if args.output:
with open(args.output, "w", encoding="utf-8") as f:
f.write(text)
logger.info(f"Saved {len(text)} chars to {args.output}")
else:
print(text)
def main():
parser = argparse.ArgumentParser(
description="Fetch a web page and extract readable content",
@@ -1146,11 +1197,18 @@ Examples:
%(prog)s -u https://example.com -e markdown markdown conversion
%(prog)s -u https://example.com -o page.txt save to file
%(prog)s -u https://example.cn -e text --encoding gbk force charset
%(prog)s -u https://example.com --format json structured JSON output
""",
)
parser.add_argument("--url", "-u", required=True, help="URL to fetch")
parser.add_argument("--extract", "-e", choices=["text", "html", "markdown"],
default="text", help="Extraction mode (default: text)")
parser.add_argument("--format", "-f", choices=["text", "json"], default="text",
help="Output format (default: text). 'json' emits a structured "
"JSON object {status, url, final_url, content_type, extract, "
"truncated, text_length, user_agent} on success, or "
"{status: error, error, error_code, status_code, url} on "
"failure — machine-readable for agents. v2.3.0.")
parser.add_argument("--timeout", "-t", type=int, default=15,
help="Request timeout in seconds (default: 15)")
parser.add_argument("--retries", type=int, default=3,
@@ -1229,6 +1287,18 @@ Examples:
logger.info(f"Hard-blocked domain detected — Wayback fallback "
f"will be prioritized if main fetch fails")
# v2.3.0: 状态收集变量。所有失败路径设置 fatal_* 后落到统一输出
# （_emit_fetch_result），json 模式输出结构化错误到 stdout，text 模式
# 保持 v2.2.x 行为（stdout 空 + stderr 日志 + exit 1）。
content = None
final_url = args.url
content_type = ""
truncated = False
user_agent = None
fatal_error = None
fatal_error_code = None
fatal_status_code = None
try:
result = fetch_url(
args.url, timeout=args.timeout, user_agent=args.user_agent,
@@ -1237,17 +1307,17 @@ Examples:
allow_redirects=not args.no_redirect,
referer=args.referer,
)
content, content_type, final_url = (
result.content, result.content_type, result.final_url,
)
content = result.content
content_type = result.content_type or ""
final_url = result.final_url
truncated = result.truncated
user_agent = result.user_agent
# 文档解析失败（PDF/DOCX/XLSX 等）时 fetch_url 不抛异常，
# 而是返回带 error_code 的 FetchResult——必须显式检查，
# 否则失败会被静默吞掉（空输出 + exit 0）。
if result.error_code:
logger.error(
f"Error: {result.error_message or result.error_code} "
f"(error_code={result.error_code}, url={args.url})")
sys.exit(1)
fatal_error = result.error_message or result.error_code
fatal_error_code = result.error_code
except Exception as e:
# 诊断信息增强：从 __cause__ 链中提取 HTTP 状态码、原始异常类型，
# 让 AI Agent 能程序化判断失败原因（404 vs 403 vs DNS 失败等），
@@ -1275,7 +1345,10 @@ Examples:
# v2.1.0: Wayback Machine 兜底
# 触发条件：兜底启用 + (错误可恢复 OR 命中被墙站点)
error_msg = str(e) if str(e) else e.__class__.__name__
fatal_error = str(e) if str(e) else e.__class__.__name__
fatal_error_code = classify_error(e)
fatal_status_code = status_code
error_msg = fatal_error
if fallback_enabled and (should_try_wayback(error_msg) or hard_blocked):
wayback_url = build_wayback_url(args.url)
wb_timeout = min(args.timeout, 10) # Wayback 独立超时，不阻塞
@@ -1289,17 +1362,31 @@ Examples:
max_size=args.max_size,
allow_redirects=True,
)
content, content_type, final_url = (
wb_result.content, wb_result.content_type,
wb_result.final_url,
)
content = wb_result.content
content_type = wb_result.content_type or ""
final_url = wb_result.final_url
truncated = wb_result.truncated
user_agent = wb_result.user_agent
if wb_result.error_code:
fatal_error = (wb_result.error_message or wb_result.error_code)
fatal_error_code = wb_result.error_code
fatal_status_code = None
else:
fatal_error = None
fatal_error_code = None
fatal_status_code = None
logger.info(f"[FALLBACK] Wayback recovery successful "
f"({len(content)} chars)")
except Exception as wb_e:
logger.error(f"[FALLBACK] Wayback also failed: {wb_e}")
sys.exit(1)
else:
sys.exit(1)
fatal_error = f"{fatal_error} ; Wayback also failed: {wb_e}"
if fatal_error:
_emit_fetch_result(args, "", args.url, final_url, content_type,
truncated, user_agent, error=fatal_error,
error_code=fatal_error_code,
status_code=fatal_status_code)
sys.exit(1)
if final_url != args.url:
logger.info(f"Redirected to: {final_url}")
@@ -1320,12 +1407,8 @@ Examples:
logger.warning(f"Warning: extracted text is very short ({len(output.strip())} chars). "
"The page may be JS-heavy or use anti-bot protection.")
if args.output:
with open(args.output, "w", encoding="utf-8") as f:
f.write(output)
logger.info(f"Saved {len(output)} chars to {args.output}")
else:
print(output)
_emit_fetch_result(args, output, args.url, final_url, content_type,
truncated, user_agent)
if __name__ == "__main__":