thzxx
4df521dc9d
迭代 1 — 正确性修复:
- 修复 --pages N 多页聚合的 unresponsive-engine 警告误判: 原用循环末次
cached 变量判断, 缓存命中时警告被错误跳过/误触发; 改用独立
performed_live_query 标记
- UA 池单一来源: 删除 common.py 手工副本 _FALLBACK_UAS_BUILTIN,
FALLBACK_UAS 直接引用 _config.UA_POOL, 消除双份漂移
- search_html 解码修复: 硬编码 utf-8 改为 detect_charset(header/meta
自动检测), 新增 --encoding 强制覆盖, 贯穿 search_multi 全链
- AdaptiveThrottle 真实并发门控: acquire_slot()/release_slot() 槽位机制,
退避降并发后新请求被快速拒绝(E_RATE_LIMIT), 实现持久降并发而非名义降并发
迭代 2 — fetch JSON 契约 + 批量并发:
- fetch.py --format json: 成功 {status,url,final_url,content_type,extract,
truncated,text_length,user_agent}; 失败 {status,error,error_code,
status_code,url}, 对齐 search.py 错误码体系
- fetch_page 采集 title + latency, 填充 --fetch-report json 空字段
- --queries-file --parallel-queries N (1-8): 并发批量, 输出保序, 受
AdaptiveThrottle 门控; 并发模式禁用 --fetch(嵌套并行不安全)
- queries 文件编码自动检测 (UTF-8 → GBK 回退)
迭代 3 — 工程化:
- 新增 pyproject.toml (searxng-search/searxng-fetch 入口点)
- 收敛 20+ 处函数内冗余导入
- --dump-schema 扩展: fetched.items 补全 15 字段, 新增 defs.batch/research
- 新增 17 个测试 (tests/test_v230_features.py), 全量 561 测试通过
- 文档同步 (SKILL.md/README.md, 版本号 2.3.0)
63 lines
3.2 KiB
Python
63 lines
3.2 KiB
Python
"""Package-level constants for searxng-cli scripts.
|
||
|
||
Import in sibling scripts with:
|
||
from _config import VERSION, USER_AGENT, SCHEMA_VERSION, UA_POOL
|
||
|
||
Retry settings and shared HTTP utilities now live in ``common.py`` so that
|
||
both ``search.py`` and ``fetch.py`` share one consistent implementation.
|
||
"""
|
||
|
||
VERSION = "2.3.0"
|
||
SCHEMA_VERSION = "1.0"
|
||
USER_AGENT = f"searxng-cli/{VERSION}"
|
||
|
||
# 浏览器 UA 池（用于 searxng-cli 默认 UA 被站点拦截时的回退）。
|
||
#
|
||
# v2.2.0 从 common.py 迁移至此统一管理（SSOT）。common.py 通过
|
||
# ``from _config import UA_POOL`` 引用，并在导入失败时回退到其内置副本。
|
||
#
|
||
# 维护原则：
|
||
# 1. 版本号保持为当前年份的主流浏览器版本，避免被识别为过时浏览器
|
||
# 2. 顺序固定——get_ua_for_domain() 用 SHA-256 哈希选索引，顺序变化会
|
||
# 改变域名→UA 的映射，导致跨版本缓存失效（可接受，但应尽量避免无谓变动）
|
||
# 3. 至少覆盖 Chrome/Edge/Firefox × Windows/macOS/Linux，保证指纹多样性
|
||
#
|
||
# 当前版本（2026 年）：Chrome 138-140 / Edge 138 / Firefox 140 / Safari 18。
|
||
UA_POOL = [
|
||
# Chrome 140 — Windows / macOS / Linux
|
||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||
"(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36",
|
||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
|
||
"(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36",
|
||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
|
||
"(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36",
|
||
# Chrome 139 — Windows / macOS / Linux
|
||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||
"(KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36",
|
||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
|
||
"(KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36",
|
||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
|
||
"(KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36",
|
||
# Chrome 138 — Windows / macOS / Linux
|
||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||
"(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
|
||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
|
||
"(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
|
||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
|
||
"(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
|
||
# Edge 138 — Windows / macOS
|
||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||
"(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 Edg/138.0.0.0",
|
||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
|
||
"(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 Edg/138.0.0.0",
|
||
# Firefox 140 — Windows / macOS / Linux
|
||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:140.0) "
|
||
"Gecko/20100101 Firefox/140.0",
|
||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:140.0) "
|
||
"Gecko/20100101 Firefox/140.0",
|
||
"Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0",
|
||
# Safari 18 — macOS（WebKit 指纹，应对 Chromium 针对性拦截）
|
||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 "
|
||
"(KHTML, like Gecko) Version/18.0 Safari/605.1.15",
|
||
]
|