thzxx
10c01a3abd
正确性修复: - 修复 Sec-Ch-Ua 构造 bug: 原实现产出 ""Not_A Brand";v="99"" 双重引号 畸形头(Chrome/Edge 两路径), 严格校验的 WAF 会忽略; 改为品牌数组拼接 - search 403 快速失败: 实例级 403 不再退避重试(~10.5s 空等), 立即 failover - AdaptiveThrottle: --throttle-failure-threshold 0 现为真正禁用语义 - number_of_results 缺失时用 len(results) 兜底(JSON/HTML 路径契约对齐) - 版本对齐: pyproject.toml 与 _config.py 同步 2.5.0 AI 代理体验: - fetch.py --extract json: 结构化骨架(title/meta/headings/links/images) - fetch.py --max-chars N: 提取后语义级截断(区别于 --max-size 字节截断) - search --fetch-total-chars N: --fetch 全局字符预算, 耗尽后 status=skipped - --dedup-fetched-content: 抓取正文 SimHash 去重, status=duplicate - --progress 新增 angle_start/ok/fail + fetch_skip/fetch_duplicate 事件 - fetch.py 补齐 --log-format json + --dump-schema - CSV 媒体列自适应(images/videos 类别自动追加媒体字段列) - --dry-run 批量模式打印实际查询列表 - search 连接复用: requests 可用时走模块级 Session(连接池) 工程治理: - 新增 scripts/release_check.py 发布一致性检查(版本/错误码表漂移) - 新增 tests/test_v250_features.py 46+4 个回归测试(全量 622 通过) - tests/conftest.py: autouse fixture 强制 stdlib 路径(本机有 requests 时 既有 urllib mock 测试不失效)
63 lines
3.2 KiB
Python
63 lines
3.2 KiB
Python
"""Package-level constants for searxng-cli scripts.
|
||
|
||
Import in sibling scripts with:
|
||
from _config import VERSION, USER_AGENT, SCHEMA_VERSION, UA_POOL
|
||
|
||
Retry settings and shared HTTP utilities now live in ``common.py`` so that
|
||
both ``search.py`` and ``fetch.py`` share one consistent implementation.
|
||
"""
|
||
|
||
VERSION = "2.5.0"
|
||
SCHEMA_VERSION = "1.0"
|
||
USER_AGENT = f"searxng-cli/{VERSION}"
|
||
|
||
# 浏览器 UA 池（用于 searxng-cli 默认 UA 被站点拦截时的回退）。
|
||
#
|
||
# v2.2.0 从 common.py 迁移至此统一管理（SSOT）。common.py 通过
|
||
# ``from _config import UA_POOL`` 引用，并在导入失败时回退到其内置副本。
|
||
#
|
||
# 维护原则：
|
||
# 1. 版本号保持为当前年份的主流浏览器版本，避免被识别为过时浏览器
|
||
# 2. 顺序固定——get_ua_for_domain() 用 SHA-256 哈希选索引，顺序变化会
|
||
# 改变域名→UA 的映射，导致跨版本缓存失效（可接受，但应尽量避免无谓变动）
|
||
# 3. 至少覆盖 Chrome/Edge/Firefox × Windows/macOS/Linux，保证指纹多样性
|
||
#
|
||
# 当前版本（2026 年）：Chrome 138-140 / Edge 138 / Firefox 140 / Safari 18。
|
||
UA_POOL = [
|
||
# Chrome 140 — Windows / macOS / Linux
|
||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||
"(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36",
|
||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
|
||
"(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36",
|
||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
|
||
"(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36",
|
||
# Chrome 139 — Windows / macOS / Linux
|
||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||
"(KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36",
|
||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
|
||
"(KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36",
|
||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
|
||
"(KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36",
|
||
# Chrome 138 — Windows / macOS / Linux
|
||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||
"(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
|
||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
|
||
"(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
|
||
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
|
||
"(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36",
|
||
# Edge 138 — Windows / macOS
|
||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||
"(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 Edg/138.0.0.0",
|
||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
|
||
"(KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 Edg/138.0.0.0",
|
||
# Firefox 140 — Windows / macOS / Linux
|
||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:140.0) "
|
||
"Gecko/20100101 Firefox/140.0",
|
||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:140.0) "
|
||
"Gecko/20100101 Firefox/140.0",
|
||
"Mozilla/5.0 (X11; Linux x86_64; rv:140.0) Gecko/20100101 Firefox/140.0",
|
||
# Safari 18 — macOS（WebKit 指纹，应对 Chromium 针对性拦截）
|
||
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 "
|
||
"(KHTML, like Gecko) Version/18.0 Safari/605.1.15",
|
||
]
|