fix(v2.1.1): 修复执行问题记录中的真实 bug + 文档对齐

源码修复(5 项):
1. search.py --time-range choices 加入 week(对齐 SearXNG API 四档)
2. fetch.py stdlib 路径处理 gzip/deflate 解压(被沙箱伪响应掩盖的真实 bug,
   无 requests 环境抓取压缩服务器会全页 U+FFFD 乱码)
3. search.py --research 模式实现跨角度合并去重,输出 merged_results 字段
   (兑现文档承诺 "Results are merged and deduplicated")
4. search.py fetch_page 返回 error_code 字段 + AdaptiveThrottle 用
   E_RATE_LIMIT 结构化检测 429(原字符串匹配 "429" 会漏判
   "Too Many Requests")
5. search.py _retry_with_backoff 复用 compute_backoff_delay(60s 封顶)
   + 处理 Retry-After header,与 fetch.py 保持一致

增强(3 项):
- common.py 精确化 baidu 子域列表(pan.baidu.com/cloud.baidu.com 不再误伤)
- search.py expand_research_queries 根据主题语言切换中英文后缀
- search.py 新增 _warn_unresponsive_engines,识别实例侧引擎挂起并提示

文档/版本:
- _config.py VERSION 2.1.0 → 2.1.1
- SKILL.md 同步更新(time-range week、merged_results、error_code、baidu 精确化)
- README.md 同步更新 + 测试数量 503 → 539

测试: 539 个全部通过,含 6 个新增验证测试
This commit is contained in:
2026-08-03 12:54:27 +08:00
parent e94cbe0783
commit 0c8fdc1e45
9 changed files with 257 additions and 41 deletions
+11 -8
View File
@@ -1,7 +1,7 @@
---
name: searxng-use-cli
description: Use when you need to search the web via your OWN SearXNG instance (no public-instance discovery). 3 CLI scripts + a shared common.py module — execute privacy-respecting searches against a user-supplied instance (with multi-instance failover, 5xx/429 retry, auto-fetch) or via SEARXNG_INSTANCE env / config file, fetch/extract readable text or markdown from web pages. Zero-config replacement for proprietary search APIs.
version: 2.1.0
version: 2.1.1
author: Metona Team
license: MIT
platforms: [linux, macos, windows]
@@ -73,16 +73,19 @@ SearXNG is a privacy-respecting metasearch engine that aggregates results from 7
- **Capped backoff** — `compute_backoff_delay()` caps at 60s (was uncapped: 1.5*2^10 = 1536s would hang the process)
- **WAF fingerprint library** — `_detect_anti_bot()` identifies Cloudflare / Imperva / PerimeterX / DataDome / Akamai / Anubis / generic challenges via `<title>` tag matching (most precise) + full-document scan of technical identifiers (cookie/header/JS variable names, not bare vendor names). v2.0.1 narrowed broad keywords (e.g. bare `cloudflare`/`captcha`/`challenge`) that caused false positives on normal articles, and added title-tag detection. Returns `waf_type` for AI-agent decisioning
- **Wayback Machine fallback** — 404/403/timeout AND anti-bot-blocked pages automatically retry via `https://web.archive.org/web/2/<url>` (latest snapshot). v2.0.1 fixed a bug where HTTP 200 anti-bot challenge pages (Cloudflare returns 200 for JS challenges) bypassed the fallback trigger. v2.1.0: `fetch.py` standalone calls now also get Wayback fallback (was only in `search.py --fetch`). Default ON; `--no-fallback` disables. Independent 10s timeout so Wayback slowness never blocks the main flow
- **Hard-blocked domain fallback** (v2.1.0) — `is_hard_blocked_domain()` detects known strong-anti-bot sites (baike.baidu.com, zhihu.com, weibo.com, mp.weixin.qq.com, douban.com, etc.) that almost always return 403 regardless of UA. These sites bypass the normal `should_try_wayback()` check and trigger Wayback immediately on failure. List maintained in `common.py` `HARD_BLOCKED_DOMAINS`
- **Hard-blocked domain fallback** (v2.1.0, refined in v2.1.1) — `is_hard_blocked_domain()` detects known strong-anti-bot sites (www.baidu.com, baike.baidu.com, zhihu.com, weibo.com, mp.weixin.qq.com, douban.com, etc.) that almost always return 403 regardless of UA. These sites bypass the normal `should_try_wayback()` check and trigger Wayback immediately on failure. v2.1.1 refined the list: moved `baidu.com` from broad subdomain matching to a precise per-subdomain list (www/baike/zhidao/tieba/wenku), so `pan.baidu.com` (netdisk) and `cloud.baidu.com` (cloud) are no longer false-positively blocked. List maintained in `common.py` `HARD_BLOCKED_DOMAINS`
- **Adaptive throttling** — `AdaptiveThrottle` state machine: 3 consecutive failures → double delay + halve concurrency; 5 consecutive successes → gradual recovery; 429 → global pause 30s. Thread-safe
- **readability-lite extraction** — when `<article>`/`<main>`/content-class `<div>` are all missing, `_readability_lite()` picks the highest text-density node (text chars / tag count + `<p>` weighting), avoiding nav/sidebar/footer noise
- **`--fetch-report`** — structured per-URL report to stderr after `--fetch N`: status, WAF type, fallback used, char count, plus adaptive throttle stats and a JSON summary line
- **`--referer`** — set Referer header for fetch requests (defaults to the instance URL when fetching result pages, disguising traffic source)
- **`--request-delay`** — configurable delay between fetch requests (default 0.3s; adaptive throttling may increase this on failures)
- New fetch result fields: `anti_bot_detected` (bool), `waf_type` (str|null), `fallback_used` (str|null)
- New fetch result fields: `anti_bot_detected` (bool), `waf_type` (str|null), `fallback_used` (str|null), `error_code` (str|null, v2.1.1 — structured error code on fetch failure, e.g. `E_RATE_LIMIT` for 429, lets AI agents programmatically distinguish rate-limit from auth/network errors)
**Research mode (v2.1.0)**
- **`--research <topic>`** — given a research topic, auto-expands into 5 multi-angle queries (overview/profile/background/works/review) and runs them in sequence. Results include `research_topic` and `research_queries` metadata so AI agents can structure their final report by angle. Mutually exclusive with `--query` and `--queries-file`. Supports all output formats (json/brief/urls/csv). Deterministic expansion (no AI judgment) — same topic always produces same queries, reproducible across processes
**Research mode (v2.1.0, enhanced in v2.1.1)**
- **`--research <topic>`** — given a research topic, auto-expands into 5 multi-angle queries (overview/profile/background/works/review) and runs them in sequence. Results include `research_topic` and `research_queries` metadata so AI agents can structure their final report by angle. v2.1.1 enhancements:
- **Cross-angle merge & dedup** — JSON output now includes a top-level `merged_results` field: all per-angle results are combined, deduplicated (same URL collapsing), and sorted, so AI agents can get a unified overview without re-deduplicating themselves. Brief/urls formats append a `[MERGED]` section after the per-angle blocks.
- **Bilingual suffixes** — `expand_research_queries()` now detects whether the topic contains CJK characters. Chinese topics use Chinese suffixes (简介/经历/作品/评价); English topics use English suffixes (profile/background/works/reviews). Avoids low-relevance cross-language combinations like "Python asyncio 经历".
- Mutually exclusive with `--query` and `--queries-file`. Supports all output formats (json/brief/urls/csv). Deterministic expansion (no AI judgment) — same topic always produces same queries, reproducible across processes
**Engineering**
- Shared `common.py` module — unified retry/charset/auth/logging/UA-pool/browser-headers/backoff logic across both scripts
@@ -109,7 +112,7 @@ SearXNG is a privacy-respecting metasearch engine that aggregates results from 7
|---------|---------|------------------|
| Instance | **required** — via `-i`, `SEARXNG_INSTANCE` env var, or config file | `-i / --instance` |
| Safe search | **0 (off)** | `-s / --safesearch {0,1,2}` |
| Time range | **year** | `-t / --time-range {day,month,year,none}` (none = disabled) |
| Time range | **year** | `-t / --time-range {day,week,month,year,none}` (none = disabled; `week` added in v2.1.1 to align with SearXNG API standard) |
| Output format | **json** | `-f / --format {json,brief,urls,csv}` |
| Engines | **google,bing,brave,duckduckgo,startpage,wikipedia,wikidata** | `--engines <list>` |
@@ -400,7 +403,7 @@ All scripts live in `scripts/`; run with `python scripts/<name>.py` from any dir
```
usage: search.py [-h] [--query QUERY] [--instance URL]
[--categories CATS] [--language LANG] [--pageno N]
[--time-range {day,month,year,none}] [--safesearch {0,1,2}]
[--time-range {day,week,month,year,none}] [--safesearch {0,1,2}]
[--engines E] [--method {GET,POST}] [--max-results N]
[--format {json,brief,urls,csv}] [--snippet-len N]
[--fetch N] [--fetch-timeout SEC] [--fetch-retries N]
@@ -442,7 +445,7 @@ usage: search.py [-h] [--query QUERY] [--instance URL]
- `--categories general,news` — comma-separated categories (whitespace auto-stripped)
- `--language zh-CN` — language filter
- `--pageno 1` — page number
- `--time-range {day,month,year,none}` — time filter (default: `year`; `none` disables time filtering)
- `--time-range {day,week,month,year,none}` — time filter (default: `year`; `none` disables time filtering; v2.1.1 adds `week` to align with SearXNG API's standard four tiers)
- `--safesearch {0,1,2}` — safe search (default: `0` = off)
- `--max-results N` — limit number of results (applied AFTER dedup+sort, so the highest-scoring/newest items are kept)
- `--sort-by {score,date,engine,none}` — sort results (default: `score` descending; `none` preserves instance order). Applied after dedup, before `--max-results`. HTML-fallback results have no score and keep their order