feat(v2.1.0): 研究模式 + fetch.py Wayback 兜底 + 被墙站点智能回退

A. fetch.py 补齐 Wayback 兜底 (修复重大 gap)

- v2.0.1 gap: fetch.py 独立调用 403 时无 Wayback 兜底 (仅 search.py --fetch 有)

- AI Agent 用 fetch.py -u URL 直接抓取被墙站点时, 403 后无任何回退

- 修复: fetch.py main() 增加 Wayback 兜底逻辑 + --no-fallback flag

- 共享逻辑抽取到 common.py: should_try_wayback() + build_wayback_url()

B. --research 研究模式

- 给定主题自动扩展 5 个多角度查询: overview/profile/background/works/review

- 确定性规则 (不依赖 AI 判断), 跨进程可复现

- 输出含 research_topic + research_queries 元数据, AI Agent 可按角度结构化汇编

- 与 --query/--queries-file 互斥, 支持所有输出格式 (json/brief/urls/csv)

- 三态退出码: 0=有结果, 2=全部空, 1=全部错误

C. 被墙站点智能回退

- common.py 增加 HARD_BLOCKED_DOMAINS: 百度百科/知乎/微博/微信公众号/豆瓣等

- is_hard_blocked_domain() 精确匹配 + 子域匹配

- 命中被墙站点时: 主抓取失败后立即 Wayback (不等 should_try_wayback 判断)

- search.py _should_try_fallback 增加 url 参数, 被墙站点直接触发兜底

真实测试验证 (search.metona.cn 实例):

- fetch.py 百度百科兜底: 403 → Wayback 恢复 150,493 chars ✓

- --research 模式: 5 角度查询扩展 + research 元数据 + 三态退出码 ✓

- 被墙站点检测: Hard-blocked domain detected 日志 + 自动 Wayback ✓

测试: 503 个全部通过 (新增 45 个: test_wayback_shared + test_research_mode)

来源: 另一个 AI Agent 反馈 Wikipedia/百度百科/知乎 fetch 失败, 需要多角度搜索+失败回退+被墙站点列表

SKILL.md

@@ -1,7 +1,7 @@
----
+---
 name: searxng-use-cli
 description: Use when you need to search the web via your OWN SearXNG instance (no public-instance discovery). 3 CLI scripts + a shared common.py module — execute privacy-respecting searches against a user-supplied instance (with multi-instance failover, 5xx/429 retry, auto-fetch) or via SEARXNG_INSTANCE env / config file, fetch/extract readable text or markdown from web pages. Zero-config replacement for proprietary search APIs.
-version: 2.0.1
+version: 2.1.0
 author: Metona Team
 license: MIT
 platforms: [linux, macos, windows]
@@ -72,7 +72,8 @@ SearXNG is a privacy-respecting metasearch engine that aggregates results from 7
 - **Retry-After compliance** — 429/503 responses read the `Retry-After` header (numeric seconds or HTTP date) and wait at least that long before retrying. Non-compliance triggers harsher rate limits
 - **Capped backoff** — `compute_backoff_delay()` caps at 60s (was uncapped: 1.5*2^10 = 1536s would hang the process)
 - **WAF fingerprint library** — `_detect_anti_bot()` identifies Cloudflare / Imperva / PerimeterX / DataDome / Akamai / Anubis / generic challenges via `<title>` tag matching (most precise) + full-document scan of technical identifiers (cookie/header/JS variable names, not bare vendor names). v2.0.1 narrowed broad keywords (e.g. bare `cloudflare`/`captcha`/`challenge`) that caused false positives on normal articles, and added title-tag detection. Returns `waf_type` for AI-agent decisioning
-- **Wayback Machine fallback** — 404/403/timeout AND anti-bot-blocked pages automatically retry via `https://web.archive.org/web/2/<url>` (latest snapshot). v2.0.1 fixed a bug where HTTP 200 anti-bot challenge pages (Cloudflare returns 200 for JS challenges) bypassed the fallback trigger. Default ON; `--no-fallback` disables. Independent 10s timeout so Wayback slowness never blocks the main flow
+- **Wayback Machine fallback** — 404/403/timeout AND anti-bot-blocked pages automatically retry via `https://web.archive.org/web/2/<url>` (latest snapshot). v2.0.1 fixed a bug where HTTP 200 anti-bot challenge pages (Cloudflare returns 200 for JS challenges) bypassed the fallback trigger. v2.1.0: `fetch.py` standalone calls now also get Wayback fallback (was only in `search.py --fetch`). Default ON; `--no-fallback` disables. Independent 10s timeout so Wayback slowness never blocks the main flow
+- **Hard-blocked domain fallback** (v2.1.0) — `is_hard_blocked_domain()` detects known strong-anti-bot sites (baike.baidu.com, zhihu.com, weibo.com, mp.weixin.qq.com, douban.com, etc.) that almost always return 403 regardless of UA. These sites bypass the normal `should_try_wayback()` check and trigger Wayback immediately on failure. List maintained in `common.py` `HARD_BLOCKED_DOMAINS`
 - **Adaptive throttling** — `AdaptiveThrottle` state machine: 3 consecutive failures → double delay + halve concurrency; 5 consecutive successes → gradual recovery; 429 → global pause 30s. Thread-safe
 - **readability-lite extraction** — when `<article>`/`<main>`/content-class `<div>` are all missing, `_readability_lite()` picks the highest text-density node (text chars / tag count + `<p>` weighting), avoiding nav/sidebar/footer noise
 - **`--fetch-report`** — structured per-URL report to stderr after `--fetch N`: status, WAF type, fallback used, char count, plus adaptive throttle stats and a JSON summary line
@@ -80,6 +81,9 @@ SearXNG is a privacy-respecting metasearch engine that aggregates results from 7
 - **`--request-delay`** — configurable delay between fetch requests (default 0.3s; adaptive throttling may increase this on failures)
 - New fetch result fields: `anti_bot_detected` (bool), `waf_type` (str|null), `fallback_used` (str|null)
 
+**Research mode (v2.1.0)**
+- **`--research <topic>`** — given a research topic, auto-expands into 5 multi-angle queries (overview/profile/background/works/review) and runs them in sequence. Results include `research_topic` and `research_queries` metadata so AI agents can structure their final report by angle. Mutually exclusive with `--query` and `--queries-file`. Supports all output formats (json/brief/urls/csv). Deterministic expansion (no AI judgment) — same topic always produces same queries, reproducible across processes
+
 **Engineering**
 - Shared `common.py` module — unified retry/charset/auth/logging/UA-pool/browser-headers/backoff logic across both scripts
 - `search.py --fetch` reuses `fetch.py`'s higher-quality text extractor (no code duplication)