feat(v2.1.0): 研究模式 + fetch.py Wayback 兜底 + 被墙站点智能回退

A. fetch.py 补齐 Wayback 兜底 (修复重大 gap)

- v2.0.1 gap: fetch.py 独立调用 403 时无 Wayback 兜底 (仅 search.py --fetch 有)

- AI Agent 用 fetch.py -u URL 直接抓取被墙站点时, 403 后无任何回退

- 修复: fetch.py main() 增加 Wayback 兜底逻辑 + --no-fallback flag

- 共享逻辑抽取到 common.py: should_try_wayback() + build_wayback_url()

B. --research 研究模式

- 给定主题自动扩展 5 个多角度查询: overview/profile/background/works/review

- 确定性规则 (不依赖 AI 判断), 跨进程可复现

- 输出含 research_topic + research_queries 元数据, AI Agent 可按角度结构化汇编

- 与 --query/--queries-file 互斥, 支持所有输出格式 (json/brief/urls/csv)

- 三态退出码: 0=有结果, 2=全部空, 1=全部错误

C. 被墙站点智能回退

- common.py 增加 HARD_BLOCKED_DOMAINS: 百度百科/知乎/微博/微信公众号/豆瓣等

- is_hard_blocked_domain() 精确匹配 + 子域匹配

- 命中被墙站点时: 主抓取失败后立即 Wayback (不等 should_try_wayback 判断)

- search.py _should_try_fallback 增加 url 参数, 被墙站点直接触发兜底

真实测试验证 (search.metona.cn 实例):

- fetch.py 百度百科兜底: 403 → Wayback 恢复 150,493 chars ✓

- --research 模式: 5 角度查询扩展 + research 元数据 + 三态退出码 ✓

- 被墙站点检测: Hard-blocked domain detected 日志 + 自动 Wayback ✓

测试: 503 个全部通过 (新增 45 个: test_wayback_shared + test_research_mode)

来源: 另一个 AI Agent 反馈 Wikipedia/百度百科/知乎 fetch 失败, 需要多角度搜索+失败回退+被墙站点列表

scripts/fetch.py

@@ -31,15 +31,18 @@ from common import (
     apply_proxy,
     build_auth_headers,
     build_browser_headers,
+    build_wayback_url,
     compute_backoff_delay,
     detect_charset,
     force_utf8_stdout,
     get_ua_for_domain,
+    is_hard_blocked_domain,
     is_retryable_error,
     parse_retry_after,
     resolve_auth_basic,
     resolve_auth_bearer,
     setup_logging,
+    should_try_wayback,
 )
 
 logger = logging.getLogger("searxng.fetch")
@@ -835,6 +838,12 @@ Examples:
                         help="Force charset for decoding (e.g. gbk, shift_jis)")
     parser.add_argument("--no-redirect", action="store_true",
                         help="Do not follow HTTP redirects")
+    parser.add_argument("--no-fallback", action="store_true",
+                        help="Disable Wayback Machine fallback (v2.1.0). "
+                             "By default, 403/404/timeout automatically retries "
+                             "via web.archive.org. Hard-blocked domains "
+                             "(baike.baidu.com, zhihu.com, etc.) always get "
+                             "Wayback priority.")
     parser.add_argument("--referer", default=None, metavar="URL",
                         help="Set Referer header (e.g. https://www.google.com/) to "
                              "disguise traffic source. v2.0.0 anti-bot measure.")
@@ -888,6 +897,13 @@ Examples:
         auth_type = "Bearer" if bearer_token else "Basic"
         logger.info(f"Auth: {auth_type} ***")
 
+    # v2.1.0: 被墙站点提示
+    fallback_enabled = not args.no_fallback
+    hard_blocked = is_hard_blocked_domain(args.url)
+    if hard_blocked:
+        logger.info(f"Hard-blocked domain detected — Wayback fallback "
+                    f"will be prioritized if main fetch fails")
+
     try:
         result = fetch_url(
             args.url, timeout=args.timeout, user_agent=args.user_agent,
@@ -923,7 +939,34 @@ Examples:
         if args.no_redirect:
             diag_parts.append("redirects=disabled")
         logger.error(" | ".join(diag_parts))
-        sys.exit(1)
+
+        # v2.1.0: Wayback Machine 兜底
+        # 触发条件：兜底启用 + (错误可恢复 OR 命中被墙站点)
+        error_msg = str(e) if str(e) else e.__class__.__name__
+        if fallback_enabled and (should_try_wayback(error_msg) or hard_blocked):
+            wayback_url = build_wayback_url(args.url)
+            wb_timeout = min(args.timeout, 10)  # Wayback 独立超时，不阻塞
+            logger.info(f"[FALLBACK] Trying Wayback Machine: {wayback_url[:70]}")
+            try:
+                wb_result = fetch_url(
+                    wayback_url, timeout=wb_timeout,
+                    user_agent=args.user_agent, encoding=args.encoding,
+                    auth_headers=None,  # Wayback 不需要原始站点的认证
+                    max_retries=min(args.retries, 2),
+                    max_size=args.max_size,
+                    allow_redirects=True,
+                )
+                content, content_type, final_url = (
+                    wb_result.content, wb_result.content_type,
+                    wb_result.final_url,
+                )
+                logger.info(f"[FALLBACK] Wayback recovery successful "
+                            f"({len(content)} chars)")
+            except Exception as wb_e:
+                logger.error(f"[FALLBACK] Wayback also failed: {wb_e}")
+                sys.exit(1)
+        else:
+            sys.exit(1)
 
     if final_url != args.url:
         logger.info(f"Redirected to: {final_url}")