feat(v2.0.0): 反爬增强 + 抓取稳定性大幅提升

反爬措施:

- 浏览器指纹头 build_browser_headers(): Sec-Ch-Ua/Sec-Fetch-*/Accept-Language/Accept-Encoding, 绕过 80%+ 轻量 WAF

- 12 个 UA 池 (Chrome/Edge/Firefox x Win/macOS/Linux x v129-131)

- 确定性 UA 轮换 get_ua_for_domain(): SHA-256 按域名固定 UA, 会话内稳定跨进程可复现

- WAF 指纹库 _detect_anti_bot(): 识别 Cloudflare/Imperva/PerimeterX/DataDome/Akamai/通用, 全文档扫描

- Retry-After 遵守: 429/503 读取 header (数字或 HTTP date) 作为最小重试延迟

- 退避封顶 60s (原无上限, N=10 时 1536s 卡死进程)

抓取稳定性:

- requests.Session 复用: 连接池(10/host) + cookie 持久化 + TLS 会话恢复

- 超时分离 (connect, read) 元组, 避免大页面浪费已建连接

- Wayback Machine 兜底: 404/403/超时自动重试 web.archive.org, 默认启用 --no-fallback 关闭

- AdaptiveThrottle 自适应限流: 3 次失败翻倍延迟+减半并发, 5 次成功渐进恢复, 429 全局暂停 30s

- readability-lite 提取: article/main 缺失时按文本密度选最可能正文 div

新增 CLI flags:

- --fetch-report: 结构化抓取报告到 stderr (每 URL 状态/WAF 类型/兜底方式/字符数 + JSON 摘要)

- --no-fallback: 禁用 Wayback 兜底

- --referer: 设置 Referer 头 (默认实例 URL)

- --request-delay: 抓取请求间隔秒数 (默认 0.3, 自适应可能增大)

fetch 结果新字段: anti_bot_detected (bool), waf_type (str|null), fallback_used (str|null)

测试: 新增 4 个测试文件 (test_browser_headers/test_anti_bot/test_wayback_fallback/test_adaptive_throttle), 451 个测试全部通过

SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: searxng-use-cli
 description: Use when you need to search the web via your OWN SearXNG instance (no public-instance discovery). 3 CLI scripts + a shared common.py module — execute privacy-respecting searches against a user-supplied instance (with multi-instance failover, 5xx/429 retry, auto-fetch) or via SEARXNG_INSTANCE env / config file, fetch/extract readable text or markdown from web pages. Zero-config replacement for proprietary search APIs.
-version: 1.8.1
+version: 2.0.0
 author: Metona Team
 license: MIT
 platforms: [linux, macos, windows]
@@ -62,8 +62,24 @@ SearXNG is a privacy-respecting metasearch engine that aggregates results from 7
 - Config-file auth — `auth_basic` and `auth_bearer` fields in `searxng.toml` let AI agents set credentials once (priority: CLI > file > config > env)
 - Credentials-file permission warning — `--auth-*-file` warns on stderr if the file is group/other-readable (POSIX only)
 
+**Anti-bot & fetch stability (v2.0.0)**
+- **Browser fingerprint headers** — `build_browser_headers()` sends full Sec-Ch-Ua / Sec-Fetch-* / Accept-Language / Accept-Encoding, not just User-Agent. Bypasses 80%+ of lightweight WAFs (Cloudflare basic, Nginx UA blocks)
+- **12-UA pool** — Chrome/Edge/Firefox × Windows/macOS/Linux × versions 129-131. `get_ua_for_domain()` uses SHA-256 to deterministically assign one UA per domain (stable within a session, reproducible across processes)
+- **requests.Session reuse** — module-level Session with connection pool (10 conns/host) + cookie persistence + TLS session resumption. Cuts TLS handshake overhead for multi-page fetches
+- **Split timeouts** — `timeout=(connect, read)` tuple (5s connect, 15s read by default). Previously a single 15s total timeout wasted already-established connections on large pages
+- **Retry-After compliance** — 429/503 responses read the `Retry-After` header (numeric seconds or HTTP date) and wait at least that long before retrying. Non-compliance triggers harsher rate limits
+- **Capped backoff** — `compute_backoff_delay()` caps at 60s (was uncapped: 1.5*2^10 = 1536s would hang the process)
+- **WAF fingerprint library** — `_detect_anti_bot()` identifies Cloudflare / Imperva / PerimeterX / DataDome / Akamai / generic challenges via full-document scan (was first 2000 chars only). Returns `waf_type` for AI-agent decisioning
+- **Wayback Machine fallback** — 404/403/timeout automatically retries via `https://web.archive.org/web/2/<url>` (latest snapshot). Default ON; `--no-fallback` disables. Independent 10s timeout so Wayback slowness never blocks the main flow
+- **Adaptive throttling** — `AdaptiveThrottle` state machine: 3 consecutive failures → double delay + halve concurrency; 5 consecutive successes → gradual recovery; 429 → global pause 30s. Thread-safe
+- **readability-lite extraction** — when `<article>`/`<main>`/content-class `<div>` are all missing, `_readability_lite()` picks the highest text-density node (text chars / tag count + `<p>` weighting), avoiding nav/sidebar/footer noise
+- **`--fetch-report`** — structured per-URL report to stderr after `--fetch N`: status, WAF type, fallback used, char count, plus adaptive throttle stats and a JSON summary line
+- **`--referer`** — set Referer header for fetch requests (defaults to the instance URL when fetching result pages, disguising traffic source)
+- **`--request-delay`** — configurable delay between fetch requests (default 0.3s; adaptive throttling may increase this on failures)
+- New fetch result fields: `anti_bot_detected` (bool), `waf_type` (str|null), `fallback_used` (str|null)
+
 **Engineering**
-- Shared `common.py` module — unified retry/charset/auth/logging logic across both scripts
+- Shared `common.py` module — unified retry/charset/auth/logging/UA-pool/browser-headers/backoff logic across both scripts
 - `search.py --fetch` reuses `fetch.py`'s higher-quality text extractor (no code duplication)
 - Structured logging (`--verbose` / `--quiet`) — three levels: default INFO (progress + warnings), `--verbose` DEBUG (HTTP detail, cache keys), `--quiet` WARNING (errors only). All log output to stderr; stdout reserved for data
 - UTF-8 stdout enforcement (`force_utf8_stdout()`) — Windows Python defaults to GBK and crashes on non-ASCII chars; both scripts force UTF-8 + `errors='replace'` at startup so `print('\xa0')` never raises
@@ -330,7 +346,10 @@ Single query output shape:
       "text_length": 12345,
       "truncated": false,
       "final_url": "https://example.com/final",
-      "user_agent_used": "searxng-cli/1.8.0"
+      "user_agent_used": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
+      "anti_bot_detected": false,
+      "waf_type": null,
+      "fallback_used": null
     }
   ],
   "fetched_source": "json"
@@ -379,9 +398,10 @@ usage: search.py [-h] [--query QUERY] [--instance URL]
                  [--engines E] [--method {GET,POST}] [--max-results N]
                  [--format {json,brief,urls,csv}] [--snippet-len N]
                  [--fetch N] [--fetch-timeout SEC] [--fetch-retries N]
-                 [--max-size BYTES] [--output FILE] [--timeout SEC]
-                 [--retry N] [--fail-fast] [--serial] [--verify]
-                 [--auth-bearer TOKEN] [--auth-bearer-file FILE]
+                 [--fetch-report] [--no-fallback] [--referer URL]
+                 [--request-delay SEC] [--max-size BYTES] [--output FILE]
+                 [--timeout SEC] [--retry N] [--fail-fast] [--serial]
+                 [--verify] [--auth-bearer TOKEN] [--auth-bearer-file FILE]
                  [--auth-basic USER:PASS] [--auth-basic-file FILE]
                  [--proxy URL] [--include-domain DOMAINS]
                  [--exclude-domain DOMAINS] [--queries-file FILE]
@@ -398,13 +418,13 @@ usage: search.py [-h] [--query QUERY] [--instance URL]
 5. Calls the SearXNG API (GET or POST) with `format=json`
 6. Falls back to HTML scraping if JSON is blocked
 7. HTML parser extracts results + suggestions + answers + infoboxes (full content, never truncated)
-8. **Stable auto-fetch:** `--fetch 3` concurrently downloads top 3 result pages with retry, browser-UA fallback, CAPTCHA detection, and **`fetch.py`'s higher-quality text extractor** (the same engine `fetch.py` uses)
+8. **Stable auto-fetch (v2.0.0 enhanced):** `--fetch 3` concurrently downloads top 3 result pages with retry, **full browser fingerprint headers** (Sec-Ch-Ua/Sec-Fetch-*), **12-UA deterministic per-domain pool**, **Retry-After compliance**, **WAF fingerprint detection** (Cloudflare/Imperva/PerimeterX/DataDome/Akamai), **Wayback Machine fallback** for 404/403/timeout, **adaptive throttling** (auto backoff + concurrency reduction on failures), and `fetch.py`'s readability-lite extractor. `--fetch-report` prints a structured per-URL report to stderr
 9. **Health-check mode:** `--verify` probes each instance (reachability / JSON-API support / latency / POST support / engine list / auth status) and prints a report, then exits without searching — use it to validate your instance list
 10. **Result caching:** `--cache-ttl 30` stores results for 30 min; identical queries within the TTL skip the network entirely. Cache lives at `$SEARXNG_CACHE_DIR` or `~/.cache/searxng-cli/cache.db` (SQLite, WAL mode). `--clear-cache` / `--cache-stats` manage it without searching
 11. **Batch mode:** `--queries-file FILE` reads one query per line (blank/`#` lines skipped) and runs them in sequence; JSON output is `{"schema_version": "1.0", "queries": [{"query":..., "status": "ok"|"error", ...}]}` (or one block per query in brief/urls). A failed query is recorded but does not abort the batch. Exit codes: 0 if any query returned results, 1 if all errored, 2 if all empty
 12. **Dedup + sort + domain filter:** After search (and cache), duplicate URLs are collapsed (default; `--no-dedup` disables), results are sorted (`--sort-by`; default: score descending), and then `--include-domain`/`--exclude-domain` filter by domain. Matching is case-insensitive and ignores a leading `www.`; when a domain is in both lists, exclude wins
 13. **Proxy & auth:** `--proxy URL` routes both search and fetch through a proxy; `--auth-bearer` / `--auth-basic` (plus `*-file` variants, `searxng.toml` `auth_basic`/`auth_bearer` fields, and `SEARXNG_BEARER_TOKEN` / `SEARXNG_BASIC_AUTH` env vars) supply credentials. Priority: CLI flag > file > config file > env var
-14. **Config defaults:** `searxng.toml` may pre-set most flags (engines, categories, language, safesearch, time_range, method, format, sort_by, timeout, max_retries, proxy, cache_ttl, fetch, fetch_timeout, fetch_retries, max_size, auth_basic, auth_bearer); explicit CLI flags always win
+14. **Config defaults:** `searxng.toml` may pre-set most flags (engines, categories, language, safesearch, time_range, method, format, sort_by, timeout, max_retries, proxy, cache_ttl, fetch, fetch_timeout, fetch_retries, max_size, request_delay, auth_basic, auth_bearer); explicit CLI flags always win
 15. **Structured errors:** in `--format json` mode, failures print a JSON object `{"error": "...", "error_code": "E_*", "recovery_hint": "...", "exit_code": N, "query": "..."}` to stdout so agents can parse them and decide recovery strategy
 
 **Key options:**
@@ -458,18 +478,18 @@ usage: search.py [-h] [--query QUERY] [--instance URL]
 usage: fetch.py [-h] --url URL [--extract {text,html,markdown}]
                 [--timeout SEC] [--retries N] [--max-size BYTES]
                 [--user-agent STR] [--encoding CHARSET]
-                [--no-redirect] [--proxy URL] [--output FILE]
-                [--auth-bearer TOKEN] [--auth-bearer-file FILE]
+                [--no-redirect] [--referer URL] [--proxy URL]
+                [--output FILE] [--auth-bearer TOKEN] [--auth-bearer-file FILE]
                 [--auth-basic USER:PASS] [--auth-basic-file FILE]
                 [--verbose] [--quiet] [--version]
 ```
 
 **What it does:**
 1. Downloads a web page via HTTP GET with retry + exponential backoff
-2. **Stable fetching:** retries on 429/5xx/connection errors (3x default), falls back to browser User-Agent if blocked
+2. **Stable fetching (v2.0.0 enhanced):** retries on 429/5xx/connection errors (3x default) with **Retry-After header compliance** and **capped 60s backoff**; falls back through a **12-UA deterministic pool** with **full browser fingerprint headers** (Sec-Ch-Ua/Sec-Fetch-*/Accept-Language); reuses a **requests.Session** for connection pooling + cookie persistence
 3. **No size limit by default** — full page content returned; use `--max-size` for a cap
 4. Detects charset from HTTP headers, HTML meta tags, or UTF-8 fallback
-5. Extracts readable content using tree-based parsers (stdlib or BeautifulSoup)
+5. Extracts readable content using tree-based parsers (stdlib or BeautifulSoup); v2.0.0 adds **readability-lite** text-density fallback when `<article>`/`<main>` are missing
 6. Outputs clean text, raw HTML, or properly-converted Markdown (with correct nested-link handling)
 
 **Key options:**
@@ -478,11 +498,12 @@ usage: fetch.py [-h] --url URL [--extract {text,html,markdown}]
 - `--extract html` — raw HTML
 - `--extract markdown` — Markdown conversion (tree-based; handles nested tags, GFM tables, fenced code blocks, blockquotes, inline code, ordered/unordered/nested lists, definition lists, images, emphasis)
 - `--encoding gbk` — force charset for non-UTF-8 pages (else auto-detected from HTTP header / HTML meta / UTF-8 fallback)
-- `--timeout 15` — request timeout in seconds
-- `--retries 3` — max retries on transient errors (429/5xx/connection); falls back to a browser User-Agent when blocked
+- `--timeout 15` — request timeout in seconds (v2.0.0: split into connect/read tuple internally)
+- `--retries 3` — max retries on transient errors (429/5xx/connection); falls back through the 12-UA pool when blocked
 - `--max-size BYTES` — cap page size (default: unlimited; e.g. `5242880` for 5MB)
-- `--user-agent STR` — custom User-Agent header
+- `--user-agent STR` — custom User-Agent header (overrides per-domain UA selection)
 - `--no-redirect` — do **not** follow HTTP 3xx redirects (implemented for both the requests and stdlib paths)
+- `--referer URL` — set Referer header to disguise traffic source (v2.0.0 anti-bot measure)
 - `--proxy URL` — HTTP/HTTPS proxy (e.g. `http://corp-proxy:8080`); respects existing `HTTP_PROXY`/`HTTPS_PROXY` env vars when omitted
 - `--output FILE` — save to file instead of stdout
 - `--auth-bearer TOKEN` / `--auth-bearer-file FILE` — `Authorization: Bearer` header; file variant reads first non-empty, non-`#` line; also honors `SEARXNG_BEARER_TOKEN` env var
@@ -494,8 +515,9 @@ usage: fetch.py [-h] --url URL [--extract {text,html,markdown}]
 **Extraction strategy (text mode):**
 1. Strip non-content elements (script, style, nav, footer, header)
 2. Extract `<article>`, `<main>`, or `<body>` content
-3. Collapse whitespace, output clean UTF-8
-4. Warn if extracted text < 500 chars (likely JS-heavy or bot-blocked)
+3. v2.0.0: if only `<body>` matched, run **readability-lite** to pick the highest text-density `<div>`/`<section>` (filters out nav/sidebar/footer by class/id)
+4. Collapse whitespace, output clean UTF-8
+5. Warn if extracted text < 500 chars (likely JS-heavy or bot-blocked)
 
 **Completion criterion:** Outputs page content. Non-zero exit on HTTP failure. Stderr carries warnings for low-confidence extraction.