feat(v2.3.0): fetch 结构化 JSON 契约 + 并发批量 + 真实并发门控

迭代 1 — 正确性修复:
- 修复 --pages N 多页聚合的 unresponsive-engine 警告误判: 原用循环末次
  cached 变量判断, 缓存命中时警告被错误跳过/误触发; 改用独立
  performed_live_query 标记
- UA 池单一来源: 删除 common.py 手工副本 _FALLBACK_UAS_BUILTIN,
  FALLBACK_UAS 直接引用 _config.UA_POOL, 消除双份漂移
- search_html 解码修复: 硬编码 utf-8 改为 detect_charset(header/meta
  自动检测), 新增 --encoding 强制覆盖, 贯穿 search_multi 全链
- AdaptiveThrottle 真实并发门控: acquire_slot()/release_slot() 槽位机制,
  退避降并发后新请求被快速拒绝(E_RATE_LIMIT), 实现持久降并发而非名义降并发

迭代 2 — fetch JSON 契约 + 批量并发:
- fetch.py --format json: 成功 {status,url,final_url,content_type,extract,
  truncated,text_length,user_agent}; 失败 {status,error,error_code,
  status_code,url}, 对齐 search.py 错误码体系
- fetch_page 采集 title + latency, 填充 --fetch-report json 空字段
- --queries-file --parallel-queries N (1-8): 并发批量, 输出保序, 受
  AdaptiveThrottle 门控; 并发模式禁用 --fetch(嵌套并行不安全)
- queries 文件编码自动检测 (UTF-8 → GBK 回退)

迭代 3 — 工程化:
- 新增 pyproject.toml (searxng-search/searxng-fetch 入口点)
- 收敛 20+ 处函数内冗余导入
- --dump-schema 扩展: fetched.items 补全 15 字段, 新增 defs.batch/research
- 新增 17 个测试 (tests/test_v230_features.py), 全量 561 测试通过
- 文档同步 (SKILL.md/README.md, 版本号 2.3.0)

SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: searxng-use-cli
 description: Use when you need to search the web via your OWN SearXNG instance (no public-instance discovery). 3 CLI scripts + a shared common.py module — execute privacy-respecting searches against a user-supplied instance (with multi-instance failover, 5xx/429 retry, auto-fetch) or via SEARXNG_INSTANCE env / config file, fetch/extract readable text or markdown from web pages. Zero-config replacement for proprietary search APIs.
-version: 2.2.1
+version: 2.3.0
 author: Metona Team
 license: MIT
 platforms: [linux, macos, windows]
@@ -45,8 +45,8 @@ SearXNG is a privacy-respecting metasearch engine that aggregates results from 7
 - Similarity deduplication (`--similarity-dedup`; off by default) — title-based SimHash + Jaccard, auto-skipped when results > 500
 - Result sorting (`--sort-by {score,date,engine,none}`; default: score descending) — applied after dedup, before `--max-results`
 - Domain allowlist/blocklist (`--include-domain` / `--exclude-domain`) — case-insensitive, ignores leading `www.`, exclude wins on conflict
+- Batch mode (`--queries-file`) — run multiple queries from a file in sequence, combined output; v2.3.0 adds `--parallel-queries N` (1-8 concurrent, output order preserved, gated by AdaptiveThrottle)
 - Pagination aggregation (`--pages N`) — fetch N pages in one run, merge with cross-page dedup
-- Batch mode (`--queries-file`) — run multiple queries from a file in sequence, combined output
 
 **Output formats**
 - JSON (default, rich metadata), brief (title+URL+snippet), urls (plain list), CSV (spreadsheet-ready)
@@ -54,6 +54,7 @@ SearXNG is a privacy-respecting metasearch engine that aggregates results from 7
 - Structured JSON error output (in `--format json` mode) with `error_code` field for machine-readable failure reporting
 - JSON Lines streaming (`--stream`) — each result emitted as a separate JSON line to stdout, enabling incremental processing
 - Progress events (`--progress`) — structured JSON Lines events to stderr for real-time execution tracking, includes `request_id` for correlation
+- `fetch.py --format json` (v2.3.0) — structured JSON contract for page fetches: success `{status, url, final_url, content_type, extract, truncated, text_length, user_agent}`, failure `{status, error, error_code, status_code, url}` (aligns with search.py's error-code system)
 
 **Caching & config**
 - SQLite result caching (`--cache-ttl`) with size cap + LRU eviction (`--cache-max-size`, default 100MB) — identical queries within a TTL skip the network entirely; `--clear-cache` / `--cache-stats` manage it
@@ -76,14 +77,14 @@ SearXNG is a privacy-respecting metasearch engine that aggregates results from 7
 - **WAF fingerprint library** — `_detect_anti_bot()` (in `search.py`, used by `--fetch` flow) identifies Cloudflare / Imperva / PerimeterX / DataDome / Akamai / Anubis / generic challenges via `<title>` tag matching + full-document scan. Returns `waf_type` for AI-agent decisioning
 - **Wayback Machine fallback** — 404/403/timeout AND anti-bot-blocked pages automatically retry via `https://web.archive.org/web/2/<url>` (latest snapshot). Default ON; `--no-fallback` disables. Wayback timeout = `min(--timeout, 10s)`; Wayback retries = `min(--retries, 2)`
 - **Hard-blocked domain fallback** — `is_hard_blocked_domain()` (in `common.py`) detects known strong-anti-bot sites (www.baidu.com, baike.baidu.com, zhihu.com, weibo.com, mp.weixin.qq.com, douban.com, etc.) that almost always return 403. These trigger Wayback immediately on failure. List maintained in `HARD_BLOCKED_DOMAINS`. Note: baidu uses exact subdomain matching (www/baike/zhidao/tieba/wenku) to avoid over-blocking pan.baidu.com; zhihu/weibo/douban use subdomain wildcard matching
-- **Adaptive throttling** — `AdaptiveThrottle` (in `search.py`) state machine: consecutive failures → double delay + halve concurrency; consecutive successes → gradual recovery; 429 → global pause. Thread-safe. Parameters configurable via `--throttle-failure-threshold` / `--throttle-pause-seconds` / `--throttle-max-delay`
+- **Adaptive throttling** — `AdaptiveThrottle` (in `search.py`) state machine: consecutive failures → double delay + halve concurrency; consecutive successes → gradual recovery; 429 → global pause. Thread-safe. Parameters configurable via `--throttle-failure-threshold` / `--throttle-pause-seconds` / `--throttle-max-delay`. v2.3.0: real concurrency gating via `acquire_slot()`/`release_slot()` — after backoff reduces concurrency, new requests are rejected until in-flight drops below the target (persistent throttling, not just nominal)
 - **readability-lite extraction** — when `<article>`/`<main>`/`role="main"`/content-class `<div>` are all missing and only `<body>` remains, picks the highest text-density `<div>`/`<section>`/`<article>` node (scored by text density + `<p>` count weighting). Text-density threshold is language-aware — CJK content uses 100 chars, other languages use 200 chars
 - **PDF/document parsing** — `fetch.py` parses PDF (via `pdftotext` subprocess) and `.docx`/`.xlsx` (via stdlib `zipfile`). Unsupported binary types return `E_UNSUPPORTED_MEDIA`
 - **`--fetch-report [json]`** — structured per-URL report to stderr after `--fetch N`: status, WAF type, fallback used, char count, plus throttle stats. `--fetch-report json` outputs a full JSON report (items array + summary)
 - **`--referer`** — set Referer header for fetch requests (defaults to the instance URL when fetching result pages)
 - **`--request-delay`** — configurable delay between fetch requests (default 0.3s; adaptive throttling may increase this on failures)
-- **`search.py --fetch` output fields** (in `fetched` array): `anti_bot_detected` (bool), `waf_type` (str|null), `fallback_used` (str|null), `error_code` (str|null — structured error code on fetch failure)
-- **`fetch.py` FetchResult fields** (standalone script output): `content` (str), `content_type` (str), `final_url` (str), `truncated` (bool), `user_agent` (str), `error_code` (str|null), `error_message` (str|null)
+- **`search.py --fetch` output fields** (in `fetched` array): `anti_bot_detected` (bool), `waf_type` (str|null), `fallback_used` (str|null), `error_code` (str|null — structured error code on fetch failure); v2.3.0 also `title` (str|null) and `latency` (float|null seconds) — consumed by `--fetch-report json`
+- **`fetch.py` FetchResult fields** (standalone script output): `content` (str), `content_type` (str), `final_url` (str), `truncated` (bool), `user_agent` (str), `error_code` (str|null), `error_message` (str|null); with `--format json` (v2.3.0) these map to a structured JSON contract
 
 **Research mode**
 - **`--research <topic>`** — given a research topic, auto-expands into 5 multi-angle queries (overview/profile/background/works/review) and runs them in sequence. Results include `research_topic` and `research_queries` metadata so AI agents can structure their final report by angle
@@ -432,7 +433,7 @@ usage: search.py [-h] [--query QUERY] [--instance URL]
                  [--auth-basic USER:PASS] [--auth-basic-file FILE]
                  [--proxy URL] [--include-domain DOMAINS]
                  [--exclude-domain DOMAINS] [--queries-file FILE]
-                 [--research TOPIC] [--research-angles ANGLES]
+                 [--parallel-queries N] [--research TOPIC] [--research-angles ANGLES]
                  [--cache-ttl MINUTES] [--cache-max-size MB] [--clear-cache]
                  [--cache-stats] [--sort-by {score,date,engine,none}] [--no-dedup]
                  [--similarity-dedup] [--similarity-threshold FLOAT]
@@ -453,7 +454,7 @@ usage: search.py [-h] [--query QUERY] [--instance URL]
 8. **Auto-fetch:** `--fetch 3` concurrently downloads top 3 result pages with retry, full browser fingerprint headers, 15-UA deterministic per-domain pool, Retry-After compliance, WAF fingerprint detection, Wayback Machine fallback for 404/403/timeout, adaptive throttling, and `fetch.py`'s readability-lite extractor. `--fetch-report` prints a structured per-URL report to stderr
 9. **Health-check mode:** `--verify` probes each instance (reachability / JSON-API support / latency / POST support / engine list / auth status) and prints a report, then exits without searching — use it to validate your instance list
 10. **Result caching:** `--cache-ttl 30` stores results for 30 min; identical queries within the TTL skip the network entirely. Cache lives at `$SEARXNG_CACHE_DIR` or `~/.cache/searxng-cli/cache.db` (SQLite, WAL mode, LRU eviction). `--clear-cache` / `--cache-stats` manage it without searching
-11. **Batch mode:** `--queries-file FILE` reads one query per line (blank/`#` lines skipped) and runs them in sequence; JSON output is `{"schema_version": "1.0", "queries": [{"query":..., "status": "ok"|"error", ...}]}` (or one block per query in brief/urls). A failed query is recorded but does not abort the batch. Exit codes: 0 if any query returned results, 1 if all errored, 2 if all empty
+11. **Batch mode:** `--queries-file FILE` reads one query per line (blank/`#` lines skipped) and runs them in sequence; JSON output is `{"schema_version": "1.0", "queries": [{"query":..., "status": "ok"|"error", ...}]}` (or one block per query in brief/urls). A failed query is recorded but does not abort the batch. v2.3.0: `--parallel-queries N` (1-8) runs queries concurrently with output order preserved; concurrent mode disables `--fetch` (nested parallel fetch is unsafe). Queries-file encoding is auto-detected (UTF-8, GBK fallback — v2.3.0). Exit codes: 0 if any query returned results, 1 if all errored, 2 if all empty
 12. **Dedup + sort + domain filter:** After search (and cache), duplicate URLs are collapsed (default; `--no-dedup` disables), results are sorted (`--sort-by`; default: score descending), and then `--include-domain`/`--exclude-domain` filter by domain. Matching is case-insensitive and ignores a leading `www.`; when a domain is in both lists, exclude wins
 13. **Proxy & auth:** `--proxy URL` routes both search and fetch through a proxy; `--auth-bearer` / `--auth-basic` (plus `*-file` variants, `searxng.toml` `auth_basic`/`auth_bearer` fields, and `SEARXNG_BEARER_TOKEN` / `SEARXNG_BASIC_AUTH` env vars) supply credentials. Priority: CLI flag > file > config file > env var
 14. **Config defaults:** `searxng.toml` may pre-set most flags; explicit CLI flags always win
@@ -520,6 +521,7 @@ usage: search.py [-h] [--query QUERY] [--instance URL]
 
 ```
 usage: fetch.py [-h] --url URL [--extract {text,html,markdown}]
+                [--format {text,json}]
                 [--timeout SEC] [--retries N] [--max-size BYTES]
                 [--user-agent STR] [--encoding CHARSET]
                 [--no-redirect] [--no-fallback] [--referer URL] [--proxy URL]
@@ -539,6 +541,7 @@ usage: fetch.py [-h] --url URL [--extract {text,html,markdown}]
 
 **Key options:**
 - `--url https://...` — required
+- `--format {text,json}` — output format (default: `text`; `json` = structured contract, v2.3.0). Success: `{status: ok, url, final_url, content_type, extract, truncated, text_length, user_agent}`; failure: `{status: error, url, error, error_code, status_code}`. stdout is pure JSON; logs stay on stderr; exit 0 on success / 1 on failure
 - `--extract text` — clean readable text (default); PDF/docx/xlsx parsed automatically regardless of extract mode (see "What it does" #7)
 - `--extract html` — raw HTML
 - `--extract markdown` — Markdown conversion (tree-based; handles nested tags, GFM tables, fenced code blocks, blockquotes, inline code, ordered/unordered/nested lists, definition lists, images, emphasis)
@@ -565,4 +568,4 @@ usage: fetch.py [-h] --url URL [--extract {text,html,markdown}]
 4. Collapse whitespace, output clean UTF-8
 5. Warn if extracted text < 500 chars (likely JS-heavy or bot-blocked)
 
-**Completion criterion:** Outputs page content. Non-zero exit on HTTP failure. Stderr carries warnings for low-confidence extraction.
+**Completion criterion:** Outputs page content (or the structured JSON object with `--format json`). Non-zero exit on HTTP failure. Stderr carries warnings for low-confidence extraction.