feat(v2.2.1): 修复 Brotli 乱码 + 缓存治理 + 多页聚合 + 研究模式增强
核心修复（v2.2.1）: - 修复 Brotli 乱码 bug: build_browser_headers 智能声明 Accept-Encoding, 仅在 brotli 可用时才声明 br; fetch.py 双路径 br 解压（requests + stdlib） 此前 Chrome/Edge UA 抓取 example.com 等返回 br 的站点输出乱码 v2.2.0 新功能: - main() 拆分为 _handle_verify/_handle_research/_handle_batch/_handle_single - --cache-max-size MB: 缓存大小上限 + LRU 淘汰（默认 100MB） - --pages N: 多页聚合 + 跨页去重 - --research 跨角度合并: 新增 merged_results 字段 - --stream / --progress: JSON Lines 流式输出 + request_id 贯穿 - --dry-run / --save-config / --log-format json - --similarity-dedup / --throttle-* 参数化 - 15-UA 池 + PDF/docx 解析 + error_code 字段 文档与测试: - SKILL.md: 版本号唯一（元数据），删除版本标记干扰 - README.md: 测试数量 539 -> 544 - 544 passed (新增 5 个 Content-Encoding 解压测试)
This commit is contained in:
@@ -1,7 +1,7 @@
|
||||
---
|
||||
name: searxng-use-cli
|
||||
description: Use when you need to search the web via your OWN SearXNG instance (no public-instance discovery). 3 CLI scripts + a shared common.py module — execute privacy-respecting searches against a user-supplied instance (with multi-instance failover, 5xx/429 retry, auto-fetch) or via SEARXNG_INSTANCE env / config file, fetch/extract readable text or markdown from web pages. Zero-config replacement for proprietary search APIs.
|
||||
version: 2.1.1
|
||||
version: 2.2.1
|
||||
author: Metona Team
|
||||
license: MIT
|
||||
platforms: [linux, macos, windows]
|
||||
@@ -31,7 +31,7 @@ metadata:
|
||||
|
||||
## Overview
|
||||
|
||||
SearXNG is a privacy-respecting metasearch engine that aggregates results from 70+ search services without tracking users. This skill provides three standalone Python CLI scripts — works with **any AI agent** (Hermes, Claude Code, Codex, OpenCode, Cursor, Trae, etc.) or directly from your terminal.
|
||||
SearXNG is a privacy-respecting metasearch engine that aggregates results from 70+ search services without tracking users. This skill provides standalone Python CLI scripts — works with **any AI agent** or directly from your terminal.
|
||||
|
||||
**Public-instance discovery has been removed.** You must supply your own SearXNG instance URL (self-hosted or one you trust). This makes behavior deterministic and avoids depending on volatile public instances.
|
||||
|
||||
@@ -39,24 +39,26 @@ SearXNG is a privacy-respecting metasearch engine that aggregates results from 7
|
||||
|
||||
**Search & results**
|
||||
- Multi-instance failover with parallel probing (faster failover, deterministic output order)
|
||||
- Exponential-backoff retry on transient errors (429/5xx/connection) via shared `common.py`
|
||||
- Exponential-backoff retry on transient errors (403/429/5xx/connection) via shared retry-policy components in `common.py` (backoff/retryable-status/Retry-After), with the retry loop in each script
|
||||
- `--verify` health-check mode (reachability / JSON-API / latency / POST / engine list / auth status)
|
||||
- Cross-engine result deduplication (default on; `--no-dedup` disables) — collapses duplicate URLs ignoring tracking params (`utm_*`, `gclid`, etc.) and fragments
|
||||
- Similarity deduplication (`--similarity-dedup`; off by default) — title-based SimHash + Jaccard, auto-skipped when results > 500
|
||||
- Result sorting (`--sort-by {score,date,engine,none}`; default: score descending) — applied after dedup, before `--max-results`
|
||||
- Domain allowlist/blocklist (`--include-domain` / `--exclude-domain`) — case-insensitive, ignores leading `www.`, exclude wins on conflict
|
||||
- Pagination aggregation (`--pages N`) — fetch N pages in one run, merge with cross-page dedup
|
||||
- Batch mode (`--queries-file`) — run multiple queries from a file in sequence, combined output
|
||||
|
||||
**Output formats**
|
||||
- JSON (default, rich metadata), brief (title+URL+snippet), urls (plain list), CSV (spreadsheet-ready)
|
||||
- Enhanced Markdown conversion — nested ordered lists (numbered), mixed `ul`/`ol` nesting, `<dl>`/`<dt>`/`<dd>` definition lists, GFM tables, fenced code blocks, blockquotes
|
||||
- Enhanced Markdown conversion — nested ordered lists, mixed `ul`/`ol` nesting, `<dl>`/`<dt>`/`<dd>` definition lists, GFM tables, fenced code blocks, blockquotes
|
||||
- Structured JSON error output (in `--format json` mode) with `error_code` field for machine-readable failure reporting
|
||||
- JSON Lines streaming (`--stream`) — each result emitted as a separate JSON line to stdout, enabling incremental processing by AI agents
|
||||
- Progress events (`--progress`) — structured JSON Lines events to stderr for real-time execution tracking
|
||||
- JSON Lines streaming (`--stream`) — each result emitted as a separate JSON line to stdout, enabling incremental processing
|
||||
- Progress events (`--progress`) — structured JSON Lines events to stderr for real-time execution tracking, includes `request_id` for correlation
|
||||
|
||||
**Caching & config**
|
||||
- SQLite result caching (`--cache-ttl`) — identical queries within a TTL skip the network entirely; `--clear-cache` / `--cache-stats` manage it
|
||||
- Config file (`searxng.toml`) pre-sets most flags; `--config FILE` loads a non-default config; `instances.txt` for plain URL lists
|
||||
- Instance resolution priority: `-i` → `SEARXNG_INSTANCE` env → config file (`./searxng.toml` → `~/.config/searxng-cli/searxng.toml` → `%APPDATA%/searxng-cli/searxng.toml` on Windows)
|
||||
- SQLite result caching (`--cache-ttl`) with size cap + LRU eviction (`--cache-max-size`, default 100MB) — identical queries within a TTL skip the network entirely; `--clear-cache` / `--cache-stats` manage it
|
||||
- Config file (`searxng.toml`) pre-sets most flags; `--config FILE` loads a non-default config; `instances.txt` for plain URL lists; `--save-config FILE` writes current args to a config file
|
||||
- Instance resolution priority: `-i` → `SEARXNG_INSTANCE` env → config file (`./searxng.toml` → `~/.config/searxng-cli/searxng.toml` → `%APPDATA%/searxng-cli/searxng.toml` on Windows) → `instances.txt` (`./instances.txt` → `~/.config/searxng-cli/instances.txt` → `%APPDATA%/searxng-cli/instances.txt`)
|
||||
|
||||
**Network & auth**
|
||||
- Proxy support (`--proxy`) for both search and fetch (sets `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY`)
|
||||
@@ -64,45 +66,53 @@ SearXNG is a privacy-respecting metasearch engine that aggregates results from 7
|
||||
- Config-file auth — `auth_basic` and `auth_bearer` fields in `searxng.toml` let AI agents set credentials once (priority: CLI > file > config > env)
|
||||
- Credentials-file permission warning — `--auth-*-file` warns on stderr if the file is group/other-readable (POSIX only)
|
||||
|
||||
**Anti-bot & fetch stability (v2.0.0)**
|
||||
- **Browser fingerprint headers** — `build_browser_headers()` sends full Sec-Ch-Ua / Sec-Fetch-* / Accept-Language / Accept-Encoding, not just User-Agent. Bypasses 80%+ of lightweight WAFs (Cloudflare basic, Nginx UA blocks)
|
||||
- **12-UA pool** — Chrome/Edge/Firefox × Windows/macOS/Linux × versions 129-131. `get_ua_for_domain()` uses SHA-256 to deterministically assign one UA per domain (stable within a session, reproducible across processes)
|
||||
- **requests.Session reuse** — module-level Session with connection pool (10 conns/host) + cookie persistence + TLS session resumption. Cuts TLS handshake overhead for multi-page fetches
|
||||
- **Split timeouts** — `timeout=(connect, read)` tuple (5s connect, 15s read by default). Previously a single 15s total timeout wasted already-established connections on large pages
|
||||
- **Retry-After compliance** — 429/503 responses read the `Retry-After` header (numeric seconds or HTTP date) and wait at least that long before retrying. Non-compliance triggers harsher rate limits
|
||||
- **Capped backoff** — `compute_backoff_delay()` caps at 60s (was uncapped: 1.5*2^10 = 1536s would hang the process)
|
||||
- **WAF fingerprint library** — `_detect_anti_bot()` identifies Cloudflare / Imperva / PerimeterX / DataDome / Akamai / Anubis / generic challenges via `<title>` tag matching (most precise) + full-document scan of technical identifiers (cookie/header/JS variable names, not bare vendor names). v2.0.1 narrowed broad keywords (e.g. bare `cloudflare`/`captcha`/`challenge`) that caused false positives on normal articles, and added title-tag detection. Returns `waf_type` for AI-agent decisioning
|
||||
- **Wayback Machine fallback** — 404/403/timeout AND anti-bot-blocked pages automatically retry via `https://web.archive.org/web/2/<url>` (latest snapshot). v2.0.1 fixed a bug where HTTP 200 anti-bot challenge pages (Cloudflare returns 200 for JS challenges) bypassed the fallback trigger. v2.1.0: `fetch.py` standalone calls now also get Wayback fallback (was only in `search.py --fetch`). Default ON; `--no-fallback` disables. Independent 10s timeout so Wayback slowness never blocks the main flow
|
||||
- **Hard-blocked domain fallback** (v2.1.0, refined in v2.1.1) — `is_hard_blocked_domain()` detects known strong-anti-bot sites (www.baidu.com, baike.baidu.com, zhihu.com, weibo.com, mp.weixin.qq.com, douban.com, etc.) that almost always return 403 regardless of UA. These sites bypass the normal `should_try_wayback()` check and trigger Wayback immediately on failure. v2.1.1 refined the list: moved `baidu.com` from broad subdomain matching to a precise per-subdomain list (www/baike/zhidao/tieba/wenku), so `pan.baidu.com` (netdisk) and `cloud.baidu.com` (cloud) are no longer false-positively blocked. List maintained in `common.py` `HARD_BLOCKED_DOMAINS`
|
||||
- **Adaptive throttling** — `AdaptiveThrottle` state machine: 3 consecutive failures → double delay + halve concurrency; 5 consecutive successes → gradual recovery; 429 → global pause 30s. Thread-safe
|
||||
- **readability-lite extraction** — when `<article>`/`<main>`/content-class `<div>` are all missing, `_readability_lite()` picks the highest text-density node (text chars / tag count + `<p>` weighting), avoiding nav/sidebar/footer noise
|
||||
- **`--fetch-report`** — structured per-URL report to stderr after `--fetch N`: status, WAF type, fallback used, char count, plus adaptive throttle stats and a JSON summary line
|
||||
- **`--referer`** — set Referer header for fetch requests (defaults to the instance URL when fetching result pages, disguising traffic source)
|
||||
**Anti-bot & fetch stability**
|
||||
- **Browser fingerprint headers** — `build_browser_headers()` sends full Sec-Ch-Ua / Sec-Fetch-* / Accept-Language / Accept-Encoding, not just User-Agent. Bypasses 80%+ of lightweight WAFs (Sec-Ch-Ua/Sec-Fetch-* only sent for Chrome/Edge; Firefox omits them). **Smart Accept-Encoding**: only advertises `br` when the brotli decompressor is actually installed, preventing servers from returning Brotli-compressed bytes that requests can't auto-decompress (which would produce garbled output)
|
||||
- **15-UA pool** — Chrome 138-140 (Win/mac/Linux), Edge 138 (Win/mac), Firefox 140 (Win/mac/Linux), Safari 18 (mac). `get_ua_for_domain()` deterministically assigns one UA per domain (stable within a session, reproducible across processes)
|
||||
- **requests.Session reuse** — module-level Session with connection pool + cookie persistence + TLS session resumption
|
||||
- **Split timeouts** — `timeout=(connect, read)` tuple (connect capped at 10s, read defaults to 15s)
|
||||
- **Retry-After compliance** — 429/503 responses read the `Retry-After` header (numeric seconds or HTTP date) and wait at least that long before retrying
|
||||
- **Capped backoff** — `compute_backoff_delay()` caps at 60s
|
||||
- **WAF fingerprint library** — `_detect_anti_bot()` (in `search.py`, used by `--fetch` flow) identifies Cloudflare / Imperva / PerimeterX / DataDome / Akamai / Anubis / generic challenges via `<title>` tag matching + full-document scan. Returns `waf_type` for AI-agent decisioning
|
||||
- **Wayback Machine fallback** — 404/403/timeout AND anti-bot-blocked pages automatically retry via `https://web.archive.org/web/2/<url>` (latest snapshot). Default ON; `--no-fallback` disables. Wayback timeout = `min(--timeout, 10s)`; Wayback retries = `min(--retries, 2)`
|
||||
- **Hard-blocked domain fallback** — `is_hard_blocked_domain()` (in `common.py`) detects known strong-anti-bot sites (www.baidu.com, baike.baidu.com, zhihu.com, weibo.com, mp.weixin.qq.com, douban.com, etc.) that almost always return 403. These trigger Wayback immediately on failure. List maintained in `HARD_BLOCKED_DOMAINS`. Note: baidu uses exact subdomain matching (www/baike/zhidao/tieba/wenku) to avoid over-blocking pan.baidu.com; zhihu/weibo/douban use subdomain wildcard matching
|
||||
- **Adaptive throttling** — `AdaptiveThrottle` (in `search.py`) state machine: consecutive failures → double delay + halve concurrency; consecutive successes → gradual recovery; 429 → global pause. Thread-safe. Parameters configurable via `--throttle-failure-threshold` / `--throttle-pause-seconds` / `--throttle-max-delay`
|
||||
- **readability-lite extraction** — when `<article>`/`<main>`/`role="main"`/content-class `<div>` are all missing and only `<body>` remains, picks the highest text-density `<div>`/`<section>`/`<article>` node (scored by text density + `<p>` count weighting). Text-density threshold is language-aware — CJK content uses 100 chars, other languages use 200 chars
|
||||
- **PDF/document parsing** — `fetch.py` parses PDF (via `pdftotext` subprocess) and `.docx`/`.xlsx` (via stdlib `zipfile`). Unsupported binary types return `E_UNSUPPORTED_MEDIA`
|
||||
- **`--fetch-report [json]`** — structured per-URL report to stderr after `--fetch N`: status, WAF type, fallback used, char count, plus throttle stats. `--fetch-report json` outputs a full JSON report (items array + summary)
|
||||
- **`--referer`** — set Referer header for fetch requests (defaults to the instance URL when fetching result pages)
|
||||
- **`--request-delay`** — configurable delay between fetch requests (default 0.3s; adaptive throttling may increase this on failures)
|
||||
- New fetch result fields: `anti_bot_detected` (bool), `waf_type` (str|null), `fallback_used` (str|null), `error_code` (str|null, v2.1.1 — structured error code on fetch failure, e.g. `E_RATE_LIMIT` for 429, lets AI agents programmatically distinguish rate-limit from auth/network errors)
|
||||
- **`search.py --fetch` output fields** (in `fetched` array): `anti_bot_detected` (bool), `waf_type` (str|null), `fallback_used` (str|null), `error_code` (str|null — structured error code on fetch failure)
|
||||
- **`fetch.py` FetchResult fields** (standalone script output): `content` (str), `content_type` (str), `final_url` (str), `truncated` (bool), `user_agent` (str), `error_code` (str|null), `error_message` (str|null)
|
||||
|
||||
**Research mode (v2.1.0, enhanced in v2.1.1)**
|
||||
- **`--research <topic>`** — given a research topic, auto-expands into 5 multi-angle queries (overview/profile/background/works/review) and runs them in sequence. Results include `research_topic` and `research_queries` metadata so AI agents can structure their final report by angle. v2.1.1 enhancements:
|
||||
- **Cross-angle merge & dedup** — JSON output now includes a top-level `merged_results` field: all per-angle results are combined, deduplicated (same URL collapsing), and sorted, so AI agents can get a unified overview without re-deduplicating themselves. Brief/urls formats append a `[MERGED]` section after the per-angle blocks.
|
||||
- **Bilingual suffixes** — `expand_research_queries()` now detects whether the topic contains CJK characters. Chinese topics use Chinese suffixes (简介/经历/作品/评价); English topics use English suffixes (profile/background/works/reviews). Avoids low-relevance cross-language combinations like "Python asyncio 经历".
|
||||
- Mutually exclusive with `--query` and `--queries-file`. Supports all output formats (json/brief/urls/csv). Deterministic expansion (no AI judgment) — same topic always produces same queries, reproducible across processes
|
||||
**Research mode**
|
||||
- **`--research <topic>`** — given a research topic, auto-expands into 5 multi-angle queries (overview/profile/background/works/review) and runs them in sequence. Results include `research_topic` and `research_queries` metadata so AI agents can structure their final report by angle
|
||||
- **Cross-angle merge & dedup** — JSON output includes a top-level `merged_results` field: all per-angle results are combined, deduplicated, and sorted. Brief/urls formats append a `[MERGED]` section after the per-angle blocks
|
||||
- **Bilingual suffixes** — detects whether the topic contains CJK characters. Chinese topics use Chinese suffixes (overview uses empty suffix = topic itself, then 简介/经历/作品/评价); English topics use English suffixes (overview uses empty suffix, then profile/background/works/reviews)
|
||||
- **`--research-angles`** — custom angles overriding the default 5 (e.g. `"overview,profile,timeline,controversy"`); each angle appended directly as a query suffix
|
||||
- Mutually exclusive with `--query` and `--queries-file`. Supports all output formats. Deterministic expansion (no AI judgment) — same topic always produces same queries
|
||||
|
||||
**Observability**
|
||||
- **Structured logging** (`--log-format {text,json}`, default text) — `json` outputs one JSON object per line `{ts, level, logger, msg, request_id}` for programmatic parsing
|
||||
- **Request ID propagation** — each run auto-generates an 8-char hex `request_id` threaded through all logs and progress events
|
||||
- **`--dry-run`** — preview mode: no HTTP requests sent; prints `{dry_run, instances, headers_count, action, ...}` JSON to stdout (`instances` is a list of instance URLs)
|
||||
- **`--progress`** — JSON Lines events to stderr with `request_id` for real-time tracking
|
||||
|
||||
**Engineering**
|
||||
- Shared `common.py` module — unified retry/charset/auth/logging/UA-pool/browser-headers/backoff logic across both scripts
|
||||
- Shared `common.py` module — unified retry policy/charset/auth resolution/logging/UA-pool/browser-headers/backoff/error classification/error codes/recovery hints/Wayback fallback/hard-blocked domains/similarity dedup/UTF-8 enforcement/proxy/Retry-After parsing across both scripts
|
||||
- `search.py --fetch` reuses `fetch.py`'s higher-quality text extractor (no code duplication)
|
||||
- Structured logging (`--verbose` / `--quiet`) — three levels: default INFO (progress + warnings), `--verbose` DEBUG (HTTP detail, cache keys), `--quiet` WARNING (errors only). All log output to stderr; stdout reserved for data
|
||||
- UTF-8 stdout enforcement (`force_utf8_stdout()`) — Windows Python defaults to GBK and crashes on non-ASCII chars; both scripts force UTF-8 + `errors='replace'` at startup so `print('\xa0')` never raises
|
||||
- Windows config discovery — `resolve_instances` / `load_config` also check `%APPDATA%/searxng-cli/` (Windows per-user app convention) in addition to `~/.config/searxng-cli/` (POSIX convention)
|
||||
- fetch.py failure diagnostics — error output includes `status_code=`, `cause=`, `url=` fields so AI agents can programmatically distinguish 404 vs 403 vs DNS failure without parsing English prose
|
||||
- Structured logging (`--verbose` / `--quiet`) — three levels: default INFO, `--verbose` DEBUG, `--quiet` WARNING. All log output to stderr; stdout reserved for data
|
||||
- UTF-8 stdout enforcement (`force_utf8_stdout()`) — Windows Python defaults to GBK and crashes on non-ASCII chars; both scripts force UTF-8 at startup
|
||||
- Windows config discovery — also checks `%APPDATA%/searxng-cli/` in addition to `~/.config/searxng-cli/`
|
||||
- Fetch failure diagnostics — error output includes `status_code=`, `cause=`, `url=` fields for programmatic distinction of 404 vs 403 vs DNS failure
|
||||
- Engine/category whitespace normalization (`"google, bing"` → `"google,bing"`)
|
||||
- `--time-range none` option to disable time filtering
|
||||
|
||||
**Scripts + shared module:**
|
||||
1. `search.py` — execute searches against a user-supplied instance, with multi-instance failover + exponential-backoff retry (429/5xx/connection) + auto-fetch + caching + batch + domain filtering
|
||||
1. `search.py` — execute searches against a user-supplied instance, with multi-instance failover + retry + auto-fetch + caching + batch + domain filtering
|
||||
2. `fetch.py` — download and extract readable text or markdown from web pages
|
||||
3. `common.py` — shared utilities (auth headers, charset detection, retry policy, fallback UAs, retry constants) used by both scripts
|
||||
4. `cache.py` — SQLite-backed result cache (SHA-256 key, TTL, WAL mode)
|
||||
5. `_config.py` — package constants (version, User-Agent)
|
||||
3. `common.py` — shared utilities (auth resolution + headers, charset detection, retry policy, UA pool, browser headers, backoff, logging, progress events, error classification + codes, recovery hints, Wayback fallback, hard-blocked domains, similarity dedup, UTF-8 enforcement, proxy, Retry-After parsing) used by both scripts
|
||||
4. `cache.py` — SQLite-backed result cache (SHA-256 key, TTL, WAL mode, size cap + LRU eviction, schema migration, module-level + class API)
|
||||
5. `_config.py` — package constants (version, schema version, default user agent, UA pool)
|
||||
|
||||
## Default settings
|
||||
|
||||
@@ -112,7 +122,7 @@ SearXNG is a privacy-respecting metasearch engine that aggregates results from 7
|
||||
|---------|---------|------------------|
|
||||
| Instance | **required** — via `-i`, `SEARXNG_INSTANCE` env var, or config file | `-i / --instance` |
|
||||
| Safe search | **0 (off)** | `-s / --safesearch {0,1,2}` |
|
||||
| Time range | **year** | `-t / --time-range {day,week,month,year,none}` (none = disabled; `week` added in v2.1.1 to align with SearXNG API standard) |
|
||||
| Time range | **year** | `-t / --time-range {day,week,month,year,none}` (none = disabled) |
|
||||
| Output format | **json** | `-f / --format {json,brief,urls,csv}` |
|
||||
| Engines | **google,bing,brave,duckduckgo,startpage,wikipedia,wikidata** | `--engines <list>` |
|
||||
|
||||
@@ -150,7 +160,7 @@ python scripts/search.py -q "python asyncio tutorial" # -i not needed
|
||||
# auth_bearer = "sk-token-123" # Bearer token (auth_bearer wins if both set)
|
||||
# # Any flag below can also be pre-set here (engines, categories, language,
|
||||
# # safesearch, time_range, method, format, timeout, max_retries, proxy,
|
||||
# # cache_ttl, fetch, fetch_timeout, fetch_retries, max_size).
|
||||
# # cache_ttl, cache_max_size, fetch, fetch_timeout, fetch_retries, max_size).
|
||||
# # Explicit CLI flags always override config values.
|
||||
# # Auth priority: --auth-* > --auth-*-file > searxng.toml > env var
|
||||
# Plain list also works in ./instances.txt (one URL per line, # for comments)
|
||||
@@ -175,7 +185,7 @@ python scripts/search.py -q "ai news" -i https://s.example.com --proxy http://co
|
||||
python scripts/search.py -q "test" -i https://private.example.com --auth-bearer-file ~/.searxng_token
|
||||
|
||||
# 12. Cache management (no search performed)
|
||||
python scripts/search.py --cache-stats # entry count, age, size, path
|
||||
python scripts/search.py --cache-stats # entries, oldest/newest, size, total, max, evicted, utilization
|
||||
python scripts/search.py --clear-cache # delete all entries
|
||||
|
||||
# 13. Export results as CSV (great for spreadsheets / data analysis)
|
||||
@@ -249,6 +259,7 @@ AI agents can use `error_code` to programmatically decide recovery strategy, and
|
||||
| `E_EMPTY` | Empty results (exit code 2) | Refine query, broaden `--time-range`, add `--categories` |
|
||||
| `E_INPUT` | Input error (bad parameters, file not found) | Check query syntax, flag combinations, file paths |
|
||||
| `E_INTERNAL` | Internal error (unexpected exception) | Re-run with `--verbose`, report bug |
|
||||
| `E_UNSUPPORTED_MEDIA` | Unsupported binary media type (PDF/docx/xlsx parse failure or unsupported content type) | No auto hint — try a different URL, or install `pdftotext` for PDF parsing |
|
||||
|
||||
### JSON Lines Streaming (`--stream`)
|
||||
|
||||
@@ -277,12 +288,14 @@ Output format (each line is a separate JSON object):
|
||||
- Exit code 0 on success, 1 on error, 2 on empty results (done event still emitted)
|
||||
|
||||
Only valid with `--format json` (single query mode). Using `--stream` with
|
||||
`--queries-file` or non-json formats raises `E_INPUT` immediately.
|
||||
`--queries-file`, `--research`, or non-json formats raises `E_INPUT` immediately.
|
||||
|
||||
### Progress Events (`--progress`)
|
||||
|
||||
For long-running operations, `--progress` emits structured JSON Lines events
|
||||
to stderr, enabling AI agents to track execution progress in real time:
|
||||
to stderr, enabling AI agents to track execution progress in real time.
|
||||
Every event includes a `request_id` field (8-char hex, auto-generated per run)
|
||||
for correlating all events from a single execution.
|
||||
|
||||
```bash
|
||||
python scripts/search.py -q "research topic" -i https://your-instance --progress --fetch 3
|
||||
@@ -291,17 +304,19 @@ python scripts/search.py -q "research topic" -i https://your-instance --progress
|
||||
Event types (each on its own line, JSON Lines format on stderr):
|
||||
|
||||
```jsonl
|
||||
{"event": "start", "query": "research topic", "instances": 2}
|
||||
{"event": "instance_try", "url": "https://instance1.example.com", "attempt": 1}
|
||||
{"event": "instance_ok", "url": "https://instance1.example.com", "latency": 0.342, "results": 10}
|
||||
{"event": "instance_fail", "url": "https://instance2.example.com", "error": "HTTP 503", "error_code": "E_NETWORK"}
|
||||
{"event": "cache_hit", "query": "research topic", "ttl": 30}
|
||||
{"event": "cache_store", "query": "research topic", "ttl": 30}
|
||||
{"event": "fetch_start", "count": 3}
|
||||
{"event": "fetch_ok", "url": "https://example.com/page", "chars": 12345}
|
||||
{"event": "fetch_fail", "url": "https://bad.example.com", "error": "HTTP 503"}
|
||||
{"event": "done", "results": 10, "query": "research topic"}
|
||||
{"event": "error", "error": "connection refused", "error_code": "E_NETWORK", "query": "..."}
|
||||
{"event": "start", "query": "research topic", "instances": 2, "request_id": "a1b2c3d4"}
|
||||
{"event": "instance_try", "url": "https://instance1.example.com", "attempt": 1, "request_id": "a1b2c3d4"}
|
||||
{"event": "instance_ok", "url": "https://instance1.example.com", "latency": 0.342, "results": 10, "request_id": "a1b2c3d4"}
|
||||
{"event": "instance_fail", "url": "https://instance2.example.com", "error": "HTTP 503", "error_code": "E_NETWORK", "request_id": "a1b2c3d4"}
|
||||
{"event": "cache_hit", "query": "research topic", "ttl": 30, "request_id": "a1b2c3d4"}
|
||||
{"event": "cache_store", "query": "research topic", "ttl": 30, "request_id": "a1b2c3d4"}
|
||||
{"event": "page_ok", "pageno": 1, "results": 10, "request_id": "a1b2c3d4"}
|
||||
{"event": "page_fail", "pageno": 2, "error": "HTTP 503", "request_id": "a1b2c3d4"}
|
||||
{"event": "fetch_start", "count": 3, "request_id": "a1b2c3d4"}
|
||||
{"event": "fetch_ok", "url": "https://example.com/page", "chars": 12345, "request_id": "a1b2c3d4"}
|
||||
{"event": "fetch_fail", "url": "https://bad.example.com", "error": "HTTP 503", "request_id": "a1b2c3d4"}
|
||||
{"event": "done", "results": 10, "query": "research topic", "request_id": "a1b2c3d4"}
|
||||
{"event": "error", "error": "connection refused", "error_code": "E_NETWORK", "query": "...", "request_id": "a1b2c3d4"}
|
||||
```
|
||||
|
||||
AI agents can parse these events to:
|
||||
@@ -309,6 +324,8 @@ AI agents can parse these events to:
|
||||
- Detect cache hits (skip waiting)
|
||||
- Monitor fetch failures and retry strategies
|
||||
- Correlate errors with specific queries in batch mode
|
||||
- Track per-page progress via `page_ok`/`page_fail` events
|
||||
- Correlate all events of a single run via `request_id`
|
||||
|
||||
`--progress` and `--verbose` can be used together (progress events on stderr,
|
||||
debug logs also on stderr). `--progress` events are JSON Lines; `--verbose`
|
||||
@@ -355,10 +372,11 @@ Single query output shape:
|
||||
"text_length": 12345,
|
||||
"truncated": false,
|
||||
"final_url": "https://example.com/final",
|
||||
"user_agent_used": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
|
||||
"user_agent_used": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36",
|
||||
"anti_bot_detected": false,
|
||||
"waf_type": null,
|
||||
"fallback_used": null
|
||||
"fallback_used": null,
|
||||
"error_code": null
|
||||
}
|
||||
],
|
||||
"fetched_source": "json"
|
||||
@@ -385,7 +403,7 @@ fields only appear when `--fetch N` is used.
|
||||
|
||||
## Cross-Agent Compatibility
|
||||
|
||||
These scripts are **agent-agnostic** — they work with any AI agent that can invoke terminal commands (Hermes, Claude Code, Codex, OpenCode, Cursor, Trae, etc.), or directly from a terminal.
|
||||
These scripts are **agent-agnostic** — they work with any AI agent that can invoke terminal commands, or directly from a terminal.
|
||||
|
||||
**Key design decisions for universal compatibility:**
|
||||
- Zero external dependencies (stdlib-only for `search.py`)
|
||||
@@ -402,42 +420,47 @@ All scripts live in `scripts/`; run with `python scripts/<name>.py` from any dir
|
||||
|
||||
```
|
||||
usage: search.py [-h] [--query QUERY] [--instance URL]
|
||||
[--categories CATS] [--language LANG] [--pageno N]
|
||||
[--categories CATS] [--language LANG] [--pageno N] [--pages N]
|
||||
[--time-range {day,week,month,year,none}] [--safesearch {0,1,2}]
|
||||
[--engines E] [--method {GET,POST}] [--max-results N]
|
||||
[--format {json,brief,urls,csv}] [--snippet-len N]
|
||||
[--fetch N] [--fetch-timeout SEC] [--fetch-retries N]
|
||||
[--fetch-report] [--no-fallback] [--referer URL]
|
||||
[--format {json,brief,urls,csv}] [--snippet-len N] [--stream]
|
||||
[--progress] [--fetch N] [--fetch-timeout SEC] [--fetch-retries N]
|
||||
[--fetch-report [json]] [--no-fallback] [--referer URL]
|
||||
[--request-delay SEC] [--max-size BYTES] [--output FILE]
|
||||
[--timeout SEC] [--retry N] [--fail-fast] [--serial]
|
||||
[--verify] [--auth-bearer TOKEN] [--auth-bearer-file FILE]
|
||||
[--auth-basic USER:PASS] [--auth-basic-file FILE]
|
||||
[--proxy URL] [--include-domain DOMAINS]
|
||||
[--exclude-domain DOMAINS] [--queries-file FILE]
|
||||
[--cache-ttl MINUTES] [--clear-cache] [--cache-stats]
|
||||
[--sort-by {score,date,engine,none}] [--no-dedup]
|
||||
[--research TOPIC] [--research-angles ANGLES]
|
||||
[--cache-ttl MINUTES] [--cache-max-size MB] [--clear-cache]
|
||||
[--cache-stats] [--sort-by {score,date,engine,none}] [--no-dedup]
|
||||
[--similarity-dedup] [--similarity-threshold FLOAT]
|
||||
[--throttle-failure-threshold N] [--throttle-pause-seconds SEC]
|
||||
[--throttle-max-delay SEC] [--log-format {text,json}]
|
||||
[--save-config FILE] [--dry-run]
|
||||
[--config FILE] [--dump-schema] [--verbose] [--quiet] [--version]
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
1. Takes a search query and resolves one or more instance URLs (`-i`, `SEARXNG_INSTANCE`, or config file — comma-separated for failover)
|
||||
2. **Multi-instance failover:** if an instance fails (429/5xx/timeout/captcha), automatically tries the next one
|
||||
3. **Exponential backoff:** retries each instance up to 3 times with jitter on transient errors (429, 502, 503, 504, and connection errors)
|
||||
3. **Exponential backoff:** retries each instance up to 3 times with jitter on transient errors (403, 429, 502, 503, 504, and connection errors)
|
||||
4. **Parallel probing (default for 2+ instances):** queries every instance concurrently and returns the first *successful* result in your original instance order — this keeps output deterministic while drastically speeding up failover when an early instance is down/slow. Use `--serial` to disable.
|
||||
5. Calls the SearXNG API (GET or POST) with `format=json`
|
||||
6. Falls back to HTML scraping if JSON is blocked
|
||||
7. HTML parser extracts results + suggestions + answers + infoboxes (full content, never truncated)
|
||||
8. **Stable auto-fetch (v2.0.0 enhanced):** `--fetch 3` concurrently downloads top 3 result pages with retry, **full browser fingerprint headers** (Sec-Ch-Ua/Sec-Fetch-*), **12-UA deterministic per-domain pool**, **Retry-After compliance**, **WAF fingerprint detection** (Cloudflare/Imperva/PerimeterX/DataDome/Akamai), **Wayback Machine fallback** for 404/403/timeout, **adaptive throttling** (auto backoff + concurrency reduction on failures), and `fetch.py`'s readability-lite extractor. `--fetch-report` prints a structured per-URL report to stderr
|
||||
8. **Auto-fetch:** `--fetch 3` concurrently downloads top 3 result pages with retry, full browser fingerprint headers, 15-UA deterministic per-domain pool, Retry-After compliance, WAF fingerprint detection, Wayback Machine fallback for 404/403/timeout, adaptive throttling, and `fetch.py`'s readability-lite extractor. `--fetch-report` prints a structured per-URL report to stderr
|
||||
9. **Health-check mode:** `--verify` probes each instance (reachability / JSON-API support / latency / POST support / engine list / auth status) and prints a report, then exits without searching — use it to validate your instance list
|
||||
10. **Result caching:** `--cache-ttl 30` stores results for 30 min; identical queries within the TTL skip the network entirely. Cache lives at `$SEARXNG_CACHE_DIR` or `~/.cache/searxng-cli/cache.db` (SQLite, WAL mode). `--clear-cache` / `--cache-stats` manage it without searching
|
||||
10. **Result caching:** `--cache-ttl 30` stores results for 30 min; identical queries within the TTL skip the network entirely. Cache lives at `$SEARXNG_CACHE_DIR` or `~/.cache/searxng-cli/cache.db` (SQLite, WAL mode, LRU eviction). `--clear-cache` / `--cache-stats` manage it without searching
|
||||
11. **Batch mode:** `--queries-file FILE` reads one query per line (blank/`#` lines skipped) and runs them in sequence; JSON output is `{"schema_version": "1.0", "queries": [{"query":..., "status": "ok"|"error", ...}]}` (or one block per query in brief/urls). A failed query is recorded but does not abort the batch. Exit codes: 0 if any query returned results, 1 if all errored, 2 if all empty
|
||||
12. **Dedup + sort + domain filter:** After search (and cache), duplicate URLs are collapsed (default; `--no-dedup` disables), results are sorted (`--sort-by`; default: score descending), and then `--include-domain`/`--exclude-domain` filter by domain. Matching is case-insensitive and ignores a leading `www.`; when a domain is in both lists, exclude wins
|
||||
13. **Proxy & auth:** `--proxy URL` routes both search and fetch through a proxy; `--auth-bearer` / `--auth-basic` (plus `*-file` variants, `searxng.toml` `auth_basic`/`auth_bearer` fields, and `SEARXNG_BEARER_TOKEN` / `SEARXNG_BASIC_AUTH` env vars) supply credentials. Priority: CLI flag > file > config file > env var
|
||||
14. **Config defaults:** `searxng.toml` may pre-set most flags (engines, categories, language, safesearch, time_range, method, format, sort_by, timeout, max_retries, proxy, cache_ttl, fetch, fetch_timeout, fetch_retries, max_size, request_delay, auth_basic, auth_bearer); explicit CLI flags always win
|
||||
14. **Config defaults:** `searxng.toml` may pre-set most flags; explicit CLI flags always win
|
||||
15. **Structured errors:** in `--format json` mode, failures print a JSON object `{"error": "...", "error_code": "E_*", "recovery_hint": "...", "exit_code": N, "query": "..."}` to stdout so agents can parse them and decide recovery strategy
|
||||
|
||||
**Key options:**
|
||||
- `--query "your search"` — **required unless** `--verify`, `--queries-file`, `--clear-cache`, `--cache-stats`, or `--dump-schema` is used
|
||||
- `--query "your search"` — **required unless** `--verify`, `--queries-file`, `--research`, `--clear-cache`, `--cache-stats`, `--save-config`, or `--dump-schema` is used
|
||||
- `--instance https://searx.example.org` — **required unless** `SEARXNG_INSTANCE` env var or a config file supplies it; comma-separated list enables failover
|
||||
- `--queries-file FILE` — read queries from a file (one per line; blank/`#` skipped) and run them in sequence; overrides `--query`
|
||||
- `--engines google,duckduckgo` — restrict to specific search engines (whitespace around commas is auto-stripped; see default list above)
|
||||
@@ -445,11 +468,14 @@ usage: search.py [-h] [--query QUERY] [--instance URL]
|
||||
- `--categories general,news` — comma-separated categories (whitespace auto-stripped)
|
||||
- `--language zh-CN` — language filter
|
||||
- `--pageno 1` — page number
|
||||
- `--time-range {day,week,month,year,none}` — time filter (default: `year`; `none` disables time filtering; v2.1.1 adds `week` to align with SearXNG API's standard four tiers)
|
||||
- `--pages N` — fetch N pages of results in one run and merge with cross-page dedup; each page cached independently (cache key includes `pageno`)
|
||||
- `--time-range {day,week,month,year,none}` — time filter (default: `year`; `none` disables time filtering)
|
||||
- `--safesearch {0,1,2}` — safe search (default: `0` = off)
|
||||
- `--max-results N` — limit number of results (applied AFTER dedup+sort, so the highest-scoring/newest items are kept)
|
||||
- `--sort-by {score,date,engine,none}` — sort results (default: `score` descending; `none` preserves instance order). Applied after dedup, before `--max-results`. HTML-fallback results have no score and keep their order
|
||||
- `--no-dedup` — disable cross-engine deduplication (by default, duplicate URLs — same page ignoring tracking params/fragment — are collapsed, keeping the first occurrence's engine/score)
|
||||
- `--similarity-dedup` — enable title-based similarity deduplication (SimHash + Jaccard); off by default; auto-skipped when results > 500
|
||||
- `--similarity-threshold FLOAT` — similarity threshold for `--similarity-dedup` (default: 0.85; higher = stricter)
|
||||
- `--config FILE` — path to a `searxng.toml` config file; overrides the default auto-discovery (`./searxng.toml` → `~/.config/searxng-cli/searxng.toml` → `%APPDATA%/searxng-cli/searxng.toml` on Windows). Must be the first flag so its values can set defaults for other flags
|
||||
- `--verbose` / `-v` — show debug-level diagnostics on stderr (HTTP request URLs, response codes, cache keys, retry detail)
|
||||
- `--quiet` — suppress progress messages and retry notices on stderr; only warnings and errors are shown (no short flag: `-q` is `--query`)
|
||||
@@ -470,13 +496,22 @@ usage: search.py [-h] [--query QUERY] [--instance URL]
|
||||
- `--fail-fast` — use only the first instance, don't fail over to the rest
|
||||
- `--proxy URL` — HTTP/HTTPS proxy for both search and fetch (e.g. `http://corp-proxy:8080`)
|
||||
- `--cache-ttl MINUTES` — cache results for N minutes (default: `0` = disabled); identical queries within the TTL skip the network
|
||||
- `--cache-max-size MB` — cache size cap in MB with LRU eviction (default: `0` = defer to `$SEARXNG_CACHE_MAX_SIZE_BYTES` env var or 100MB built-in default; when cap exceeded, least-recently-accessed entries evicted; set a very large value for effectively unlimited)
|
||||
- `--clear-cache` — delete all cached entries and exit (no search)
|
||||
- `--cache-stats` — print cache statistics (entries, age, size, path) and exit
|
||||
- `--cache-stats` — print cache statistics (entries, oldest_created_at, newest_created_at, path, size_bytes, total_bytes, max_size_bytes, evicted_count, utilization_pct) and exit
|
||||
- `--dump-schema` — print the JSON Schema for `--format json` output and exit; lets AI agents programmatically discover field names and types without parsing prose docs
|
||||
- `--auth-bearer TOKEN` — `Authorization: Bearer` header for private instances
|
||||
- `--auth-bearer-file FILE` — read Bearer token from a file (first non-empty, non-`#` line); also honors `SEARXNG_BEARER_TOKEN` env var
|
||||
- `--auth-basic USER:PASS` — `Authorization: Basic` header (auto base64-encoded)
|
||||
- `--auth-basic-file FILE` — read `user:pass` from a file (first non-empty, non-`#` line); also honors `SEARXNG_BASIC_AUTH` env var
|
||||
- `--log-format {text,json}` — structured logging format (default: `text`); `json` outputs one JSON object per line `{ts, level, logger, msg, request_id}`
|
||||
- `--dry-run` — preview mode: no HTTP requests; prints `{dry_run, instances, headers_count, action, ...}` JSON to stdout (`instances` is a list); supports search/research/batch/verify
|
||||
- `--throttle-failure-threshold N` — consecutive failures before doubling delay + halving concurrency (default: 3)
|
||||
- `--throttle-pause-seconds SEC` — global pause on 429 (default: 30)
|
||||
- `--throttle-max-delay SEC` — max adaptive delay (default: 10)
|
||||
- `--research-angles "a,b,c"` — custom research angles overriding the default 5; each angle appended directly as a query suffix
|
||||
- `--save-config FILE` — save current CLI args as a `searxng.toml` config file and exit
|
||||
- `--fetch-report [json]` — `--fetch-report json` outputs a full JSON report (items + summary); original `--fetch-report` (no arg) keeps text format
|
||||
- `--version` — print version and exit
|
||||
|
||||
**Completion criterion:** Outputs valid JSON with `results` array. Non-zero exit on total failure (all instances exhausted).
|
||||
@@ -487,7 +522,7 @@ usage: search.py [-h] [--query QUERY] [--instance URL]
|
||||
usage: fetch.py [-h] --url URL [--extract {text,html,markdown}]
|
||||
[--timeout SEC] [--retries N] [--max-size BYTES]
|
||||
[--user-agent STR] [--encoding CHARSET]
|
||||
[--no-redirect] [--referer URL] [--proxy URL]
|
||||
[--no-redirect] [--no-fallback] [--referer URL] [--proxy URL]
|
||||
[--output FILE] [--auth-bearer TOKEN] [--auth-bearer-file FILE]
|
||||
[--auth-basic USER:PASS] [--auth-basic-file FILE]
|
||||
[--verbose] [--quiet] [--version]
|
||||
@@ -495,24 +530,26 @@ usage: fetch.py [-h] --url URL [--extract {text,html,markdown}]
|
||||
|
||||
**What it does:**
|
||||
1. Downloads a web page via HTTP GET with retry + exponential backoff
|
||||
2. **Stable fetching (v2.0.0 enhanced):** retries on 429/5xx/connection errors (3x default) with **Retry-After header compliance** and **capped 60s backoff**; falls back through a **12-UA deterministic pool** with **full browser fingerprint headers** (Sec-Ch-Ua/Sec-Fetch-*/Accept-Language); reuses a **requests.Session** for connection pooling + cookie persistence
|
||||
2. Retries on 429/5xx/connection errors (3x default) with Retry-After header compliance and capped 60s backoff; falls back through the 15-UA deterministic pool with full browser fingerprint headers; reuses a requests.Session for connection pooling + cookie persistence. **Both paths handle Content-Encoding decompression**: stdlib urllib path handles gzip/deflate/br (urllib doesn't auto-decompress any of them); requests path handles br manually (requests auto-decompresses gzip/deflate but not Brotli unless the brotli package is installed)
|
||||
3. **No size limit by default** — full page content returned; use `--max-size` for a cap
|
||||
4. Detects charset from HTTP headers, HTML meta tags, or UTF-8 fallback
|
||||
5. Extracts readable content using tree-based parsers (stdlib or BeautifulSoup); v2.0.0 adds **readability-lite** text-density fallback when `<article>`/`<main>` are missing
|
||||
5. Extracts readable content using tree-based parsers (stdlib or BeautifulSoup); readability-lite text-density fallback when `<article>`/`<main>`/`role="main"`/content-class `<div>` are all missing and only `<body>` remains
|
||||
6. Outputs clean text, raw HTML, or properly-converted Markdown (with correct nested-link handling)
|
||||
7. **PDF/document parsing** (regardless of `--extract` mode): PDF via `pdftotext` subprocess, `.docx`/`.xlsx` via stdlib `zipfile`. Unsupported binary types return `E_UNSUPPORTED_MEDIA`
|
||||
|
||||
**Key options:**
|
||||
- `--url https://...` — required
|
||||
- `--extract text` — clean readable text (default)
|
||||
- `--extract text` — clean readable text (default); PDF/docx/xlsx parsed automatically regardless of extract mode (see "What it does" #7)
|
||||
- `--extract html` — raw HTML
|
||||
- `--extract markdown` — Markdown conversion (tree-based; handles nested tags, GFM tables, fenced code blocks, blockquotes, inline code, ordered/unordered/nested lists, definition lists, images, emphasis)
|
||||
- `--encoding gbk` — force charset for non-UTF-8 pages (else auto-detected from HTTP header / HTML meta / UTF-8 fallback)
|
||||
- `--timeout 15` — request timeout in seconds (v2.0.0: split into connect/read tuple internally)
|
||||
- `--retries 3` — max retries on transient errors (429/5xx/connection); falls back through the 12-UA pool when blocked
|
||||
- `--timeout 15` — request timeout in seconds (split into connect/read tuple internally)
|
||||
- `--retries 3` — max retries on transient errors (429/5xx/connection); falls back through the 15-UA pool when blocked
|
||||
- `--max-size BYTES` — cap page size (default: unlimited; e.g. `5242880` for 5MB)
|
||||
- `--user-agent STR` — custom User-Agent header (overrides per-domain UA selection)
|
||||
- `--no-redirect` — do **not** follow HTTP 3xx redirects (implemented for both the requests and stdlib paths)
|
||||
- `--referer URL` — set Referer header to disguise traffic source (v2.0.0 anti-bot measure)
|
||||
- `--no-fallback` — disable Wayback Machine fallback for 404/403/timeout and anti-bot-blocked pages (hard-blocked domains like baike.baidu.com, zhihu.com still prefer Wayback)
|
||||
- `--referer URL` — set Referer header to disguise traffic source
|
||||
- `--proxy URL` — HTTP/HTTPS proxy (e.g. `http://corp-proxy:8080`); respects existing `HTTP_PROXY`/`HTTPS_PROXY` env vars when omitted
|
||||
- `--output FILE` — save to file instead of stdout
|
||||
- `--auth-bearer TOKEN` / `--auth-bearer-file FILE` — `Authorization: Bearer` header; file variant reads first non-empty, non-`#` line; also honors `SEARXNG_BEARER_TOKEN` env var
|
||||
@@ -523,10 +560,9 @@ usage: fetch.py [-h] --url URL [--extract {text,html,markdown}]
|
||||
|
||||
**Extraction strategy (text mode):**
|
||||
1. Strip non-content elements (script, style, nav, footer, header)
|
||||
2. Extract `<article>`, `<main>`, or `<body>` content
|
||||
3. v2.0.0: if only `<body>` matched, run **readability-lite** to pick the highest text-density `<div>`/`<section>` (filters out nav/sidebar/footer by class/id)
|
||||
2. Extract main content in priority order: `<article>` → `<main>` → `role="main"` → content-class `<div>` (class~=`content`/`article`/`post`/`entry`) → `<body>`
|
||||
3. If only `<body>` matched (all higher-priority elements missing), run readability-lite to pick the highest text-density `<div>`/`<section>`/`<article>` node (scored by text density + `<p>` count weighting; filters out nav/sidebar/footer by class/id)
|
||||
4. Collapse whitespace, output clean UTF-8
|
||||
5. Warn if extracted text < 500 chars (likely JS-heavy or bot-blocked)
|
||||
|
||||
**Completion criterion:** Outputs page content. Non-zero exit on HTTP failure. Stderr carries warnings for low-confidence extraction.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user