searxng-use-cli/SKILL.md

name, description, version, author, license, platforms, metadata

name	description	version	author	license	platforms	metadata
searxng-use-cli	Use when you need to search the web via your OWN SearXNG instance (no public-instance discovery). 3 CLI scripts + a shared common.py module — execute privacy-respecting searches against a user-supplied instance (with multi-instance failover, 5xx/429 retry, auto-fetch) or via SEARXNG_INSTANCE env / config file, fetch/extract readable text or markdown from web pages. Zero-config replacement for proprietary search APIs.	1.7.0	Metona Team	MIT	linux macos windows	skill tags related_skills search searxng cli web-scraping privacy no-api-key

SearXNG CLI Toolkit

Overview

SearXNG is a privacy-respecting metasearch engine that aggregates results from 70+ search services without tracking users. This skill provides three standalone Python CLI scripts — works with any AI agent (Hermes, Claude Code, Codex, OpenCode, Cursor, Trae, etc.) or directly from your terminal.

Public-instance discovery has been removed. You must supply your own SearXNG instance URL (self-hosted or one you trust). This makes behavior deterministic and avoids depending on volatile public instances.

Key capabilities:

Search & results

• Multi-instance failover with parallel probing (faster failover, deterministic output order)
• Exponential-backoff retry on transient errors (429/5xx/connection) via shared common.py
• --verify health-check mode (reachability / JSON-API / latency / POST / engine list / auth status)
• Cross-engine result deduplication (default on; --no-dedup disables) — collapses duplicate URLs ignoring tracking params (utm_*, gclid, etc.) and fragments
• Result sorting (--sort-by {score,date,engine,none}; default: score descending) — applied after dedup, before --max-results
• Domain allowlist/blocklist (--include-domain / --exclude-domain) — case-insensitive, ignores leading www., exclude wins on conflict
• Batch mode (--queries-file) — run multiple queries from a file in sequence, combined output

Output formats

• JSON (default, rich metadata), brief (title+URL+snippet), urls (plain list), CSV (spreadsheet-ready)
• Enhanced Markdown conversion — nested ordered lists (numbered), mixed ul/ol nesting, <dl>/<dt>/<dd> definition lists, GFM tables, fenced code blocks, blockquotes
• Structured JSON error output (in --format json mode) with error_code field for machine-readable failure reporting
• JSON Lines streaming (--stream) — each result emitted as a separate JSON line to stdout, enabling incremental processing by AI agents
• Progress events (--progress) — structured JSON Lines events to stderr for real-time execution tracking

Caching & config

• SQLite result caching (--cache-ttl) — identical queries within a TTL skip the network entirely; --clear-cache / --cache-stats manage it
• Config file (searxng.toml) pre-sets most flags; --config FILE loads a non-default config; instances.txt for plain URL lists
• Instance resolution priority: -i → SEARXNG_INSTANCE env → config file

Network & auth

• Proxy support (--proxy) for both search and fetch (sets HTTP_PROXY/HTTPS_PROXY/NO_PROXY)
• Auth via CLI flag, file, config file, or env var (--auth-bearer / --auth-basic + *-file variants) to avoid leaking secrets in shell history
• Config-file auth — auth_basic and auth_bearer fields in searxng.toml let AI agents set credentials once (priority: CLI > file > config > env)
• Credentials-file permission warning — --auth-*-file warns on stderr if the file is group/other-readable (POSIX only)

Engineering

• Shared common.py module — unified retry/charset/auth/logging logic across both scripts
• search.py --fetch reuses fetch.py's higher-quality text extractor (no code duplication)
• Structured logging (--verbose / --quiet) — three levels: default INFO (progress + warnings), --verbose DEBUG (HTTP detail, cache keys), --quiet WARNING (errors only). All log output to stderr; stdout reserved for data
• Engine/category whitespace normalization ("google, bing" → "google,bing")
• --time-range none option to disable time filtering

Scripts + shared module:

1. search.py — execute searches against a user-supplied instance, with multi-instance failover + exponential-backoff retry (429/5xx/connection) + auto-fetch + caching + batch + domain filtering
2. fetch.py — download and extract readable text or markdown from web pages
3. common.py — shared utilities (auth headers, charset detection, retry policy, fallback UAs, retry constants) used by both scripts
4. cache.py — SQLite-backed result cache (SHA-256 key, TTL, WAL mode)
5. _config.py — package constants (version, User-Agent)

Default settings

search.py ships with opinionated defaults tuned for AI research:

Setting	Default	Flag to override
Instance	required — via -i, SEARXNG_INSTANCE env var, or config file	-i / --instance
Safe search	0 (off)	-s / --safesearch {0,1,2}
Time range	year	-t / --time-range {day,month,year,none} (none = disabled)
Output format	json	-f / --format {json,brief,urls,csv}
Engines	google,bing,brave,duckduckgo,startpage,wikipedia,wikidata	--engines <list>

Quick Start

# Prerequisites: Python 3.8+
# Optional but recommended:
pip install requests beautifulsoup4

# 1. Search against YOUR instance (instance URL is required)
python scripts/search.py -q "python asyncio tutorial" -i https://my-searxng.example.com

# 2. Multiple instances for failover (comma-separated)
python scripts/search.py -q "rust memory safety" \
  -i https://a.example.com,https://b.example.com --format brief

# 3. Search + auto-fetch top 3 result pages in one command
python scripts/search.py -q "climate policy" -i https://my-searxng.example.com --fetch 3

# 4. Fetch a result page
python scripts/fetch.py -u "https://example.com" --extract text

# 5. Skip -i by configuring the instance once (env var, current shell)
export SEARXNG_INSTANCE="https://my-searxng.example.com,https://backup.example.com"
python scripts/search.py -q "python asyncio tutorial"          # -i not needed

# 6. Or use a config file (./searxng.toml or ~/.config/searxng-cli/searxng.toml)
#    [searxng]
#    instance = "https://my-searxng.example.com"
#    # or: instances = ["https://a.example.com", "https://b.example.com"]
#    # Auth (optional, for private instances):
#    auth_basic = "user:password"        # Basic auth
#    auth_bearer = "sk-token-123"        # Bearer token (auth_bearer wins if both set)
#    # Any flag below can also be pre-set here (engines, categories, language,
#    # safesearch, time_range, method, format, timeout, max_retries, proxy,
#    # cache_ttl, fetch, fetch_timeout, fetch_retries, max_size).
#    # Explicit CLI flags always override config values.
#    # Auth priority: --auth-* > --auth-*-file > searxng.toml > env var
#    Plain list also works in ./instances.txt (one URL per line, # for comments)

# 7. Cache results for 30 minutes (identical queries skip the network)
python scripts/search.py -q "python asyncio" -i https://s.example.com --cache-ttl 30

# 7b. Sort by date (newest first) or disable dedup for raw engine output
python scripts/search.py -q "ai news" -i https://s.example.com --sort-by date --no-dedup

# 8. Batch: run queries from a file (one per line; blank/# lines skipped)
python scripts/search.py --queries-file queries.txt -i https://s.example.com --format json > batch.json

# 9. Domain allowlist + blocklist (applied after search)
python scripts/search.py -q "rust async" -i https://s.example.com \
  --include-domain doc.rust-lang.org,wikipedia.org --exclude-domain pinterest.com

# 10. Route through a corporate proxy (applies to search and fetch)
python scripts/search.py -q "ai news" -i https://s.example.com --proxy http://corp-proxy:8080

# 11. Auth from a file (avoids leaking tokens in shell history)
python scripts/search.py -q "test" -i https://private.example.com --auth-bearer-file ~/.searxng_token

# 12. Cache management (no search performed)
python scripts/search.py --cache-stats        # entry count, age, size, path
python scripts/search.py --clear-cache        # delete all entries

# 13. Export results as CSV (great for spreadsheets / data analysis)
python scripts/search.py -q "rust async" -i https://s.example.com --format csv > results.csv

# 14. Use a specific config file (overrides auto-discovered searxng.toml)
python scripts/search.py --config ./my-config.toml -q "test"

# 15. Control log verbosity on stderr
python scripts/search.py -q "test" -i https://s.example.com --verbose   # debug detail
python scripts/search.py -q "test" -i https://s.example.com --quiet     # errors only

Dependency levels:

Level	Scripts	What you get
Zero deps (stdlib only)	search.py, _config.py	Full search + auto-fetch
pip install requests	fetch.py	Better HTTP (session reuse, redirect handling)
pip install beautifulsoup4	fetch.py	Higher-quality text extraction

When to Use

• Web search without API keys — programmatic search results against your own SearXNG instance
• Privacy-conscious research — queries routed through your instance, not ad-tech infrastructure
• Scraping search results — batch query multiple terms and collect structured results as JSON
• Fetching search result pages — follow links from search results and extract clean text

Don't use for:

• High-frequency production search without a properly-scaled instance — respect your instance's rate limits
• Guaranteed uptime/accuracy — depends entirely on the instance you supply

AI Agent Integration Guide

This section documents the structured interfaces that AI agents can rely on for programmatic integration. All features are designed to be machine-readable and machine-actionable.

Output Channels

Channel	Content	Description
stdout	Data	JSON/CSV/text — the only source AI should parse
stderr	Logs + Progress	Human-readable logs (default) or JSON Lines events (--progress)
exit 0	Success	Results available on stdout
exit 1	Fatal error	Error JSON on stdout (in --format json mode) or stderr
exit 2	Empty results	Search succeeded but returned no results

Error Code System

In --format json mode, errors are emitted as structured JSON on stdout:

{
  "error": "All 2 instances failed. Last error: connection refused",
  "error_code": "E_NETWORK",
  "exit_code": 1,
  "query": "search term"
}

AI agents can use error_code to programmatically decide recovery strategy:

Code	Meaning	AI Recovery Strategy
E_CONFIG	Configuration error (no instance resolved)	Prompt user to set -i / SEARXNG_INSTANCE / config file
E_AUTH	Authentication failed (401/403)	Check token/credentials, prompt user to re-authenticate
E_NETWORK	Network error (connection refused, timeout, 5xx, all instances failed)	Retry with backoff, switch instance or proxy
E_RATE_LIMIT	Rate limited (429)	Wait and retry with reduced frequency
E_PARSE	Parse error (JSON/HTML parsing failed)	Check if instance supports JSON, try HTML fallback
E_EMPTY	Empty results (exit code 2)	Adjust query terms or time range
E_INPUT	Input error (bad parameters, file not found)	Fix CLI arguments or file paths
E_INTERNAL	Internal error (unexpected exception)	Report bug with full error message

JSON Lines Streaming (--stream)

For large result sets, --stream outputs results as JSON Lines (one JSON object per line) to stdout, allowing AI agents to process results incrementally without waiting for the full response:

python scripts/search.py -q "large topic" -i https://your-instance --stream

Output format (each line is a separate JSON object):

{"type": "result", "result": {"title": "...", "url": "...", "content": "..."}}
{"type": "result", "result": {"title": "...", "url": "...", "content": "..."}}
{"type": "done", "count": 2, "query": "large topic"}

• type: "result" — one per search result, emitted as soon as available
• type: "done" — terminal event with total count, always emitted last
• Exit code 0 on success, 2 on empty results (done event still emitted)

Only valid with --format json (the default).

Progress Events (--progress)

For long-running operations, --progress emits structured JSON Lines events to stderr, enabling AI agents to track execution progress in real time:

python scripts/search.py -q "research topic" -i https://your-instance --progress --fetch 3

Event types (each on its own line, JSON Lines format on stderr):

{"event": "start", "query": "research topic", "instances": 2}
{"event": "cache_hit", "query": "research topic", "ttl": 30}
{"event": "cache_store", "query": "research topic", "ttl": 30}
{"event": "fetch_start", "count": 3}
{"event": "fetch_ok", "url": "https://example.com/page", "chars": 12345}
{"event": "fetch_fail", "url": "https://bad.example.com", "error": "HTTP 503"}
{"event": "done", "results": 10, "query": "research topic"}
{"event": "error", "error": "connection refused", "error_code": "E_NETWORK", "query": "..."}

AI agents can parse these events to:

• Show progress indicators to users
• Detect cache hits (skip waiting)
• Monitor fetch failures and retry strategies
• Correlate errors with specific queries in batch mode

--progress and --verbose can be used together (progress events on stderr, debug logs also on stderr). --progress events are JSON Lines; --verbose logs are human-readable text.

Output JSON Schema

The default --format json output shape (for reference, not enforced):

{
  "query": "search term",
  "number_of_results": 10,
  "results": [
    {
      "title": "Result title",
      "url": "https://example.com/page",
      "content": "Snippet text...",
      "engine": "google",
      "score": 1.0,
      "category": "general",
      "published_date": "2024-01-15T10:30:00"
    }
  ],
  "answers": ["Direct answer if available"],
  "corrections": [],
  "suggestions": ["related suggestion"],
  "infoboxes": [],
  "unresponsive_engines": [["engine_name", "error reason"]],
  "fetched": [
    {
      "url": "https://example.com/page",
      "status": "ok",
      "text": "Extracted page content...",
      "text_length": 12345,
      "truncated": false,
      "final_url": "https://example.com/final",
      "user_agent_used": "searxng-cli/1.7.0"
    }
  ],
  "fetched_source": "json"
}

Fields marked as optional may be absent. The fetched and fetched_source fields only appear when --fetch N is used.

Cross-Agent Compatibility

These scripts are agent-agnostic — they work with any AI agent that can invoke terminal commands:

Agent	How to invoke
Hermes	python scripts/search.py -q "..." -i https://your-instance
Claude Code	Same — call via terminal tool
Codex (OpenAI)	Same — call via terminal tool
OpenCode	Same — call via terminal tool
Cursor	Same — call via terminal tool
Trae	Same — call via terminal tool
Standalone (human)	Run directly in any terminal

Key design decisions for universal compatibility:

• Zero external dependencies (stdlib-only for search.py)
• Scripts inject their own directory into sys.path, so they run from any working directory
• Stdout carries data (JSON/text), stderr carries progress/warnings
• Exit codes: 0=success, 1=fatal error, 2=no results/empty
• NO agent-specific API calls or tool dependencies — purely CLI-based, portable across all agent platforms

Standalone usage (no agent):

# Clone or download the scripts/ directory, then run from anywhere:
python /path/to/scripts/search.py -q "your query" -i https://your-instance
python /path/to/scripts/fetch.py -u "https://example.com"

SearXNG Search API (Quick Reference)

Source: docs.searxng.org/dev/search_api.html

GET/POST  /search?q=<query>&format=json        GET /search
POST      /search  (form data)                 POST /

Parameters

Parameter	Required	Values	Description
q	yes	string	Search query (supports engine syntax like site:)
format	no	json, csv, rss	Output format (many instances only allow html)
categories	no	comma-separated	e.g. general,images,news,science
language	no	language code	e.g. en, zh-CN, de
pageno	no	integer (default 1)	Result page number — fully supported by search.py
time_range	no	day, month, year, none	Time filter (search.py default: year; none = disabled)
safesearch	no	0, 1, 2	Safe search (search.py default: 0 = off)
engines	no	comma-separated	search.py default: google,bing,brave,duckduckgo,startpage,wikipedia,wikidata

Authentication

SearXNG supports Authorization headers for private instances (configured via settings.yml → server.secret_key). Both search.py and fetch.py accept:

# Bearer token (most common for API-style auth)
python search.py -q "test" --auth-bearer "sk-abc123..." -i https://private-searx.example.com

# Basic auth (base64-encoded user:pass)
python search.py -q "test" --auth-basic "admin:secret123" -i https://private-searx.example.com

# fetch.py also supports auth for authenticated endpoints
python fetch.py -u "https://protected.example.com/page" --auth-bearer "tok_xxx"

If both are provided, Bearer takes precedence. Auth headers are forwarded to both search API calls and auto-fetched page requests.

Avoid leaking secrets in shell history — credentials can be read from a file or env var instead of a CLI flag (priority: CLI flag > file > env var):

# From a file (first non-empty, non-# line is used)
python search.py -q "test" -i https://private.example.com --auth-bearer-file ~/.searxng_token
python search.py -q "test" -i https://private.example.com --auth-basic-file ~/.searxng_auth

# From an environment variable
export SEARXNG_BEARER_TOKEN="sk-abc123..."
export SEARXNG_BASIC_AUTH="admin:secret123"
python search.py -q "test" -i https://private.example.com      # credentials picked up automatically

JSON Response Structure

{
  "query": "searxng",
  "number_of_results": 1234,
  "results": [
    {
      "title": "...",
      "url": "https://...",
      "content": "Snippet text...",
      "engine": "google",
      "score": 0.0,
      "category": "general"
    }
  ],
  "answers": [],
  "corrections": [],
  "suggestions": ["searxng docker", "searxng api"],
  "infoboxes": [],
  "unresponsive_engines": [["bing", "timeout"]]
}

Key gotcha: Many instances disable format=json in their settings. search.py auto-detects and falls back to HTML scraping. Enable format: [html, json] under search.formats in your instance's settings.yml to get rich JSON metadata.

Scripts

All scripts live in scripts/; run with python scripts/<name>.py from any directory. They import _config.py for shared constants and self-inject their own directory into sys.path.

1. search.py — Execute SearXNG Search

usage: search.py [-h] [--query QUERY] [--instance URL]
                 [--categories CATS] [--language LANG] [--pageno N]
                 [--time-range {day,month,year,none}] [--safesearch {0,1,2}]
                 [--engines E] [--method {GET,POST}] [--max-results N]
                 [--format {json,brief,urls,csv}] [--snippet-len N]
                 [--fetch N] [--fetch-timeout SEC] [--fetch-retries N]
                 [--max-size BYTES] [--output FILE] [--timeout SEC]
                 [--retry N] [--fail-fast] [--serial] [--verify]
                 [--auth-bearer TOKEN] [--auth-bearer-file FILE]
                 [--auth-basic USER:PASS] [--auth-basic-file FILE]
                 [--proxy URL] [--include-domain DOMAINS]
                 [--exclude-domain DOMAINS] [--queries-file FILE]
                 [--cache-ttl MINUTES] [--clear-cache] [--cache-stats]
                 [--sort-by {score,date,engine,none}] [--no-dedup]
                 [--config FILE] [--verbose] [--quiet] [--version]

What it does:

1. Takes a search query and resolves one or more instance URLs (-i, SEARXNG_INSTANCE, or config file — comma-separated for failover)
2. Multi-instance failover: if an instance fails (429/5xx/timeout/captcha), automatically tries the next one
3. Exponential backoff: retries each instance up to 3 times with jitter on transient errors (429, 502, 503, 504, and connection errors)
4. Parallel probing (default for 2+ instances): queries every instance concurrently and returns the first successful result in your original instance order — this keeps output deterministic while drastically speeding up failover when an early instance is down/slow. Use --serial to disable.
5. Calls the SearXNG API (GET or POST) with format=json
6. Falls back to HTML scraping if JSON is blocked
7. HTML parser extracts results + suggestions + answers + infoboxes (full content, never truncated)
8. Stable auto-fetch: --fetch 3 concurrently downloads top 3 result pages with retry, browser-UA fallback, CAPTCHA detection, and fetch.py's higher-quality text extractor (the same engine fetch.py uses)
9. Health-check mode: --verify probes each instance (reachability / JSON-API support / latency / POST support / engine list / auth status) and prints a report, then exits without searching — use it to validate your instance list
10. Result caching: --cache-ttl 30 stores results for 30 min; identical queries within the TTL skip the network entirely. Cache lives at $SEARXNG_CACHE_DIR or ~/.cache/searxng-cli/cache.db (SQLite, WAL mode). --clear-cache / --cache-stats manage it without searching
11. Batch mode: --queries-file FILE reads one query per line (blank/# lines skipped) and runs them in sequence; output is a JSON array (or one block per query in brief/urls). A failed query is recorded but does not abort the batch
12. Dedup + sort + domain filter: After search (and cache), duplicate URLs are collapsed (default; --no-dedup disables), results are sorted (--sort-by; default: score descending), and then --include-domain/--exclude-domain filter by domain. Matching is case-insensitive and ignores a leading www.; when a domain is in both lists, exclude wins
13. Proxy & auth: --proxy URL routes both search and fetch through a proxy; --auth-bearer / --auth-basic (plus *-file variants, searxng.toml auth_basic/auth_bearer fields, and SEARXNG_BEARER_TOKEN / SEARXNG_BASIC_AUTH env vars) supply credentials. Priority: CLI flag > file > config file > env var
14. Config defaults: searxng.toml may pre-set most flags (engines, categories, language, safesearch, time_range, method, format, sort_by, timeout, max_retries, proxy, cache_ttl, fetch, fetch_timeout, fetch_retries, max_size, auth_basic, auth_bearer); explicit CLI flags always win
15. Structured errors: in --format json mode, failures print a JSON object {"error": "...", "exit_code": N, "query": "..."} to stdout so agents can parse them

Key options:

• --query "your search" — required unless --verify, --queries-file, --clear-cache, or --cache-stats is used
• --instance https://searx.example.org — required unless SEARXNG_INSTANCE env var or a config file supplies it; comma-separated list enables failover
• --queries-file FILE — read queries from a file (one per line; blank/# skipped) and run them in sequence; overrides --query
• --engines google,duckduckgo — restrict to specific search engines (whitespace around commas is auto-stripped; see default list above)
• --method POST — use POST instead of GET (better for long queries)
• --categories general,news — comma-separated categories (whitespace auto-stripped)
• --language zh-CN — language filter
• --pageno 1 — page number
• --time-range {day,month,year,none} — time filter (default: year; none disables time filtering)
• --safesearch {0,1,2} — safe search (default: 0 = off)
• --max-results N — limit number of results (applied AFTER dedup+sort, so the highest-scoring/newest items are kept)
• --sort-by {score,date,engine,none} — sort results (default: score descending; none preserves instance order). Applied after dedup, before --max-results. HTML-fallback results have no score and keep their order
• --no-dedup — disable cross-engine deduplication (by default, duplicate URLs — same page ignoring tracking params/fragment — are collapsed, keeping the first occurrence's engine/score)
• --config FILE — path to a searxng.toml config file; overrides the default auto-discovery (./searxng.toml → ~/.config/searxng-cli/searxng.toml). Must be the first flag so its values can set defaults for other flags
• --verbose / -v — show debug-level diagnostics on stderr (HTTP request URLs, response codes, cache keys, retry detail)
• --quiet — suppress progress messages and retry notices on stderr; only warnings and errors are shown (no short flag: -q is --query)
• --include-domain a.com,b.org — allowlist; only results from these domains are kept (applied after search)
• --exclude-domain pinterest.com — blocklist; results from these domains are dropped (applied after search; wins over include on conflict)
• --serial — disable parallel multi-instance probing; search instances strictly one at a time
• --verify — health-check mode: verify instances and exit (no search); combine with --format brief for a table or --format json for machine-readable output
• --format json — full JSON (default); includes fetched array when --fetch is used; errors are emitted as JSON to stdout
• --format brief — title + URL + full snippet (no truncation by default; use --snippet-len 200 to cap)
• --format urls — only result URLs
• --format csv — CSV export (title,url,engine,score,published_date,content); in batch mode (--queries-file), all queries merge into one CSV with a query column
• --fetch N — after search, auto-fetch full text of top N result pages (concurrent, stdlib only)
• --fetch-timeout 10 — timeout per page fetch (default: 10s)
• --fetch-retries 3 — max retries per page fetch
• --max-size BYTES — cap page size (default: unlimited; e.g. 5242880 for 5MB)
• --retry 5 — max retries per instance (default: 3)
• --timeout 15 — request timeout in seconds
• --fail-fast — use only the first instance, don't fail over to the rest
• --proxy URL — HTTP/HTTPS proxy for both search and fetch (e.g. http://corp-proxy:8080)
• --cache-ttl MINUTES — cache results for N minutes (default: 0 = disabled); identical queries within the TTL skip the network
• --clear-cache — delete all cached entries and exit (no search)
• --cache-stats — print cache statistics (entries, age, size, path) and exit
• --auth-bearer TOKEN — Authorization: Bearer header for private instances
• --auth-bearer-file FILE — read Bearer token from a file (first non-empty, non-# line); also honors SEARXNG_BEARER_TOKEN env var
• --auth-basic USER:PASS — Authorization: Basic header (auto base64-encoded)
• --auth-basic-file FILE — read user:pass from a file (first non-empty, non-# line); also honors SEARXNG_BASIC_AUTH env var
• --version — print version and exit

Completion criterion: Outputs valid JSON with results array. Non-zero exit on total failure (all instances exhausted).

2. fetch.py — Fetch & Extract Web Page Content

usage: fetch.py [-h] --url URL [--extract {text,html,markdown}]
                [--timeout SEC] [--retries N] [--max-size BYTES]
                [--user-agent STR] [--encoding CHARSET]
                [--no-redirect] [--proxy URL] [--output FILE]
                [--auth-bearer TOKEN] [--auth-bearer-file FILE]
                [--auth-basic USER:PASS] [--auth-basic-file FILE]
                [--verbose] [--quiet] [--version]

What it does:

1. Downloads a web page via HTTP GET with retry + exponential backoff
2. Stable fetching: retries on 429/5xx/connection errors (3x default), falls back to browser User-Agent if blocked
3. No size limit by default — full page content returned; use --max-size for a cap
4. Detects charset from HTTP headers, HTML meta tags, or UTF-8 fallback
5. Extracts readable content using tree-based parsers (stdlib or BeautifulSoup)
6. Outputs clean text, raw HTML, or properly-converted Markdown (with correct nested-link handling)

Key options:

• --url https://... — required
• --extract text — clean readable text (default)
• --extract html — raw HTML
• --extract markdown — Markdown conversion (tree-based; handles nested tags, GFM tables, fenced code blocks, blockquotes, inline code, ordered/unordered/nested lists, definition lists, images, emphasis)
• --encoding gbk — force charset for non-UTF-8 pages (else auto-detected from HTTP header / HTML meta / UTF-8 fallback)
• --timeout 15 — request timeout in seconds
• --retries 3 — max retries on transient errors (429/5xx/connection); falls back to a browser User-Agent when blocked
• --max-size BYTES — cap page size (default: unlimited; e.g. 5242880 for 5MB)
• --user-agent STR — custom User-Agent header
• --no-redirect — do not follow HTTP 3xx redirects (implemented for both the requests and stdlib paths)
• --proxy URL — HTTP/HTTPS proxy (e.g. http://corp-proxy:8080); respects existing HTTP_PROXY/HTTPS_PROXY env vars when omitted
• --output FILE — save to file instead of stdout
• --auth-bearer TOKEN / --auth-bearer-file FILE — Authorization: Bearer header; file variant reads first non-empty, non-# line; also honors SEARXNG_BEARER_TOKEN env var
• --auth-basic USER:PASS / --auth-basic-file FILE — Authorization: Basic header (auto base64-encoded); file variant + SEARXNG_BASIC_AUTH env var also supported
• --verbose / -v — show debug-level diagnostics on stderr
• --quiet — suppress progress messages on stderr; only warnings and errors are shown
• --version — print version and exit

Extraction strategy (text mode):

1. Strip non-content elements (script, style, nav, footer, header)
2. Extract <article>, <main>, or <body> content
3. Collapse whitespace, output clean UTF-8
4. Warn if extracted text < 500 chars (likely JS-heavy or bot-blocked)

Completion criterion: Outputs page content. Non-zero exit on HTTP failure. Stderr carries warnings for low-confidence extraction.

Common Workflow

INSTANCE=https://my-searxng.example.com

# 1. Search + auto-fetch top 3 result pages (uses defaults: safesearch off,
#    time_range year, json output, the 7 default engines)
python scripts/search.py -q "python asyncio tutorial" -i "$INSTANCE" --fetch 3

# 2. Brief format with full snippets (no truncation by default)
python scripts/search.py -q "rust memory safety" -i "$INSTANCE" --format brief --fetch 2

# 3. Pipe-friendly: search -> extract first URL
TOP_URL=$(python scripts/search.py -q "rust book" -i "$INSTANCE" --format urls --fail-fast | head -1)

# 4. Override engines and use POST for a long query
python scripts/search.py -q "ai news" -i "$INSTANCE" --engines google,duckduckgo \
  --method POST --fetch 5 --fetch-timeout 15 --format json

# 5. Failover across two instances
python scripts/search.py -q "quantum computing" -i "$INSTANCE,https://backup.example.com"

JSON vs HTML fallback difference: When an instance supports format=json, search results include rich metadata (engine name, score, category) and fetched_source is "json". With HTML fallback, results are less structured and fetched_source is "html". The auto-fetch feature works identically in both cases.

Troubleshooting

--instance is required (unless configured)

Public-instance discovery has been removed. You must supply an instance, but you have three options (in priority order):

1. -i https://your-instance (CLI flag, comma-separated for failover)
2. SEARXNG_INSTANCE environment variable (e.g. export SEARXNG_INSTANCE="https://a,https://b")
3. A config file: ./searxng.toml or ~/.config/searxng-cli/searxng.toml (instance = "..." or instances = [...]), or ./instances.txt / ~/.config/searxng-cli/instances.txt (one URL per line, # comments)

Self-host SearXNG (Docker: docker run -d -p 8080:8080 searxng/searxng) or use an instance you trust.

All instances return 429 / timeout

Your instance is rate-limited. Wait a few minutes, pass multiple instances via -i a,b, or tune your instance's limiter settings.

Validate your instance list with --verify

Before relying on a multi-instance setup, run python scripts/search.py --verify -i a,b,c (optionally with --format brief for a human-readable table). It reports each instance's reachability, JSON-API support, and latency, then exits without searching. Combine with SEARXNG_INSTANCE or a config file to check your persisted list in one go.

JSON format blocked (HTML fallback)

Some instances disable format=json. Enable search.formats: [html, json] in your instance's settings.yml. The HTML fallback parser handles the rest automatically.

Certificate errors on Windows

Some instances use Let's Encrypt certificates. Run pip install certifi or upgrade Python's certifi bundle.

fetch.py output is empty or gibberish

Try --encoding gbk for Chinese sites, --encoding shift_jis for Japanese. If the page requires JavaScript, use a headless browser instead. If a page 3xx-redirects and you passed --no-redirect, an empty body is expected (that's the redirect response itself).

Caching, proxy, domain filter, batch

• Cache: --cache-ttl 30 caches results for 30 min; repeat the same query and you'll see [cache hit] on stderr. --cache-stats shows entries/age/size/path; --clear-cache wipes them. The cache key covers query + engines + categories + language + time_range + safesearch + pageno + method — different params get separate entries. Cache lives at $SEARXNG_CACHE_DIR or ~/.cache/searxng-cli/cache.db (SQLite, WAL mode).
• Proxy: --proxy http://corp-proxy:8080 sets HTTP_PROXY/HTTPS_PROXY for both search and fetch; NO_PROXY defaults to localhost,127.0.0.1,::1 so local traffic stays direct. Existing proxy env vars are honored when --proxy is omitted.
• Domain filter: --include-domain (allowlist) and --exclude-domain (blocklist) run after search. Matching is case-insensitive and ignores a leading www.. If a domain appears in both lists, exclude wins (the result is dropped).
• Batch: --queries-file FILE runs one query per line (blank/# skipped) in sequence. JSON output is an array [{"query":..., "results":...}, ...]; a failed query becomes {"query":..., "error":...} but does not abort the batch. Exit 0 if any query succeeded, 1 only if all failed.
• Config defaults: searxng.toml can pre-set most flags (see the Quick Start example). Explicit CLI flags always override config values; string-valued ints from TOML are normalized automatically.

"ModuleNotFoundError: No module named '_config'"

Each script self-injects its directory into sys.path, so this should no longer occur. If it does, ensure _config.py sits next to search.py / fetch.py in the same scripts/ directory.

Common Pitfalls

1. JSON format blocked. Many instances disable format=json. Enable it in your instance settings for rich metadata; otherwise the HTML fallback parser is used.

2. Rate limiting. search.py retries each instance up to 3 times with exponential backoff and jitter, then fails over to the next instance. Supply multiple instances via -i a,b for resilience.

3. fetch.py extraction quality varies. The heuristic text extractor works well on articles and documentation but poorly on SPAs, login walls, and JavaScript-heavy pages. For JS-heavy pages, use a headless browser.

4. Encoding issues on Windows. The scripts output UTF-8. On cmd.exe, run chcp 65001 first. PowerShell handles UTF-8 natively. For non-UTF-8 pages, use fetch.py --encoding gbk.

5. POST vs GET. Some instances handle POST differently or block it entirely. If POST search fails, try the default GET method.

6. Overriding defaults. Remember the new defaults (safesearch off, time_range year, the 7-engine list). Pass the corresponding flag to change any of them per query.

7. Cache vs. freshness. --cache-ttl returns cached results without hitting the network — fast, but stale. For time-sensitive queries (news, prices), use --cache-ttl 0 or --clear-cache; the cache key includes time_range/pageno but not wall-clock time, so a day-range query cached at 09:00 is served as-is until the TTL expires.

8. Domain filter semantics. --include-domain/--exclude-domain run after search and only prune the already-returned results — they do not make the instance fetch more. An overly strict allowlist can yield zero results. Exclude wins over include when a domain is in both lists.

9. Dedup changes output. Dedup is on by default — duplicate URLs (ignoring utm_*/gclid/fragment, normalizing scheme/host case and param order) are collapsed. Use --no-dedup if you need the raw per-engine result set (e.g., comparing engine coverage).

10. Sort changes --max-results behavior. With the default --sort-by score, --max-results N keeps the highest-scoring N results, not the first N in instance order. Use --sort-by none to preserve the original order before limiting.

Verification Checklist

• python scripts/search.py -q "test" -i <URL> --format json returns results
• python scripts/search.py -q "test" -i <URL> --method POST works (optional)
• python scripts/search.py -q "test" -i a,b fails over across instances
• python scripts/fetch.py -u "https://example.com" returns readable text
• python scripts/fetch.py -u "https://example.com" -e markdown produces valid markdown
• python scripts/fetch.py -u "http://<redirecting>" --no-redirect does not follow the redirect
• All scripts have --help and --version
• Scripts run from any working directory (sys.path self-injection)
• Scripts exit 0 on success, non-zero on failure
• python scripts/search.py --verify -i <URL> reports instance health (reachable / JSON support / latency), exit 0
• python scripts/search.py -q "test" -i <URL> shows Dedup: on stderr when duplicates exist; --no-dedup suppresses it
• python scripts/search.py -q "test" -i <URL> --sort-by date returns newest-first; --sort-by none preserves instance order
• python scripts/search.py -q "test" -i <URL> --format csv outputs a CSV with header title,url,engine,score,published_date,content
• python scripts/search.py --config ./my-config.toml -q "test" loads the specified config and applies its defaults
• python scripts/search.py -q "test" -i <URL> --verbose shows debug-level diagnostics on stderr
• python scripts/search.py -q "test" -i <URL> --quiet suppresses progress messages; only warnings/errors on stderr
• python scripts/fetch.py -u <URL> --verbose shows debug-level diagnostics on stderr
• python scripts/search.py -q "test" -i <URL> --cache-ttl 30 then re-run → [cache hit] on stderr
• python scripts/search.py --cache-stats prints entries/size/path; --clear-cache reports count deleted
• python scripts/search.py --queries-file queries.txt -i <URL> --format json emits a JSON array
• python scripts/search.py -q "test" -i <URL> --include-domain example.com --exclude-domain spam.com filters as expected
• python scripts/search.py -q "test" -i <URL> --proxy http://proxy:8080 routes through the proxy
• python scripts/search.py -q "test" -i <URL> --auth-bearer-file <FILE> authenticates without CLI token leakage
• A failing search with --format json emits {"error":..., "exit_code":1} to stdout
• python scripts/fetch.py -u <URL> --proxy http://proxy:8080 routes through the proxy
• python scripts/fetch.py -u <URL> --auth-bearer-file <FILE> authenticates
• Stderr carries warnings; stdout carries data