MetonaTeam/searxng-use-cli
2
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: SearXNG CLI Toolkit v1.6.0
CI / test (3.8) (push) Canceled after 0s
Details
CI / test (3.9) (push) Canceled after 0s
Details
CI / test (3.10) (push) Canceled after 0s
Details
CI / test (3.11) (push) Canceled after 0s
Details
CI / test (3.12) (push) Canceled after 0s
Details

Browse Source 
Multi-instance failover, exponential-backoff retry, SQLite cache, batch mode, domain filter, cross-engine dedup, result sorting, CSV export, structured logging, enhanced Markdown conversion, 155 pytest tests, Gitea Actions CI
This commit is contained in:
Metona Team
2026-08-01 17:02:34 +08:00
commit 0468dd4e9d
17 changed files with 4791 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
SKILL.md
+508
View File
@@ -0,0 +1,508 @@
---
name: searxng-use-cli
description: Use when you need to search the web via your OWN SearXNG instance (no public-instance discovery). 3 CLI scripts + a shared common.py module — execute privacy-respecting searches against a user-supplied instance (with multi-instance failover, 5xx/429 retry, auto-fetch) or via SEARXNG_INSTANCE env / config file, fetch/extract readable text or markdown from web pages. Zero-config replacement for proprietary search APIs.
version: 1.6.0
author: Metona Team
license: MIT
platforms: [linux, macos, windows]
metadata:
skill:
tags: [search, searxng, cli, web-scraping, privacy, no-api-key]
related_skills: []
---
# SearXNG CLI Toolkit
## Overview
SearXNG is a privacy-respecting metasearch engine that aggregates results from 70+ search services without tracking users. This skill provides three standalone Python CLI scripts — works with **any AI agent** (Hermes, Claude Code, Codex, OpenCode, Cursor, Trae, etc.) or directly from your terminal.
**Public-instance discovery has been removed.** You must supply your own SearXNG instance URL (self-hosted or one you trust). This makes behavior deterministic and avoids depending on volatile public instances.
**Key capabilities:**
**Search & results**
- Multi-instance failover with parallel probing (faster failover, deterministic output order)
- Exponential-backoff retry on transient errors (429/5xx/connection) via shared `common.py`
- `--verify` health-check mode (reachability / JSON-API / latency / POST / engine list / auth status)
- Cross-engine result deduplication (default on; `--no-dedup` disables) — collapses duplicate URLs ignoring tracking params (`utm_*`, `gclid`, etc.) and fragments
- Result sorting (`--sort-by {score,date,engine,none}`; default: score descending) — applied after dedup, before `--max-results`
- Domain allowlist/blocklist (`--include-domain` / `--exclude-domain`) — case-insensitive, ignores leading `www.`, exclude wins on conflict
- Batch mode (`--queries-file`) — run multiple queries from a file in sequence, combined output
**Output formats**
- JSON (default, rich metadata), brief (title+URL+snippet), urls (plain list), CSV (spreadsheet-ready)
- Enhanced Markdown conversion — nested ordered lists (numbered), mixed `ul`/`ol` nesting, `<dl>`/`<dt>`/`<dd>` definition lists, GFM tables, fenced code blocks, blockquotes
- Structured JSON error output (in `--format json` mode) for machine-readable failure reporting
**Caching & config**
- SQLite result caching (`--cache-ttl`) — identical queries within a TTL skip the network entirely; `--clear-cache` / `--cache-stats` manage it
- Config file (`searxng.toml`) pre-sets most flags; `--config FILE` loads a non-default config; `instances.txt` for plain URL lists
- Instance resolution priority: `-i` → `SEARXNG_INSTANCE` env → config file
**Network & auth**
- Proxy support (`--proxy`) for both search and fetch (sets `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY`)
- Auth via CLI flag, file, or env var (`--auth-bearer` / `--auth-basic` + `*-file` variants) to avoid leaking secrets in shell history
- Credentials-file permission warning — `--auth-*-file` warns on stderr if the file is group/other-readable (POSIX only)
**Engineering**
- Shared `common.py` module — unified retry/charset/auth/logging logic across both scripts
- `search.py --fetch` reuses `fetch.py`'s higher-quality text extractor (no code duplication)
- Structured logging (`--verbose` / `--quiet`) — three levels: default INFO (progress + warnings), `--verbose` DEBUG (HTTP detail, cache keys), `--quiet` WARNING (errors only). All log output to stderr; stdout reserved for data
- Engine/category whitespace normalization (`"google, bing"` → `"google,bing"`)
- `--time-range none` option to disable time filtering
**Scripts + shared module:**
1. `search.py` — execute searches against a user-supplied instance, with multi-instance failover + exponential-backoff retry (429/5xx/connection) + auto-fetch + caching + batch + domain filtering
2. `fetch.py` — download and extract readable text or markdown from web pages
3. `common.py` — shared utilities (auth headers, charset detection, retry policy, fallback UAs, retry constants) used by both scripts
4. `cache.py` — SQLite-backed result cache (SHA-256 key, TTL, WAL mode)
5. `_config.py` — package constants (version, User-Agent)
## Default settings
`search.py` ships with opinionated defaults tuned for AI research:
| Setting | Default | Flag to override |
|---------|---------|------------------|
| Instance | **required** — via `-i`, `SEARXNG_INSTANCE` env var, or config file | `-i / --instance` |
| Safe search | **0 (off)** | `-s / --safesearch {0,1,2}` |
| Time range | **year** | `-t / --time-range {day,month,year,none}` (none = disabled) |
| Output format | **json** | `-f / --format {json,brief,urls,csv}` |
| Engines | **google,bing,brave,duckduckgo,startpage,wikipedia,wikidata** | `--engines <list>` |
## Quick Start
```bash
# Prerequisites: Python 3.8+
# Optional but recommended:
pip install requests beautifulsoup4
# 1. Search against YOUR instance (instance URL is required)
python scripts/search.py -q "python asyncio tutorial" -i https://my-searxng.example.com
# 2. Multiple instances for failover (comma-separated)
python scripts/search.py -q "rust memory safety" \
-i https://a.example.com,https://b.example.com --format brief
# 3. Search + auto-fetch top 3 result pages in one command
python scripts/search.py -q "climate policy" -i https://my-searxng.example.com --fetch 3
# 4. Fetch a result page
python scripts/fetch.py -u "https://example.com" --extract text
# 5. Skip -i by configuring the instance once (env var, current shell)
export SEARXNG_INSTANCE="https://my-searxng.example.com,https://backup.example.com"
python scripts/search.py -q "python asyncio tutorial" # -i not needed
# 6. Or use a config file (./searxng.toml or ~/.config/searxng-cli/searxng.toml)
# [searxng]
# instance = "https://my-searxng.example.com"
# # or: instances = ["https://a.example.com", "https://b.example.com"]
# # Any flag below can also be pre-set here (engines, categories, language,
# # safesearch, time_range, method, format, timeout, max_retries, proxy,
# # cache_ttl, fetch, fetch_timeout, fetch_retries, max_size).
# # Explicit CLI flags always override config values.
# Plain list also works in ./instances.txt (one URL per line, # for comments)
# 7. Cache results for 30 minutes (identical queries skip the network)
python scripts/search.py -q "python asyncio" -i https://s.example.com --cache-ttl 30
# 7b. Sort by date (newest first) or disable dedup for raw engine output
python scripts/search.py -q "ai news" -i https://s.example.com --sort-by date --no-dedup
# 8. Batch: run queries from a file (one per line; blank/# lines skipped)
python scripts/search.py --queries-file queries.txt -i https://s.example.com --format json > batch.json
# 9. Domain allowlist + blocklist (applied after search)
python scripts/search.py -q "rust async" -i https://s.example.com \
--include-domain doc.rust-lang.org,wikipedia.org --exclude-domain pinterest.com
# 10. Route through a corporate proxy (applies to search and fetch)
python scripts/search.py -q "ai news" -i https://s.example.com --proxy http://corp-proxy:8080
# 11. Auth from a file (avoids leaking tokens in shell history)
python scripts/search.py -q "test" -i https://private.example.com --auth-bearer-file ~/.searxng_token
# 12. Cache management (no search performed)
python scripts/search.py --cache-stats # entry count, age, size, path
python scripts/search.py --clear-cache # delete all entries
# 13. Export results as CSV (great for spreadsheets / data analysis)
python scripts/search.py -q "rust async" -i https://s.example.com --format csv > results.csv
# 14. Use a specific config file (overrides auto-discovered searxng.toml)
python scripts/search.py --config ./my-config.toml -q "test"
# 15. Control log verbosity on stderr
python scripts/search.py -q "test" -i https://s.example.com --verbose # debug detail
python scripts/search.py -q "test" -i https://s.example.com --quiet # errors only
```
**Dependency levels:**
| Level | Scripts | What you get |
|-------|---------|-------------|
| Zero deps (stdlib only) | `search.py`, `_config.py` | Full search + auto-fetch |
| `pip install requests` | `fetch.py` | Better HTTP (session reuse, redirect handling) |
| `pip install beautifulsoup4` | `fetch.py` | Higher-quality text extraction |
## When to Use
- **Web search without API keys** — programmatic search results against your own SearXNG instance
- **Privacy-conscious research** — queries routed through your instance, not ad-tech infrastructure
- **Scraping search results** — batch query multiple terms and collect structured results as JSON
- **Fetching search result pages** — follow links from search results and extract clean text
**Don't use for:**
- High-frequency production search without a properly-scaled instance — respect your instance's rate limits
- Guaranteed uptime/accuracy — depends entirely on the instance you supply
## Cross-Agent Compatibility
These scripts are **agent-agnostic** — they work with any AI agent that can invoke terminal commands:
| Agent | How to invoke |
|-------|--------------|
| **Hermes** | `python scripts/search.py -q "..." -i https://your-instance` |
| **Claude Code** | Same — call via terminal tool |
| **Codex (OpenAI)** | Same — call via terminal tool |
| **OpenCode** | Same — call via terminal tool |
| **Cursor** | Same — call via terminal tool |
| **Trae** | Same — call via terminal tool |
| **Standalone (human)** | Run directly in any terminal |
**Key design decisions for universal compatibility:**
- Zero external dependencies (stdlib-only for `search.py`)
- Scripts inject their own directory into `sys.path`, so they run from **any** working directory
- Stdout carries data (JSON/text), stderr carries progress/warnings
- Exit codes: 0=success, 1=fatal error, 2=no results/empty
- NO agent-specific API calls or tool dependencies — purely CLI-based, portable across all agent platforms
**Standalone usage (no agent):**
```bash
# Clone or download the scripts/ directory, then run from anywhere:
python /path/to/scripts/search.py -q "your query" -i https://your-instance
python /path/to/scripts/fetch.py -u "https://example.com"
```
## SearXNG Search API (Quick Reference)
Source: [docs.searxng.org/dev/search_api.html](https://docs.searxng.org/dev/search_api.html)
```
GET/POST /search?q=<query>&format=json GET /search
POST /search (form data) POST /
```
**Parameters**
| Parameter | Required | Values | Description |
|-------------|----------|---------------------------|--------------------------------------------------|
| `q` | yes | string | Search query (supports engine syntax like `site:`)|
| `format` | no | `json`, `csv`, `rss` | Output format (**many instances only allow html**) |
| `categories`| no | comma-separated | e.g. `general,images,news,science` |
| `language` | no | language code | e.g. `en`, `zh-CN`, `de` |
| `pageno` | no | integer (default 1) | Result page number — **fully supported** by search.py |
| `time_range`| no | `day`, `month`, `year`, `none` | Time filter (search.py default: `year`; `none` = disabled) |
| `safesearch`| no | `0`, `1`, `2` | Safe search (search.py default: `0` = off) |
| `engines` | no | comma-separated | search.py default: `google,bing,brave,duckduckgo,startpage,wikipedia,wikidata` |
### Authentication
SearXNG supports `Authorization` headers for private instances (configured via `settings.yml` → `server.secret_key`). Both `search.py` and `fetch.py` accept:
```bash
# Bearer token (most common for API-style auth)
python search.py -q "test" --auth-bearer "sk-abc123..." -i https://private-searx.example.com
# Basic auth (base64-encoded user:pass)
python search.py -q "test" --auth-basic "admin:secret123" -i https://private-searx.example.com
# fetch.py also supports auth for authenticated endpoints
python fetch.py -u "https://protected.example.com/page" --auth-bearer "tok_xxx"
```
If both are provided, Bearer takes precedence. Auth headers are forwarded to both search API calls and auto-fetched page requests.
**Avoid leaking secrets in shell history** — credentials can be read from a file or env var instead of a CLI flag (priority: CLI flag > file > env var):
```bash
# From a file (first non-empty, non-# line is used)
python search.py -q "test" -i https://private.example.com --auth-bearer-file ~/.searxng_token
python search.py -q "test" -i https://private.example.com --auth-basic-file ~/.searxng_auth
# From an environment variable
export SEARXNG_BEARER_TOKEN="sk-abc123..."
export SEARXNG_BASIC_AUTH="admin:secret123"
python search.py -q "test" -i https://private.example.com # credentials picked up automatically
```
### JSON Response Structure
```json
{
"query": "searxng",
"number_of_results": 1234,
"results": [
{
"title": "...",
"url": "https://...",
"content": "Snippet text...",
"engine": "google",
"score": 0.0,
"category": "general"
}
],
"answers": [],
"corrections": [],
"suggestions": ["searxng docker", "searxng api"],
"infoboxes": [],
"unresponsive_engines": [["bing", "timeout"]]
}
```
**Key gotcha:** Many instances disable `format=json` in their settings. `search.py` auto-detects and falls back to HTML scraping. Enable `format: [html, json]` under `search.formats` in your instance's `settings.yml` to get rich JSON metadata.
## Scripts
All scripts live in `scripts/`; run with `python scripts/<name>.py` from any directory. They import `_config.py` for shared constants and self-inject their own directory into `sys.path`.
### 1. `search.py` — Execute SearXNG Search
```
usage: search.py [-h] [--query QUERY] [--instance URL]
[--categories CATS] [--language LANG] [--pageno N]
[--time-range {day,month,year,none}] [--safesearch {0,1,2}]
[--engines E] [--method {GET,POST}] [--max-results N]
[--format {json,brief,urls,csv}] [--snippet-len N]
[--fetch N] [--fetch-timeout SEC] [--fetch-retries N]
[--max-size BYTES] [--output FILE] [--timeout SEC]
[--retry N] [--fail-fast] [--serial] [--verify]
[--auth-bearer TOKEN] [--auth-bearer-file FILE]
[--auth-basic USER:PASS] [--auth-basic-file FILE]
[--proxy URL] [--include-domain DOMAINS]
[--exclude-domain DOMAINS] [--queries-file FILE]
[--cache-ttl MINUTES] [--clear-cache] [--cache-stats]
[--sort-by {score,date,engine,none}] [--no-dedup]
[--config FILE] [--verbose] [--quiet] [--version]
```
**What it does:**
1. Takes a search query and resolves one or more instance URLs (`-i`, `SEARXNG_INSTANCE`, or config file — comma-separated for failover)
2. **Multi-instance failover:** if an instance fails (429/5xx/timeout/captcha), automatically tries the next one
3. **Exponential backoff:** retries each instance up to 3 times with jitter on transient errors (429, 502, 503, 504, and connection errors)
4. **Parallel probing (default for 2+ instances):** queries every instance concurrently and returns the first *successful* result in your original instance order — this keeps output deterministic while drastically speeding up failover when an early instance is down/slow. Use `--serial` to disable.
5. Calls the SearXNG API (GET or POST) with `format=json`
6. Falls back to HTML scraping if JSON is blocked
7. HTML parser extracts results + suggestions + answers + infoboxes (full content, never truncated)
8. **Stable auto-fetch:** `--fetch 3` concurrently downloads top 3 result pages with retry, browser-UA fallback, CAPTCHA detection, and **`fetch.py`'s higher-quality text extractor** (the same engine `fetch.py` uses)
9. **Health-check mode:** `--verify` probes each instance (reachability / JSON-API support / latency / POST support / engine list / auth status) and prints a report, then exits without searching — use it to validate your instance list
10. **Result caching:** `--cache-ttl 30` stores results for 30 min; identical queries within the TTL skip the network entirely. Cache lives at `$SEARXNG_CACHE_DIR` or `~/.cache/searxng-cli/cache.db` (SQLite, WAL mode). `--clear-cache` / `--cache-stats` manage it without searching
11. **Batch mode:** `--queries-file FILE` reads one query per line (blank/`#` lines skipped) and runs them in sequence; output is a JSON array (or one block per query in brief/urls). A failed query is recorded but does not abort the batch
12. **Dedup + sort + domain filter:** After search (and cache), duplicate URLs are collapsed (default; `--no-dedup` disables), results are sorted (`--sort-by`; default: score descending), and then `--include-domain`/`--exclude-domain` filter by domain. Matching is case-insensitive and ignores a leading `www.`; when a domain is in both lists, exclude wins
13. **Proxy & auth:** `--proxy URL` routes both search and fetch through a proxy; `--auth-bearer` / `--auth-basic` (plus `*-file` variants and `SEARXNG_BEARER_TOKEN` / `SEARXNG_BASIC_AUTH` env vars) supply credentials without leaking them via shell history
14. **Config defaults:** `searxng.toml` may pre-set most flags (engines, categories, language, safesearch, time_range, method, format, sort_by, timeout, max_retries, proxy, cache_ttl, fetch, fetch_timeout, fetch_retries, max_size); explicit CLI flags always win
15. **Structured errors:** in `--format json` mode, failures print a JSON object `{"error": "...", "exit_code": N, "query": "..."}` to stdout so agents can parse them
**Key options:**
- `--query "your search"` — **required unless** `--verify`, `--queries-file`, `--clear-cache`, or `--cache-stats` is used
- `--instance https://searx.example.org` — **required unless** `SEARXNG_INSTANCE` env var or a config file supplies it; comma-separated list enables failover
- `--queries-file FILE` — read queries from a file (one per line; blank/`#` skipped) and run them in sequence; overrides `--query`
- `--engines google,duckduckgo` — restrict to specific search engines (whitespace around commas is auto-stripped; see default list above)
- `--method POST` — use POST instead of GET (better for long queries)
- `--categories general,news` — comma-separated categories (whitespace auto-stripped)
- `--language zh-CN` — language filter
- `--pageno 1` — page number
- `--time-range {day,month,year,none}` — time filter (default: `year`; `none` disables time filtering)
- `--safesearch {0,1,2}` — safe search (default: `0` = off)
- `--max-results N` — limit number of results (applied AFTER dedup+sort, so the highest-scoring/newest items are kept)
- `--sort-by {score,date,engine,none}` — sort results (default: `score` descending; `none` preserves instance order). Applied after dedup, before `--max-results`. HTML-fallback results have no score and keep their order
- `--no-dedup` — disable cross-engine deduplication (by default, duplicate URLs — same page ignoring tracking params/fragment — are collapsed, keeping the first occurrence's engine/score)
- `--config FILE` — path to a `searxng.toml` config file; overrides the default auto-discovery (`./searxng.toml` → `~/.config/searxng-cli/searxng.toml`). Must be the first flag so its values can set defaults for other flags
- `--verbose` / `-v` — show debug-level diagnostics on stderr (HTTP request URLs, response codes, cache keys, retry detail)
- `--quiet` — suppress progress messages and retry notices on stderr; only warnings and errors are shown (no short flag: `-q` is `--query`)
- `--include-domain a.com,b.org` — allowlist; only results from these domains are kept (applied after search)
- `--exclude-domain pinterest.com` — blocklist; results from these domains are dropped (applied after search; wins over include on conflict)
- `--serial` — disable parallel multi-instance probing; search instances strictly one at a time
- `--verify` — health-check mode: verify instances and exit (no search); combine with `--format brief` for a table or `--format json` for machine-readable output
- `--format json` — full JSON (default); includes `fetched` array when `--fetch` is used; errors are emitted as JSON to stdout
- `--format brief` — title + URL + full snippet (no truncation by default; use `--snippet-len 200` to cap)
- `--format urls` — only result URLs
- `--format csv` — CSV export (title,url,engine,score,published_date,content); in batch mode (`--queries-file`), all queries merge into one CSV with a `query` column
- `--fetch N` — after search, auto-fetch full text of top N result pages (concurrent, stdlib only)
- `--fetch-timeout 10` — timeout per page fetch (default: 10s)
- `--fetch-retries 3` — max retries per page fetch
- `--max-size BYTES` — cap page size (default: unlimited; e.g. `5242880` for 5MB)
- `--retry 5` — max retries per instance (default: 3)
- `--timeout 15` — request timeout in seconds
- `--fail-fast` — use only the first instance, don't fail over to the rest
- `--proxy URL` — HTTP/HTTPS proxy for both search and fetch (e.g. `http://corp-proxy:8080`)
- `--cache-ttl MINUTES` — cache results for N minutes (default: `0` = disabled); identical queries within the TTL skip the network
- `--clear-cache` — delete all cached entries and exit (no search)
- `--cache-stats` — print cache statistics (entries, age, size, path) and exit
- `--auth-bearer TOKEN` — `Authorization: Bearer` header for private instances
- `--auth-bearer-file FILE` — read Bearer token from a file (first non-empty, non-`#` line); also honors `SEARXNG_BEARER_TOKEN` env var
- `--auth-basic USER:PASS` — `Authorization: Basic` header (auto base64-encoded)
- `--auth-basic-file FILE` — read `user:pass` from a file (first non-empty, non-`#` line); also honors `SEARXNG_BASIC_AUTH` env var
- `--version` — print version and exit
**Completion criterion:** Outputs valid JSON with `results` array. Non-zero exit on total failure (all instances exhausted).
### 2. `fetch.py` — Fetch & Extract Web Page Content
```
usage: fetch.py [-h] --url URL [--extract {text,html,markdown}]
[--timeout SEC] [--retries N] [--max-size BYTES]
[--user-agent STR] [--encoding CHARSET]
[--no-redirect] [--proxy URL] [--output FILE]
[--auth-bearer TOKEN] [--auth-bearer-file FILE]
[--auth-basic USER:PASS] [--auth-basic-file FILE]
[--verbose] [--quiet] [--version]
```
**What it does:**
1. Downloads a web page via HTTP GET with retry + exponential backoff
2. **Stable fetching:** retries on 429/5xx/connection errors (3x default), falls back to browser User-Agent if blocked
3. **No size limit by default** — full page content returned; use `--max-size` for a cap
4. Detects charset from HTTP headers, HTML meta tags, or UTF-8 fallback
5. Extracts readable content using tree-based parsers (stdlib or BeautifulSoup)
6. Outputs clean text, raw HTML, or properly-converted Markdown (with correct nested-link handling)
**Key options:**
- `--url https://...` — required
- `--extract text` — clean readable text (default)
- `--extract html` — raw HTML
- `--extract markdown` — Markdown conversion (tree-based; handles nested tags, GFM tables, fenced code blocks, blockquotes, inline code, ordered/unordered/nested lists, definition lists, images, emphasis)
- `--encoding gbk` — force charset for non-UTF-8 pages (else auto-detected from HTTP header / HTML meta / UTF-8 fallback)
- `--timeout 15` — request timeout in seconds
- `--retries 3` — max retries on transient errors (429/5xx/connection); falls back to a browser User-Agent when blocked
- `--max-size BYTES` — cap page size (default: unlimited; e.g. `5242880` for 5MB)
- `--user-agent STR` — custom User-Agent header
- `--no-redirect` — do **not** follow HTTP 3xx redirects (implemented for both the requests and stdlib paths)
- `--proxy URL` — HTTP/HTTPS proxy (e.g. `http://corp-proxy:8080`); respects existing `HTTP_PROXY`/`HTTPS_PROXY` env vars when omitted
- `--output FILE` — save to file instead of stdout
- `--auth-bearer TOKEN` / `--auth-bearer-file FILE` — `Authorization: Bearer` header; file variant reads first non-empty, non-`#` line; also honors `SEARXNG_BEARER_TOKEN` env var
- `--auth-basic USER:PASS` / `--auth-basic-file FILE` — `Authorization: Basic` header (auto base64-encoded); file variant + `SEARXNG_BASIC_AUTH` env var also supported
- `--verbose` / `-v` — show debug-level diagnostics on stderr
- `--quiet` — suppress progress messages on stderr; only warnings and errors are shown
- `--version` — print version and exit
**Extraction strategy (text mode):**
1. Strip non-content elements (script, style, nav, footer, header)
2. Extract `<article>`, `<main>`, or `<body>` content
3. Collapse whitespace, output clean UTF-8
4. Warn if extracted text < 500 chars (likely JS-heavy or bot-blocked)
**Completion criterion:** Outputs page content. Non-zero exit on HTTP failure. Stderr carries warnings for low-confidence extraction.
## Common Workflow
```bash
INSTANCE=https://my-searxng.example.com
# 1. Search + auto-fetch top 3 result pages (uses defaults: safesearch off,
# time_range year, json output, the 7 default engines)
python scripts/search.py -q "python asyncio tutorial" -i "$INSTANCE" --fetch 3
# 2. Brief format with full snippets (no truncation by default)
python scripts/search.py -q "rust memory safety" -i "$INSTANCE" --format brief --fetch 2
# 3. Pipe-friendly: search -> extract first URL
TOP_URL=$(python scripts/search.py -q "rust book" -i "$INSTANCE" --format urls --fail-fast | head -1)
# 4. Override engines and use POST for a long query
python scripts/search.py -q "ai news" -i "$INSTANCE" --engines google,duckduckgo \
--method POST --fetch 5 --fetch-timeout 15 --format json
# 5. Failover across two instances
python scripts/search.py -q "quantum computing" -i "$INSTANCE,https://backup.example.com"
```
**JSON vs HTML fallback difference:** When an instance supports `format=json`, search results include rich metadata (engine name, score, category) and `fetched_source` is `"json"`. With HTML fallback, results are less structured and `fetched_source` is `"html"`. The auto-fetch feature works identically in both cases.
## Troubleshooting
### `--instance` is required (unless configured)
Public-instance discovery has been removed. You must supply an instance, but you have three options (in priority order):
1. `-i https://your-instance` (CLI flag, comma-separated for failover)
2. `SEARXNG_INSTANCE` environment variable (e.g. `export SEARXNG_INSTANCE="https://a,https://b"`)
3. A config file: `./searxng.toml` or `~/.config/searxng-cli/searxng.toml` (`instance = "..."` or `instances = [...]`), or `./instances.txt` / `~/.config/searxng-cli/instances.txt` (one URL per line, `#` comments)
Self-host SearXNG (Docker: `docker run -d -p 8080:8080 searxng/searxng`) or use an instance you trust.
### All instances return 429 / timeout
Your instance is rate-limited. Wait a few minutes, pass multiple instances via `-i a,b`, or tune your instance's limiter settings.
### Validate your instance list with `--verify`
Before relying on a multi-instance setup, run `python scripts/search.py --verify -i a,b,c` (optionally with `--format brief` for a human-readable table). It reports each instance's reachability, JSON-API support, and latency, then exits without searching. Combine with `SEARXNG_INSTANCE` or a config file to check your persisted list in one go.
### JSON format blocked (HTML fallback)
Some instances disable `format=json`. Enable `search.formats: [html, json]` in your instance's `settings.yml`. The HTML fallback parser handles the rest automatically.
### Certificate errors on Windows
Some instances use Let's Encrypt certificates. Run `pip install certifi` or upgrade Python's certifi bundle.
### fetch.py output is empty or gibberish
Try `--encoding gbk` for Chinese sites, `--encoding shift_jis` for Japanese. If the page requires JavaScript, use a headless browser instead. If a page 3xx-redirects and you passed `--no-redirect`, an empty body is expected (that's the redirect response itself).
### Caching, proxy, domain filter, batch
- **Cache:** `--cache-ttl 30` caches results for 30 min; repeat the same query and you'll see `[cache hit]` on stderr. `--cache-stats` shows entries/age/size/path; `--clear-cache` wipes them. The cache key covers query + engines + categories + language + time_range + safesearch + pageno + method — different params get separate entries. Cache lives at `$SEARXNG_CACHE_DIR` or `~/.cache/searxng-cli/cache.db` (SQLite, WAL mode).
- **Proxy:** `--proxy http://corp-proxy:8080` sets `HTTP_PROXY`/`HTTPS_PROXY` for both search and fetch; `NO_PROXY` defaults to `localhost,127.0.0.1,::1` so local traffic stays direct. Existing proxy env vars are honored when `--proxy` is omitted.
- **Domain filter:** `--include-domain` (allowlist) and `--exclude-domain` (blocklist) run after search. Matching is case-insensitive and ignores a leading `www.`. If a domain appears in both lists, exclude wins (the result is dropped).
- **Batch:** `--queries-file FILE` runs one query per line (blank/`#` skipped) in sequence. JSON output is an array `[{"query":..., "results":...}, ...]`; a failed query becomes `{"query":..., "error":...}` but does not abort the batch. Exit 0 if any query succeeded, 1 only if all failed.
- **Config defaults:** `searxng.toml` can pre-set most flags (see the Quick Start example). Explicit CLI flags always override config values; string-valued ints from TOML are normalized automatically.
### "ModuleNotFoundError: No module named '_config'"
Each script self-injects its directory into `sys.path`, so this should no longer occur. If it does, ensure `_config.py` sits next to `search.py` / `fetch.py` in the same `scripts/` directory.
## Common Pitfalls
1. **JSON format blocked.** Many instances disable `format=json`. Enable it in your instance settings for rich metadata; otherwise the HTML fallback parser is used.
2. **Rate limiting.** `search.py` retries each instance up to 3 times with exponential backoff and jitter, then fails over to the next instance. Supply multiple instances via `-i a,b` for resilience.
3. **fetch.py extraction quality varies.** The heuristic text extractor works well on articles and documentation but poorly on SPAs, login walls, and JavaScript-heavy pages. For JS-heavy pages, use a headless browser.
4. **Encoding issues on Windows.** The scripts output UTF-8. On cmd.exe, run `chcp 65001` first. PowerShell handles UTF-8 natively. For non-UTF-8 pages, use `fetch.py --encoding gbk`.
5. **POST vs GET.** Some instances handle POST differently or block it entirely. If POST search fails, try the default GET method.
6. **Overriding defaults.** Remember the new defaults (safesearch off, time_range year, the 7-engine list). Pass the corresponding flag to change any of them per query.
7. **Cache vs. freshness.** `--cache-ttl` returns cached results without hitting the network — fast, but stale. For time-sensitive queries (news, prices), use `--cache-ttl 0` or `--clear-cache`; the cache key includes `time_range`/`pageno` but not wall-clock time, so a `day`-range query cached at 09:00 is served as-is until the TTL expires.
8. **Domain filter semantics.** `--include-domain`/`--exclude-domain` run *after* search and only prune the already-returned results — they do not make the instance fetch more. An overly strict allowlist can yield zero results. Exclude wins over include when a domain is in both lists.
9. **Dedup changes output.** Dedup is on by default — duplicate URLs (ignoring `utm_*`/`gclid`/fragment, normalizing scheme/host case and param order) are collapsed. Use `--no-dedup` if you need the raw per-engine result set (e.g., comparing engine coverage).
10. **Sort changes `--max-results` behavior.** With the default `--sort-by score`, `--max-results N` keeps the *highest-scoring* N results, not the first N in instance order. Use `--sort-by none` to preserve the original order before limiting.
## Verification Checklist
- [ ] `python scripts/search.py -q "test" -i <URL> --format json` returns results
- [ ] `python scripts/search.py -q "test" -i <URL> --method POST` works (optional)
- [ ] `python scripts/search.py -q "test" -i a,b` fails over across instances
- [ ] `python scripts/fetch.py -u "https://example.com"` returns readable text
- [ ] `python scripts/fetch.py -u "https://example.com" -e markdown` produces valid markdown
- [ ] `python scripts/fetch.py -u "http://<redirecting>" --no-redirect` does not follow the redirect
- [ ] All scripts have `--help` and `--version`
- [ ] Scripts run from any working directory (sys.path self-injection)
- [ ] Scripts exit 0 on success, non-zero on failure
- [ ] `python scripts/search.py --verify -i <URL>` reports instance health (reachable / JSON support / latency), exit 0
- [ ] `python scripts/search.py -q "test" -i <URL>` shows `Dedup:` on stderr when duplicates exist; `--no-dedup` suppresses it
- [ ] `python scripts/search.py -q "test" -i <URL> --sort-by date` returns newest-first; `--sort-by none` preserves instance order
- [ ] `python scripts/search.py -q "test" -i <URL> --format csv` outputs a CSV with header `title,url,engine,score,published_date,content`
- [ ] `python scripts/search.py --config ./my-config.toml -q "test"` loads the specified config and applies its defaults
- [ ] `python scripts/search.py -q "test" -i <URL> --verbose` shows debug-level diagnostics on stderr
- [ ] `python scripts/search.py -q "test" -i <URL> --quiet` suppresses progress messages; only warnings/errors on stderr
- [ ] `python scripts/fetch.py -u <URL> --verbose` shows debug-level diagnostics on stderr
- [ ] `python scripts/search.py -q "test" -i <URL> --cache-ttl 30` then re-run → `[cache hit]` on stderr
- [ ] `python scripts/search.py --cache-stats` prints entries/size/path; `--clear-cache` reports count deleted
- [ ] `python scripts/search.py --queries-file queries.txt -i <URL> --format json` emits a JSON array
- [ ] `python scripts/search.py -q "test" -i <URL> --include-domain example.com --exclude-domain spam.com` filters as expected
- [ ] `python scripts/search.py -q "test" -i <URL> --proxy http://proxy:8080` routes through the proxy
- [ ] `python scripts/search.py -q "test" -i <URL> --auth-bearer-file <FILE>` authenticates without CLI token leakage
- [ ] A failing search with `--format json` emits `{"error":..., "exit_code":1}` to stdout
- [ ] `python scripts/fetch.py -u <URL> --proxy http://proxy:8080` routes through the proxy
- [ ] `python scripts/fetch.py -u <URL> --auth-bearer-file <FILE>` authenticates
- [ ] Stderr carries warnings; stdout carries data