diff --git a/SKILL.md b/SKILL.md index 419e7b3..94c8444 100644 --- a/SKILL.md +++ b/SKILL.md @@ -1,4 +1,4 @@ ---- +--- name: searxng-use-cli description: Use when you need to search the web via your OWN SearXNG instance (no public-instance discovery). 3 CLI scripts + a shared common.py module — execute privacy-respecting searches against a user-supplied instance (with multi-instance failover, 5xx/429 retry, auto-fetch) or via SEARXNG_INSTANCE env / config file, fetch/extract readable text or markdown from web pages. Zero-config replacement for proprietary search APIs. version: 1.8.0 @@ -339,17 +339,7 @@ fields only appear when `--fetch N` is used. ## Cross-Agent Compatibility -These scripts are **agent-agnostic** — they work with any AI agent that can invoke terminal commands: - -| Agent | How to invoke | -|-------|--------------| -| **Hermes** | `python scripts/search.py -q "..." -i https://your-instance` | -| **Claude Code** | Same — call via terminal tool | -| **Codex (OpenAI)** | Same — call via terminal tool | -| **OpenCode** | Same — call via terminal tool | -| **Cursor** | Same — call via terminal tool | -| **Trae** | Same — call via terminal tool | -| **Standalone (human)** | Run directly in any terminal | +These scripts are **agent-agnostic** — they work with any AI agent that can invoke terminal commands (Hermes, Claude Code, Codex, OpenCode, Cursor, Trae, etc.), or directly from a terminal. **Key design decisions for universal compatibility:** - Zero external dependencies (stdlib-only for `search.py`) @@ -358,91 +348,6 @@ These scripts are **agent-agnostic** — they work with any AI agent that can in - Exit codes: 0=success, 1=fatal error, 2=no results/empty - NO agent-specific API calls or tool dependencies — purely CLI-based, portable across all agent platforms -**Standalone usage (no agent):** -```bash -# Clone or download the scripts/ directory, then run from anywhere: -python /path/to/scripts/search.py -q "your query" -i https://your-instance -python /path/to/scripts/fetch.py -u "https://example.com" -``` - -## SearXNG Search API (Quick Reference) - -Source: [docs.searxng.org/dev/search_api.html](https://docs.searxng.org/dev/search_api.html) - -``` -GET/POST /search?q=&format=json GET /search -POST /search (form data) POST / -``` - -**Parameters** - -| Parameter | Required | Values | Description | -|-------------|----------|---------------------------|--------------------------------------------------| -| `q` | yes | string | Search query (supports engine syntax like `site:`)| -| `format` | no | `json`, `csv`, `rss` | Output format (**many instances only allow html**) | -| `categories`| no | comma-separated | e.g. `general,images,news,science` | -| `language` | no | language code | e.g. `en`, `zh-CN`, `de` | -| `pageno` | no | integer (default 1) | Result page number — **fully supported** by search.py | -| `time_range`| no | `day`, `month`, `year`, `none` | Time filter (search.py default: `year`; `none` = disabled) | -| `safesearch`| no | `0`, `1`, `2` | Safe search (search.py default: `0` = off) | -| `engines` | no | comma-separated | search.py default: `google,bing,brave,duckduckgo,startpage,wikipedia,wikidata` | - -### Authentication - -SearXNG supports `Authorization` headers for private instances (configured via `settings.yml` → `server.secret_key`). Both `search.py` and `fetch.py` accept: - -```bash -# Bearer token (most common for API-style auth) -python search.py -q "test" --auth-bearer "sk-abc123..." -i https://private-searx.example.com - -# Basic auth (base64-encoded user:pass) -python search.py -q "test" --auth-basic "admin:secret123" -i https://private-searx.example.com - -# fetch.py also supports auth for authenticated endpoints -python fetch.py -u "https://protected.example.com/page" --auth-bearer "tok_xxx" -``` - -If both are provided, Bearer takes precedence. Auth headers are forwarded to both search API calls and auto-fetched page requests. - -**Avoid leaking secrets in shell history** — credentials can be read from a file or env var instead of a CLI flag (priority: CLI flag > file > env var): - -```bash -# From a file (first non-empty, non-# line is used) -python search.py -q "test" -i https://private.example.com --auth-bearer-file ~/.searxng_token -python search.py -q "test" -i https://private.example.com --auth-basic-file ~/.searxng_auth - -# From an environment variable -export SEARXNG_BEARER_TOKEN="sk-abc123..." -export SEARXNG_BASIC_AUTH="admin:secret123" -python search.py -q "test" -i https://private.example.com # credentials picked up automatically -``` - -### JSON Response Structure - -```json -{ - "query": "searxng", - "number_of_results": 1234, - "results": [ - { - "title": "...", - "url": "https://...", - "content": "Snippet text...", - "engine": "google", - "score": 0.0, - "category": "general" - } - ], - "answers": [], - "corrections": [], - "suggestions": ["searxng docker", "searxng api"], - "infoboxes": [], - "unresponsive_engines": [["bing", "timeout"]] -} -``` - -**Key gotcha:** Many instances disable `format=json` in their settings. `search.py` auto-detects and falls back to HTML scraping. Enable `format: [html, json]` under `search.formats` in your instance's `settings.yml` to get rich JSON metadata. - ## Scripts All scripts live in `scripts/`; run with `python scripts/.py` from any directory. They import `_config.py` for shared constants and self-inject their own directory into `sys.path`. @@ -576,114 +481,3 @@ usage: fetch.py [-h] --url URL [--extract {text,html,markdown}] **Completion criterion:** Outputs page content. Non-zero exit on HTTP failure. Stderr carries warnings for low-confidence extraction. -## Common Workflow - -```bash -INSTANCE=https://my-searxng.example.com - -# 1. Search + auto-fetch top 3 result pages (uses defaults: safesearch off, -# time_range year, json output, the 7 default engines) -python scripts/search.py -q "python asyncio tutorial" -i "$INSTANCE" --fetch 3 - -# 2. Brief format with full snippets (no truncation by default) -python scripts/search.py -q "rust memory safety" -i "$INSTANCE" --format brief --fetch 2 - -# 3. Pipe-friendly: search -> extract first URL -TOP_URL=$(python scripts/search.py -q "rust book" -i "$INSTANCE" --format urls --fail-fast | head -1) - -# 4. Override engines and use POST for a long query -python scripts/search.py -q "ai news" -i "$INSTANCE" --engines google,duckduckgo \ - --method POST --fetch 5 --fetch-timeout 15 --format json - -# 5. Failover across two instances -python scripts/search.py -q "quantum computing" -i "$INSTANCE,https://backup.example.com" -``` - -**JSON vs HTML fallback difference:** When an instance supports `format=json`, search results include rich metadata (engine name, score, category) and `fetched_source` is `"json"`. With HTML fallback, results are less structured and `fetched_source` is `"html"`. The auto-fetch feature works identically in both cases. - -## Troubleshooting - -### `--instance` is required (unless configured) -Public-instance discovery has been removed. You must supply an instance, but you have three options (in priority order): -1. `-i https://your-instance` (CLI flag, comma-separated for failover) -2. `SEARXNG_INSTANCE` environment variable (e.g. `export SEARXNG_INSTANCE="https://a,https://b"`) -3. A config file: `./searxng.toml` or `~/.config/searxng-cli/searxng.toml` (`instance = "..."` or `instances = [...]`), or `./instances.txt` / `~/.config/searxng-cli/instances.txt` (one URL per line, `#` comments) - -Self-host SearXNG (Docker: `docker run -d -p 8080:8080 searxng/searxng`) or use an instance you trust. - -### All instances return 429 / timeout -Your instance is rate-limited. Wait a few minutes, pass multiple instances via `-i a,b`, or tune your instance's limiter settings. - -### Validate your instance list with `--verify` -Before relying on a multi-instance setup, run `python scripts/search.py --verify -i a,b,c` (optionally with `--format brief` for a human-readable table). It reports each instance's reachability, JSON-API support, and latency, then exits without searching. Combine with `SEARXNG_INSTANCE` or a config file to check your persisted list in one go. - -### JSON format blocked (HTML fallback) -Some instances disable `format=json`. Enable `search.formats: [html, json]` in your instance's `settings.yml`. The HTML fallback parser handles the rest automatically. - -### Certificate errors on Windows -Some instances use Let's Encrypt certificates. Run `pip install certifi` or upgrade Python's certifi bundle. - -### fetch.py output is empty or gibberish -Try `--encoding gbk` for Chinese sites, `--encoding shift_jis` for Japanese. If the page requires JavaScript, use a headless browser instead. If a page 3xx-redirects and you passed `--no-redirect`, an empty body is expected (that's the redirect response itself). - -### Caching, proxy, domain filter, batch -- **Cache:** `--cache-ttl 30` caches results for 30 min; repeat the same query and you'll see `[cache hit]` on stderr. `--cache-stats` shows entries/age/size/path; `--clear-cache` wipes them. The cache key covers query + engines + categories + language + time_range + safesearch + pageno + method — different params get separate entries. Cache lives at `$SEARXNG_CACHE_DIR` or `~/.cache/searxng-cli/cache.db` (SQLite, WAL mode). -- **Proxy:** `--proxy http://corp-proxy:8080` sets `HTTP_PROXY`/`HTTPS_PROXY` for both search and fetch; `NO_PROXY` defaults to `localhost,127.0.0.1,::1` so local traffic stays direct. Existing proxy env vars are honored when `--proxy` is omitted. -- **Domain filter:** `--include-domain` (allowlist) and `--exclude-domain` (blocklist) run after search. Matching is case-insensitive and ignores a leading `www.`. If a domain appears in both lists, exclude wins (the result is dropped). -- **Batch:** `--queries-file FILE` runs one query per line (blank/`#` skipped) in sequence. JSON output is an array `[{"query":..., "results":...}, ...]`; a failed query becomes `{"query":..., "error":...}` but does not abort the batch. Exit 0 if any query succeeded, 1 only if all failed. -- **Config defaults:** `searxng.toml` can pre-set most flags (see the Quick Start example). Explicit CLI flags always override config values; string-valued ints from TOML are normalized automatically. - -### "ModuleNotFoundError: No module named '_config'" -Each script self-injects its directory into `sys.path`, so this should no longer occur. If it does, ensure `_config.py` sits next to `search.py` / `fetch.py` in the same `scripts/` directory. - -## Common Pitfalls - -1. **JSON format blocked.** Many instances disable `format=json`. Enable it in your instance settings for rich metadata; otherwise the HTML fallback parser is used. - -2. **Rate limiting.** `search.py` retries each instance up to 3 times with exponential backoff and jitter, then fails over to the next instance. Supply multiple instances via `-i a,b` for resilience. - -3. **fetch.py extraction quality varies.** The heuristic text extractor works well on articles and documentation but poorly on SPAs, login walls, and JavaScript-heavy pages. For JS-heavy pages, use a headless browser. - -4. **Encoding issues on Windows.** The scripts output UTF-8. On cmd.exe, run `chcp 65001` first. PowerShell handles UTF-8 natively. For non-UTF-8 pages, use `fetch.py --encoding gbk`. - -5. **POST vs GET.** Some instances handle POST differently or block it entirely. If POST search fails, try the default GET method. - -6. **Overriding defaults.** Remember the new defaults (safesearch off, time_range year, the 7-engine list). Pass the corresponding flag to change any of them per query. - -7. **Cache vs. freshness.** `--cache-ttl` returns cached results without hitting the network — fast, but stale. For time-sensitive queries (news, prices), use `--cache-ttl 0` or `--clear-cache`; the cache key includes `time_range`/`pageno` but not wall-clock time, so a `day`-range query cached at 09:00 is served as-is until the TTL expires. - -8. **Domain filter semantics.** `--include-domain`/`--exclude-domain` run *after* search and only prune the already-returned results — they do not make the instance fetch more. An overly strict allowlist can yield zero results. Exclude wins over include when a domain is in both lists. - -9. **Dedup changes output.** Dedup is on by default — duplicate URLs (ignoring `utm_*`/`gclid`/fragment, normalizing scheme/host case and param order) are collapsed. Use `--no-dedup` if you need the raw per-engine result set (e.g., comparing engine coverage). - -10. **Sort changes `--max-results` behavior.** With the default `--sort-by score`, `--max-results N` keeps the *highest-scoring* N results, not the first N in instance order. Use `--sort-by none` to preserve the original order before limiting. - -## Verification Checklist - -- [ ] `python scripts/search.py -q "test" -i --format json` returns results -- [ ] `python scripts/search.py -q "test" -i --method POST` works (optional) -- [ ] `python scripts/search.py -q "test" -i a,b` fails over across instances -- [ ] `python scripts/fetch.py -u "https://example.com"` returns readable text -- [ ] `python scripts/fetch.py -u "https://example.com" -e markdown` produces valid markdown -- [ ] `python scripts/fetch.py -u "http://" --no-redirect` does not follow the redirect -- [ ] All scripts have `--help` and `--version` -- [ ] Scripts run from any working directory (sys.path self-injection) -- [ ] Scripts exit 0 on success, non-zero on failure -- [ ] `python scripts/search.py --verify -i ` reports instance health (reachable / JSON support / latency), exit 0 -- [ ] `python scripts/search.py -q "test" -i ` shows `Dedup:` on stderr when duplicates exist; `--no-dedup` suppresses it -- [ ] `python scripts/search.py -q "test" -i --sort-by date` returns newest-first; `--sort-by none` preserves instance order -- [ ] `python scripts/search.py -q "test" -i --format csv` outputs a CSV with header `title,url,engine,score,published_date,content` -- [ ] `python scripts/search.py --config ./my-config.toml -q "test"` loads the specified config and applies its defaults -- [ ] `python scripts/search.py -q "test" -i --verbose` shows debug-level diagnostics on stderr -- [ ] `python scripts/search.py -q "test" -i --quiet` suppresses progress messages; only warnings/errors on stderr -- [ ] `python scripts/fetch.py -u --verbose` shows debug-level diagnostics on stderr -- [ ] `python scripts/search.py -q "test" -i --cache-ttl 30` then re-run → `[cache hit]` on stderr -- [ ] `python scripts/search.py --cache-stats` prints entries/size/path; `--clear-cache` reports count deleted -- [ ] `python scripts/search.py --queries-file queries.txt -i --format json` emits a JSON array -- [ ] `python scripts/search.py -q "test" -i --include-domain example.com --exclude-domain spam.com` filters as expected -- [ ] `python scripts/search.py -q "test" -i --proxy http://proxy:8080` routes through the proxy -- [ ] `python scripts/search.py -q "test" -i --auth-bearer-file ` authenticates without CLI token leakage -- [ ] A failing search with `--format json` emits `{"error":..., "exit_code":1}` to stdout -- [ ] `python scripts/fetch.py -u --proxy http://proxy:8080` routes through the proxy -- [ ] `python scripts/fetch.py -u --auth-bearer-file ` authenticates -- [ ] Stderr carries warnings; stdout carries data