searxng-use-cli/README.md

# SearXNG CLI Toolkit

[![CI](https://git.metona.cn/MetonaTeam/searxng-use-cli/actions/workflows/ci.yml/badge.svg)](https://git.metona.cn/MetonaTeam/searxng-use-cli/actions)
[![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

A privacy-respecting metasearch CLI toolkit that runs against your own SearXNG instance. Zero external dependencies for search (stdlib only), optional `requests` + `beautifulsoup4` for enhanced page fetching. Works with any AI agent (Hermes, Claude Code, Codex, OpenCode, Cursor, Trae) or directly from your terminal.

## Features

**Search & results**
- Multi-instance failover with parallel probing
- Exponential-backoff retry on transient errors (429/5xx/connection)
- `--verify` health-check mode (reachability / JSON-API / latency / POST / engines / auth)
- Cross-engine result deduplication (default on; `--no-dedup` disables)
- Result sorting (`--sort-by {score,date,engine,none}`)
- Domain allowlist/blocklist (`--include-domain` / `--exclude-domain`)
- Batch mode (`--queries-file`)

**Output formats**
- JSON (default), brief, urls, CSV
- Enhanced Markdown conversion (GFM tables, code blocks, blockquotes, nested lists, definition lists)
- Structured JSON error output for machine-readable failure reporting

**Caching & config**
- SQLite result caching (`--cache-ttl`) with TTL management
- `searxng.toml` config file for defaults; `--config FILE` for explicit loading
- Instance resolution: `-i` → `SEARXNG_INSTANCE` env → config file

**Network & auth**
- Proxy support (`--proxy`) for both search and fetch
- Auth via CLI flag, file, or env var (`--auth-bearer` / `--auth-basic` + `*-file` variants)
- Credentials-file permission warning (POSIX)

**Engineering**
- Shared `common.py` (unified retry/charset/auth/logging)
- Structured logging (`--verbose` / `--quiet`)
- 155 unit + integration tests with pytest

## Quick Start

```bash
# Prerequisites: Python 3.8+ (optional: pip install requests beautifulsoup4)

# Search against YOUR instance (instance URL is required)
python scripts/search.py -q "python asyncio tutorial" -i https://my-searxng.example.com

# Multiple instances for failover (comma-separated)
python scripts/search.py -q "rust memory safety" \
  -i https://a.example.com,https://b.example.com --format brief

# Search + auto-fetch top 3 result pages
python scripts/search.py -q "climate policy" -i https://my-searxng.example.com --fetch 3

# Fetch a web page
python scripts/fetch.py -u "https://example.com" --extract markdown

# Skip -i via env var
export SEARXNG_INSTANCE="https://my-searxng.example.com"
python scripts/search.py -q "python asyncio tutorial"
```

## Installation

```bash
git clone https://git.metona.cn/MetonaTeam/searxng-use-cli.git
cd searxng-use-cli

# Zero deps — search.py runs on stdlib alone
python scripts/search.py --version

# Optional: enhanced fetch quality
pip install requests beautifulsoup4
```

## Usage

### search.py — Execute SearXNG Search

```bash
python scripts/search.py -q "your query" -i https://your-instance \
  [--format json|brief|urls|csv] \
  [--engines google,bing,brave] \
  [--time-range day|month|year|none] \
  [--language en] \
  [--sort-by score|date|engine|none] \
  [--no-dedup] \
  [--max-results 10] \
  [--fetch 3] \
  [--cache-ttl 30] \
  [--queries-file queries.txt] \
  [--include-domain example.com] \
  [--exclude-domain spam.com] \
  [--proxy http://corp:8080] \
  [--auth-bearer-file ~/.token] \
  [--verify] \
  [--verbose|-v] [--quiet]
```

### fetch.py — Fetch & Extract Web Page Content

```bash
python scripts/fetch.py -u https://example.com \
  --extract text|html|markdown \
  [--encoding gbk] \
  [--max-size 5242880] \
  [--proxy http://corp:8080] \
  [--auth-bearer-file ~/.token]
```

## Configuration

Instance URLs resolve in priority order:
1. `-i / --instance` (comma-separated for failover)
2. `SEARXNG_INSTANCE` environment variable
3. Config file: `./searxng.toml` → `~/.config/searxng-cli/searxng.toml` → `./instances.txt` → `~/.config/searxng-cli/instances.txt`

Example `searxng.toml`:
```toml
[searxng]
instance = "https://my-searxng.example.com"
# or: instances = ["https://a.example.com", "https://b.example.com"]
engines = "google,bing,brave,duckduckgo,startpage,wikipedia,wikidata"
time_range = "year"
safesearch = 0
format = "json"
cache_ttl = 30
```

## Testing

```bash
pip install pytest
pytest -q
```

155 tests cover: cache operations, auth resolution, domain filtering, Markdown conversion, search logic, integration flows, and logging configuration.

## Project Structure

```
├── scripts/
│   ├── search.py      # Search with multi-instance failover, cache, batch, domain filter
│   ├── fetch.py       # Web page fetcher with text/markdown extraction
│   ├── common.py      # Shared utilities (auth, retry, charset, logging)
│   ├── cache.py       # SQLite-backed result cache
│   └── _config.py     # Version + User-Agent constants
├── tests/             # pytest unit + integration tests
├── .gitea/workflows/  # Gitea Actions CI
├── SKILL.md           # Full skill documentation (agent-facing)
├── pytest.ini         # Test configuration
└── README.md
```

## Defaults

| Setting | Default | Flag |
|---------|---------|------|
| Instance | **required** | `-i` / `SEARXNG_INSTANCE` / config |
| Safe search | 0 (off) | `-s` |
| Time range | year | `-t` |
| Output format | json | `-f` |
| Engines | google,bing,brave,duckduckgo,startpage,wikipedia,wikidata | `--engines` |
| Sort | score descending | `--sort-by` |
| Dedup | on | `--no-dedup` |

## Cross-Agent Compatibility

These scripts are agent-agnostic — they work with any AI agent that can invoke terminal commands:

| Agent | How to invoke |
|-------|--------------|
| Hermes | `python scripts/search.py -q "..." -i https://your-instance` |
| Claude Code | Same — call via terminal tool |
| Codex (OpenAI) | Same — call via terminal tool |
| OpenCode | Same — call via terminal tool |
| Cursor | Same — call via terminal tool |
| Trae | Same — call via terminal tool |
| Standalone (human) | Run directly in any terminal |

Key design decisions for universal compatibility:
- Zero external dependencies (stdlib-only for `search.py`)
- Scripts self-inject their directory into `sys.path` — run from any working directory
- Stdout carries data (JSON/CSV/text), stderr carries progress/warnings
- Exit codes: 0=success, 1=fatal error, 2=no results/empty
- NO agent-specific API calls — purely CLI-based, portable across all agent platforms

## License

MIT