Initial commit: SearXNG CLI Toolkit v1.6.0
Multi-instance failover, exponential-backoff retry, SQLite cache, batch mode, domain filter, cross-engine dedup, result sorting, CSV export, structured logging, enhanced Markdown conversion, 155 pytest tests, Gitea Actions CI
This commit is contained in:
Metona Team
@@ -0,0 +1,190 @@
|
||||
# SearXNG CLI Toolkit
|
||||
|
||||
[](https://git.metona.cn/MetonaTeam/searxng-use-cli/actions)
|
||||
[](https://www.python.org/downloads/)
|
||||
[](LICENSE)
|
||||
|
||||
A privacy-respecting metasearch CLI toolkit that runs against your own SearXNG instance. Zero external dependencies for search (stdlib only), optional `requests` + `beautifulsoup4` for enhanced page fetching. Works with any AI agent (Hermes, Claude Code, Codex, OpenCode, Cursor, Trae) or directly from your terminal.
|
||||
|
||||
## Features
|
||||
|
||||
**Search & results**
|
||||
- Multi-instance failover with parallel probing
|
||||
- Exponential-backoff retry on transient errors (429/5xx/connection)
|
||||
- `--verify` health-check mode (reachability / JSON-API / latency / POST / engines / auth)
|
||||
- Cross-engine result deduplication (default on; `--no-dedup` disables)
|
||||
- Result sorting (`--sort-by {score,date,engine,none}`)
|
||||
- Domain allowlist/blocklist (`--include-domain` / `--exclude-domain`)
|
||||
- Batch mode (`--queries-file`)
|
||||
|
||||
**Output formats**
|
||||
- JSON (default), brief, urls, CSV
|
||||
- Enhanced Markdown conversion (GFM tables, code blocks, blockquotes, nested lists, definition lists)
|
||||
- Structured JSON error output for machine-readable failure reporting
|
||||
|
||||
**Caching & config**
|
||||
- SQLite result caching (`--cache-ttl`) with TTL management
|
||||
- `searxng.toml` config file for defaults; `--config FILE` for explicit loading
|
||||
- Instance resolution: `-i` → `SEARXNG_INSTANCE` env → config file
|
||||
|
||||
**Network & auth**
|
||||
- Proxy support (`--proxy`) for both search and fetch
|
||||
- Auth via CLI flag, file, or env var (`--auth-bearer` / `--auth-basic` + `*-file` variants)
|
||||
- Credentials-file permission warning (POSIX)
|
||||
|
||||
**Engineering**
|
||||
- Shared `common.py` (unified retry/charset/auth/logging)
|
||||
- Structured logging (`--verbose` / `--quiet`)
|
||||
- 155 unit + integration tests with pytest
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Prerequisites: Python 3.8+ (optional: pip install requests beautifulsoup4)
|
||||
|
||||
# Search against YOUR instance (instance URL is required)
|
||||
python scripts/search.py -q "python asyncio tutorial" -i https://my-searxng.example.com
|
||||
|
||||
# Multiple instances for failover (comma-separated)
|
||||
python scripts/search.py -q "rust memory safety" \
|
||||
-i https://a.example.com,https://b.example.com --format brief
|
||||
|
||||
# Search + auto-fetch top 3 result pages
|
||||
python scripts/search.py -q "climate policy" -i https://my-searxng.example.com --fetch 3
|
||||
|
||||
# Fetch a web page
|
||||
python scripts/fetch.py -u "https://example.com" --extract markdown
|
||||
|
||||
# Skip -i via env var
|
||||
export SEARXNG_INSTANCE="https://my-searxng.example.com"
|
||||
python scripts/search.py -q "python asyncio tutorial"
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
git clone https://git.metona.cn/MetonaTeam/searxng-use-cli.git
|
||||
cd searxng-use-cli
|
||||
|
||||
# Zero deps — search.py runs on stdlib alone
|
||||
python scripts/search.py --version
|
||||
|
||||
# Optional: enhanced fetch quality
|
||||
pip install requests beautifulsoup4
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### search.py — Execute SearXNG Search
|
||||
|
||||
```bash
|
||||
python scripts/search.py -q "your query" -i https://your-instance \
|
||||
[--format json|brief|urls|csv] \
|
||||
[--engines google,bing,brave] \
|
||||
[--time-range day|month|year|none] \
|
||||
[--language en] \
|
||||
[--sort-by score|date|engine|none] \
|
||||
[--no-dedup] \
|
||||
[--max-results 10] \
|
||||
[--fetch 3] \
|
||||
[--cache-ttl 30] \
|
||||
[--queries-file queries.txt] \
|
||||
[--include-domain example.com] \
|
||||
[--exclude-domain spam.com] \
|
||||
[--proxy http://corp:8080] \
|
||||
[--auth-bearer-file ~/.token] \
|
||||
[--verify] \
|
||||
[--verbose|-v] [--quiet]
|
||||
```
|
||||
|
||||
### fetch.py — Fetch & Extract Web Page Content
|
||||
|
||||
```bash
|
||||
python scripts/fetch.py -u https://example.com \
|
||||
--extract text|html|markdown \
|
||||
[--encoding gbk] \
|
||||
[--max-size 5242880] \
|
||||
[--proxy http://corp:8080] \
|
||||
[--auth-bearer-file ~/.token]
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Instance URLs resolve in priority order:
|
||||
1. `-i / --instance` (comma-separated for failover)
|
||||
2. `SEARXNG_INSTANCE` environment variable
|
||||
3. Config file: `./searxng.toml` → `~/.config/searxng-cli/searxng.toml` → `./instances.txt` → `~/.config/searxng-cli/instances.txt`
|
||||
|
||||
Example `searxng.toml`:
|
||||
```toml
|
||||
[searxng]
|
||||
instance = "https://my-searxng.example.com"
|
||||
# or: instances = ["https://a.example.com", "https://b.example.com"]
|
||||
engines = "google,bing,brave,duckduckgo,startpage,wikipedia,wikidata"
|
||||
time_range = "year"
|
||||
safesearch = 0
|
||||
format = "json"
|
||||
cache_ttl = 30
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
pip install pytest
|
||||
pytest -q
|
||||
```
|
||||
|
||||
155 tests cover: cache operations, auth resolution, domain filtering, Markdown conversion, search logic, integration flows, and logging configuration.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
├── scripts/
|
||||
│ ├── search.py # Search with multi-instance failover, cache, batch, domain filter
|
||||
│ ├── fetch.py # Web page fetcher with text/markdown extraction
|
||||
│ ├── common.py # Shared utilities (auth, retry, charset, logging)
|
||||
│ ├── cache.py # SQLite-backed result cache
|
||||
│ └── _config.py # Version + User-Agent constants
|
||||
├── tests/ # pytest unit + integration tests
|
||||
├── .gitea/workflows/ # Gitea Actions CI
|
||||
├── SKILL.md # Full skill documentation (agent-facing)
|
||||
├── pytest.ini # Test configuration
|
||||
└── README.md
|
||||
```
|
||||
|
||||
## Defaults
|
||||
|
||||
| Setting | Default | Flag |
|
||||
|---------|---------|------|
|
||||
| Instance | **required** | `-i` / `SEARXNG_INSTANCE` / config |
|
||||
| Safe search | 0 (off) | `-s` |
|
||||
| Time range | year | `-t` |
|
||||
| Output format | json | `-f` |
|
||||
| Engines | google,bing,brave,duckduckgo,startpage,wikipedia,wikidata | `--engines` |
|
||||
| Sort | score descending | `--sort-by` |
|
||||
| Dedup | on | `--no-dedup` |
|
||||
|
||||
## Cross-Agent Compatibility
|
||||
|
||||
These scripts are agent-agnostic — they work with any AI agent that can invoke terminal commands:
|
||||
|
||||
| Agent | How to invoke |
|
||||
|-------|--------------|
|
||||
| Hermes | `python scripts/search.py -q "..." -i https://your-instance` |
|
||||
| Claude Code | Same — call via terminal tool |
|
||||
| Codex (OpenAI) | Same — call via terminal tool |
|
||||
| OpenCode | Same — call via terminal tool |
|
||||
| Cursor | Same — call via terminal tool |
|
||||
| Trae | Same — call via terminal tool |
|
||||
| Standalone (human) | Run directly in any terminal |
|
||||
|
||||
Key design decisions for universal compatibility:
|
||||
- Zero external dependencies (stdlib-only for `search.py`)
|
||||
- Scripts self-inject their directory into `sys.path` — run from any working directory
|
||||
- Stdout carries data (JSON/CSV/text), stderr carries progress/warnings
|
||||
- Exit codes: 0=success, 1=fatal error, 2=no results/empty
|
||||
- NO agent-specific API calls — purely CLI-based, portable across all agent platforms
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
Reference in New Issue
Block a user