feat(v1.7.0): AI 友好度增强 + 测试补全 (155→309)

核心新增（面向 AI Agent 程序化使用）：
- 结构化错误码体系：E_CONFIG/E_AUTH/E_NETWORK/E_RATE_LIMIT/E_PARSE/E_EMPTY/E_INPUT/E_INTERNAL
  classify_error() 自动分类异常，JSON 错误输出含 error_code 字段
- JSON Lines 流式输出 (--stream)：每条结果独立一行，AI 可增量处理
- 进度事件 (--progress)：JSON Lines 事件流到 stderr（start/cache_hit/fetch_ok/done 等）

测试补全（+154 例，覆盖全部高风险盲区）：
- HTML 回退搜索路径 (19)
- --fetch 自动抓取 (21)
- --verify 健康检查 (15)
- 输出格式化 (15)
- 实例解析链 (20)
- 并行多实例搜索 (10)
- CLI 入口与端到端 (17)
- 错误码分类 (27)
- 流式输出与进度事件 (10)

源码改进：
- search.py: h3 内 a 标签 href 作为 url fallback，提升 SearXNG 主题兼容性
- common.py: 新增 classify_error/emit_progress/set_progress_enabled

文档同步：SKILL.md 新增 AI Agent Integration Guide 章节，README.md 更新参数与错误码表

README.md

@@ -55,7 +55,40 @@ AI 调用后，stdout 输出网页正文（text/html/markdown 三种格式），
 | exit 1 | 失败 | 致命错误（所有实例不可用、参数错误等） |
 | exit 2 | 空结果 | 搜索成功但无结果 |
 
-**错误处理**：`--format json` 模式下，错误以 JSON 输出到 stdout（非 stderr），格式为 `{"error": "...", "exit_code": 1}`，AI 可程序化捕获。
+**错误处理**：`--format json` 模式下，错误以 JSON 输出到 stdout（非 stderr），格式为 `{"error": "...", "error_code": "E_NETWORK", "exit_code": 1, "query": "..."}`，AI 可程序化捕获。
+
+**错误码体系**（`error_code` 字段）：
+
+| 错误码 | 含义 | AI 恢复策略 |
+|--------|------|-------------|
+| `E_CONFIG` | 配置错误（无实例） | 提示用户设置 `-i` / `SEARXNG_INSTANCE` |
+| `E_AUTH` | 认证失败（401/403） | 检查 token/凭证 |
+| `E_NETWORK` | 网络错误（连接失败、5xx） | 重试或切换实例/代理 |
+| `E_RATE_LIMIT` | 限流（429） | 等待后重试 |
+| `E_PARSE` | 解析错误 | 检查实例 JSON 支持 |
+| `E_INPUT` | 输入错误（参数/文件） | 修正参数 |
+| `E_INTERNAL` | 内部错误 | 报告 bug |
+
+### 流式输出与进度事件（AI 高级用法）
+
+**`--stream`**：JSON Lines 流式输出，每条结果一行 JSON，AI 可增量处理：
+```
+{"type": "result", "result": {"title": "...", "url": "..."}}
+{"type": "done", "count": 10, "query": "..."}
+```
+
+**`--progress`**：进度事件流（JSON Lines 到 stderr），AI 可实时跟踪执行：
+```
+{"event": "start", "query": "...", "instances": 2}
+{"event": "cache_hit", "query": "...", "ttl": 30}
+{"event": "fetch_ok", "url": "...", "chars": 12345}
+{"event": "done", "results": 10, "query": "..."}
+```
+
+```bash
+# AI 推荐用法：流式输出 + 进度事件
+python scripts/search.py -q "research" -i https://your-instance --stream --progress
+```
 
 ## 部署
 
@@ -112,6 +145,8 @@ python scripts/search.py -q "查询词" -i https://your-instance \
   [--proxy http://corp:8080] \
   [--auth-bearer-file ~/.token] \
   [--verify] \
+  [--stream] \
+  [--progress] \
   [--verbose|-v] [--quiet]
 ```
 
@@ -138,6 +173,8 @@ python scripts/search.py -q "查询词" -i https://your-instance \
 | `--auth-basic-file` | Basic Auth 文件 | — |
 | `--verify` | 实例健康检查模式 | — |
 | `--config` | 指定配置文件 | 自动发现 |
+| `--stream` | JSON Lines 流式输出（每条结果一行） | 关闭 |
+| `--progress` | 进度事件（JSON Lines 到 stderr） | 关闭 |
 | `-v / --verbose` | 调试日志 | — |
 | `--quiet` | 仅输出警告和错误 | — |
 
@@ -203,7 +240,10 @@ python scripts/fetch.py -u https://example.com \
 **工程**
 - 共享 `common.py`（统一重试/字符集/认证/日志）
 - 结构化日志（`--verbose` / `--quiet`）
-- 155 个单元+集成测试
+- 结构化错误码（`E_NETWORK` / `E_AUTH` / `E_RATE_LIMIT` 等）
+- JSON Lines 流式输出（`--stream`）
+- 进度事件（`--progress`，JSON Lines 到 stderr）
+- 309 个单元+集成测试
 
 ## 跨 Agent 兼容性
 
@@ -231,7 +271,7 @@ pip install pytest
 pytest -q
 ```
 
-155 个测试覆盖：缓存操作、认证解析、域名过滤、Markdown 转换、搜索逻辑、集成流程、日志配置。
+309 个测试覆盖：缓存操作、认证解析、域名过滤、Markdown 转换、搜索逻辑、集成流程、日志配置、HTML 回退、自动抓取、健康检查、输出格式化、实例解析、并行搜索、CLI 端到端、错误码分类、流式输出、进度事件。
 
 ## 项目结构
 

SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: searxng-use-cli
 description: Use when you need to search the web via your OWN SearXNG instance (no public-instance discovery). 3 CLI scripts + a shared common.py module — execute privacy-respecting searches against a user-supplied instance (with multi-instance failover, 5xx/429 retry, auto-fetch) or via SEARXNG_INSTANCE env / config file, fetch/extract readable text or markdown from web pages. Zero-config replacement for proprietary search APIs.
-version: 1.6.0
+version: 1.7.0
 author: Metona Team
 license: MIT
 platforms: [linux, macos, windows]
@@ -33,7 +33,9 @@ SearXNG is a privacy-respecting metasearch engine that aggregates results from 7
 **Output formats**
 - JSON (default, rich metadata), brief (title+URL+snippet), urls (plain list), CSV (spreadsheet-ready)
 - Enhanced Markdown conversion — nested ordered lists (numbered), mixed `ul`/`ol` nesting, `<dl>`/`<dt>`/`<dd>` definition lists, GFM tables, fenced code blocks, blockquotes
-- Structured JSON error output (in `--format json` mode) for machine-readable failure reporting
+- Structured JSON error output (in `--format json` mode) with `error_code` field for machine-readable failure reporting
+- JSON Lines streaming (`--stream`) — each result emitted as a separate JSON line to stdout, enabling incremental processing by AI agents
+- Progress events (`--progress`) — structured JSON Lines events to stderr for real-time execution tracking
 
 **Caching & config**
 - SQLite result caching (`--cache-ttl`) — identical queries within a TTL skip the network entirely; `--clear-cache` / `--cache-stats` manage it
@@ -157,6 +159,146 @@ python scripts/search.py -q "test" -i https://s.example.com --quiet     # errors
 - High-frequency production search without a properly-scaled instance — respect your instance's rate limits
 - Guaranteed uptime/accuracy — depends entirely on the instance you supply
 
+## AI Agent Integration Guide
+
+This section documents the structured interfaces that AI agents can rely on
+for programmatic integration. All features are designed to be machine-readable
+and machine-actionable.
+
+### Output Channels
+
+| Channel | Content | Description |
+|---------|---------|-------------|
+| stdout | Data | JSON/CSV/text — the only source AI should parse |
+| stderr | Logs + Progress | Human-readable logs (default) or JSON Lines events (`--progress`) |
+| exit 0 | Success | Results available on stdout |
+| exit 1 | Fatal error | Error JSON on stdout (in `--format json` mode) or stderr |
+| exit 2 | Empty results | Search succeeded but returned no results |
+
+### Error Code System
+
+In `--format json` mode, errors are emitted as structured JSON on stdout:
+
+```json
+{
+  "error": "All 2 instances failed. Last error: connection refused",
+  "error_code": "E_NETWORK",
+  "exit_code": 1,
+  "query": "search term"
+}
+```
+
+AI agents can use `error_code` to programmatically decide recovery strategy:
+
+| Code | Meaning | AI Recovery Strategy |
+|------|---------|---------------------|
+| `E_CONFIG` | Configuration error (no instance resolved) | Prompt user to set `-i` / `SEARXNG_INSTANCE` / config file |
+| `E_AUTH` | Authentication failed (401/403) | Check token/credentials, prompt user to re-authenticate |
+| `E_NETWORK` | Network error (connection refused, timeout, 5xx, all instances failed) | Retry with backoff, switch instance or proxy |
+| `E_RATE_LIMIT` | Rate limited (429) | Wait and retry with reduced frequency |
+| `E_PARSE` | Parse error (JSON/HTML parsing failed) | Check if instance supports JSON, try HTML fallback |
+| `E_EMPTY` | Empty results (exit code 2) | Adjust query terms or time range |
+| `E_INPUT` | Input error (bad parameters, file not found) | Fix CLI arguments or file paths |
+| `E_INTERNAL` | Internal error (unexpected exception) | Report bug with full error message |
+
+### JSON Lines Streaming (`--stream`)
+
+For large result sets, `--stream` outputs results as JSON Lines (one JSON
+object per line) to stdout, allowing AI agents to process results
+incrementally without waiting for the full response:
+
+```bash
+python scripts/search.py -q "large topic" -i https://your-instance --stream
+```
+
+Output format (each line is a separate JSON object):
+
+```
+{"type": "result", "result": {"title": "...", "url": "...", "content": "..."}}
+{"type": "result", "result": {"title": "...", "url": "...", "content": "..."}}
+{"type": "done", "count": 2, "query": "large topic"}
+```
+
+- `type: "result"` — one per search result, emitted as soon as available
+- `type: "done"` — terminal event with total count, always emitted last
+- Exit code 0 on success, 2 on empty results (done event still emitted)
+
+Only valid with `--format json` (the default).
+
+### Progress Events (`--progress`)
+
+For long-running operations, `--progress` emits structured JSON Lines events
+to stderr, enabling AI agents to track execution progress in real time:
+
+```bash
+python scripts/search.py -q "research topic" -i https://your-instance --progress --fetch 3
+```
+
+Event types (each on its own line, JSON Lines format on stderr):
+
+```jsonl
+{"event": "start", "query": "research topic", "instances": 2}
+{"event": "cache_hit", "query": "research topic", "ttl": 30}
+{"event": "cache_store", "query": "research topic", "ttl": 30}
+{"event": "fetch_start", "count": 3}
+{"event": "fetch_ok", "url": "https://example.com/page", "chars": 12345}
+{"event": "fetch_fail", "url": "https://bad.example.com", "error": "HTTP 503"}
+{"event": "done", "results": 10, "query": "research topic"}
+{"event": "error", "error": "connection refused", "error_code": "E_NETWORK", "query": "..."}
+```
+
+AI agents can parse these events to:
+- Show progress indicators to users
+- Detect cache hits (skip waiting)
+- Monitor fetch failures and retry strategies
+- Correlate errors with specific queries in batch mode
+
+`--progress` and `--verbose` can be used together (progress events on stderr,
+debug logs also on stderr). `--progress` events are JSON Lines; `--verbose`
+logs are human-readable text.
+
+### Output JSON Schema
+
+The default `--format json` output shape (for reference, not enforced):
+
+```json
+{
+  "query": "search term",
+  "number_of_results": 10,
+  "results": [
+    {
+      "title": "Result title",
+      "url": "https://example.com/page",
+      "content": "Snippet text...",
+      "engine": "google",
+      "score": 1.0,
+      "category": "general",
+      "published_date": "2024-01-15T10:30:00"
+    }
+  ],
+  "answers": ["Direct answer if available"],
+  "corrections": [],
+  "suggestions": ["related suggestion"],
+  "infoboxes": [],
+  "unresponsive_engines": [["engine_name", "error reason"]],
+  "fetched": [
+    {
+      "url": "https://example.com/page",
+      "status": "ok",
+      "text": "Extracted page content...",
+      "text_length": 12345,
+      "truncated": false,
+      "final_url": "https://example.com/final",
+      "user_agent_used": "searxng-cli/1.7.0"
+    }
+  ],
+  "fetched_source": "json"
+}
+```
+
+Fields marked as optional may be absent. The `fetched` and `fetched_source`
+fields only appear when `--fetch N` is used.
+
 ## Cross-Agent Compatibility
 
 These scripts are **agent-agnostic** — they work with any AI agent that can invoke terminal commands:

scripts/_config.py

@@ -7,5 +7,5 @@ Retry settings and shared HTTP utilities now live in ``common.py`` so that
 both ``search.py`` and ``fetch.py`` share one consistent implementation.
 """
 
-VERSION = "1.6.0"
+VERSION = "1.7.0"
 USER_AGENT = f"searxng-cli/{VERSION}"

scripts/common.py

@@ -259,3 +259,136 @@ def is_retryable_error(exc: BaseException) -> bool:
         # requests connection/timeout error without a response -> transient
         return True
     return False
+
+
+# ----- Structured error classification -----
+#
+# 错误码体系：让 AI Agent 程序化地判断错误类型并采取恢复策略。
+# 所有错误码以 E_ 前缀，在 --format json 模式下随 error_code 字段输出。
+#
+# AI 可根据 error_code 决策：
+#   E_CONFIG     → 检查实例配置/环境变量，提示用户设置
+#   E_AUTH       → 检查 token/凭证，提示用户重新认证
+#   E_NETWORK    → 重试或切换实例/代理
+#   E_RATE_LIMIT → 等待后重试，降低请求频率
+#   E_PARSE      → 检查实例是否支持 JSON，尝试 HTML 回退
+#   E_EMPTY      → 调整查询词或时间范围
+#   E_INPUT      → 修正参数/文件路径
+#   E_INTERNAL   → 报告 bug，附带完整错误信息
+
+# 错误码常量（供 search.py / fetch.py 引用）
+E_CONFIG = "E_CONFIG"
+E_AUTH = "E_AUTH"
+E_NETWORK = "E_NETWORK"
+E_RATE_LIMIT = "E_RATE_LIMIT"
+E_PARSE = "E_PARSE"
+E_EMPTY = "E_EMPTY"
+E_INPUT = "E_INPUT"
+E_INTERNAL = "E_INTERNAL"
+
+
+def classify_error(exc: BaseException) -> str:
+    """将异常分类为结构化错误码，供 AI Agent 程序化处理。
+
+    分类逻辑（按优先级）：
+      1. 429 → E_RATE_LIMIT
+      2. 401/403 → E_AUTH
+      3. 4xx（非上述）→ E_INPUT（请求参数问题）
+      4. 5xx / URLError / OSError / TimeoutError → E_NETWORK
+      5. json.JSONDecodeError / ValueError → E_PARSE
+      6. FileNotFoundError → E_INPUT
+      7. RuntimeError → 尝试从消息中提取线索，否则 E_INTERNAL
+      8. 其他 → E_INTERNAL
+    """
+    import json as _json
+
+    # HTTP 错误（urllib 和 requests 都有 .code 或 .status_code）
+    status = None
+    if isinstance(exc, urllib.error.HTTPError):
+        status = exc.code
+    else:
+        resp = getattr(exc, "response", None)
+        status = getattr(resp, "status_code", None)
+
+    if status is not None:
+        if status == 429:
+            return E_RATE_LIMIT
+        if status in (401, 403):
+            return E_AUTH
+        if 400 <= status < 500:
+            return E_INPUT
+        if 500 <= status < 600:
+            return E_NETWORK
+
+    # 文件/输入错误（FileNotFoundError 是 OSError 子类，必须先于 OSError 检查）
+    if isinstance(exc, FileNotFoundError):
+        return E_INPUT
+
+    # 连接级错误
+    if isinstance(exc, (urllib.error.URLError, OSError, TimeoutError)):
+        return E_NETWORK
+    if isinstance(exc, ConnectionError):
+        return E_NETWORK
+
+    # 解析错误
+    if isinstance(exc, (ValueError, _json.JSONDecodeError)):
+        return E_PARSE
+
+    # RuntimeError：从消息中推断（search_multi 的 "All N instances failed" 等）
+    msg = str(exc).lower()
+    if isinstance(exc, RuntimeError):
+        if "auth" in msg or "403" in msg or "401" in msg:
+            return E_AUTH
+        if "rate" in msg or "429" in msg:
+            return E_RATE_LIMIT
+        if "all" in msg and "instance" in msg and "fail" in msg:
+            return E_NETWORK
+        if "parse" in msg or "json" in msg or "html" in msg:
+            return E_PARSE
+        if "not found" in msg or "no " in msg and "instance" in msg:
+            return E_CONFIG
+        return E_INTERNAL
+
+    return E_INTERNAL
+
+
+# ----- Progress event emitter (for --progress flag) -----
+#
+# 当 --progress 启用时，search.py 会调用 emit_progress() 发射结构化事件到
+# stderr（JSON Lines 格式）。AI Agent 可解析这些事件来跟踪执行进度。
+#
+# 事件类型：
+#   {"event": "start", "query": "...", "instances": N}
+#   {"event": "instance_try", "url": "...", "attempt": 1}
+#   {"event": "instance_ok", "url": "...", "latency": 0.5, "results": 10}
+#   {"event": "instance_fail", "url": "...", "error": "...", "error_code": "E_*"}
+#   {"event": "cache_hit", "query": "...", "ttl": 30}
+#   {"event": "cache_store", "query": "...", "ttl": 30}
+#   {"event": "fetch_start", "count": 3}
+#   {"event": "fetch_ok", "url": "...", "chars": 1234}
+#   {"event": "fetch_fail", "url": "...", "error": "..."}
+#   {"event": "done", "results": N, "query": "..."}
+#   {"event": "error", "error": "...", "error_code": "E_*", "query": "..."}
+
+_progress_enabled = False
+
+
+def set_progress_enabled(enabled: bool) -> None:
+    """全局开关：是否向 stderr 输出 JSON Lines 格式的进度事件。"""
+    global _progress_enabled
+    _progress_enabled = enabled
+
+
+def emit_progress(event: str, **kwargs) -> None:
+    """向 stderr 输出一行 JSON 格式的进度事件。
+
+    仅当 --progress 启用时才输出。``event`` 是事件类型字符串，
+    ``kwargs`` 是事件的附加字段。输出格式为 JSON Lines（每行一个 JSON 对象）。
+    """
+    if not _progress_enabled:
+        return
+    import json as _json
+    payload = {"event": event}
+    payload.update(kwargs)
+    print(_json.dumps(payload, ensure_ascii=False), file=sys.stderr, flush=True)
+

scripts/search.py

@@ -31,9 +31,20 @@ from common import (
     MAX_RETRIES,
     apply_proxy,
     build_auth_headers,
+    classify_error,
+    emit_progress,
     resolve_auth_basic,
     resolve_auth_bearer,
+    set_progress_enabled,
     setup_logging,
+    E_CONFIG,
+    E_AUTH,
+    E_NETWORK,
+    E_RATE_LIMIT,
+    E_PARSE,
+    E_EMPTY,
+    E_INPUT,
+    E_INTERNAL,
 )
 from fetch import extract_text, fetch_url
 import cache as cache_module
@@ -112,6 +123,13 @@ class SearXNGHTMLParser(HTMLParser):
             elif tag == "h3":
                 self._in_h3 = True
                 self._text_buf = []
+            elif tag == "a" and self._in_h3:
+                # h3 内的 <a href> 作为 url 的 fallback：某些 SearXNG 主题
+                # 不使用 url_header class，URL 仅出现在 h3 的链接中。
+                # url_header 优先（上面已处理），此处仅填充空 url。
+                href = attrs_dict.get("href", "")
+                if href and not self._current.get("url"):
+                    self._current["url"] = href
             elif tag == "p" and "content" in classes:
                 self._in_content = True
                 self._text_buf = []
@@ -1095,16 +1113,19 @@ def _run_single_query(query: str, args, instance_urls: list,
                       auth_headers: dict, ttl_seconds: int):
     """Run one query end-to-end: search → limit → domain-filter → fetch.
 
-    Returns ``(results_dict, error_str)``. On success ``error_str`` is None.
-    Cache hits skip the network entirely. Post-processing (limit / filter /
-    fetch) always runs so batch callers see the same shape as single-query
-    callers.
+    Returns ``(results_dict, error_str, error_code)``。成功时后两者为 None。
+    ``error_code`` 是结构化错误码（E_NETWORK/E_AUTH 等），让 AI Agent
+    程序化判断错误类型。Cache hits 跳过网络。后处理（limit/filter/fetch）
+    总是执行，保证 batch 调用方看到与单查询一致的形状。
     """
     params = _build_params(query, args)
 
+    emit_progress("start", query=query, instances=len(instance_urls))
+
     cached = cache_module.get(params, ttl_seconds) if ttl_seconds > 0 else None
     if cached is not None:
         logger.info(f"[cache hit] q={query!r} TTL={args.cache_ttl}min, skipping network")
+        emit_progress("cache_hit", query=query, ttl=args.cache_ttl)
         results = cached
     else:
         try:
@@ -1117,10 +1138,13 @@ def _run_single_query(query: str, args, instance_urls: list,
                 parallel=not args.serial,
             )
         except Exception as e:
-            return None, str(e)
+            err_code = classify_error(e)
+            emit_progress("error", error=str(e), error_code=err_code, query=query)
+            return None, str(e), err_code
         if ttl_seconds > 0:
             cache_module.put(params, results, ttl_seconds)
             logger.info(f"[cache stored] q={query!r} TTL={args.cache_ttl}min")
+            emit_progress("cache_store", query=query, ttl=args.cache_ttl)
 
     # Dedup (default on; --no-dedup disables) then sort, both BEFORE limit
     # so --max-results keeps the highest-scoring / newest items.
@@ -1143,6 +1167,7 @@ def _run_single_query(query: str, args, instance_urls: list,
         logger.info(f"Domain filter: {before} -> {after} results")
 
     if args.fetch > 0 and results.get("results"):
+        emit_progress("fetch_start", count=args.fetch)
         fetched = fetch_top_results(
             results, args.fetch,
             timeout=args.fetch_timeout,
@@ -1150,30 +1175,48 @@ def _run_single_query(query: str, args, instance_urls: list,
             max_retries=args.fetch_retries,
             max_size=args.max_size,
         )
+        # Emit fetch_ok / fetch_fail events
+        for f in fetched:
+            if f.get("status") == "ok":
+                emit_progress("fetch_ok", url=f.get("url", ""),
+                              chars=f.get("text_length", 0))
+            else:
+                emit_progress("fetch_fail", url=f.get("url", ""),
+                              error=f.get("error", "unknown"))
         results["fetched"] = fetched
         results["fetched_source"] = results.get("_fallback", "json")
 
-    return results, None
+    result_count = len(results.get("results", []))
+    emit_progress("done", results=result_count, query=query)
+    return results, None, None
 
 
-def _emit_error(message: str, args, query: str = None, exit_code: int = 1):
+def _emit_error(message: str, args, query: str = None, exit_code: int = 1,
+                error_code: str = None):
     """Emit an error and exit.
 
     In ``--format json`` mode the error is printed to **stdout** as a
     structured JSON object so agents piping stdout can parse it. All other
     formats print to stderr (keeping stdout clean for data) and exit.
 
-    The JSON shape is ``{"error": "...", "exit_code": N, "query": "..."}``
-    (query only included when provided).
+    The JSON shape is::
+
+        {"error": "...", "exit_code": N, "error_code": "E_*", "query": "..."}
+
+    ``error_code`` 是结构化错误码（E_CONFIG/E_AUTH/E_NETWORK 等），让 AI
+    Agent 程序化判断错误类型并采取恢复策略。``query`` 仅在提供时包含。
     """
     if getattr(args, "format", None) == "json":
         payload = {"error": message, "exit_code": exit_code}
+        if error_code:
+            payload["error_code"] = error_code
         if query:
             payload["query"] = query
         print(json.dumps(payload, indent=2, ensure_ascii=False))
     else:
         prefix = f"[query: {query}] " if query else ""
-        logger.error(f"{prefix}Error: {message}")
+        code_prefix = f"[{error_code}] " if error_code else ""
+        logger.error(f"{prefix}{code_prefix}Error: {message}")
     sys.exit(exit_code)
 
 
@@ -1318,6 +1361,17 @@ Use --config FILE to load a non-default config file (overrides the auto-discover
                              "Env var SEARXNG_BASIC_AUTH is also honored.")
     parser.add_argument("--output", "-o", default=None,
                         help="Save to file instead of stdout")
+    parser.add_argument("--stream", action="store_true",
+                        help="Stream results as JSON Lines (one JSON object per line) to stdout. "
+                             "Each line is a {\"type\": \"result\", \"result\": {...}} event. "
+                             "Ends with {\"type\": \"done\", \"count\": N}. "
+                             "AI Agent can process results incrementally without waiting for full output. "
+                             "Only valid with --format json.")
+    parser.add_argument("--progress", action="store_true",
+                        help="Emit structured progress events as JSON Lines to stderr. "
+                             "Events: start, instance_try, instance_ok, instance_fail, "
+                             "cache_hit, cache_store, fetch_start, fetch_ok, fetch_fail, done. "
+                             "AI Agent can track execution progress programmatically.")
     parser.add_argument("--timeout", type=int, default=_cfg_int(config, "timeout", 15),
                         help="Request timeout in seconds (default: 15)")
     parser.add_argument("--retry", type=int, default=_cfg_int(config, "max_retries", None),
@@ -1356,6 +1410,9 @@ Use --config FILE to load a non-default config file (overrides the auto-discover
 
     args = parser.parse_args()
 
+    # 启用 --progress 进度事件（JSON Lines 到 stderr）
+    set_progress_enabled(getattr(args, "progress", False))
+
     # --query is required unless we're doing a non-search operation.
     # --queries-file is an alternative to --query for batch mode.
     if (not args.verify and not args.query and not args.queries_file
@@ -1375,7 +1432,7 @@ Use --config FILE to load a non-default config file (overrides the auto-discover
     if not instance_urls:
         _emit_error("no SearXNG instance resolved. Provide -i/--instance, set the "
                     "SEARXNG_INSTANCE environment variable, or create a searxng.toml / "
-                    "instances.txt config file.", args)
+                    "instances.txt config file.", args, error_code=E_CONFIG)
 
     # Build auth headers if provided (needed by both verify and search).
     # Credentials may come from CLI flag, file, or env var (in priority order)
@@ -1384,7 +1441,7 @@ Use --config FILE to load a non-default config file (overrides the auto-discover
         bearer_token = resolve_auth_bearer(args.auth_bearer, args.auth_bearer_file)
         basic_auth = resolve_auth_basic(args.auth_basic, args.auth_basic_file)
     except RuntimeError as e:
-        _emit_error(str(e), args)
+        _emit_error(str(e), args, error_code=E_AUTH)
     auth_headers = build_auth_headers(
         bearer_token=bearer_token,
         basic_auth=basic_auth,
@@ -1440,20 +1497,24 @@ Use --config FILE to load a non-default config file (overrides the auto-discover
         try:
             queries = _read_queries_file(args.queries_file)
         except RuntimeError as e:
-            _emit_error(str(e), args)
+            _emit_error(str(e), args, error_code=E_INPUT)
         if not queries:
-            _emit_error(f"no queries found in '{args.queries_file}'", args)
+            _emit_error(f"no queries found in '{args.queries_file}'", args,
+                        error_code=E_INPUT)
 
         logger.info(f"Running {len(queries)} queries from {args.queries_file}...")
         batch = []
         any_ok = False
         for i, q in enumerate(queries, 1):
             logger.info(f"\n[{i}/{len(queries)}] {q}")
-            results, err = _run_single_query(q, args, instance_urls,
-                                             auth_headers, ttl_seconds)
+            results, err, err_code = _run_single_query(q, args, instance_urls,
+                                                        auth_headers, ttl_seconds)
             if err:
                 logger.error(f"  [ERROR] {err}")
-                batch.append({"query": q, "error": err})
+                entry = {"query": q, "error": err}
+                if err_code:
+                    entry["error_code"] = err_code
+                batch.append(entry)
             else:
                 any_ok = True
                 batch.append({"query": q, "results": results})
@@ -1516,10 +1577,22 @@ Use --config FILE to load a non-default config file (overrides the auto-discover
         sys.exit(0 if any_ok else 1)
 
     # ----- Single query mode -----
-    results, err = _run_single_query(args.query, args, instance_urls,
-                                     auth_headers, ttl_seconds)
+    results, err, err_code = _run_single_query(args.query, args, instance_urls,
+                                                auth_headers, ttl_seconds)
     if err:
-        _emit_error(err, args, query=args.query)
+        _emit_error(err, args, query=args.query, error_code=err_code)
+
+    # --stream: JSON Lines 流式输出，每条结果一行，AI 可增量处理
+    if getattr(args, "stream", False) and args.format == "json":
+        for r in results.get("results", []):
+            print(json.dumps({"type": "result", "result": r},
+                             ensure_ascii=False), flush=True)
+        print(json.dumps({"type": "done",
+                          "count": len(results.get("results", [])),
+                          "query": args.query}, ensure_ascii=False), flush=True)
+        if not results.get("results"):
+            sys.exit(2)
+        sys.exit(0)
 
     output = _format_results(results, args)
     if args.output:

tests/test_cli_e2e.py

@@ -0,0 +1,296 @@
+"""End-to-end tests for scripts/search.py.
+
+Covers three layers of the CLI:
+  * ``_run_single_query`` — full orchestration (search → dedup → sort →
+    limit → domain-filter → fetch), stubbed at the ``search_multi`` /
+    ``fetch_top_results`` boundary so no network is touched.
+  * ``_emit_error``       — error output + exit (json→stdout, others→stderr).
+  * ``main()``            — real CLI via subprocess (--version / --help /
+    no-args / --cache-stats).
+
+``_run_single_query`` tests build a ``SimpleNamespace`` args object that
+matches the argparse.Namespace shape produced by ``main()``. Cache-related
+tests use the ``isolated_cache`` fixture from conftest.py so each test gets
+a fresh SQLite cache under a temp dir. ``main()`` tests spawn a real
+subprocess with cwd pinned to the project root.
+"""
+import json
+import os
+import subprocess
+import sys
+from pathlib import Path
+from types import SimpleNamespace
+
+import pytest
+
+from search import _run_single_query, _emit_error, _build_params
+import cache as cache_module
+
+PROJECT_ROOT = Path(__file__).resolve().parent.parent
+
+
+def _make_args(**overrides):
+    """Build a minimal args object matching the argparse.Namespace shape
+    that ``_run_single_query`` reads. All fields _run_single_query touches
+    are present with sensible defaults; tests override only what they need.
+    """
+    base = dict(
+        query="test", format="json", method="GET", timeout=15, retry=0,
+        serial=False, no_dedup=False, sort_by="none", max_results=None,
+        include_domain=None, exclude_domain=None, fetch=0, fetch_timeout=10,
+        fetch_retries=3, max_size=None, cache_ttl=0, snippet_len=0,
+        categories=None, language=None, pageno=1, time_range="year",
+        safesearch=0, engines="google,bing",
+    )
+    base.update(overrides)
+    return SimpleNamespace(**base)
+
+
+# ===== _run_single_query =====
+
+def test_run_single_query_success(monkeypatch):
+    """Successful search returns (results, None)."""
+    args = _make_args()
+    fake = {"results": [{"url": "https://a.com/1", "title": "A"}]}
+    monkeypatch.setattr("search.search_multi", lambda *a, **kw: fake)
+
+    results, err, _ = _run_single_query("test", args, ["https://x.example.com"], {}, 0)
+    assert err is None
+    assert results is not None
+    assert results["results"][0]["url"] == "https://a.com/1"
+
+
+def test_run_single_query_failure_returns_error(monkeypatch):
+    """When search_multi raises, returns (None, error_str)."""
+    args = _make_args()
+
+    def boom(*a, **kw):
+        raise RuntimeError("network down")
+    monkeypatch.setattr("search.search_multi", boom)
+
+    results, err, _ = _run_single_query("test", args, ["https://x.example.com"], {}, 0)
+    assert results is None
+    assert "network down" in err
+
+
+def test_run_single_query_cache_hit_skips_network(isolated_cache, monkeypatch):
+    """Cache hit returns the cached result without calling search_multi.
+
+    Pre-populates the cache with the exact params _run_single_query builds,
+    then asserts search_multi is never reached (a call would raise).
+    """
+    args = _make_args(cache_ttl=5)
+    cached = {"results": [{"url": "https://a.com/1", "title": "cached"}]}
+
+    params = _build_params("test", args)
+    cache_module.put(params, cached, 300)
+
+    def should_not_call(*a, **kw):
+        raise AssertionError("search_multi must not be called on cache hit")
+    monkeypatch.setattr("search.search_multi", should_not_call)
+
+    results, err, _ = _run_single_query("test", args, ["https://x.example.com"], {}, 300)
+    assert err is None
+    assert results is not None
+    assert results["results"][0]["title"] == "cached"
+
+
+def test_run_single_query_cache_miss_stores_result(isolated_cache, monkeypatch):
+    """Cache miss calls search_multi and stores the result for next time."""
+    args = _make_args(cache_ttl=5)
+    fake = {"results": [{"url": "https://a.com/1", "title": "A"}]}
+    monkeypatch.setattr("search.search_multi", lambda *a, **kw: fake)
+
+    assert cache_module.stats()["entries"] == 0
+
+    results, err, _ = _run_single_query("test", args, ["https://x.example.com"], {}, 300)
+    assert err is None
+
+    assert cache_module.stats()["entries"] >= 1
+
+
+def test_run_single_query_no_dedup_keeps_duplicates(monkeypatch):
+    """no_dedup=True skips cross-engine URL deduplication.
+
+    Two results with the same URL but different engines are both kept.
+    """
+    args = _make_args(no_dedup=True)
+    fake = {"results": [
+        {"url": "https://a.com/1", "engine": "google"},
+        {"url": "https://a.com/1", "engine": "bing"},
+    ]}
+    monkeypatch.setattr("search.search_multi", lambda *a, **kw: fake)
+
+    results, err, _ = _run_single_query("test", args, ["https://x.example.com"], {}, 0)
+    assert err is None
+    assert len(results["results"]) == 2
+
+
+def test_run_single_query_max_results_truncation(monkeypatch):
+    """--max-results truncates the list (applied AFTER dedup+sort)."""
+    args = _make_args(max_results=2)
+    fake = {"results": [{"url": f"https://a.com/{i}"} for i in range(5)]}
+    monkeypatch.setattr("search.search_multi", lambda *a, **kw: fake)
+
+    results, err, _ = _run_single_query("test", args, ["https://x.example.com"], {}, 0)
+    assert err is None
+    assert len(results["results"]) == 2
+
+
+def test_run_single_query_include_domain_filter(monkeypatch):
+    """--include-domain keeps only results whose domain matches."""
+    args = _make_args(include_domain="a.com")
+    fake = {"results": [
+        {"url": "https://a.com/1"},
+        {"url": "https://b.com/2"},
+        {"url": "https://a.com/3"},
+    ]}
+    monkeypatch.setattr("search.search_multi", lambda *a, **kw: fake)
+
+    results, err, _ = _run_single_query("test", args, ["https://x.example.com"], {}, 0)
+    assert err is None
+    urls = [r["url"] for r in results["results"]]
+    assert urls == ["https://a.com/1", "https://a.com/3"]
+
+
+def test_run_single_query_exclude_domain_filter(monkeypatch):
+    """--exclude-domain drops results whose domain matches."""
+    args = _make_args(exclude_domain="b.com")
+    fake = {"results": [
+        {"url": "https://a.com/1"},
+        {"url": "https://b.com/2"},
+        {"url": "https://c.com/3"},
+    ]}
+    monkeypatch.setattr("search.search_multi", lambda *a, **kw: fake)
+
+    results, err, _ = _run_single_query("test", args, ["https://x.example.com"], {}, 0)
+    assert err is None
+    urls = [r["url"] for r in results["results"]]
+    assert urls == ["https://a.com/1", "https://c.com/3"]
+
+
+def test_run_single_query_fetch_attaches_fetched(monkeypatch):
+    """--fetch N attaches a 'fetched' list (and fetched_source) to results."""
+    args = _make_args(fetch=2)
+    fake = {"results": [
+        {"url": "https://a.com/1"},
+        {"url": "https://b.com/2"},
+    ]}
+    monkeypatch.setattr("search.search_multi", lambda *a, **kw: fake)
+    fake_fetched = [
+        {"url": "https://a.com/1", "status": "ok", "text": "page A"},
+        {"url": "https://b.com/2", "status": "ok", "text": "page B"},
+    ]
+    monkeypatch.setattr("search.fetch_top_results", lambda *a, **kw: fake_fetched)
+
+    results, err, _ = _run_single_query("test", args, ["https://x.example.com"], {}, 0)
+    assert err is None
+    assert results["fetched"] == fake_fetched
+    assert results["fetched_source"] == "json"
+
+
+# ===== _emit_error =====
+
+def test_emit_error_json_to_stdout(capsys):
+    """In json mode the error is printed as JSON to stdout and exits 1."""
+    args = _make_args(format="json")
+    with pytest.raises(SystemExit) as exc_info:
+        _emit_error("boom", args)
+    assert exc_info.value.code == 1
+    captured = capsys.readouterr()
+    data = json.loads(captured.out)
+    assert data["error"] == "boom"
+    assert data["exit_code"] == 1
+
+
+def test_emit_error_brief_keeps_stdout_clean(capsys):
+    """In non-json mode stdout stays clean (error routed via logger to stderr)."""
+    args = _make_args(format="brief")
+    with pytest.raises(SystemExit) as exc_info:
+        _emit_error("boom", args)
+    assert exc_info.value.code == 1
+    captured = capsys.readouterr()
+    assert captured.out == ""
+
+
+def test_emit_error_includes_query_when_provided(capsys):
+    """JSON error includes the 'query' field when a query is supplied."""
+    args = _make_args(format="json")
+    with pytest.raises(SystemExit):
+        _emit_error("not found", args, query="hello world")
+    captured = capsys.readouterr()
+    data = json.loads(captured.out)
+    assert data["query"] == "hello world"
+    assert data["error"] == "not found"
+
+
+def test_emit_error_omits_query_when_absent(capsys):
+    """JSON error omits the 'query' field entirely when query is None."""
+    args = _make_args(format="json")
+    with pytest.raises(SystemExit):
+        _emit_error("boom", args, query=None)
+    captured = capsys.readouterr()
+    data = json.loads(captured.out)
+    assert "query" not in data
+
+
+# ===== main() via real subprocess =====
+
+def _run_cli(*args, env=None):
+    """Run scripts/search.py as a real subprocess at the project root.
+
+    Uses ``sys.executable`` so the same interpreter that runs pytest runs
+    the CLI. cwd is pinned to PROJECT_ROOT so the script's config-file
+    auto-discovery (./searxng.toml) behaves deterministically.
+    """
+    full_env = {**os.environ, **(env or {})}
+    cmd = [sys.executable, str(PROJECT_ROOT / "scripts" / "search.py")] + list(args)
+    return subprocess.run(
+        cmd, cwd=str(PROJECT_ROOT),
+        capture_output=True, text=True, timeout=30,
+        env=full_env,
+    )
+
+
+def test_cli_version_prints_version():
+    """`--version` exits 0 and prints the version string."""
+    r = _run_cli("--version")
+    assert r.returncode == 0
+    assert "1.7.0" in r.stdout
+    assert "searxng-cli" in r.stdout
+
+
+def test_cli_help_exits_zero():
+    """`--help` exits 0 and prints usage text on stdout."""
+    r = _run_cli("--help")
+    assert r.returncode == 0
+    assert "Search via a user-supplied SearXNG instance" in r.stdout
+    assert "--query" in r.stdout
+
+
+def test_cli_no_args_exits_nonzero():
+    """No arguments → parser.error → non-zero exit.
+
+    argparse uses exit code 2 for parser.error (not 1); we assert non-zero
+    plus the message mentions --query.
+    """
+    r = _run_cli()
+    assert r.returncode != 0
+    assert "--query" in r.stderr or "required" in r.stderr.lower()
+
+
+def test_cli_cache_stats_outputs_json(tmp_path):
+    """`--cache-stats` prints cache statistics as JSON and exits 0.
+
+    An -i instance is supplied because instance resolution runs before the
+    cache-stats branch in main(); the instance is never contacted.
+    SEARXNG_CACHE_DIR is redirected to a temp dir for isolation.
+    """
+    r = _run_cli(
+        "--cache-stats", "-i", "https://example.com",
+        env={"SEARXNG_CACHE_DIR": str(tmp_path)},
+    )
+    assert r.returncode == 0
+    data = json.loads(r.stdout)
+    assert "entries" in data
+    assert "path" in data

tests/test_error_codes.py

@@ -0,0 +1,233 @@
+"""Tests for structured error classification (error code system).
+
+Covers: classify_error() mapping exceptions to E_* codes, _emit_error()
+outputting error_code in JSON mode, _run_single_query() propagating
+error_code from caught exceptions.
+"""
+import json
+import urllib.error
+from types import SimpleNamespace
+from unittest.mock import patch
+import pytest
+
+from common import (
+    classify_error,
+    E_CONFIG, E_AUTH, E_NETWORK, E_RATE_LIMIT,
+    E_PARSE, E_EMPTY, E_INPUT, E_INTERNAL,
+)
+from search import _emit_error, _run_single_query
+
+
+# ----- classify_error: HTTP errors -----
+
+def test_classify_429_is_rate_limit():
+    err = urllib.error.HTTPError("url", 429, "Too Many Requests", {}, None)
+    assert classify_error(err) == E_RATE_LIMIT
+
+
+def test_classify_401_is_auth():
+    err = urllib.error.HTTPError("url", 401, "Unauthorized", {}, None)
+    assert classify_error(err) == E_AUTH
+
+
+def test_classify_403_is_auth():
+    err = urllib.error.HTTPError("url", 403, "Forbidden", {}, None)
+    assert classify_error(err) == E_AUTH
+
+
+def test_classify_404_is_input():
+    err = urllib.error.HTTPError("url", 404, "Not Found", {}, None)
+    assert classify_error(err) == E_INPUT
+
+
+def test_classify_400_is_input():
+    err = urllib.error.HTTPError("url", 400, "Bad Request", {}, None)
+    assert classify_error(err) == E_INPUT
+
+
+def test_classify_500_is_network():
+    err = urllib.error.HTTPError("url", 500, "Internal Server Error", {}, None)
+    assert classify_error(err) == E_NETWORK
+
+
+def test_classify_503_is_network():
+    err = urllib.error.HTTPError("url", 503, "Service Unavailable", {}, None)
+    assert classify_error(err) == E_NETWORK
+
+
+# ----- classify_error: connection errors -----
+
+def test_classify_urlerror_is_network():
+    err = urllib.error.URLError("connection refused")
+    assert classify_error(err) == E_NETWORK
+
+
+def test_classify_timeout_is_network():
+    assert classify_error(TimeoutError("timed out")) == E_NETWORK
+
+
+def test_classify_oserror_is_network():
+    assert classify_error(OSError("network unreachable")) == E_NETWORK
+
+
+# ----- classify_error: parse errors -----
+
+def test_classify_value_error_is_parse():
+    assert classify_error(ValueError("invalid JSON")) == E_PARSE
+
+
+def test_classify_json_decode_error_is_parse():
+    import json as _json
+    try:
+        _json.loads("{bad}")
+        assert False
+    except _json.JSONDecodeError as e:
+        assert classify_error(e) == E_PARSE
+
+
+# ----- classify_error: file/input errors -----
+
+def test_classify_filenotfound_is_input():
+    assert classify_error(FileNotFoundError("no such file")) == E_INPUT
+
+
+# ----- classify_error: RuntimeError message inference -----
+
+def test_classify_runtime_all_instances_failed_is_network():
+    err = RuntimeError("All 3 instances failed. Last error: timeout")
+    assert classify_error(err) == E_NETWORK
+
+
+def test_classify_runtime_auth_message_is_auth():
+    err = RuntimeError("Auth failed: 403 Forbidden")
+    assert classify_error(err) == E_AUTH
+
+
+def test_classify_runtime_rate_limit_message():
+    err = RuntimeError("Rate limited: 429")
+    assert classify_error(err) == E_RATE_LIMIT
+
+
+def test_classify_runtime_parse_message_is_parse():
+    err = RuntimeError("Failed to parse JSON response")
+    assert classify_error(err) == E_PARSE
+
+
+def test_classify_runtime_no_instance_is_config():
+    err = RuntimeError("No instance found in config")
+    assert classify_error(err) == E_CONFIG
+
+
+def test_classify_runtime_unknown_is_internal():
+    err = RuntimeError("Something unexpected happened")
+    assert classify_error(err) == E_INTERNAL
+
+
+# ----- classify_error: fallback -----
+
+def test_classify_unknown_exception_is_internal():
+    assert classify_error(Exception("unknown")) == E_INTERNAL
+
+
+# ----- _emit_error: error_code in JSON output -----
+
+def test_emit_error_json_includes_error_code():
+    """JSON mode: error_code appears in the JSON output."""
+    args = SimpleNamespace(format="json")
+    with pytest.raises(SystemExit) as exc_info:
+        _emit_error("connection refused", args, query="test",
+                    error_code=E_NETWORK)
+    assert exc_info.value.code == 1
+    # Output already printed to stdout; we can't easily capture it here
+    # without capsys, so we test via capsys below.
+
+
+def test_emit_error_json_with_code_and_query(capsys):
+    """JSON mode: both error_code and query are in the output."""
+    args = SimpleNamespace(format="json")
+    with pytest.raises(SystemExit):
+        _emit_error("rate limited", args, query="news",
+                    error_code=E_RATE_LIMIT)
+    out, _ = capsys.readouterr()
+    data = json.loads(out)
+    assert data["error"] == "rate limited"
+    assert data["error_code"] == E_RATE_LIMIT
+    assert data["query"] == "news"
+    assert data["exit_code"] == 1
+
+
+def test_emit_error_json_without_error_code(capsys):
+    """JSON mode: error_code field omitted when not provided (backwards compat)."""
+    args = SimpleNamespace(format="json")
+    with pytest.raises(SystemExit):
+        _emit_error("some error", args)
+    out, _ = capsys.readouterr()
+    data = json.loads(out)
+    assert "error_code" not in data
+
+
+def test_emit_error_brief_includes_code_prefix(caplog):
+    """Non-JSON mode: error_code appears as [E_*] prefix in log output."""
+    args = SimpleNamespace(format="brief")
+    with pytest.raises(SystemExit):
+        with caplog.at_level("ERROR"):
+            _emit_error("not found", args, query="test", error_code=E_INPUT)
+    assert "[E_INPUT]" in caplog.text
+
+
+# ----- _run_single_query: error_code propagation -----
+
+def test_run_single_query_propagates_error_code():
+    """When search_multi raises, _run_single_query returns the classified code."""
+    import search as search_mod
+    err = urllib.error.URLError("connection refused")
+    args = SimpleNamespace(
+        query="test", format="json", method="GET", timeout=15, retry=0,
+        serial=False, no_dedup=False, sort_by="none", max_results=None,
+        include_domain=None, exclude_domain=None, fetch=0, fetch_timeout=10,
+        fetch_retries=3, max_size=None, cache_ttl=0, snippet_len=0,
+        categories=None, language=None, pageno=1, time_range="year",
+        safesearch=0, engines="google,bing",
+    )
+    with patch.object(search_mod, "search_multi", side_effect=err):
+        results, err_str, err_code = _run_single_query(
+            "test", args, ["https://x.example.com"], {}, 0)
+    assert results is None
+    assert "connection refused" in err_str
+    assert err_code == E_NETWORK
+
+
+def test_run_single_query_propagates_auth_error():
+    """HTTP 403 → error_code = E_AUTH."""
+    import search as search_mod
+    err = urllib.error.HTTPError("url", 403, "Forbidden", {}, None)
+    args = SimpleNamespace(
+        query="test", format="json", method="GET", timeout=15, retry=0,
+        serial=False, no_dedup=False, sort_by="none", max_results=None,
+        include_domain=None, exclude_domain=None, fetch=0, fetch_timeout=10,
+        fetch_retries=3, max_size=None, cache_ttl=0, snippet_len=0,
+        categories=None, language=None, pageno=1, time_range="year",
+        safesearch=0, engines="google,bing",
+    )
+    with patch.object(search_mod, "search_multi", side_effect=err):
+        _, _, err_code = _run_single_query(
+            "test", args, ["https://x.example.com"], {}, 0)
+    assert err_code == E_AUTH
+
+
+def test_run_single_query_propagates_rate_limit():
+    """HTTP 429 → error_code = E_RATE_LIMIT."""
+    import search as search_mod
+    err = urllib.error.HTTPError("url", 429, "Too Many Requests", {}, None)
+    args = SimpleNamespace(
+        query="test", format="json", method="GET", timeout=15, retry=0,
+        serial=False, no_dedup=False, sort_by="none", max_results=None,
+        include_domain=None, exclude_domain=None, fetch=0, fetch_timeout=10,
+        fetch_retries=3, max_size=None, cache_ttl=0, snippet_len=0,
+        categories=None, language=None, pageno=1, time_range="year",
+        safesearch=0, engines="google,bing",
+    )
+    with patch.object(search_mod, "search_multi", side_effect=err):
+        _, _, err_code = _run_single_query(
+            "test", args, ["https://x.example.com"], {}, 0)
+    assert err_code == E_RATE_LIMIT

tests/test_fetch_auto.py

@@ -0,0 +1,232 @@
+"""Tests for the --fetch auto-fetch feature in search.py.
+
+Covers: _is_blocked_page (CAPTCHA/bot detection heuristics), fetch_page
+(ok/error/blocked paths), fetch_top_results (URL dedup, ordering,
+concurrency, empty input).
+"""
+from unittest.mock import patch, MagicMock
+
+import search as search_mod
+from search import _is_blocked_page, fetch_page, fetch_top_results
+import fetch as fetch_mod
+from fetch import FetchResult
+
+
+# ----- _is_blocked_page -----
+
+def test_blocked_page_detects_captcha():
+    assert _is_blocked_page("<html>Please complete the CAPTCHA</html>")
+
+
+def test_blocked_page_detects_cloudflare():
+    assert _is_blocked_page("<html>Just a moment... cf-browser-verification</html>")
+
+
+def test_blocked_page_detects_challenge():
+    assert _is_blocked_page("<html>Checking your browser before accessing</html>")
+
+
+def test_blocked_page_detects_anubis():
+    assert _is_blocked_page("<html>anubis_challenge</html>")
+
+
+def test_blocked_page_clean_html():
+    """Normal HTML must not be flagged as blocked."""
+    assert not _is_blocked_page("<html><body><article>Real content</article></body></html>")
+
+
+def test_blocked_page_empty():
+    assert not _is_blocked_page("")
+
+
+def test_blocked_page_checks_first_2000_chars():
+    """Detection only scans the first 2000 chars for performance."""
+    padding = "x" * 2500
+    html = f"<html>{padding}captcha</html>"
+    assert not _is_blocked_page(html)
+
+
+# ----- fetch_page -----
+
+def _mock_fetch_result(content: str, content_type="text/html",
+                       final_url="https://example.com",
+                       truncated=False, ua="searxng-cli/1.6.0"):
+    return FetchResult(
+        content=content,
+        content_type=content_type,
+        final_url=final_url,
+        truncated=truncated,
+        user_agent=ua,
+    )
+
+
+def test_fetch_page_ok_html():
+    """Successful HTML fetch extracts text and returns status=ok."""
+    html = "<html><body><article>Hello world</article></body></html>"
+    with patch.object(search_mod, "fetch_url",
+                      return_value=_mock_fetch_result(html)):
+        r = fetch_page("https://example.com")
+    assert r["status"] == "ok"
+    assert "Hello world" in r["text"]
+    assert r["text_length"] > 0
+    assert r["truncated"] is False
+
+
+def test_fetch_page_ok_non_html():
+    """Non-HTML content is returned as-is without text extraction."""
+    with patch.object(search_mod, "fetch_url",
+                      return_value=_mock_fetch_result(
+                          "plain text", content_type="text/plain")):
+        r = fetch_page("https://example.com/file.txt")
+    assert r["status"] == "ok"
+    assert r["text"] == "plain text"
+
+
+def test_fetch_page_blocked_detection():
+    """CAPTCHA pages are marked as error, not ok."""
+    html = "<html>Please complete the CAPTCHA to continue</html>"
+    with patch.object(search_mod, "fetch_url",
+                      return_value=_mock_fetch_result(html)):
+        r = fetch_page("https://example.com")
+    assert r["status"] == "error"
+    assert "Bot protection" in r["error"]
+    assert r["text"] == ""
+
+
+def test_fetch_page_network_error():
+    """Network exceptions return status=error with message."""
+    with patch.object(search_mod, "fetch_url",
+                      side_effect=RuntimeError("HTTP 503 for url")):
+        r = fetch_page("https://example.com")
+    assert r["status"] == "error"
+    assert "503" in r["error"]
+    assert r["text"] == ""
+
+
+def test_fetch_page_truncated_flag():
+    """truncated=True is propagated from fetch_url."""
+    html = "<html>" + "x" * 100 + "</html>"
+    with patch.object(search_mod, "fetch_url",
+                      return_value=_mock_fetch_result(html, truncated=True)):
+        r = fetch_page("https://example.com", max_size=50)
+    assert r["status"] == "ok"
+    assert r["truncated"] is True
+    assert r["truncated_at"] == 50
+
+
+def test_fetch_page_preserves_final_url():
+    """Redirect final_url is captured in the result."""
+    with patch.object(search_mod, "fetch_url",
+                      return_value=_mock_fetch_result(
+                          "<html>x</html>",
+                          final_url="https://final.example.com/page")):
+        r = fetch_page("https://example.com")
+    assert r["final_url"] == "https://final.example.com/page"
+
+
+def test_fetch_page_records_fallback_ua():
+    """user_agent_used is propagated for logging/debugging."""
+    with patch.object(search_mod, "fetch_url",
+                      return_value=_mock_fetch_result(
+                          "<html>x</html>",
+                          ua="Mozilla/5.0 fallback")):
+        r = fetch_page("https://example.com")
+    assert r["user_agent_used"] == "Mozilla/5.0 fallback"
+
+
+# ----- fetch_top_results -----
+
+def test_fetch_top_results_empty():
+    """No results → empty list, no fetch attempts."""
+    assert fetch_top_results({"results": []}, 3) == []
+
+
+def test_fetch_top_results_no_results_key():
+    assert fetch_top_results({}, 3) == []
+
+
+def test_fetch_top_results_dedup_urls():
+    """Duplicate URLs are fetched only once."""
+    results = {"results": [
+        {"url": "https://a.com"},
+        {"url": "https://a.com"},  # dup
+        {"url": "https://b.com"},
+    ]}
+    fetched_urls = []
+
+    def _fake_fetch(url, **kwargs):
+        fetched_urls.append(url)
+        return {"url": url, "status": "ok", "text": "x", "text_length": 1,
+                "truncated": False}
+
+    with patch.object(search_mod, "fetch_page", side_effect=_fake_fetch):
+        out = fetch_top_results(results, 3, request_delay=0)
+    assert len(fetched_urls) == 2  # dedup
+    assert "https://a.com" in fetched_urls
+    assert "https://b.com" in fetched_urls
+
+
+def test_fetch_top_results_limits_count():
+    """Only top N unique URLs are fetched."""
+    results = {"results": [{"url": f"https://x{i}.com"} for i in range(10)]}
+
+    def _fake_fetch(url, **kwargs):
+        return {"url": url, "status": "ok", "text": "x", "text_length": 1,
+                "truncated": False}
+
+    with patch.object(search_mod, "fetch_page", side_effect=_fake_fetch):
+        out = fetch_top_results(results, 3, request_delay=0)
+    assert len(out) == 3
+
+
+def test_fetch_top_results_preserves_order():
+    """Fetched results are reordered to match original result order."""
+    results = {"results": [
+        {"url": "https://a.com"},
+        {"url": "https://b.com"},
+        {"url": "https://c.com"},
+    ]}
+
+    def _fake_fetch(url, **kwargs):
+        return {"url": url, "status": "ok", "text": url[-1], "text_length": 1,
+                "truncated": False}
+
+    with patch.object(search_mod, "fetch_page", side_effect=_fake_fetch):
+        out = fetch_top_results(results, 3, request_delay=0)
+    urls = [f["url"] for f in out]
+    assert urls == ["https://a.com", "https://b.com", "https://c.com"]
+
+
+def test_fetch_top_results_handles_errors():
+    """A failed fetch still appears in output with status=error."""
+    results = {"results": [{"url": "https://ok.com"}, {"url": "https://bad.com"}]}
+
+    def _fake_fetch(url, **kwargs):
+        if "bad" in url:
+            return {"url": url, "status": "error", "error": "503",
+                    "text": "", "text_length": 0, "truncated": False}
+        return {"url": url, "status": "ok", "text": "content",
+                "text_length": 7, "truncated": False}
+
+    with patch.object(search_mod, "fetch_page", side_effect=_fake_fetch):
+        out = fetch_top_results(results, 2, request_delay=0)
+    statuses = {f["url"]: f["status"] for f in out}
+    assert statuses["https://ok.com"] == "ok"
+    assert statuses["https://bad.com"] == "error"
+
+
+def test_fetch_top_results_skips_empty_urls():
+    """Results without a URL are skipped."""
+    results = {"results": [
+        {"url": ""},
+        {"url": "https://real.com"},
+    ]}
+
+    def _fake_fetch(url, **kwargs):
+        return {"url": url, "status": "ok", "text": "x", "text_length": 1,
+                "truncated": False}
+
+    with patch.object(search_mod, "fetch_page", side_effect=_fake_fetch):
+        out = fetch_top_results(results, 3, request_delay=0)
+    assert len(out) == 1
+    assert out[0]["url"] == "https://real.com"

tests/test_html_fallback.py

@@ -0,0 +1,264 @@
+"""Tests for the HTML fallback search path.
+
+Covers: SearXNGHTMLParser (result/suggestion/answer extraction, <time> tag,
+script/style skipping), parse_html_results (shape), search_html (HTTP +
+error), search_single (JSON→HTML fallback ordering).
+"""
+import urllib.error
+from unittest.mock import patch, MagicMock
+
+from search import (
+    SearXNGHTMLParser,
+    parse_html_results,
+    search_html,
+    search_single,
+)
+
+
+# ----- SearXNGHTMLParser: result extraction -----
+
+def test_parser_single_result():
+    """A complete <article class="result"> with title/url/content."""
+    html = """
+    <article class="result result-default category-general">
+      <h3><a href="https://example.com/page">Example Title</a></h3>
+      <p class="content">Snippet text here</p>
+    </article>
+    """
+    parser = SearXNGHTMLParser()
+    parser.feed(html)
+    assert len(parser.results) == 1
+    r = parser.results[0]
+    assert r["title"] == "Example Title"
+    assert r["url"] == "https://example.com/page"
+    assert r["content"] == "Snippet text here"
+
+
+def test_parser_url_header_class():
+    """<a class="url_header"> populates the url field."""
+    html = """
+    <article class="result">
+      <a href="https://hdr.example.com" class="url_header">link</a>
+      <h3><a href="https://title.example.com">Title</a></h3>
+    </article>
+    """
+    parser = SearXNGHTMLParser()
+    parser.feed(html)
+    assert len(parser.results) == 1
+    # url_header takes precedence over h3's href
+    assert parser.results[0]["url"] == "https://hdr.example.com"
+
+
+def test_parser_time_datetime_attr():
+    """<time datetime="..."> populates published_date from the attribute."""
+    html = """
+    <article class="result">
+      <h3><a href="https://x.com">T</a></h3>
+      <time datetime="2024-01-15T10:30:00">Jan 15, 2024</time>
+    </article>
+    """
+    parser = SearXNGHTMLParser()
+    parser.feed(html)
+    assert parser.results[0]["published_date"] == "2024-01-15T10:30:00"
+
+
+def test_parser_time_text_fallback():
+    """When datetime attr is absent, text content is used as published_date."""
+    html = """
+    <article class="result">
+      <h3><a href="https://x.com">T</a></h3>
+      <time>3 days ago</time>
+    </article>
+    """
+    parser = SearXNGHTMLParser()
+    parser.feed(html)
+    assert parser.results[0]["published_date"] == "3 days ago"
+
+
+def test_parser_skips_script_and_style():
+    """<script> and <style> content must not leak into results."""
+    html = """
+    <article class="result">
+      <h3><a href="https://x.com">T</a></h3>
+      <script>var evil = "ignore me";</script>
+      <style>.css { color: red; }</style>
+      <p class="content">real content</p>
+    </article>
+    """
+    parser = SearXNGHTMLParser()
+    parser.feed(html)
+    assert len(parser.results) == 1
+    assert "evil" not in parser.results[0]["content"]
+    assert "css" not in parser.results[0]["content"]
+    assert parser.results[0]["content"] == "real content"
+
+
+def test_parser_multiple_results():
+    html = """
+    <article class="result"><h3><a href="https://a.com">A</a></h3></article>
+    <article class="result"><h3><a href="https://b.com">B</a></h3></article>
+    <article class="result"><h3><a href="https://c.com">C</a></h3></article>
+    """
+    parser = SearXNGHTMLParser()
+    parser.feed(html)
+    assert len(parser.results) == 3
+    assert [r["title"] for r in parser.results] == ["A", "B", "C"]
+
+
+def test_parser_empty_article_skipped():
+    """An <article> with neither title nor url is dropped."""
+    html = '<article class="result"><p class="content">no title or url</p></article>'
+    parser = SearXNGHTMLParser()
+    parser.feed(html)
+    assert len(parser.results) == 0
+
+
+def test_parser_suggestions():
+    """<div id="suggestions"> with <a> tags populates suggestions list."""
+    html = """
+    <div id="suggestions">
+      <a>first suggestion</a>
+      <a>second suggestion</a>
+    </div>
+    """
+    parser = SearXNGHTMLParser()
+    parser.feed(html)
+    assert "first suggestion" in parser.suggestions
+    assert "second suggestion" in parser.suggestions
+
+
+def test_parser_suggestions_class():
+    """class="suggestion" also triggers suggestion capture."""
+    html = '<div class="suggestion"><a>via class</a></div>'
+    parser = SearXNGHTMLParser()
+    parser.feed(html)
+    assert "via class" in parser.suggestions
+
+
+def test_parser_answer_box():
+    """<div class="answer"> or id="answer" populates answers list."""
+    html = '<div class="answer">The answer is 42</div>'
+    parser = SearXNGHTMLParser()
+    parser.feed(html)
+    assert len(parser.answers) == 1
+    assert "42" in parser.answers[0]
+
+
+def test_parser_answer_by_id():
+    html = '<div id="answer">42</div>'
+    parser = SearXNGHTMLParser()
+    parser.feed(html)
+    assert len(parser.answers) == 1
+
+
+# ----- parse_html_results -----
+
+def test_parse_html_results_shape():
+    """parse_html_results returns a dict with all expected keys."""
+    html = """
+    <article class="result"><h3><a href="https://x.com">T</a></h3>
+    <p class="content">c</p></article>
+    <div id="suggestions"><a>s1</a></div>
+    <div class="answer">a1</div>
+    """
+    r = parse_html_results(html, query="test")
+    assert r["query"] == "test"
+    assert len(r["results"]) == 1
+    assert r["suggestions"] == ["s1"]
+    assert r["answers"] == ["a1"]
+    assert r["corrections"] == []
+    assert r["infoboxes"] == []
+    assert r["unresponsive_engines"] == []
+    assert r["_fallback"] == "html"
+    assert r["number_of_results"] == 1
+
+
+def test_parse_html_results_empty():
+    """Empty HTML yields an empty-but-well-shaped result."""
+    r = parse_html_results("", query="q")
+    assert r["results"] == []
+    assert r["query"] == "q"
+
+
+# ----- search_html -----
+
+def _mock_urlopen(data: bytes, content_type="text/html"):
+    resp = MagicMock()
+    resp.read.return_value = data
+    resp.headers = {"Content-Type": content_type}
+    resp.__enter__.return_value = resp
+    resp.__exit__.return_value = None
+    return resp
+
+
+def test_search_html_success():
+    """search_html fetches and parses an HTML results page."""
+    html = """
+    <article class="result"><h3><a href="https://r.com">R</a></h3>
+    <p class="content">snippet</p></article>
+    """
+    with patch("urllib.request.urlopen",
+               return_value=_mock_urlopen(html.encode())):
+        r = search_html("https://s.example.com", {"q": "test"})
+    assert r["results"][0]["title"] == "R"
+    assert r["results"][0]["url"] == "https://r.com"
+
+
+def test_search_html_strips_format_param():
+    """search_html must not pass format=json to the HTML endpoint."""
+    captured_req = {}
+
+    def _capture(req, timeout=None):
+        captured_req["url"] = req.full_url
+        return _mock_urlopen(b"<article class='result'></article>")
+
+    with patch("urllib.request.urlopen", side_effect=_capture):
+        search_html("https://s.example.com", {"q": "test", "format": "json"})
+    assert "format=json" not in captured_req["url"]
+    assert "q=test" in captured_req["url"]
+
+
+def test_search_html_error_raises_runtime():
+    """Network errors are wrapped in RuntimeError with context."""
+    err = urllib.error.URLError("connection refused")
+    with patch("urllib.request.urlopen", side_effect=err):
+        try:
+            search_html("https://s.example.com", {"q": "test"})
+            assert False, "should raise"
+        except RuntimeError as e:
+            assert "HTML search failed" in str(e)
+
+
+# ----- search_single -----
+
+def test_search_single_prefers_json():
+    """When JSON works, search_single returns JSON results directly."""
+    import json as json_mod
+    payload = json_mod.dumps({"results": [{"title": "json", "url": "https://j.com"}]})
+    with patch("urllib.request.urlopen",
+               return_value=_mock_urlopen(payload.encode(),
+                                          content_type="application/json")):
+        r = search_single("https://s.example.com", {"q": "t", "format": "json"})
+    assert r["results"][0]["title"] == "json"
+
+
+def test_search_single_falls_back_to_html():
+    """When JSON returns None, search_single falls back to HTML parsing."""
+    html = """
+    <article class="result"><h3><a href="https://h.com">html</a></h3></article>
+    """
+    # First call returns HTML (JSON unsupported), second call (HTML search) also HTML
+    with patch("urllib.request.urlopen",
+               return_value=_mock_urlopen(html.encode())):
+        r = search_single("https://s.example.com", {"q": "t", "format": "json"})
+    assert r["results"][0]["title"] == "html"
+    assert r.get("_fallback") == "html"
+
+
+def test_search_single_html_fallback_marks_fallback():
+    """HTML fallback path sets _fallback='html' in the result."""
+    html = '<article class="result"><h3><a href="https://x.com">x</a></h3></article>'
+    with patch("urllib.request.urlopen",
+               return_value=_mock_urlopen(html.encode())):
+        r = search_single("https://s.example.com", {"q": "t", "format": "json"})
+    assert r["_fallback"] == "html"

tests/test_instance_resolution.py

@@ -0,0 +1,193 @@
+"""Tests for scripts/search.py — instance resolution chain.
+
+Covers:
+  * ``_load_toml`` — TOML file loading (Python 3.11+ tomllib / 3.8-3.10 tomli)
+  * ``_read_instance_file`` — config file parsing (.toml single/list/table,
+    .txt per-line / comments / comma-separated / blanks, missing file,
+    corrupted toml)
+  * ``resolve_instances`` — priority chain (CLI > env > config file > empty)
+    and config-file auto-discovery (cwd/home, .toml before .txt)
+
+Tests are hermetic: cwd and home are redirected to ``tmp_path`` via
+monkeypatch so the real user environment never interferes.
+"""
+import pytest
+from pathlib import Path
+
+from search import _load_toml, _read_instance_file, resolve_instances
+
+
+# ----- _load_toml -----
+
+def test_load_toml_valid_file(tmp_path):
+    f = tmp_path / "test.toml"
+    f.write_text('key = "value"\nnumber = 42\n', encoding="utf-8")
+    data = _load_toml(f)
+    assert data["key"] == "value"
+    assert data["number"] == 42
+
+
+def test_load_toml_returns_dict(tmp_path):
+    f = tmp_path / "test.toml"
+    f.write_text('instance = "https://x.example.com"\n', encoding="utf-8")
+    data = _load_toml(f)
+    assert isinstance(data, dict)
+
+
+# ----- _read_instance_file: .toml -----
+
+def test_read_toml_single_instance(tmp_path):
+    """Top-level instance = "url" → parse_instances single URL."""
+    f = tmp_path / "searxng.toml"
+    f.write_text('instance = "https://x.example.com"\n', encoding="utf-8")
+    assert _read_instance_file(f) == ["https://x.example.com"]
+
+
+def test_read_toml_instances_list(tmp_path):
+    """Top-level instances = ["a", "b"] → normalized list."""
+    f = tmp_path / "searxng.toml"
+    f.write_text('instances = ["a.com", "b.com"]\n', encoding="utf-8")
+    assert _read_instance_file(f) == ["https://a.com", "https://b.com"]
+
+
+def test_read_toml_searxng_table_instance(tmp_path):
+    """Instance under [searxng] table is read correctly."""
+    f = tmp_path / "searxng.toml"
+    f.write_text('[searxng]\ninstance = "https://t.example.com"\n',
+                 encoding="utf-8")
+    assert _read_instance_file(f) == ["https://t.example.com"]
+
+
+def test_read_corrupted_toml_returns_empty(tmp_path):
+    """Corrupted (unparseable) toml → empty list, no exception raised."""
+    f = tmp_path / "bad.toml"
+    f.write_text("[searxng\ninstance = broken\n", encoding="utf-8")
+    assert _read_instance_file(f) == []
+
+
+# ----- _read_instance_file: .txt -----
+
+def test_read_txt_one_url_per_line(tmp_path):
+    f = tmp_path / "instances.txt"
+    f.write_text("https://a.com\nhttps://b.com\n", encoding="utf-8")
+    assert _read_instance_file(f) == ["https://a.com", "https://b.com"]
+
+
+def test_read_txt_skips_comments(tmp_path):
+    f = tmp_path / "instances.txt"
+    f.write_text("# comment\nhttps://a.com\n# another\nhttps://b.com\n",
+                 encoding="utf-8")
+    assert _read_instance_file(f) == ["https://a.com", "https://b.com"]
+
+
+def test_read_txt_comma_separated(tmp_path):
+    """Comma-separated URLs on one line are split."""
+    f = tmp_path / "instances.txt"
+    f.write_text("a.com,b.com\n", encoding="utf-8")
+    assert _read_instance_file(f) == ["https://a.com", "https://b.com"]
+
+
+def test_read_txt_skips_blank_lines(tmp_path):
+    f = tmp_path / "instances.txt"
+    f.write_text("https://a.com\n\n\nhttps://b.com\n", encoding="utf-8")
+    assert _read_instance_file(f) == ["https://a.com", "https://b.com"]
+
+
+# ----- _read_instance_file: edge cases -----
+
+def test_read_nonexistent_file_returns_empty(tmp_path):
+    """Missing file → empty list (FileNotFoundError caught internally)."""
+    f = tmp_path / "nonexistent.toml"
+    assert _read_instance_file(f) == []
+
+
+# ----- resolve_instances: priority chain -----
+
+def test_resolve_cli_arg_takes_priority_over_env(monkeypatch, tmp_path):
+    """CLI arg wins even when SEARXNG_INSTANCE env var is set."""
+    monkeypatch.setenv("SEARXNG_INSTANCE", "https://env.example.com")
+    assert resolve_instances("https://cli.example.com") == ["https://cli.example.com"]
+
+
+def test_resolve_env_takes_priority_over_config_file(monkeypatch, tmp_path):
+    """Env var wins over a present config file."""
+    monkeypatch.chdir(tmp_path)
+    monkeypatch.setattr(Path, "home", lambda: tmp_path)
+    (tmp_path / "searxng.toml").write_text(
+        'instance = "https://file.example.com"\n', encoding="utf-8")
+    monkeypatch.setenv("SEARXNG_INSTANCE", "https://env.example.com")
+    assert resolve_instances(None) == ["https://env.example.com"]
+
+
+def test_resolve_no_source_returns_empty(monkeypatch, tmp_path):
+    """No CLI, no env, no config file → empty list."""
+    monkeypatch.chdir(tmp_path)
+    monkeypatch.setattr(Path, "home", lambda: tmp_path)
+    monkeypatch.delenv("SEARXNG_INSTANCE", raising=False)
+    assert resolve_instances(None) == []
+
+
+def test_resolve_cli_none_falls_back_to_env(monkeypatch, tmp_path):
+    """cli_arg=None → use SEARXNG_INSTANCE env var."""
+    monkeypatch.chdir(tmp_path)
+    monkeypatch.setattr(Path, "home", lambda: tmp_path)
+    monkeypatch.setenv("SEARXNG_INSTANCE", "https://env.example.com")
+    assert resolve_instances(None) == ["https://env.example.com"]
+
+
+def test_resolve_no_env_falls_back_to_config_file(monkeypatch, tmp_path):
+    """No CLI, no env → fall back to config file."""
+    monkeypatch.chdir(tmp_path)
+    monkeypatch.setattr(Path, "home", lambda: tmp_path)
+    monkeypatch.delenv("SEARXNG_INSTANCE", raising=False)
+    (tmp_path / "searxng.toml").write_text(
+        'instance = "https://file.example.com"\n', encoding="utf-8")
+    assert resolve_instances(None) == ["https://file.example.com"]
+
+
+# ----- resolve_instances: config file discovery -----
+
+def test_resolve_discovers_cwd_searxng_toml(monkeypatch, tmp_path):
+    """./searxng.toml is auto-discovered."""
+    monkeypatch.chdir(tmp_path)
+    monkeypatch.setattr(Path, "home", lambda: tmp_path)
+    monkeypatch.delenv("SEARXNG_INSTANCE", raising=False)
+    (tmp_path / "searxng.toml").write_text(
+        'instance = "https://cwd.example.com"\n', encoding="utf-8")
+    assert resolve_instances(None) == ["https://cwd.example.com"]
+
+
+def test_resolve_discovers_home_searxng_toml(monkeypatch, tmp_path):
+    """~/.config/searxng-cli/searxng.toml is auto-discovered."""
+    monkeypatch.chdir(tmp_path)  # cwd has no config file
+    home = tmp_path / "fake_home"
+    cfg_dir = home / ".config" / "searxng-cli"
+    cfg_dir.mkdir(parents=True)
+    (cfg_dir / "searxng.toml").write_text(
+        'instance = "https://home.example.com"\n', encoding="utf-8")
+    monkeypatch.setattr(Path, "home", lambda: home)
+    monkeypatch.delenv("SEARXNG_INSTANCE", raising=False)
+    assert resolve_instances(None) == ["https://home.example.com"]
+
+
+def test_resolve_discovers_cwd_instances_txt(monkeypatch, tmp_path):
+    """./instances.txt is auto-discovered when no .toml present."""
+    monkeypatch.chdir(tmp_path)
+    monkeypatch.setattr(Path, "home", lambda: tmp_path)
+    monkeypatch.delenv("SEARXNG_INSTANCE", raising=False)
+    (tmp_path / "instances.txt").write_text("https://txt.example.com\n",
+                                            encoding="utf-8")
+    assert resolve_instances(None) == ["https://txt.example.com"]
+
+
+def test_resolve_prefers_toml_over_txt(monkeypatch, tmp_path):
+    """When both ./searxng.toml and ./instances.txt exist, .toml wins."""
+    monkeypatch.chdir(tmp_path)
+    monkeypatch.setattr(Path, "home", lambda: tmp_path)
+    monkeypatch.delenv("SEARXNG_INSTANCE", raising=False)
+    (tmp_path / "searxng.toml").write_text(
+        'instance = "https://toml.example.com"\n', encoding="utf-8")
+    (tmp_path / "instances.txt").write_text("https://txt.example.com\n",
+                                            encoding="utf-8")
+    result = resolve_instances(None)
+    assert result == ["https://toml.example.com"]

tests/test_output_format.py

@@ -0,0 +1,187 @@
+"""Tests for scripts/search.py — output formatting functions.
+
+Pure-function tests (no network, no mocking) covering:
+  * format_brief — title/url/content rendering, snippet truncation,
+    suggestions, answers, empty input
+  * format_urls — basic list, skip-empty-url, empty input
+  * _format_results — json/urls/brief/csv dispatch + fetched-pages section
+"""
+import json
+import types
+
+from search import format_brief, format_urls, _format_results
+
+
+# ----- format_brief -----
+
+def test_format_brief_basic():
+    """Basic brief output includes title, URL, and content lines."""
+    results = {"results": [
+        {"title": "Hello World", "url": "https://example.com/1",
+         "content": "A short snippet."},
+    ]}
+    out = format_brief(results)
+    assert "1. Hello World" in out
+    assert "https://example.com/1" in out
+    assert "A short snippet." in out
+
+
+def test_format_brief_no_content():
+    """When content is empty/missing, no content line is emitted."""
+    results = {"results": [
+        {"title": "No Snippet", "url": "https://example.com/2"},
+    ]}
+    out = format_brief(results)
+    assert "1. No Snippet" in out
+    assert "https://example.com/2" in out
+    # Only title + url lines plus a trailing blank line — no content line.
+    lines = out.split("\n")
+    assert len(lines) == 3
+    assert lines[2] == ""
+
+
+def test_format_brief_snippet_truncation():
+    """snippet_len > 0 truncates content to that many characters."""
+    long_text = "abcdefghijklmnopqrstuvwxyz"
+    results = {"results": [
+        {"title": "T", "url": "https://example.com", "content": long_text},
+    ]}
+    out = format_brief(results, snippet_len=5)
+    assert "abcde" in out
+    assert "abcdef" not in out  # truncated beyond 5 chars
+
+
+def test_format_brief_with_suggestions():
+    """Suggestions list is rendered as a comma-separated line."""
+    results = {
+        "results": [{"title": "T", "url": "https://example.com", "content": "c"}],
+        "suggestions": ["python asyncio", "python requests"],
+    }
+    out = format_brief(results)
+    assert "Suggestions: python asyncio, python requests" in out
+
+
+def test_format_brief_with_answers():
+    """Each answer is emitted on its own 'Answer:' line."""
+    results = {
+        "results": [{"title": "T", "url": "https://example.com", "content": "c"}],
+        "answers": ["42", "the answer"],
+    }
+    out = format_brief(results)
+    assert "Answer: 42" in out
+    assert "Answer: the answer" in out
+
+
+def test_format_brief_empty_results():
+    """Empty results list (or missing key) yields an empty string."""
+    assert format_brief({"results": []}) == ""
+    assert format_brief({}) == ""
+
+
+# ----- format_urls -----
+
+def test_format_urls_basic():
+    """Each result URL appears on its own line."""
+    results = {"results": [
+        {"url": "https://a.com/1"},
+        {"url": "https://b.com/2"},
+    ]}
+    out = format_urls(results)
+    assert out == "https://a.com/1\nhttps://b.com/2"
+
+
+def test_format_urls_skips_empty():
+    """Results with empty/missing URLs are skipped."""
+    results = {"results": [
+        {"url": "https://a.com/1"},
+        {"url": ""},
+        {"title": "no url here"},
+        {"url": "https://b.com/2"},
+    ]}
+    out = format_urls(results)
+    assert out == "https://a.com/1\nhttps://b.com/2"
+
+
+def test_format_urls_empty_results():
+    """Empty results yield an empty string."""
+    assert format_urls({"results": []}) == ""
+    assert format_urls({}) == ""
+
+
+# ----- _format_results -----
+
+def test_format_results_json():
+    """json format emits valid, parseable JSON mirroring the input."""
+    results = {"results": [
+        {"title": "T", "url": "https://example.com", "content": "c"},
+    ], "suggestions": ["x"]}
+    args = types.SimpleNamespace(format="json")
+    out = _format_results(results, args)
+    parsed = json.loads(out)
+    assert parsed == results
+
+
+def test_format_results_brief():
+    """brief format dispatches to format_brief."""
+    results = {"results": [
+        {"title": "T", "url": "https://example.com", "content": "snippet"},
+    ]}
+    args = types.SimpleNamespace(format="brief", snippet_len=0, fetch=0)
+    out = _format_results(results, args)
+    assert "1. T" in out
+    assert "https://example.com" in out
+    assert "snippet" in out
+
+
+def test_format_results_urls():
+    """urls format dispatches to format_urls."""
+    results = {"results": [
+        {"url": "https://a.com/1"},
+        {"url": "https://b.com/2"},
+    ]}
+    args = types.SimpleNamespace(format="urls")
+    out = _format_results(results, args)
+    assert out == "https://a.com/1\nhttps://b.com/2"
+
+
+def test_format_results_csv_smoke():
+    """csv format emits the header row + one row per result (smoke test)."""
+    results = {"results": [
+        {"title": "T1", "url": "https://a.com", "engine": "google",
+         "score": 1.0, "published_date": "2024-01-01", "content": "snip"},
+    ]}
+    args = types.SimpleNamespace(format="csv", snippet_len=0, fetch=0)
+    out = _format_results(results, args)
+    assert "title,url,engine,score,published_date,content" in out
+    assert "T1" in out
+    assert "https://a.com" in out
+
+
+def test_format_results_brief_with_fetched():
+    """brief + args.fetch>0 + results['fetched'] appends a fetched section."""
+    results = {
+        "results": [{"title": "T", "url": "https://example.com", "content": "c"}],
+        "fetched": [
+            {"url": "https://example.com", "status": "ok",
+             "text": "Full page content here."},
+        ],
+    }
+    args = types.SimpleNamespace(format="brief", snippet_len=0, fetch=3)
+    out = _format_results(results, args)
+    assert "FETCHED PAGES (1 pages)" in out
+    assert "--- https://example.com ---" in out
+    assert "Full page content here." in out
+
+
+def test_format_results_brief_with_fetched_error():
+    """Fetched entries with status != 'ok' render an [ERROR: ...] line."""
+    results = {
+        "results": [{"title": "T", "url": "https://example.com", "content": "c"}],
+        "fetched": [
+            {"url": "https://broken.example", "status": "error", "error": "timeout"},
+        ],
+    }
+    args = types.SimpleNamespace(format="brief", snippet_len=0, fetch=3)
+    out = _format_results(results, args)
+    assert "FETCHED PAGES (1 pages)" in out
+    assert "[ERROR: timeout]" in out

tests/test_parallel_search.py

@@ -0,0 +1,276 @@
+"""Parallel-mode tests for search_multi (parallel=True branch).
+
+The serial path (parallel=False / single instance) is covered by
+``test_integration.py``. This module focuses on the ThreadPoolExecutor
+branch: result ordering by user input, failover, all-fail, auth-header
+forwarding, retry_per=0 semantics, and exception capture.
+
+Thread-safety note: ``urllib.request.urlopen`` is patched with a
+URL-dispatching *function* (not a ``side_effect`` list). A function with
+no mutable shared state is safe to call concurrently from multiple
+worker threads, whereas a list-based ``side_effect`` would race on the
+shared iterator. ``MagicMock.call_count`` is itself thread-safe.
+"""
+import json
+import logging
+import time
+import urllib.error
+from unittest.mock import MagicMock, patch
+
+from search import search_multi
+
+
+def _mock_urlopen(data: bytes, content_type="application/json"):
+    """Build a MagicMock that quacks like an urlopen context manager."""
+    resp = MagicMock()
+    resp.read.return_value = data
+    resp.headers = {"Content-Type": content_type}
+    resp.__enter__.return_value = resp
+    resp.__exit__.return_value = None
+    return resp
+
+
+def _url_router(routing=None, default=None):
+    """Build a thread-safe side_effect that dispatches by request URL.
+
+    ``routing`` maps a URL substring to either a response object
+    (returned) or a ``BaseException`` (raised). Entries are checked in
+    insertion order (dicts are ordered on Python 3.7+). ``default`` is
+    used when no key matches; if it is an exception it is raised,
+    otherwise returned. An ``AssertionError`` is raised when nothing
+    matches and no default is set — this makes unexpected calls loud
+    rather than silently returning a MagicMock.
+    """
+    routing = routing or {}
+
+    def _side_effect(req, *args, **kwargs):
+        url = getattr(req, "full_url", str(req))
+        for key, resp in routing.items():
+            if key in url:
+                if isinstance(resp, BaseException):
+                    raise resp
+                return resp
+        if default is not None:
+            if isinstance(default, BaseException):
+                raise default
+            return default
+        raise AssertionError(
+            f"unexpected urlopen for {url!r} (routing={list(routing)})"
+        )
+    return _side_effect
+
+
+def _payload(title: str, url: str = "https://x.com") -> bytes:
+    return json.dumps({"results": [{"title": title, "url": url}]}).encode()
+
+
+# ------------------------------------------------------------------
+# parallel=True: success paths
+# ------------------------------------------------------------------
+
+def test_parallel_both_success_returns_first_in_order():
+    """Both instances succeed → return the first one in user-supplied order."""
+    router = _url_router({
+        "a.example.com": _mock_urlopen(_payload("from-a", "https://a.com")),
+        "b.example.com": _mock_urlopen(_payload("from-b", "https://b.com")),
+    })
+    with patch("urllib.request.urlopen", side_effect=router):
+        r = search_multi(
+            ["https://a.example.com", "https://b.example.com"],
+            {"q": "test", "format": "json"},
+            parallel=True, retry_per=0,
+        )
+    assert r["results"][0]["title"] == "from-a"
+
+
+def test_parallel_first_fails_second_succeeds():
+    """First instance fails, second succeeds → return second's results."""
+    router = _url_router({
+        "a.example.com": urllib.error.URLError("connection refused"),
+        "b.example.com": _mock_urlopen(_payload("from-b", "https://b.com")),
+    })
+    with patch("urllib.request.urlopen", side_effect=router):
+        r = search_multi(
+            ["https://a.example.com", "https://b.example.com"],
+            {"q": "test", "format": "json"},
+            parallel=True, retry_per=0,
+        )
+    assert r["results"][0]["title"] == "from-b"
+
+
+def test_parallel_all_fail_raises():
+    """All instances fail → RuntimeError carrying the '(parallel)' marker."""
+    err = urllib.error.URLError("down")
+    with patch("urllib.request.urlopen", side_effect=_url_router(default=err)):
+        try:
+            search_multi(
+                ["https://a.example.com", "https://b.example.com"],
+                {"q": "test", "format": "json"},
+                parallel=True, retry_per=0,
+            )
+            assert False, "should have raised RuntimeError"
+        except RuntimeError as e:
+            assert "(parallel)" in str(e)
+            assert "2" in str(e)
+
+
+def test_parallel_three_middle_succeeds():
+    """Three instances, only the middle one succeeds → return middle."""
+    router = _url_router({
+        "a.example.com": urllib.error.URLError("down"),
+        "b.example.com": _mock_urlopen(_payload("middle-b", "https://b.com")),
+        "c.example.com": urllib.error.URLError("down"),
+    })
+    with patch("urllib.request.urlopen", side_effect=router):
+        r = search_multi(
+            ["https://a.example.com", "https://b.example.com",
+             "https://c.example.com"],
+            {"q": "test", "format": "json"},
+            parallel=True, retry_per=0,
+        )
+    assert r["results"][0]["title"] == "middle-b"
+
+
+def test_parallel_returns_user_order_not_completion_order():
+    """Result follows user input order, NOT completion order.
+
+    Instance 'a' is deliberately slowed down so 'b' finishes first, yet
+    'a' (listed first) must still be the returned result when both
+    succeed. This guards the ``for u in instance_urls`` deterministic
+    selection at the end of the parallel branch.
+    """
+    resp_a = _mock_urlopen(_payload("slow-a", "https://a.com"))
+    resp_b = _mock_urlopen(_payload("fast-b", "https://b.com"))
+
+    def _side_effect(req, *args, **kwargs):
+        url = getattr(req, "full_url", str(req))
+        if "a.example.com" in url:
+            time.sleep(0.15)  # 'a' finishes after 'b'
+            return resp_a
+        if "b.example.com" in url:
+            return resp_b
+        raise AssertionError(f"unexpected urlopen for {url!r}")
+
+    with patch("urllib.request.urlopen", side_effect=_side_effect):
+        r = search_multi(
+            ["https://a.example.com", "https://b.example.com"],
+            {"q": "test", "format": "json"},
+            parallel=True, retry_per=0,
+        )
+    assert r["results"][0]["title"] == "slow-a"
+
+
+# ------------------------------------------------------------------
+# parallel=True: serial-path fallback conditions
+# ------------------------------------------------------------------
+
+def test_parallel_single_instance_uses_serial_path():
+    """A single instance (len <= 1) takes the serial path even with
+    parallel=True. Distinguishable by error message: serial path says
+    'Last error', parallel path says '(parallel)'.
+    """
+    err = urllib.error.URLError("down")
+    with patch("urllib.request.urlopen", side_effect=err):
+        try:
+            search_multi(
+                ["https://a.example.com"],
+                {"q": "test", "format": "json"},
+                parallel=True, retry_per=0,
+            )
+            assert False, "should have raised RuntimeError"
+        except RuntimeError as e:
+            assert "Last error" in str(e)
+            assert "(parallel)" not in str(e)
+
+
+def test_parallel_false_explicit_serial_path():
+    """parallel=False explicitly forces the serial path for >1 instances."""
+    err = urllib.error.URLError("down")
+    with patch("urllib.request.urlopen", side_effect=err):
+        try:
+            search_multi(
+                ["https://a.example.com", "https://b.example.com"],
+                {"q": "test", "format": "json"},
+                parallel=False, retry_per=0,
+            )
+            assert False, "should have raised RuntimeError"
+        except RuntimeError as e:
+            assert "Last error" in str(e)
+            assert "(parallel)" not in str(e)
+
+
+# ------------------------------------------------------------------
+# parallel=True: header forwarding & retry semantics
+# ------------------------------------------------------------------
+
+def test_parallel_auth_headers_passed():
+    """auth_headers are forwarded to every instance request."""
+    captured = []  # list.append is atomic under CPython's GIL
+    payload = _payload("ok", "https://x.com")
+
+    def _side_effect(req, *args, **kwargs):
+        captured.append(req.headers)
+        return _mock_urlopen(payload)
+
+    with patch("urllib.request.urlopen", side_effect=_side_effect):
+        r = search_multi(
+            ["https://a.example.com", "https://b.example.com"],
+            {"q": "test", "format": "json"},
+            parallel=True, retry_per=0,
+            auth_headers={"Authorization": "Bearer secret-token"},
+        )
+    assert r["results"][0]["title"] == "ok"
+    # Both instances were called exactly once (success path, no HTML fallback)
+    assert len(captured) == 2
+    for hdrs in captured:
+        assert hdrs.get("Authorization") == "Bearer secret-token"
+
+
+def test_parallel_retry_per_zero_no_retry():
+    """retry_per=0 → exactly one attempt per instance, no backoff retries.
+
+    Three failing instances must produce exactly 3 urlopen calls total.
+    MagicMock.call_count is thread-safe, so concurrent increments are
+    observed correctly after the executor joins.
+    """
+    err = urllib.error.URLError("down")
+    mock_open = MagicMock(side_effect=err)
+    with patch("urllib.request.urlopen", mock_open):
+        try:
+            search_multi(
+                ["https://a.example.com", "https://b.example.com",
+                 "https://c.example.com"],
+                {"q": "test", "format": "json"},
+                parallel=True, retry_per=0,
+            )
+        except RuntimeError:
+            pass  # expected: all failed
+    # 3 instances × 1 attempt (retry_per=0) = 3 calls, no retries
+    assert mock_open.call_count == 3
+
+
+def test_parallel_exception_caught_and_logged(caplog):
+    """A raising instance is caught and logged; the survivor's result wins.
+
+    Instance 'a' raises HTTPError 403 (not retryable, re-raised by
+    _retry_with_backoff). The parallel _task wrapper must catch it,
+    record it as a failure, and let instance 'b' succeed.
+    """
+    router = _url_router({
+        "a.example.com": urllib.error.HTTPError(
+            "url", 403, "Forbidden", {}, None),
+        "b.example.com": _mock_urlopen(_payload("survivor-b", "https://b.com")),
+    })
+    with patch("urllib.request.urlopen", side_effect=router), \
+         caplog.at_level(logging.INFO, logger="searxng.search"):
+        r = search_multi(
+            ["https://a.example.com", "https://b.example.com"],
+            {"q": "test", "format": "json"},
+            parallel=True, retry_per=0,
+        )
+    assert r["results"][0]["title"] == "survivor-b"
+    # The failure was logged (line: logger.info(f"  Failed {u}: {res}"))
+    assert any(
+        "Failed" in rec.message and "a.example.com" in rec.message
+        for rec in caplog.records
+    ), f"expected a 'Failed ... a.example.com' log line, got: {[r.message for r in caplog.records]}"

tests/test_stream_progress.py

@@ -0,0 +1,207 @@
+"""Tests for --stream (JSON Lines output) and --progress (progress events).
+
+Covers: stream output format (result/done/error events), progress event
+emission (start/cache_hit/cache_store/done/error/fetch_*), progress
+disabled by default, stream only works with --format json.
+"""
+import json
+import sys
+from types import SimpleNamespace
+from unittest.mock import patch
+import pytest
+
+import common
+from common import emit_progress, set_progress_enabled
+import search as search_mod
+from search import _run_single_query, main
+
+
+# ----- emit_progress: default disabled -----
+
+def test_progress_disabled_by_default(capsys):
+    """Without --progress, emit_progress is a no-op."""
+    set_progress_enabled(False)
+    emit_progress("start", query="test", instances=1)
+    out, err = capsys.readouterr()
+    assert out == ""
+    assert err == ""
+
+
+def test_progress_enabled_emits_json(capsys):
+    """With --progress, emit_progress outputs JSON Lines to stderr."""
+    set_progress_enabled(True)
+    try:
+        emit_progress("start", query="test", instances=2)
+        _, err = capsys.readouterr()
+        data = json.loads(err.strip())
+        assert data["event"] == "start"
+        assert data["query"] == "test"
+        assert data["instances"] == 2
+    finally:
+        set_progress_enabled(False)
+
+
+def test_progress_multiple_events(capsys):
+    """Multiple events produce multiple JSON Lines."""
+    set_progress_enabled(True)
+    try:
+        emit_progress("start", query="q", instances=1)
+        emit_progress("cache_hit", query="q", ttl=30)
+        emit_progress("done", results=5, query="q")
+        _, err = capsys.readouterr()
+        lines = [l for l in err.strip().split("\n") if l]
+        assert len(lines) == 3
+        events = [json.loads(l)["event"] for l in lines]
+        assert events == ["start", "cache_hit", "done"]
+    finally:
+        set_progress_enabled(False)
+
+
+def test_progress_event_with_error_code(capsys):
+    """Error events include error_code field."""
+    set_progress_enabled(True)
+    try:
+        emit_progress("error", error="timeout", error_code="E_NETWORK",
+                      query="test")
+        _, err = capsys.readouterr()
+        data = json.loads(err.strip())
+        assert data["error_code"] == "E_NETWORK"
+    finally:
+        set_progress_enabled(False)
+
+
+# ----- _run_single_query: progress events -----
+
+def _make_args(**overrides):
+    """Construct a minimal args object for _run_single_query."""
+    defaults = dict(
+        query="test", format="json", method="GET", timeout=15, retry=0,
+        serial=False, no_dedup=False, sort_by="none", max_results=None,
+        include_domain=None, exclude_domain=None, fetch=0, fetch_timeout=10,
+        fetch_retries=3, max_size=None, cache_ttl=0, snippet_len=0,
+        categories=None, language=None, pageno=1, time_range="year",
+        safesearch=0, engines="google,bing",
+    )
+    defaults.update(overrides)
+    return SimpleNamespace(**defaults)
+
+
+def test_run_single_query_emits_start_and_done(capsys):
+    """_run_single_query emits start and done events when progress is enabled."""
+    set_progress_enabled(True)
+    try:
+        mock_results = {"results": [{"title": "t", "url": "https://x.com"}]}
+        with patch.object(search_mod, "search_multi", return_value=mock_results):
+            _run_single_query("test", _make_args(),
+                              ["https://x.example.com"], {}, 0)
+        _, err = capsys.readouterr()
+        lines = [json.loads(l) for l in err.strip().split("\n") if l]
+        events = [e["event"] for e in lines]
+        assert "start" in events
+        assert "done" in events
+        done_event = next(e for e in lines if e["event"] == "done")
+        assert done_event["results"] == 1
+    finally:
+        set_progress_enabled(False)
+
+
+def test_run_single_query_emits_cache_hit(capsys, isolated_cache):
+    """Cache hit emits cache_hit event."""
+    set_progress_enabled(True)
+    try:
+        mock_results = {"results": [{"title": "t", "url": "https://x.com"}]}
+        with patch.object(search_mod, "search_multi", return_value=mock_results):
+            # First call: cache miss, stores result
+            _run_single_query("test", _make_args(cache_ttl=30),
+                              ["https://x.example.com"], {}, 1800)
+            capsys.readouterr()  # clear
+            # Second call: cache hit
+            _run_single_query("test", _make_args(cache_ttl=30),
+                              ["https://x.example.com"], {}, 1800)
+        _, err = capsys.readouterr()
+        lines = [json.loads(l) for l in err.strip().split("\n") if l]
+        events = [e["event"] for e in lines]
+        assert "cache_hit" in events
+    finally:
+        set_progress_enabled(False)
+
+
+def test_run_single_query_emits_error_on_failure(capsys):
+    """Search failure emits error event with error_code."""
+    set_progress_enabled(True)
+    try:
+        import urllib.error
+        err = urllib.error.HTTPError("url", 403, "Forbidden", {}, None)
+        with patch.object(search_mod, "search_multi", side_effect=err):
+            _run_single_query("test", _make_args(),
+                              ["https://x.example.com"], {}, 0)
+        _, err_out = capsys.readouterr()
+        lines = [json.loads(l) for l in err_out.strip().split("\n") if l]
+        error_events = [e for e in lines if e["event"] == "error"]
+        assert len(error_events) == 1
+        assert error_events[0]["error_code"] == "E_AUTH"
+    finally:
+        set_progress_enabled(False)
+
+
+# ----- --stream: JSON Lines output -----
+
+def test_stream_outputs_json_lines(capsys):
+    """--stream outputs each result as a JSON Line + a done event."""
+    mock_results = {"results": [
+        {"title": "first", "url": "https://a.com"},
+        {"title": "second", "url": "https://b.com"},
+    ]}
+    with patch.object(search_mod, "search_multi", return_value=mock_results):
+        with pytest.raises(SystemExit) as exc_info:
+            main.__wrapped__ if hasattr(main, "__wrapped__") else None
+            # Call main with --stream
+            with patch.object(sys, "argv", ["search.py", "-q", "test",
+                                            "-i", "https://x.example.com",
+                                            "--stream", "--format", "json"]):
+                # Mock the early argparse for logging
+                with patch.object(search_mod, "setup_logging"):
+                    main()
+        assert exc_info.value.code == 0
+    out, _ = capsys.readouterr()
+    lines = [json.loads(l) for l in out.strip().split("\n") if l]
+    # Should have 2 result events + 1 done event
+    result_events = [e for e in lines if e["type"] == "result"]
+    done_events = [e for e in lines if e["type"] == "done"]
+    assert len(result_events) == 2
+    assert len(done_events) == 1
+    assert done_events[0]["count"] == 2
+    assert result_events[0]["result"]["title"] == "first"
+
+
+def test_stream_empty_results_exit_2(capsys):
+    """--stream with empty results exits with code 2 and emits done with count=0."""
+    mock_results = {"results": []}
+    with patch.object(search_mod, "search_multi", return_value=mock_results):
+        with pytest.raises(SystemExit) as exc_info:
+            with patch.object(sys, "argv", ["search.py", "-q", "test",
+                                            "-i", "https://x.example.com",
+                                            "--stream", "--format", "json"]):
+                with patch.object(search_mod, "setup_logging"):
+                    main()
+    assert exc_info.value.code == 2
+    out, _ = capsys.readouterr()
+    lines = [json.loads(l) for l in out.strip().split("\n") if l]
+    done_events = [e for e in lines if e["type"] == "done"]
+    assert len(done_events) == 1
+    assert done_events[0]["count"] == 0
+
+
+def test_stream_result_event_shape():
+    """Each result event has type=result and result=<result dict>."""
+    # Unit test the stream output logic directly
+    results = [{"title": "t", "url": "https://x.com"}]
+    lines = []
+    for r in results:
+        lines.append(json.dumps({"type": "result", "result": r}))
+    lines.append(json.dumps({"type": "done", "count": len(results)}))
+    parsed = [json.loads(l) for l in lines]
+    assert parsed[0]["type"] == "result"
+    assert parsed[0]["result"]["url"] == "https://x.com"
+    assert parsed[-1]["type"] == "done"
+    assert parsed[-1]["count"] == 1

tests/test_verify.py

@@ -0,0 +1,214 @@
+"""Tests for the --verify instance health-check feature.
+
+Covers: _probe_config_endpoint (reachable/disabled/error), verify_instances
+(reachability, JSON support, POST probe, latency, auth status, error
+classification), _print_verify_report (JSON and table output).
+"""
+import json
+import urllib.error
+from unittest.mock import patch, MagicMock
+
+from search import _probe_config_endpoint, verify_instances, _print_verify_report
+
+
+def _mock_urlopen(data: bytes, content_type="application/json"):
+    """Build a MagicMock that quacks like an urlopen context manager."""
+    resp = MagicMock()
+    resp.read.return_value = data
+    resp.headers = {"Content-Type": content_type}
+    resp.__enter__.return_value = resp
+    resp.__exit__.return_value = None
+    return resp
+
+
+# ----- _probe_config_endpoint -----
+
+def test_probe_config_reachable():
+    """A working /config endpoint returns engines and categories."""
+    payload = json.dumps({
+        "engines": [{"name": "google"}, {"name": "bing"}, {"name": "wikipedia"}],
+        "categories": {"general": [], "news": [], "images": []},
+    })
+    with patch("urllib.request.urlopen",
+               return_value=_mock_urlopen(payload.encode())):
+        r = _probe_config_endpoint("https://s.example.com", timeout=10)
+    assert r["reachable"] is True
+    assert r["engines"] == ["google", "bing", "wikipedia"]
+    assert set(r["categories"]) == {"general", "news", "images"}
+    assert r["error"] is None
+
+
+def test_probe_config_categories_as_list():
+    """Some instances return categories as a list instead of a dict."""
+    payload = json.dumps({
+        "engines": [{"name": "google"}],
+        "categories": ["general", "news"],
+    })
+    with patch("urllib.request.urlopen",
+               return_value=_mock_urlopen(payload.encode())):
+        r = _probe_config_endpoint("https://s.example.com", timeout=10)
+    assert r["reachable"] is True
+    assert r["categories"] == ["general", "news"]
+
+
+def test_probe_config_http_error():
+    """/config returns 403 → reachable=False with HTTP code in error."""
+    err = urllib.error.HTTPError("url", 403, "Forbidden", {}, None)
+    with patch("urllib.request.urlopen", side_effect=err):
+        r = _probe_config_endpoint("https://s.example.com", timeout=10)
+    assert r["reachable"] is False
+    assert "403" in r["error"]
+    assert r["engines"] == []
+
+
+def test_probe_config_connection_error():
+    """Connection error → reachable=False with error message."""
+    err = urllib.error.URLError("connection refused")
+    with patch("urllib.request.urlopen", side_effect=err):
+        r = _probe_config_endpoint("https://s.example.com", timeout=10)
+    assert r["reachable"] is False
+    assert r["engines"] == []
+    assert "connection refused" in r["error"]
+
+
+# ----- verify_instances -----
+
+def _mock_json_search_response(results=None):
+    """Mock a successful JSON search response."""
+    payload = json.dumps({"results": results or [{"title": "t", "url": "https://x.com"}]})
+    return _mock_urlopen(payload.encode(), content_type="application/json")
+
+
+def _mock_html_search_response():
+    """Mock an HTML search response (JSON disabled)."""
+    return _mock_urlopen(b"<html>not json</html>", content_type="text/html")
+
+
+def test_verify_reachable_json_supported():
+    """Instance returns JSON → reachable + json_supported + post probe."""
+    with patch("urllib.request.urlopen",
+               return_value=_mock_json_search_response()):
+        # /config also needs to respond
+        config_payload = json.dumps({"engines": [{"name": "google"}], "categories": {}})
+        config_resp = _mock_urlopen(config_payload.encode())
+        with patch("urllib.request.urlopen",
+                   return_value=_mock_json_search_response()):
+            report = verify_instances(["https://s.example.com"])
+    r = report[0]
+    assert r["url"] == "https://s.example.com"
+    assert r["reachable"] is True
+    assert r["json_supported"] is True
+
+
+def test_verify_html_only_instance():
+    """Instance returns HTML (no JSON) → reachable but json_supported=False."""
+    with patch("urllib.request.urlopen",
+               return_value=_mock_html_search_response()):
+        report = verify_instances(["https://s.example.com"])
+    r = report[0]
+    assert r["reachable"] is True
+    assert r["json_supported"] is False
+
+
+def test_verify_auth_rejected():
+    """401/403 with auth → auth_status='rejected'."""
+    err = urllib.error.HTTPError("url", 403, "Forbidden", {}, None)
+    auth_headers = {"Authorization": "Bearer token123"}
+    with patch("urllib.request.urlopen", side_effect=err):
+        report = verify_instances(["https://s.example.com"],
+                                  auth_headers=auth_headers)
+    r = report[0]
+    assert r["auth_status"] == "rejected"
+
+
+def test_verify_no_auth_returns_na():
+    """Without auth headers, auth_status is 'n/a'."""
+    err = urllib.error.URLError("timeout")
+    with patch("urllib.request.urlopen", side_effect=err):
+        report = verify_instances(["https://s.example.com"])
+    r = report[0]
+    assert r["auth_status"] == "n/a"
+
+
+def test_verify_connection_error():
+    """Connection error → reachable=False, latency=None (no response received)."""
+    err = urllib.error.URLError("connection refused")
+    with patch("urllib.request.urlopen", side_effect=err):
+        report = verify_instances(["https://s.example.com"])
+    r = report[0]
+    assert r["reachable"] is False
+    assert r["json_supported"] is False
+    assert r["latency"] is None  # no response → no latency
+    assert "connection refused" in r["error"]
+
+
+def test_verify_preserves_input_order():
+    """Multiple instances: report preserves the original input order."""
+    urls = ["https://a.example.com", "https://b.example.com", "https://c.example.com"]
+    with patch("urllib.request.urlopen",
+               return_value=_mock_json_search_response()):
+        report = verify_instances(urls)
+    assert [r["url"] for r in report] == urls
+
+
+def test_verify_has_latency():
+    """Latency is a positive float for reachable instances."""
+    with patch("urllib.request.urlopen",
+               return_value=_mock_json_search_response()):
+        report = verify_instances(["https://s.example.com"])
+    assert report[0]["latency"] is not None
+    assert report[0]["latency"] >= 0
+
+
+def test_verify_has_result_count():
+    """Result count from the test query is recorded."""
+    with patch("urllib.request.urlopen",
+               return_value=_mock_json_search_response(
+                   [{"title": "a"}, {"title": "b"}, {"title": "c"}])):
+        report = verify_instances(["https://s.example.com"])
+    assert report[0]["result_count"] == 3
+
+
+# ----- _print_verify_report -----
+
+def test_print_report_json(capsys):
+    """JSON output is valid JSON with the full report."""
+    report = [
+        {"url": "https://a.com", "reachable": True, "json_supported": True,
+         "post_supported": True, "latency": 0.5, "result_count": 10,
+         "engines": ["google"], "auth_status": "n/a", "error": None,
+         "config_endpoint": True},
+    ]
+    _print_verify_report(report, as_json=True)
+    out, _ = capsys.readouterr()
+    data = json.loads(out)
+    assert data[0]["url"] == "https://a.com"
+    assert data[0]["reachable"] is True
+
+
+def test_print_report_table(capsys):
+    """Table output includes the header and a summary line."""
+    report = [
+        {"url": "https://a.com", "reachable": True, "json_supported": True,
+         "post_supported": True, "latency": 0.5, "result_count": 10,
+         "engines": ["google", "bing"], "auth_status": "n/a", "error": None,
+         "config_endpoint": True},
+        {"url": "https://b.com", "reachable": False, "json_supported": False,
+         "post_supported": None, "latency": None, "result_count": None,
+         "engines": [], "auth_status": "n/a", "error": "timeout",
+         "config_endpoint": None},
+    ]
+    _print_verify_report(report, as_json=False)
+    out, _ = capsys.readouterr()
+    assert "URL" in out
+    assert "REACH" in out
+    assert "https://a.com" in out
+    assert "https://b.com" in out
+    assert "1/2 instances reachable" in out
+
+
+def test_print_report_empty(capsys):
+    """Empty report produces a table with 0/0 summary."""
+    _print_verify_report([], as_json=False)
+    out, _ = capsys.readouterr()
+    assert "0/0 instances reachable" in out