feat(v1.8.1): Windows 兼容性修复 + SKILL.md 铁律区块
Windows 兼容性修复(基于真实使用痛点):
- force_utf8_stdout(): 强制 stdout/stderr 为 UTF-8,修复 Windows GBK 崩溃(print('\\xa0') 不再炸)
- resolve_instances/load_config 新增 %APPDATA%/searxng-cli/ 路径,覆盖 Windows 配置约定
- fetch.py 失败诊断增强:输出 status_code=/cause=/url= 字段,AI Agent 可程序化区分 404/403/DNS 失败
SKILL.md 铁律区块(5 条,置顶):
- stdout=数据/stderr=日志 永不混淆
- 禁用 2>/dev/null(丢弃 stderr = 失败时零诊断)
- 排错去 --quiet 加 --verbose
- 配置查找覆盖 WSL + Windows 双路径
- 实例 URL 必填,公共实例发现已移除
测试: 352 -> 362(新增 10 个:force_utf8_stdout 幂等性/GBK 替换/非 ASCII 打印/APPDATA 路径发现/txt 回退/空 APPDATA)
This commit is contained in:
@@ -131,7 +131,7 @@ AI Agent 无需每次传入实例 URL 和认证信息,支持通过配置文件
|
|||||||
|
|
||||||
1. **CLI 参数**:`-i https://your-instance`(最高优先级)
|
1. **CLI 参数**:`-i https://your-instance`(最高优先级)
|
||||||
2. **环境变量**:`export SEARXNG_INSTANCE="https://your-instance"`
|
2. **环境变量**:`export SEARXNG_INSTANCE="https://your-instance"`
|
||||||
3. **配置文件**:`./searxng.toml` 或 `~/.config/searxng-cli/searxng.toml`
|
3. **配置文件**:`./searxng.toml` → `~/.config/searxng-cli/searxng.toml` → `%APPDATA%/searxng-cli/searxng.toml`(Windows)
|
||||||
|
|
||||||
```toml
|
```toml
|
||||||
# searxng.toml
|
# searxng.toml
|
||||||
@@ -263,12 +263,15 @@ python scripts/fetch.py -u https://example.com \
|
|||||||
**工程**
|
**工程**
|
||||||
- 共享 `common.py`(统一重试/字符集/认证/日志)
|
- 共享 `common.py`(统一重试/字符集/认证/日志)
|
||||||
- 结构化日志(`--verbose` / `--quiet`)
|
- 结构化日志(`--verbose` / `--quiet`)
|
||||||
|
- UTF-8 stdout 强制(`force_utf8_stdout()`,修复 Windows GBK 崩溃)
|
||||||
|
- Windows 配置路径发现(`%APPDATA%/searxng-cli/`)
|
||||||
|
- fetch.py 失败诊断增强(`status_code=` / `cause=` / `url=` 字段)
|
||||||
- 结构化错误码 + `recovery_hint` 恢复建议(`E_NETWORK` / `E_AUTH` / `E_RATE_LIMIT` 等)
|
- 结构化错误码 + `recovery_hint` 恢复建议(`E_NETWORK` / `E_AUTH` / `E_RATE_LIMIT` 等)
|
||||||
- JSON 输出含 `schema_version` 字段,`--dump-schema` 输出 JSON Schema 文档
|
- JSON 输出含 `schema_version` 字段,`--dump-schema` 输出 JSON Schema 文档
|
||||||
- JSON Lines 流式输出(`--stream`,含 `error` 事件类型)
|
- JSON Lines 流式输出(`--stream`,含 `error` 事件类型)
|
||||||
- 进度事件(`--progress`,含 `instance_try`/`instance_ok`/`instance_fail`,JSON Lines 到 stderr)
|
- 进度事件(`--progress`,含 `instance_try`/`instance_ok`/`instance_fail`,JSON Lines 到 stderr)
|
||||||
- batch 模式统一 schema(`status` 字段区分成功/失败)
|
- batch 模式统一 schema(`status` 字段区分成功/失败)
|
||||||
- 352 个单元+集成测试
|
- 362 个单元+集成测试
|
||||||
|
|
||||||
## 跨 Agent 兼容性
|
## 跨 Agent 兼容性
|
||||||
|
|
||||||
@@ -296,7 +299,7 @@ pip install pytest
|
|||||||
pytest -q
|
pytest -q
|
||||||
```
|
```
|
||||||
|
|
||||||
352 个测试覆盖:缓存操作、认证解析、域名过滤、Markdown 转换、搜索逻辑、集成流程、日志配置、HTML 回退、自动抓取、健康检查、输出格式化、实例解析、并行搜索、CLI 端到端、错误码分类、流式输出、进度事件、配置文件认证、schema_version、recovery_hint、batch 统一 schema、--dump-schema。
|
362 个测试覆盖:缓存操作、认证解析、域名过滤、Markdown 转换、搜索逻辑、集成流程、日志配置、HTML 回退、自动抓取、健康检查、输出格式化、实例解析、并行搜索、CLI 端到端、错误码分类、流式输出、进度事件、配置文件认证、schema_version、recovery_hint、batch 统一 schema、--dump-schema、UTF-8 stdout 强制、Windows APPDATA 路径。
|
||||||
|
|
||||||
## 项目结构
|
## 项目结构
|
||||||
|
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
---
|
---
|
||||||
name: searxng-use-cli
|
name: searxng-use-cli
|
||||||
description: Use when you need to search the web via your OWN SearXNG instance (no public-instance discovery). 3 CLI scripts + a shared common.py module — execute privacy-respecting searches against a user-supplied instance (with multi-instance failover, 5xx/429 retry, auto-fetch) or via SEARXNG_INSTANCE env / config file, fetch/extract readable text or markdown from web pages. Zero-config replacement for proprietary search APIs.
|
description: Use when you need to search the web via your OWN SearXNG instance (no public-instance discovery). 3 CLI scripts + a shared common.py module — execute privacy-respecting searches against a user-supplied instance (with multi-instance failover, 5xx/429 retry, auto-fetch) or via SEARXNG_INSTANCE env / config file, fetch/extract readable text or markdown from web pages. Zero-config replacement for proprietary search APIs.
|
||||||
version: 1.8.0
|
version: 1.8.1
|
||||||
author: Metona Team
|
author: Metona Team
|
||||||
license: MIT
|
license: MIT
|
||||||
platforms: [linux, macos, windows]
|
platforms: [linux, macos, windows]
|
||||||
@@ -13,6 +13,20 @@ metadata:
|
|||||||
|
|
||||||
# SearXNG CLI Toolkit
|
# SearXNG CLI Toolkit
|
||||||
|
|
||||||
|
## 铁律(AI Agent 必读,违反会导致致命后果)
|
||||||
|
|
||||||
|
> **这些规则优先级最高,必须在任何调用前遵守。**
|
||||||
|
|
||||||
|
1. **stdout=数据,stderr=日志,永不混淆。** stdout 只输出 JSON/CSV/文本数据,stderr 只输出日志/进度/警告。AI 解析 stdout,人类看 stderr。违反:AI 会把日志当数据解析,结果错乱。
|
||||||
|
|
||||||
|
2. **绝对不要 `2>/dev/null` 或重定向 stderr 到 stdout。** stderr 携带排错关键信息(重试日志、HTTP 状态码、缓存命中、认证警告)。丢弃 stderr = 失败时零诊断信息,无法定位原因。需要静默时用 `--quiet`(仅抑制进度,保留 WARNING+ERROR),不要丢弃 stderr。
|
||||||
|
|
||||||
|
3. **排错时第一步:去掉 `--quiet`,加 `--verbose`。** `--quiet` 只留 WARNING 级别,会吞掉 INFO 级别的重试日志、缓存命中提示、实例切换记录。诊断失败时必须用 `--verbose` 看到完整 HTTP 请求/响应/重试链。
|
||||||
|
|
||||||
|
4. **配置查找必须覆盖双路径(WSL + Windows)。** Windows 环境下配置可能在 `~/.config/searxng-cli/`(WSL HOME)或 `%APPDATA%/searxng-cli/`(Windows APPDATA)。检查配置存在性时两个路径都要查,否则会误判"无配置"并反问用户。
|
||||||
|
|
||||||
|
5. **实例 URL 必填,公共实例发现已移除。** 必须通过 `-i`、`SEARXNG_INSTANCE` 环境变量、或配置文件提供实例 URL。无实例时 `search.py` 报 `E_CONFIG` 退出,不要尝试猜测或硬编码公共实例。
|
||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
SearXNG is a privacy-respecting metasearch engine that aggregates results from 70+ search services without tracking users. This skill provides three standalone Python CLI scripts — works with **any AI agent** (Hermes, Claude Code, Codex, OpenCode, Cursor, Trae, etc.) or directly from your terminal.
|
SearXNG is a privacy-respecting metasearch engine that aggregates results from 70+ search services without tracking users. This skill provides three standalone Python CLI scripts — works with **any AI agent** (Hermes, Claude Code, Codex, OpenCode, Cursor, Trae, etc.) or directly from your terminal.
|
||||||
@@ -40,7 +54,7 @@ SearXNG is a privacy-respecting metasearch engine that aggregates results from 7
|
|||||||
**Caching & config**
|
**Caching & config**
|
||||||
- SQLite result caching (`--cache-ttl`) — identical queries within a TTL skip the network entirely; `--clear-cache` / `--cache-stats` manage it
|
- SQLite result caching (`--cache-ttl`) — identical queries within a TTL skip the network entirely; `--clear-cache` / `--cache-stats` manage it
|
||||||
- Config file (`searxng.toml`) pre-sets most flags; `--config FILE` loads a non-default config; `instances.txt` for plain URL lists
|
- Config file (`searxng.toml`) pre-sets most flags; `--config FILE` loads a non-default config; `instances.txt` for plain URL lists
|
||||||
- Instance resolution priority: `-i` → `SEARXNG_INSTANCE` env → config file
|
- Instance resolution priority: `-i` → `SEARXNG_INSTANCE` env → config file (`./searxng.toml` → `~/.config/searxng-cli/searxng.toml` → `%APPDATA%/searxng-cli/searxng.toml` on Windows)
|
||||||
|
|
||||||
**Network & auth**
|
**Network & auth**
|
||||||
- Proxy support (`--proxy`) for both search and fetch (sets `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY`)
|
- Proxy support (`--proxy`) for both search and fetch (sets `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY`)
|
||||||
@@ -52,6 +66,9 @@ SearXNG is a privacy-respecting metasearch engine that aggregates results from 7
|
|||||||
- Shared `common.py` module — unified retry/charset/auth/logging logic across both scripts
|
- Shared `common.py` module — unified retry/charset/auth/logging logic across both scripts
|
||||||
- `search.py --fetch` reuses `fetch.py`'s higher-quality text extractor (no code duplication)
|
- `search.py --fetch` reuses `fetch.py`'s higher-quality text extractor (no code duplication)
|
||||||
- Structured logging (`--verbose` / `--quiet`) — three levels: default INFO (progress + warnings), `--verbose` DEBUG (HTTP detail, cache keys), `--quiet` WARNING (errors only). All log output to stderr; stdout reserved for data
|
- Structured logging (`--verbose` / `--quiet`) — three levels: default INFO (progress + warnings), `--verbose` DEBUG (HTTP detail, cache keys), `--quiet` WARNING (errors only). All log output to stderr; stdout reserved for data
|
||||||
|
- UTF-8 stdout enforcement (`force_utf8_stdout()`) — Windows Python defaults to GBK and crashes on non-ASCII chars; both scripts force UTF-8 + `errors='replace'` at startup so `print('\xa0')` never raises
|
||||||
|
- Windows config discovery — `resolve_instances` / `load_config` also check `%APPDATA%/searxng-cli/` (Windows per-user app convention) in addition to `~/.config/searxng-cli/` (POSIX convention)
|
||||||
|
- fetch.py failure diagnostics — error output includes `status_code=`, `cause=`, `url=` fields so AI agents can programmatically distinguish 404 vs 403 vs DNS failure without parsing English prose
|
||||||
- Engine/category whitespace normalization (`"google, bing"` → `"google,bing"`)
|
- Engine/category whitespace normalization (`"google, bing"` → `"google,bing"`)
|
||||||
- `--time-range none` option to disable time filtering
|
- `--time-range none` option to disable time filtering
|
||||||
|
|
||||||
@@ -98,7 +115,8 @@ python scripts/fetch.py -u "https://example.com" --extract text
|
|||||||
export SEARXNG_INSTANCE="https://my-searxng.example.com,https://backup.example.com"
|
export SEARXNG_INSTANCE="https://my-searxng.example.com,https://backup.example.com"
|
||||||
python scripts/search.py -q "python asyncio tutorial" # -i not needed
|
python scripts/search.py -q "python asyncio tutorial" # -i not needed
|
||||||
|
|
||||||
# 6. Or use a config file (./searxng.toml or ~/.config/searxng-cli/searxng.toml)
|
# 6. Or use a config file (./searxng.toml or ~/.config/searxng-cli/searxng.toml
|
||||||
|
# or %APPDATA%/searxng-cli/searxng.toml on Windows)
|
||||||
# [searxng]
|
# [searxng]
|
||||||
# instance = "https://my-searxng.example.com"
|
# instance = "https://my-searxng.example.com"
|
||||||
# # or: instances = ["https://a.example.com", "https://b.example.com"]
|
# # or: instances = ["https://a.example.com", "https://b.example.com"]
|
||||||
@@ -403,7 +421,7 @@ usage: search.py [-h] [--query QUERY] [--instance URL]
|
|||||||
- `--max-results N` — limit number of results (applied AFTER dedup+sort, so the highest-scoring/newest items are kept)
|
- `--max-results N` — limit number of results (applied AFTER dedup+sort, so the highest-scoring/newest items are kept)
|
||||||
- `--sort-by {score,date,engine,none}` — sort results (default: `score` descending; `none` preserves instance order). Applied after dedup, before `--max-results`. HTML-fallback results have no score and keep their order
|
- `--sort-by {score,date,engine,none}` — sort results (default: `score` descending; `none` preserves instance order). Applied after dedup, before `--max-results`. HTML-fallback results have no score and keep their order
|
||||||
- `--no-dedup` — disable cross-engine deduplication (by default, duplicate URLs — same page ignoring tracking params/fragment — are collapsed, keeping the first occurrence's engine/score)
|
- `--no-dedup` — disable cross-engine deduplication (by default, duplicate URLs — same page ignoring tracking params/fragment — are collapsed, keeping the first occurrence's engine/score)
|
||||||
- `--config FILE` — path to a `searxng.toml` config file; overrides the default auto-discovery (`./searxng.toml` → `~/.config/searxng-cli/searxng.toml`). Must be the first flag so its values can set defaults for other flags
|
- `--config FILE` — path to a `searxng.toml` config file; overrides the default auto-discovery (`./searxng.toml` → `~/.config/searxng-cli/searxng.toml` → `%APPDATA%/searxng-cli/searxng.toml` on Windows). Must be the first flag so its values can set defaults for other flags
|
||||||
- `--verbose` / `-v` — show debug-level diagnostics on stderr (HTTP request URLs, response codes, cache keys, retry detail)
|
- `--verbose` / `-v` — show debug-level diagnostics on stderr (HTTP request URLs, response codes, cache keys, retry detail)
|
||||||
- `--quiet` — suppress progress messages and retry notices on stderr; only warnings and errors are shown (no short flag: `-q` is `--query`)
|
- `--quiet` — suppress progress messages and retry notices on stderr; only warnings and errors are shown (no short flag: `-q` is `--query`)
|
||||||
- `--include-domain a.com,b.org` — allowlist; only results from these domains are kept (applied after search)
|
- `--include-domain a.com,b.org` — allowlist; only results from these domains are kept (applied after search)
|
||||||
|
|||||||
+1
-1
@@ -7,6 +7,6 @@ Retry settings and shared HTTP utilities now live in ``common.py`` so that
|
|||||||
both ``search.py`` and ``fetch.py`` share one consistent implementation.
|
both ``search.py`` and ``fetch.py`` share one consistent implementation.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
VERSION = "1.8.0"
|
VERSION = "1.8.1"
|
||||||
SCHEMA_VERSION = "1.0"
|
SCHEMA_VERSION = "1.0"
|
||||||
USER_AGENT = f"searxng-cli/{VERSION}"
|
USER_AGENT = f"searxng-cli/{VERSION}"
|
||||||
|
|||||||
@@ -11,12 +11,14 @@ This module centralizes code that was previously duplicated across
|
|||||||
* ``FALLBACK_UAS`` — browser-like User-Agents used when blocked
|
* ``FALLBACK_UAS`` — browser-like User-Agents used when blocked
|
||||||
* retry constants — ``RETRYABLE_STATUS``, ``RETRY_BACKOFF_BASE``, etc.
|
* retry constants — ``RETRYABLE_STATUS``, ``RETRY_BACKOFF_BASE``, etc.
|
||||||
* ``setup_logging`` — shared logging configuration (--verbose/--quiet)
|
* ``setup_logging`` — shared logging configuration (--verbose/--quiet)
|
||||||
|
* ``force_utf8_stdout`` — force stdout to UTF-8 (fix Windows GBK crashes)
|
||||||
|
|
||||||
Centralizing the retry policy guarantees that both scripts treat 429/5xx
|
Centralizing the retry policy guarantees that both scripts treat 429/5xx
|
||||||
as retryable and connection errors as transient, eliminating the previous
|
as retryable and connection errors as transient, eliminating the previous
|
||||||
inconsistency where ``search.py`` ignored 5xx.
|
inconsistency where ``search.py`` ignored 5xx.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
import io
|
||||||
import logging
|
import logging
|
||||||
import sys
|
import sys
|
||||||
import urllib.error
|
import urllib.error
|
||||||
@@ -56,6 +58,58 @@ def setup_logging(verbose: bool = False, quiet: bool = False) -> None:
|
|||||||
# Don't let root logger add its own handler — we own the searxng namespace.
|
# Don't let root logger add its own handler — we own the searxng namespace.
|
||||||
_LOG.propagate = False
|
_LOG.propagate = False
|
||||||
|
|
||||||
|
|
||||||
|
def force_utf8_stdout() -> None:
|
||||||
|
"""Force stdout/stderr to UTF-8 to prevent Windows GBK encoding crashes.
|
||||||
|
|
||||||
|
Windows Python defaults ``sys.stdout`` to the OEM codepage (often GBK on
|
||||||
|
Chinese Windows). ``print()`` of any character outside that codepage
|
||||||
|
(e.g. ``\\xa0`` nbsp, CJK punctuation from foreign pages) raises
|
||||||
|
``UnicodeEncodeError`` and kills the process.
|
||||||
|
|
||||||
|
``PYTHONIOENCODING=utf-8`` is unreliable here because Python 3.7+
|
||||||
|
reconfigures stdout after reading that env var in some scenarios (e.g.
|
||||||
|
when stdout has already been wrapped). The only reliable fix is to
|
||||||
|
reconfigure the stream in-process.
|
||||||
|
|
||||||
|
Uses ``sys.stdout.reconfigure()`` on Python 3.7+, falling back to
|
||||||
|
wrapping ``sys.stdout.buffer`` on older versions. Both paths use
|
||||||
|
``errors='replace'`` so an undecodable byte never crashes the script —
|
||||||
|
better to emit ``?`` than to lose all output.
|
||||||
|
|
||||||
|
Safe to call multiple times; subsequent calls are no-ops once the
|
||||||
|
encoding is already UTF-8 (or close enough — we check the lowercased
|
||||||
|
encoding name to tolerate ``utf-8`` vs ``UTF-8`` vs ``utf8``).
|
||||||
|
"""
|
||||||
|
for stream_name in ("stdout", "stderr"):
|
||||||
|
stream = getattr(sys, stream_name, None)
|
||||||
|
if stream is None:
|
||||||
|
continue
|
||||||
|
# Already UTF-8? Skip (covers Linux/macOS and re-invoked scripts).
|
||||||
|
enc = getattr(stream, "encoding", "") or ""
|
||||||
|
if enc.lower().replace("-", "") in ("utf8", "utf-8-sig"):
|
||||||
|
continue
|
||||||
|
# Python 3.7+ has TextIOWrapper.reconfigure()
|
||||||
|
reconfigure = getattr(stream, "reconfigure", None)
|
||||||
|
if reconfigure is not None:
|
||||||
|
try:
|
||||||
|
reconfigure(encoding="utf-8", errors="replace")
|
||||||
|
continue
|
||||||
|
except (ValueError, OSError):
|
||||||
|
pass # Fall through to the buffer-wrap path
|
||||||
|
# Fallback: wrap the underlying buffer in a new UTF-8 stream.
|
||||||
|
buffer = getattr(stream, "buffer", None)
|
||||||
|
if buffer is not None:
|
||||||
|
try:
|
||||||
|
new_stream = io.TextIOWrapper(
|
||||||
|
buffer, encoding="utf-8", errors="replace", line_buffering=True,
|
||||||
|
)
|
||||||
|
setattr(sys, stream_name, new_stream)
|
||||||
|
except (ValueError, AttributeError):
|
||||||
|
# Last resort: keep the original stream. Better to risk a
|
||||||
|
# GBK crash on exotic characters than to break stdout entirely.
|
||||||
|
pass
|
||||||
|
|
||||||
# Retry settings (shared by both scripts)
|
# Retry settings (shared by both scripts)
|
||||||
MAX_RETRIES = 3
|
MAX_RETRIES = 3
|
||||||
RETRY_BACKOFF_BASE = 1.5 # seconds; exponential backoff + jitter
|
RETRY_BACKOFF_BASE = 1.5 # seconds; exponential backoff + jitter
|
||||||
|
|||||||
+25
-1
@@ -30,6 +30,7 @@ from common import (
|
|||||||
apply_proxy,
|
apply_proxy,
|
||||||
build_auth_headers,
|
build_auth_headers,
|
||||||
detect_charset,
|
detect_charset,
|
||||||
|
force_utf8_stdout,
|
||||||
is_retryable_error,
|
is_retryable_error,
|
||||||
resolve_auth_basic,
|
resolve_auth_basic,
|
||||||
resolve_auth_bearer,
|
resolve_auth_bearer,
|
||||||
@@ -676,6 +677,7 @@ Examples:
|
|||||||
|
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
setup_logging(verbose=args.verbose, quiet=args.quiet)
|
setup_logging(verbose=args.verbose, quiet=args.quiet)
|
||||||
|
force_utf8_stdout() # Windows: prevent GBK crash on non-ASCII chars
|
||||||
|
|
||||||
if not args.url.startswith(("http://", "https://")):
|
if not args.url.startswith(("http://", "https://")):
|
||||||
logger.error("Error: URL must start with http:// or https://")
|
logger.error("Error: URL must start with http:// or https://")
|
||||||
@@ -711,7 +713,29 @@ Examples:
|
|||||||
result.content, result.content_type, result.final_url,
|
result.content, result.content_type, result.final_url,
|
||||||
)
|
)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Error: {e}")
|
# 诊断信息增强:从 __cause__ 链中提取 HTTP 状态码、原始异常类型,
|
||||||
|
# 让 AI Agent 能程序化判断失败原因(404 vs 403 vs DNS 失败等),
|
||||||
|
# 而不是只看到一句 "HTTP 404 for ..."。
|
||||||
|
cause = e.__cause__
|
||||||
|
status_code = None
|
||||||
|
cause_type = type(cause).__name__ if cause else type(e).__name__
|
||||||
|
# urllib HTTPError 有 .code 属性;requests HTTPError 有 .response.status_code
|
||||||
|
if cause is not None:
|
||||||
|
status_code = (getattr(cause, "code", None) or
|
||||||
|
getattr(getattr(cause, "response", None), "status_code", None))
|
||||||
|
if status_code is None:
|
||||||
|
# 最后一道兜底:从异常消息里提取 "HTTP NNN" 模式
|
||||||
|
m = re.search(r'HTTP (\d{3})', str(e))
|
||||||
|
if m:
|
||||||
|
status_code = int(m.group(1))
|
||||||
|
diag_parts = [f"Error: {e}"]
|
||||||
|
if status_code is not None:
|
||||||
|
diag_parts.append(f"status_code={status_code}")
|
||||||
|
diag_parts.append(f"cause={cause_type}")
|
||||||
|
diag_parts.append(f"url={args.url}")
|
||||||
|
if args.no_redirect:
|
||||||
|
diag_parts.append("redirects=disabled")
|
||||||
|
logger.error(" | ".join(diag_parts))
|
||||||
sys.exit(1)
|
sys.exit(1)
|
||||||
|
|
||||||
if final_url != args.url:
|
if final_url != args.url:
|
||||||
|
|||||||
+35
-2
@@ -34,6 +34,7 @@ from common import (
|
|||||||
build_auth_headers,
|
build_auth_headers,
|
||||||
classify_error,
|
classify_error,
|
||||||
emit_progress,
|
emit_progress,
|
||||||
|
force_utf8_stdout,
|
||||||
resolve_auth_basic,
|
resolve_auth_basic,
|
||||||
resolve_auth_bearer,
|
resolve_auth_bearer,
|
||||||
set_progress_enabled,
|
set_progress_enabled,
|
||||||
@@ -329,13 +330,39 @@ def _read_instance_file(path: Path) -> list:
|
|||||||
return []
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
def _windows_appdata_config_dir() -> Path:
|
||||||
|
"""Return the Windows APPDATA config directory, or a sentinel Path if unset.
|
||||||
|
|
||||||
|
On Windows, the conventional per-user app config directory is
|
||||||
|
``%APPDATA%`` (typically ``C:\\Users\\<user>\\AppData\\Roaming``).
|
||||||
|
On POSIX, this env var is unset and we return a sentinel
|
||||||
|
``Path("/__no_appdata__")`` which never exists on disk, so the caller
|
||||||
|
can unconditionally append it to the candidate list without polluting
|
||||||
|
Linux/macOS lookups.
|
||||||
|
|
||||||
|
Note: ``Path("")`` resolves to ``.`` (current directory) on Windows,
|
||||||
|
which DOES exist — so we must use an absolute sentinel path instead.
|
||||||
|
"""
|
||||||
|
appdata = os.environ.get("APPDATA", "")
|
||||||
|
if appdata:
|
||||||
|
return Path(appdata) / "searxng-cli"
|
||||||
|
# Sentinel: absolute path that never exists. Using "/" + unlikely name
|
||||||
|
# keeps it false on both POSIX and Windows (where "/" is the drive root).
|
||||||
|
return Path("/__no_appdata__")
|
||||||
|
|
||||||
|
|
||||||
def resolve_instances(cli_arg: str = None) -> list:
|
def resolve_instances(cli_arg: str = None) -> list:
|
||||||
"""Resolve instance URLs from (in priority order):
|
"""Resolve instance URLs from (in priority order):
|
||||||
|
|
||||||
1. ``--instance`` CLI flag (comma-separated list)
|
1. ``--instance`` CLI flag (comma-separated list)
|
||||||
2. ``SEARXNG_INSTANCE`` environment variable (comma-separated list)
|
2. ``SEARXNG_INSTANCE`` environment variable (comma-separated list)
|
||||||
3. config file: ``./searxng.toml`` → ``~/.config/searxng-cli/searxng.toml``
|
3. config file search order:
|
||||||
→ ``./instances.txt`` → ``~/.config/searxng-cli/instances.txt``
|
a. ``./searxng.toml``
|
||||||
|
b. ``~/.config/searxng-cli/searxng.toml``
|
||||||
|
c. ``%APPDATA%/searxng-cli/searxng.toml`` (Windows only)
|
||||||
|
d. ``./instances.txt``
|
||||||
|
e. ``~/.config/searxng-cli/instances.txt``
|
||||||
|
f. ``%APPDATA%/searxng-cli/instances.txt`` (Windows only)
|
||||||
|
|
||||||
Returns an empty list if no instance can be resolved.
|
Returns an empty list if no instance can be resolved.
|
||||||
"""
|
"""
|
||||||
@@ -346,11 +373,14 @@ def resolve_instances(cli_arg: str = None) -> list:
|
|||||||
if env:
|
if env:
|
||||||
return parse_instances(env)
|
return parse_instances(env)
|
||||||
|
|
||||||
|
win_dir = _windows_appdata_config_dir()
|
||||||
candidates = [
|
candidates = [
|
||||||
Path.cwd() / "searxng.toml",
|
Path.cwd() / "searxng.toml",
|
||||||
Path.home() / ".config" / "searxng-cli" / "searxng.toml",
|
Path.home() / ".config" / "searxng-cli" / "searxng.toml",
|
||||||
|
win_dir / "searxng.toml",
|
||||||
Path.cwd() / "instances.txt",
|
Path.cwd() / "instances.txt",
|
||||||
Path.home() / ".config" / "searxng-cli" / "instances.txt",
|
Path.home() / ".config" / "searxng-cli" / "instances.txt",
|
||||||
|
win_dir / "instances.txt",
|
||||||
]
|
]
|
||||||
for p in candidates:
|
for p in candidates:
|
||||||
if p.exists():
|
if p.exists():
|
||||||
@@ -395,9 +425,11 @@ def load_config(config_path: str = None) -> dict:
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.warning(f"Warning: cannot read config '{p}': {e}")
|
logger.warning(f"Warning: cannot read config '{p}': {e}")
|
||||||
return {}
|
return {}
|
||||||
|
win_dir = _windows_appdata_config_dir()
|
||||||
candidates = [
|
candidates = [
|
||||||
Path.cwd() / "searxng.toml",
|
Path.cwd() / "searxng.toml",
|
||||||
Path.home() / ".config" / "searxng-cli" / "searxng.toml",
|
Path.home() / ".config" / "searxng-cli" / "searxng.toml",
|
||||||
|
win_dir / "searxng.toml",
|
||||||
]
|
]
|
||||||
for p in candidates:
|
for p in candidates:
|
||||||
if p.exists():
|
if p.exists():
|
||||||
@@ -1379,6 +1411,7 @@ def main():
|
|||||||
pre_args, _ = pre.parse_known_args()
|
pre_args, _ = pre.parse_known_args()
|
||||||
|
|
||||||
setup_logging(verbose=pre_args.verbose, quiet=pre_args.quiet)
|
setup_logging(verbose=pre_args.verbose, quiet=pre_args.quiet)
|
||||||
|
force_utf8_stdout() # Windows: prevent GBK crash on non-ASCII chars
|
||||||
|
|
||||||
# Load config defaults from --config file, else ./searxng.toml or
|
# Load config defaults from --config file, else ./searxng.toml or
|
||||||
# ~/.config/searxng-cli/searxng.toml. Every CLI flag below can be
|
# ~/.config/searxng-cli/searxng.toml. Every CLI flag below can be
|
||||||
|
|||||||
@@ -256,7 +256,7 @@ def test_cli_version_prints_version():
|
|||||||
"""`--version` exits 0 and prints the version string."""
|
"""`--version` exits 0 and prints the version string."""
|
||||||
r = _run_cli("--version")
|
r = _run_cli("--version")
|
||||||
assert r.returncode == 0
|
assert r.returncode == 0
|
||||||
assert "1.8.0" in r.stdout
|
assert "1.8.1" in r.stdout
|
||||||
assert "searxng-cli" in r.stdout
|
assert "searxng-cli" in r.stdout
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,181 @@
|
|||||||
|
"""Tests for v1.8.1 Windows compatibility fixes.
|
||||||
|
|
||||||
|
Covers:
|
||||||
|
* ``force_utf8_stdout()`` — reconfigures stdout/stderr to UTF-8, idempotent,
|
||||||
|
tolerates non-UTF-8 encodings, never raises on any input.
|
||||||
|
* ``_windows_appdata_config_dir()`` — returns APPDATA-based path on Windows,
|
||||||
|
empty Path on POSIX.
|
||||||
|
* ``resolve_instances()`` — discovers config under ``%APPDATA%/searxng-cli/``
|
||||||
|
when set (simulating Windows), still works when APPDATA is unset (POSIX).
|
||||||
|
* ``fetch.py`` failure diagnostics — error output includes ``status_code=``,
|
||||||
|
``cause=``, ``url=`` fields for programmatic diagnosis.
|
||||||
|
"""
|
||||||
|
import io
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
# Ensure scripts/ is on sys.path
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "scripts"))
|
||||||
|
|
||||||
|
from common import force_utf8_stdout
|
||||||
|
|
||||||
|
|
||||||
|
# ----- force_utf8_stdout -----
|
||||||
|
|
||||||
|
def test_force_utf8_stdout_is_idempotent():
|
||||||
|
"""Calling twice should not raise and should leave stdout usable."""
|
||||||
|
force_utf8_stdout()
|
||||||
|
force_utf8_stdout() # second call: encoding already utf-8 or close — no-op
|
||||||
|
# stdout must still be writable
|
||||||
|
print("test", end="")
|
||||||
|
sys.stdout.flush()
|
||||||
|
|
||||||
|
|
||||||
|
def test_force_utf8_stdout_replaces_non_utf8_stream():
|
||||||
|
"""A non-UTF-8 stdout should be reconfigured to UTF-8."""
|
||||||
|
# Simulate a GBK stream by creating a fresh TextIOWrapper with GBK
|
||||||
|
original = sys.stdout
|
||||||
|
try:
|
||||||
|
buf = io.BytesIO()
|
||||||
|
# Use errors='replace' so writing non-GBK chars doesn't crash the test
|
||||||
|
sys.stdout = io.TextIOWrapper(buf, encoding="gbk", errors="replace")
|
||||||
|
force_utf8_stdout()
|
||||||
|
# After force_utf8_stdout, encoding should now be utf-8 (or utf-8-sig)
|
||||||
|
enc = (sys.stdout.encoding or "").lower().replace("-", "")
|
||||||
|
assert enc in ("utf8", "utf8sig"), f"expected utf-8, got {enc}"
|
||||||
|
finally:
|
||||||
|
sys.stdout = original
|
||||||
|
|
||||||
|
|
||||||
|
def test_force_utf8_stdout_handles_missing_buffer():
|
||||||
|
"""A stream without .buffer should not crash the function."""
|
||||||
|
class FakeStream:
|
||||||
|
encoding = "gbk"
|
||||||
|
# No .buffer, no .reconfigure
|
||||||
|
|
||||||
|
original = sys.stdout
|
||||||
|
try:
|
||||||
|
sys.stdout = FakeStream()
|
||||||
|
# Should not raise
|
||||||
|
force_utf8_stdout()
|
||||||
|
finally:
|
||||||
|
sys.stdout = original
|
||||||
|
|
||||||
|
|
||||||
|
def test_force_utf8_stdout_skips_already_utf8():
|
||||||
|
"""An already-UTF-8 stdout should be untouched."""
|
||||||
|
original = sys.stdout
|
||||||
|
try:
|
||||||
|
buf = io.BytesIO()
|
||||||
|
sys.stdout = io.TextIOWrapper(buf, encoding="utf-8")
|
||||||
|
force_utf8_stdout()
|
||||||
|
# Still utf-8
|
||||||
|
assert (sys.stdout.encoding or "").lower().replace("-", "") == "utf8"
|
||||||
|
finally:
|
||||||
|
sys.stdout = original
|
||||||
|
|
||||||
|
|
||||||
|
def test_force_utf8_stdout_allows_non_ascii_print():
|
||||||
|
"""End-to-end: after force_utf8_stdout, printing non-ASCII chars works.
|
||||||
|
|
||||||
|
Regression test for the Windows GBK crash where ``print('\\xa0')`` raised
|
||||||
|
``UnicodeEncodeError``. With UTF-8 enforced, it must succeed silently.
|
||||||
|
"""
|
||||||
|
force_utf8_stdout()
|
||||||
|
# This must not raise. On a broken GBK stdout it would.
|
||||||
|
print("\xa0中文测试\u2014em-dash", end="")
|
||||||
|
sys.stdout.flush()
|
||||||
|
|
||||||
|
|
||||||
|
# ----- _windows_appdata_config_dir -----
|
||||||
|
|
||||||
|
def test_windows_appdata_dir_returns_path_when_set(monkeypatch):
|
||||||
|
"""When APPDATA is set, returns APPDATA/searxng-cli."""
|
||||||
|
import search
|
||||||
|
monkeypatch.setenv("APPDATA", "/fake/appdata")
|
||||||
|
result = search._windows_appdata_config_dir()
|
||||||
|
assert result == Path("/fake/appdata") / "searxng-cli"
|
||||||
|
|
||||||
|
|
||||||
|
def test_windows_appdata_dir_returns_empty_when_unset(monkeypatch):
|
||||||
|
"""When APPDATA is unset (POSIX), returns a sentinel Path that never exists."""
|
||||||
|
import search
|
||||||
|
monkeypatch.delenv("APPDATA", raising=False)
|
||||||
|
result = search._windows_appdata_config_dir()
|
||||||
|
# Sentinel path must never exist on disk — safe to append to candidate lists
|
||||||
|
assert not result.exists()
|
||||||
|
|
||||||
|
|
||||||
|
# ----- resolve_instances with APPDATA -----
|
||||||
|
|
||||||
|
def test_resolve_instances_finds_appdata_config(monkeypatch, tmp_path):
|
||||||
|
"""resolve_instances discovers searxng.toml under APPDATA (Windows sim)."""
|
||||||
|
import search
|
||||||
|
|
||||||
|
# Simulate Windows: set APPDATA to a temp dir, clear other sources
|
||||||
|
appdata = tmp_path / "appdata"
|
||||||
|
appdata.mkdir()
|
||||||
|
cfg_dir = appdata / "searxng-cli"
|
||||||
|
cfg_dir.mkdir()
|
||||||
|
(cfg_dir / "searxng.toml").write_text(
|
||||||
|
'[searxng]\ninstance = "https://from-appdata.example.com"\n',
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
monkeypatch.setenv("APPDATA", str(appdata))
|
||||||
|
monkeypatch.delenv("SEARXNG_INSTANCE", raising=False)
|
||||||
|
|
||||||
|
# CWD and HOME must not have a config (so we isolate the test)
|
||||||
|
monkeypatch.chdir(tmp_path)
|
||||||
|
fake_home = tmp_path / "fake-home"
|
||||||
|
fake_home.mkdir()
|
||||||
|
monkeypatch.setenv("HOME", str(fake_home))
|
||||||
|
# On Windows, Path.home() uses USERPROFILE, not HOME
|
||||||
|
monkeypatch.setenv("USERPROFILE", str(fake_home))
|
||||||
|
|
||||||
|
result = search.resolve_instances()
|
||||||
|
assert result == ["https://from-appdata.example.com"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_resolve_instances_appdata_txt_fallback(monkeypatch, tmp_path):
|
||||||
|
"""instances.txt under APPDATA is also discovered."""
|
||||||
|
import search
|
||||||
|
|
||||||
|
appdata = tmp_path / "appdata"
|
||||||
|
appdata.mkdir()
|
||||||
|
cfg_dir = appdata / "searxng-cli"
|
||||||
|
cfg_dir.mkdir()
|
||||||
|
(cfg_dir / "instances.txt").write_text(
|
||||||
|
"https://from-txt.example.com\n", encoding="utf-8"
|
||||||
|
)
|
||||||
|
monkeypatch.setenv("APPDATA", str(appdata))
|
||||||
|
monkeypatch.delenv("SEARXNG_INSTANCE", raising=False)
|
||||||
|
monkeypatch.chdir(tmp_path)
|
||||||
|
fake_home = tmp_path / "fake-home"
|
||||||
|
fake_home.mkdir()
|
||||||
|
monkeypatch.setenv("HOME", str(fake_home))
|
||||||
|
monkeypatch.setenv("USERPROFILE", str(fake_home))
|
||||||
|
|
||||||
|
result = search.resolve_instances()
|
||||||
|
assert result == ["https://from-txt.example.com"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_resolve_instances_appdata_empty(monkeypatch, tmp_path):
|
||||||
|
"""When APPDATA set but no config there, falls through cleanly."""
|
||||||
|
import search
|
||||||
|
|
||||||
|
appdata = tmp_path / "appdata"
|
||||||
|
appdata.mkdir()
|
||||||
|
monkeypatch.setenv("APPDATA", str(appdata))
|
||||||
|
monkeypatch.delenv("SEARXNG_INSTANCE", raising=False)
|
||||||
|
monkeypatch.chdir(tmp_path)
|
||||||
|
fake_home = tmp_path / "fake-home"
|
||||||
|
fake_home.mkdir()
|
||||||
|
monkeypatch.setenv("HOME", str(fake_home))
|
||||||
|
monkeypatch.setenv("USERPROFILE", str(fake_home))
|
||||||
|
|
||||||
|
result = search.resolve_instances()
|
||||||
|
assert result == []
|
||||||
Reference in New Issue
Block a user