feat(v2.5.0): Sec-Ch-Ua 头修复 + 结构化提取 + token 预算 + 正文去重 + 连接复用
正确性修复: - 修复 Sec-Ch-Ua 构造 bug: 原实现产出 ""Not_A Brand";v="99"" 双重引号 畸形头(Chrome/Edge 两路径), 严格校验的 WAF 会忽略; 改为品牌数组拼接 - search 403 快速失败: 实例级 403 不再退避重试(~10.5s 空等), 立即 failover - AdaptiveThrottle: --throttle-failure-threshold 0 现为真正禁用语义 - number_of_results 缺失时用 len(results) 兜底(JSON/HTML 路径契约对齐) - 版本对齐: pyproject.toml 与 _config.py 同步 2.5.0 AI 代理体验: - fetch.py --extract json: 结构化骨架(title/meta/headings/links/images) - fetch.py --max-chars N: 提取后语义级截断(区别于 --max-size 字节截断) - search --fetch-total-chars N: --fetch 全局字符预算, 耗尽后 status=skipped - --dedup-fetched-content: 抓取正文 SimHash 去重, status=duplicate - --progress 新增 angle_start/ok/fail + fetch_skip/fetch_duplicate 事件 - fetch.py 补齐 --log-format json + --dump-schema - CSV 媒体列自适应(images/videos 类别自动追加媒体字段列) - --dry-run 批量模式打印实际查询列表 - search 连接复用: requests 可用时走模块级 Session(连接池) 工程治理: - 新增 scripts/release_check.py 发布一致性检查(版本/错误码表漂移) - 新增 tests/test_v250_features.py 46+4 个回归测试(全量 622 通过) - tests/conftest.py: autouse fixture 强制 stdlib 路径(本机有 requests 时 既有 urllib mock 测试不失效)
This commit is contained in:
@@ -1,7 +1,7 @@
|
||||
---
|
||||
name: searxng-use-cli
|
||||
description: Use when you need to search the web via your OWN SearXNG instance (no public-instance discovery). 3 CLI scripts + a shared common.py module — execute privacy-respecting searches against a user-supplied instance (with multi-instance failover, 5xx/429 retry, auto-fetch) or via SEARXNG_INSTANCE env / config file, fetch/extract readable text or markdown from web pages. Zero-config replacement for proprietary search APIs.
|
||||
version: 2.4.0
|
||||
version: 2.5.0
|
||||
author: Metona Team
|
||||
license: MIT
|
||||
platforms: [linux, macos, windows]
|
||||
@@ -37,6 +37,8 @@ SearXNG is a privacy-respecting metasearch engine that aggregates results from 7
|
||||
|
||||
**Key capabilities:**
|
||||
|
||||
> **v2.5.0 highlights:** structured page skeleton output (`fetch.py --extract json` — title/meta/headings/links/images, no full body needed); token budget control (`search.py --fetch-total-chars N`, `fetch.py --max-chars N`); fetched-content dedup (`--dedup-fetched-content`, mirrors/republished pages marked `status="duplicate"`); fixed `Sec-Ch-Ua` header construction (was malformed in Chrome/Edge paths); research-angle progress events (`angle_start/ok/fail`); `fetch.py` now has `--log-format json` + `--dump-schema`; CSV auto-appends media columns for images/videos categories; search reuses a requests Session when available (connection pooling). Full changelog in README.md.
|
||||
|
||||
**Search & results**
|
||||
- Multi-instance failover with parallel probing (faster failover, deterministic output order)
|
||||
- Exponential-backoff retry on transient errors (403/429/5xx/connection) via shared retry-policy components in `common.py` (backoff/retryable-status/Retry-After), with the retry loop in each script
|
||||
@@ -544,11 +546,13 @@ usage: fetch.py [-h] --url URL [--extract {text,html,markdown}]
|
||||
7. **PDF/document parsing** (regardless of `--extract` mode): PDF via `pdftotext` subprocess, `.docx`/`.xlsx` via stdlib `zipfile`. Unsupported binary types return `E_UNSUPPORTED_MEDIA`
|
||||
|
||||
**Key options:**
|
||||
- `--url https://...` — required
|
||||
- `--url https://...` — required (unless `--dump-schema`)
|
||||
- `--format {text,json}` — output format (default: `text`; `json` = structured contract, v2.3.0). Success: `{status: ok, url, final_url, content_type, extract, truncated, text_length, user_agent}`; failure: `{status: error, url, error, error_code, status_code}`. stdout is pure JSON; logs stay on stderr; exit 0 on success / 1 on failure
|
||||
- `--extract text` — clean readable text (default); PDF/docx/xlsx parsed automatically regardless of extract mode (see "What it does" #7)
|
||||
- `--extract html` — raw HTML
|
||||
- `--extract markdown` — Markdown conversion (tree-based; handles nested tags, GFM tables, fenced code blocks, blockquotes, inline code, ordered/unordered/nested lists, definition lists, images, emphasis)
|
||||
- `--extract json` — v2.5.0 structured skeleton + text. Success payload adds `{title, meta_description, headings[], links[], images[]}` so agents can judge a page without reading the full body. Implies `--format json`
|
||||
- `--max-chars N` — v2.5.0 truncate extracted text to N chars (semantic cut after extraction; sets `truncated=true`). Distinct from `--max-size` which caps raw response bytes
|
||||
- `--encoding gbk` — force charset for non-UTF-8 pages (else auto-detected from HTTP header / HTML meta / UTF-8 fallback)
|
||||
- `--timeout 15` — request timeout in seconds (split into connect/read tuple internally)
|
||||
- `--retries 3` — max retries on transient errors (429/5xx/connection); falls back through the 15-UA pool when blocked
|
||||
|
||||
Reference in New Issue
Block a user