fix(v2.1.1): 修复执行问题记录中的真实 bug + 文档对齐
源码修复(5 项): 1. search.py --time-range choices 加入 week(对齐 SearXNG API 四档) 2. fetch.py stdlib 路径处理 gzip/deflate 解压(被沙箱伪响应掩盖的真实 bug, 无 requests 环境抓取压缩服务器会全页 U+FFFD 乱码) 3. search.py --research 模式实现跨角度合并去重,输出 merged_results 字段 (兑现文档承诺 "Results are merged and deduplicated") 4. search.py fetch_page 返回 error_code 字段 + AdaptiveThrottle 用 E_RATE_LIMIT 结构化检测 429(原字符串匹配 "429" 会漏判 "Too Many Requests") 5. search.py _retry_with_backoff 复用 compute_backoff_delay(60s 封顶) + 处理 Retry-After header,与 fetch.py 保持一致 增强(3 项): - common.py 精确化 baidu 子域列表(pan.baidu.com/cloud.baidu.com 不再误伤) - search.py expand_research_queries 根据主题语言切换中英文后缀 - search.py 新增 _warn_unresponsive_engines,识别实例侧引擎挂起并提示 文档/版本: - _config.py VERSION 2.1.0 → 2.1.1 - SKILL.md 同步更新(time-range week、merged_results、error_code、baidu 精确化) - README.md 同步更新 + 测试数量 503 → 539 测试: 539 个全部通过,含 6 个新增验证测试
This commit is contained in:
@@ -9,6 +9,7 @@ for improved extraction quality (optional, falls back to stdlib).
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import gzip
|
||||
import logging
|
||||
import random
|
||||
import re
|
||||
@@ -16,6 +17,7 @@ import sys
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import zlib
|
||||
from collections import namedtuple
|
||||
from html.parser import HTMLParser
|
||||
from pathlib import Path
|
||||
@@ -737,6 +739,28 @@ def fetch_url(url: str, timeout=15, user_agent: str = None,
|
||||
content_type = resp.headers.get("Content-Type", "")
|
||||
final_url = resp.geturl()
|
||||
|
||||
# v2.1.1 修复:stdlib urllib 不自动解压 gzip/deflate
|
||||
# 服务器返回压缩字节流时 raw.decode() 会失败 →
|
||||
# errors="replace" → 全页 U+FFFD 乱码。
|
||||
# requests 库会自动处理 Content-Encoding,但 stdlib 不会。
|
||||
# 此前该 bug 被沙箱伪响应掩盖(两者都产生 U+FFFD),
|
||||
# 实际在无 requests 的真实环境中会复现。
|
||||
content_encoding = (resp.headers.get("Content-Encoding", "")
|
||||
.lower().strip())
|
||||
if content_encoding and raw:
|
||||
try:
|
||||
if "gzip" in content_encoding:
|
||||
raw = gzip.decompress(raw)
|
||||
elif "deflate" in content_encoding:
|
||||
# deflate 可能是 zlib 包装或裸 deflate
|
||||
try:
|
||||
raw = zlib.decompress(raw)
|
||||
except zlib.error:
|
||||
raw = zlib.decompress(raw, -zlib.MAX_WBITS)
|
||||
except (OSError, zlib.error) as e:
|
||||
logger.debug(f" decompress failed ({content_encoding}): {e}")
|
||||
# 解压失败保留原 raw,让下游 decode 兜底
|
||||
|
||||
if encoding:
|
||||
charset = encoding
|
||||
else:
|
||||
|
||||
Reference in New Issue
Block a user