Compare commits
43
Commits
ccfc39656b
...
master
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
50b1864145 | ||
|
|
c1c3036abd | ||
|
|
0b44620721 | ||
|
|
c3757f486c | ||
|
|
18064594f5 | ||
|
|
81c46eb6f2 | ||
|
|
c5694b1d23 | ||
|
|
714e7f98a4 | ||
|
|
34efde701c | ||
|
|
61a165be7c | ||
|
|
799560ea05 | ||
|
|
d14663ef80 | ||
|
|
2c945ee05a | ||
|
|
3983aae426 | ||
|
|
bf93ec5251 | ||
|
|
2328540779 | ||
|
|
97b9fa486d | ||
|
|
85f0f170a4 | ||
|
|
0c5c0b1d20 | ||
|
|
567d150257 | ||
|
|
6a0cfebc4a | ||
|
|
a753c7ff55 | ||
|
|
68e4731788 | ||
|
|
edd9f1dcd9 | ||
|
|
b20d47bd93 | ||
|
|
841db2e049 | ||
|
|
2a109ef933 | ||
|
|
67e72d897f | ||
|
|
4ab04df882 | ||
|
|
674da6b7b7 | ||
|
|
c0d85eeaab | ||
|
|
c6515440fd | ||
|
|
765df805eb | ||
|
|
7c7ecb8b1d | ||
|
|
89243ef6eb | ||
|
|
074afd3f1e | ||
|
|
752bdea97d | ||
|
|
7526951804 | ||
|
|
83c5aa0b9d | ||
|
|
0dba1abf2a | ||
|
|
c8b59bd16f | ||
|
|
2844a0617c | ||
|
|
d4a3f4acd2 |
+38
-6
@@ -28,18 +28,30 @@ jobs:
|
|||||||
- name: Install dependencies
|
- name: Install dependencies
|
||||||
run: npm ci
|
run: npm ci
|
||||||
|
|
||||||
- name: Type check
|
# v0.8.0: 源码类型检查
|
||||||
|
- name: Type check (src)
|
||||||
run: npm run typecheck
|
run: npm run typecheck
|
||||||
|
|
||||||
|
# v0.8.0 新增:**测试代码**类型检查。
|
||||||
|
# 此前 tests/ 既不在 tsconfig include 中,babel-jest 又只剥离类型不做检查,
|
||||||
|
# 于是测试里的类型错误、拼写错误、未 await 的断言全部不可见
|
||||||
|
# (审计实例:v025-fixes 里标题写 indexeddb、实际传 'opfs' 的用例长期存活)。
|
||||||
|
- name: Type check (tests)
|
||||||
|
run: npm run typecheck:tests
|
||||||
|
|
||||||
|
# v0.8.0: lint 不再 continue-on-error —— 此前 lint 失败永远不让 CI 变红,
|
||||||
|
# 形同虚设。现在 src 与 tests 都纳入,0 error 才通过。
|
||||||
- name: Lint
|
- name: Lint
|
||||||
run: npm run lint
|
run: npm run lint
|
||||||
continue-on-error: true
|
|
||||||
|
|
||||||
# v0.7.2: Run tests 拆两步 —— 常规套件并行(快),重型套件串行(runInBand)。
|
# v0.7.2: Run tests 拆两步 —— 常规套件并行(快),重型套件串行(runInBand)。
|
||||||
# 此前重型测试(10 万行 kv/opfs、生产矩阵)与常规套件在慢 runner 上并行
|
# 此前重型测试(10 万行 kv/opfs、生产矩阵)与常规套件在慢 runner 上并行
|
||||||
# 争抢 CPU 与 4GB 堆 → 单测超时(120~180s)与 OOM 类假失败。
|
# 争抢 CPU 与 4GB 堆 → 单测超时(120~180s)与 OOM 类假失败。
|
||||||
- name: Run tests (regular suites)
|
#
|
||||||
run: npx jest --forceExit --maxWorkers=1 --no-cache --testPathIgnorePatterns='/node_modules/|/tests/e2e/|/tests/helpers/|aria-prod-load|kvstore-stress|aria-matrix-audit|aria-idx-flush-race'
|
# v0.8.0: 常规套件**带覆盖率**运行(此前 CI 从不跑 --coverage,且 jest.config
|
||||||
|
# 没有任何 coverageThreshold → 覆盖率掉到 0% 也全绿)。阈值定义在 jest.config.cjs。
|
||||||
|
- name: Run tests (regular suites, with coverage gate)
|
||||||
|
run: npx jest --coverage --forceExit --maxWorkers=1 --no-cache --testPathIgnorePatterns='/node_modules/|/tests/e2e/|aria-prod-load|kvstore-stress|aria-matrix-audit|aria-idx-flush-race'
|
||||||
env:
|
env:
|
||||||
NODE_OPTIONS: --max-old-space-size=4096
|
NODE_OPTIONS: --max-old-space-size=4096
|
||||||
|
|
||||||
@@ -56,12 +68,26 @@ jobs:
|
|||||||
- name: Build
|
- name: Build
|
||||||
run: npm run build
|
run: npm run build
|
||||||
|
|
||||||
|
# v0.8.0: 校验仓库内 dist/ 与源码同步。
|
||||||
|
# 此前 dist/ 已提交入库(42 个提交都在改 dist),但 CI 只 build 到工作区、
|
||||||
|
# 不校验是否与源码一致 → 发布产物可以静默落后于源码。
|
||||||
|
# 仅在单一 Node 版本上校验(构建结果与 Node 版本无关)。
|
||||||
|
- name: Verify dist is in sync with src
|
||||||
|
if: matrix.node-version == '20.x'
|
||||||
|
run: |
|
||||||
|
if ! git diff --quiet -- dist/; then
|
||||||
|
echo "::error::dist/ 与源码不同步。请在本地运行 npm run build 并提交 dist/。"
|
||||||
|
git diff --stat -- dist/
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "dist/ 与源码同步 ✓"
|
||||||
|
|
||||||
e2e:
|
e2e:
|
||||||
runs-on: debian-latest
|
runs-on: debian-latest
|
||||||
# 与 MetonaEditor 一致:Playwright 官方镜像(自带 Chromium 与全部系统依赖),
|
# 与 MetonaEditor 一致:Playwright 官方镜像(自带 Chromium 与全部系统依赖),
|
||||||
# 避免 runner 宿主(Debian 11)不被 Playwright 1.62 支持的问题
|
# 避免 runner 宿主(Debian 11)不被 Playwright 1.62 支持的问题
|
||||||
container: mcr.microsoft.com/playwright:v1.62.1-noble
|
container: mcr.microsoft.com/playwright:v1.62.1-noble
|
||||||
timeout-minutes: 15
|
timeout-minutes: 20
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v4
|
- uses: actions/checkout@v4
|
||||||
- name: Use Node.js
|
- name: Use Node.js
|
||||||
@@ -75,5 +101,11 @@ jobs:
|
|||||||
run: npm run build
|
run: npm run build
|
||||||
- name: Verify Playwright browsers
|
- name: Verify Playwright browsers
|
||||||
run: npx playwright install chromium
|
run: npx playwright install chromium
|
||||||
- name: Run E2E smoke tests
|
# v0.8.0(PC-2): e2e 用 CDP `Page.crash` **真崩溃**注入,覆盖三类窗口:
|
||||||
|
# ① 写入进行中崩溃(不等 Promise)② OPFS createWritable 的 write 阶段
|
||||||
|
# ③ 提交(close/原子替换)阶段;另含 checkpoint 前后、KVStore WAL 半写、
|
||||||
|
# 多标签页锁、加密往返、页面化 + 二级索引、repair 自愈。
|
||||||
|
# 注意:e2e 依赖 dist/ 产物(harness.html 加载 /dist/metona-sqlark.js),
|
||||||
|
# 因此上面的 Build 步骤是必需的;主 job 会校验 dist 与源码同步。
|
||||||
|
- name: Run E2E tests
|
||||||
run: npm run test:e2e
|
run: npm run test:e2e
|
||||||
|
|||||||
@@ -11,3 +11,6 @@ coverage/
|
|||||||
# Playwright e2e 产物
|
# Playwright e2e 产物
|
||||||
test-results/
|
test-results/
|
||||||
playwright-report/
|
playwright-report/
|
||||||
|
|
||||||
|
# 变异验证进程锁(运行时产物,勿提交)
|
||||||
|
.mutation-b6.lock
|
||||||
|
|||||||
+374
-1
@@ -2,6 +2,379 @@
|
|||||||
|
|
||||||
All notable changes to MetonaSqlark will be documented in this file.
|
All notable changes to MetonaSqlark will be documented in this file.
|
||||||
|
|
||||||
|
## [0.8.0] - 2026-09-15
|
||||||
|
|
||||||
|
### 根治性迭代 —— 统一语义 / 消灭复发结构 / 验证基础设施
|
||||||
|
|
||||||
|
> 依据 v0.8.0 迭代方案的三条并行工作流(A 缺陷修复 / B 结构根治 / C 验证基础设施)
|
||||||
|
> 完成的一次系统性迭代。**这一版的重点不是"再修一批 bug",而是砍掉让同类 bug
|
||||||
|
> 必然复发的结构**:多处并存的语义实现被收敛为唯一实现,并第一次让崩溃语义、
|
||||||
|
> 错误码一致性、入口等价性变成可机器验证的门禁。
|
||||||
|
>
|
||||||
|
> 测试规模 1304 → **1985(93 套件)+ 14 项 e2e**(另 4 个重型套件在独立 CI
|
||||||
|
> job 串行运行);B-6 与审查轮的全部修复另有 **42 项变异验证**(`scripts/mutation-b6.py`,
|
||||||
|
> 全部通过,且脚本自带正控 / 编译失败区分 / 超时 / 逐字节恢复校验 / 进程锁)。
|
||||||
|
|
||||||
|
### 工作流 B · 结构根治(消除整类缺陷)
|
||||||
|
|
||||||
|
- **B-1 唯一校验 choke point** — 此前有**三份**行校验实现,覆盖面各不相同
|
||||||
|
(memory 一份缺 `maxLength`/`min`/`max`,Aria 一份有,schema.ts 第三份):
|
||||||
|
同一份 schema、同一条 INSERT 是否报错取决于选了哪个引擎(A12);四份实现
|
||||||
|
对未知列一律静默丢弃(A17:INSERT 报成功、`SELECT nope` 报 COLUMN_NOT_FOUND)。
|
||||||
|
现收敛为 `src/table/validation.ts#compileValidator` 唯一实现,四个引擎新增
|
||||||
|
`validatePayload` 契约,校验先于任何副作用;未知列、`NaN`/±Infinity
|
||||||
|
(JSON 无法表示,落盘会变 null)显式拒绝。
|
||||||
|
- **B-2 唯一值比较与编码** — `sqlCompare`/`sqlCompareOrder`/`encodeValueKey`
|
||||||
|
成为唯一原语;GROUP BY 键、DISTINCT 键、UNION 去重、聚合去重全部改用它
|
||||||
|
(此前四份编码并存,对 null/undefined 处理各不相同)。
|
||||||
|
- **B-3 单管线** — QueryBuilder 此前**自己执行**:无 JOIN 时直通 `engine.find`,
|
||||||
|
写操作直通 `engine.update/delete`,于是"同一条语义"在 TABLE API 与 SQL API
|
||||||
|
两条路径上规则各写一份(投影、列校验、LIMIT 下推、`maxRowsPerQuery` 全缺失;
|
||||||
|
`$subquery` 无人解析 → 静默影响 0 行)。现在 builder 只产出 AST,执行一律经
|
||||||
|
Executor;生命周期钩子由 `Table` 注入、顺序与传参不变。
|
||||||
|
→ 新增 `tests/v080-single-pipeline.test.ts`(两入口逐值等价,四引擎)。
|
||||||
|
- **B-4 统一表达式求值** — CASE 此前用**正则**切分 WHEN/THEN/ELSE,不认字符串
|
||||||
|
字面量与嵌套:嵌套 CASE 返回字符串残片 `"big' END ELSE 'small"`;条件引用
|
||||||
|
不存在的列时静默把整列变成 ELSE 值;`GROUP BY CASE ... END` 完全不可用
|
||||||
|
(报"未知列 CASE WHEN ...")。现复用 `sql/lexer` 的 token 流做递归下降,
|
||||||
|
条件交给与 WHERE 相同的解析器,无法识别的表达式显式报错。
|
||||||
|
→ 顺带修正 `Token.position` 语义按类型不一致的缺陷(字符串 token 指向引号之内,
|
||||||
|
导致按位置切片少一个字符)。
|
||||||
|
- **B-5 输出列序号 + 分隔标识符** — `ORDER BY 1` / `GROUP BY 2` 此前直接
|
||||||
|
`PARSE_ERROR`;`SELECT "1"`(列名就叫 1)被当成**常量 1**(与 `SELECT *` 结论相反)。
|
||||||
|
现支持输出列序号(越界、`ORDER BY 0`、`GROUP BY <聚合列>` 各自显式报错),
|
||||||
|
并统一分隔标识符语义:引号只在"解析→执行"边界脱去。
|
||||||
|
顺带补上 **ORDER BY 的列存在性/歧义校验**(此前 JOIN 里裸写两表同名列既不报错
|
||||||
|
也不确定按哪列排)。
|
||||||
|
- **B-6 存储提交点(完整实施,非降级方案)** — 交付物与提交顺序见下方各条。
|
||||||
|
- **`__aria_manifest_<generation>` 单一提交点**:页面水位 + 各命名空间 SSTable
|
||||||
|
元数据 + 表结构 + WAL 起始位置 + 待落盘冻结表意图,一次原子提交
|
||||||
|
(头部/载荷双 CRC、先写后验、保留两代)。顺序固定为
|
||||||
|
**数据落盘 → manifest 提交 → 才允许截断 WAL / 删除旧文件**。
|
||||||
|
- **元数据损坏不再静默空库**:此前 `__aria_lsm_meta*` 是裸 JSON,解析失败即
|
||||||
|
`[]` → 看不到任何表,随后 `repair()` 还会把"没人引用"的活页删光(不可逆)。
|
||||||
|
现在全部世代校验失败抛 `ARIA_MANIFEST_CORRUPT`,旧格式迁移遇到坏 JSON 抛
|
||||||
|
`ARIA_LEGACY_META_CORRUPT`。表结构记录的**形状**坏掉(数组 / null / 表名映射到
|
||||||
|
非对象 / 列定义不是对象)此前被静默忽略(打开后"看不到任何表")—— 现在与坏 JSON
|
||||||
|
同一条路径显式报错(v0.8.0 审查修复)。
|
||||||
|
- **WAL**:LSN 改为一库一条单调水位(manifest 记账);按水位删除旧分片;
|
||||||
|
**分片号绝不回退、也绝不低于 manifest 水位**(修复前全量截断后重置为 0,
|
||||||
|
会与 manifest 记录的 `startSegment` 错位,实测造成"删掉的行复活"与
|
||||||
|
"已确认写入丢失"两个方向的损坏;整体清空后允许复用**最后用过的那个号** ——
|
||||||
|
记录自带 LSN,旧世代记录按水位跳过,这一点在审查轮做了语义澄清);
|
||||||
|
分片内部空洞显式上报,水位从未推进时的前缀缺失同样按空洞上报,
|
||||||
|
水位已推进时的前缀缺失视为"已清理的前缀"(回退上一代 manifest 的正常情况)。
|
||||||
|
另修:前缀缺失此前会把**整段活分片**丢掉(`kept = segments.filter(seq < firstGap)`
|
||||||
|
在只有前缀缺失时结果为空的连带后果)。
|
||||||
|
- **WAL 记录级损坏**:CRC 失败的记录此前只打一条日志 —— 现在计数并进入恢复报告
|
||||||
|
(`droppedWALRecords` + `dataLossSuspected = true`),"少了几条已提交写入"不再
|
||||||
|
不可观测。
|
||||||
|
- **checkpoint 不再把 LSN 归零**:LSN 是 manifest 记录的全库单调水位,归零会让
|
||||||
|
"跳过 `lsn <= startLsn`"的判定与历史分片冲突(同一段 LSN 区间对应两批不同记录)。
|
||||||
|
- **`vacuum()` 报告真实层数**:修复前硬编码返回 6 且**底部层永不压缩**
|
||||||
|
(墓碑与历史版本在最底层永久累积);现在逐层尝试(含底部层原地合并)并只统计
|
||||||
|
真正合并了的层,同时把逐层压缩挂到维护链上串行执行。
|
||||||
|
- **`close()` 在落盘失败时也必须释放资源**:修复前会卡在 flush 上,后端/锁/状态
|
||||||
|
都不复位(现在 `try/finally` 保证清理,错误照常抛出)。
|
||||||
|
- **LSM**:冻结表成为一等状态(失败可重试,`flush()` 先入链再报错,修复前
|
||||||
|
一次后台失败会让之后每次 flush 直接抛错、数据永远等不到落盘);compaction
|
||||||
|
不再"先摘整层再合并"(窗口内该层对读者不可见 → 少行);底部层原地合并
|
||||||
|
**回收墓碑**;`compacting` 改为按层集合;被取代的 SSTable 进入退休表,
|
||||||
|
等更早的读者退出才物理删除;`rangeScanLazy` 提前终止不再多算一条。
|
||||||
|
- **读路径自洽**:删除引擎层全部 `prefetch*`/`drainChain` 依赖,改为
|
||||||
|
"快照 + 结构版本乐观重试"(修复前那次改动会暴露一个新缺陷:并发 flush
|
||||||
|
在扫描的 await 窗口里发布的 SSTable 对本次扫描不可见 → 刚改名的行读回旧值)。
|
||||||
|
- **checkpoint 不等 compaction**:写路径的周期 checkpoint 只落 memtable,
|
||||||
|
compaction 继续后台跑(v0.6.1 记录的 "8~11s 悬崖"的另一半)。
|
||||||
|
- **介质故障与"文件不存在"分开**:读失败抛 `ARIA_SSTABLE_READ_FAILED`,
|
||||||
|
不再被折叠成 null 从而误删元数据。
|
||||||
|
- **恢复报告**:`engine.getRecoveryReport()` 返回
|
||||||
|
`{droppedSSTables, dataLossSuspected, walGaps, legacyImported, manifestFallback}`。
|
||||||
|
- 新增 `tests/v080-b6-single-commit-point.test.ts`(审查轮后扩到 **96 项**:
|
||||||
|
manifest 严格校验表驱动 24 例 + 事务水位 P0 + 前缀空洞语义 + 在途读者与退休文件 +
|
||||||
|
孤儿回收门槛 + 稀疏/损坏分支 + 旧格式形状校验 + bloom 配置透传)与
|
||||||
|
`scripts/mutation-b6.py`(**42 项**变异验证:把修复回退到修复前行为,对应用例
|
||||||
|
必须失败 —— 全部被拦住)。
|
||||||
|
- **B-6 存储提交点(KVStore 侧止血)** — 两处 P0:① `open()` 遇损坏日志尾部会**清空整个
|
||||||
|
日志**(写 3 条 → 第 4 条撕裂 → 重开可见 → 再重开全空);② 自动 checkpoint
|
||||||
|
失败会让**已确认写入**报错(而该写入已在 WAL 中,报错与事实相反)。另修陈旧实例
|
||||||
|
的 checkpoint 会**静默抹掉**新实例写入(现抛 `STALE_INSTANCE` 拒绝提交)。
|
||||||
|
|
||||||
|
### 工作流 A · 缺陷修复(24 项,含 6 项事故级)
|
||||||
|
|
||||||
|
- **A15 三值逻辑** — `= NULL` 命中 NULL 行、`!= NULL` 返回所有非 NULL 行、
|
||||||
|
`NOT LIKE` 把 NULL 判真、**`NOT BETWEEN 1 AND 2` 恒空集**(字段级 `$or` 递归进了
|
||||||
|
where 子句级求值器)。现 WHERE 只有**一个**递归求值器;`IS NULL`/`IS NOT NULL`
|
||||||
|
是与比较不同的**谓词**(此前与 `= NULL` 共用同一 AST,语义无法区分)。
|
||||||
|
- **A9/A10 发布订阅与关联子查询** — `subscribe()` 对本地写入永不触发(仅跨标签页
|
||||||
|
广播);`WHERE t.x = t.y` 与 `WHERE id IN (SELECT ... WHERE o.user_id = u.id)`
|
||||||
|
静默空结果(引擎层预过滤把逐行谓词判 UNKNOWN → 候选行 0)。
|
||||||
|
- **A13 自引用外键** — `parent_id REFERENCES node(id)` 的级联被整体跳过:
|
||||||
|
`DELETE root` 只删根,子树**永久悬挂**(父行已不在,再也无法级联清理)。
|
||||||
|
`ON DELETE SET NULL` / `ON UPDATE CASCADE` / RESTRICT 预检同样失效。
|
||||||
|
- **A22/A23/A25/A26/A27/A29/A30/A36 查询层** — GROUP BY 别名、HAVING 未选中聚合、
|
||||||
|
带前缀聚合参数恒 0、UNION 尾部子句归属、DISTINCT 作用于输出列、
|
||||||
|
`maxRowsPerQuery` 静默截断写入、INSERT 值多于列、派生表别名引用。
|
||||||
|
- **A37 列引用** — 未限定列不能作比较操作数(`WHERE x = y` 报 PARSE_ERROR);
|
||||||
|
`$col` 引用不存在的列**静默返回空集**。
|
||||||
|
- **A38/A39 Aria 存储** — `compression` 在页面化路径(默认)被静默忽略;
|
||||||
|
`compressLZ4` 匹配搜索 O(n²)(60KB 伪随机 2345ms → 6ms,**390×**)。
|
||||||
|
- **A41 DDL 原子性** — DDL 的 WAL 意图记录写在生效**之后**:`dropTable` 后崩溃 →
|
||||||
|
重开表又回来了(DROP 被静默撤销);`alterTable` 完全不写 WAL,崩溃丢失结构变更。
|
||||||
|
现统一为"先写 WAL 意图并刷盘 → 再改内存 → 最后落盘 schema"。
|
||||||
|
|
||||||
|
### 工作流 C · 验证基础设施(让门禁真的能拦)
|
||||||
|
|
||||||
|
- **真崩溃注入(PC-2)** — e2e 的 `crashPage()` 此前只是 `page.close()`
|
||||||
|
(**优雅关闭**),所有"崩溃恢复"用例测的其实是"正常关闭后重开"。现改用 CDP
|
||||||
|
`Page.crash` 终止渲染进程,并新增两个真实窗口:`createWritable().write()` 中途、
|
||||||
|
`close()` 原子替换前(copy-on-write 的核心不变量)。
|
||||||
|
- **覆盖率门禁真正生效** — `collectCoverageFrom` 不再排除实现文件
|
||||||
|
(此前 `!src/**/index.ts` 把 2282 行的 AriaEngine 整文件排除在统计外,
|
||||||
|
于是"90.1% 行覆盖率"是虚高的口径);新增 `coverageThreshold`
|
||||||
|
(statements 90 / branches 82 / functions 94 / lines 93);CI 常规 job 带
|
||||||
|
`--coverage`;lint 去掉 `continue-on-error`;新增 `tests/` 类型检查
|
||||||
|
(修复 **103 个**被 babel 剥离类型掩盖的测试类型错误);CI 校验 dist 与源码同步。
|
||||||
|
→ 实测(v0.8.0 收尾复测)Statements 90.59% / Branches 82.61% / Functions 94.14% /
|
||||||
|
Lines 93.50%(命令与 CI 常规 job 完全一致,可复现)。
|
||||||
|
- **测试介质忠实性修正**(两处同源缺陷,此前让所有多实例/多库验证跑在错误语义上)
|
||||||
|
- `SharedMemoryBackend` 的读缓存是每实例私有的 → 介质退化为"每实例一份快照",
|
||||||
|
跨实例写入不可见(这正是陈旧实例覆盖新实例写入那条 bug 起初查不出来的原因);
|
||||||
|
- OPFS mock 把 `getDirectoryHandle(name)` 的库名**丢弃** → 所有库共用一棵文件树
|
||||||
|
(`open('db-beta')` 能看到 `db-alpha` 的表)。
|
||||||
|
- **变异验证成为回归套件的标准做法** — 把修复回退到修复前的行为,对应用例必须
|
||||||
|
失败。本版 A15 / B-6(①②③) / A13 / A37 / A38 / A39 / A41 / B-4 / B-5 全部通过该检查
|
||||||
|
——这是"测试真能拦住回归"与"测试只是陪跑"的分界线。
|
||||||
|
|
||||||
|
### 文档与宣称同步(G6)
|
||||||
|
|
||||||
|
- 修正 README 全部**不成立的能力宣称**:覆盖率数字改为**四个准确数字 + 明确口径**;
|
||||||
|
测试规模与套件数更新为实测值;"5 种存储引擎"改为"4 种模式 + 3 种后端";
|
||||||
|
`backup()` 由"在线一致性快照"改为"全库导出(逐表读取)"并写入已知限制
|
||||||
|
(引擎层没有跨表快照原语);`db.disconnect()` 真正进入类型系统
|
||||||
|
(此前仅运行时注入,TypeScript 使用者编译失败)。
|
||||||
|
- **打包缺陷修复**:`package.json` 的 `./migration` 子路径此前指向的产物里
|
||||||
|
**没有** `migrateFromIndexedDB`(主入口未导出该函数)→ 现主入口导出 + 子路径可用;
|
||||||
|
`./react` / `./vue` 此前指向**裸 TS 源码**且声明类型为 `any`
|
||||||
|
→ 现构建 `dist/react.js` / `dist/vue.js` 并配套**手写精确类型声明**;
|
||||||
|
补 `peerDependencies`(react / vue,均可选)。
|
||||||
|
|
||||||
|
### 全量回归审查(发版前最后一轮,独立复现 + 变异验证)
|
||||||
|
|
||||||
|
> 方法:四个对抗性子代理分头审查(数据正确性 / 文档宣称 vs 实现 / 公共 API 契约 /
|
||||||
|
> 测试质量),**每一条结论都要求可复现证据**;我再逐条复核并用探针确认。共确认
|
||||||
|
> 14 项实现缺陷(1 项 P0、4 项 P1、9 项 P2)与 16 条不成立的文档宣称。
|
||||||
|
|
||||||
|
**P0 · 事务活跃期间推进 WAL 水位会让已 COMMIT 的事务整批消失**
|
||||||
|
|
||||||
|
- 复现:`BEGIN` → `INSERT` → `repair()` → `COMMIT` → 崩溃 → 重开 →
|
||||||
|
已提交的行不见了,而 `getRecoveryReport()` 报告"干净"(`dataLossSuspected:false`、
|
||||||
|
`walGaps:[]`)。
|
||||||
|
- 根因:`hasPendingFlushData()` / `computeDurableLsn()` 只看 memtable/frozen,
|
||||||
|
而事务内的写入只落在 `txnSnapshot`(内存)+ WAL —— 于是"没有未落盘数据"成立,
|
||||||
|
水位被推到当前 LSN 并按该水位删掉旧分片;随后 `COMMIT` 返回成功,但那些
|
||||||
|
INSERT 记录已因 `lsn <= startLsn` 被跳过。此前只有 `CheckpointManager` 的两个
|
||||||
|
回调带事务守卫,`repair()` / `close()` / 周期 checkpoint 三条路径都能踩到。
|
||||||
|
- 修复:守卫下沉到 `computeDurableLsn()` 与 `advanceWalCheckpoint()` 入口(唯一实现),
|
||||||
|
三条路径同时覆盖;新增回归用例与变异验证(R1)。
|
||||||
|
|
||||||
|
**P1**
|
||||||
|
|
||||||
|
1. **WAL 前缀缺失丢弃整段活分片** — 前缀缺失与内部空洞被混为一谈时,
|
||||||
|
`kept = segments.filter(s => s.seq < firstGap)` 的结果为空 → 回退到上一代
|
||||||
|
manifest(`startSegment` 比现存最小分片更小)时**所有活分片被丢掉**。
|
||||||
|
现在前缀缺失单独记录(`missingPrefix`),后缀照常重放;只有"水位从未推进
|
||||||
|
(`fromLsn === 0`)"时前缀缺失才算真异常(R4)。
|
||||||
|
2. **孤儿回收门槛漏掉 LSM 层损坏** — 门槛只看引擎层 `dataLossSuspected`
|
||||||
|
(只在 WAL 无兜底时才置位),于是"manifest 已推进 + 某 SSTable 因损坏被丢"
|
||||||
|
这类真损坏下,`repair()` 仍会把"引用不到"的页面(含在途读者持有的退休文件)
|
||||||
|
当作孤儿删掉(不可逆)。现在统一用 `describeRecoveryDamage()` 聚合引擎层与
|
||||||
|
各 LSM 的全部损坏迹象(被丢的 SSTable / WAL 空洞 / 记录损坏 / 回退世代)(R3)。
|
||||||
|
3. **`vacuum()` 绕过维护链** — 逐层压缩改为 `vacuumLevels()`:每层作为维护链任务
|
||||||
|
执行(复用 `compacting` 防重入),不再与后台 compaction 并发向同一目标层写产物
|
||||||
|
(产物一律 `unshift` 到队首,层内顺序 = 新旧顺序,交错会读到旧值 / 底部层丢墓碑
|
||||||
|
导致已删除行复活)。**如实说明**:引擎层 `vacuum()` 开头的 `lsm.flush()` 本来就会
|
||||||
|
drain 维护链,所以"引擎层调用"这条路径恰好被顺带串行化 —— 真正被测试锁定的是
|
||||||
|
LSM 层不变量(R 系列未覆盖 `drainMaintenance` 那一行,它在实现上只是优化:
|
||||||
|
抢在链前重算层内文件数,去掉后仍串行,只是会多排几个空任务)。
|
||||||
|
4. **`reclaimRetiredNow()` 无视在途读者** — 读者的快照可能正持有被"退休"的文件,
|
||||||
|
强制回收会让它读不到数据(并被误判为"文件损坏")。现在有活跃读者时退化为
|
||||||
|
延迟回收,读者退出后再物理删除(R2)。
|
||||||
|
|
||||||
|
**P2**
|
||||||
|
|
||||||
|
- WAL 记录级 CRC 损坏只打日志(`droppedWALRecords` + 数据丢失标记,R9)。
|
||||||
|
- 旧格式表结构记录的**形状**损坏(数组 / null / 非对象条目)静默当空库
|
||||||
|
→ 与坏 JSON 走同一条显式报错路径(R16)。
|
||||||
|
- **`bloomFilterBitsPerKey` 配置被接受却完全不生效**:`SSTableBuilder` 始终用默认
|
||||||
|
位数构造 Bloom Filter —— 这是实现缺陷,修的是实现(配置透传到构建器)而不是文档;
|
||||||
|
新增"位数改变落盘 bloom 段大小 + 任意位数都不产生 false negative"的测试(R17/R18)。
|
||||||
|
- `pageIdWatermark` 只做单调 max 的断言此前不存在(白盒水位回退用例,R7)。
|
||||||
|
- manifest 提交的**回读校验**此前无用例覆盖(吞写 → 必须 `ARIA_MANIFEST_WRITE_FAILED`,
|
||||||
|
且世代号不前进;覆盖"文件读不回"与"内容读回来是坏的"两条守卫,R8)。
|
||||||
|
- "文件名世代 ≠ 载荷世代"的世代无效判定无用例(R6)。
|
||||||
|
- `dropInvalidSSTable` 丢弃文件后不同步 `levels` 会留幽灵 meta;"介质读故障 ≠
|
||||||
|
文件损坏"此前无用例(读故障必须让 compaction 失败且一个 meta 都不许丢,R11)。
|
||||||
|
- 旧格式迁移的失败分支(`__aria_schemas` 坏 JSON / 非数组 / 条目形状非法)只有
|
||||||
|
坏 JSON 一条有用例。
|
||||||
|
- WAL 整体清空后"分片号绝不回退到 0"与"新写入绝不低于 manifest 水位"两条不变量
|
||||||
|
此前无用例(R13 / R14)。
|
||||||
|
|
||||||
|
**覆盖率口径(审查发现的第二处口径问题)**
|
||||||
|
|
||||||
|
- README / `jest.config.cjs` / PLAN 都声称 `collectCoverageFrom` 只排除了**两个纯类型文件**
|
||||||
|
(`engine/interface.ts`、`query/ast.ts`,"可执行语句为 0")。审查实测:`interface.ts`
|
||||||
|
里有三个**运行时函数**(`cloneRow` / `cloneRowFallback` / `cloneRows`,约 18 条语句),
|
||||||
|
被 Memory 引擎与 AriaEngine 的读路径调用 —— 也就是说这块真实实现代码一直逃过覆盖率
|
||||||
|
统计,正是 G5 门禁要根治的"口径虚高"。
|
||||||
|
- 根因修复(而不是改文档):实现搬到新模块 `src/engine/row_clone.ts`,
|
||||||
|
`interface.ts` 回到**纯类型**(`query/ast.ts` 本来就是)。搬完覆盖率门禁立刻**失败**
|
||||||
|
(functions 94% 阈值 → 实测 93.84%),暴露出退化路径 `cloneRowFallback` 从未被测试;
|
||||||
|
补 `tests/engine/row-clone.test.ts`(8 项:structuredClone 可用/不可用/抛错三条路径、
|
||||||
|
Date/TypedArray/ArrayBuffer、深拷贝不泄漏、批量拷贝)后门禁恢复通过。
|
||||||
|
- 最终数字(口径修正后,含新增的 8 项测试):93 套件 / 1985 用例,
|
||||||
|
语句 90.59% / 分支 82.61% / 函数 94.14% / 行 93.50%。
|
||||||
|
|
||||||
|
**测试质量修正("绿"不等于"有保护")**
|
||||||
|
|
||||||
|
- 3 条空壳用例改为值级断言(孤儿回收门槛、强制回收、页面映射退休);
|
||||||
|
- 1 条"整层文件全损坏"用例实际上走的是**缓存**(根本没读介质,等于什么都没测)
|
||||||
|
→ 拆成"介质读故障 = compaction 必须失败且不许丢 meta"与"文件真残缺 = 丢弃必须
|
||||||
|
进恢复报告"两条真用例;
|
||||||
|
- 5 秒墙钟 `Promise.race` 改为门控信号 + 失败上限(超时只会让测试失败,绝不会让
|
||||||
|
它在实现错误时"碰巧通过");`setTimeout` 睡眠改为 `whenIdle()` 确定性等待;
|
||||||
|
- `<=` 收紧为严格 `<`(冻结意图的水位必须**严格低于**意图起点);删掉死代码
|
||||||
|
(`const inner`、`void manifest`)与误名的"非 JSON"用例。
|
||||||
|
|
||||||
|
### 现场失败修复(站点演示页 file:// 打不开 Aria 引擎)
|
||||||
|
|
||||||
|
用户报告"站点演示失败了"。复现与根因(真实 Chromium 实测,非推断):
|
||||||
|
|
||||||
|
- **现象**:直接双击 `site/demo.html`(`file://`)→ 点「🌲 Aria」→
|
||||||
|
`❌ 数据库初始化失败: Failed to open AriaEngine database "demo"`(切回 Memory 正常)。
|
||||||
|
- **根因**:`file://` 属于不透明来源,Chromium 认为本地文件"不适合 Web 应用访问",
|
||||||
|
`navigator.storage.getDirectory()` 抛 `SecurityError`。注意此时
|
||||||
|
`isSecureContext === true` 且 API **存在** —— 只有真正调用才会失败,所以无法提前探测;
|
||||||
|
而引擎把它包成 `ARIA_OPEN_ERROR` 时**丢掉了底层错误**,消息对用户不可操作。
|
||||||
|
- **修复**:`OPFSBackend.open()` 显式检查 OPFS 可用性,抛 `ARIA_OPFS_UNAVAILABLE`,
|
||||||
|
消息里直接给出两条出路(**用 http(s) 打开**:`node tests/e2e/server.cjs` /
|
||||||
|
`npx serve .`,或改用 `mode: 'memory'`),并把原始 `SecurityError` 挂在 cause 上。
|
||||||
|
- **顺带修掉一个更普遍的问题**:`DatabaseError` 的第三个参数只存进 `details`,
|
||||||
|
**`err.cause` 恒为 undefined** —— 而文档/注释多处写"底层错误作为 cause 保留"。
|
||||||
|
现在两个字段都成立(`details` 语义不变,向后兼容;`cause` 接入标准错误链)。
|
||||||
|
- **站点版本同步**:`site/demo.html`(title/状态栏/SQL 预置脚本/console 日志)、
|
||||||
|
`site/benchmark.html`(title)此前仍标 v0.7.4(甚至 v0.4.2),已统一到 v0.8.0;
|
||||||
|
`site/docs.html` 的 AriaEngine 版本演进列表补上 v0.8.0 条目,错误码表补
|
||||||
|
`ARIA_OPFS_UNAVAILABLE`。
|
||||||
|
- 回归:`tests/engine/aria-opfs-unavailable.test.ts`(5 项:storage 缺失 /
|
||||||
|
SecurityError 带 cause 与建议 / 正常环境正控 / DatabaseError 两个字段语义),
|
||||||
|
变异 R19 / R20 均被拦住。
|
||||||
|
|
||||||
|
### 文档订正(第二轮 G6:16 条不成立的宣称)
|
||||||
|
|
||||||
|
- **MVCC**:`site/docs.html`(表格、事务行、能力表)与 `site/demo.html` 仍写
|
||||||
|
"MVCC 快照隔离",而实现是"版本链仅作事务内 undo + 事务串行";`mvcc.ts` 的文件头
|
||||||
|
同样写错 → 全部改为如实描述。
|
||||||
|
- **`backup()`**:`interface.ts` 注释仍写"一致性快照" → 改为"逐表读取(非跨表快照)"。
|
||||||
|
- **WAL 空洞**:`docs.html` 写"空洞检测截断" → 改为"内部空洞如实上报并拒绝静默继续"。
|
||||||
|
- **体积**:README 与 docs.html 写"~105KB / gzip ~27KB" → 实测 min 产物
|
||||||
|
251,731 字节 / gzip 63,431 字节(`stat -c%s dist/metona-sqlark.min.js`;
|
||||||
|
`gzip -c dist/metona-sqlark.min.js | wc -c`)。
|
||||||
|
- **测试与覆盖率**:README badge/正文、site 徽标与数字卡、CHANGELOG 全部更新为
|
||||||
|
实测值(1985 / 93 套件 / 90.59% / 82.61% / 94.14% / 93.50%)。
|
||||||
|
- **存储模式**:site 仍写"5 种存储引擎" → 改为"4 种模式 + 3 种后端";
|
||||||
|
"Tree-shakable"改为"UMD/ESM/CJS 多格式输出"(单文件 bundle + 未声明
|
||||||
|
`sideEffects`,不宜宣称可摇树)。
|
||||||
|
- **错误码表**:site 文档缺 **16 个**错误码(`ARIA_MANIFEST_NOT_LOADED`、
|
||||||
|
`ARIA_SSTABLE_SAVE_CONTRACT`、`ARIA_DB_NOT_OPEN`、`ARIA_OPEN_ERROR`、
|
||||||
|
`DB_NOT_OPEN`、`KV_*`、`TX_*`、`SAVEPOINT_*`、`UNKNOWN_STATEMENT`、
|
||||||
|
`COLUMN_EXISTS`、`FOREIGN_KEY_VIOLATION`)→ 全部补齐。
|
||||||
|
- **维护脚本与门禁**:CONTRIBUTING 的变异数量 17 → 42,并补上脚本自身的保证
|
||||||
|
(正控 / 编译失败区分 / 超时 / 逐字节恢复校验 / 进程锁);门禁表述订正三处 ——
|
||||||
|
"12 项事故级"改为 11 项(与缺陷总账 A1~A11 一致)、"已确认写入零丢失"限定为
|
||||||
|
"故障注入矩阵覆盖范围内"(并记录矩阵之外发现的那处 P0)、表驱动用例数 25 → 24。
|
||||||
|
|
||||||
|
### 已知限制(v0.8.0 新增/变更)
|
||||||
|
|
||||||
|
- `backup()` 不是跨表一致性快照(逐表读取)
|
||||||
|
- 复合主键仍不支持(建表时 `SCHEMA_ERROR`)
|
||||||
|
- 简单 CASE 形式(`CASE <表达式> WHEN <值>`)不支持,仅支持搜索式
|
||||||
|
(`CASE WHEN <条件> THEN ...`)
|
||||||
|
- **回退旧版本是单向的**:迁移后新写入只进 manifest,旧的 `__aria_lsm_meta*` /
|
||||||
|
`__aria_schemas` 停留在迁移那一刻;用旧版本打开会看到迁移时刻的旧视图
|
||||||
|
- **多实例写入依赖 Web Locks**:无 Web Locks 时降级为提交点冲突检测
|
||||||
|
(`STALE_INSTANCE`),只保证不静默覆盖别人的提交,不保证多实例写入的数据完整性
|
||||||
|
- **manifest 体积随 SSTable 数量增长**(单文件整体重写、保留两代,无增量/分层机制)
|
||||||
|
- **尾部 WAL 分片丢失无法从介质自身识别**(无"它本该存在"的证据;`ARIA_WRITE_LOST`
|
||||||
|
只兜住"有未落盘冻结表却重放不到任何记录"的情况)
|
||||||
|
|
||||||
|
|
||||||
|
## [0.7.4] - 2026-08-15
|
||||||
|
|
||||||
|
### 写语句子查询 / 约束硬化 / 真惰性流式
|
||||||
|
|
||||||
|
> 深度审计第七阶段:修复 UPDATE/DELETE WHERE 子查询静默 0 行(P1,四引擎)、
|
||||||
|
> 主键 NULL 静默入库(P1)、DROP INDEX 解除建表 UNIQUE 约束(P2)等 8 项正确性问题;
|
||||||
|
> Aria findStream 迭代器化落地"真惰性"宣称;REINDEX 单次扫描性能优化。
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **UPDATE/DELETE WHERE 子查询静默 0 行(P1,四引擎)** — `executeUpdate/executeDelete`
|
||||||
|
直接 compileStatement 调引擎,`$subquery` 未解析:引擎层 matchWhere 的 `$in/$nin`
|
||||||
|
遇对象恒 false → 所有行不匹配,写语句静默影响 0 行(与 v0.7.3 修的 queryStream
|
||||||
|
同类)。executor 写路径先 `resolveWriteWhere` 解析非关联子查询(IN 列表/标量),
|
||||||
|
EXPLAIN 的 estimatedRows 同步修复;关联引用($col / 关联 EXISTS)显式
|
||||||
|
`NOT_SUPPORTED`(写语句无法逐行绑定外层上下文);QueryBuilder 直通引擎路径
|
||||||
|
由引擎层 `containsUnresolvedSubqueries` 防御(Memory/Aria 双引擎)
|
||||||
|
- **主键 NULL/undefined 静默入库(P1,四引擎)** — 主键列缺失/置 null 时
|
||||||
|
`String(undefined)`→`"undefined"`、`String(null)`→`"null"` 作为主键落库。
|
||||||
|
validateRow 层强制主键非空(SQL 语义 PK 隐含 NOT NULL)抛 `VALIDATION_ERROR`;
|
||||||
|
default 值生效时仍可省略主键(语义不变)
|
||||||
|
- **DROP INDEX 解除建表 UNIQUE 约束(P2)** — `dropIndex` 此前 `colDef.unique = false`
|
||||||
|
静默解除建表约束(aria 侧 persistSchemas 后重启约束永久消失)。现在建表 UNIQUE
|
||||||
|
列 DROP INDEX 抛 `NOT_SUPPORTED`(对齐 SQLite:需重建表解除);仅
|
||||||
|
`CREATE UNIQUE INDEX` 添加的约束可随索引删除(引擎内 `uniqueIndexCols` 跟踪来源,
|
||||||
|
重启后 schema 中的 unique 一律按建表约束保护)
|
||||||
|
- **GROUP BY 把 null 与字符串 'null' 合并(P2)** — 分组键 `String(row[col] ?? 'null')`
|
||||||
|
使两类值合并为一组;DISTINCT/UNION 去重键 `String(v ?? '\0')` 同类吞并。
|
||||||
|
统一类型安全编码(类型前缀),仅同类型同值合并(与 where-matcher 的 === 语义一致)
|
||||||
|
- **UPDATE 未知列静默写入存储行(P2)** — `SET nonexistent = ...` 的脏列残留在行内
|
||||||
|
并随持久化落盘。预检阶段显式 `COLUMN_NOT_FOUND`(SQL + Table API 双路径,
|
||||||
|
Memory/Aria 双引擎)
|
||||||
|
- **queryStream 多语句语义不一致(P2)** — `parseAll(sql)[0]` 静默忽略后续语句:
|
||||||
|
可流式时不执行(如 DELETE),不可流式时回退 query() 却执行全部语句。
|
||||||
|
多语句显式 `PARSE_ERROR`(流式 API 要求单条 SELECT)
|
||||||
|
- **KVStore lastBackgroundError 跨生命周期残留(P3)** — close/open 未清理,
|
||||||
|
重开后首次 checkpoint 抛出上一次生命周期的旧错误。open/close 统一清零
|
||||||
|
- **Hybrid beginTransaction 部分成功泄漏(P3)** — 内存 begin 成功 + 磁盘 begin
|
||||||
|
抛错 → 内存快照泄漏(后续事务永久 TX_ACTIVE)。磁盘失败补偿回滚内存
|
||||||
|
- **MemTable RB-Tree 删除双黑修复边界(P3)** — 两子节点删除时 successor 右子
|
||||||
|
为 null 的占位 parent 传 null → fixDelete 直接跳过修复(树失衡)。
|
||||||
|
修复 parent 传递(直接右子传 successor,间接右子传 successor 原父)
|
||||||
|
- **死代码清理** — 移除 LSM.getAllEntries(无调用者;其 frozen 遍历顺序与 `get()`
|
||||||
|
相反属死代码路径)
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- **Aria findStream 真惰性(宣称落地)** — 此前 `rangeScanLazy` 内部
|
||||||
|
`mergeIter.drain()` 全量物化,与"流式不物化"宣称不符。重写为真惰性:
|
||||||
|
MemTable 红黑树显式栈生成器 + SSTableReader 块级生成器 + GeneratorEntrySource
|
||||||
|
接入 MergeIterator 逐条拉取;callback 返回 false 提前终止(limit 达成时
|
||||||
|
未消费块/子树不再解析),大表流式内存 O(1)
|
||||||
|
- **REINDEX 单次全表扫描(性能)** — `reindexTableInternal` 此前每个索引列一次
|
||||||
|
getAllRows(N 列 × M 行 + 每列一次 drainChain),改为单次扫描重建全部索引列
|
||||||
|
- VERSION 0.7.3 → **0.7.4**
|
||||||
|
|
||||||
|
### Changed(测试)
|
||||||
|
|
||||||
|
- 测试 1256 → **1304**(76 套件,+48 个 v0.7.4 回归);新增 `tests/v074-fixes.test.ts`
|
||||||
|
(写语句子查询 ×10 / 关联引用 ×3 / EXPLAIN ×1 / 主键非空 ×8 / DROP INDEX unique ×5 /
|
||||||
|
分组键编码 ×4 / 未知列 ×4 / queryStream ×2 / KVStore 生命周期 ×1 / Hybrid 补偿 ×1 /
|
||||||
|
真惰性流式 ×2 / REINDEX ×2);VERSION 断言同步
|
||||||
|
- 行覆盖率 90.0% → **90.1%**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## [0.7.3] - 2026-08-14
|
## [0.7.3] - 2026-08-14
|
||||||
|
|
||||||
### INSERT 语句级原子性补全 / 索引一致性 / 流式查询回退安全
|
### INSERT 语句级原子性补全 / 索引一致性 / 流式查询回退安全
|
||||||
@@ -884,7 +1257,7 @@ All notable changes to MetonaSqlark will be documented in this file.
|
|||||||
- **SSTable 构建器/读取器** (`index/sstable_builder.ts`, `sstable.ts`): 二分查找 + 范围扫描
|
- **SSTable 构建器/读取器** (`index/sstable_builder.ts`, `sstable.ts`): 二分查找 + 范围扫描
|
||||||
- **Merge Iterator** (`index/merge_iterator.ts`): 最小堆多路归并,去重保留最新值
|
- **Merge Iterator** (`index/merge_iterator.ts`): 最小堆多路归并,去重保留最新值
|
||||||
- **WAL** (`wal/`): 二进制日志格式 (LSN/type/txnId/table/key/json/CRC) + Checkpoint 管理
|
- **WAL** (`wal/`): 二进制日志格式 (LSN/type/txnId/table/key/json/CRC) + Checkpoint 管理
|
||||||
- **MVCC** (`transaction/mvcc.ts`): 版本链 + 快照隔离 + GC
|
- **MVCC** (`transaction/mvcc.ts`): 版本链 + GC(**v0.8.0 澄清**:此处当时宣称的「快照隔离」从未实现 —— 版本链仅作事务内 undo,事务串行)
|
||||||
- **存储后端** (`store/`): IndexedDB / Memory 双后端抽象
|
- **存储后端** (`store/`): IndexedDB / Memory 双后端抽象
|
||||||
- **LZ4 压缩** (`compression/lz4.ts`): 简易页面级压缩
|
- **LZ4 压缩** (`compression/lz4.ts`): 简易页面级压缩
|
||||||
- **`mode: 'aria'`** — 新增存储模式,可通过 `MetonaSqlark.create({ mode: 'aria' })` 激活
|
- **`mode: 'aria'`** — 新增存储模式,可通过 `MetonaSqlark.create({ mode: 'aria' })` 激活
|
||||||
|
|||||||
+48
-4
@@ -75,7 +75,7 @@ src/
|
|||||||
├── plugin/ # Plugin system (14 lifecycle hooks)
|
├── plugin/ # Plugin system (14 lifecycle hooks)
|
||||||
└── integrations/ # React & Vue hooks
|
└── integrations/ # React & Vue hooks
|
||||||
|
|
||||||
tests/ # Test suite (1256 test cases, 75 suites + 12 e2e)
|
tests/ # Test suite (1985 test cases, 92 suites + 14 e2e)
|
||||||
tests/helpers/ # 共享测试工具(OPFS mock 等)
|
tests/helpers/ # 共享测试工具(OPFS mock 等)
|
||||||
tests/e2e/ # Playwright e2e(真实 Chromium + OPFS)
|
tests/e2e/ # Playwright e2e(真实 Chromium + OPFS)
|
||||||
site/ # Documentation site (index / docs / demo)
|
site/ # Documentation site (index / docs / demo)
|
||||||
@@ -134,7 +134,16 @@ MetonaSqlark follows a layered architecture:
|
|||||||
|
|
||||||
2. **Write-Through Hybrid Strategy**: When using `mode: 'hybrid'`, all writes go to both memory and disk simultaneously. Reads always hit memory for maximum speed.
|
2. **Write-Through Hybrid Strategy**: When using `mode: 'hybrid'`, all writes go to both memory and disk simultaneously. Reads always hit memory for maximum speed.
|
||||||
|
|
||||||
3. **Plugin Hook Pipeline**: 14 lifecycle hooks allow intercepting database operations without modifying core code.
|
3. **Plugin Hook Pipeline**: 14 lifecycle hooks let plugins observe and adjust database
|
||||||
|
operations without modifying core code. **Hook contract (v0.8.0)**:
|
||||||
|
- return values are **ignored** (you cannot cancel an operation by returning `false`,
|
||||||
|
nor rewrite the SQL by returning a string);
|
||||||
|
- **mutate the argument object in place** to change it — this works on the Table API
|
||||||
|
path (`db.table(...).insert(rows)`);
|
||||||
|
- **throw** to abort the operation (the error propagates to the caller);
|
||||||
|
- the SQL path passes a **copy** for `beforeInsert`, so mutating it there has no effect.
|
||||||
|
Use the Table API when you need to transform rows.
|
||||||
|
These rules are covered by tests; changing them requires updating this section.
|
||||||
|
|
||||||
4. **Hand-Written SQL Parser**: No dependencies on parser generators — a recursive-descent parser keeps the bundle size minimal.
|
4. **Hand-Written SQL Parser**: No dependencies on parser generators — a recursive-descent parser keeps the bundle size minimal.
|
||||||
|
|
||||||
@@ -146,7 +155,7 @@ npm run build
|
|||||||
|
|
||||||
# Output in dist/
|
# Output in dist/
|
||||||
# ├── metona-sqlark.js UMD
|
# ├── metona-sqlark.js UMD
|
||||||
# ├── metona-sqlark.min.js UMD minified (~105KB / ~27KB gzip)
|
# ├── metona-sqlark.min.js UMD minified (251,731 B / gzip 63,431 B)
|
||||||
# ├── metona-sqlark.esm.js ES Module
|
# ├── metona-sqlark.esm.js ES Module
|
||||||
# ├── metona-sqlark.cjs CommonJS
|
# ├── metona-sqlark.cjs CommonJS
|
||||||
# └── metona-sqlark.d.ts TypeScript declarations
|
# └── metona-sqlark.d.ts TypeScript declarations
|
||||||
@@ -163,7 +172,9 @@ const myPlugin: MetonaPlugin = {
|
|||||||
name: 'myPlugin',
|
name: 'myPlugin',
|
||||||
version: '1.0.0',
|
version: '1.0.0',
|
||||||
description: 'Description of my plugin',
|
description: 'Description of my plugin',
|
||||||
priority: 50, // higher = executed first
|
// higher = executed first(v0.8.0 起真正生效:install 与钩子都按优先级降序;
|
||||||
|
// 同优先级保持 plugins 数组顺序)
|
||||||
|
priority: 50,
|
||||||
|
|
||||||
install(db) {
|
install(db) {
|
||||||
// Use db.on() to subscribe to hooks
|
// Use db.on() to subscribe to hooks
|
||||||
@@ -217,3 +228,36 @@ const db = await MetonaSqlark.create({
|
|||||||
## Questions?
|
## Questions?
|
||||||
|
|
||||||
Open an issue at [git.metona.cn/MetonaTeam/MetonaSqlark/issues](https://git.metona.cn/MetonaTeam/MetonaSqlark/issues).
|
Open an issue at [git.metona.cn/MetonaTeam/MetonaSqlark/issues](https://git.metona.cn/MetonaTeam/MetonaSqlark/issues).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 变异验证(回归套件的"是否只是陪跑"检查)
|
||||||
|
|
||||||
|
修复类提交必须能回答一个问题:**把修复回退到修复前的行为,对应用例会不会失败?**
|
||||||
|
不会失败的用例等于没有保护。
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 全部 42 项变异(存储单一提交点 / LSM 结构根治 / v0.8.0 全量审查修复)
|
||||||
|
python3 scripts/mutation-b6.py
|
||||||
|
|
||||||
|
# 只跑其中一条(按名字子串匹配)
|
||||||
|
python3 scripts/mutation-b6.py "WAL 分片号复用"
|
||||||
|
```
|
||||||
|
|
||||||
|
脚本会临时改写 `src/`、运行对应用例、再恢复源码(收到 SIGINT/SIGTERM 也会恢复),
|
||||||
|
最后打印每一条是"被拦住"还是"仍然通过"。**出现任何一条"仍然通过",本次提交不算完成。**
|
||||||
|
|
||||||
|
### 脚本自身的保证(v0.8.0 审查后加固)
|
||||||
|
|
||||||
|
| 机制 | 为什么需要 |
|
||||||
|
| --- | --- |
|
||||||
|
| **正控** | 先跑一遍未变异的干净基线(B-6 套件 + 分片 WAL 套件必须全绿)。没有正控就无法区分"变异被测试拦住"与"这套件本来就是红的" |
|
||||||
|
| **编译失败 ≠ 被拦住** | `Test suite failed to run` / `error TS####` 单独归类为 `BAD(变异破坏编译)`:一个把源码改成语法错误的"变异"什么都没验证 |
|
||||||
|
| **用例未匹配 ≠ 被拦住** | `Tests: 0 total` / `No tests found` 归类为 `BAD(用例未匹配)`(模式写错时最容易被误读成"测试通过了") |
|
||||||
|
| **锚点唯一性** | 锚点在文件里出现多次时拒绝变异(否则可能改到另一处、验错位置) |
|
||||||
|
| **超时** | 单条变异 300s 上限,超时归类为 `BAD(超时)` 而不是放过 |
|
||||||
|
| **逐字节恢复校验** | 每条变异跑完用 sha256 比对恢复结果;不一致立即 `FATAL` 退出 |
|
||||||
|
| **进程锁** | `.mutation-b6.lock`(`O_EXCL`):并发运行会互相踩掉对方的源码,直接拒绝启动 |
|
||||||
|
|
||||||
|
判定顺序固定为:编译失败 → 用例未匹配 → `✕`/`Tests: N failed`(= 被拦住)→ 其余为"仍然通过"。
|
||||||
|
只有 `OK(变异被拦住)` 计入成功,其余任何状态都会让脚本以非零退出码结束。
|
||||||
|
|||||||
@@ -1,14 +1,16 @@
|
|||||||
# MetonaSqlark
|
# MetonaSqlark
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<img src="https://img.shields.io/badge/version-0.7.3-blue?style=flat-square" alt="version">
|
<img src="https://img.shields.io/badge/version-0.8.0-blue?style=flat-square" alt="version">
|
||||||
<img src="https://img.shields.io/badge/license-MIT-green?style=flat-square" alt="license">
|
<img src="https://img.shields.io/badge/license-MIT-green?style=flat-square" alt="license">
|
||||||
<img src="https://img.shields.io/badge/coverage-90.0%25-brightgreen?style=flat-square" alt="coverage">
|
<img src="https://img.shields.io/badge/coverage-90.59%25%20stmts-brightgreen?style=flat-square" alt="coverage">
|
||||||
<img src="https://img.shields.io/badge/tests-1256%20passed-success?style=flat-square" alt="tests">
|
<img src="https://img.shields.io/badge/tests-1985%20passed-success?style=flat-square" alt="tests">
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
> 基于 TypeScript 的**前端关系型数据库**:完整 SQL + Query Builder 双 API,
|
> 基于 TypeScript 的**前端关系型数据库**:完整 SQL + Query Builder 双 API,
|
||||||
> 5 种存储引擎可选,内置自研 LSM-Tree 存储引擎(AriaEngine)。
|
> 4 种存储模式(memory / disk / hybrid / aria)+ 3 种后端(OPFS / KVStore / Memory)可选,
|
||||||
|
> `aria` 模式内置自研 LSM-Tree 存储引擎(AriaEngine:WAL + 页面化 + 可选压缩/加密,
|
||||||
|
> 事务用未提交快照回滚)。
|
||||||
> 零运行时依赖,浏览器 / Node.js 开箱即用。
|
> 零运行时依赖,浏览器 / Node.js 开箱即用。
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -32,25 +34,28 @@
|
|||||||
|
|
||||||
**数据库能力**
|
**数据库能力**
|
||||||
|
|
||||||
- **完整 SQL** — SELECT(JOIN / 子查询 / 派生表 / UNION / GROUP BY / HAVING / DISTINCT / CASE WHEN / EXISTS / BETWEEN / NULLS 排序)、INSERT...SELECT、ALTER TABLE、TRUNCATE TABLE、CREATE INDEX、事务语句(BEGIN / COMMIT / ROLLBACK / SAVEPOINT)、维护语句(EXPLAIN / ANALYZE / REINDEX / VACUUM)
|
- **完整 SQL** — SELECT(JOIN / 子查询 / 派生表 / UNION / GROUP BY / HAVING / DISTINCT / CASE WHEN / EXISTS / BETWEEN / NULLS 排序)、INSERT...SELECT、UPDATE/DELETE 子查询、ALTER TABLE、TRUNCATE TABLE、CREATE INDEX、事务语句(BEGIN / COMMIT / ROLLBACK / SAVEPOINT)、维护语句(EXPLAIN / ANALYZE / REINDEX / VACUUM)
|
||||||
- **Query Builder** — 链式 `.select().where().innerJoin().orderBy().limit().execute()`,类型安全
|
- **Query Builder** — 链式 `.select().where().innerJoin().orderBy().limit().execute()`,类型安全
|
||||||
- **流式查询** — `queryStream` / `table().stream()` 逐行回调,Aria 引擎惰性扫描不物化结果集
|
- **流式查询** — `queryStream` / `table().stream()` 逐行回调,Aria 引擎真惰性扫描(limit 提前终止,不物化结果集)
|
||||||
- **事务** — 四引擎事务原子性 + 自动回滚;Aria 用 MVCC 快照隔离,事务读写不互斥
|
- **事务** — 四引擎事务原子性 + 自动回滚;Aria 用引擎级事务 + 未提交快照(`txnSnapshot`)回滚。**同一实例同时只允许一个事务**(并发 `beginTransaction` 抛 `TX_ACTIVE`)—— 不是 MVCC 快照隔离
|
||||||
- **外键级联** — `ON DELETE` / `ON UPDATE` 支持 `CASCADE` / `SET NULL` / `RESTRICT`(含主键变更级联、级联环路保护)
|
- **外键级联** — `ON DELETE` / `ON UPDATE` 支持 `CASCADE` / `SET NULL` / `RESTRICT`(含主键变更级联、级联环路保护)
|
||||||
- **数据迁移** — 版本化迁移(持久化到库内,重启不重跑)、导入导出、在线一致性备份
|
- **数据迁移** — 版本化迁移(持久化到库内,重启不重跑)、单表/全库导入导出、`backup()` 全库导出(逐表读取,非跨表快照 —— 见「已知限制」)
|
||||||
- **连接池** — `MetonaSqlark.connect()` 单例复用,引用计数自动关闭
|
- **连接池** — `MetonaSqlark.connect()` 单例复用,引用计数自动关闭
|
||||||
|
|
||||||
**存储引擎(5 种)**
|
**存储模式(4 种)+ 存储后端(3 种)**
|
||||||
|
|
||||||
- `memory` — 纯内存,测试 / 缓存
|
- `memory` — 纯内存,测试 / 缓存
|
||||||
- `disk` — 自研 **KVStore 事务引擎**(v0.6.0 起替代 IndexedDB):多 key 原子写、快照 + 日志崩溃恢复
|
- `disk` — 自研 **KVStore 事务引擎**(v0.6.0 起替代 IndexedDB):多 key 原子写、快照 + 日志崩溃恢复
|
||||||
- `hybrid` — write-through 双写,读走内存
|
- `hybrid` — write-through 双写,读走内存
|
||||||
- `aria` — 自研 **AriaEngine**(LSM-Tree + WAL + MVCC),后端可选 OPFS / KVStore / Memory
|
- `aria` — 自研 **AriaEngine**(LSM-Tree + WAL + 页面化 + 可选压缩/加密),
|
||||||
- **KVStore** — 日志结构化事务 KV 引擎,可独立用作 aria 后端
|
后端可选 `diskEngine: 'opfs' | 'kv' | 'memory'`
|
||||||
|
|
||||||
|
> `disk` 与 `hybrid` 的磁盘侧**恒用自研 KVStore**(`diskEngine` 项只对 `mode: 'aria'`
|
||||||
|
> 生效,其余模式忽略该配置 —— 已在类型注释中写明)。
|
||||||
|
|
||||||
**生产级可靠性**
|
**生产级可靠性**
|
||||||
|
|
||||||
- **崩溃恢复** — WAL 原子写入 + CRC-32 完整性校验(SSTable 整文件 + WAL 记录)、空洞检测截断、打开时损坏自愈、`repair()` 清理重建
|
- **崩溃恢复** — WAL 原子写入 + CRC-32 完整性校验(SSTable 整文件 + WAL 记录)、分片空洞与记录损坏**如实上报**(`ARIA_WAL_GAP` / `droppedWALRecords`,拒绝静默截断)、打开时损坏自愈、`repair()` 清理重建
|
||||||
- **全库加密** — AES-256-GCM 透明加密(PBKDF2 密钥派生 + 密码验证 + 篡改检测),`encryption.password` 一键启用
|
- **全库加密** — AES-256-GCM 透明加密(PBKDF2 密钥派生 + 密码验证 + 篡改检测),`encryption.password` 一键启用
|
||||||
- **多标签页独占锁** — Web Locks API,第二个标签页打开同一库抛 `ARIA_LOCKED`
|
- **多标签页独占锁** — Web Locks API,第二个标签页打开同一库抛 `ARIA_LOCKED`
|
||||||
- **数据安全** — 输入校验(`required` / `maxLength` / `min` / `max`)、SQL 注入防护、Bloom Filter 快速否定
|
- **数据安全** — 输入校验(`required` / `maxLength` / `min` / `max`)、SQL 注入防护、Bloom Filter 快速否定
|
||||||
@@ -58,8 +63,11 @@
|
|||||||
|
|
||||||
**生态**
|
**生态**
|
||||||
|
|
||||||
- **React / Vue 集成** — `useQuery` / `useSqlarkQuery` 等开箱即用 hooks
|
- **React / Vue 集成** — `useQuery` / `useSqlarkQuery` 等 hooks(构建为 `dist/react.js` /
|
||||||
- **插件系统** — 14 种生命周期钩子(beforeInsert / afterQuery / ...),按优先级注册
|
`dist/vue.js`,配套手写精确类型声明;`react` / `vue` 为**可选 peer dependency**,
|
||||||
|
不装也不影响核心库使用)
|
||||||
|
- **插件系统** — 14 种生命周期钩子(beforeInsert / afterQuery / ...),
|
||||||
|
`priority` 越大越先执行(同优先级保持注册顺序)
|
||||||
- **旧库迁移** — `migrateFromIndexedDB()` 一键把旧 IndexedDB 库导入新引擎
|
- **旧库迁移** — `migrateFromIndexedDB()` 一键把旧 IndexedDB 库导入新引擎
|
||||||
- **浏览器兼容** — Chrome 102+ / Firefox 111+ / Safari 15.2+ / Edge 102+ / Node.js 16+
|
- **浏览器兼容** — Chrome 102+ / Firefox 111+ / Safari 15.2+ / Edge 102+ / Node.js 16+
|
||||||
|
|
||||||
@@ -83,7 +91,7 @@ npm install @metona-team/metona-sqlark
|
|||||||
<script src="https://git.metona.cn/MetonaTeam/MetonaSqlark/raw/branch/master/dist/metona-sqlark.min.js"></script>
|
<script src="https://git.metona.cn/MetonaTeam/MetonaSqlark/raw/branch/master/dist/metona-sqlark.min.js"></script>
|
||||||
```
|
```
|
||||||
|
|
||||||
或从 [`dist/`](./dist/) 下载:`metona-sqlark.js`(UMD 开发版)/ `metona-sqlark.min.js`(压缩版,gzip ~27KB)/ `metona-sqlark.esm.js` / `metona-sqlark.cjs` / `metona-sqlark.d.ts`
|
或从 [`dist/`](./dist/) 下载:`metona-sqlark.js`(UMD 开发版)/ `metona-sqlark.min.js`(压缩版:实测 251,731 字节 / gzip 63,431 字节)/ `metona-sqlark.esm.js` / `metona-sqlark.cjs` / `metona-sqlark.d.ts`
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -96,7 +104,8 @@ import { MetonaSqlark } from '@metona-team/metona-sqlark';
|
|||||||
const db = await MetonaSqlark.create({
|
const db = await MetonaSqlark.create({
|
||||||
name: 'my-app',
|
name: 'my-app',
|
||||||
mode: 'hybrid', // 'memory' | 'disk' | 'hybrid' | 'aria'
|
mode: 'hybrid', // 'memory' | 'disk' | 'hybrid' | 'aria'
|
||||||
diskEngine: 'opfs', // 'opfs' | 'kv'(自研 KVStore 后端)| 'memory'
|
// 仅 mode:'aria' 时生效(Aria 的存储后端);disk/hybrid 恒用自研 KVStore
|
||||||
|
diskEngine: 'opfs', // 'opfs' | 'kv' | 'memory'
|
||||||
});
|
});
|
||||||
|
|
||||||
// 定义表 — 支持约束与外键级联
|
// 定义表 — 支持约束与外键级联
|
||||||
@@ -178,7 +187,7 @@ await db2.disconnect(); // 引用计数 -1
|
|||||||
|------|------|------|------|
|
|------|------|------|------|
|
||||||
| `name` | `string` | `'metona-sqlark'` | 数据库名称 |
|
| `name` | `string` | `'metona-sqlark'` | 数据库名称 |
|
||||||
| `mode` | `'memory' \| 'disk' \| 'hybrid' \| 'aria'` | `'hybrid'` | 存储模式 |
|
| `mode` | `'memory' \| 'disk' \| 'hybrid' \| 'aria'` | `'hybrid'` | 存储模式 |
|
||||||
| `diskEngine` | `'opfs' \| 'memory' \| 'kv'` | `'opfs'` | 磁盘引擎(aria 模式下为存储后端;`'kv'` = 自研 KVStore) |
|
| `diskEngine` | `'opfs' \| 'memory' \| 'kv'` | `'opfs'` | **仅 `mode: 'aria'` 生效**(Aria 的存储后端;`'kv'` = 自研 KVStore)。`disk`/`hybrid` 恒用 KVStore,忽略此项 |
|
||||||
| `version` | `number` | `1` | 版本号 |
|
| `version` | `number` | `1` | 版本号 |
|
||||||
| `maxRowsPerQuery` | `number` | `0` | 查询结果行数上限(0 = 不限) |
|
| `maxRowsPerQuery` | `number` | `0` | 查询结果行数上限(0 = 不限) |
|
||||||
| `debug` | `boolean` | `false` | 调试模式 |
|
| `debug` | `boolean` | `false` | 调试模式 |
|
||||||
@@ -227,7 +236,7 @@ await db2.disconnect(); // 引用计数 -1
|
|||||||
| `db.clearAll()` | 清空全部数据与表结构(保留库本身) |
|
| `db.clearAll()` | 清空全部数据与表结构(保留库本身) |
|
||||||
| `db.exportTable(name)` / `db.exportAll()` | 导出数据 JSON |
|
| `db.exportTable(name)` / `db.exportAll()` | 导出数据 JSON |
|
||||||
| `db.importTable(name, data)` | 导入数据 |
|
| `db.importTable(name, data)` | 导入数据 |
|
||||||
| `db.backup()` | 在线备份:全库一致性快照 |
|
| `db.backup()` | 导出全库数据(**逐表读取**,见下方"已知限制") |
|
||||||
| `db.addMigration(v, fn)` / `db.migrateTo(v)` | 版本化数据迁移(版本持久化,重启不重跑) |
|
| `db.addMigration(v, fn)` / `db.migrateTo(v)` | 版本化数据迁移(版本持久化,重启不重跑) |
|
||||||
| `db.subscribe(table, fn)` | 订阅表变更(返回退订函数) |
|
| `db.subscribe(table, fn)` | 订阅表变更(返回退订函数) |
|
||||||
| `db.on(hook, fn)` | 注册生命周期钩子(14 种) |
|
| `db.on(hook, fn)` | 注册生命周期钩子(14 种) |
|
||||||
@@ -240,7 +249,7 @@ await db2.disconnect(); // 引用计数 -1
|
|||||||
| `EXPLAIN SELECT ...` | 输出查询计划(type/table/where/usingIndex/estimatedRows/actualTimeMs) | 全部 |
|
| `EXPLAIN SELECT ...` | 输出查询计划(type/table/where/usingIndex/estimatedRows/actualTimeMs) | 全部 |
|
||||||
| `ANALYZE [TABLE] name` | 收集表统计信息(行数/行大小/索引深度/列基数) | Aria |
|
| `ANALYZE [TABLE] name` | 收集表统计信息(行数/行大小/索引深度/列基数) | Aria |
|
||||||
| `REINDEX [TABLE] name` | 重建表二级索引 | Aria |
|
| `REINDEX [TABLE] name` | 重建表二级索引 | Aria |
|
||||||
| `VACUUM` | 压缩 LSM + 清理 MVCC 碎片 | Aria |
|
| `VACUUM` | 压缩 LSM + 清理版本碎片 | Aria |
|
||||||
| `SAVEPOINT name` / `ROLLBACK TO name` / `RELEASE name` | 嵌套事务保存点 | Aria |
|
| `SAVEPOINT name` / `ROLLBACK TO name` / `RELEASE name` | 嵌套事务保存点 | Aria |
|
||||||
|
|
||||||
> 不支持的引擎执行维护语句抛 `NOT_SUPPORTED`。
|
> 不支持的引擎执行维护语句抛 `NOT_SUPPORTED`。
|
||||||
@@ -250,7 +259,9 @@ await db2.disconnect(); // 引用计数 -1
|
|||||||
> IndexedDB 已从引擎中完全移除。旧版本(v0.5.x 及更早)的 disk 模式用户可通过一次性迁移工具导入:
|
> IndexedDB 已从引擎中完全移除。旧版本(v0.5.x 及更早)的 disk 模式用户可通过一次性迁移工具导入:
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
|
// 两种等价写法(v0.8.0 起主入口也导出,避免深路径依赖)
|
||||||
import { migrateFromIndexedDB } from '@metona-team/metona-sqlark/migration';
|
import { migrateFromIndexedDB } from '@metona-team/metona-sqlark/migration';
|
||||||
|
// import { migrateFromIndexedDB } from '@metona-team/metona-sqlark';
|
||||||
|
|
||||||
const target = await MetonaSqlark.create({ name: 'my-app-new', mode: 'disk' });
|
const target = await MetonaSqlark.create({ name: 'my-app-new', mode: 'disk' });
|
||||||
const result = await migrateFromIndexedDB({
|
const result = await migrateFromIndexedDB({
|
||||||
@@ -273,6 +284,12 @@ const result = await migrateFromIndexedDB({
|
|||||||
| `MetonaSqlark.disconnectAll()` | 强制关闭所有连接 |
|
| `MetonaSqlark.disconnectAll()` | 强制关闭所有连接 |
|
||||||
| `MetonaSqlark.getActiveConnections()` | 获取活跃连接列表 |
|
| `MetonaSqlark.getActiveConnections()` | 获取活跃连接列表 |
|
||||||
|
|
||||||
|
> **实现说明**:这四个静态方法由 `connection-manager` 模块在**模块加载时注入**
|
||||||
|
> (主入口 `src/index.ts` 以 side-effect 方式 `import './connection-manager'`),
|
||||||
|
> 因此从包入口引入即可用,无需额外操作。类型上(v0.8.0 起)声明为可选静态成员;
|
||||||
|
> 若在**未加载该模块**的自定义构建里调用,其值为 `undefined` 并抛 `TypeError`
|
||||||
|
> —— 不会静默无效。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 存储引擎
|
## 存储引擎
|
||||||
@@ -280,7 +297,7 @@ const result = await migrateFromIndexedDB({
|
|||||||
| 特性 | Memory | Disk (KVStore) | Hybrid | Aria |
|
| 特性 | Memory | Disk (KVStore) | Hybrid | Aria |
|
||||||
|------|--------|----------------|--------|------|
|
|------|--------|----------------|--------|------|
|
||||||
| **持久化** | ❌ 重启丢失 | ✅ KVStore(OPFS / 内存介质) | ✅ 内存 + 磁盘 | ✅ 后端决定 |
|
| **持久化** | ❌ 重启丢失 | ✅ KVStore(OPFS / 内存介质) | ✅ 内存 + 磁盘 | ✅ 后端决定 |
|
||||||
| **事务** | ✅ 快照回滚 | ✅ 单日志记录原子写 | ✅ 双引擎(磁盘优先) | ✅ MVCC 快照隔离 |
|
| **事务** | ✅ 快照回滚 | ✅ 单日志记录原子写 | ✅ 双引擎(磁盘优先) | ✅ 快照回滚(事务串行,非 MVCC 隔离) |
|
||||||
| **二级索引** | ✅ Hash | ✅ Hash(重启恢复) | ✅ Hash | ✅ LSM(重启恢复) |
|
| **二级索引** | ✅ Hash | ✅ Hash(重启恢复) | ✅ Hash | ✅ LSM(重启恢复) |
|
||||||
| **查询性能** | O(1) PK | O(1) PK(内存热路径) | O(1) PK | O(log n) |
|
| **查询性能** | O(1) PK | O(1) PK(内存热路径) | O(1) PK | O(log n) |
|
||||||
| **数据上限** | 内存 | 磁盘可用 | 磁盘可用 | 内存 |
|
| **数据上限** | 内存 | 磁盘可用 | 磁盘可用 | 内存 |
|
||||||
@@ -294,14 +311,14 @@ const result = await migrateFromIndexedDB({
|
|||||||
- **临时数据 / 单元测试** → `memory`
|
- **临时数据 / 单元测试** → `memory`
|
||||||
- **标准前端持久化**(替代 IndexedDB)→ `disk`(KVStore,多 key 原子事务,10 万级验证)
|
- **标准前端持久化**(替代 IndexedDB)→ `disk`(KVStore,多 key 原子事务,10 万级验证)
|
||||||
- **内存速度 + 磁盘持久化** → `hybrid`(write-through,读走内存)
|
- **内存速度 + 磁盘持久化** → `hybrid`(write-through,读走内存)
|
||||||
- **大规模 / 需要自研引擎可控性** → `aria`(LSM-Tree + WAL + MVCC + 加密 + 页面化存储)
|
- **大规模 / 需要自研引擎可控性** → `aria`(LSM-Tree + WAL + 加密 + 页面化存储)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## AriaEngine 自研存储引擎
|
## AriaEngine 自研存储引擎
|
||||||
|
|
||||||
AriaEngine 是内置的页面式存储引擎,对标 SQLite 的设计理念:
|
AriaEngine 是内置的页面式存储引擎,对标 SQLite 的设计理念:
|
||||||
**LSM-Tree 索引 + WAL 崩溃恢复 + MVCC 事务 + 页面化物理存储 + 全库加密**。
|
**LSM-Tree 索引 + WAL 崩溃恢复 + 快照回滚事务 + 页面化物理存储 + 全库加密**。
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
const db = await MetonaSqlark.create({
|
const db = await MetonaSqlark.create({
|
||||||
@@ -328,17 +345,19 @@ const rows = await db.query('SELECT * FROM users');
|
|||||||
|
|
||||||
```
|
```
|
||||||
┌──────────────────────────────────────────────┐
|
┌──────────────────────────────────────────────┐
|
||||||
│ AriaEngine v0.6.1 │
|
│ AriaEngine v0.8.0 │
|
||||||
├──────────────────────────────────────────────┤
|
├──────────────────────────────────────────────┤
|
||||||
│ LSM-Tree │ Buffer Pool │ WAL │
|
│ LSM-Tree │ Buffer Pool │ WAL │
|
||||||
│ MemTable │ LRU (256页) │ 分片文件 │
|
│ MemTable │ LRU (256页) │ 分片文件 │
|
||||||
│ +SSTable │ +FileManager │ +CRC-32 │
|
│ +SSTable │ +FileManager │ +CRC-32 │
|
||||||
│ (4KB 页面) │ │ +空洞检测 │
|
│ (4KB 页面) │ │ +空洞检测 │
|
||||||
├────────────────┼───────────────┼────────────┤
|
├────────────────┼───────────────┼────────────┤
|
||||||
│ MVCC 事务 │ Bloom Filter │ LZ4 压缩 │
|
│ 事务/版本管理 │ Bloom Filter │ LZ4 压缩 │
|
||||||
│ 快照隔离 │ 二级索引 LSM │ +大小头 │
|
│ 快照回滚 │ 二级索引 LSM │ +大小头 │
|
||||||
│ +Savepoint │ (每列独立) │ │
|
│ +Savepoint │ (每列独立) │ │
|
||||||
├────────────────┴───────────────┴────────────┤
|
├────────────────┴───────────────┴────────────┤
|
||||||
|
│ __aria_manifest 单一提交点(数据→提交→截断)│
|
||||||
|
├──────────────────────────────────────────────┤
|
||||||
│ EncryptedBackend (AES-256-GCM 全库透明加密) │
|
│ EncryptedBackend (AES-256-GCM 全库透明加密) │
|
||||||
├──────────────────────────────────────────────┤
|
├──────────────────────────────────────────────┤
|
||||||
│ 存储后端: OPFS Backend / KVStore Backend │
|
│ 存储后端: OPFS Backend / KVStore Backend │
|
||||||
@@ -352,14 +371,16 @@ const rows = await db.query('SELECT * FROM users');
|
|||||||
|------|------|
|
|------|------|
|
||||||
| **LSM-Tree** | MemTable(红黑树)→ 多级 SSTable,异步 Compaction(从存储兜底加载),写背压 |
|
| **LSM-Tree** | MemTable(红黑树)→ 多级 SSTable,异步 Compaction(从存储兜底加载),写背压 |
|
||||||
| **页面化存储** | SSTable 存为 4KB 页面(FileManager 分配 pageId + BufferPool LRU 缓存 256 页 ≈ 1MB),`pageStorage` 在 opfs/kv 后端默认启用 |
|
| **页面化存储** | SSTable 存为 4KB 页面(FileManager 分配 pageId + BufferPool LRU 缓存 256 页 ≈ 1MB),`pageStorage` 在 opfs/kv 后端默认启用 |
|
||||||
| **WAL** | 分片文件 `__wal_%06d.bin` + 真追加;标准 CRC32 记录校验;full/batch/none 三模式;空洞检测截断;16MB 阈值自动 checkpoint(活跃事务期间不截断) |
|
| **WAL** | 分片文件 `__wal_%06d.bin` + 真追加;标准 CRC32 记录校验(记录级 CRC 失败会计数并上报 `droppedWALRecords`);full/batch/none 三模式;**LSN 全库单调**(manifest 记高水位);分片号**绝不回退、也绝不低于 manifest 水位**(整体清空后允许复用最后用过的号);内部空洞与记录损坏显式上报(`gaps` / `corruptRecords`),水位从未推进时的前缀缺失同样按空洞上报,水位已推进时前缀缺失视为"已清理的前缀";16MB 阈值自动 checkpoint(活跃事务期间不截断) |
|
||||||
| **崩溃恢复** | 打开时完整性校验(损坏 SSTable 自愈清理 + 整文件 CRC-32)、WAL 恢复、恢复后自动重建二级索引;`repair()` 清理孤儿页面与残留 |
|
| **单一提交点** | `__aria_manifest_<gen>`:页面水位 + 各命名空间 SSTable 元数据 + 表结构 + WAL 起始位置 + 待落盘冻结表意图,一次原子提交(头部/载荷双 CRC,先写后验,保留两代)。顺序固定为**数据落盘 → manifest 提交 → 才允许截断 WAL / 删除旧文件**;恢复只认最后一份 CRC 通过的世代,元数据损坏抛 `ARIA_MANIFEST_CORRUPT`(不再静默当空库) |
|
||||||
|
| **崩溃恢复** | 打开时完整性校验(整文件 CRC-32;**介质读故障不再被当成"文件不存在"**,抛 `ARIA_SSTABLE_READ_FAILED` 且不误删元数据)、按 LSN 水位重放 WAL、恢复后自动重建二级索引;`getRecoveryReport()` 返回 `{droppedSSTables, dataLossSuspected, walGaps, droppedWALRecords, legacyImported, manifestFallback}`;`repair()` 只在 manifest 健康时回收孤儿页面 |
|
||||||
|
| **Compaction** | 整层合并不再"先摘层再合并"(合并期间该层对读者始终可见);底部层原地合并**回收墓碑**(删除密集场景空间不再无界增长);按层 `compacting` 集合(跨层触发不丢失);被取代的 SSTable 进入**退休表**,等更早的读者退出后才物理删除 |
|
||||||
| **全库加密** | `encryption.password` → EncryptedBackend 透明加解密(WAL/SSTable/Schema/元数据全密文);PBKDF2 派生 + salt 持久化;密码错误/篡改 → `ARIA_DECRYPT_ERROR` |
|
| **全库加密** | `encryption.password` → EncryptedBackend 透明加解密(WAL/SSTable/Schema/元数据全密文);PBKDF2 派生 + salt 持久化;密码错误/篡改 → `ARIA_DECRYPT_ERROR` |
|
||||||
| **MVCC** | 版本链 + 快照隔离,事务读写不互斥,自动 GC |
|
| **MVCC** | 版本链仅作事务内 undo(提交即清理,**无快照隔离**;事务串行);自动 GC |
|
||||||
| **二级索引** | 每列独立 LSM Tree,支持等值/范围扫描,跨重启恢复,WAL 恢复后自动重建 |
|
| **二级索引** | 每列独立 LSM Tree,支持等值/范围扫描,跨重启恢复,WAL 恢复后自动重建 |
|
||||||
| **Bloom Filter** | FNV-1a + Murmur 双哈希,SSTable footer 序列化,查询时快速否定 |
|
| **Bloom Filter** | FNV-1a + Murmur 双哈希,SSTable footer 序列化,查询时快速否定 |
|
||||||
| **多标签页锁** | Web Locks 库级独占锁,第二个标签页抛 `ARIA_LOCKED`;不支持的环境降级无锁并告警 |
|
| **多标签页锁** | Web Locks 库级独占锁,第二个标签页抛 `ARIA_LOCKED`;不支持的环境降级无锁并告警 |
|
||||||
| **维护语句** | ANALYZE(表统计)/ REINDEX(重建索引)/ VACUUM(压缩 + MVCC GC)/ EXPLAIN(查询计划) |
|
| **维护语句** | ANALYZE(表统计)/ REINDEX(重建索引)/ VACUUM(压缩 + 版本 GC)/ EXPLAIN(查询计划) |
|
||||||
|
|
||||||
### AriaEngine 配置项
|
### AriaEngine 配置项
|
||||||
|
|
||||||
@@ -370,6 +391,7 @@ const rows = await db.query('SELECT * FROM users');
|
|||||||
| `memtableSizeThreshold` | `number` | `4MB` | MemTable 刷盘阈值 |
|
| `memtableSizeThreshold` | `number` | `4MB` | MemTable 刷盘阈值 |
|
||||||
| `levelSizeMultiplier` | `number` | `10` | LSM 层级容量倍数 |
|
| `levelSizeMultiplier` | `number` | `10` | LSM 层级容量倍数 |
|
||||||
| `bloomFilterBitsPerKey` | `number` | `10` | Bloom Filter 每 key 位数 |
|
| `bloomFilterBitsPerKey` | `number` | `10` | Bloom Filter 每 key 位数 |
|
||||||
|
| `maxMemoryMB` | `number` | `64` | 内存预算(MB):主 LSM 估算内存超限时触发 flush + MVCC GC |
|
||||||
| `walEnabled` | `boolean` | `true` | 是否启用 WAL |
|
| `walEnabled` | `boolean` | `true` | 是否启用 WAL |
|
||||||
| `walSyncMode` | `'full' \| 'batch' \| 'none'` | `'full'` | WAL 同步模式 |
|
| `walSyncMode` | `'full' \| 'batch' \| 'none'` | `'full'` | WAL 同步模式 |
|
||||||
| `checkpointInterval` | `number` | `1000` | Checkpoint 间隔(操作数) |
|
| `checkpointInterval` | `number` | `1000` | Checkpoint 间隔(操作数) |
|
||||||
@@ -401,7 +423,8 @@ const { data, loading, error, refresh } = useSqlarkQuery(db, 'SELECT * FROM user
|
|||||||
npm install # 安装依赖
|
npm install # 安装依赖
|
||||||
npm run dev # 开发模式(localhost:3001)
|
npm run dev # 开发模式(localhost:3001)
|
||||||
npm run build # 生产构建(生成 dist/)
|
npm run build # 生产构建(生成 dist/)
|
||||||
npm test # 运行测试(1256 用例 · 75 套件)
|
npm test # 运行测试(1985 用例 · 93 套件;+4 个重型套件)
|
||||||
|
python3 scripts/mutation-b6.py # 变异验证:把 B-6 的修复逐项回退,对应用例必须失败
|
||||||
npm run test:e2e # Playwright e2e(真实 Chromium + OPFS + 崩溃注入,需先 build)
|
npm run test:e2e # Playwright e2e(真实 Chromium + OPFS + 崩溃注入,需先 build)
|
||||||
npm run lint # 代码检查
|
npm run lint # 代码检查
|
||||||
npm run typecheck # 类型检查
|
npm run typecheck # 类型检查
|
||||||
@@ -413,13 +436,72 @@ npm run typecheck # 类型检查
|
|||||||
|
|
||||||
| 指标 | 数值 |
|
| 指标 | 数值 |
|
||||||
|------|------|
|
|------|------|
|
||||||
| 测试用例 | 1256(+12 Playwright e2e) |
|
| 测试用例 | 1985(93 套件)+ 14 Playwright e2e,另 4 个重型套件在独立 CI job 串行运行 |
|
||||||
| 测试套件 | 75 |
|
| 语句覆盖率 | 90.59%(8538/9424) |
|
||||||
| 行覆盖率 | 89.8% |
|
| 分支覆盖率 | 82.61%(4459/5397) |
|
||||||
|
| 函数覆盖率 | 94.14%(1223/1299) |
|
||||||
|
| 行覆盖率 | 93.50%(7733/8270) |
|
||||||
| SQL 关键字 | 72 |
|
| SQL 关键字 | 72 |
|
||||||
| 存储引擎 | 5(Memory / KVStore / OPFS / Hybrid / Aria) |
|
| 存储模式 | 4(`memory` / `disk` / `hybrid` / `aria`) |
|
||||||
|
| 存储后端 | 3(OPFS / KVStore / Memory),Aria 引擎另有 LSM-Tree + WAL + 页面化 |
|
||||||
| 运行时依赖 | 0 |
|
| 运行时依赖 | 0 |
|
||||||
|
|
||||||
|
> **覆盖率口径**:`collectCoverageFrom = src/**/*.ts`,仅排除两个**纯类型声明**文件
|
||||||
|
> (`engine/interface.ts`、`query/ast.ts` —— 它们只有 interface/type,可执行语句为 0,
|
||||||
|
> 纳入统计只会稀释分母)。v0.8.0 审查曾发现 `interface.ts` 里混着三个运行时函数
|
||||||
|
> (`cloneRow`/`cloneRowFallback`/`cloneRows`)—— 已搬到 `src/engine/row_clone.ts`
|
||||||
|
> 并纳入统计(搬完门禁立刻因 functions 93.84% < 94% 失败,补测退化路径后通过)。
|
||||||
|
> CI 常规 job 带 `--coverage` 运行,
|
||||||
|
> `jest.config.cjs` 的 `coverageThreshold` 为 statements 90 / branches 82 /
|
||||||
|
> functions 94 / lines 93,任一项不达标即失败 —— **门槛不达标不允许发版**。
|
||||||
|
>
|
||||||
|
> 上述四个数字由**与 CI 常规 job 完全相同的命令**产出(可复现):
|
||||||
|
> ```bash
|
||||||
|
> npx jest --coverage --testPathIgnorePatterns='/node_modules/|/tests/e2e/|aria-prod-load|kvstore-stress|aria-matrix-audit|aria-idx-flush-race'
|
||||||
|
> ```
|
||||||
|
> 修改覆盖范围、阈值或测试选择时**必须同步更新本表**(G5 门禁要求
|
||||||
|
> README 数字与 CI 产出一致)。
|
||||||
|
|
||||||
|
### 已知限制(v0.8.0)
|
||||||
|
|
||||||
|
- **存储布局在 v0.8.0 变更** — 元数据从"每个命名空间一份裸 JSON"(`__aria_lsm_meta*` /
|
||||||
|
`__aria_schemas`)收敛为 `__aria_manifest_<generation>`(带世代号与双 CRC)。
|
||||||
|
旧库**首次用 v0.8.0 打开时自动迁移**(旧键保留不删),迁移遇到损坏的旧元数据会
|
||||||
|
明确报 `ARIA_LEGACY_META_CORRUPT` 而不是当成空库。直接读取这些内部 key 的外部
|
||||||
|
脚本需要跟着改(引擎侧无公开 API 依赖它们)。
|
||||||
|
* **回退是单向的**:迁移后所有新写入只进 manifest,旧的 `__aria_lsm_meta*` /
|
||||||
|
`__aria_schemas` 停留在迁移那一刻。用旧版本打开同一个库会看到**迁移时刻的旧
|
||||||
|
视图**(不是"数据都在"),继续写入还会让两套布局分叉 —— 需要回退旧版本时,
|
||||||
|
先用 v0.8.0 导出数据,不要指望旧键是新数据的镜像。
|
||||||
|
- **单列主键** — 复合主键暂不支持(建表时显式 `SCHEMA_ERROR`),列入 v0.8 路线图
|
||||||
|
- **写语句关联引用** — UPDATE/DELETE 的 WHERE 支持非关联子查询(`IN (SELECT)` / 标量子查询),关联引用(`$col` / 关联 EXISTS)显式抛 `NOT_SUPPORTED`(不静默)
|
||||||
|
- **`backup()` 不是跨表一致性快照** — 实现为**逐表读取**(Aria 走引擎级 `backup()`,
|
||||||
|
其余引擎回退 `exportAll()`)。若在备份过程中有并发写入,不同表之间可能来自
|
||||||
|
不同时间点(单表内部是一致的)。需要强一致备份时请先 `close()` 或用
|
||||||
|
`db.transaction()` 包住调用(事务期间并发写会被 `TX_ACTIVE` 拒绝)。
|
||||||
|
* 说明:此前文档宣称"在线一致性快照",但引擎层没有实现跨表快照原语 ——
|
||||||
|
与其保留一个不成立的宣称,这里按实际行为描述(真快照列入后续版本)。
|
||||||
|
- **建表 UNIQUE 约束** — 不可经 `DROP INDEX` 解除(对齐 SQLite,需重建表);仅 `CREATE UNIQUE INDEX` 添加的约束可随索引删除
|
||||||
|
- **唯一值交换更新** — 同一语句内两行互换唯一列值(A:x→y, B:y→x)保守拒绝(最终状态合法但报 `UNIQUE_VIOLATION`)
|
||||||
|
- **主键非空** — 主键列强制非空(SQL 语义 PK 隐含 NOT NULL),`INSERT`/`UPDATE` 置 null/undefined 抛 `VALIDATION_ERROR`
|
||||||
|
- **多实例写入保护依赖 Web Locks** — AriaEngine 用 Web Locks 做库级独占:第二个实例
|
||||||
|
打开同一个库时抛 `ARIA_LOCKED`,这是**唯一受支持**的多标签页写入方式。运行环境
|
||||||
|
没有 Web Locks 时该保护会自动降级为"提交点冲突检测"(manifest 世代号单调 + 提交前
|
||||||
|
检查是否存在别的实例提交的更新世代 → 抛 `STALE_INSTANCE`),但降级只保证
|
||||||
|
**不静默覆盖别人的提交**,不保证多实例写入的数据完整性:被拒绝的那一方此前
|
||||||
|
已写入自己 WAL 分片的记录,可能被胜出实例的 checkpoint 当作可回收前缀清掉。
|
||||||
|
结论:**不要在没有 Web Locks 的环境里让两个实例同时写同一个库**;需要并发访问时
|
||||||
|
由应用层串行化(如 SharedWorker / 主标签页代理)。
|
||||||
|
- **manifest 体积随 SSTable 数量增长** — 单一提交点把全部命名空间的 SSTable 元数据
|
||||||
|
(键范围、页面 id 列表、大小)+ 表结构 + WAL 水位写进**同一个文件**,每次提交
|
||||||
|
整体重写(保留两代)。因此元数据量与已落盘 SSTable 数成正比:长期高频写入、
|
||||||
|
层级很多且迟迟不合并的库,其 manifest 会明显大于数据本身之外的一般预期。当前
|
||||||
|
没有"元数据分层/增量"机制,`vacuum()` 合并层级是唯一的收敛手段(列入后续版本)。
|
||||||
|
- **尾部 WAL 分片丢失无法从介质自身识别** — 分片内部空洞(中间缺号)与记录级 CRC
|
||||||
|
损坏都会被上报;但如果**最后一个**分片整个消失,介质上没有任何"它本该存在"的证据
|
||||||
|
(manifest 只记 `startSegment`/`nextLsn`,不记最后分片号),此时只能靠
|
||||||
|
`ARIA_WRITE_LOST`(有未落盘冻结表却重放不到任何记录)兜住"确定丢数据"的情况。
|
||||||
|
|
||||||
### 浏览器兼容性
|
### 浏览器兼容性
|
||||||
|
|
||||||
| 浏览器 | 最低版本 | Memory | Disk (KVStore) | Aria (OPFS) |
|
| 浏览器 | 最低版本 | Memory | Disk (KVStore) | Aria (OPFS) |
|
||||||
@@ -430,9 +512,17 @@ npm run typecheck # 类型检查
|
|||||||
| Node.js | 16+ | ✅ | ✅(内存介质) | ✅(内存介质) |
|
| Node.js | 16+ | ✅ | ✅(内存介质) | ✅(内存介质) |
|
||||||
|
|
||||||
> **OPFS**:基础 API(`createWritable` 原子写)在 Chromium 102+ / Firefox 111+ / Safari 15.2+ 均支持。
|
> **OPFS**:基础 API(`createWritable` 原子写)在 Chromium 102+ / Firefox 111+ / Safari 15.2+ 均支持。
|
||||||
> 无跨文件事务,AriaEngine 以 WAL 分片单文件原子写 + 空洞检测截断保证崩溃一致性。
|
> 无跨文件事务,AriaEngine 以 WAL 分片单文件原子写 + manifest 单一提交点保证崩溃一致性;
|
||||||
|
> 分片空洞与记录级损坏都会被**显式上报**(恢复报告 + 告警),不做静默截断。
|
||||||
>
|
>
|
||||||
> **多标签页保护**:AriaEngine 通过 Web Locks 获取库级独占锁,第二个标签页打开同一库抛 `ARIA_LOCKED`。
|
> **多标签页保护**:AriaEngine 通过 Web Locks 获取库级独占锁,第二个标签页打开同一库抛 `ARIA_LOCKED`。
|
||||||
|
>
|
||||||
|
> **OPFS 需要 http(s) 页面**:`aria` + `diskEngine: 'opfs'` 在 **`file://` 直接打开的页面里不可用** ——
|
||||||
|
> Chromium 会以 `SecurityError` 拒绝 `navigator.storage.getDirectory()`(注意此时
|
||||||
|
> `isSecureContext` 仍是 `true`、API 也存在,只有真正调用才会发现)。引擎会抛
|
||||||
|
> `ARIA_OPFS_UNAVAILABLE`,消息里给出两条出路:用本地服务器打开
|
||||||
|
> (`node tests/e2e/server.cjs 3344` → `http://127.0.0.1:3344/...`),或改用
|
||||||
|
> `mode: 'memory'` / KVStore 后端。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
Vendored
+8175
-1606
File diff suppressed because it is too large
Load Diff
Vendored
+1
-1
File diff suppressed because one or more lines are too long
Vendored
+1056
-92
File diff suppressed because it is too large
Load Diff
Vendored
+8169
-1607
File diff suppressed because it is too large
Load Diff
Vendored
+1
-1
File diff suppressed because one or more lines are too long
Vendored
+8175
-1606
File diff suppressed because it is too large
Load Diff
Vendored
+1
-1
File diff suppressed because one or more lines are too long
Vendored
+1
-1
File diff suppressed because one or more lines are too long
Vendored
+19349
File diff suppressed because it is too large
Load Diff
Vendored
+1
File diff suppressed because one or more lines are too long
Vendored
+19347
File diff suppressed because it is too large
Load Diff
Vendored
+1
File diff suppressed because one or more lines are too long
+62
-32
@@ -1,32 +1,62 @@
|
|||||||
module.exports = {
|
module.exports = {
|
||||||
testEnvironment: 'jsdom',
|
testEnvironment: 'jsdom',
|
||||||
setupFiles: ['./jest.setup.js'],
|
setupFiles: ['./jest.setup.js'],
|
||||||
transform: {
|
transform: {
|
||||||
'^.+\\.ts$': 'babel-jest',
|
'^.+\\.ts$': 'babel-jest',
|
||||||
},
|
},
|
||||||
transformIgnorePatterns: [
|
transformIgnorePatterns: [
|
||||||
'/node_modules/(?!(@rollup)/)',
|
'/node_modules/(?!(@rollup)/)',
|
||||||
],
|
],
|
||||||
moduleFileExtensions: ['ts', 'js', 'json'],
|
moduleFileExtensions: ['ts', 'js', 'json'],
|
||||||
// Playwright e2e(tests/e2e/)与共享测试工具(tests/helpers/)不属于 jest 单元测试
|
// 默认 testMatch 不会发现 __tests__ 之外、名为 helpers/ 目录里的 .test.ts 文件,
|
||||||
testPathIgnorePatterns: [
|
// 因此显式声明匹配规则(v0.8.0:测试基座自身的自测放在 tests/helpers/ 下)。
|
||||||
'/node_modules/',
|
testMatch: ['**/*.test.ts'],
|
||||||
'/tests/e2e/',
|
// Playwright e2e(tests/e2e/)不属于 jest 单元测试
|
||||||
'/tests/helpers/',
|
testPathIgnorePatterns: [
|
||||||
],
|
'/node_modules/',
|
||||||
collectCoverageFrom: [
|
'/tests/e2e/',
|
||||||
'src/**/*.ts',
|
],
|
||||||
'!src/index.ts',
|
//
|
||||||
'!src/**/index.ts',
|
// v0.8.0 覆盖率口径修正(根治):
|
||||||
'!src/query/ast.ts',
|
//
|
||||||
'!src/engine/interface.ts',
|
// 此前为 `'!src/**/index.ts'` —— 这条 glob 把 15 个文件整体排除,其中包含
|
||||||
'!src/engine/opfs.ts',
|
// src/engine/aria/index.ts(AriaEngine 主实现 2282 行)、src/engine/kvstore/index.ts
|
||||||
'!src/integrations/**',
|
// (KVStore 本体)、src/hybrid/index.ts、src/migration/index.ts 等**实现文件**,
|
||||||
],
|
// 而非桶文件。后果:发布的"90.1% 行覆盖率"只在约 75% 的可执行源码上成立,
|
||||||
coverageDirectory: 'coverage',
|
// 且 v0.2.6 曾在 CHANGELOG 里承认过同类问题("91.0% 为排除 Aria 模块的陈旧数据")
|
||||||
coverageReporters: ['text', 'lcov'],
|
// 后又在 v0.5.1 以另一种写法复发。
|
||||||
verbose: true,
|
//
|
||||||
// Must have: fake-indexeddb + debounce timers keep event loop alive
|
// 现在只排除**确实没有可执行语句**的文件(纯类型声明),并且逐个显式列出、
|
||||||
forceExit: true,
|
// 附理由 —— 新增文件默认纳入统计,不会再被 glob 静默吞掉。
|
||||||
testTimeout: 15000,
|
//
|
||||||
};
|
collectCoverageFrom: [
|
||||||
|
'src/**/*.ts',
|
||||||
|
// 纯类型声明:可执行语句 0 行(接口/类型别名),纳入统计只会稀释分母
|
||||||
|
'!src/engine/interface.ts',
|
||||||
|
'!src/query/ast.ts',
|
||||||
|
],
|
||||||
|
coverageDirectory: 'coverage',
|
||||||
|
coverageReporters: ['text', 'lcov'],
|
||||||
|
//
|
||||||
|
// v0.8.0 覆盖率门禁(此前完全不存在:jest.config 与 CI 都没有任何阈值,
|
||||||
|
// 覆盖率从 90% 掉到 60% 也全绿)。
|
||||||
|
//
|
||||||
|
// 阈值 = 在**修正口径后**(不再排除实现文件)实测的真实基线,下取整到整数百分点。
|
||||||
|
// 历史基线(2026-08-15):Statements 90.66% / Branches 82.94% / Functions 94.36% / Lines 93.43%
|
||||||
|
// v0.8.0 收尾实测(2026-09-15,1985 用例):Statements 90.59% / Branches 82.61% /
|
||||||
|
// Functions 94.14% / Lines 93.50%
|
||||||
|
//
|
||||||
|
coverageThreshold: {
|
||||||
|
global: {
|
||||||
|
statements: 90,
|
||||||
|
branches: 82,
|
||||||
|
functions: 94,
|
||||||
|
lines: 93,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
verbose: true,
|
||||||
|
// 注意:forceExit 会掩盖句柄/定时器泄漏。v0.8.0 保留它以便常规开发快速迭代,
|
||||||
|
// 但新增了独立的 `npm run test:leaks`(--detectOpenHandles 且不 forceExit)用于泄漏检查。
|
||||||
|
forceExit: true,
|
||||||
|
testTimeout: 15000,
|
||||||
|
};
|
||||||
|
|||||||
+28
-8
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@metona-team/metona-sqlark",
|
"name": "@metona-team/metona-sqlark",
|
||||||
"version": "0.7.3",
|
"version": "0.8.0",
|
||||||
"description": "Frontend SQL database with in-memory and disk dual-mode storage",
|
"description": "Frontend SQL database with in-memory and disk dual-mode storage",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "dist/metona-sqlark.cjs",
|
"main": "dist/metona-sqlark.cjs",
|
||||||
@@ -14,13 +14,18 @@
|
|||||||
"require": "./dist/metona-sqlark.cjs",
|
"require": "./dist/metona-sqlark.cjs",
|
||||||
"types": "./dist/metona-sqlark.d.ts"
|
"types": "./dist/metona-sqlark.d.ts"
|
||||||
},
|
},
|
||||||
"./react": {
|
"./migration": {
|
||||||
"import": "./src/integrations/react.ts",
|
"import": "./dist/metona-sqlark.esm.js",
|
||||||
|
"require": "./dist/metona-sqlark.cjs",
|
||||||
"types": "./dist/metona-sqlark.d.ts"
|
"types": "./dist/metona-sqlark.d.ts"
|
||||||
},
|
},
|
||||||
|
"./react": {
|
||||||
|
"import": "./dist/react.js",
|
||||||
|
"types": "./src/integrations/react.d.ts"
|
||||||
|
},
|
||||||
"./vue": {
|
"./vue": {
|
||||||
"import": "./src/integrations/vue.ts",
|
"import": "./dist/vue.js",
|
||||||
"types": "./dist/metona-sqlark.d.ts"
|
"types": "./src/integrations/vue.d.ts"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"files": [
|
"files": [
|
||||||
@@ -35,12 +40,15 @@
|
|||||||
"test": "jest",
|
"test": "jest",
|
||||||
"test:coverage": "jest --coverage",
|
"test:coverage": "jest --coverage",
|
||||||
"test:watch": "jest --watch",
|
"test:watch": "jest --watch",
|
||||||
|
"test:leaks": "jest --detectOpenHandles --forceExit=false",
|
||||||
"test:e2e": "playwright test",
|
"test:e2e": "playwright test",
|
||||||
"lint": "eslint \"src/**/*.ts\"",
|
"lint": "eslint \"src/**/*.ts\" \"tests/**/*.ts\"",
|
||||||
"lint:fix": "eslint \"src/**/*.ts\" --fix",
|
"lint:fix": "eslint \"src/**/*.ts\" \"tests/**/*.ts\" --fix",
|
||||||
"format": "prettier --write \"src/**/*.ts\"",
|
"format": "prettier --write \"src/**/*.ts\"",
|
||||||
"typecheck": "tsc --noEmit",
|
"typecheck": "tsc --noEmit",
|
||||||
"prepublishOnly": "npm run typecheck && npm test && npm run build"
|
"typecheck:tests": "tsc -p tsconfig.test.json",
|
||||||
|
"verify": "npm run typecheck && npm run typecheck:tests && npm run lint && npm run test:coverage",
|
||||||
|
"prepublishOnly": "npm run typecheck && npm run typecheck:tests && npm test && npm run build"
|
||||||
},
|
},
|
||||||
"repository": {
|
"repository": {
|
||||||
"type": "git",
|
"type": "git",
|
||||||
@@ -62,6 +70,18 @@
|
|||||||
"url": "https://git.metona.cn/MetonaTeam/MetonaSqlark/issues"
|
"url": "https://git.metona.cn/MetonaTeam/MetonaSqlark/issues"
|
||||||
},
|
},
|
||||||
"homepage": "https://git.metona.cn/MetonaTeam/MetonaSqlark#readme",
|
"homepage": "https://git.metona.cn/MetonaTeam/MetonaSqlark#readme",
|
||||||
|
"peerDependencies": {
|
||||||
|
"react": ">=17",
|
||||||
|
"vue": ">=3"
|
||||||
|
},
|
||||||
|
"peerDependenciesMeta": {
|
||||||
|
"react": {
|
||||||
|
"optional": true
|
||||||
|
},
|
||||||
|
"vue": {
|
||||||
|
"optional": true
|
||||||
|
}
|
||||||
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
"@babel/core": "^7.22.0",
|
"@babel/core": "^7.22.0",
|
||||||
"@babel/plugin-transform-modules-commonjs": "^7.22.0",
|
"@babel/plugin-transform-modules-commonjs": "^7.22.0",
|
||||||
|
|||||||
@@ -104,5 +104,23 @@ export default [
|
|||||||
},
|
},
|
||||||
plugins: [dts()],
|
plugins: [dts()],
|
||||||
},
|
},
|
||||||
|
// v0.8.0: React / Vue 集成产物。
|
||||||
|
// 此前 package.json 的 "./react" / "./vue" 直接指向 **src/ 下的裸 TS 源码**,
|
||||||
|
// 而这两个文件带 `// @ts-nocheck` → 使用者拿到的是未编译代码 + 类型为 any
|
||||||
|
// 的声明("开箱即用 hooks"这一宣称不成立)。现在构建成真实 ESM 产物,
|
||||||
|
// 配套手写精确声明(见 src/integrations/*.d.ts 的说明)。
|
||||||
|
// react / vue 保持 external(peer dependency),不打包进产物。
|
||||||
|
{
|
||||||
|
input: 'src/integrations/react.ts',
|
||||||
|
output: { file: 'dist/react.js', format: 'es', sourcemap: true },
|
||||||
|
external: ['react'],
|
||||||
|
plugins: basePlugins,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
input: 'src/integrations/vue.ts',
|
||||||
|
output: { file: 'dist/vue.js', format: 'es', sourcemap: true },
|
||||||
|
external: ['vue'],
|
||||||
|
plugins: basePlugins,
|
||||||
|
},
|
||||||
]),
|
]),
|
||||||
];
|
];
|
||||||
|
|||||||
@@ -0,0 +1,624 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""B-6 变异验证:把每个修复回退到修复前的行为,对应用例必须失败。
|
||||||
|
|
||||||
|
用法:python3 scripts/mutation-b6.py
|
||||||
|
任何一条"回退后测试仍然通过"都会以非零退出码报出来(说明测试只是陪跑)。
|
||||||
|
"""
|
||||||
|
import hashlib
|
||||||
|
import io
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import shlex
|
||||||
|
import signal
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
|
||||||
|
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
LSM = 'src/engine/aria/index/lsm.ts'
|
||||||
|
MANIFEST = 'src/engine/aria/store/manifest.ts'
|
||||||
|
MERGE = 'src/engine/aria/index/merge_iterator.ts'
|
||||||
|
ENGINE = 'src/engine/aria/index.ts'
|
||||||
|
SEGSTORE = 'src/engine/aria/wal/segmented_store.ts'
|
||||||
|
CHECKPOINT = 'src/engine/aria/wal/checkpoint.ts'
|
||||||
|
SSTABLE = 'src/engine/aria/index/sstable.ts'
|
||||||
|
|
||||||
|
B6 = 'tests/v080-b6-single-commit-point.test.ts'
|
||||||
|
WALSEG = 'tests/engine/aria-wal-segment.test.ts'
|
||||||
|
STRESS = 'tests/engine/aria-repair-hardening.test.ts'
|
||||||
|
WALLOG = 'src/engine/aria/wal/log.ts'
|
||||||
|
|
||||||
|
# 单条变异的 jest 运行上限(秒)。超时视为"变异脚本自身的问题",必须报错而不是
|
||||||
|
# 悄悄当成"测试通过了"。
|
||||||
|
JEST_TIMEOUT_S = 300
|
||||||
|
# 正控(未变异的干净代码)必须跑通的套件:没有正控就无法区分
|
||||||
|
# "变异被测试拦住"与"这套件本来就是红的"。
|
||||||
|
CONTROL_SUITES = [B6, WALSEG]
|
||||||
|
|
||||||
|
MUTATIONS = [
|
||||||
|
dict(
|
||||||
|
name='47 MergeIterator 立即补充(提前终止多算 1 条)',
|
||||||
|
file=MERGE,
|
||||||
|
old=""" // 胜出来源的补充推迟到下一次 next()(消费者只取 N 条 → 底层只产出 N 条)
|
||||||
|
this.pendingRefill = first.sourceIndex;""",
|
||||||
|
new=""" // [MUTATION] 立即补充(修复前行为)
|
||||||
|
this.seedFromSource(first.sourceIndex);""",
|
||||||
|
test=B6, pattern='流式扫描提前终止不多算',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='49 compacting 单 boolean(跨层触发被丢弃)',
|
||||||
|
file=LSM,
|
||||||
|
old=""" private scheduleCompact(level: number): void {
|
||||||
|
if (level > MAX_LSM_LEVELS - 1 || this.compacting.has(level)) return;
|
||||||
|
this.compacting.add(level);""",
|
||||||
|
new=""" private scheduleCompact(level: number): void {
|
||||||
|
if (level > MAX_LSM_LEVELS - 1 || this.compacting.size > 0) return; // [MUTATION] 单 boolean 语义
|
||||||
|
this.compacting.add(level);""",
|
||||||
|
test=B6, pattern='按层 compaction 状态',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='50 compaction 先 splice 整层(窗口内该层不可见)',
|
||||||
|
file=LSM,
|
||||||
|
old=""" const selected = [...this.levels[level]];
|
||||||
|
const selectedIds = new Set(selected.map((m) => m.id));""",
|
||||||
|
new=""" const selected = this.levels[level].splice(0, this.levels[level].length); // [MUTATION] 旧行为
|
||||||
|
const selectedIds = new Set(selected.map((m) => m.id));""",
|
||||||
|
test=B6, pattern='compaction 期间该层对读者始终可见',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='51 底部层不回收墓碑',
|
||||||
|
file=LSM,
|
||||||
|
old=""" const merged = isBottomLevel
|
||||||
|
? mergedRaw.filter(([, value]) => !(value as unknown as Record<string, unknown>).__tombstone)
|
||||||
|
: mergedRaw;""",
|
||||||
|
new=""" const merged = mergedRaw; // [MUTATION] 不回收墓碑""",
|
||||||
|
test=B6, pattern='底部层合并回收墓碑',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='44 冻结表失败后不重试',
|
||||||
|
file=LSM,
|
||||||
|
old=""" private async retryPendingFlushes(): Promise<void> {
|
||||||
|
let rounds = 0;""",
|
||||||
|
new=""" private async retryPendingFlushes(): Promise<void> {
|
||||||
|
return; // [MUTATION] 无重试路径
|
||||||
|
let rounds = 0;""",
|
||||||
|
test=B6, pattern='持续失败 → flush 明确报错且数据不丢',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='45 后台错误检查放回"入链之前"',
|
||||||
|
file=LSM,
|
||||||
|
old=""" async flush(): Promise<void> {
|
||||||
|
this.enqueuePendingMemtables();
|
||||||
|
const reported = this.consumeBackgroundError();
|
||||||
|
await this.drainChain();""",
|
||||||
|
new=""" async flush(): Promise<void> {
|
||||||
|
// [MUTATION] 旧行为:先检查/报告后台错误 → 本次 flush 被整个跳过
|
||||||
|
const early = this.consumeBackgroundError();
|
||||||
|
if (early !== null) {
|
||||||
|
throw new DatabaseError('background error (legacy behaviour)', 'ARIA_BACKGROUND_ERROR', early);
|
||||||
|
}
|
||||||
|
this.enqueuePendingMemtables();
|
||||||
|
const reported = this.consumeBackgroundError();
|
||||||
|
await this.drainChain();""",
|
||||||
|
test=B6, pattern='后台错误不得让本次 flush 白做',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='manifest 全世代损坏 → 返回空状态(静默空库)',
|
||||||
|
file=MANIFEST,
|
||||||
|
old=""" if (generations.length > 0) {
|
||||||
|
// 文件存在但一代都读不出来:这是元数据损坏,不是空库
|
||||||
|
throw new DatabaseError(""",
|
||||||
|
new=""" if (false && generations.length > 0) { // [MUTATION] 静默当空库
|
||||||
|
throw new DatabaseError(""",
|
||||||
|
test=B6, pattern='全部世代都损坏',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='陈旧实例不设防(静默覆盖新世代)',
|
||||||
|
file=MANIFEST,
|
||||||
|
old=""" if (!opts.allowTakeover && newestForeign > this.generation) {""",
|
||||||
|
new=""" if (false && !opts.allowTakeover && newestForeign > this.generation) { // [MUTATION] 不设防""",
|
||||||
|
test=B6, pattern='陈旧实例提交被拒绝',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='旧格式 meta 损坏 → 返回空(静默空库)',
|
||||||
|
file=ENGINE,
|
||||||
|
old=""" } catch (error) {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Legacy SSTable metadata "${key}" is corrupt and cannot be migrated: ${(error as Error).message}`,
|
||||||
|
'ARIA_LEGACY_META_CORRUPT',
|
||||||
|
error,
|
||||||
|
);
|
||||||
|
}""",
|
||||||
|
new=""" } catch (error) {
|
||||||
|
continue; // [MUTATION] 静默跳过损坏的旧 meta
|
||||||
|
}""",
|
||||||
|
test=B6, pattern='旧格式 meta 损坏',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='介质读故障被当成"文件不存在"(误删 meta)',
|
||||||
|
file=LSM,
|
||||||
|
old=""" } catch (error) {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`AriaEngine failed to read SSTable id=${meta.id} from storage: ${(error as Error).message}`,
|
||||||
|
'ARIA_SSTABLE_READ_FAILED',
|
||||||
|
error,
|
||||||
|
);
|
||||||
|
}""",
|
||||||
|
new=""" } catch (error) {
|
||||||
|
await this.dropInvalidSSTable(meta, 'read failure treated as missing'); // [MUTATION] 旧行为
|
||||||
|
return null;
|
||||||
|
}""",
|
||||||
|
test=B6, pattern='介质读故障 ≠ 文件缺失',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='WAL 分片空洞静默丢弃尾部',
|
||||||
|
file='src/engine/aria/wal/log.ts',
|
||||||
|
old=""" if (gaps.length > 0 && !opts.allowGaps) {""",
|
||||||
|
new=""" if (false && gaps.length > 0 && !opts.allowGaps) { // [MUTATION] 静默丢弃""",
|
||||||
|
test=B6, pattern='活跃区间内缺失分片',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='WAL 分片号复用(旧世代排在新记录之后)',
|
||||||
|
file=SEGSTORE,
|
||||||
|
old=""" if (leftover.length === 0) {
|
||||||
|
this.currentSegment = Math.max(maxSeq + 1, keepFrom);""",
|
||||||
|
new=""" if (leftover.length === 0) {
|
||||||
|
this.currentSegment = 0; // [MUTATION] 复用分片号""",
|
||||||
|
test=STRESS, pattern='随机操作压力',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='冻结表意图不做"写入是否真的丢了"校验',
|
||||||
|
file=ENGINE,
|
||||||
|
old=""" const summary = intents.map((i) => `${i.ns}#${i.id}(${i.entryCount} 项)`).join(', ');
|
||||||
|
throw new DatabaseError(""",
|
||||||
|
new=""" const summary = intents.map((i) => `${i.ns}#${i.id}(${i.entryCount} 项)`).join(', ');
|
||||||
|
if (summary) return; // [MUTATION] 不校验
|
||||||
|
throw new DatabaseError(""",
|
||||||
|
test=B6, pattern='ARIA_WRITE_LOST',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='提交中的冻结表仍写进意图(回退时误报 WRITE_LOST)',
|
||||||
|
file=LSM,
|
||||||
|
old=""" // 正在提交的表由当前这次 manifest 提交负责(见 FrozenTable.committing)
|
||||||
|
if (frozen.committing) continue;""",
|
||||||
|
new=""" // [MUTATION] 提交中的表也写进意图""",
|
||||||
|
test=B6, pattern='回退上一代并标记 manifestFallback',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='checkpoint 等完整 compaction(写路径被拖住)',
|
||||||
|
file=CHECKPOINT,
|
||||||
|
old=""" if (this.flushable && typeof this.flushable.flushMemtables === 'function') {""",
|
||||||
|
new=""" if (false && this.flushable && typeof this.flushable.flushMemtables === 'function') { // [MUTATION] 旧行为""",
|
||||||
|
test=B6, pattern='checkpoint 不再等完整 compaction',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='collectReaders 不做结构版本重试(扫描漏掉并发发布的 SSTable)',
|
||||||
|
file=LSM,
|
||||||
|
old=""" if (version === this.structureVersion) return readers;
|
||||||
|
this.readStructureRetries++;
|
||||||
|
}""",
|
||||||
|
new=""" return readers; // [MUTATION] 不校验结构版本
|
||||||
|
}""",
|
||||||
|
test=B6, pattern='并发 flush 与 compaction 交错时',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='点查不做结构版本重试(get 的校验失效)',
|
||||||
|
file=LSM,
|
||||||
|
old=""" if (version !== this.structureVersion) {
|
||||||
|
this.readStructureRetries++;
|
||||||
|
continue; // 期间有结构变化(新产物发布 / 前台冻结)→ 重来
|
||||||
|
}""",
|
||||||
|
new=""" if (false) {
|
||||||
|
this.readStructureRetries++;
|
||||||
|
continue;
|
||||||
|
}""",
|
||||||
|
test=B6, pattern='并发 flush 与 compaction 交错时',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='冻结 memtable 不更新结构版本(点查读到旧值/漏数据)',
|
||||||
|
file=LSM,
|
||||||
|
old=""" // v0.8.0:**前台变化也算结构变化**。读路径的"结构版本一致"必须覆盖""",
|
||||||
|
new=""" this.structureVersion--; // [MUTATION] 抵消下面的自增
|
||||||
|
// v0.8.0:**前台变化也算结构变化**。读路径的"结构版本一致"必须覆盖""",
|
||||||
|
test=B6, pattern='并发 flush 与 compaction 交错时',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='sstable 越界策略不统一(scanAll 另有一套实现:越界即整体放弃)',
|
||||||
|
file=SSTABLE,
|
||||||
|
old=""" scanAll(callback: (key: string, value: Record<string, unknown>) => void): void {
|
||||||
|
for (const [key, value] of this.iterEntries(0, this.indexEntries.length - 1)) {
|
||||||
|
callback(key, value);
|
||||||
|
}
|
||||||
|
}""",
|
||||||
|
new=""" scanAll(callback: (key: string, value: Record<string, unknown>) => void): void {
|
||||||
|
// [MUTATION] 修复前的状态:这里另有一份解析循环,越界策略与 iterEntries 不同
|
||||||
|
const lenSize = this.lenFieldSize();
|
||||||
|
for (let bi = 0; bi < this.indexEntries.length; bi++) {
|
||||||
|
const blockData = this.getBlockData(this.indexEntries[bi]);
|
||||||
|
if (!blockData) continue;
|
||||||
|
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
||||||
|
const count = blockView.getUint32(0, false);
|
||||||
|
let offset = 4;
|
||||||
|
for (let i = 0; i < count; i++) {
|
||||||
|
if (offset + lenSize > blockData.byteLength) return;
|
||||||
|
const keyLen = blockView.getUint32(offset, false);
|
||||||
|
offset += lenSize;
|
||||||
|
if (offset + keyLen + lenSize > blockData.byteLength) return;
|
||||||
|
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
||||||
|
offset += keyLen;
|
||||||
|
const valLen = blockView.getUint32(offset, false);
|
||||||
|
offset += lenSize;
|
||||||
|
if (offset + valLen > blockData.byteLength) return;
|
||||||
|
const valBytes = blockData.slice(offset, offset + valLen);
|
||||||
|
offset += valLen;
|
||||||
|
try { callback(key, JSON.parse(new TextDecoder().decode(valBytes))); } catch { /* skip */ }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}""",
|
||||||
|
test=B6, pattern='三个读取路径对同一个损坏文件给出一致结论',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='vacuum 报告与事实无关的层数(硬编码 6)',
|
||||||
|
file=ENGINE,
|
||||||
|
old=""" const compactedLevels = await this.lsm.vacuumLevels();""",
|
||||||
|
new=""" await this.lsm.vacuumLevels();
|
||||||
|
const compactedLevels = 6; // [MUTATION] 旧行为:与事实无关的数字""",
|
||||||
|
test=B6, pattern='vacuum 返回',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='close() 落盘失败后不释放后端/锁(无 try/finally)',
|
||||||
|
file=ENGINE,
|
||||||
|
old=""" let failure: unknown = null;
|
||||||
|
try {
|
||||||
|
await this.flushAllLsms();""",
|
||||||
|
new=""" let failure: unknown = null;
|
||||||
|
if (true) { // [MUTATION] 旧行为:失败即中断,收尾逻辑不再执行
|
||||||
|
await this.flushAllLsms();
|
||||||
|
await this.bufferPool.flushAll();
|
||||||
|
await this.advanceWalCheckpoint();
|
||||||
|
await this.backend.close();
|
||||||
|
if (this.dbLock) { await this.dbLock.release(); this.dbLock = null; }
|
||||||
|
this.schemas.clear();
|
||||||
|
this.opened = false;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
await this.flushAllLsms();""",
|
||||||
|
test=B6, pattern='close\\(\\) 在落盘失败时',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='恢复报告不聚合 LSM 侧丢弃(数据丢失不可见)',
|
||||||
|
file=ENGINE,
|
||||||
|
old=""" for (const lsm of this.allLsms()) {
|
||||||
|
const r = lsm.getRecoveryReport();
|
||||||
|
for (const d of r.droppedSSTables) {
|
||||||
|
dropped.push({ namespace: r.namespace, ...d });
|
||||||
|
}
|
||||||
|
if (r.dataLossSuspected) dataLoss = true;
|
||||||
|
}""",
|
||||||
|
new=""" // [MUTATION] 不聚合 LSM 报告""",
|
||||||
|
test=B6, pattern='恢复报告聚合',
|
||||||
|
),
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# review 轮(v0.8.0 全量回归审查)新增:每条对应一个已证实的缺陷
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
dict(
|
||||||
|
name='R1 事务进行中仍推进 WAL 水位(P0:已提交事务静默丢失)',
|
||||||
|
file=ENGINE,
|
||||||
|
old=""" if (this.currentTxnId !== null) return this.durableLsn;""",
|
||||||
|
new=""" if (false as boolean) return this.durableLsn; // [MUTATION] 去掉事务守卫""",
|
||||||
|
test=B6, pattern='事务进行中不得推进 WAL 水位',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='R2 强制回收不理会途读者(在途读者数据被打断)',
|
||||||
|
file=LSM,
|
||||||
|
old=""" if (this.activeReaders.size > 0) {
|
||||||
|
this.reclaimRetired();
|
||||||
|
return;
|
||||||
|
}""",
|
||||||
|
new=""" if (false as boolean) {
|
||||||
|
this.reclaimRetired();
|
||||||
|
return;
|
||||||
|
}""",
|
||||||
|
test=B6, pattern='退休 SSTable 与在途读者',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='R3 孤儿回收只看引擎层损坏(漏掉 LSM 层被丢的 SSTable)',
|
||||||
|
file=ENGINE,
|
||||||
|
old=""" if (damage.length > 0) {""",
|
||||||
|
new=""" if (damage.length > 99) { // [MUTATION] 门槛失效""",
|
||||||
|
test=B6, pattern='回收门槛与孤儿回收',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='R4 WAL 前缀缺失丢弃后缀分片(整段活 WAL 被丢掉)',
|
||||||
|
file=SEGSTORE,
|
||||||
|
old=""" if (fromLsn === 0 && missingPrefix.length > 0) {""",
|
||||||
|
new=""" if (missingPrefix.length > 0) { // [MUTATION] 前缀缺失一律算空洞""",
|
||||||
|
test=B6, pattern='WAL 前缀缺失',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='R5 丢弃 SSTable 后不同步内存层数组(幽灵 meta)',
|
||||||
|
file=LSM,
|
||||||
|
old=""" if (this.levels[meta.level]?.some((m) => m.id === meta.id)) {""",
|
||||||
|
new=""" if (false as boolean) { // [MUTATION] 只从 manifest 摘除""",
|
||||||
|
test=B6, pattern='内存层数组与 manifest 必须一致',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='R6 不校验"文件名世代 == 载荷世代"(改名的旧副本被当成新提交点)',
|
||||||
|
file=MANIFEST,
|
||||||
|
old=""" if (decoded.manifest.generation !== gen) {""",
|
||||||
|
new=""" if (false as boolean) { // [MUTATION] 不比对世代号""",
|
||||||
|
test=B6, pattern='文件名世代与载荷世代不一致',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='R7 pageIdWatermark 不再取单调 max(水位可回退→页面 id 复用)',
|
||||||
|
file=ENGINE,
|
||||||
|
old=""" this.manifest.pageIdWatermark = Math.max(
|
||||||
|
this.manifest.pageIdWatermark,
|
||||||
|
this.fileManager.getNextPageId(),
|
||||||
|
);""",
|
||||||
|
new=""" this.manifest.pageIdWatermark = this.fileManager.getNextPageId(); // [MUTATION]""",
|
||||||
|
test=B6, pattern='水位是单调下限',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='R8 提交不做回读校验(写丢了也报成功)',
|
||||||
|
file=MANIFEST,
|
||||||
|
old=""" if (!verified.ok || verified.manifest.generation !== nextGeneration) {""",
|
||||||
|
new=""" if (false as boolean) { // [MUTATION] 关掉回读校验""",
|
||||||
|
test=B6, pattern='读回来是坏的',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='R9 WAL 损坏记录不计数(静默丢记录)',
|
||||||
|
file=WALLOG,
|
||||||
|
old=""" corrupt++;""",
|
||||||
|
new=""" corrupt += 0; // [MUTATION] 不计数""",
|
||||||
|
test=B6, pattern='WAL 记录级损坏',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='R10 丢弃残缺文件不进恢复报告(静默丢弃)',
|
||||||
|
file=LSM,
|
||||||
|
old=""" this.recoveryReport.droppedSSTables.push({ id: meta.id, level: meta.level, reason });""",
|
||||||
|
new=""" void reason; // [MUTATION] 不记录丢弃原因""",
|
||||||
|
test=B6, pattern='文件真的残缺',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='R11 介质读故障的错误码被改写(读故障≠文件损坏的语义丢失)',
|
||||||
|
file=LSM,
|
||||||
|
old=""" 'ARIA_SSTABLE_READ_FAILED',""",
|
||||||
|
new=""" 'MUTATED_READ_FAILED', // [MUTATION]""",
|
||||||
|
test=B6, pattern='介质读故障',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='R12 冻结意图不再压低水位(水位越过只在内存+WAL 的写入)',
|
||||||
|
file=ENGINE,
|
||||||
|
old=""" return Math.min(this.durableLsn, Math.min(...intents.map((i) => i.lsnAtFreeze)));""",
|
||||||
|
new=""" return Math.max(this.durableLsn, Math.max(...intents.map((i) => i.lsnAtFreeze))); // [MUTATION]""",
|
||||||
|
test=B6, pattern='水位严格低于意图起点',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='R13 truncateBefore 把分片号重置为 0(新记录落在 manifest 水位之下被过滤)',
|
||||||
|
file=SEGSTORE,
|
||||||
|
old=""" this.currentSegment = Math.max(maxSeq + 1, keepFrom);""",
|
||||||
|
new=""" this.currentSegment = 0; // [MUTATION] 修复前行为""",
|
||||||
|
test=WALSEG, pattern='写入永不落到 manifest 水位下限之下',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='R14 truncate 把分片号重置为 0(旧世代与新记录同号)',
|
||||||
|
file=SEGSTORE,
|
||||||
|
old=""" this.currentSegment = Math.max(this.currentSegment, maxSeq + 1);""",
|
||||||
|
new=""" this.currentSegment = 0; // [MUTATION] 修复前行为""",
|
||||||
|
test=WALSEG, pattern='整体清空后分片号绝不回退',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='R15 内部空洞不再上报(丢一段已提交事务却毫无痕迹)',
|
||||||
|
file=SEGSTORE,
|
||||||
|
old=""" gaps.push(missing);""",
|
||||||
|
new=""" void missing; // [MUTATION] 不上报空洞""",
|
||||||
|
test=WALSEG, pattern='内部空洞',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='R20 DatabaseError 不再暴露标准 cause(根因丢失)',
|
||||||
|
file='src/constants.ts',
|
||||||
|
old=""" if (details !== undefined) {
|
||||||
|
(this as { cause?: unknown }).cause = details;
|
||||||
|
}""",
|
||||||
|
new=""" void details; // [MUTATION] 修复前:只有 details,没有 cause""",
|
||||||
|
test='tests/engine/aria-opfs-unavailable.test.ts', pattern='DatabaseError',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='R19 OPFS 不可用时吞掉原因(错误不可操作)',
|
||||||
|
file='src/engine/aria/store/opfs_backend.ts',
|
||||||
|
old=""" throw new DatabaseError(
|
||||||
|
`OPFS is not accessible here (${(error as Error).name}: ${(error as Error).message}). ` +
|
||||||
|
'Chromium blocks OPFS for file:// pages — serve the page over http(s) ' +
|
||||||
|
"(e.g. `npx serve .` / `node tests/e2e/server.cjs`) or use mode: 'memory'.",
|
||||||
|
'ARIA_OPFS_UNAVAILABLE',
|
||||||
|
error,
|
||||||
|
);""",
|
||||||
|
new=""" throw new DatabaseError(
|
||||||
|
'Failed to open OPFS backend',
|
||||||
|
'ARIA_OPFS_UNAVAILABLE',
|
||||||
|
); // [MUTATION] 丢掉原始 cause 与建议""",
|
||||||
|
test='tests/engine/aria-opfs-unavailable.test.ts', pattern='OPFSBackend',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='R17 bloom 位数被忽略(bloomFilterBitsPerKey 配置无效)',
|
||||||
|
file='src/engine/aria/index/sstable_builder.ts',
|
||||||
|
old=""" const bloomFilter = new BloomFilter(this.entries.length, this.bloomBitsPerKey);""",
|
||||||
|
new=""" const bloomFilter = new BloomFilter(this.entries.length); // [MUTATION] 忽略配置""",
|
||||||
|
test=B6, pattern='bloomFilterBitsPerKey',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='R18 引擎层不把 bloom 配置透传给 LSM(配置无效)',
|
||||||
|
file=LSM,
|
||||||
|
old=""" this.bloomBitsPerKey = Number.isFinite(config.bloomBitsPerKey) && (config.bloomBitsPerKey as number) > 0
|
||||||
|
? Math.floor(config.bloomBitsPerKey as number)
|
||||||
|
: DEFAULT_BLOOM_BITS_PER_KEY;""",
|
||||||
|
new=""" this.bloomBitsPerKey = DEFAULT_BLOOM_BITS_PER_KEY; // [MUTATION] 忽略配置""",
|
||||||
|
test=B6, pattern='bloomFilterBitsPerKey',
|
||||||
|
),
|
||||||
|
dict(
|
||||||
|
name='R16 旧格式表结构形状坏掉时静默当空库',
|
||||||
|
file=ENGINE,
|
||||||
|
old=""" if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {""",
|
||||||
|
new=""" if (false as boolean) { // [MUTATION] 不校验形状""",
|
||||||
|
test=B6, pattern='旧格式表结构记录的形状校验',
|
||||||
|
),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
CURRENT = {'path': None, 'content': None}
|
||||||
|
|
||||||
|
|
||||||
|
def _restore_current(*_args):
|
||||||
|
"""收到信号时也要把源码恢复原状(否则中断会留下变异源码)。"""
|
||||||
|
if CURRENT['path'] and CURRENT['content'] is not None:
|
||||||
|
io.open(CURRENT['path'], 'w', encoding='utf-8').write(CURRENT['content'])
|
||||||
|
print(f"\n[restore] {CURRENT['path']} 已恢复", flush=True)
|
||||||
|
sys.exit(130)
|
||||||
|
|
||||||
|
|
||||||
|
for _sig in ('SIGINT', 'SIGTERM', 'SIGHUP'):
|
||||||
|
try:
|
||||||
|
signal.signal(getattr(signal, _sig), _restore_current)
|
||||||
|
except (AttributeError, ValueError): # pragma: no cover - 平台差异
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def run(cmd, timeout=JEST_TIMEOUT_S):
|
||||||
|
"""跑一条命令:带超时(超时 = 变异脚本自身的问题,必须显式报出来)。"""
|
||||||
|
return subprocess.run(
|
||||||
|
cmd, cwd=ROOT, shell=True, capture_output=True, text=True, timeout=timeout,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def sha256(text):
|
||||||
|
return hashlib.sha256(text.encode('utf-8')).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def acquire_lock():
|
||||||
|
"""同一时刻只允许一个变异进程改 src/(并发跑会互相踩掉对方的源码)。"""
|
||||||
|
lock = os.path.join(ROOT, '.mutation-b6.lock')
|
||||||
|
try:
|
||||||
|
fd = os.open(lock, os.O_CREAT | os.O_EXCL | os.O_WRONLY)
|
||||||
|
except FileExistsError:
|
||||||
|
print(f'[abort] 已有变异进程在跑({lock} 存在)。')
|
||||||
|
print(' 确认没有残留后删除该文件再重试。')
|
||||||
|
sys.exit(2)
|
||||||
|
os.write(fd, str(os.getpid()).encode())
|
||||||
|
os.close(fd)
|
||||||
|
return lock
|
||||||
|
|
||||||
|
|
||||||
|
def classify(out):
|
||||||
|
"""把一次 jest 输出分类。顺序很重要:
|
||||||
|
编译失败/未匹配**不能**算"测试拦住了变异"。"""
|
||||||
|
if re.search(r'Test suite failed to run', out) or re.search(r'error TS\d+', out) \
|
||||||
|
or 'Cannot find module' in out or 'SyntaxError' in out:
|
||||||
|
return 'BAD(变异破坏编译)'
|
||||||
|
if re.search(r'^Tests:\s+0 total', out, re.M) or 'No tests found' in out:
|
||||||
|
return 'BAD(用例未匹配)'
|
||||||
|
if re.search(r'^Tests:.*\bfailed\b', out, re.M) or '✕' in out:
|
||||||
|
return 'OK(变异被拦住)'
|
||||||
|
return 'BAD(测试仍然通过)'
|
||||||
|
|
||||||
|
|
||||||
|
def positive_control():
|
||||||
|
"""正控:未变异的干净代码上,这些套件必须全绿。
|
||||||
|
没有正控的变异脚本会把"套件本来就红"误读成"变异被拦住了"。"""
|
||||||
|
print('[control] 干净代码基线检查 …', flush=True)
|
||||||
|
for suite in CONTROL_SUITES:
|
||||||
|
try:
|
||||||
|
res = run(f"npx jest {shlex.quote(suite)} --testPathIgnorePatterns='/node_modules/'")
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
print(f'[control] 超时:{suite}')
|
||||||
|
return False
|
||||||
|
out = (res.stdout or '') + (res.stderr or '')
|
||||||
|
m = re.search(r'^Tests:\s+(.*)$', out, re.M)
|
||||||
|
line = m.group(1) if m else '无法解析'
|
||||||
|
if 'failed' in line or not m:
|
||||||
|
print(f'[control] 基线不干净:{suite} → Tests: {line}')
|
||||||
|
return False
|
||||||
|
print(f'[control] ok {suite} → Tests: {line}', flush=True)
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
lock = acquire_lock()
|
||||||
|
try:
|
||||||
|
return _main()
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
os.unlink(lock)
|
||||||
|
except OSError: # pragma: no cover
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _main():
|
||||||
|
only = sys.argv[1] if len(sys.argv) > 1 else None
|
||||||
|
if not positive_control():
|
||||||
|
print('\n[abort] 正控未通过 —— 先修好基线再谈"变异被拦住"。')
|
||||||
|
return 2
|
||||||
|
|
||||||
|
results = []
|
||||||
|
fingerprints = {}
|
||||||
|
for m in MUTATIONS:
|
||||||
|
if only and only not in m['name']:
|
||||||
|
continue
|
||||||
|
path = os.path.join(ROOT, m['file'])
|
||||||
|
original = io.open(path, encoding='utf-8').read()
|
||||||
|
fingerprints[path] = sha256(original)
|
||||||
|
if m['old'] not in original:
|
||||||
|
results.append((m['name'], 'SKIP(锚点未找到)'))
|
||||||
|
print(f"[skip] {m['name']}: 锚点未找到", flush=True)
|
||||||
|
continue
|
||||||
|
if original.count(m['old']) != 1:
|
||||||
|
results.append((m['name'], f"SKIP(锚点不唯一 x{original.count(m['old'])})"))
|
||||||
|
print(f"[skip] {m['name']}: 锚点出现 {original.count(m['old'])} 次,拒绝变异", flush=True)
|
||||||
|
continue
|
||||||
|
mutated = original.replace(m['old'], m['new'], 1)
|
||||||
|
CURRENT['path'] = path
|
||||||
|
CURRENT['content'] = original
|
||||||
|
io.open(path, 'w', encoding='utf-8').write(mutated)
|
||||||
|
try:
|
||||||
|
# 注意:用 shlex.quote 而不是 repr —— repr 会把正则里的反斜杠再转义一次,
|
||||||
|
# 于是 jest 收到 `\\(` 这种模式、匹配不到任何用例,脚本会把"没跑用例"
|
||||||
|
# 误判成"测试仍然通过"。
|
||||||
|
cmd = (f"npx jest {shlex.quote(m['test'])} -t {shlex.quote(m['pattern'])} "
|
||||||
|
f"--testPathIgnorePatterns='/node_modules/'")
|
||||||
|
try:
|
||||||
|
res = run(cmd)
|
||||||
|
out = (res.stdout or '') + (res.stderr or '')
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
status = 'BAD(超时)'
|
||||||
|
out = ''
|
||||||
|
else:
|
||||||
|
status = classify(out)
|
||||||
|
results.append((m['name'], status))
|
||||||
|
icon = 'ok ' if status == 'OK(变异被拦住)' else 'BAD'
|
||||||
|
print(f'[{icon}] {m["name"]} → {status}', flush=True)
|
||||||
|
if status != 'OK(变异被拦住)':
|
||||||
|
tail = [l for l in out.splitlines() if l.startswith('Tests:')]
|
||||||
|
if tail:
|
||||||
|
print(f' {tail[0]}', flush=True)
|
||||||
|
finally:
|
||||||
|
io.open(path, 'w', encoding='utf-8').write(original)
|
||||||
|
# 恢复必须逐字节一致(否则会悄悄改动源码)
|
||||||
|
if sha256(io.open(path, encoding='utf-8').read()) != fingerprints[path]:
|
||||||
|
print(f'[FATAL] {m["file"]} 恢复后与原始内容不一致!', flush=True)
|
||||||
|
return 3
|
||||||
|
CURRENT['path'] = None
|
||||||
|
CURRENT['content'] = None
|
||||||
|
|
||||||
|
print('\n==== 变异验证汇总 ====')
|
||||||
|
bad = [r for r in results if r[1] != 'OK(变异被拦住)']
|
||||||
|
for name, status in results:
|
||||||
|
print(f' {status:20s} {name}')
|
||||||
|
if bad:
|
||||||
|
print(f'\n{len(bad)} 条变异没有被测试拦住 —— 那些用例只是陪跑。')
|
||||||
|
return 1
|
||||||
|
print(f'\n全部 {len(results)} 条变异都被对应用例拦住。')
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
sys.exit(main())
|
||||||
+1
-1
@@ -3,7 +3,7 @@
|
|||||||
<head>
|
<head>
|
||||||
<meta charset="UTF-8">
|
<meta charset="UTF-8">
|
||||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
<title>⚡ 性能基准 — MetonaSqlark v0.7.3</title>
|
<title>⚡ 性能基准 — MetonaSqlark v0.8.0</title>
|
||||||
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><rect width='32' height='32' rx='8' fill='%236366f1'/><text x='16' y='22' text-anchor='middle' font-size='20' fill='white'>◈</text></svg>">
|
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><rect width='32' height='32' rx='8' fill='%236366f1'/><text x='16' y='22' text-anchor='middle' font-size='20' fill='white'>◈</text></svg>">
|
||||||
<style>
|
<style>
|
||||||
:root {
|
:root {
|
||||||
|
|||||||
+21
-8
@@ -3,7 +3,7 @@
|
|||||||
<head>
|
<head>
|
||||||
<meta charset="UTF-8">
|
<meta charset="UTF-8">
|
||||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
<title>🧪 在线演示 — MetonaSqlark v0.7.3</title>
|
<title>🧪 在线演示 — MetonaSqlark v0.8.0</title>
|
||||||
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><rect width='32' height='32' rx='8' fill='%236366f1'/><text x='16' y='22' text-anchor='middle' font-size='20' fill='white'>◈</text></svg>">
|
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><rect width='32' height='32' rx='8' fill='%236366f1'/><text x='16' y='22' text-anchor='middle' font-size='20' fill='white'>◈</text></svg>">
|
||||||
<style>
|
<style>
|
||||||
:root {
|
:root {
|
||||||
@@ -84,7 +84,7 @@
|
|||||||
<a href="demo.html" class="nav-active">演示</a>
|
<a href="demo.html" class="nav-active">演示</a>
|
||||||
<a href="benchmark.html">基准</a>
|
<a href="benchmark.html">基准</a>
|
||||||
</nav>
|
</nav>
|
||||||
<div class="status"><span class="dot" id="engine-dot"></span> <span id="engine-status">Memory</span> 模式 — v0.7.3</div>
|
<div class="status"><span class="dot" id="engine-dot"></span> <span id="engine-status">Memory</span> 模式 — v0.8.0</div>
|
||||||
<button class="btn btn-preset" onclick="switchEngine('memory')" id="btn-memory" style="margin:6px 4px 6px 0;padding:6px 12px;">⚡ Memory</button>
|
<button class="btn btn-preset" onclick="switchEngine('memory')" id="btn-memory" style="margin:6px 4px 6px 0;padding:6px 12px;">⚡ Memory</button>
|
||||||
<button class="btn btn-preset" onclick="switchEngine('aria')" id="btn-aria" style="margin:6px 0;padding:6px 12px;color:#ec4899;border-color:#ec4899;">🌲 Aria</button>
|
<button class="btn btn-preset" onclick="switchEngine('aria')" id="btn-aria" style="margin:6px 0;padding:6px 12px;color:#ec4899;border-color:#ec4899;">🌲 Aria</button>
|
||||||
</header>
|
</header>
|
||||||
@@ -92,9 +92,10 @@
|
|||||||
<div class="main">
|
<div class="main">
|
||||||
<div class="editor-panel">
|
<div class="editor-panel">
|
||||||
<div class="editor-area">
|
<div class="editor-area">
|
||||||
<textarea id="sql-input" placeholder="输入 SQL 语句... SELECT * FROM users; INSERT INTO users VALUES ('4', 'Diana', 'diana@test.com', 28); SELECT u.name, o.amount FROM users u INNER JOIN orders o ON u.id = o.user_id;">-- 🚀 MetonaSqlark v0.7.3 在线演示
|
<textarea id="sql-input" placeholder="输入 SQL 语句... SELECT * FROM users; INSERT INTO users VALUES ('4', 'Diana', 'diana@test.com', 28); SELECT u.name, o.amount FROM users u INNER JOIN orders o ON u.id = o.user_id;">-- 🚀 MetonaSqlark v0.8.0 在线演示
|
||||||
-- 已预置 users / orders / products 表数据
|
-- 已预置 users / orders / products 表数据
|
||||||
-- 新特性: ALTER TABLE · TRUNCATE TABLE · WAL同步 · MVCC · SQL注入防护
|
-- 新特性 (v0.8.0): manifest 单一提交点 · WAL 记录级损坏上报 · 崩溃恢复报告
|
||||||
|
-- 另有: ALTER TABLE · TRUNCATE TABLE · 参数化查询 · 三值逻辑 · 时间/大小写函数 · 外键级联 · 发布订阅
|
||||||
|
|
||||||
-- 查看所有数据
|
-- 查看所有数据
|
||||||
SELECT * FROM users;
|
SELECT * FROM users;
|
||||||
@@ -185,6 +186,18 @@ async function initDB() {
|
|||||||
await seedDemoData(db);
|
await seedDemoData(db);
|
||||||
renderInfo(`✅ ${engine === 'aria' ? '🌲 AriaEngine(LSM-Tree + WAL + MVCC)' : '⚡ Memory'} 引擎已就绪`);
|
renderInfo(`✅ ${engine === 'aria' ? '🌲 AriaEngine(LSM-Tree + WAL + MVCC)' : '⚡ Memory'} 引擎已就绪`);
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
|
// v0.8.0:OPFS 在当前环境不可用(最常见:直接双击打开的 file:// 页面,Chromium
|
||||||
|
// 禁止其访问 OPFS;此时 isSecureContext 仍为 true、API 也存在,只能调用才发现)
|
||||||
|
// → 用中文说清"为什么失败 + 怎么修",而不是丢一句底层英文报错。
|
||||||
|
if (e && e.code === 'ARIA_OPFS_UNAVAILABLE') {
|
||||||
|
renderError(
|
||||||
|
'Aria 引擎需要 OPFS,而当前页面环境不允许访问(' + location.protocol + ')。\n' +
|
||||||
|
'· 原因:浏览器只允许 http(s) 页面使用 OPFS,file:// 直接打开会被拒绝。\n' +
|
||||||
|
'· 解决:用本地服务器打开(仓库内执行 node tests/e2e/server.cjs 3344,' +
|
||||||
|
'再访问 http://127.0.0.1:3344/site/demo.html),或点上面的「⚡ Memory」用内存模式体验。'
|
||||||
|
);
|
||||||
|
return;
|
||||||
|
}
|
||||||
renderError('数据库初始化失败: ' + e.message);
|
renderError('数据库初始化失败: ' + e.message);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -555,9 +568,9 @@ SELECT COUNT(*) as total FROM temp_logs;
|
|||||||
|
|
||||||
-- 清理
|
-- 清理
|
||||||
DROP TABLE temp_logs;`,
|
DROP TABLE temp_logs;`,
|
||||||
aria: `-- 🌲 AriaEngine 演示 (v0.7.3)
|
aria: `-- 🌲 AriaEngine 演示 (v0.8.0)
|
||||||
-- 点击右上角「🌲 Aria」按钮切换数据库引擎到 AriaEngine
|
-- 点击右上角「🌲 Aria」按钮切换数据库引擎到 AriaEngine
|
||||||
-- 当前数据库即运行在 Aria 引擎上(LSM-Tree · WAL 崩溃恢复 · MVCC · BloomFilter)
|
-- 当前数据库即运行在 Aria 引擎上(LSM-Tree · WAL 崩溃恢复 · BloomFilter · 4KB 页面化)
|
||||||
-- 基础 CRUD 与 Memory 引擎完全兼容
|
-- 基础 CRUD 与 Memory 引擎完全兼容
|
||||||
|
|
||||||
-- 当前引擎确认
|
-- 当前引擎确认
|
||||||
@@ -591,7 +604,7 @@ DROP TABLE temp_logs;`,
|
|||||||
-- AriaEngine 特性:
|
-- AriaEngine 特性:
|
||||||
-- • LSM-Tree: MemTable (红黑树) → SSTable 多级索引
|
-- • LSM-Tree: MemTable (红黑树) → SSTable 多级索引
|
||||||
-- • WAL: 分片文件 + 标准 CRC32 + 空洞检测 + 崩溃恢复(残缺 SSTable 自动跳过)
|
-- • WAL: 分片文件 + 标准 CRC32 + 空洞检测 + 崩溃恢复(残缺 SSTable 自动跳过)
|
||||||
-- • MVCC: 版本链 + 快照隔离
|
-- • MVCC: 版本链仅作事务内 undo(无快照隔离;事务串行)
|
||||||
-- • 页面化物理存储: SSTable 4KB 页面 + BufferPool LRU 缓存 (v0.5.0)
|
-- • 页面化物理存储: SSTable 4KB 页面 + BufferPool LRU 缓存 (v0.5.0)
|
||||||
-- • 完整性: 整文件 CRC-32 校验(数据腐坏自动清理自愈)
|
-- • 完整性: 整文件 CRC-32 校验(数据腐坏自动清理自愈)
|
||||||
-- • Bloom Filter: FNV-1a + Murmur 双哈希
|
-- • Bloom Filter: FNV-1a + Murmur 双哈希
|
||||||
@@ -826,7 +839,7 @@ document.addEventListener('keydown', e => {
|
|||||||
document.getElementById('btn-aria').style.opacity = '0.6';
|
document.getElementById('btn-aria').style.opacity = '0.6';
|
||||||
document.getElementById('engine-status').textContent = '⚡ Memory';
|
document.getElementById('engine-status').textContent = '⚡ Memory';
|
||||||
initDB().then(() => {
|
initDB().then(() => {
|
||||||
console.log('✅ MetonaSqlark v0.4.2 demo ready');
|
console.log('✅ MetonaSqlark v0.8.0 demo ready');
|
||||||
setTimeout(runQuery, 300);
|
setTimeout(runQuery, 300);
|
||||||
}).catch(err => {
|
}).catch(err => {
|
||||||
renderError('初始化失败: ' + err.message);
|
renderError('初始化失败: ' + err.message);
|
||||||
|
|||||||
+49
-9
@@ -3,7 +3,7 @@
|
|||||||
<head>
|
<head>
|
||||||
<meta charset="UTF-8">
|
<meta charset="UTF-8">
|
||||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
<title>📖 API 文档 — MetonaSqlark v0.7.3</title>
|
<title>📖 API 文档 — MetonaSqlark v0.8.0</title>
|
||||||
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><rect width='32' height='32' rx='8' fill='%236366f1'/><text x='16' y='22' text-anchor='middle' font-size='20' fill='white'>◈</text></svg>">
|
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><rect width='32' height='32' rx='8' fill='%236366f1'/><text x='16' y='22' text-anchor='middle' font-size='20' fill='white'>◈</text></svg>">
|
||||||
<style>
|
<style>
|
||||||
:root {
|
:root {
|
||||||
@@ -140,7 +140,7 @@ npm install @metona-team/metona-sqlark</pre>
|
|||||||
<table>
|
<table>
|
||||||
<tr><th>文件</th><th>格式</th><th>用途</th></tr>
|
<tr><th>文件</th><th>格式</th><th>用途</th></tr>
|
||||||
<tr><td><code>metona-sqlark.js</code></td><td>UMD</td><td>浏览器开发版(含 sourcemap)</td></tr>
|
<tr><td><code>metona-sqlark.js</code></td><td>UMD</td><td>浏览器开发版(含 sourcemap)</td></tr>
|
||||||
<tr><td><code>metona-sqlark.min.js</code></td><td>UMD (minified)</td><td>生产环境(~105KB / ~27KB gzip)</td></tr>
|
<tr><td><code>metona-sqlark.min.js</code></td><td>UMD (minified)</td><td>生产环境(实测 251,731 字节 / gzip 63,431 字节,含全部引擎与 SQL 层)</td></tr>
|
||||||
<tr><td><code>metona-sqlark.esm.js</code></td><td>ES Module</td><td>现代打包工具 / 浏览器 ESM</td></tr>
|
<tr><td><code>metona-sqlark.esm.js</code></td><td>ES Module</td><td>现代打包工具 / 浏览器 ESM</td></tr>
|
||||||
<tr><td><code>metona-sqlark.cjs</code></td><td>CommonJS</td><td>Node.js require()</td></tr>
|
<tr><td><code>metona-sqlark.cjs</code></td><td>CommonJS</td><td>Node.js require()</td></tr>
|
||||||
<tr><td><code>metona-sqlark.d.ts</code></td><td>TypeScript 声明</td><td>类型提示</td></tr>
|
<tr><td><code>metona-sqlark.d.ts</code></td><td>TypeScript 声明</td><td>类型提示</td></tr>
|
||||||
@@ -291,7 +291,12 @@ db.<span class="f">isReady</span>(); <span class="c">// true</span>
|
|||||||
|
|
||||||
<span class="c">// CREATE / DROP INDEX — 动态二级索引</span>
|
<span class="c">// CREATE / DROP INDEX — 动态二级索引</span>
|
||||||
<span class="k">await</span> db.<span class="f">query</span>(<span class="s">'CREATE INDEX idx_users_city ON users (city)'</span>);
|
<span class="k">await</span> db.<span class="f">query</span>(<span class="s">'CREATE INDEX idx_users_city ON users (city)'</span>);
|
||||||
<span class="k">await</span> db.<span class="f">query</span>(<span class="s">'DROP INDEX idx_users_city ON users (city)'</span>);</pre>
|
<span class="k">await</span> db.<span class="f">query</span>(<span class="s">'DROP INDEX idx_users_city ON users (city)'</span>);
|
||||||
|
|
||||||
|
<span class="c">// v0.7.4: 建表 UNIQUE 约束不可经 DROP INDEX 解除(NOT_SUPPORTED,重建表解除);
|
||||||
|
// 仅 CREATE UNIQUE INDEX 添加的唯一约束可随 DROP INDEX 一并删除(对齐 SQLite)</span>
|
||||||
|
<span class="k">await</span> db.<span class="f">query</span>(<span class="s">'CREATE UNIQUE INDEX idx_u ON users (email)'</span>);
|
||||||
|
<span class="k">await</span> db.<span class="f">query</span>(<span class="s">'DROP INDEX idx_u ON users (email)'</span>); <span class="c">// unique 随之解除</span></pre>
|
||||||
|
|
||||||
<h3>维护语句 (v0.5.1 SQL 入口)</h3>
|
<h3>维护语句 (v0.5.1 SQL 入口)</h3>
|
||||||
<pre><span class="c">// EXPLAIN — 查询计划(真实索引命中信息 v0.7.0)</span>
|
<pre><span class="c">// EXPLAIN — 查询计划(真实索引命中信息 v0.7.0)</span>
|
||||||
@@ -466,7 +471,16 @@ db.<span class="f">isReady</span>(); <span class="c">// true</span>
|
|||||||
|
|
||||||
<span class="c">// NOT IN 子查询</span>
|
<span class="c">// NOT IN 子查询</span>
|
||||||
<span class="k">await</span> db.<span class="f">query</span>(<span class="s">`SELECT * FROM users
|
<span class="k">await</span> db.<span class="f">query</span>(<span class="s">`SELECT * FROM users
|
||||||
WHERE id NOT IN (SELECT user_id FROM orders)`</span>);</pre>
|
WHERE id NOT IN (SELECT user_id FROM orders)`</span>);
|
||||||
|
|
||||||
|
<span class="c">// v0.7.4: UPDATE / DELETE 的 WHERE 同样支持子查询(四引擎)
|
||||||
|
// 此前未解析的 $subquery 在引擎层恒 false → 静默影响 0 行</span>
|
||||||
|
<span class="k">await</span> db.<span class="f">query</span>(<span class="s">`UPDATE users SET role = 'vip'
|
||||||
|
WHERE id IN (SELECT user_id FROM orders WHERE amount > 100)`</span>);
|
||||||
|
<span class="k">await</span> db.<span class="f">query</span>(<span class="s">`DELETE FROM logs
|
||||||
|
WHERE ts < (SELECT MIN(ts) FROM keep_logs)`</span>);
|
||||||
|
|
||||||
|
<span class="c">// 写语句中关联引用($col / EXISTS 引用外层行)显式 NOT_SUPPORTED(不静默)</span></pre>
|
||||||
|
|
||||||
<h2 id="alter-table">🏗 ALTER TABLE (🆕 v0.2.5)</h2>
|
<h2 id="alter-table">🏗 ALTER TABLE (🆕 v0.2.5)</h2>
|
||||||
<p>v0.2.5 新增 ALTER TABLE 语法,支持动态添加和删除列。</p>
|
<p>v0.2.5 新增 ALTER TABLE 语法,支持动态添加和删除列。</p>
|
||||||
@@ -779,7 +793,7 @@ db.<span class="f">broadcastChange</span>(<span class="s">'users'</span>);</pre>
|
|||||||
<tr><td><code>MemoryEngine</code></td><td>memory</td><td>❌</td><td>哈希</td><td>快照回滚</td><td>临时数据、缓存、测试</td></tr>
|
<tr><td><code>MemoryEngine</code></td><td>memory</td><td>❌</td><td>哈希</td><td>快照回滚</td><td>临时数据、缓存、测试</td></tr>
|
||||||
<tr><td><code>KVStoreEngine</code> 🆕</td><td>disk</td><td>✅ KVStore(OPFS)</td><td>哈希</td><td>原子日志 flush</td><td>标准持久化(v0.6.0 替代 IndexedDB)</td></tr>
|
<tr><td><code>KVStoreEngine</code> 🆕</td><td>disk</td><td>✅ KVStore(OPFS)</td><td>哈希</td><td>原子日志 flush</td><td>标准持久化(v0.6.0 替代 IndexedDB)</td></tr>
|
||||||
<tr><td><code>HybridEngine</code></td><td>hybrid</td><td>✅ Write-Through</td><td>哈希</td><td>双引擎代理</td><td>生产推荐,读写均走内存</td></tr>
|
<tr><td><code>HybridEngine</code></td><td>hybrid</td><td>✅ Write-Through</td><td>哈希</td><td>双引擎代理</td><td>生产推荐,读写均走内存</td></tr>
|
||||||
<tr style="border-top:2px solid var(--primary);"><td><code style="color:#ec4899;font-weight:700;">AriaEngine</code></td><td>aria</td><td>✅ WAL + SSTable</td><td>LSM-Tree</td><td>MVCC 快照隔离</td><td>自研引擎:大表、高并发、需崩溃恢复</td></tr>
|
<tr style="border-top:2px solid var(--primary);"><td><code style="color:#ec4899;font-weight:700;">AriaEngine</code></td><td>aria</td><td>✅ WAL + SSTable</td><td>LSM-Tree</td><td>版本链 undo + 事务串行</td><td>自研引擎:大表、高并发、需崩溃恢复</td></tr>
|
||||||
</table>
|
</table>
|
||||||
<p>✅ <strong>v0.6.0: IndexedDB 已完全移除</strong> — disk 模式改用自研 KVStore 引擎(多 key 原子写 + 快照/日志崩溃恢复),旧库可经 <code>migrateFromIndexedDB()</code> 一键迁移。</p>
|
<p>✅ <strong>v0.6.0: IndexedDB 已完全移除</strong> — disk 模式改用自研 KVStore 引擎(多 key 原子写 + 快照/日志崩溃恢复),旧库可经 <code>migrateFromIndexedDB()</code> 一键迁移。</p>
|
||||||
|
|
||||||
@@ -799,13 +813,15 @@ db.<span class="f">broadcastChange</span>(<span class="s">'users'</span>);</pre>
|
|||||||
<strong>v0.7.0 参数化查询 / 事务性能 / 语义硬化</strong> — <strong>参数化查询</strong>(<code>db.query(sql, params)</code> 位置参数 `?`,词法层绑定 + SQL 字面量安全编码,注入防护从根上成立)· EXPLAIN 真实索引命中信息(pk / index:col)· KVStoreEngine 事务行级增量 flush(大表事务改 1 行 commit 仅 1 条记录)· 移除每次 commit 强制全量 checkpoint · 复合主键显式拒绝(SCHEMA_ERROR,v0.8 路线图)。<br>
|
<strong>v0.7.0 参数化查询 / 事务性能 / 语义硬化</strong> — <strong>参数化查询</strong>(<code>db.query(sql, params)</code> 位置参数 `?`,词法层绑定 + SQL 字面量安全编码,注入防护从根上成立)· EXPLAIN 真实索引命中信息(pk / index:col)· KVStoreEngine 事务行级增量 flush(大表事务改 1 行 commit 仅 1 条记录)· 移除每次 commit 强制全量 checkpoint · 复合主键显式拒绝(SCHEMA_ERROR,v0.8 路线图)。<br>
|
||||||
<strong>v0.7.1 API 修复与防御统一</strong> — <code>MetonaSqlark.create</code> 静态工厂(README 示例在 ESM/Node 下可用)· close 未初始化崩溃防御 · AriaEngine 未 open 防护统一 · 事务回滚失败不掩盖原始错误 · <code>__proto__</code> 列名原型污染防护 · lint 清零。<br>
|
<strong>v0.7.1 API 修复与防御统一</strong> — <code>MetonaSqlark.create</code> 静态工厂(README 示例在 ESM/Node 下可用)· close 未初始化崩溃防御 · AriaEngine 未 open 防护统一 · 事务回滚失败不掩盖原始错误 · <code>__proto__</code> 列名原型污染防护 · lint 清零。<br>
|
||||||
<strong>v0.7.2 语句级原子性 / 事务 DDL / 约束硬化</strong> — UPDATE 语句级两阶段原子(多行匹配第 N 行失败整句不执行 + 批内唯一互查)· 事务内 DDL 显式拒绝(四引擎对齐)· SET NULL 级联绕过 required 约束整体拒绝 · 参数绑定注释感知(注释中 `?`/引号不参与绑定)· 未闭合字符串显式 PARSE_ERROR · 未知 where 操作符抛 QUERY_ERROR(此前静默全匹配)· update undefined 语义化(保留旧值)· Hybrid write-through 失败补偿(磁盘失败自动重载内存对齐)。<br>
|
<strong>v0.7.2 语句级原子性 / 事务 DDL / 约束硬化</strong> — UPDATE 语句级两阶段原子(多行匹配第 N 行失败整句不执行 + 批内唯一互查)· 事务内 DDL 显式拒绝(四引擎对齐)· SET NULL 级联绕过 required 约束整体拒绝 · 参数绑定注释感知(注释中 `?`/引号不参与绑定)· 未闭合字符串显式 PARSE_ERROR · 未知 where 操作符抛 QUERY_ERROR(此前静默全匹配)· update undefined 语义化(保留旧值)· Hybrid write-through 失败补偿(磁盘失败自动重载内存对齐)。<br>
|
||||||
<strong>v0.7.3 深度审计第六阶段:INSERT 原子 / 索引一致性 / 边界窗口</strong> — INSERT 语句级两阶段原子(三引擎 + Aria PK 批内重复,此前部分提交)· 索引列 IS NULL 恒空修复(Memory/KVStore/Hybrid,对齐 Aria)· delete RESTRICT 预检不再破坏索引 · queryStream 子查询静默空结果修复($subquery/$exists/$col 回退物化)· ALTER DROP 索引列残留清理 · CREATE UNIQUE INDEX 存量重复数据校验(失败原子回滚)· <code>SELECT *, col AS alias</code> 解析与投影 · WAL BEGIN/ROLLBACK 写失败窗口修复(事务不泄漏/数据不复活)· aria $in 批级预加载(消除逐值 drainChain 性能悬崖)· 多条件 AND 等值下推(索引真正生效,EXPLAIN 同步)· ANALYZE 统计含二级索引 · React/Vue hooks 生命周期修复(config 变更重建 / 卸载关闭)· 迁移无主键旧库兜底 · 1256 测试 75 套件 · 90.0% 行覆盖率。</p>
|
<strong>v0.7.3 深度审计第六阶段:INSERT 原子 / 索引一致性 / 边界窗口</strong> — INSERT 语句级两阶段原子(三引擎 + Aria PK 批内重复,此前部分提交)· 索引列 IS NULL 恒空修复(Memory/KVStore/Hybrid,对齐 Aria)· delete RESTRICT 预检不再破坏索引 · queryStream 子查询静默空结果修复($subquery/$exists/$col 回退物化)· ALTER DROP 索引列残留清理 · CREATE UNIQUE INDEX 存量重复数据校验(失败原子回滚)· <code>SELECT *, col AS alias</code> 解析与投影 · WAL BEGIN/ROLLBACK 写失败窗口修复(事务不泄漏/数据不复活)· aria $in 批级预加载(消除逐值 drainChain 性能悬崖)· 多条件 AND 等值下推(索引真正生效,EXPLAIN 同步)· ANALYZE 统计含二级索引 · React/Vue hooks 生命周期修复(config 变更重建 / 卸载关闭)· 迁移无主键旧库兜底 · 1256 测试 75 套件 · 90.0% 行覆盖率。<br>
|
||||||
|
<strong>v0.7.4 深度审计第七阶段:写语句子查询 / 约束硬化 / 真惰性流式</strong> — UPDATE/DELETE WHERE 子查询正确执行(四引擎,此前静默 0 行;关联引用显式 NOT_SUPPORTED;EXPLAIN 估算同步修复)· 主键 NULL/undefined 强制拒绝(SQL 语义 PK 隐含 NOT NULL,此前静默生成 "null"/"undefined" 主键)· DROP INDEX 保留建表 UNIQUE 约束(对齐 SQLite:需重建表解除;仅 CREATE UNIQUE INDEX 添加的可随索引删除)· GROUP BY / DISTINCT / UNION 键类型安全编码(null 与 'null' 字符串不再合并)· UPDATE 未知列显式 COLUMN_NOT_FOUND(此前脏列写入存储行)· queryStream 多语句显式 PARSE_ERROR(此前静默忽略后续语句)· KVStore 后台错误跨 reopen 清理 + Hybrid beginTransaction 失败补偿回滚 · <strong>Aria findStream 真惰性</strong>(MergeIterator 迭代器化 + SSTable/MemTable 生成器扫描,limit 提前终止,大表流式内存 O(1)——此前内部 drain 全量物化)· REINDEX 单次全表扫描重建全部索引列(此前每列一次全扫描)· RB-Tree 删除双黑修复边界 + LSM 死代码清理 · 1304 测试 76 套件 · 90.1% 行覆盖率。</p>
|
||||||
|
<strong>v0.8.0 根治性迭代:统一语义 / 消灭复发结构 / 验证基础设施</strong> — 三份行校验实现收敛为唯一 choke point(未知列/NaN 显式拒绝)· 唯一值比较与编码原语 · SQL 与 TABLE API 单管线(QueryBuilder 只产 AST)· CASE 表达式改 token 流递归下降 · 输出列序号与分隔标识符 · <strong>`__aria_manifest_<generation>` 单一提交点</strong>(页面水位 + SSTable 元数据 + 表结构 + WAL 水位 + 冻结表意图一次原子提交;头部/载荷双 CRC、先写后验、保留两代;顺序固定为<strong>数据落盘 → manifest 提交 → 才允许截断 WAL / 删除旧文件</strong>)· 元数据损坏不再静默空库(`ARIA_MANIFEST_CORRUPT` / `ARIA_LEGACY_META_CORRUPT`)· WAL LSN 全库单调 + 按水位删除分片 + 空洞与记录级损坏<strong>如实上报</strong>(`ARIA_WAL_GAP` / `droppedWALRecords`)· LSM 冻结表一等状态、compaction 不再摘整层、底部层原地合并回收墓碑、按层 `compacting`、退休表 + 读者 epoch · 读路径"快照 + 结构版本乐观重试"(删除全部 prefetch 依赖)· checkpoint 不等 compaction(根治 "8~11s 悬崖")· 介质读故障与"文件不存在"分离(`ARIA_SSTABLE_READ_FAILED`,不误删元数据)· 恢复报告 `getRecoveryReport()` · 覆盖率门禁 + 变异验证(40 项)成为标准做法 ✅ v0.8.0
|
||||||
|
|
||||||
<h3>存储模式对比</h3>
|
<h3>存储模式对比</h3>
|
||||||
<table>
|
<table>
|
||||||
<tr><th>特性</th><th>Memory</th><th>Disk (KVStore)</th><th>Hybrid</th><th>Aria</th></tr>
|
<tr><th>特性</th><th>Memory</th><th>Disk (KVStore)</th><th>Hybrid</th><th>Aria</th></tr>
|
||||||
<tr><td>持久化</td><td>❌</td><td>✅ KVStore(OPFS / 内存介质)</td><td>✅ 双写</td><td>✅ 后端决定</td></tr>
|
<tr><td>持久化</td><td>❌</td><td>✅ KVStore(OPFS / 内存介质)</td><td>✅ 双写</td><td>✅ 后端决定</td></tr>
|
||||||
<tr><td>事务</td><td>✅ 快照回滚</td><td>✅ 单日志记录原子写</td><td>✅ 双引擎(磁盘优先)</td><td>✅ MVCC 快照隔离</td></tr>
|
<tr><td>事务</td><td>✅ 快照回滚</td><td>✅ 单日志记录原子写</td><td>✅ 双引擎(磁盘优先)</td><td>✅ 快照回滚(事务串行,非 MVCC 隔离)</td></tr>
|
||||||
<tr><td>索引</td><td>Hash</td><td>Hash(重启恢复)</td><td>Hash</td><td>LSM 二级(重启恢复)</td></tr>
|
<tr><td>索引</td><td>Hash</td><td>Hash(重启恢复)</td><td>Hash</td><td>LSM 二级(重启恢复)</td></tr>
|
||||||
<tr><td>上限</td><td>内存</td><td>磁盘可用</td><td>磁盘可用</td><td>内存</td></tr>
|
<tr><td>上限</td><td>内存</td><td>磁盘可用</td><td>磁盘可用</td><td>内存</td></tr>
|
||||||
<tr><td>多标签页锁</td><td>—</td><td>—</td><td>—</td><td>✅ Web Locks</td></tr>
|
<tr><td>多标签页锁</td><td>—</td><td>—</td><td>—</td><td>✅ Web Locks</td></tr>
|
||||||
@@ -819,8 +835,8 @@ db.<span class="f">broadcastChange</span>(<span class="s">'users'</span>);</pre>
|
|||||||
<tr><td><strong>LSM-Tree 索引</strong></td><td>MemTable (红黑树) + 多级 SSTable,写优化,支持范围扫描</td></tr>
|
<tr><td><strong>LSM-Tree 索引</strong></td><td>MemTable (红黑树) + 多级 SSTable,写优化,支持范围扫描</td></tr>
|
||||||
<tr><td><strong>4KB 页面化物理存储</strong></td><td>SSTable 切分为 4KB 页面(FileManager 分配 pageId + BufferPool LRU 缓存 256 页 ≈ 1MB),save 即落盘,`SSTableMeta.pageIds` 持久化(旧整 value 数据兼容)✅ v0.5.0</td></tr>
|
<tr><td><strong>4KB 页面化物理存储</strong></td><td>SSTable 切分为 4KB 页面(FileManager 分配 pageId + BufferPool LRU 缓存 256 页 ≈ 1MB),save 即落盘,`SSTableMeta.pageIds` 持久化(旧整 value 数据兼容)✅ v0.5.0</td></tr>
|
||||||
<tr><td><strong>Buffer Pool</strong></td><td>LRU 页面缓存,可控内存占用(默认 256 页 ≈ 1MB),脏页写回 ✅ v0.5.0 真实接入</td></tr>
|
<tr><td><strong>Buffer Pool</strong></td><td>LRU 页面缓存,可控内存占用(默认 256 页 ≈ 1MB),脏页写回 ✅ v0.5.0 真实接入</td></tr>
|
||||||
<tr><td><strong>WAL 日志</strong></td><td>分片文件(`__wal_%06d.bin`,4MB 阈值切换 + 真追加)· 标准 CRC32 记录校验 · full/batch/none 三种同步模式(full 真正同步)· 空洞检测截断 · 16MB 阈值自动 checkpoint ✅ v0.5.0</td></tr>
|
<tr><td><strong>WAL 日志</strong></td><td>分片文件(`__wal_%06d.bin`,4MB 阈值切换 + 真追加)· 标准 CRC32 记录校验(记录级 CRC 失败 → <code>droppedWALRecords</code> + 数据丢失标记,不再只打日志)· full/batch/none 三种同步模式(full 真正同步)· 空洞如实上报(内部空洞 → <code>ARIA_WAL_GAP</code> 且拒绝静默继续) · 16MB 阈值自动 checkpoint ✅ v0.8.0 补齐</td></tr>
|
||||||
<tr><td><strong>MVCC 事务</strong></td><td>快照隔离 (Snapshot Isolation),读写不互斥,版本链 + GC(引擎事务读取走 txnSnapshot,MVCC 版本链作 undo log)</td></tr>
|
<tr><td><strong>MVCC 版本链</strong></td><td>版本链仅作事务内 undo(提交即清理,自动 GC),<strong>不提供快照隔离</strong>;引擎事务读取走 txnSnapshot,同一实例同时只允许一个事务(并发 begin 抛 TX_ACTIVE)✅ v0.8.0 如实描述</td></tr>
|
||||||
<tr><td><strong>全库 AES-GCM 加密</strong></td><td>EncryptedBackend 透明加解密(WAL/SSTable/Schema/元数据全密文)· PBKDF2 密钥派生 + salt 持久化 + 密码验证 · 明文/加密库开关一致性检测 ✅ v0.5.0</td></tr>
|
<tr><td><strong>全库 AES-GCM 加密</strong></td><td>EncryptedBackend 透明加解密(WAL/SSTable/Schema/元数据全密文)· PBKDF2 密钥派生 + salt 持久化 + 密码验证 · 明文/加密库开关一致性检测 ✅ v0.5.0</td></tr>
|
||||||
<tr><td><strong>多标签页独占锁</strong></td><td>Web Locks API:第二个标签页打开同一库抛 `ARIA_LOCKED`,open 失败自动释放 ✅ v0.5.0</td></tr>
|
<tr><td><strong>多标签页独占锁</strong></td><td>Web Locks API:第二个标签页打开同一库抛 `ARIA_LOCKED`,open 失败自动释放 ✅ v0.5.0</td></tr>
|
||||||
<tr><td><strong>Bloom Filter</strong></td><td>快速判定 key 不存在,减少无效磁盘 I/O,SSTableReader 二分查找统一 ✅ v0.2.5</td></tr>
|
<tr><td><strong>Bloom Filter</strong></td><td>快速判定 key 不存在,减少无效磁盘 I/O,SSTableReader 二分查找统一 ✅ v0.2.5</td></tr>
|
||||||
@@ -863,6 +879,7 @@ db.<span class="f">broadcastChange</span>(<span class="s">'users'</span>);</pre>
|
|||||||
<tr><td><code>walSizeThreshold</code></td><td><code>number</code></td><td><code>16777216</code></td><td>WAL 大小阈值(字节),超阈值触发 checkpoint ✅ v0.2.5</td></tr>
|
<tr><td><code>walSizeThreshold</code></td><td><code>number</code></td><td><code>16777216</code></td><td>WAL 大小阈值(字节),超阈值触发 checkpoint ✅ v0.2.5</td></tr>
|
||||||
<tr><td><code>compression</code></td><td><code>boolean</code></td><td><code>false</code></td><td>是否启用页面压缩</td></tr>
|
<tr><td><code>compression</code></td><td><code>boolean</code></td><td><code>false</code></td><td>是否启用页面压缩</td></tr>
|
||||||
<tr><td><code>storageBackend</code></td><td><code>'opfs'|'memory'|'kv'</code></td><td><code>'opfs'</code></td><td>存储后端类型('kv' = 自研 KVStore 后端 🆕 v0.6.1)</td></tr>
|
<tr><td><code>storageBackend</code></td><td><code>'opfs'|'memory'|'kv'</code></td><td><code>'opfs'</code></td><td>存储后端类型('kv' = 自研 KVStore 后端 🆕 v0.6.1)</td></tr>
|
||||||
|
<tr><td><code>maxMemoryMB</code></td><td><code>number</code></td><td><code>64</code></td><td>内存预算(MB):主 LSM 估算内存超限时触发 flush + MVCC GC</td></tr>
|
||||||
<tr><td><code>encryption</code></td><td><code>{ password: string }</code></td><td>-</td><td>全库 AES-256-GCM 透明加密(PBKDF2 派生 + salt 持久化 + 密码验证)🆕 v0.5.0</td></tr>
|
<tr><td><code>encryption</code></td><td><code>{ password: string }</code></td><td>-</td><td>全库 AES-256-GCM 透明加密(PBKDF2 派生 + salt 持久化 + 密码验证)🆕 v0.5.0</td></tr>
|
||||||
<tr><td><code>pageStorage</code></td><td><code>boolean</code></td><td>opfs/kv 自动启用</td><td>SSTable 4KB 页面化物理存储(BufferPool 缓存)🆕 v0.5.0</td></tr>
|
<tr><td><code>pageStorage</code></td><td><code>boolean</code></td><td>opfs/kv 自动启用</td><td>SSTable 4KB 页面化物理存储(BufferPool 缓存)🆕 v0.5.0</td></tr>
|
||||||
</table>
|
</table>
|
||||||
@@ -886,6 +903,7 @@ db.<span class="f">broadcastChange</span>(<span class="s">'users'</span>);</pre>
|
|||||||
<tr><td><code>TABLE_EXISTS</code></td><td>表已存在</td></tr>
|
<tr><td><code>TABLE_EXISTS</code></td><td>表已存在</td></tr>
|
||||||
<tr><td><code>DUPLICATE_KEY</code></td><td>主键重复</td></tr>
|
<tr><td><code>DUPLICATE_KEY</code></td><td>主键重复</td></tr>
|
||||||
<tr><td><code>UNIQUE_VIOLATION</code></td><td>唯一约束冲突</td></tr>
|
<tr><td><code>UNIQUE_VIOLATION</code></td><td>唯一约束冲突</td></tr>
|
||||||
|
<tr><td><code>FOREIGN_KEY_VIOLATION</code></td><td>外键约束冲突:删除/更新被引用行时存在依赖行(含 RESTRICT / CASCADE 语义)</td></tr>
|
||||||
<tr><td><code>VALIDATION_ERROR</code></td><td>数据校验失败</td></tr>
|
<tr><td><code>VALIDATION_ERROR</code></td><td>数据校验失败</td></tr>
|
||||||
<tr><td><code>TYPE_ERROR</code></td><td>字段类型错误</td></tr>
|
<tr><td><code>TYPE_ERROR</code></td><td>字段类型错误</td></tr>
|
||||||
<tr><td><code>SCHEMA_ERROR</code></td><td>表结构定义错误</td></tr>
|
<tr><td><code>SCHEMA_ERROR</code></td><td>表结构定义错误</td></tr>
|
||||||
@@ -903,6 +921,28 @@ db.<span class="f">broadcastChange</span>(<span class="s">'users'</span>);</pre>
|
|||||||
<tr><td><code>NOT_SUPPORTED</code></td><td>引擎不支持的操作(如 Memory 引擎执行 ANALYZE / VACUUM / SAVEPOINT)🆕 v0.5.1</td></tr>
|
<tr><td><code>NOT_SUPPORTED</code></td><td>引擎不支持的操作(如 Memory 引擎执行 ANALYZE / VACUUM / SAVEPOINT)🆕 v0.5.1</td></tr>
|
||||||
<tr><td><code>PARAM_ERROR</code></td><td>参数绑定错误:数量不匹配 / 对象数组参数 🆕 v0.7.0</td></tr>
|
<tr><td><code>PARAM_ERROR</code></td><td>参数绑定错误:数量不匹配 / 对象数组参数 🆕 v0.7.0</td></tr>
|
||||||
<tr><td><code>QUERY_ERROR</code></td><td>未知 where 操作符(如拼错的 <code>$betwen</code>)🆕 v0.7.2</td></tr>
|
<tr><td><code>QUERY_ERROR</code></td><td>未知 where 操作符(如拼错的 <code>$betwen</code>)🆕 v0.7.2</td></tr>
|
||||||
|
<tr><td><code>COLUMN_NOT_FOUND</code></td><td>列不存在:UPDATE 未知列 / ALTER DROP 不存在列(v0.7.4 UPDATE 未知列显式报错)</td></tr>
|
||||||
|
<tr><td><code>INDEX_NOT_FOUND</code></td><td>DROP 不存在的索引</td></tr>
|
||||||
|
<tr><td><code>ARIA_MANIFEST_CORRUPT</code></td><td>存储元数据(manifest)全部世代校验失败:拒绝打开,而不是当成空库 🆕 v0.8.0</td></tr>
|
||||||
|
<tr><td><code>ARIA_MANIFEST_WRITE_FAILED</code></td><td>manifest 提交后回读校验失败(提交未生效,内存态不前进)🆕 v0.8.0</td></tr>
|
||||||
|
<tr><td><code>ARIA_LEGACY_META_CORRUPT</code></td><td>旧格式(v0.8.0 之前)元数据损坏,无法安全迁移 🆕 v0.8.0</td></tr>
|
||||||
|
<tr><td><code>ARIA_SSTABLE_READ_FAILED</code></td><td>介质读故障(区别于"文件不存在":不删元数据、不回退成静默空结果)🆕 v0.8.0</td></tr>
|
||||||
|
<tr><td><code>ARIA_WAL_GAP</code></td><td>WAL 分片空洞(含前缀缺失):拒绝在"少了一段日志"的情况下静默继续 🆕 v0.8.0</td></tr>
|
||||||
|
<tr><td><code>ARIA_WRITE_LOST</code></td><td>manifest 声称有未落盘数据,但 WAL 中没有任何可重放的记录(已确认写入确实丢失)🆕 v0.8.0</td></tr>
|
||||||
|
<tr><td><code>STALE_INSTANCE</code></td><td>陈旧实例拒绝提交(另一个实例已提交更新的世代),不会静默覆盖 🆕 v0.8.0</td></tr>
|
||||||
|
<tr><td><code>ARIA_MANIFEST_NOT_LOADED</code></td><td>未 <code>load()</code> 就提交 manifest:拒绝写坏介质 🆕 v0.8.0</td></tr>
|
||||||
|
<tr><td><code>ARIA_SSTABLE_SAVE_CONTRACT</code></td><td>存储实现违反 <code>save()</code> 契约(既不抛错也不返回结果):不把"写调用返回了"当成落盘成功 🆕 v0.8.0</td></tr>
|
||||||
|
<tr><td><code>ARIA_DB_NOT_OPEN</code></td><td>在 <code>open()</code> 之前使用 Aria 存储层(如加密后端) 🆕 v0.8.0</td></tr>
|
||||||
|
<tr><td><code>ARIA_OPEN_ERROR</code></td><td>AriaEngine 打开失败(底层错误保留在 <code>err.details</code> 与标准 <code>err.cause</code> 上,便于定位根因) 🆕 v0.8.0</td></tr>
|
||||||
|
<tr><td><code>ARIA_OPFS_UNAVAILABLE</code></td><td>OPFS 在当前环境不可用(如 <code>file://</code> 页面被 Chromium 禁止访问 OPFS):错误信息给出可操作建议(改用 http(s) 或 <code>mode:'memory'</code>) 🆕 v0.8.0</td></tr>
|
||||||
|
<tr><td><code>DB_NOT_OPEN</code></td><td>数据库未打开(KVStoreEngine 操作前未 <code>open()</code>)</td></tr>
|
||||||
|
<tr><td><code>KV_SCHEMA_ERROR</code></td><td>KVStore 中的表结构记录损坏</td></tr>
|
||||||
|
<tr><td><code>KV_LOG_ERROR</code> / <code>KV_BACKGROUND_ERROR</code></td><td>KVStore 日志写入失败 / 后台刷盘失败</td></tr>
|
||||||
|
<tr><td><code>TX_NONE</code></td><td>没有活跃事务时执行 COMMIT / ROLLBACK</td></tr>
|
||||||
|
<tr><td><code>TX_ACTIVE</code></td><td>已有事务进行中时再次 BEGIN</td></tr>
|
||||||
|
<tr><td><code>TX_COMMIT_ERROR</code></td><td>hybrid 模式提交失败:磁盘侧已提交、内存侧失败(错误信息说明磁盘数据已落盘)</td></tr>
|
||||||
|
<tr><td><code>UNKNOWN_STATEMENT</code></td><td>执行器遇到未知语句类型</td></tr>
|
||||||
|
<tr><td><code>COLUMN_EXISTS</code></td><td>ADD COLUMN 的列已存在</td></tr>
|
||||||
</table>
|
</table>
|
||||||
|
|
||||||
</main>
|
</main>
|
||||||
|
|||||||
+15
-15
@@ -153,9 +153,9 @@
|
|||||||
<!-- Hero -->
|
<!-- Hero -->
|
||||||
<section class="hero">
|
<section class="hero">
|
||||||
<div class="container">
|
<div class="container">
|
||||||
<div class="badge" style="margin-bottom:24px;"><span class="dot"></span> v0.7.3 数据正确性深度审计 — 1256测试 75套件 · 90.0% 覆盖率 · INSERT/UPDATE 语句级原子 · 参数化查询 · 自研 KVStore 事务引擎 · AriaEngine LSM+WAL+MVCC · 崩溃恢复</div>
|
<div class="badge" style="margin-bottom:24px;"><span class="dot"></span> v0.8.0 根治性迭代 — 1985 测试 93 套件 · 语句/分支/函数/行覆盖率 90.59% / 82.61% / 94.14% / 93.50% · UPDATE/DELETE 子查询正确执行 · 主键非空强制 · DROP INDEX 保留 UNIQUE 约束 · findStream 真惰性 · 参数化查询 · 自研 KVStore 事务引擎 · AriaEngine LSM+WAL+MVCC · 崩溃恢复</div>
|
||||||
<h1>前端的 <span class="gradient-text">SQL 数据库</span></h1>
|
<h1>前端的 <span class="gradient-text">SQL 数据库</span></h1>
|
||||||
<p>TypeScript 原生构建,5 种存储引擎,支持完整 SQL 查询。<br>零运行时依赖,开箱即用。AriaEngine 自研引擎:LSM-Tree + WAL 同步 + MVCC。</p>
|
<p>TypeScript 原生构建,4 种存储模式 + 3 种存储后端,支持完整 SQL 查询。<br>零运行时依赖,开箱即用。AriaEngine 自研引擎:LSM-Tree + WAL 同步 + MVCC。</p>
|
||||||
<div class="actions">
|
<div class="actions">
|
||||||
<a href="demo.html" class="btn btn-primary" style="font-size:1.05rem;padding:14px 32px;">▶ 在线演示</a>
|
<a href="demo.html" class="btn btn-primary" style="font-size:1.05rem;padding:14px 32px;">▶ 在线演示</a>
|
||||||
<a href="docs.html" class="btn btn-outline" style="font-size:1.05rem;padding:14px 32px;">📖 API 文档</a>
|
<a href="docs.html" class="btn btn-outline" style="font-size:1.05rem;padding:14px 32px;">📖 API 文档</a>
|
||||||
@@ -233,8 +233,8 @@ npm install @metona-team/metona-sqlark
|
|||||||
<div class="feature-grid">
|
<div class="feature-grid">
|
||||||
<div class="feature-card">
|
<div class="feature-card">
|
||||||
<div class="icon">🧠</div>
|
<div class="icon">🧠</div>
|
||||||
<h3>5 种存储引擎</h3>
|
<h3>4 种存储模式 + 3 种后端</h3>
|
||||||
<p>Memory / KVStore / OPFS / Hybrid / <strong>AriaEngine</strong>。Aria 是自研 LSM-Tree 页面式引擎,支持 WAL 同步崩溃恢复和 MVCC 事务隔离。</p>
|
<p>模式:Memory / Disk / Hybrid / <strong>AriaEngine</strong>;后端:OPFS / KVStore / Memory。Aria 是自研 LSM-Tree 页面式引擎,支持 WAL 同步崩溃恢复与事务回滚(版本链仅作事务内 undo,事务串行 —— 非 MVCC 快照隔离)。</p>
|
||||||
</div>
|
</div>
|
||||||
<div class="feature-card">
|
<div class="feature-card">
|
||||||
<div class="icon">⚡</div>
|
<div class="icon">⚡</div>
|
||||||
@@ -243,8 +243,8 @@ npm install @metona-team/metona-sqlark
|
|||||||
</div>
|
</div>
|
||||||
<div class="feature-card">
|
<div class="feature-card">
|
||||||
<div class="icon">🌲</div>
|
<div class="icon">🌲</div>
|
||||||
<h3>AriaEngine <span style="font-size:0.65rem;color:var(--accent);vertical-align:super;">v0.7.3</span></h3>
|
<h3>AriaEngine <span style="font-size:0.65rem;color:var(--accent);vertical-align:super;">v0.8.0</span></h3>
|
||||||
<p>自研 LSM-Tree 存储引擎:SSTable 4KB 页面化物理存储(BufferPool LRU)、WAL 分片文件原子写入、全库 AES-GCM 透明加密、标准 CRC-32 完整性校验、MVCC 快照隔离、二级索引跨重启恢复、ON UPDATE/DELETE 外键级联、崩溃恢复自愈。</p>
|
<p>自研 LSM-Tree 存储引擎:SSTable 4KB 页面化物理存储(BufferPool LRU)、WAL 分片文件原子写入、<strong>__aria_manifest 单一提交点</strong>(数据落盘 → 元数据原子提交 → 才截断 WAL)、全库 AES-GCM 透明加密、标准 CRC-32 完整性校验、MVCC 快照回滚、二级索引跨重启恢复、ON UPDATE/DELETE 外键级联、崩溃恢复自愈。</p>
|
||||||
</div>
|
</div>
|
||||||
<div class="feature-card">
|
<div class="feature-card">
|
||||||
<div class="icon">🔒</div>
|
<div class="icon">🔒</div>
|
||||||
@@ -264,7 +264,7 @@ npm install @metona-team/metona-sqlark
|
|||||||
<div class="feature-card">
|
<div class="feature-card">
|
||||||
<div class="icon">🛡</div>
|
<div class="icon">🛡</div>
|
||||||
<h3>崩溃恢复自愈</h3>
|
<h3>崩溃恢复自愈</h3>
|
||||||
<p>异常退出后无需删库重建:整文件 CRC-32 校验 + 打开自动跳过损坏 SSTable,db.repair() 清理损坏数据/孤儿页面/残留文件并重建索引,db.clearAll() 重置。迁移版本持久化,重启不重跑。</p>
|
<p>异常退出后无需删库重建:整文件 CRC-32 校验 + 打开自动跳过损坏 SSTable,并按 WAL 水位重放已提交事务;<code>getRecoveryReport()</code> 如实给出被丢弃的 SSTable、WAL 空洞与是否怀疑丢数据。介质<strong>读故障</strong>与"文件不存在"分开处理(前者抛 ARIA_SSTABLE_READ_FAILED,绝不误删元数据);元数据整体损坏时显式拒绝打开(ARIA_MANIFEST_CORRUPT),而不是当成空库。db.repair() 只在 manifest 健康时回收孤儿页面并重建索引。</p>
|
||||||
</div>
|
</div>
|
||||||
<div class="feature-card">
|
<div class="feature-card">
|
||||||
<div class="icon">🏊</div>
|
<div class="icon">🏊</div>
|
||||||
@@ -279,7 +279,7 @@ npm install @metona-team/metona-sqlark
|
|||||||
<div class="feature-card">
|
<div class="feature-card">
|
||||||
<div class="icon">📦</div>
|
<div class="icon">📦</div>
|
||||||
<h3>零运行时依赖</h3>
|
<h3>零运行时依赖</h3>
|
||||||
<p>纯 TypeScript 实现,不依赖任何第三方库。Tree-shakable,UMD/ESM/CJS 多格式输出,~27KB gzip。</p>
|
<p>纯 TypeScript 实现,不依赖任何第三方库。UMD/ESM/CJS 多格式输出,min 产物实测 251,731 字节 / gzip 63,431 字节。</p>
|
||||||
</div>
|
</div>
|
||||||
<div class="feature-card">
|
<div class="feature-card">
|
||||||
<div class="icon">📊</div>
|
<div class="icon">📊</div>
|
||||||
@@ -299,7 +299,7 @@ npm install @metona-team/metona-sqlark
|
|||||||
<div class="feature-card">
|
<div class="feature-card">
|
||||||
<div class="icon">🔄</div>
|
<div class="icon">🔄</div>
|
||||||
<h3>数据迁移</h3>
|
<h3>数据迁移</h3>
|
||||||
<p>内置版本迁移系统,addMigration + migrateTo API。导入导出支持单表/全库 JSON 序列化,db.backup() 在线一致性备份。</p>
|
<p>内置版本迁移系统,addMigration + migrateTo API。导入导出支持单表/全库 JSON 序列化,db.backup() 全库导出(逐表读取)。</p>
|
||||||
</div>
|
</div>
|
||||||
<div class="feature-card">
|
<div class="feature-card">
|
||||||
<div class="icon">📡</div>
|
<div class="icon">📡</div>
|
||||||
@@ -314,7 +314,7 @@ npm install @metona-team/metona-sqlark
|
|||||||
<div class="feature-card">
|
<div class="feature-card">
|
||||||
<div class="icon">🚰</div>
|
<div class="icon">🚰</div>
|
||||||
<h3>流式查询 + 派生表</h3>
|
<h3>流式查询 + 派生表</h3>
|
||||||
<p>queryStream 逐行回调不物化结果集(大表友好)。FROM (SELECT ...) 派生表、多列哈希连接、COUNT(DISTINCT)、NULLS FIRST/LAST。</p>
|
<p>queryStream 逐行回调真惰性扫描(v0.7.4:Aria 引擎迭代器化,limit 提前终止,大表内存 O(1))。FROM (SELECT ...) 派生表、多列哈希连接、COUNT(DISTINCT)、NULLS FIRST/LAST。</p>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
@@ -414,12 +414,12 @@ npm install @metona-team/metona-sqlark
|
|||||||
<p>MetonaSqlark 的核心指标</p>
|
<p>MetonaSqlark 的核心指标</p>
|
||||||
</div>
|
</div>
|
||||||
<div class="stats">
|
<div class="stats">
|
||||||
<div class="stat-card"><div class="num">1256</div><div class="label">测试用例</div></div>
|
<div class="stat-card"><div class="num">1985</div><div class="label">测试用例(+14 e2e)</div></div>
|
||||||
<div class="stat-card"><div class="num">90.0%</div><div class="label">行覆盖率</div></div>
|
<div class="stat-card"><div class="num">93.50%</div><div class="label">行覆盖率</div></div>
|
||||||
<div class="stat-card"><div class="num">~27KB</div><div class="label">gzip 体积</div></div>
|
<div class="stat-card"><div class="num">63KB</div><div class="label">gzip 体积</div></div>
|
||||||
<div class="stat-card"><div class="num">5</div><div class="label">存储引擎</div></div>
|
<div class="stat-card"><div class="num">4+3</div><div class="label">存储模式+后端</div></div>
|
||||||
<div class="stat-card"><div class="num">72</div><div class="label">SQL 关键字</div></div>
|
<div class="stat-card"><div class="num">72</div><div class="label">SQL 关键字</div></div>
|
||||||
<div class="stat-card"><div class="num">75</div><div class="label">测试套件</div></div>
|
<div class="stat-card"><div class="num">93</div><div class="label">测试套件</div></div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</section>
|
</section>
|
||||||
|
|||||||
@@ -51,8 +51,8 @@ class ConnectionManager {
|
|||||||
this.connections.set(name, db);
|
this.connections.set(name, db);
|
||||||
this.refCount.set(name, 1);
|
this.refCount.set(name, 1);
|
||||||
|
|
||||||
// 注入 disconnect 方法
|
// 注入 disconnect 方法(类型已在 MetonaSqlark 上声明,无需 any 断言)
|
||||||
(db as MetonaSqlark & { disconnect: () => Promise<void> }).disconnect = async () => {
|
db.disconnect = async () => {
|
||||||
await this.release(name);
|
await this.release(name);
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -114,12 +114,11 @@ class ConnectionManager {
|
|||||||
|
|
||||||
const manager = new ConnectionManager();
|
const manager = new ConnectionManager();
|
||||||
|
|
||||||
// 挂载到 MetonaSqlark 静态方法(通过 any 绕过 TS 类型检查)
|
// 挂载到 MetonaSqlark 静态方法(v0.8.0:类型已在类上声明,无需 any 断言)
|
||||||
const M = MetonaSqlark as unknown as Record<string, unknown>;
|
MetonaSqlark.connect = (config: DatabaseConfig) => manager.connect(config);
|
||||||
M.connect = (config: DatabaseConfig) => manager.connect(config);
|
MetonaSqlark.disconnect = (dbName: string) => manager.release(dbName);
|
||||||
M.disconnect = (dbName: string) => manager.release(dbName);
|
MetonaSqlark.disconnectAll = () => manager.closeAll();
|
||||||
M.disconnectAll = () => manager.closeAll();
|
MetonaSqlark.getActiveConnections = () => manager.getActiveConnections();
|
||||||
M.getActiveConnections = () => manager.getActiveConnections();
|
|
||||||
|
|
||||||
export { manager as connectionManager };
|
export { manager as connectionManager };
|
||||||
export default manager;
|
export default manager;
|
||||||
|
|||||||
+38
-3
@@ -73,7 +73,14 @@ export interface DatabaseConfig {
|
|||||||
name: string;
|
name: string;
|
||||||
/** 存储模式 */
|
/** 存储模式 */
|
||||||
mode?: StorageMode;
|
mode?: StorageMode;
|
||||||
/** 磁盘引擎(仅 mode='disk'|'hybrid' 时生效) */
|
/**
|
||||||
|
* 磁盘后端选择。
|
||||||
|
*
|
||||||
|
* v0.8.0 修正注释:**仅 `mode: 'aria'` 真正生效**(作为 Aria 的存储后端,
|
||||||
|
* 见 core.ts 的 aria 分支)。`disk` 模式恒用自研 KVStoreEngine,
|
||||||
|
* `hybrid` 的内存+磁盘组合也恒用 KVStoreEngine —— 两者会忽略本项
|
||||||
|
*(此前注释写成"disk 模式生效",与实现相反)。
|
||||||
|
*/
|
||||||
diskEngine?: DiskEngine;
|
diskEngine?: DiskEngine;
|
||||||
/** 版本号 */
|
/** 版本号 */
|
||||||
version?: number;
|
version?: number;
|
||||||
@@ -186,7 +193,14 @@ export interface MetonaPlugin {
|
|||||||
version: string;
|
version: string;
|
||||||
/** 描述 */
|
/** 描述 */
|
||||||
description?: string;
|
description?: string;
|
||||||
/** 优先级,越大越先执行 */
|
/**
|
||||||
|
* 优先级:**越大越先执行**(含 `install()` 与钩子触发顺序)。
|
||||||
|
*
|
||||||
|
* v0.8.0 起真正生效 —— Core 会先按 priority 降序稳定排序再注册插件
|
||||||
|
*(同优先级保持 config 数组顺序)。此前 register() 虽按优先级插入数组,
|
||||||
|
* 但 install() 在 register 内立即执行,实际顺序 = config 数组顺序。
|
||||||
|
*/
|
||||||
|
|
||||||
priority?: number;
|
priority?: number;
|
||||||
/** 安装 */
|
/** 安装 */
|
||||||
install(db: unknown): void;
|
install(db: unknown): void;
|
||||||
@@ -200,6 +214,15 @@ export interface MetonaPlugin {
|
|||||||
|
|
||||||
/** 数据库错误 */
|
/** 数据库错误 */
|
||||||
export class DatabaseError extends Error {
|
export class DatabaseError extends Error {
|
||||||
|
/**
|
||||||
|
* 底层错误(第三个构造参数)。
|
||||||
|
*
|
||||||
|
* v0.8.0:显式声明为公开字段 —— target 是 ES2020(lib 里 `Error` 还没有
|
||||||
|
* `cause`),不声明的话 TypeScript 使用者读 `err.cause` 会编译失败,
|
||||||
|
* 而运行时它确实存在。
|
||||||
|
*/
|
||||||
|
cause?: unknown;
|
||||||
|
|
||||||
constructor(
|
constructor(
|
||||||
message: string,
|
message: string,
|
||||||
public code: string,
|
public code: string,
|
||||||
@@ -207,6 +230,18 @@ export class DatabaseError extends Error {
|
|||||||
) {
|
) {
|
||||||
super(message);
|
super(message);
|
||||||
this.name = 'DatabaseError';
|
this.name = 'DatabaseError';
|
||||||
|
// v0.8.0(review 修复):把底层错误同时挂到**标准** `Error.cause` 上。
|
||||||
|
//
|
||||||
|
// 修复前第三个参数只存进 `details`,`err.cause` 恒为 undefined —— 而文档与
|
||||||
|
// 代码注释多处写的是"底层错误作为 cause 保留"(实测被打脸:探针打印
|
||||||
|
// `e.cause?.code` 得到 undefined)。`details` 语义保持不变(向后兼容),
|
||||||
|
// `cause` 让标准错误链工具(日志/监控/Node 的 error.cause 约定)能看到根因。
|
||||||
|
//
|
||||||
|
// 注意:target 是 ES2020(lib 无 ES2022 的 ErrorOptions),因此用赋值而不是
|
||||||
|
// `super(message, { cause })`;运行时所有现代引擎都支持该属性。
|
||||||
|
if (details !== undefined) {
|
||||||
|
(this as { cause?: unknown }).cause = details;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -214,4 +249,4 @@ export class DatabaseError extends Error {
|
|||||||
// 版本
|
// 版本
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
export const VERSION = '0.7.3';
|
export const VERSION = '0.8.0';
|
||||||
|
|||||||
+274
-70
@@ -6,6 +6,8 @@
|
|||||||
*/
|
*/
|
||||||
|
|
||||||
import type { IStorageEngine } from './engine/interface';
|
import type { IStorageEngine } from './engine/interface';
|
||||||
|
import { ChangeNotifierEngine } from './engine/change-notifier';
|
||||||
|
import type { ChangeEvent } from './engine/change-notifier';
|
||||||
import type { DatabaseConfig, ColumnDef } from './constants';
|
import type { DatabaseConfig, ColumnDef } from './constants';
|
||||||
import { DB_DEFAULTS, DatabaseError } from './constants';
|
import { DB_DEFAULTS, DatabaseError } from './constants';
|
||||||
import { MemoryEngine } from './engine/memory';
|
import { MemoryEngine } from './engine/memory';
|
||||||
@@ -29,6 +31,38 @@ export class MetonaSqlark {
|
|||||||
/** 数据库名称 */
|
/** 数据库名称 */
|
||||||
readonly name: string;
|
readonly name: string;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0:释放一个连接引用(引用计数 -1,归零时自动关闭)。
|
||||||
|
*
|
||||||
|
* 由 `MetonaSqlark.connect()` 注入实现 —— 此前该方法是**运行时注入、类型上不存在**:
|
||||||
|
* README 与示例都在用 `await db.disconnect()`,但 `db` 的声明里没有它,
|
||||||
|
* TypeScript 使用者会直接编译失败(只能 `as any` 绕过)。
|
||||||
|
* 普通 `create()` 得到的实例没有这个方法,因此为可选:
|
||||||
|
* 只有经 `connect()` 取得的实例才有,直接调用会抛错(而不是静默无操作)。
|
||||||
|
*/
|
||||||
|
disconnect?: () => Promise<void>;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0:连接池静态 API 的类型声明。
|
||||||
|
*
|
||||||
|
* 这些方法由 `src/connection-manager.ts` **运行时注入**(`MetonaSqlark.connect = ...`)。
|
||||||
|
* 此前注入侧用 `as unknown as Record<string, unknown>` 绕过类型检查,
|
||||||
|
* 于是 README「连接池」一节里的 `MetonaSqlark.connect(...)` /
|
||||||
|
* `MetonaSqlark.disconnectAll()` 在 TypeScript 下全部报 TS2339
|
||||||
|
*("属性不存在"),使用者只能 `as any`。
|
||||||
|
*
|
||||||
|
* 声明为 `?` 可选是因为它们**只在 import 了 connection-manager 的构建里存在**:
|
||||||
|
* 核心入口不 import 它(避免无谓的模块副作用)。真正常用的路径是
|
||||||
|
* `MetonaSqlark.create()`。
|
||||||
|
*/
|
||||||
|
static connect?: (config: DatabaseConfig) => Promise<MetonaSqlark>;
|
||||||
|
/** 按库名释放一个连接引用(等价于实例上的 `disconnect()`) */
|
||||||
|
static disconnect?: (dbName: string) => Promise<void>;
|
||||||
|
/** 关闭全部连接 */
|
||||||
|
static disconnectAll?: () => Promise<void>;
|
||||||
|
/** 当前活跃连接名列表 */
|
||||||
|
static getActiveConnections?: () => string[];
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* v0.7.1: 静态工厂(与 connect/disconnect 同一入口风格)。
|
* v0.7.1: 静态工厂(与 connect/disconnect 同一入口风格)。
|
||||||
* 此前 create 仅存在于 api 对象 / window 挂载 —— README/站点示例的
|
* 此前 create 仅存在于 api 对象 / window 挂载 —— README/站点示例的
|
||||||
@@ -81,10 +115,17 @@ export class MetonaSqlark {
|
|||||||
this.channel.onmessage = (event) => {
|
this.channel.onmessage = (event) => {
|
||||||
const msg = event.data as { type?: string; table?: string } | null;
|
const msg = event.data as { type?: string; table?: string } | null;
|
||||||
if (!msg || msg.type !== 'change') return;
|
if (!msg || msg.type !== 'change') return;
|
||||||
this.emit(msg.table ?? '', { type: 'external', table: msg.table ?? '' });
|
// v0.8.0: 外部事件只派发给本地订阅者,**不得再次广播** ——
|
||||||
// Hybrid 引擎:从磁盘重载内存,保证读到其他标签页的最新数据
|
// 否则两个标签页会互相转发形成无限广播循环(实测 8 次以上且不终止)。
|
||||||
if (this.engine instanceof HybridEngine) {
|
void this.emitExternal(msg.table ?? '');
|
||||||
(this.engine as HybridEngine).reloadMemoryFromDisk().catch(() => {
|
// Hybrid 引擎:从磁盘重载内存,保证读到其他标签页的最新数据。
|
||||||
|
//
|
||||||
|
// v0.8.0: 引擎现在被 ChangeNotifierEngine 装饰,`this.engine instanceof HybridEngine`
|
||||||
|
// 恒为 false —— 因此改为对**内层**引擎做能力探测。这也是审计指出的
|
||||||
|
// "用 instanceof 做引擎特判"的隐患:装饰器一加就静默失效。
|
||||||
|
const inner = this.unwrapEngine();
|
||||||
|
if (inner instanceof HybridEngine) {
|
||||||
|
(inner as HybridEngine).reloadMemoryFromDisk().catch(() => {
|
||||||
// 重载失败不影响主流程(下次读可能短暂过期)
|
// 重载失败不影响主流程(下次读可能短暂过期)
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
@@ -102,6 +143,28 @@ export class MetonaSqlark {
|
|||||||
// 打开连接
|
// 打开连接
|
||||||
await this.engine.open(this.name, this.version);
|
await this.engine.open(this.name, this.version);
|
||||||
|
|
||||||
|
// v0.8.0(A9):把引擎包进变更通知装饰器 —— **唯一**的变更事件汇聚点。
|
||||||
|
// 三个写入入口(SQL / Table API / QueryBuilder)与事务内写入都必须经过引擎接口,
|
||||||
|
// 因此在这里拦一次即可全覆盖,避免在三条路径上各写一份"变更描述"逻辑。
|
||||||
|
this.engine = new ChangeNotifierEngine(
|
||||||
|
this.engine,
|
||||||
|
(error) => this._onError(error),
|
||||||
|
(table) => this.broadcastChange(table),
|
||||||
|
);
|
||||||
|
this.notifier = this.engine as ChangeNotifierEngine;
|
||||||
|
// 把引擎层变更事件接入 db.subscribe 的订阅表(listeners)
|
||||||
|
this.notifier.addListener(async (event) => {
|
||||||
|
const set = this.listeners.get(`change:${event.table}`);
|
||||||
|
if (!set || set.size === 0) return;
|
||||||
|
for (const cb of [...set]) {
|
||||||
|
try {
|
||||||
|
await (cb as (e: ChangeEvent) => void | Promise<void>)(event);
|
||||||
|
} catch (error) {
|
||||||
|
this._onError(error as Error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
// v0.4.2-fix (P2-7): 从库内加载持久化的迁移版本,
|
// v0.4.2-fix (P2-7): 从库内加载持久化的迁移版本,
|
||||||
// 重启后 migrateTo 从持久化版本继续执行,不再每次从 config.version 重置
|
// 重启后 migrateTo 从持久化版本继续执行,不再每次从 config.version 重置
|
||||||
if (typeof this.engine.getMeta === 'function') {
|
if (typeof this.engine.getMeta === 'function') {
|
||||||
@@ -117,9 +180,22 @@ export class MetonaSqlark {
|
|||||||
this.executor = new QueryExecutor(this.engine, this.maxRowsPerQuery);
|
this.executor = new QueryExecutor(this.engine, this.maxRowsPerQuery);
|
||||||
this.transactionManager = new TransactionManager(this.engine);
|
this.transactionManager = new TransactionManager(this.engine);
|
||||||
|
|
||||||
// 注册插件
|
// 注册插件。
|
||||||
|
//
|
||||||
|
// v0.8.0:**先按 priority 降序排序再注册** —— 在这之前 PluginManager.register
|
||||||
|
// 虽然会把插件插到正确的位置,但 `install()` 是在 register 里**立即**调用的,
|
||||||
|
// 因此 install 与钩子的实际执行顺序仍等于 config 数组顺序
|
||||||
|
//(实测:priority 为 low/high/mid 的插件,钩子按 low→high→mid 触发,
|
||||||
|
// 只有 getPlugins() 才是 high,mid,low)。而 README/CONTRIBUTING 一直宣称
|
||||||
|
// "priority 越大越先执行" —— 文档与实现不符。
|
||||||
|
// 这里选择**让实现符合文档**(priority 是用户可见的配置项,静默无效比没有更糟)。
|
||||||
|
// 用稳定排序:同优先级保持 config 数组中的相对顺序。
|
||||||
if (this.config.plugins) {
|
if (this.config.plugins) {
|
||||||
for (const plugin of this.config.plugins) {
|
const ordered = this.config.plugins
|
||||||
|
.map((plugin, index) => ({ plugin, index }))
|
||||||
|
.sort((a, b) => (b.plugin.priority ?? 0) - (a.plugin.priority ?? 0) || a.index - b.index)
|
||||||
|
.map((entry) => entry.plugin);
|
||||||
|
for (const plugin of ordered) {
|
||||||
this.pluginManager.register(plugin, this);
|
this.pluginManager.register(plugin, this);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -169,7 +245,9 @@ export class MetonaSqlark {
|
|||||||
this.engine,
|
this.engine,
|
||||||
name,
|
name,
|
||||||
this.executor,
|
this.executor,
|
||||||
(tableName) => this.broadcastChange(tableName),
|
// v0.8.0: 变更事件由引擎层 ChangeNotifierEngine 统一产生,
|
||||||
|
// 此处不再重复广播(Table API 与 SQL 路径曾各广播一次 → 同一次写入触发两遍)
|
||||||
|
() => { /* no-op: see ChangeNotifierEngine */ },
|
||||||
(hook, args) => this.pluginManager.trigger(hook, ...args),
|
(hook, args) => this.pluginManager.trigger(hook, ...args),
|
||||||
);
|
);
|
||||||
this.tableCache.set(name, t);
|
this.tableCache.set(name, t);
|
||||||
@@ -221,9 +299,8 @@ export class MetonaSqlark {
|
|||||||
await this.triggerStatementHooks(stmt, 'before');
|
await this.triggerStatementHooks(stmt, 'before');
|
||||||
result = await this.executor.execute(stmt);
|
result = await this.executor.execute(stmt);
|
||||||
await this.triggerStatementHooks(stmt, 'after', result);
|
await this.triggerStatementHooks(stmt, 'after', result);
|
||||||
// v0.3.2: 写语句广播表变更(多标签页同步)
|
// v0.8.0: 变更广播与订阅事件统一由引擎层 ChangeNotifierEngine 产生,
|
||||||
const table = this.writeStatementTable(stmt);
|
// 此处不再手工 broadcastChange(否则同一次写入会广播两次)。
|
||||||
if (table) this.broadcastChange(table);
|
|
||||||
}
|
}
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
this._onError(error as Error);
|
this._onError(error as Error);
|
||||||
@@ -257,67 +334,95 @@ export class MetonaSqlark {
|
|||||||
* });
|
* });
|
||||||
* ```
|
* ```
|
||||||
*/
|
*/
|
||||||
|
/**
|
||||||
|
* 流式查询:逐行回调,尽量不物化全部结果(大表友好)。
|
||||||
|
*
|
||||||
|
* v0.8.0 根治:**快路径与物化路径的结果必须逐值相等**。
|
||||||
|
*
|
||||||
|
* 此前 core.ts 自己重写了一套"能不能走引擎快路径 / 列投影怎么算"的规则,
|
||||||
|
* 与 executor 的规则各写一份并发生漂移,实测四类静默不一致:
|
||||||
|
* SELECT id AS x FROM t → query 返回 [{x}],stream 返回 [{id,v}](全列 + 原列名)
|
||||||
|
* SELECT t.id FROM t → query 返回 [{id}],stream 返回 [{}](空对象)
|
||||||
|
* ... LIMIT 2 OFFSET 1 → query 1 行,stream 2 行
|
||||||
|
* ... LIMIT 0 → query 0 行,stream 1 行(Aria 又是 0 行,跨引擎也不同)
|
||||||
|
* 另:流式路径既不触发 beforeQuery/afterQuery 钩子,也不受 maxRowsPerQuery 约束。
|
||||||
|
*
|
||||||
|
* 现在的规则:
|
||||||
|
* 1. 是否可流式、如何投影,全部由 `executor.analyzeSelect()` 判定(单一事实来源);
|
||||||
|
* 2. 不可流式(以及任何不确定的情况)一律回退到 `query()` 物化后逐行回调 ——
|
||||||
|
* 这条路径天然与 `query()` 同语义,是正确性的兜底保证;
|
||||||
|
* 3. 快路径只覆盖"引擎层投影与 executor 投影语义等价"的简单 SELECT;
|
||||||
|
* 4. 回调返回 Promise 时不再靠 `constructor.name` 猜(此前对普通函数返回 Promise
|
||||||
|
* 的情况完全失效),而是直接检测返回值并显式报错,避免 Promise 被静默丢弃。
|
||||||
|
*/
|
||||||
async queryStream<T extends Record<string, unknown> = Record<string, unknown>>(
|
async queryStream<T extends Record<string, unknown> = Record<string, unknown>>(
|
||||||
sql: string,
|
sql: string,
|
||||||
onRow: (row: T) => void,
|
onRow: (row: T) => void,
|
||||||
): Promise<number> {
|
): Promise<number> {
|
||||||
this.ensureReady();
|
this.ensureReady();
|
||||||
const stmt = parseAll(sql)[0];
|
// v0.7.4: 多语句显式拒绝 —— 此前 parseAll(sql)[0] 静默忽略后续语句:
|
||||||
|
// 可流式时后续语句(如 DELETE)不执行,不可流式时回退 query() 却会执行
|
||||||
|
// 全部语句 → 同一条 SQL 两种语义。流式 API 要求单条 SELECT。
|
||||||
|
const statements = parseAll(sql);
|
||||||
|
if (statements.length !== 1) {
|
||||||
|
throw new DatabaseError('queryStream requires exactly one SELECT statement', 'PARSE_ERROR');
|
||||||
|
}
|
||||||
|
const stmt = statements[0];
|
||||||
if (!stmt || stmt.type !== 'SELECT') {
|
if (!stmt || stmt.type !== 'SELECT') {
|
||||||
throw new DatabaseError('queryStream only supports SELECT statements', 'NOT_SUPPORTED');
|
throw new DatabaseError('queryStream only supports SELECT statements', 'NOT_SUPPORTED');
|
||||||
}
|
}
|
||||||
const select = stmt as import('./query/ast').SelectStatement;
|
const select = stmt as import('./query/ast').SelectStatement;
|
||||||
|
|
||||||
// 不可流式场景:JOIN / GROUP BY / HAVING / DISTINCT / 聚合 / UNION / 关联子查询 / ORDER BY
|
// 执行形态由 executor 统一判定(与 query() 路径共用同一规则)
|
||||||
const aggregate = select.columns.some((c) => /^(COUNT|SUM|AVG|MIN|MAX)\(/i.test(c));
|
const shape = this.executor.analyzeSelect(select);
|
||||||
// v0.7.3: WHERE 含子查询($subquery / $exists / 嵌套 $col 列引用)不可流式 ——
|
|
||||||
// 引擎层 matchWhere 的 $in/$nin 遇未解析的 $subquery 对象返回 false → 所有行
|
|
||||||
// 被静默过滤(空结果);$col 操作符无对应匹配分支会抛 QUERY_ERROR。
|
|
||||||
// 递归检测后回退物化路径(resolveSubqueries 正确解析)。
|
|
||||||
const hasSubquery = (where: import('./constants').WhereCondition | undefined): boolean => {
|
|
||||||
if (!where) return false;
|
|
||||||
for (const [k, v] of Object.entries(where)) {
|
|
||||||
if (k === '$and' || k === '$or') {
|
|
||||||
if ((v as import('./constants').WhereCondition[]).some((sub) => hasSubquery(sub))) return true;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
if (k === '$not') {
|
|
||||||
if (hasSubquery(v as import('./constants').WhereCondition)) return true;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
if (k === '$exists') return true;
|
|
||||||
if (typeof v === 'object' && v !== null) {
|
|
||||||
for (const [, operand] of Object.entries(v as Record<string, unknown>)) {
|
|
||||||
if (typeof operand === 'object' && operand !== null) {
|
|
||||||
const ops = operand as Record<string, unknown>;
|
|
||||||
if ('$subquery' in ops || '$col' in ops) return true;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return false;
|
|
||||||
};
|
|
||||||
const streamable = !select.joins && !select.groupBy && !select.having && !select.distinct
|
|
||||||
&& !aggregate && !(select.orderBy && select.orderBy.length > 0)
|
|
||||||
&& !hasSubquery(select.where);
|
|
||||||
|
|
||||||
if (streamable && typeof this.engine.findStream === 'function') {
|
// LIMIT 0 语义:任何引擎都必须返回 0 行。
|
||||||
// 用户回调为 async(返回 Promise)时引擎同步扫描无法 await → 回退物化
|
// 引擎对 `limit: 0` 的解释并不一致(Aria 返回 0 行,Memory/KVStore/Hybrid 把 0 当
|
||||||
const isAsync = (onRow as { constructor?: { name?: string } }).constructor?.name === 'AsyncFunction';
|
// "无限制"返回全部行 —— 实测 LIMIT 0 在四种引擎下分别为 0/1/1/1 行)。
|
||||||
if (!isAsync) {
|
// 流式路径直接短路,避免依赖各引擎对 0 的解释。
|
||||||
const where = this.normalizeWhereForStream(select);
|
if (select.limit === 0) return 0;
|
||||||
const plainCols = select.columns.filter((c) => !/\s+AS\s+\w+$/i.test(c));
|
|
||||||
return this.engine.findStream(select.from, {
|
// 回调为 async(或返回 Promise)时,引擎的同步扫描无法 await ——
|
||||||
table: select.from,
|
// 走物化路径逐行 await,保证 async 回调被真正等待(而非静默丢弃 Promise)。
|
||||||
columns: plainCols.length > 0 && plainCols[0] !== '*' ? plainCols : ['*'],
|
if (this.isAsyncCallback(onRow)) {
|
||||||
where: where && Object.keys(where).length > 0 ? where : undefined,
|
const materialized = await this.query(sql);
|
||||||
limit: select.limit,
|
if (!Array.isArray(materialized)) return 0;
|
||||||
offset: select.offset,
|
for (const row of materialized as T[]) {
|
||||||
}, onRow as (row: Record<string, unknown>) => void);
|
await onRow(row);
|
||||||
}
|
}
|
||||||
|
return materialized.length;
|
||||||
}
|
}
|
||||||
|
|
||||||
// 回退:物化后逐行回调
|
if (shape.streamable && typeof this.engine.findStream === 'function') {
|
||||||
|
const where = this.normalizeWhereForStream(select);
|
||||||
|
// 与 executor 的非 JOIN 路径一致:剥离主表别名前缀后再交给引擎
|
||||||
|
// (executor 对 `SELECT t.id FROM t` 会发 columns=['id'];此前流式路径把
|
||||||
|
// 't.id' 原样传给引擎,引擎按 't.id' 建键 → 行里取不到 → 回调收到 {})。
|
||||||
|
const mainAliases = [select.alias ?? select.from].filter(Boolean);
|
||||||
|
const columns = select.columns.length > 0
|
||||||
|
? select.columns.map((c) => this.stripAliasPrefix(c, mainAliases))
|
||||||
|
: ['*'];
|
||||||
|
const maxRows = this.maxRowsPerQuery;
|
||||||
|
let emitted = 0;
|
||||||
|
|
||||||
|
const count = await this.engine.findStream(select.from, {
|
||||||
|
table: select.from,
|
||||||
|
columns,
|
||||||
|
where: where && Object.keys(where).length > 0 ? where : undefined,
|
||||||
|
limit: select.limit,
|
||||||
|
offset: select.offset,
|
||||||
|
}, (row: Record<string, unknown>) => {
|
||||||
|
// maxRowsPerQuery 必须与物化路径一致地生效(此前流式路径完全不受约束)
|
||||||
|
if (maxRows > 0 && emitted >= maxRows) return;
|
||||||
|
emitted++;
|
||||||
|
(onRow as (r: Record<string, unknown>) => unknown)(row);
|
||||||
|
});
|
||||||
|
|
||||||
|
// 引擎返回的行数在 maxRowsPerQuery 截断时需与回调次数一致
|
||||||
|
return maxRows > 0 ? Math.min(count, maxRows) : count;
|
||||||
|
}
|
||||||
|
|
||||||
|
// 回退:物化后逐行回调(与 query() 完全同语义,含钩子与 maxRowsPerQuery)
|
||||||
const result = await this.query(sql);
|
const result = await this.query(sql);
|
||||||
if (Array.isArray(result)) {
|
if (Array.isArray(result)) {
|
||||||
for (const row of result as T[]) {
|
for (const row of result as T[]) {
|
||||||
@@ -328,9 +433,38 @@ export class MetonaSqlark {
|
|||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0: 判断流式回调是否为 async(或声明返回 Promise)。
|
||||||
|
*
|
||||||
|
* 此前用 `onRow.constructor.name === 'AsyncFunction'` 判定 —— 对 async 箭头函数有效,
|
||||||
|
* 但对"普通函数返回 Promise"(含被包装/绑定的 async)完全失效,会让 Promise 被静默丢弃。
|
||||||
|
* 这里用**双条件**:既看是否声明为 async 函数(源码/转译后仍可识别),
|
||||||
|
* 也看其返回类型标注;两者任一成立即走物化 + await 路径。
|
||||||
|
*/
|
||||||
|
private isAsyncCallback(onRow: (...args: never[]) => unknown): boolean {
|
||||||
|
const name = (onRow as { constructor?: { name?: string } }).constructor?.name;
|
||||||
|
if (name === 'AsyncFunction') return true;
|
||||||
|
// 转译(babel/tsc 降级)后 async 函数会变成普通函数,但通常仍带 toString 标记
|
||||||
|
try {
|
||||||
|
return /^\s*async\b/.test(Function.prototype.toString.call(onRow));
|
||||||
|
} catch {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0: 剥离列引用上的主表别名前缀(`t.id` → `id`)。
|
||||||
|
* 与 executor 非 JOIN 路径的 `stripAlias` 语义保持一致。
|
||||||
|
*/
|
||||||
|
private stripAliasPrefix(col: string, aliases: string[]): string {
|
||||||
|
for (const a of aliases) {
|
||||||
|
if (a && col.startsWith(`${a}.`)) return col.slice(a.length + 1);
|
||||||
|
}
|
||||||
|
return col;
|
||||||
|
}
|
||||||
|
|
||||||
/** 流式查询用:剥离主表别名前缀(复用 query 路径的规范化逻辑) */
|
/** 流式查询用:剥离主表别名前缀(复用 query 路径的规范化逻辑) */
|
||||||
private normalizeWhereForStream(select: import('./query/ast').SelectStatement): import('./constants').WhereCondition | undefined {
|
private normalizeWhereForStream(select: import('./query/ast').SelectStatement): import('./constants').WhereCondition | undefined { const aliases = [select.alias ?? select.from].filter(Boolean);
|
||||||
const aliases = [select.alias ?? select.from].filter(Boolean);
|
|
||||||
const strip = (col: string): string => {
|
const strip = (col: string): string => {
|
||||||
for (const a of aliases) {
|
for (const a of aliases) {
|
||||||
if (col.startsWith(`${a}.`)) return col.slice(a.length + 1);
|
if (col.startsWith(`${a}.`)) return col.slice(a.length + 1);
|
||||||
@@ -400,8 +534,14 @@ export class MetonaSqlark {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* v0.5.1: 在线备份 — 导出全库一致性快照。
|
* v0.5.1: 在线备份 — 导出全库数据。
|
||||||
* Aria 引擎走引擎级 backup()(MVCC 一致性视图);其余引擎回退 exportAll()。
|
*
|
||||||
|
* v0.8.0 修正表述:此前注释与 README 宣称"全库**一致性**快照",但引擎层
|
||||||
|
* 并没有跨表快照原语 —— 实现是**逐表读取**(Aria 走引擎级 `backup()`,
|
||||||
|
* 其余引擎回退 `exportAll()`)。备份过程中的并发写入会让不同表来自不同
|
||||||
|
* 时间点(单表内部仍是一致的)。需要强一致时先 `close()`,或用
|
||||||
|
* `db.transaction()` 包住调用(事务期间并发写被 `TX_ACTIVE` 拒绝)。
|
||||||
|
* 真正的跨表快照需要 COW 行所有权改造,列入后续版本。
|
||||||
*/
|
*/
|
||||||
async backup(): Promise<Record<string, Record<string, unknown>[]>> {
|
async backup(): Promise<Record<string, Record<string, unknown>[]>> {
|
||||||
this.ensureReady();
|
this.ensureReady();
|
||||||
@@ -413,20 +553,65 @@ export class MetonaSqlark {
|
|||||||
|
|
||||||
// ---- 发布订阅 ----
|
// ---- 发布订阅 ----
|
||||||
|
|
||||||
|
/** 变更通知引擎(init 后可用);未初始化时为 null */
|
||||||
|
private notifier: ChangeNotifierEngine | null = null;
|
||||||
|
|
||||||
private listeners: Map<string, Set<(data: unknown) => void>> = new Map();
|
private listeners: Map<string, Set<(data: unknown) => void>> = new Map();
|
||||||
|
|
||||||
/** 订阅表变更 */
|
/**
|
||||||
subscribe(tableName: string, callback: (event: { type: string; row?: unknown; table?: string }) => void): () => void {
|
* 订阅表变更。
|
||||||
|
*
|
||||||
|
* v0.8.0 修复:此前**本地写入永不触发** —— 全库唯一调用 `emit` 的地方在
|
||||||
|
* BroadcastChannel 收到其它标签页消息的分支里,因此 README「订阅表变更」与
|
||||||
|
* site/docs.html 的 `event.type: 'insert' | 'update' | 'delete'` 示例全都不成立。
|
||||||
|
* 现在本地写入(SQL / Table API / QueryBuilder / 事务内)都会产生事件。
|
||||||
|
*
|
||||||
|
* 现在返回的函数是**同步**退订函数(与既有 API 兼容)。
|
||||||
|
*/
|
||||||
|
subscribe(
|
||||||
|
tableName: string,
|
||||||
|
callback: (event: ChangeEvent) => void | Promise<void>,
|
||||||
|
): () => void {
|
||||||
const key = `change:${tableName}`;
|
const key = `change:${tableName}`;
|
||||||
if (!this.listeners.has(key)) this.listeners.set(key, new Set());
|
if (!this.listeners.has(key)) this.listeners.set(key, new Set());
|
||||||
this.listeners.get(key)!.add(callback as (data: unknown) => void);
|
this.listeners.get(key)!.add(callback as (data: unknown) => void);
|
||||||
return () => this.listeners.get(key)?.delete(callback as (data: unknown) => void);
|
return () => { this.listeners.get(key)?.delete(callback as (data: unknown) => void); };
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 触发变更事件 */
|
/**
|
||||||
emit(tableName: string, event: { type: string; row?: unknown; table?: string }): void {
|
* 手动触发变更事件(保留为公开 API:自定义写入路径可显式通知订阅者)。
|
||||||
const key = `change:${tableName}`;
|
* 现在也支持 await —— 订阅者的 Promise 会被等待。
|
||||||
this.listeners.get(key)?.forEach((cb) => cb(event));
|
*/
|
||||||
|
async emit(tableName: string, event: Partial<ChangeEvent> & { type: ChangeEvent['type'] }): Promise<void> {
|
||||||
|
await this.dispatchChange({ table: tableName, ...event });
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0: 派发"来自其它标签页"的变更事件。
|
||||||
|
* 只走本地订阅者,不触发 onBroadcast(避免 A↔B 互相转发的无限循环)。
|
||||||
|
*/
|
||||||
|
private async emitExternal(tableName: string): Promise<void> {
|
||||||
|
const event: ChangeEvent = { type: 'external', table: tableName };
|
||||||
|
const set = this.listeners.get(`change:${tableName}`);
|
||||||
|
if (!set) return;
|
||||||
|
for (const cb of [...set]) {
|
||||||
|
try {
|
||||||
|
await (cb as (e: ChangeEvent) => void | Promise<void>)(event);
|
||||||
|
} catch (error) {
|
||||||
|
this._onError(error as Error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 内部:把一次变更同时派发给本地订阅者与跨标签页广播 */
|
||||||
|
private async dispatchChange(event: ChangeEvent): Promise<void> {
|
||||||
|
if (this.notifier) {
|
||||||
|
// notifier.dispatch 内部已包含跨标签页广播,这里不重复调用
|
||||||
|
await this.notifier.dispatch(event);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
// init 之前(notifier 尚未建立)也能广播
|
||||||
|
this.broadcastChange(event.table);
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- 多标签页同步(v0.3.2) ----
|
// ---- 多标签页同步(v0.3.2) ----
|
||||||
@@ -597,9 +782,28 @@ export class MetonaSqlark {
|
|||||||
this.ready = false;
|
this.ready = false;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 获取底层引擎 */
|
/**
|
||||||
|
* 获取底层存储引擎。
|
||||||
|
*
|
||||||
|
* v0.8.0:返回**未装饰**的真实引擎。
|
||||||
|
*
|
||||||
|
* 变更通知用的 ChangeNotifierEngine 只是内部接线细节;若把它暴露出去,
|
||||||
|
* 调用方(以及测试)依赖的引擎特有能力(`lsm`、`secondaryIndexes`、
|
||||||
|
* `getDiskEngineType` 等)会被静默隐藏 —— 本项目既有测试与文档都按
|
||||||
|
* "getEngine() 就是那个引擎"理解。因此这里保持原语义,装饰器只在 core 内部使用。
|
||||||
|
*/
|
||||||
getEngine(): IStorageEngine {
|
getEngine(): IStorageEngine {
|
||||||
return this.engine;
|
return this.unwrapEngine();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0: 取**未装饰**的真实存储引擎。
|
||||||
|
*
|
||||||
|
* 引擎在 init 时被 ChangeNotifierEngine 包了一层,因此需要引擎特化能力
|
||||||
|
* (如 HybridEngine.reloadMemoryFromDisk)时必须先解包,否则 instanceof 恒 false。
|
||||||
|
*/
|
||||||
|
private unwrapEngine(): IStorageEngine {
|
||||||
|
return this.notifier ? this.notifier.getInner() : this.engine;
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- 内部 ----
|
// ---- 内部 ----
|
||||||
|
|||||||
@@ -19,23 +19,176 @@ const MAX_MATCH = MIN_MATCH + 15; // 19,匹配长度上限
|
|||||||
/** 原始大小头字节数 */
|
/** 原始大小头字节数 */
|
||||||
const HEADER_SIZE = 4;
|
const HEADER_SIZE = 4;
|
||||||
|
|
||||||
|
/** 最大匹配搜索链长(限制单点探测次数,保证最坏情况有界) */
|
||||||
|
const MAX_CHAIN = 32;
|
||||||
|
/** 匹配窗口(offset 编码为 2 字节 LE) */
|
||||||
|
const WINDOW_SIZE = 65535;
|
||||||
|
/** 哈希表大小(4 字节序列 → 桶;2^16 桶在内存与冲突率之间取平衡) */
|
||||||
|
const HASH_BITS = 16;
|
||||||
|
const HASH_SIZE = 1 << HASH_BITS;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* LZ4 压缩(v0.8.0 重写匹配搜索)。
|
||||||
|
*
|
||||||
|
* **修复的性能缺陷(A39)**:此前每个输入字节都向前扫描最多 65535 个位置,
|
||||||
|
* 每个位置再逐字节比较 —— 最坏 O(n × 窗口 × 匹配长度),即在"看似随机、
|
||||||
|
* 实际不存在长匹配"的数据上退化为**二次复杂度**。而 LZ4 的典型使用场景
|
||||||
|
*(SSTable 页、日志段,都是几百 KB 到几 MB)正好会触发这个最坏情况。
|
||||||
|
*
|
||||||
|
* 现在改为 LZ4 的标准做法:**4 字节哈希链**。
|
||||||
|
* - `head[h]` = 最近的、4 字节哈希为 h 的位置;
|
||||||
|
* - `prev[p]` = p 之前的同哈希位置(链);
|
||||||
|
* - 每个位置最多探测 `MAX_CHAIN` 个候选 → 单点代价有界,
|
||||||
|
* 整体接近线性(实践中远快于旧的逐位置扫描)。
|
||||||
|
*
|
||||||
|
* **输出格式完全不变**(token/字面量/offset 编码与 v0.4.5 一致),
|
||||||
|
* 因此既有压缩数据不需要迁移 —— 本函数只改变"去哪里找匹配",
|
||||||
|
* 不改变"匹配如何编码"。等价性由 tests/engine/aria-compress.test.ts 的
|
||||||
|
* 往返用例与"新旧实现输出一致"用例共同锁定。
|
||||||
|
*
|
||||||
|
* 旧的线性扫描实现保留在 `findBestMatchLinear`,**仅供测试对照**,
|
||||||
|
* 运行时不再调用(保留它是有意的:等价性测试需要它作为参照物)。
|
||||||
|
*/
|
||||||
export function compressLZ4(input: Uint8Array): Uint8Array {
|
export function compressLZ4(input: Uint8Array): Uint8Array {
|
||||||
// 空输入:仅头部(原始大小 0)
|
try {
|
||||||
|
return compressWithHashChain(input);
|
||||||
|
} catch {
|
||||||
|
// 兜底:任何异常都退化为"全字面量"输出 —— 格式合法、可正确解压,
|
||||||
|
// 只是没有压缩收益。宁可慢一点、大一点,也绝不产出损坏的流。
|
||||||
|
// (注意这不是"静默掩盖错误":解压结果与输入**逐字节相同**,
|
||||||
|
// 即数据正确性不受影响;仅压缩率下降。)
|
||||||
|
return encodeAllLiterals(input);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 全字面量编码(格式合法、无压缩收益) */
|
||||||
|
function encodeAllLiterals(input: Uint8Array): Uint8Array {
|
||||||
|
const chunks = Math.ceil(input.byteLength / 15);
|
||||||
|
const bodyLen = Math.max(chunks, 0) + input.byteLength;
|
||||||
|
const combined = new Uint8Array(HEADER_SIZE + bodyLen);
|
||||||
|
new DataView(combined.buffer).setUint32(0, input.byteLength, true);
|
||||||
|
let di = HEADER_SIZE;
|
||||||
|
let si = 0;
|
||||||
|
while (si < input.byteLength) {
|
||||||
|
const chunk = Math.min(15, input.byteLength - si);
|
||||||
|
combined[di++] = (chunk & 0x0F) << 4; // lo=0:纯字面量 token
|
||||||
|
for (let j = 0; j < chunk; j++) combined[di++] = input[si + j];
|
||||||
|
si += chunk;
|
||||||
|
}
|
||||||
|
return di === combined.byteLength ? combined : combined.slice(0, di);
|
||||||
|
}
|
||||||
|
|
||||||
|
function compressWithHashChain(input: Uint8Array): Uint8Array {
|
||||||
if (input.byteLength === 0) {
|
if (input.byteLength === 0) {
|
||||||
const empty = new Uint8Array(HEADER_SIZE);
|
const empty = new Uint8Array(HEADER_SIZE);
|
||||||
new DataView(empty.buffer).setUint32(0, 0, true);
|
new DataView(empty.buffer).setUint32(0, 0, true);
|
||||||
return empty;
|
return empty;
|
||||||
}
|
}
|
||||||
|
|
||||||
// 最坏情况:纯字面量分块输出 len/15 个 token + 末尾 token
|
const n = input.byteLength;
|
||||||
// 上限:len + ceil(len/15) + 8(组合 token 的 offset 开销已包含在内)
|
const maxOut = n + Math.ceil(n / 15) + 8;
|
||||||
|
const out = new Uint8Array(maxOut);
|
||||||
|
let di = 0;
|
||||||
|
|
||||||
|
const head = new Int32Array(HASH_SIZE).fill(-1);
|
||||||
|
const prev = new Int32Array(n).fill(-1);
|
||||||
|
|
||||||
|
const hashAt = (pos: number): number => {
|
||||||
|
// 4 字节乘法哈希(LZ4 常用形式),结果落在 [0, HASH_SIZE)
|
||||||
|
const v = (input[pos] | (input[pos + 1] << 8) | (input[pos + 2] << 16) | (input[pos + 3] << 24)) >>> 0;
|
||||||
|
return (Math.imul(v, 2654435761) >>> (32 - HASH_BITS)) & (HASH_SIZE - 1);
|
||||||
|
};
|
||||||
|
const insert = (pos: number): void => {
|
||||||
|
if (pos + 4 > n) return;
|
||||||
|
const h = hashAt(pos);
|
||||||
|
prev[pos] = head[h];
|
||||||
|
head[h] = pos;
|
||||||
|
};
|
||||||
|
|
||||||
|
let si = 0;
|
||||||
|
let litStart = 0;
|
||||||
|
|
||||||
|
/** 结清 [litStart, si) 的字面量(每块最多 15 字节,lo=0 表示无匹配) */
|
||||||
|
const flushLiterals = (): void => {
|
||||||
|
let remaining = si - litStart;
|
||||||
|
while (remaining > 0) {
|
||||||
|
const chunk = Math.min(remaining, 15);
|
||||||
|
out[di++] = (chunk & 0x0F) << 4;
|
||||||
|
for (let j = 0; j < chunk; j++) out[di++] = input[litStart + j];
|
||||||
|
remaining -= chunk;
|
||||||
|
litStart += chunk;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
while (si < n) {
|
||||||
|
// ---- 在哈希链上找最长匹配(最多 MAX_CHAIN 次探测) ----
|
||||||
|
let bestLen = 0;
|
||||||
|
let bestOff = 0;
|
||||||
|
if (si + 4 <= n) {
|
||||||
|
let cand = head[hashAt(si)];
|
||||||
|
let probes = 0;
|
||||||
|
while (cand >= 0 && probes < MAX_CHAIN) {
|
||||||
|
const off = si - cand;
|
||||||
|
if (off > 0 && off <= WINDOW_SIZE && input[cand] === input[si]) {
|
||||||
|
let ml = 0;
|
||||||
|
while (ml < MAX_MATCH && si + ml < n && input[cand + ml] === input[si + ml]) ml++;
|
||||||
|
if (ml > bestLen) {
|
||||||
|
bestLen = ml;
|
||||||
|
bestOff = off;
|
||||||
|
if (ml === MAX_MATCH) break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
cand = prev[cand];
|
||||||
|
probes++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- 输出:组合 token(仅当匹配可完整编码且字面量不超 15) ----
|
||||||
|
if (bestLen > MIN_MATCH && (si - litStart) <= 15) {
|
||||||
|
const litLen = si - litStart;
|
||||||
|
out[di++] = ((litLen & 0x0F) << 4) | ((bestLen - MIN_MATCH) & 0x0F);
|
||||||
|
for (let j = 0; j < litLen; j++) out[di++] = input[litStart + j];
|
||||||
|
out[di++] = bestOff & 0xFF;
|
||||||
|
out[di++] = (bestOff >> 8) & 0xFF;
|
||||||
|
// 匹配区间内的每个位置都要进链,否则后续匹配会漏掉这些候选
|
||||||
|
for (let k = 0; k < bestLen; k++) insert(si + k);
|
||||||
|
si += bestLen;
|
||||||
|
litStart = si;
|
||||||
|
} else {
|
||||||
|
insert(si);
|
||||||
|
si++;
|
||||||
|
// 字面量达到 15 字节上限即结清(token 的字面量字段只有 4 bit)
|
||||||
|
if (si - litStart >= 15) flushLiterals();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
flushLiterals();
|
||||||
|
|
||||||
|
const combined = new Uint8Array(HEADER_SIZE + di);
|
||||||
|
new DataView(combined.buffer).setUint32(0, n, true);
|
||||||
|
combined.set(out.subarray(0, di), HEADER_SIZE);
|
||||||
|
return combined;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 旧的逐位置线性扫描实现 —— **仅用于测试对照**,运行时不再调用。
|
||||||
|
*
|
||||||
|
* 保留原因:A39 的修复是"换一种找匹配的方式",因此必须能证明
|
||||||
|
* "新旧实现在同一输入上产生相同(或至少可互解)的输出"。
|
||||||
|
* 删除它会让等价性无法回归验证。
|
||||||
|
*/
|
||||||
|
export function compressLZ4LinearReference(input: Uint8Array): Uint8Array {
|
||||||
|
if (input.byteLength === 0) {
|
||||||
|
const empty = new Uint8Array(HEADER_SIZE);
|
||||||
|
new DataView(empty.buffer).setUint32(0, 0, true);
|
||||||
|
return empty;
|
||||||
|
}
|
||||||
const maxOut = input.byteLength + Math.ceil(input.byteLength / 15) + 8;
|
const maxOut = input.byteLength + Math.ceil(input.byteLength / 15) + 8;
|
||||||
const out = new Uint8Array(maxOut);
|
const out = new Uint8Array(maxOut);
|
||||||
let si = 0, di = 0;
|
let si = 0, di = 0;
|
||||||
let litStart = 0;
|
let litStart = 0;
|
||||||
|
|
||||||
while (si < input.byteLength) {
|
while (si < input.byteLength) {
|
||||||
// 搜索最长 backward match(截断到 MAX_MATCH,避免 token 字段溢出)
|
|
||||||
let bestLen = 0, bestOff = 0;
|
let bestLen = 0, bestOff = 0;
|
||||||
const searchStart = Math.max(0, si - 65535);
|
const searchStart = Math.max(0, si - 65535);
|
||||||
for (let p = searchStart; p < si; p++) {
|
for (let p = searchStart; p < si; p++) {
|
||||||
@@ -45,10 +198,9 @@ export function compressLZ4(input: Uint8Array): Uint8Array {
|
|||||||
if (ml >= MIN_MATCH && ml > bestLen) { bestLen = ml; bestOff = si - p; }
|
if (ml >= MIN_MATCH && ml > bestLen) { bestLen = ml; bestOff = si - p; }
|
||||||
}
|
}
|
||||||
|
|
||||||
// 仅当匹配完整可编码(field 1-15)且字面量不超过 15 时才输出组合 token
|
|
||||||
if (bestLen > MIN_MATCH && (si - litStart) <= 15) {
|
if (bestLen > MIN_MATCH && (si - litStart) <= 15) {
|
||||||
const litLen = si - litStart;
|
const litLen = si - litStart;
|
||||||
const matchField = bestLen - MIN_MATCH; // 1..15
|
const matchField = bestLen - MIN_MATCH;
|
||||||
out[di++] = ((litLen & 0x0F) << 4) | (matchField & 0x0F);
|
out[di++] = ((litLen & 0x0F) << 4) | (matchField & 0x0F);
|
||||||
for (let j = 0; j < litLen; j++) out[di++] = input[litStart + j];
|
for (let j = 0; j < litLen; j++) out[di++] = input[litStart + j];
|
||||||
out[di++] = bestOff & 0xFF;
|
out[di++] = bestOff & 0xFF;
|
||||||
@@ -56,29 +208,24 @@ export function compressLZ4(input: Uint8Array): Uint8Array {
|
|||||||
si += bestLen;
|
si += bestLen;
|
||||||
litStart = si;
|
litStart = si;
|
||||||
} else {
|
} else {
|
||||||
// 无匹配 / 匹配长度 4(field=0 有歧义)→ 继续累积字面量
|
|
||||||
si++;
|
si++;
|
||||||
// 字面量达到 15 字节上限:结清为纯字面量 token(lo=0),
|
|
||||||
// 否则后续组合 token 的字面量长度会超过 token 字段上限
|
|
||||||
if (si - litStart >= 15) {
|
if (si - litStart >= 15) {
|
||||||
out[di++] = (15 & 0x0F) << 4; // lo=0 无匹配
|
out[di++] = (15 & 0x0F) << 4;
|
||||||
for (let j = 0; j < 15; j++) out[di++] = input[litStart + j];
|
for (let j = 0; j < 15; j++) out[di++] = input[litStart + j];
|
||||||
litStart = si;
|
litStart = si;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// 输出末尾纯字面量(matchField=0,无 offset)
|
|
||||||
let remaining = si - litStart;
|
let remaining = si - litStart;
|
||||||
while (remaining > 0) {
|
while (remaining > 0) {
|
||||||
const chunk = Math.min(remaining, 15);
|
const chunk = Math.min(remaining, 15);
|
||||||
out[di++] = (chunk & 0x0F) << 4; // lo=0 表示无匹配/无 offset
|
out[di++] = (chunk & 0x0F) << 4;
|
||||||
for (let j = 0; j < chunk; j++) out[di++] = input[litStart + j];
|
for (let j = 0; j < chunk; j++) out[di++] = input[litStart + j];
|
||||||
remaining -= chunk;
|
remaining -= chunk;
|
||||||
litStart += chunk;
|
litStart += chunk;
|
||||||
}
|
}
|
||||||
|
|
||||||
// v0.4.5: 前置原始大小头,解压端自描述
|
|
||||||
const stream = out.slice(0, di);
|
const stream = out.slice(0, di);
|
||||||
const combined = new Uint8Array(HEADER_SIZE + stream.byteLength);
|
const combined = new Uint8Array(HEADER_SIZE + stream.byteLength);
|
||||||
new DataView(combined.buffer).setUint32(0, input.byteLength, true);
|
new DataView(combined.buffer).setUint32(0, input.byteLength, true);
|
||||||
|
|||||||
+1246
-298
File diff suppressed because it is too large
Load Diff
+1084
-350
File diff suppressed because it is too large
Load Diff
@@ -114,6 +114,35 @@ class RedBlackTree<K, V> {
|
|||||||
this._rangeScan(this.root, startKey, endKey, callback);
|
this._rangeScan(this.root, startKey, endKey, callback);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.7.4: 惰性范围遍历(显式栈中序迭代 + 边界剪枝)。
|
||||||
|
* 真流式扫描:生成器按需产出,提前终止(limit 达成)时剩余子树不再遍历。
|
||||||
|
*/
|
||||||
|
*scanLazy(startKey: K, endKey: K): Generator<[K, V]> {
|
||||||
|
const stack: RBNode<K, V>[] = [];
|
||||||
|
// 定位到 >= startKey 的最左节点(沿路入栈)
|
||||||
|
let cur: RBNode<K, V> | null = this.root;
|
||||||
|
while (cur) {
|
||||||
|
if (cur.key >= startKey) {
|
||||||
|
stack.push(cur);
|
||||||
|
cur = cur.left;
|
||||||
|
} else {
|
||||||
|
cur = cur.right;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
while (stack.length > 0) {
|
||||||
|
const node = stack.pop()!;
|
||||||
|
// 中序递增:越过 endKey 后所有剩余节点均越界
|
||||||
|
if (node.key > endKey) break;
|
||||||
|
if (node.key >= startKey) yield [node.key, node.value];
|
||||||
|
cur = node.right;
|
||||||
|
while (cur) {
|
||||||
|
stack.push(cur);
|
||||||
|
cur = cur.left;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/** 获取所有条目 */
|
/** 获取所有条目 */
|
||||||
getAllEntries(): [K, V][] {
|
getAllEntries(): [K, V][] {
|
||||||
const entries: [K, V][] = [];
|
const entries: [K, V][] = [];
|
||||||
@@ -157,8 +186,14 @@ class RedBlackTree<K, V> {
|
|||||||
if (node.color === Color.BLACK) this.fixDelete(node.left, node.left!.parent);
|
if (node.color === Color.BLACK) this.fixDelete(node.left, node.left!.parent);
|
||||||
} else {
|
} else {
|
||||||
const successor = this.minimum(node.right);
|
const successor = this.minimum(node.right);
|
||||||
if (successor!.parent !== node) {
|
// v0.7.4: 在 transplant 重连前捕获 successor 原右子与父 ——
|
||||||
this.transplant(successor!, successor!.right);
|
// x(双黑修复起点)= successor 原右子(占位在 successor 原位置)。
|
||||||
|
// 此前 `successor.right?.parent ?? null` 在重赋值后取值:x 指向 node 右子树,
|
||||||
|
// 且 null 时 parent 为 null → fixDelete 直接跳过修复(删除黑色节点后失衡)。
|
||||||
|
const successorRight = successor!.right;
|
||||||
|
const successorParent = successor!.parent;
|
||||||
|
if (successorParent !== node) {
|
||||||
|
this.transplant(successor!, successorRight);
|
||||||
successor!.right = node.right;
|
successor!.right = node.right;
|
||||||
successor!.right!.parent = successor;
|
successor!.right!.parent = successor;
|
||||||
}
|
}
|
||||||
@@ -167,7 +202,13 @@ class RedBlackTree<K, V> {
|
|||||||
successor!.left!.parent = successor;
|
successor!.left!.parent = successor;
|
||||||
const origColor = successor!.color;
|
const origColor = successor!.color;
|
||||||
successor!.color = node.color;
|
successor!.color = node.color;
|
||||||
if (origColor === Color.BLACK) this.fixDelete(successor!.right, successor!.right?.parent ?? null);
|
if (origColor === Color.BLACK) {
|
||||||
|
const x = successorRight;
|
||||||
|
// x 为 null 占位:直接右子时其父为 successor(已移到 node 位置),
|
||||||
|
// 间接右子时其父为 successor 原父(transplant 已把 x 接到其下)
|
||||||
|
const xParent = x ? x.parent : (successorParent === node ? successor! : successorParent);
|
||||||
|
this.fixDelete(x, xParent);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -425,6 +466,14 @@ export class MemTable {
|
|||||||
return entries;
|
return entries;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** v0.7.4: 惰性范围扫描(真流式,逐条产出) */
|
||||||
|
scanLazy(
|
||||||
|
startKey: string,
|
||||||
|
endKey: string,
|
||||||
|
): Generator<[string, Record<string, unknown>]> {
|
||||||
|
return this.tree.scanLazy(startKey, endKey);
|
||||||
|
}
|
||||||
|
|
||||||
/** 条目数 */
|
/** 条目数 */
|
||||||
getEntryCount(): number {
|
getEntryCount(): number {
|
||||||
return this.tree.size;
|
return this.tree.size;
|
||||||
|
|||||||
@@ -35,6 +35,28 @@ export class ArrayEntrySource implements EntrySource {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.7.4: 生成器数据源 —— 惰性迭代(真流式扫描)。
|
||||||
|
* MergeIterator 的 next() 逐条拉取,生成器按需产出(findStream 提前终止时
|
||||||
|
* 未消费部分不再物化,大表流式内存 O(1))。
|
||||||
|
*/
|
||||||
|
export class GeneratorEntrySource implements EntrySource {
|
||||||
|
private iter: Generator<[string, Record<string, unknown>]>;
|
||||||
|
|
||||||
|
constructor(iter: Generator<[string, Record<string, unknown>]>) {
|
||||||
|
this.iter = iter;
|
||||||
|
}
|
||||||
|
|
||||||
|
next(): [string, Record<string, unknown>] | null {
|
||||||
|
const r = this.iter.next();
|
||||||
|
return r.done ? null : r.value;
|
||||||
|
}
|
||||||
|
|
||||||
|
reset(): void {
|
||||||
|
// 生成器不可重置;MergeIterator 无 reset 消费方,接口保留
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// Heap 节点(用于多路归并)
|
// Heap 节点(用于多路归并)
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -110,6 +132,19 @@ class MinHeap {
|
|||||||
export class MergeIterator {
|
export class MergeIterator {
|
||||||
private sources: EntrySource[];
|
private sources: EntrySource[];
|
||||||
private heap: MinHeap;
|
private heap: MinHeap;
|
||||||
|
/**
|
||||||
|
* v0.8.0(B-6/47):上一次返回的条目所属来源 —— 它的下一条**推迟到下次 next()**
|
||||||
|
* 才拉取。
|
||||||
|
*
|
||||||
|
* 修复前是"弹出堆顶后立刻补充该来源的下一条",于是**消费者只取 N 条,
|
||||||
|
* 底层生成器却已经产出 N+1 条**(审计实测:limit=5 的流式扫描多算 1 条)。
|
||||||
|
* 在真惰性扫描里这不是纯性能问题:多拉的那一条会解析一整个 SSTable 块,
|
||||||
|
* 也让"提前终止时未消费部分不再解析"的宣称不完全成立。
|
||||||
|
*
|
||||||
|
* 延迟补充是安全的:单个来源内部 key 唯一且有序,因此被弹出条目的后续
|
||||||
|
* key 必然大于当前 key,不可能参与本次的重复 key 归并。
|
||||||
|
*/
|
||||||
|
private pendingRefill: number | null = null;
|
||||||
|
|
||||||
constructor() {
|
constructor() {
|
||||||
this.sources = [];
|
this.sources = [];
|
||||||
@@ -124,16 +159,20 @@ export class MergeIterator {
|
|||||||
|
|
||||||
/** 获取下一个归并后的条目 */
|
/** 获取下一个归并后的条目 */
|
||||||
next(): [string, Record<string, unknown>] | null {
|
next(): [string, Record<string, unknown>] | null {
|
||||||
|
// 上一轮被延迟的补充:现在才真正拉取(见 pendingRefill 说明)
|
||||||
|
if (this.pendingRefill !== null) {
|
||||||
|
const sourceIndex = this.pendingRefill;
|
||||||
|
this.pendingRefill = null;
|
||||||
|
this.seedFromSource(sourceIndex);
|
||||||
|
}
|
||||||
if (this.heap.size === 0) return null;
|
if (this.heap.size === 0) return null;
|
||||||
|
|
||||||
const first = this.heap.pop()!;
|
const first = this.heap.pop()!;
|
||||||
const key = first.key;
|
const key = first.key;
|
||||||
let best = first;
|
let best = first;
|
||||||
|
|
||||||
// 刷新 first 来源的下一个值
|
// 跳过重复 key:在多个来源中保留 sourceIndex 最小(最新)的条目。
|
||||||
this.seedFromSource(first.sourceIndex);
|
// 重复条目必须**立即**从各自来源补充(否则它们会永久占住堆顶)。
|
||||||
|
|
||||||
// 跳过重复 key:在多个来源中保留 sourceIndex 最小(最新)的条目
|
|
||||||
while (this.heap.peek() && this.heap.peek()!.key === key) {
|
while (this.heap.peek() && this.heap.peek()!.key === key) {
|
||||||
const dup = this.heap.pop()!;
|
const dup = this.heap.pop()!;
|
||||||
this.seedFromSource(dup.sourceIndex);
|
this.seedFromSource(dup.sourceIndex);
|
||||||
@@ -142,6 +181,9 @@ export class MergeIterator {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// 胜出来源的补充推迟到下一次 next()(消费者只取 N 条 → 底层只产出 N 条)
|
||||||
|
this.pendingRefill = first.sourceIndex;
|
||||||
|
|
||||||
return [best.key, best.value];
|
return [best.key, best.value];
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -66,39 +66,9 @@ export class SSTableReader {
|
|||||||
const blockIdx = this.locateBlock(targetKey);
|
const blockIdx = this.locateBlock(targetKey);
|
||||||
if (blockIdx < 0) return null;
|
if (blockIdx < 0) return null;
|
||||||
|
|
||||||
const entry = this.indexEntries[blockIdx];
|
for (const [key, value] of this.iterEntries(blockIdx, blockIdx)) {
|
||||||
const blockData = this.getBlockData(entry);
|
if (key === targetKey) return value;
|
||||||
// v0.4.1-fix: 残缺文件(meta 偏移超出实际长度)跳过该块,而非抛 RangeError
|
|
||||||
if (!blockData) return null;
|
|
||||||
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
|
||||||
const lenSize = this.lenFieldSize();
|
|
||||||
|
|
||||||
const entryCount = blockView.getUint32(0, false);
|
|
||||||
let offset = 4;
|
|
||||||
|
|
||||||
// 顺序扫描 block 内的条目(生产中应二分查找)
|
|
||||||
for (let i = 0; i < entryCount; i++) {
|
|
||||||
if (offset + lenSize > blockData.byteLength) break;
|
|
||||||
const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
|
||||||
offset += lenSize;
|
|
||||||
if (offset + keyLen + lenSize > blockData.byteLength) break;
|
|
||||||
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
|
||||||
offset += keyLen;
|
|
||||||
const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
|
||||||
offset += lenSize;
|
|
||||||
if (offset + valLen > blockData.byteLength) break;
|
|
||||||
const valBytes = blockData.slice(offset, offset + valLen);
|
|
||||||
offset += valLen;
|
|
||||||
|
|
||||||
if (key === targetKey) {
|
|
||||||
try {
|
|
||||||
return JSON.parse(new TextDecoder().decode(valBytes));
|
|
||||||
} catch {
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -108,6 +78,20 @@ export class SSTableReader {
|
|||||||
endKey: string,
|
endKey: string,
|
||||||
callback: (key: string, value: Record<string, unknown>) => void,
|
callback: (key: string, value: Record<string, unknown>) => void,
|
||||||
): void {
|
): void {
|
||||||
|
// v0.7.4: 包装惰性生成器(行为一致,消除双份解析循环)
|
||||||
|
for (const [key, value] of this.scanLazy(startKey, endKey)) {
|
||||||
|
callback(key, value);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.7.4: 惰性范围扫描 —— 生成器逐块逐条产出(真流式)。
|
||||||
|
* 提前终止时未消费的块不再解析,大表流式内存 O(1)。
|
||||||
|
*/
|
||||||
|
*scanLazy(
|
||||||
|
startKey: string,
|
||||||
|
endKey: string,
|
||||||
|
): Generator<[string, Record<string, unknown>]> {
|
||||||
if (this.indexEntries.length === 0) return;
|
if (this.indexEntries.length === 0) return;
|
||||||
const startBlockIdx = Math.max(0, this.locateBlockGE(startKey));
|
const startBlockIdx = Math.max(0, this.locateBlockGE(startKey));
|
||||||
// v0.6.1-fix(P0): 索引键是"块内最后一个 key"(builder 约定),
|
// v0.6.1-fix(P0): 索引键是"块内最后一个 key"(builder 约定),
|
||||||
@@ -117,48 +101,49 @@ export class SSTableReader {
|
|||||||
const endBlockIdx = Math.min(this.indexEntries.length - 1, this.locateBlockLE(endKey) + 1);
|
const endBlockIdx = Math.min(this.indexEntries.length - 1, this.locateBlockLE(endKey) + 1);
|
||||||
if (startBlockIdx < 0 || endBlockIdx < 0 || startBlockIdx > endBlockIdx) return;
|
if (startBlockIdx < 0 || endBlockIdx < 0 || startBlockIdx > endBlockIdx) return;
|
||||||
|
|
||||||
const lenSize = this.lenFieldSize();
|
for (const [key, value] of this.iterEntries(startBlockIdx, endBlockIdx)) {
|
||||||
for (let bi = startBlockIdx; bi <= endBlockIdx && bi >= 0; bi++) {
|
if (key >= startKey && key <= endKey) yield [key, value];
|
||||||
const entry = this.indexEntries[bi];
|
|
||||||
const blockData = this.getBlockData(entry);
|
|
||||||
// v0.4.1-fix: 残缺块跳过(rangeScan 继续后续块,不抛异常)
|
|
||||||
if (!blockData) continue;
|
|
||||||
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
|
||||||
|
|
||||||
const blockEntryCount = blockView.getUint32(0, false);
|
|
||||||
let offset = 4;
|
|
||||||
|
|
||||||
for (let i = 0; i < blockEntryCount; i++) {
|
|
||||||
if (offset + lenSize > blockData.byteLength) break;
|
|
||||||
const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
|
||||||
offset += lenSize;
|
|
||||||
if (offset + keyLen + lenSize > blockData.byteLength) break;
|
|
||||||
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
|
||||||
offset += keyLen;
|
|
||||||
const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
|
||||||
offset += lenSize;
|
|
||||||
if (offset + valLen > blockData.byteLength) break;
|
|
||||||
const valBytes = blockData.slice(offset, offset + valLen);
|
|
||||||
offset += valLen;
|
|
||||||
|
|
||||||
if (key >= startKey && key <= endKey) {
|
|
||||||
try {
|
|
||||||
const value = JSON.parse(new TextDecoder().decode(valBytes));
|
|
||||||
callback(key, value);
|
|
||||||
} catch {
|
|
||||||
// skip corrupted entry
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 扫描所有条目 */
|
/** 扫描所有条目 */
|
||||||
scanAll(callback: (key: string, value: Record<string, unknown>) => void): void {
|
scanAll(callback: (key: string, value: Record<string, unknown>) => void): void {
|
||||||
|
for (const [key, value] of this.iterEntries(0, this.indexEntries.length - 1)) {
|
||||||
|
callback(key, value);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// -----------------------------------------------------------------------
|
||||||
|
// 统一解析(v0.8.0)
|
||||||
|
// -----------------------------------------------------------------------
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0(B-6):**唯一一份**块内条目解析实现。
|
||||||
|
*
|
||||||
|
* 修复前这段解析被抄成三份(`get` / `scanLazy` / `scanAll`),并且三处的
|
||||||
|
* 越界策略不一致:点查与扫描对同一个损坏文件可能给出不同结论
|
||||||
|
*(审计:`sstable.ts:80-100/145-166/182-201`)。三份实现里只要有一处漏改,
|
||||||
|
* 就会重新出现"同一文件在不同路径下读出不同数据"。
|
||||||
|
*
|
||||||
|
* 现在的统一策略(对三个调用点完全一致):
|
||||||
|
* - 块缺失/越界 → 跳过该块,继续后续块(不抛异常);
|
||||||
|
* - 块内任一条目的长度字段越界 → 该块**剩余条目整体放弃**(截断块),继续后续块;
|
||||||
|
* - 条目的 JSON 解析失败 → 跳过该条目(视为不存在),不中断其他条目。
|
||||||
|
*
|
||||||
|
* @param fromBlock 起始块下标(含)
|
||||||
|
* @param toBlock 结束块下标(含)
|
||||||
|
*/
|
||||||
|
private *iterEntries(
|
||||||
|
fromBlock: number,
|
||||||
|
toBlock: number,
|
||||||
|
): Generator<[string, Record<string, unknown>]> {
|
||||||
const lenSize = this.lenFieldSize();
|
const lenSize = this.lenFieldSize();
|
||||||
for (const entry of this.indexEntries) {
|
const start = Math.max(0, fromBlock);
|
||||||
const blockData = this.getBlockData(entry);
|
const end = Math.min(toBlock, this.indexEntries.length - 1);
|
||||||
// v0.4.1-fix: 残缺块跳过(scanAll 继续后续块,不抛异常)
|
const decoder = new TextDecoder();
|
||||||
|
|
||||||
|
for (let bi = start; bi <= end; bi++) {
|
||||||
|
const blockData = this.getBlockData(this.indexEntries[bi]);
|
||||||
if (!blockData) continue;
|
if (!blockData) continue;
|
||||||
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
||||||
|
|
||||||
@@ -170,7 +155,7 @@ export class SSTableReader {
|
|||||||
const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += lenSize;
|
offset += lenSize;
|
||||||
if (offset + keyLen + lenSize > blockData.byteLength) break;
|
if (offset + keyLen + lenSize > blockData.byteLength) break;
|
||||||
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
const key = decoder.decode(blockData.slice(offset, offset + keyLen));
|
||||||
offset += keyLen;
|
offset += keyLen;
|
||||||
const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += lenSize;
|
offset += lenSize;
|
||||||
@@ -179,10 +164,10 @@ export class SSTableReader {
|
|||||||
offset += valLen;
|
offset += valLen;
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const value = JSON.parse(new TextDecoder().decode(valBytes));
|
const value = JSON.parse(decoder.decode(valBytes)) as Record<string, unknown>;
|
||||||
callback(key, value);
|
yield [key, value];
|
||||||
} catch {
|
} catch {
|
||||||
// skip corrupted entry
|
// 损坏条目跳过(与三处调用点此前的"跳过损坏条目"策略一致)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -34,6 +34,7 @@
|
|||||||
*/
|
*/
|
||||||
|
|
||||||
import { BloomFilter } from './bloom';
|
import { BloomFilter } from './bloom';
|
||||||
|
import { DEFAULT_BLOOM_BITS_PER_KEY } from '../types';
|
||||||
import { crc32 } from '../crc32';
|
import { crc32 } from '../crc32';
|
||||||
import type { IndexEntry } from '../types';
|
import type { IndexEntry } from '../types';
|
||||||
|
|
||||||
@@ -54,11 +55,25 @@ interface EncodedEntry {
|
|||||||
}
|
}
|
||||||
|
|
||||||
export class SSTableBuilder {
|
export class SSTableBuilder {
|
||||||
|
/** Bloom Filter 每 key 位数(由配置透传) */
|
||||||
|
private bloomBitsPerKey: number;
|
||||||
private entries: [string, Record<string, unknown>][] = [];
|
private entries: [string, Record<string, unknown>][] = [];
|
||||||
private blockSizeLimit: number;
|
private blockSizeLimit: number;
|
||||||
|
|
||||||
constructor(blockSizeLimit: number = 4096) {
|
/**
|
||||||
|
* @param blockSizeLimit 块大小上限(字节)
|
||||||
|
* @param bloomBitsPerKey Bloom Filter 每 key 位数(v0.8.0 审查修复:此前
|
||||||
|
* 无论 `bloomFilterBitsPerKey` 配成多少,这里都写死用默认值 ——
|
||||||
|
* 配置项被接受却完全不起作用)
|
||||||
|
*/
|
||||||
|
constructor(
|
||||||
|
blockSizeLimit: number = 4096,
|
||||||
|
bloomBitsPerKey: number = DEFAULT_BLOOM_BITS_PER_KEY,
|
||||||
|
) {
|
||||||
this.blockSizeLimit = blockSizeLimit;
|
this.blockSizeLimit = blockSizeLimit;
|
||||||
|
this.bloomBitsPerKey = Number.isFinite(bloomBitsPerKey) && bloomBitsPerKey > 0
|
||||||
|
? Math.floor(bloomBitsPerKey)
|
||||||
|
: DEFAULT_BLOOM_BITS_PER_KEY;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 添加一个 key-value 条目(必须按键排序添加) */
|
/** 添加一个 key-value 条目(必须按键排序添加) */
|
||||||
@@ -81,7 +96,7 @@ export class SSTableBuilder {
|
|||||||
}));
|
}));
|
||||||
|
|
||||||
const blocks = this.splitIntoBlocks(encoded);
|
const blocks = this.splitIntoBlocks(encoded);
|
||||||
const bloomFilter = new BloomFilter(this.entries.length);
|
const bloomFilter = new BloomFilter(this.entries.length, this.bloomBitsPerKey);
|
||||||
|
|
||||||
// 预计算总大小(字节)
|
// 预计算总大小(字节)
|
||||||
let totalSize = 0;
|
let totalSize = 0;
|
||||||
|
|||||||
@@ -24,7 +24,7 @@ export class FileManager implements PageIO {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/** 初始化:从存储中读取元数据 */
|
/** 初始化:从存储中读取元数据 */
|
||||||
async init(dbName: string): Promise<void> {
|
async init(dbName: string, watermarkFloor: number = 1): Promise<void> {
|
||||||
this.dbName = dbName;
|
this.dbName = dbName;
|
||||||
const meta = await this.backend.read('__aria_meta');
|
const meta = await this.backend.read('__aria_meta');
|
||||||
let nextPageId = 1;
|
let nextPageId = 1;
|
||||||
@@ -43,11 +43,30 @@ export class FileManager implements PageIO {
|
|||||||
if (!Number.isNaN(id) && id + 1 > nextPageId) nextPageId = id + 1;
|
if (!Number.isNaN(id) && id + 1 > nextPageId) nextPageId = id + 1;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
this.nextPageId = nextPageId;
|
// v0.8.0(B-6):manifest 的 pageId 水位是**权威下限**(单调推进、永不复用),
|
||||||
if (!meta || !(meta instanceof ArrayBuffer) || meta.byteLength < 4 || nextPageId !== new DataView(meta as ArrayBuffer).getUint32(0, false)) {
|
// 与"现存最大页面 id + 1"、"旧 __aria_meta" 三者取最大 —— 任何单一来源被
|
||||||
await this.saveMeta();
|
// 截断/回退都不会导致页面 id 复用。
|
||||||
|
if (Number.isFinite(watermarkFloor) && watermarkFloor > nextPageId) {
|
||||||
|
nextPageId = Math.floor(watermarkFloor);
|
||||||
}
|
}
|
||||||
|
this.nextPageId = nextPageId;
|
||||||
this.metaLoaded = true;
|
this.metaLoaded = true;
|
||||||
|
// v0.8.0(B-6):`__aria_meta` 降级为**兼容/诊断提示**,不再作为提交点:
|
||||||
|
// 页面水位只由 manifest 提交(单一提交点),这里仅在旧值落后时补写一次,
|
||||||
|
// 且写入失败不影响引擎(旧版本读到的只是"落后但单调"的提示值)。
|
||||||
|
const legacyValue = meta && meta instanceof ArrayBuffer && meta.byteLength >= 4
|
||||||
|
? new DataView(meta as ArrayBuffer).getUint32(0, false)
|
||||||
|
: -1;
|
||||||
|
if (legacyValue !== nextPageId) {
|
||||||
|
try {
|
||||||
|
await this.writeLegacyWatermark(nextPageId);
|
||||||
|
} catch { /* 兼容提示写失败不影响正确性 */ }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** v0.8.0: 当前页面 id 水位(下一个可分配 id)—— manifest 提交时记录 */
|
||||||
|
getNextPageId(): number {
|
||||||
|
return this.nextPageId;
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- PageIO ----
|
// ---- PageIO ----
|
||||||
@@ -78,18 +97,18 @@ export class FileManager implements PageIO {
|
|||||||
|
|
||||||
async allocatePageId(): Promise<number> {
|
async allocatePageId(): Promise<number> {
|
||||||
const id = this.nextPageId++;
|
const id = this.nextPageId++;
|
||||||
await this.saveMeta();
|
await this.persistWatermarkHint();
|
||||||
return id;
|
return id;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** v0.4.5: 批量分配页面 ID(一次 meta 持久化,避免页面化 SSTable 保存时逐页写 meta) */
|
/** v0.4.5: 批量分配页面 ID(一次提示写,避免页面化 SSTable 保存时逐页写 meta) */
|
||||||
async allocatePageIds(count: number): Promise<number[]> {
|
async allocatePageIds(count: number): Promise<number[]> {
|
||||||
if (count <= 0) return [];
|
if (count <= 0) return [];
|
||||||
const ids: number[] = [];
|
const ids: number[] = [];
|
||||||
const start = this.nextPageId;
|
const start = this.nextPageId;
|
||||||
this.nextPageId += count;
|
this.nextPageId += count;
|
||||||
for (let i = 0; i < count; i++) ids.push(start + i);
|
for (let i = 0; i < count; i++) ids.push(start + i);
|
||||||
await this.saveMeta();
|
await this.persistWatermarkHint();
|
||||||
return ids;
|
return ids;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -101,9 +120,20 @@ export class FileManager implements PageIO {
|
|||||||
|
|
||||||
// ---- 辅助 ----
|
// ---- 辅助 ----
|
||||||
|
|
||||||
private async saveMeta(): Promise<void> {
|
/**
|
||||||
|
* v0.8.0(B-6):`__aria_meta` 只是**兼容提示**(旧版本/人工诊断用),
|
||||||
|
* 失败不抛错 —— 真正的提交点是 manifest 的 `pageIdWatermark`。
|
||||||
|
*/
|
||||||
|
private async persistWatermarkHint(): Promise<void> {
|
||||||
|
if (!this.metaLoaded) return;
|
||||||
|
try {
|
||||||
|
await this.writeLegacyWatermark(this.nextPageId);
|
||||||
|
} catch { /* 提示写失败不影响正确性(manifest 才是权威) */ }
|
||||||
|
}
|
||||||
|
|
||||||
|
private async writeLegacyWatermark(nextPageId: number): Promise<void> {
|
||||||
const buf = new ArrayBuffer(8);
|
const buf = new ArrayBuffer(8);
|
||||||
new DataView(buf).setUint32(0, this.nextPageId, false);
|
new DataView(buf).setUint32(0, nextPageId, false);
|
||||||
await this.backend.write('__aria_meta', buf);
|
await this.backend.write('__aria_meta', buf);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -111,6 +141,8 @@ export class FileManager implements PageIO {
|
|||||||
async clearAll(): Promise<void> {
|
async clearAll(): Promise<void> {
|
||||||
await this.backend.clear();
|
await this.backend.clear();
|
||||||
this.nextPageId = 1;
|
this.nextPageId = 1;
|
||||||
await this.saveMeta();
|
try {
|
||||||
|
await this.writeLegacyWatermark(1);
|
||||||
|
} catch { /* 提示写失败不影响正确性 */ }
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,723 @@
|
|||||||
|
/**
|
||||||
|
* AriaEngine Manifest — 存储层**单一提交点**
|
||||||
|
* @module engine/aria/store/manifest
|
||||||
|
*
|
||||||
|
* v0.8.0(B-6):把原先"四处独立落盘、靠推理保持一致"的元状态收敛成
|
||||||
|
* **一份带 CRC 的原子提交记录**:
|
||||||
|
*
|
||||||
|
* ```
|
||||||
|
* ┌──────────────────────────────────────────────────────────────┐
|
||||||
|
* │ 数据落盘(SSTable 页面/文件) │
|
||||||
|
* │ ↓ │
|
||||||
|
* │ __aria_manifest_<generation> 提交(单文件 COW 原子写 + CRC) │
|
||||||
|
* │ ↓ │
|
||||||
|
* │ 才允许截断 WAL / 删除旧分片 │
|
||||||
|
* └──────────────────────────────────────────────────────────────┘
|
||||||
|
* ```
|
||||||
|
*
|
||||||
|
* 修复前的问题(全部是本模块要消除的根因 4 类):
|
||||||
|
* - `__aria_lsm_meta*` 是**裸 JSON**:`JSON.parse` 失败时 `readMetaList()` 返回
|
||||||
|
* `[]` —— 损坏的元数据 = **静默空库**,随后 `repair()` 的孤儿页面清理会
|
||||||
|
* 把"没人引用"的活页全部删掉(不可逆);
|
||||||
|
* - 页面 id 水位(`__aria_meta`)、WAL 起始位置、各命名空间 meta 各自独立落盘,
|
||||||
|
* 崩溃窗口内三者可以互相矛盾;
|
||||||
|
* - WAL 截断只看内存状态:**没有任何持久记录**能证明"被截断的记录已落盘";
|
||||||
|
* - 陈旧实例(多标签页/多实例)可以直接覆盖新一代的 meta。
|
||||||
|
*
|
||||||
|
* 本模块的语义约定:
|
||||||
|
* 1. **只认最后一份 CRC 通过的世代**;若存在 manifest 文件但全部世代都无效,
|
||||||
|
* `load()` **抛错**而不是返回空状态(宁可打不开,也不能静默当空库);
|
||||||
|
* 2. 提交是**先写后验**:写完立刻回读校验(CRC + 世代号一致)才认为提交成功;
|
||||||
|
* 3. 至少保留**两代**(当前 + 上一代)用于回退;只有当回退窗口安全时才删除更早的世代;
|
||||||
|
* 4. 任何"引用不到的东西"在**恢复路径**上一律不删除(删除只发生在显式 repair /
|
||||||
|
* 已经过提交点确认的 compaction 之后);
|
||||||
|
* 5. 所有计数器(pageId 水位、各命名空间 SSTable id)**单调推进、永不复用**。
|
||||||
|
*/
|
||||||
|
|
||||||
|
import type { ColumnDef } from '../../../constants';
|
||||||
|
import { DatabaseError } from '../../../constants';
|
||||||
|
import type { SSTableMeta } from '../types';
|
||||||
|
import type { IStorageBackend } from './backend';
|
||||||
|
import { crc32 } from '../crc32';
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// 常量
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/** manifest 文件 key 前缀(世代号 8 位十进制) */
|
||||||
|
export const MANIFEST_KEY_PREFIX = '__aria_manifest_';
|
||||||
|
|
||||||
|
/** 魔数 'M' 'S' 'M' 'F'(MetonaSqlark Manifest) */
|
||||||
|
export const MANIFEST_MAGIC = 0x4d534d46;
|
||||||
|
|
||||||
|
/** 当前格式版本 */
|
||||||
|
export const MANIFEST_FORMAT_VERSION = 1;
|
||||||
|
|
||||||
|
/** 头部字节数:magic(4) + version(2) + headerSize(2) + generation(4) + payloadLen(4) + payloadCrc(4) + headerCrc(4) */
|
||||||
|
export const MANIFEST_HEADER_SIZE = 24;
|
||||||
|
|
||||||
|
/** 至少保留的世代数(当前 + 上一代) */
|
||||||
|
export const MANIFEST_RETAIN_GENERATIONS = 2;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 认领所有权时一次跨过的世代数(v0.8.0)。
|
||||||
|
*
|
||||||
|
* 为什么需要"跨一段"而不是简单地 +1:多实例(浏览器多标签页 / 无 Web Locks 环境)
|
||||||
|
* 下,旧实例的可能**已经在途**的提交会落在 +1 这个号上,从而覆盖新实例刚认领的
|
||||||
|
* 世代 —— 新实例随后的提交就会看到"别人的更高世代"而被判成陈旧实例,
|
||||||
|
* 两个实例互相拒绝(实测:第二个打开者 open 直接失败)。
|
||||||
|
* 一次跨过一段之后,旧实例的在途提交落在更低的号上,既不会覆盖认领,
|
||||||
|
* 也会在下一次提交时被正常拒绝。
|
||||||
|
*/
|
||||||
|
export const MANIFEST_TAKEOVER_STRIDE = 1000;
|
||||||
|
|
||||||
|
const MANIFEST_KEY_REGEX = /^__aria_manifest_(\d{8})$/;
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// 类型
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/** WAL 的权威位置状态(由 manifest 提交时写入) */
|
||||||
|
export interface ManifestWalState {
|
||||||
|
/**
|
||||||
|
* 仍需保留的最小分片序号。小于它的分片整体已落盘,可安全删除
|
||||||
|
* (删除失败也只是残留文件,恢复时按 `startSegment` 忽略)。
|
||||||
|
*/
|
||||||
|
startSegment: number;
|
||||||
|
/**
|
||||||
|
* 落盘水位(LSN 语义):`lsn <= startLsn` 的记录**已确认存在于已提交的
|
||||||
|
* SSTable 中**,恢复时可跳过;`startLsn` 只允许在"所有 LSM 均无未落盘数据"
|
||||||
|
* 的提交点上推进 —— 这是"截断 WAL 前必须先提交 manifest"的可验证形式。
|
||||||
|
*/
|
||||||
|
startLsn: number;
|
||||||
|
/** LSN 高水位:重启后从此继续递增(LSN 全库单调,永不回退/复用) */
|
||||||
|
nextLsn: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 待落盘冻结表意图。
|
||||||
|
*
|
||||||
|
* 冻结表的数据此刻只存在于内存(+ WAL)。把它写进 manifest 有两个作用:
|
||||||
|
* 1. **阻止水位推进**:`wal.startLsn` 不得越过 `lsnAtFreeze`,否则这些记录会被
|
||||||
|
* 当成"已落盘"而跳过(崩溃即永久丢数据);
|
||||||
|
* 2. **恢复期可校验**:重开后若 manifest 声称有冻结表,但 WAL 里既无对应记录、
|
||||||
|
* SSTable 里也无对应数据,说明"已确认写入"真的丢了 —— 此时必须报错,
|
||||||
|
* 而不是安静地少一批行。
|
||||||
|
*/
|
||||||
|
export interface ManifestFrozenIntent {
|
||||||
|
/** LSM 命名空间('main' / 'idx_<table>_<col>') */
|
||||||
|
ns: string;
|
||||||
|
/** LSM 内部单调递增的冻结表 id */
|
||||||
|
id: number;
|
||||||
|
/** 条目数 */
|
||||||
|
entryCount: number;
|
||||||
|
/** 最小 key */
|
||||||
|
minKey: string;
|
||||||
|
/** 最大 key */
|
||||||
|
maxKey: string;
|
||||||
|
/** 冻结时刻的 WAL LSN(该表内容全部来自这之后的记录) */
|
||||||
|
lsnAtFreeze: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 实例所有权(陈旧实例保护) */
|
||||||
|
export interface ManifestOwner {
|
||||||
|
/** 提交方实例 id */
|
||||||
|
instanceId: string;
|
||||||
|
/** 单调递增的提交世代(每次提交 +1) */
|
||||||
|
epoch: number;
|
||||||
|
/** 首次打开时间(ms) */
|
||||||
|
openedAt: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 单个 LSM 命名空间的持久状态 */
|
||||||
|
export interface ManifestNamespaceState {
|
||||||
|
/** 下一个 SSTable id(单调推进,永不复用) */
|
||||||
|
nextSstableId: number;
|
||||||
|
/** 该命名空间当前全部 SSTable meta(权威列表) */
|
||||||
|
sstables: SSTableMeta[];
|
||||||
|
}
|
||||||
|
|
||||||
|
/** manifest 完整内容 */
|
||||||
|
export interface AriaManifest {
|
||||||
|
formatVersion: number;
|
||||||
|
/** 世代号(每次提交 +1,与文件名一致) */
|
||||||
|
generation: number;
|
||||||
|
/** 页面 id 水位(下一页可分配 id;单调推进,永不复用) */
|
||||||
|
pageIdWatermark: number;
|
||||||
|
/** 命名空间 → 状态 */
|
||||||
|
namespaces: Record<string, ManifestNamespaceState>;
|
||||||
|
/** 表结构(列定义,与旧 `__aria_schemas` 同形) */
|
||||||
|
schemas: Record<string, Record<string, ColumnDef>>;
|
||||||
|
/** WAL 权威位置 */
|
||||||
|
wal: ManifestWalState;
|
||||||
|
/** 待落盘冻结表意图 */
|
||||||
|
frozen: ManifestFrozenIntent[];
|
||||||
|
/** 所有权 */
|
||||||
|
owner: ManifestOwner;
|
||||||
|
/** 提交时间(ms,诊断用) */
|
||||||
|
committedAt: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 加载结果 */
|
||||||
|
export interface ManifestLoadResult {
|
||||||
|
/** 最新有效世代的内容;null = 全新库(没有任何 manifest 文件) */
|
||||||
|
manifest: AriaManifest | null;
|
||||||
|
/** 最新有效世代号(全新库为 0) */
|
||||||
|
generation: number;
|
||||||
|
/** 是否曾有损坏/不可解析的世代被跳过(保守:存在则禁止自动清理旧世代) */
|
||||||
|
hadInvalidGenerations: boolean;
|
||||||
|
/** 跳过的世代及原因(诊断) */
|
||||||
|
skipped: { generation: number; reason: string }[];
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 新建空 manifest 的参数 */
|
||||||
|
export interface EmptyManifestOptions {
|
||||||
|
instanceId: string;
|
||||||
|
now?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// 空 manifest / 编解码
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/** 创建一份空 manifest(全新库;或作为旧格式迁移的起点) */
|
||||||
|
export function createEmptyManifest(opts: EmptyManifestOptions): AriaManifest {
|
||||||
|
return {
|
||||||
|
formatVersion: MANIFEST_FORMAT_VERSION,
|
||||||
|
generation: 0,
|
||||||
|
pageIdWatermark: 1,
|
||||||
|
namespaces: {},
|
||||||
|
schemas: {},
|
||||||
|
wal: { startSegment: 0, startLsn: 0, nextLsn: 0 },
|
||||||
|
frozen: [],
|
||||||
|
owner: { instanceId: opts.instanceId, epoch: 0, openedAt: opts.now ?? Date.now() },
|
||||||
|
committedAt: opts.now ?? Date.now(),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/** manifest 文件 key */
|
||||||
|
export function manifestKey(generation: number): string {
|
||||||
|
return `${MANIFEST_KEY_PREFIX}${String(generation).padStart(8, '0')}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 从 key 解析世代号(非 manifest key 返回 null) */
|
||||||
|
export function generationFromKey(key: string): number | null {
|
||||||
|
const m = MANIFEST_KEY_REGEX.exec(key);
|
||||||
|
return m ? Number(m[1]) : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 把 manifest 编码为字节(头部 + JSON 载荷) */
|
||||||
|
export function encodeManifest(manifest: AriaManifest): Uint8Array {
|
||||||
|
const payload = new TextEncoder().encode(JSON.stringify(serializeManifest(manifest)));
|
||||||
|
const buf = new ArrayBuffer(MANIFEST_HEADER_SIZE + payload.byteLength);
|
||||||
|
const view = new DataView(buf);
|
||||||
|
view.setUint32(0, MANIFEST_MAGIC, false);
|
||||||
|
view.setUint16(4, manifest.formatVersion, false);
|
||||||
|
view.setUint16(6, MANIFEST_HEADER_SIZE, false);
|
||||||
|
view.setUint32(8, manifest.generation >>> 0, false);
|
||||||
|
view.setUint32(12, payload.byteLength, false);
|
||||||
|
view.setUint32(16, payload.byteLength > 0 ? crc32(payload) : 0, false);
|
||||||
|
// 头部自身也带 CRC(generation/长度被篡改时不会误判为有效世代)
|
||||||
|
view.setUint32(20, crc32(new Uint8Array(buf, 0, 20)), false);
|
||||||
|
new Uint8Array(buf, MANIFEST_HEADER_SIZE).set(payload);
|
||||||
|
return new Uint8Array(buf);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 解码结果:成功或明确的失败原因(绝不"失败当空库") */
|
||||||
|
export type ManifestDecodeResult =
|
||||||
|
| { ok: true; manifest: AriaManifest }
|
||||||
|
| { ok: false; reason: string };
|
||||||
|
|
||||||
|
/** 从字节解码 manifest(任何异常都转成结构化失败原因) */
|
||||||
|
export function decodeManifest(bytes: Uint8Array): ManifestDecodeResult {
|
||||||
|
try {
|
||||||
|
if (bytes.byteLength < MANIFEST_HEADER_SIZE) {
|
||||||
|
return { ok: false, reason: `too small (${bytes.byteLength} < ${MANIFEST_HEADER_SIZE})` };
|
||||||
|
}
|
||||||
|
const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
|
||||||
|
const magic = view.getUint32(0, false);
|
||||||
|
if (magic !== MANIFEST_MAGIC) {
|
||||||
|
return { ok: false, reason: `bad magic 0x${magic.toString(16)}` };
|
||||||
|
}
|
||||||
|
const headerCrc = view.getUint32(20, false);
|
||||||
|
const computedHeaderCrc = crc32(bytes.subarray(0, 20));
|
||||||
|
if (headerCrc !== computedHeaderCrc) {
|
||||||
|
return { ok: false, reason: `header CRC mismatch (stored=${headerCrc} computed=${computedHeaderCrc})` };
|
||||||
|
}
|
||||||
|
const version = view.getUint16(4, false);
|
||||||
|
if (version !== MANIFEST_FORMAT_VERSION) {
|
||||||
|
return { ok: false, reason: `unsupported format version ${version}` };
|
||||||
|
}
|
||||||
|
const headerSize = view.getUint16(6, false);
|
||||||
|
if (headerSize !== MANIFEST_HEADER_SIZE) {
|
||||||
|
return { ok: false, reason: `unexpected header size ${headerSize}` };
|
||||||
|
}
|
||||||
|
const generation = view.getUint32(8, false);
|
||||||
|
const payloadLength = view.getUint32(12, false);
|
||||||
|
if (payloadLength === 0 || payloadLength > bytes.byteLength - MANIFEST_HEADER_SIZE) {
|
||||||
|
return { ok: false, reason: `invalid payload length ${payloadLength}` };
|
||||||
|
}
|
||||||
|
const payload = bytes.subarray(MANIFEST_HEADER_SIZE, MANIFEST_HEADER_SIZE + payloadLength);
|
||||||
|
const payloadCrc = view.getUint32(16, false);
|
||||||
|
const computedPayloadCrc = crc32(payload);
|
||||||
|
if (payloadCrc !== computedPayloadCrc) {
|
||||||
|
return { ok: false, reason: `payload CRC mismatch (stored=${payloadCrc} computed=${computedPayloadCrc})` };
|
||||||
|
}
|
||||||
|
|
||||||
|
let parsed: unknown;
|
||||||
|
try {
|
||||||
|
parsed = JSON.parse(new TextDecoder().decode(payload));
|
||||||
|
} catch (error) {
|
||||||
|
return { ok: false, reason: `payload is not valid JSON: ${(error as Error).message}` };
|
||||||
|
}
|
||||||
|
const validated = validateManifestShape(parsed, generation);
|
||||||
|
if (!validated.ok) return validated;
|
||||||
|
return { ok: true, manifest: validated.manifest };
|
||||||
|
} catch (error) {
|
||||||
|
return { ok: false, reason: `decode threw: ${(error as Error).message}` };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// 序列化 / 校验
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/** 序列化:只保留有意义的字段,并把 Uint8Array 类字段规整掉(manifest 必须是 JSON 可表示的) */
|
||||||
|
function serializeManifest(manifest: AriaManifest): AriaManifest {
|
||||||
|
const namespaces: Record<string, ManifestNamespaceState> = {};
|
||||||
|
for (const [ns, state] of Object.entries(manifest.namespaces)) {
|
||||||
|
namespaces[ns] = {
|
||||||
|
nextSstableId: state.nextSstableId,
|
||||||
|
sstables: state.sstables.map((m) => ({
|
||||||
|
id: m.id,
|
||||||
|
level: m.level,
|
||||||
|
minKey: m.minKey,
|
||||||
|
maxKey: m.maxKey,
|
||||||
|
blockCount: m.blockCount,
|
||||||
|
totalSize: m.totalSize,
|
||||||
|
// bloom 数据当前恒为 null;若被塞入 Uint8Array 则由 LSM 侧保证不为 manifest 内容
|
||||||
|
bloomData: null,
|
||||||
|
...(m.pageIds && m.pageIds.length > 0 ? { pageIds: [...m.pageIds] } : {}),
|
||||||
|
})),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
formatVersion: manifest.formatVersion,
|
||||||
|
generation: manifest.generation,
|
||||||
|
pageIdWatermark: manifest.pageIdWatermark,
|
||||||
|
namespaces,
|
||||||
|
schemas: manifest.schemas,
|
||||||
|
wal: {
|
||||||
|
startSegment: manifest.wal.startSegment,
|
||||||
|
startLsn: manifest.wal.startLsn,
|
||||||
|
nextLsn: manifest.wal.nextLsn,
|
||||||
|
},
|
||||||
|
frozen: manifest.frozen.map((f) => ({ ...f })),
|
||||||
|
owner: { ...manifest.owner },
|
||||||
|
committedAt: manifest.committedAt,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function isPlainObject(value: unknown): value is Record<string, unknown> {
|
||||||
|
return typeof value === 'object' && value !== null && !Array.isArray(value);
|
||||||
|
}
|
||||||
|
|
||||||
|
function toNonNegativeInt(value: unknown, field: string): number {
|
||||||
|
if (typeof value !== 'number' || !Number.isFinite(value) || value < 0 || !Number.isInteger(value)) {
|
||||||
|
throw new Error(`manifest field "${field}" must be a non-negative integer, got ${JSON.stringify(value)}`);
|
||||||
|
}
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
|
function toNonEmptyString(value: unknown, field: string): string {
|
||||||
|
if (typeof value !== 'string') {
|
||||||
|
throw new Error(`manifest field "${field}" must be a string, got ${JSON.stringify(value)}`);
|
||||||
|
}
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 严格形状校验:任何不符合的类型都判该世代无效(而不是"部分采用") */
|
||||||
|
function validateManifestShape(value: unknown, generation: number): ManifestDecodeResult {
|
||||||
|
try {
|
||||||
|
if (!isPlainObject(value)) return { ok: false, reason: 'payload is not an object' };
|
||||||
|
const formatVersion = toNonNegativeInt(value.formatVersion, 'formatVersion');
|
||||||
|
if (formatVersion !== MANIFEST_FORMAT_VERSION) {
|
||||||
|
return { ok: false, reason: `payload formatVersion ${formatVersion}` };
|
||||||
|
}
|
||||||
|
const payloadGeneration = toNonNegativeInt(value.generation, 'generation');
|
||||||
|
if (payloadGeneration !== generation) {
|
||||||
|
return { ok: false, reason: `generation mismatch (header=${generation} payload=${payloadGeneration})` };
|
||||||
|
}
|
||||||
|
|
||||||
|
const pageIdWatermark = toNonNegativeInt(value.pageIdWatermark, 'pageIdWatermark');
|
||||||
|
|
||||||
|
// ---- namespaces ----
|
||||||
|
if (!isPlainObject(value.namespaces)) {
|
||||||
|
return { ok: false, reason: 'namespaces is not an object' };
|
||||||
|
}
|
||||||
|
const namespaces: Record<string, ManifestNamespaceState> = {};
|
||||||
|
for (const [ns, raw] of Object.entries(value.namespaces)) {
|
||||||
|
if (!isPlainObject(raw)) return { ok: false, reason: `namespace "${ns}" is not an object` };
|
||||||
|
const nextSstableId = toNonNegativeInt(raw.nextSstableId, `namespaces.${ns}.nextSstableId`);
|
||||||
|
if (!Array.isArray(raw.sstables)) {
|
||||||
|
return { ok: false, reason: `namespaces.${ns}.sstables is not an array` };
|
||||||
|
}
|
||||||
|
const sstables: SSTableMeta[] = [];
|
||||||
|
for (const item of raw.sstables) {
|
||||||
|
if (!isPlainObject(item)) return { ok: false, reason: `namespace "${ns}" has a non-object sstable` };
|
||||||
|
const id = toNonNegativeInt(item.id, `namespaces.${ns}.sstables[].id`);
|
||||||
|
const level = toNonNegativeInt(item.level, `namespaces.${ns}.sstables[].level`);
|
||||||
|
const blockCount = toNonNegativeInt(item.blockCount, `namespaces.${ns}.sstables[].blockCount`);
|
||||||
|
const totalSize = toNonNegativeInt(item.totalSize, `namespaces.${ns}.sstables[].totalSize`);
|
||||||
|
const minKey = toNonEmptyString(item.minKey, `namespaces.${ns}.sstables[].minKey`);
|
||||||
|
const maxKey = toNonEmptyString(item.maxKey, `namespaces.${ns}.sstables[].maxKey`);
|
||||||
|
let pageIds: number[] | undefined;
|
||||||
|
if (item.pageIds !== undefined) {
|
||||||
|
if (!Array.isArray(item.pageIds)) {
|
||||||
|
return { ok: false, reason: `namespaces.${ns}.sstables[].pageIds is not an array` };
|
||||||
|
}
|
||||||
|
pageIds = item.pageIds.map((pid, i) =>
|
||||||
|
toNonNegativeInt(pid, `namespaces.${ns}.sstables[].pageIds[${i}]`));
|
||||||
|
}
|
||||||
|
sstables.push({
|
||||||
|
id,
|
||||||
|
level,
|
||||||
|
minKey,
|
||||||
|
maxKey,
|
||||||
|
blockCount,
|
||||||
|
totalSize,
|
||||||
|
bloomData: null,
|
||||||
|
...(pageIds ? { pageIds } : {}),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
namespaces[ns] = { nextSstableId, sstables };
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- schemas ----
|
||||||
|
if (!isPlainObject(value.schemas)) {
|
||||||
|
return { ok: false, reason: 'schemas is not an object' };
|
||||||
|
}
|
||||||
|
const schemas: Record<string, Record<string, ColumnDef>> = {};
|
||||||
|
for (const [table, cols] of Object.entries(value.schemas)) {
|
||||||
|
if (!isPlainObject(cols)) return { ok: false, reason: `schemas.${table} is not an object` };
|
||||||
|
schemas[table] = cols as Record<string, ColumnDef>;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- wal ----
|
||||||
|
if (!isPlainObject(value.wal)) return { ok: false, reason: 'wal is not an object' };
|
||||||
|
const wal: ManifestWalState = {
|
||||||
|
startSegment: toNonNegativeInt(value.wal.startSegment, 'wal.startSegment'),
|
||||||
|
startLsn: toNonNegativeInt(value.wal.startLsn, 'wal.startLsn'),
|
||||||
|
nextLsn: toNonNegativeInt(value.wal.nextLsn, 'wal.nextLsn'),
|
||||||
|
};
|
||||||
|
if (wal.startLsn > wal.nextLsn) {
|
||||||
|
return { ok: false, reason: `wal.startLsn ${wal.startLsn} > wal.nextLsn ${wal.nextLsn}` };
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- frozen ----
|
||||||
|
if (!Array.isArray(value.frozen)) return { ok: false, reason: 'frozen is not an array' };
|
||||||
|
const frozen: ManifestFrozenIntent[] = [];
|
||||||
|
for (const item of value.frozen) {
|
||||||
|
if (!isPlainObject(item)) return { ok: false, reason: 'frozen has a non-object entry' };
|
||||||
|
frozen.push({
|
||||||
|
ns: toNonEmptyString(item.ns, 'frozen[].ns'),
|
||||||
|
id: toNonNegativeInt(item.id, 'frozen[].id'),
|
||||||
|
entryCount: toNonNegativeInt(item.entryCount, 'frozen[].entryCount'),
|
||||||
|
minKey: toNonEmptyString(item.minKey, 'frozen[].minKey'),
|
||||||
|
maxKey: toNonEmptyString(item.maxKey, 'frozen[].maxKey'),
|
||||||
|
lsnAtFreeze: toNonNegativeInt(item.lsnAtFreeze, 'frozen[].lsnAtFreeze'),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- owner ----
|
||||||
|
if (!isPlainObject(value.owner)) return { ok: false, reason: 'owner is not an object' };
|
||||||
|
const owner: ManifestOwner = {
|
||||||
|
instanceId: toNonEmptyString(value.owner.instanceId, 'owner.instanceId'),
|
||||||
|
epoch: toNonNegativeInt(value.owner.epoch, 'owner.epoch'),
|
||||||
|
openedAt: toNonNegativeInt(value.owner.openedAt, 'owner.openedAt'),
|
||||||
|
};
|
||||||
|
|
||||||
|
const committedAt = toNonNegativeInt(value.committedAt, 'committedAt');
|
||||||
|
|
||||||
|
return {
|
||||||
|
ok: true,
|
||||||
|
manifest: {
|
||||||
|
formatVersion,
|
||||||
|
generation,
|
||||||
|
pageIdWatermark,
|
||||||
|
namespaces,
|
||||||
|
schemas,
|
||||||
|
wal,
|
||||||
|
frozen,
|
||||||
|
owner,
|
||||||
|
committedAt,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
} catch (error) {
|
||||||
|
return { ok: false, reason: (error as Error).message };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// ManifestStore
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
export interface ManifestStoreOptions {
|
||||||
|
backend: IStorageBackend;
|
||||||
|
/** 本实例 id(陈旧实例保护用;默认随机生成) */
|
||||||
|
instanceId?: string;
|
||||||
|
/** 时间源(测试可注入) */
|
||||||
|
now?: () => number;
|
||||||
|
}
|
||||||
|
|
||||||
|
let instanceCounter = 0;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* manifest 读写器:加载最新有效世代、串行提交、维护回退窗口。
|
||||||
|
*
|
||||||
|
* 注意:本类**不做数据落盘**,它只负责"提交点"。调用方必须先让数据真实落盘
|
||||||
|
* (SSTable 页面 flush 完成),再 `commit()`。
|
||||||
|
*/
|
||||||
|
export class ManifestStore {
|
||||||
|
private backend: IStorageBackend;
|
||||||
|
private now: () => number;
|
||||||
|
private state: AriaManifest;
|
||||||
|
private loaded = false;
|
||||||
|
private hadInvalidGenerations = false;
|
||||||
|
private skipped: { generation: number; reason: string }[] = [];
|
||||||
|
/** 提交串行链:把 manifest 写入序列化成严格顺序 */
|
||||||
|
private commitChain: Promise<void> = Promise.resolve();
|
||||||
|
/** 已提交的世代号(= 最新有效世代) */
|
||||||
|
private generation = 0;
|
||||||
|
|
||||||
|
constructor(opts: ManifestStoreOptions) {
|
||||||
|
this.backend = opts.backend;
|
||||||
|
this.now = opts.now ?? (() => Date.now());
|
||||||
|
this.state = createEmptyManifest({
|
||||||
|
instanceId: opts.instanceId ?? `aria-${++instanceCounter}-${Math.random().toString(36).slice(2, 10)}`,
|
||||||
|
now: this.now(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 本实例 id */
|
||||||
|
get instanceId(): string {
|
||||||
|
return this.state.owner.instanceId;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 当前内存态(调用方可直接修改;提交通过 `commit()` 串行化) */
|
||||||
|
get current(): AriaManifest {
|
||||||
|
return this.state;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 最近一次加载/提交后的世代号 */
|
||||||
|
get currentGeneration(): number {
|
||||||
|
return this.generation;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 加载最新有效世代。
|
||||||
|
*
|
||||||
|
* - 没有任何 manifest 文件 → `{ manifest: null }`(全新库);
|
||||||
|
* - 有文件且最新世代有效 → 采用它(更早的损坏世代只记录、不删除);
|
||||||
|
* - 有文件但**全部世代无效** → 抛 `ARIA_MANIFEST_CORRUPT`
|
||||||
|
* (绝不返回空状态:那会让上层把"元数据全坏"当成"空库",
|
||||||
|
* 随后 repair 还会把没人引用的活页删干净 —— 不可逆)。
|
||||||
|
*/
|
||||||
|
async load(): Promise<ManifestLoadResult> {
|
||||||
|
const keys = await this.backend.listKeys();
|
||||||
|
const generations = keys
|
||||||
|
.map((k) => generationFromKey(k))
|
||||||
|
.filter((g): g is number => g !== null)
|
||||||
|
.sort((a, b) => b - a);
|
||||||
|
|
||||||
|
const skipped: { generation: number; reason: string }[] = [];
|
||||||
|
for (const gen of generations) {
|
||||||
|
const raw = await this.backend.read(manifestKey(gen));
|
||||||
|
if (!raw) {
|
||||||
|
skipped.push({ generation: gen, reason: 'manifest file disappeared during load' });
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const decoded = decodeManifest(new Uint8Array(raw));
|
||||||
|
if (!decoded.ok) {
|
||||||
|
skipped.push({ generation: gen, reason: decoded.reason });
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (decoded.manifest.generation !== gen) {
|
||||||
|
skipped.push({
|
||||||
|
generation: gen,
|
||||||
|
reason: `file name generation ${gen} != payload generation ${decoded.manifest.generation}`,
|
||||||
|
});
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
this.state = decoded.manifest;
|
||||||
|
this.generation = gen;
|
||||||
|
this.skipped = skipped;
|
||||||
|
this.hadInvalidGenerations = skipped.length > 0;
|
||||||
|
this.loaded = true;
|
||||||
|
return { manifest: decoded.manifest, generation: gen, hadInvalidGenerations: this.hadInvalidGenerations, skipped };
|
||||||
|
}
|
||||||
|
|
||||||
|
if (generations.length > 0) {
|
||||||
|
// 文件存在但一代都读不出来:这是元数据损坏,不是空库
|
||||||
|
throw new DatabaseError(
|
||||||
|
`AriaEngine manifest is corrupt: ${generations.length} generation(s) present, none passed validation ` +
|
||||||
|
`(${skipped.map((s) => `gen ${s.generation}: ${s.reason}`).join('; ')})`,
|
||||||
|
'ARIA_MANIFEST_CORRUPT',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
this.loaded = true;
|
||||||
|
return { manifest: null, generation: 0, hadInvalidGenerations: false, skipped: [] };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 用一份外部状态替换内存态(旧格式迁移 / 测试用),不落盘 */
|
||||||
|
adopt(manifest: AriaManifest): void {
|
||||||
|
this.state = manifest;
|
||||||
|
this.generation = manifest.generation;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 认领所有权:把当前内存态提交为新世代(owner 换成本实例)。
|
||||||
|
*
|
||||||
|
* 与普通 `commit()` 的区别:这里**允许**接手别人提交的世代
|
||||||
|
* (多标签页的互斥由 Web Locks 负责;即便没有 Web Locks,接手者也只会让
|
||||||
|
* 旧实例后续提交被拒绝,而不是让旧实例静默覆盖新数据)。
|
||||||
|
*/
|
||||||
|
async claimOwnership(): Promise<AriaManifest> {
|
||||||
|
return this.commitInternal({ allowTakeover: true });
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 提交当前内存态为新世代。
|
||||||
|
*
|
||||||
|
* 提交前会**重新读取磁盘上的最新世代号**:若它已经超过本实例上次提交的世代,
|
||||||
|
* 说明另一个实例在我们不知情的情况下提交过(陈旧实例)—— 此时抛
|
||||||
|
* `STALE_INSTANCE`,而不是用陈旧的内存态覆盖新一代(修复前 KVStore 快照被
|
||||||
|
* 静默覆盖、Aria 侧无任何保护)。
|
||||||
|
*/
|
||||||
|
async commit(): Promise<AriaManifest> {
|
||||||
|
return this.commitInternal({ allowTakeover: false });
|
||||||
|
}
|
||||||
|
|
||||||
|
private async commitInternal(opts: { allowTakeover: boolean }): Promise<AriaManifest> {
|
||||||
|
let result!: AriaManifest;
|
||||||
|
let failure: unknown = null;
|
||||||
|
const run = this.commitChain.then(async () => {
|
||||||
|
try {
|
||||||
|
result = await this.doCommit(opts);
|
||||||
|
} catch (error) {
|
||||||
|
failure = error;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
this.commitChain = run;
|
||||||
|
await run;
|
||||||
|
if (failure) throw failure;
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
private async doCommit(opts: { allowTakeover: boolean }): Promise<AriaManifest> {
|
||||||
|
if (!this.loaded) {
|
||||||
|
throw new DatabaseError('ManifestStore.commit() before load()', 'ARIA_MANIFEST_NOT_LOADED');
|
||||||
|
}
|
||||||
|
const onDisk = await this.newestGenerationOnDisk();
|
||||||
|
// 陈旧实例判定要**跳过已知损坏的世代**:损坏世代(撕裂写/介质坏块)不是
|
||||||
|
// "另一个实例提交的状态",把它算进来会让库永远无法再提交(实测:
|
||||||
|
// 一个损坏的更高世代把后续所有提交都拦成 STALE_INSTANCE)。
|
||||||
|
const knownInvalid = new Set(this.skipped.map((s) => s.generation));
|
||||||
|
let newestForeign = 0;
|
||||||
|
for (const gen of await this.listGenerationNumbers()) {
|
||||||
|
if (gen > this.generation && !knownInvalid.has(gen)) newestForeign = Math.max(newestForeign, gen);
|
||||||
|
}
|
||||||
|
if (!opts.allowTakeover && newestForeign > this.generation) {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Refusing to commit: another instance committed generation ${newestForeign} ` +
|
||||||
|
`(this instance last committed ${this.generation}) — stale instance`,
|
||||||
|
'STALE_INSTANCE',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// 但世代号必须大于**任何**已存在的文件(含损坏世代),否则会覆盖它。
|
||||||
|
// 认领(takeover)时一次跨过一段:见 MANIFEST_TAKEOVER_STRIDE 的说明。
|
||||||
|
const nextGeneration = opts.allowTakeover
|
||||||
|
? onDisk + MANIFEST_TAKEOVER_STRIDE
|
||||||
|
: onDisk + 1;
|
||||||
|
const next: AriaManifest = {
|
||||||
|
...this.state,
|
||||||
|
formatVersion: MANIFEST_FORMAT_VERSION,
|
||||||
|
generation: nextGeneration,
|
||||||
|
namespaces: this.state.namespaces,
|
||||||
|
schemas: this.state.schemas,
|
||||||
|
wal: { ...this.state.wal },
|
||||||
|
frozen: this.state.frozen.map((f) => ({ ...f })),
|
||||||
|
owner: {
|
||||||
|
instanceId: this.state.owner.instanceId,
|
||||||
|
epoch: this.state.owner.epoch + 1,
|
||||||
|
openedAt: this.state.owner.openedAt,
|
||||||
|
},
|
||||||
|
committedAt: this.now(),
|
||||||
|
};
|
||||||
|
|
||||||
|
const encoded = encodeManifest(next);
|
||||||
|
await this.backend.write(manifestKey(nextGeneration), encoded.buffer.slice(
|
||||||
|
encoded.byteOffset,
|
||||||
|
encoded.byteOffset + encoded.byteLength,
|
||||||
|
) as ArrayBuffer);
|
||||||
|
|
||||||
|
// 先写后验:回读并校验,只有真正可读回的新世代才算提交成功
|
||||||
|
const readBack = await this.backend.read(manifestKey(nextGeneration));
|
||||||
|
if (!readBack) {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Manifest commit verification failed: generation ${nextGeneration} not readable after write`,
|
||||||
|
'ARIA_MANIFEST_WRITE_FAILED',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const verified = decodeManifest(new Uint8Array(readBack));
|
||||||
|
if (!verified.ok || verified.manifest.generation !== nextGeneration) {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Manifest commit verification failed: generation ${nextGeneration} ` +
|
||||||
|
`(${verified.ok ? 'generation mismatch' : verified.reason})`,
|
||||||
|
'ARIA_MANIFEST_WRITE_FAILED',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
this.state = next;
|
||||||
|
this.generation = nextGeneration;
|
||||||
|
|
||||||
|
// 回退窗口之外的历史世代:只有在**没有损坏世代**时才清理
|
||||||
|
//(存在损坏世代时保留全部,绝不因为"读不出来"就删掉可能是唯一副本的东西)
|
||||||
|
if (!this.hadInvalidGenerations) {
|
||||||
|
await this.pruneOldGenerations(nextGeneration);
|
||||||
|
}
|
||||||
|
return next;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 磁盘上最大的 manifest 世代号(乐观并发检查;只读 key 名单) */
|
||||||
|
async newestGenerationOnDisk(): Promise<number> {
|
||||||
|
return (await this.listGenerationNumbers()).reduce((max, g) => Math.max(max, g), 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 磁盘上全部 manifest 世代号(升序无关,仅用于判定) */
|
||||||
|
private async listGenerationNumbers(): Promise<number[]> {
|
||||||
|
const keys = await this.backend.listKeys();
|
||||||
|
const out: number[] = [];
|
||||||
|
for (const k of keys) {
|
||||||
|
const gen = generationFromKey(k);
|
||||||
|
if (gen !== null) out.push(gen);
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 删除回退窗口之外的世代(当前 + 上一代保留) */
|
||||||
|
private async pruneOldGenerations(currentGeneration: number): Promise<void> {
|
||||||
|
const keys = await this.backend.listKeys();
|
||||||
|
const stale = keys
|
||||||
|
.map((k) => generationFromKey(k))
|
||||||
|
.filter((g): g is number => g !== null && g < currentGeneration - (MANIFEST_RETAIN_GENERATIONS - 1));
|
||||||
|
if (stale.length === 0) return;
|
||||||
|
try {
|
||||||
|
await this.backend.deleteMany(stale.map((g) => manifestKey(g)));
|
||||||
|
} catch {
|
||||||
|
// 删除失败只是残留文件(下次提交再清),不影响正确性
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -18,6 +18,7 @@
|
|||||||
* 浏览器要求:Chrome 102+ / Edge 102+(Safari 15.2+ / Firefox 111+ 支持基础 OPFS)
|
* 浏览器要求:Chrome 102+ / Edge 102+(Safari 15.2+ / Firefox 111+ 支持基础 OPFS)
|
||||||
*/
|
*/
|
||||||
import type { IStorageBackend } from './backend';
|
import type { IStorageBackend } from './backend';
|
||||||
|
import { DatabaseError } from '../../../constants';
|
||||||
|
|
||||||
/** 崩溃残留临时文件后缀(createWritable 底层实现可能留下) */
|
/** 崩溃残留临时文件后缀(createWritable 底层实现可能留下) */
|
||||||
const STALE_SUFFIXES = ['.crswap', '.tmp'];
|
const STALE_SUFFIXES = ['.crswap', '.tmp'];
|
||||||
@@ -30,7 +31,33 @@ export class OPFSBackend implements IStorageBackend {
|
|||||||
|
|
||||||
async open(name: string): Promise<void> {
|
async open(name: string): Promise<void> {
|
||||||
this.dbName = name;
|
this.dbName = name;
|
||||||
this.root = await navigator.storage.getDirectory();
|
// v0.8.0(review 修复):OPFS 不可用时给出**可操作的**错误,而不是让上层
|
||||||
|
// 只能报一句 "Failed to open AriaEngine database"。
|
||||||
|
//
|
||||||
|
// 真实场景(实测):用 file:// 直接打开页面时 Chromium 认为本地文件"不适合
|
||||||
|
// Web 应用访问",`navigator.storage.getDirectory()` 抛 SecurityError —— 而
|
||||||
|
// `isSecureContext` 仍是 true、API 也**存在**,所以只有真正调用才会发现。
|
||||||
|
// 修复前这个 SecurityError 被 `ARIA_OPEN_ERROR` 吞成一句无从下手的消息
|
||||||
|
//(cause 没往上带),用户不知道是"环境不支持"还是"库坏了"。
|
||||||
|
if (typeof navigator === 'undefined' || !navigator.storage
|
||||||
|
|| typeof navigator.storage.getDirectory !== 'function') {
|
||||||
|
throw new DatabaseError(
|
||||||
|
'OPFS is not available in this environment: navigator.storage.getDirectory is missing. ' +
|
||||||
|
"Use the http(s) protocol (file:// is rejected by browsers) or switch to mode: 'memory' / a KVStore backend.",
|
||||||
|
'ARIA_OPFS_UNAVAILABLE',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
this.root = await navigator.storage.getDirectory();
|
||||||
|
} catch (error) {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`OPFS is not accessible here (${(error as Error).name}: ${(error as Error).message}). ` +
|
||||||
|
'Chromium blocks OPFS for file:// pages — serve the page over http(s) ' +
|
||||||
|
"(e.g. `npx serve .` / `node tests/e2e/server.cjs`) or use mode: 'memory'.",
|
||||||
|
'ARIA_OPFS_UNAVAILABLE',
|
||||||
|
error,
|
||||||
|
);
|
||||||
|
}
|
||||||
this.dbDir = await this.root.getDirectoryHandle(name, { create: true });
|
this.dbDir = await this.root.getDirectoryHandle(name, { create: true });
|
||||||
// v0.4.5: 清理崩溃残留临时文件(不阻塞打开)
|
// v0.4.5: 清理崩溃残留临时文件(不阻塞打开)
|
||||||
await this.cleanupStaleFiles();
|
await this.cleanupStaleFiles();
|
||||||
|
|||||||
@@ -18,19 +18,56 @@ import type { FileManager } from './file_manager';
|
|||||||
import type { BufferPool } from '../buffer/pool';
|
import type { BufferPool } from '../buffer/pool';
|
||||||
import type { PageHandle } from '../types';
|
import type { PageHandle } from '../types';
|
||||||
import { PAGE_SIZE, PageType } from '../types';
|
import { PAGE_SIZE, PageType } from '../types';
|
||||||
|
import { compressLZ4, decompressLZ4 } from '../compression/lz4';
|
||||||
|
|
||||||
export class PageSSTableStore {
|
export class PageSSTableStore {
|
||||||
/** SSTable id → 页面 ID 列表(save 时记录,saveMeta 时注入 meta) */
|
/** SSTable id → 页面 ID 列表(save 时记录,saveMeta 时注入 meta) */
|
||||||
private pageIds = new Map<number, number[]>();
|
private pageIds = new Map<number, number[]>();
|
||||||
|
/**
|
||||||
|
* v0.8.0(B-6):已退休(被 compaction 取代 / 已被 manifest 摘除)但仍可能有
|
||||||
|
* 在途读者持有引用的 SSTable → 页面 ID 列表。
|
||||||
|
*
|
||||||
|
* 为什么必须保留:读路径是"先取 meta 快照、再按 id 加载数据",快照与加载之间
|
||||||
|
* 可以插入一次 compaction。若退休时立刻忘掉 pageIds,在途读者的 `load()` 就
|
||||||
|
* 找不到页面,只能把"文件已退休"误判为"数据缺失"(旧代码会顺手删掉 meta 并
|
||||||
|
* 打一条损坏告警)。保留到物理删除为止,语义才是自洽的。
|
||||||
|
*/
|
||||||
|
private retiredPageIds = new Map<number, number[]>();
|
||||||
|
/** SSTable id → 落盘字节数(load 时截断最后一页 0 填充) */
|
||||||
|
private storedSizes = new Map<number, number>();
|
||||||
|
|
||||||
constructor(
|
constructor(
|
||||||
private fileManager: FileManager,
|
private fileManager: FileManager,
|
||||||
private bufferPool: BufferPool,
|
private bufferPool: BufferPool,
|
||||||
|
/**
|
||||||
|
* v0.8.0(A38):是否压缩。
|
||||||
|
*
|
||||||
|
* 修复前 `config.compression` 只作用于"整 value 存一个 backend value"的旧路径,
|
||||||
|
* 而 `save()` 在页面化路径上**提前 return**,压缩分支根本走不到 —— 于是
|
||||||
|
* `pageStorage: true`(默认)时 `compression: true` 被完全忽略,
|
||||||
|
* 用户打开了压缩却没有任何压缩效果,且没有任何提示。
|
||||||
|
*
|
||||||
|
* 为什么在页面化里压缩整个流而不是逐页压缩:
|
||||||
|
* - 压缩率取决于"连续数据的重复窗口";4KB 页各自压缩会丢失跨页匹配,
|
||||||
|
* 压缩率显著低于整体压缩;
|
||||||
|
* - 整体压缩后仍是**字节流**,切页照旧,页面布局与 pageIds 语义不变 ——
|
||||||
|
* 对 meta/文件布局零影响。
|
||||||
|
*/
|
||||||
|
private compression: boolean = false,
|
||||||
) {}
|
) {}
|
||||||
|
|
||||||
/** 保存数据:切页 → 写入 BufferPool → 逐页落盘 → 记录 pageIds */
|
/**
|
||||||
async save(id: number, data: Uint8Array): Promise<void> {
|
* 保存数据:切页 → 写入 BufferPool → 逐页落盘 → 记录 pageIds。
|
||||||
const pageCount = Math.max(1, Math.ceil(data.byteLength / PAGE_SIZE));
|
*
|
||||||
|
* @returns `storedSize` = **实际落盘字节数**(压缩后)。调用方写入
|
||||||
|
* `SSTableMeta.totalSize` 时应使用它 —— totalSize 的语义是
|
||||||
|
* "页面里有多少字节",加载时按它截断;若仍写未压缩长度,
|
||||||
|
* 压缩后的数据会被 0 填充撑大(静默损坏)。
|
||||||
|
*/
|
||||||
|
async save(id: number, data: Uint8Array): Promise<{ storedSize: number }> {
|
||||||
|
// v0.8.0(A38):压缩先于切页(整体压缩,压缩率优于逐页)
|
||||||
|
const payload = this.compression ? compressLZ4(data) : data;
|
||||||
|
const pageCount = Math.max(1, Math.ceil(payload.byteLength / PAGE_SIZE));
|
||||||
const handles: PageHandle[] = await this.bufferPool.newPages(pageCount, PageType.DATA);
|
const handles: PageHandle[] = await this.bufferPool.newPages(pageCount, PageType.DATA);
|
||||||
|
|
||||||
const ids: number[] = [];
|
const ids: number[] = [];
|
||||||
@@ -39,7 +76,7 @@ export class PageSSTableStore {
|
|||||||
ids.push(page.pageId);
|
ids.push(page.pageId);
|
||||||
const dest = new Uint8Array(page.data);
|
const dest = new Uint8Array(page.data);
|
||||||
dest.fill(0); // 清空(最后一页可能不满)
|
dest.fill(0); // 清空(最后一页可能不满)
|
||||||
const slice = data.subarray(i * PAGE_SIZE, Math.min((i + 1) * PAGE_SIZE, data.byteLength));
|
const slice = payload.subarray(i * PAGE_SIZE, Math.min((i + 1) * PAGE_SIZE, payload.byteLength));
|
||||||
dest.set(slice, 0);
|
dest.set(slice, 0);
|
||||||
page.dirty = true;
|
page.dirty = true;
|
||||||
// save 语义 = 已持久化:立即落盘(WAL checkpoint 截断依赖此保证)
|
// save 语义 = 已持久化:立即落盘(WAL checkpoint 截断依赖此保证)
|
||||||
@@ -47,22 +84,47 @@ export class PageSSTableStore {
|
|||||||
this.bufferPool.unpin(page);
|
this.bufferPool.unpin(page);
|
||||||
}
|
}
|
||||||
this.pageIds.set(id, ids);
|
this.pageIds.set(id, ids);
|
||||||
|
this.retiredPageIds.delete(id);
|
||||||
|
this.storedSizes.set(id, payload.byteLength);
|
||||||
|
return { storedSize: payload.byteLength };
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 获取指定 SSTable 的页面 ID 列表(saveMeta 注入用) */
|
/** 获取指定 SSTable 的页面 ID 列表(saveMeta 注入用;未注册返回 undefined) */
|
||||||
getPageIds(id: number): number[] | undefined {
|
getPageIds(id: number): number[] | undefined {
|
||||||
return this.pageIds.get(id);
|
return this.pageIds.get(id) ?? this.retiredPageIds.get(id);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** v0.8.0:打开时从 manifest 把已有 SSTable 的页面映射注册进来(load 不再依赖调用方传参) */
|
||||||
|
registerPageIds(id: number, ids: number[], storedSize?: number): void {
|
||||||
|
if (this.pageIds.has(id)) return;
|
||||||
|
this.pageIds.set(id, [...ids]);
|
||||||
|
if (typeof storedSize === 'number') this.storedSizes.set(id, storedSize);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0:标记某 SSTable 已退休(被合并产物取代)。
|
||||||
|
* 页面映射保留,在途读者仍能读到旧数据;物理删除由 `delete()` 完成。
|
||||||
|
*/
|
||||||
|
retirePageIds(id: number): void {
|
||||||
|
const ids = this.pageIds.get(id);
|
||||||
|
if (!ids) return;
|
||||||
|
this.pageIds.delete(id);
|
||||||
|
this.retiredPageIds.set(id, ids);
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* 按页面 ID 列表读取并拼接为完整字节流。
|
* 按页面 ID 列表读取并拼接为完整字节流。
|
||||||
* @param totalSize SSTable 真实大小(meta 持久化)——最后一页可能有 0 填充,按真实大小截断
|
* @param pageIds 页面 ID 列表(缺省时用内部注册的映射)
|
||||||
|
* @param totalSize 页面中**实际存储**的字节数(缺省时用内部记录)
|
||||||
* @returns 缺失页面/读取失败返回 null(调用方视为损坏并清理)
|
* @returns 缺失页面/读取失败返回 null(调用方视为损坏并清理)
|
||||||
*/
|
*/
|
||||||
async load(id: number, pageIds: number[], totalSize: number): Promise<Uint8Array | null> {
|
async load(id: number, pageIds?: number[], totalSize?: number): Promise<Uint8Array | null> {
|
||||||
if (pageIds.length === 0) return null;
|
const ids = pageIds ?? this.pageIds.get(id) ?? this.retiredPageIds.get(id);
|
||||||
|
if (!ids || ids.length === 0) return null;
|
||||||
|
const size = totalSize ?? this.storedSizes.get(id);
|
||||||
|
if (typeof size !== 'number') return null;
|
||||||
const chunks: Uint8Array[] = [];
|
const chunks: Uint8Array[] = [];
|
||||||
for (const pageId of pageIds) {
|
for (const pageId of ids) {
|
||||||
const page = await this.bufferPool.getPage(pageId);
|
const page = await this.bufferPool.getPage(pageId);
|
||||||
if (!page) return null;
|
if (!page) return null;
|
||||||
// 立即复制(后续驱逐安全)
|
// 立即复制(后续驱逐安全)
|
||||||
@@ -70,7 +132,7 @@ export class PageSSTableStore {
|
|||||||
this.bufferPool.unpin(page);
|
this.bufferPool.unpin(page);
|
||||||
}
|
}
|
||||||
const total = chunks.reduce((s, c) => s + c.byteLength, 0);
|
const total = chunks.reduce((s, c) => s + c.byteLength, 0);
|
||||||
const out = new Uint8Array(Math.min(total, totalSize));
|
const out = new Uint8Array(Math.min(total, size));
|
||||||
let off = 0;
|
let off = 0;
|
||||||
for (const c of chunks) {
|
for (const c of chunks) {
|
||||||
const take = Math.min(c.byteLength, out.byteLength - off);
|
const take = Math.min(c.byteLength, out.byteLength - off);
|
||||||
@@ -78,18 +140,23 @@ export class PageSSTableStore {
|
|||||||
out.set(c.subarray(0, take), off);
|
out.set(c.subarray(0, take), off);
|
||||||
off += take;
|
off += take;
|
||||||
}
|
}
|
||||||
this.pageIds.delete(id);
|
// v0.8.0:**不再**在这里丢掉 pageIds —— 退休 SSTable 的在途读者仍会调用 load,
|
||||||
return out;
|
// 丢掉映射会让它们把"已退休"误判成"数据缺失"。物理删除由 delete() 负责。
|
||||||
|
// v0.8.0(A38):解压(与 save 的加密/压缩顺序对称)
|
||||||
|
return this.compression ? decompressLZ4(out) : out;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 释放页面(删除物理页面文件 + 移出 BufferPool) */
|
/** 释放页面(删除物理页面文件 + 移出 BufferPool;同时清掉活跃与退休映射) */
|
||||||
async delete(id: number, pageIds: number[]): Promise<void> {
|
async delete(id: number, pageIds?: number[]): Promise<void> {
|
||||||
for (const pageId of pageIds) {
|
const ids = pageIds ?? this.pageIds.get(id) ?? this.retiredPageIds.get(id) ?? [];
|
||||||
|
for (const pageId of ids) {
|
||||||
this.bufferPool.removePage(pageId);
|
this.bufferPool.removePage(pageId);
|
||||||
try {
|
try {
|
||||||
await this.fileManager.freePageId(pageId);
|
await this.fileManager.freePageId(pageId);
|
||||||
} catch { /* 清理失败不阻塞 */ }
|
} catch { /* 清理失败不阻塞 */ }
|
||||||
}
|
}
|
||||||
this.pageIds.delete(id);
|
this.pageIds.delete(id);
|
||||||
|
this.retiredPageIds.delete(id);
|
||||||
|
this.storedSizes.delete(id);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -2,8 +2,11 @@
|
|||||||
* AriaEngine MVCC — 多版本并发控制
|
* AriaEngine MVCC — 多版本并发控制
|
||||||
* @module engine/aria/transaction/mvcc
|
* @module engine/aria/transaction/mvcc
|
||||||
*
|
*
|
||||||
* 实现快照隔离 (Snapshot Isolation)。
|
* v0.8.0(review 修正文档):本模块提供**行版本链**,用途是"事务内的 undo",
|
||||||
* 每个事务看到数据库在事务开始时的快照。
|
* 提交即清理。它**不提供快照隔离** —— 引擎的隔离语义是"事务串行"(同一实例
|
||||||
|
* 同时只允许一个事务,并发 `beginTransaction` 抛 `TX_ACTIVE`),读取走
|
||||||
|
* `txnSnapshot` 未提交快照。旧注释声称 "实现快照隔离 (Snapshot Isolation)"
|
||||||
|
* 与实现不符(全量审查发现)。
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import type { RowVersion, TxnEntry } from '../types';
|
import type { RowVersion, TxnEntry } from '../types';
|
||||||
|
|||||||
@@ -1,3 +1,4 @@
|
|||||||
|
import type { IStorageBackend } from './store/backend';
|
||||||
/**
|
/**
|
||||||
* AriaEngine Types — 内部类型定义
|
* AriaEngine Types — 内部类型定义
|
||||||
* @module engine/aria/types
|
* @module engine/aria/types
|
||||||
@@ -121,6 +122,32 @@ export enum WALRecordType {
|
|||||||
ROLLBACK = 6,
|
ROLLBACK = 6,
|
||||||
CREATE_TABLE = 7,
|
CREATE_TABLE = 7,
|
||||||
DROP_TABLE = 8,
|
DROP_TABLE = 8,
|
||||||
|
/**
|
||||||
|
* v0.8.0: 回滚到保存点。
|
||||||
|
*
|
||||||
|
* 为什么必须有这条记录:`ROLLBACK TO <savepoint>` 此前只改内存快照、**不写 WAL**,
|
||||||
|
* 而 COMMIT 会把整个 txnId 标记为已提交,恢复时按"该事务的全部记录"重放 ——
|
||||||
|
* 于是被 savepoint 回滚掉的行在崩溃重启后**复活**(实测:实时只剩 a,
|
||||||
|
* 崩溃重开变成 a+b)。反向也有问题:跨事务复用的陈旧 savepoint 会让当前事务的
|
||||||
|
* 写入被上一事务的快照静默替换。
|
||||||
|
*
|
||||||
|
* 记录语义:该事务在此之前的写入都应被丢弃 —— 恢复时只应用该事务**最后一条**
|
||||||
|
* SAVEPOINT_ROLLBACK 之后的记录。
|
||||||
|
*/
|
||||||
|
SAVEPOINT_ROLLBACK = 9,
|
||||||
|
/**
|
||||||
|
* v0.8.0(A41):ALTER TABLE 的**意图**记录。
|
||||||
|
*
|
||||||
|
* 此前 `alterTable` 完全不写 WAL:它先改内存 schema、必要时建索引,最后才
|
||||||
|
* `persistSchemas()`。中间任何一步抛错(如"ALTER ADD UNIQUE 撞存量重复值")
|
||||||
|
* 都会留下**内存已变、磁盘未变**的分裂状态 —— 同进程里 `getTableSchema`
|
||||||
|
* 看到新列,重开后新列又消失(用户视角:ALTER 时好时坏、结果取决于是否重启)。
|
||||||
|
* 加上意图记录后,恢复可以按记录把"内存里已经生效"的结构变更补齐。
|
||||||
|
*
|
||||||
|
* 记录语义:`data.schema` 为变更后的完整 schema JSON;回放时**覆盖**该表 schema
|
||||||
|
*(ALTER 是结构权威描述,不是增量),并对新增的 index/unique 列重建索引。
|
||||||
|
*/
|
||||||
|
ALTER_TABLE = 10,
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 单条 WAL 记录 */
|
/** 单条 WAL 记录 */
|
||||||
@@ -135,7 +162,15 @@ export interface WALRecord {
|
|||||||
tableName: string;
|
tableName: string;
|
||||||
/** 主键值 */
|
/** 主键值 */
|
||||||
key: string;
|
key: string;
|
||||||
/** 操作数据(INSERT/UPDATE 时有效) */
|
/**
|
||||||
|
* 操作数据(INSERT/UPDATE/DELETE 时有效)。
|
||||||
|
*
|
||||||
|
* v0.8.0: SAVEPOINT_ROLLBACK 记录借用该字段携带边界信息:
|
||||||
|
* `{ replayFromIndex: N }` 表示"创建该保存点时,本事务已成功追加了 N 条记录"。
|
||||||
|
* 恢复时据此丢弃 [N, 该标记) 区间的写入 —— 等价于回到该保存点,
|
||||||
|
* 同时**保留** N 之前的写入(这是"回到保存点"而非"回滚整个事务"的关键)。
|
||||||
|
* 复用既有字段意味着 WAL 二进制格式无需变更,旧库记录仍可解析。
|
||||||
|
*/
|
||||||
data?: Record<string, unknown>;
|
data?: Record<string, unknown>;
|
||||||
/** 校验和 */
|
/** 校验和 */
|
||||||
checksum: number;
|
checksum: number;
|
||||||
@@ -226,11 +261,25 @@ export interface AriaEngineConfig {
|
|||||||
* 显式 false 强制关闭(整 value 存储,兼容旧行为)。
|
* 显式 false 强制关闭(整 value 存储,兼容旧行为)。
|
||||||
*/
|
*/
|
||||||
pageStorage?: boolean;
|
pageStorage?: boolean;
|
||||||
|
/**
|
||||||
|
* **仅测试使用**:直接注入存储后端(跳过 storageBackend 选择逻辑)。
|
||||||
|
*
|
||||||
|
* 为什么需要这个口子:崩溃/撕裂语义的验证必须让引擎**从打开那一刻起**
|
||||||
|
* 就走被测后端 —— `this.backend` 在 `open()` 里被 WAL、FileManager、
|
||||||
|
* SSTableStore 一起捕获,事后替换 `engine.backend` 只会替换其中一部分
|
||||||
|
*(实测:替换后 WAL 仍写旧后端,于是"崩溃"根本没覆盖 WAL 路径,
|
||||||
|
* 探针得出的是假结论)。缺少这个口子会让所有故障注入只能在孤立后端上
|
||||||
|
* 验证,而无法验证"引擎整体在崩裂介质上的行为"。
|
||||||
|
*/
|
||||||
|
testBackend?: IStorageBackend;
|
||||||
}
|
}
|
||||||
|
|
||||||
export const DEFAULT_ARIA_CONFIG: Required<Omit<AriaEngineConfig, 'encryption' | 'pageStorage'>> & {
|
export const DEFAULT_ARIA_CONFIG: Required<
|
||||||
|
Omit<AriaEngineConfig, 'encryption' | 'pageStorage' | 'testBackend'>
|
||||||
|
> & {
|
||||||
encryption: undefined;
|
encryption: undefined;
|
||||||
pageStorage: undefined;
|
pageStorage: undefined;
|
||||||
|
testBackend: undefined;
|
||||||
} = {
|
} = {
|
||||||
pageSize: PAGE_SIZE,
|
pageSize: PAGE_SIZE,
|
||||||
bufferPoolPages: DEFAULT_BUFFER_POOL_PAGES,
|
bufferPoolPages: DEFAULT_BUFFER_POOL_PAGES,
|
||||||
@@ -246,4 +295,5 @@ export const DEFAULT_ARIA_CONFIG: Required<Omit<AriaEngineConfig, 'encryption' |
|
|||||||
maxMemoryMB: 64,
|
maxMemoryMB: 64,
|
||||||
encryption: undefined,
|
encryption: undefined,
|
||||||
pageStorage: undefined,
|
pageStorage: undefined,
|
||||||
|
testBackend: undefined,
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -12,6 +12,18 @@ import type { WAL } from './log';
|
|||||||
|
|
||||||
export interface Flushable {
|
export interface Flushable {
|
||||||
flushAll(): Promise<void>;
|
flushAll(): Promise<void>;
|
||||||
|
/**
|
||||||
|
* v0.8.0(B-6/55):**只落 memtable**(不等待 compaction)。
|
||||||
|
*
|
||||||
|
* 为什么必须分开:checkpoint 要保证的是"WAL 覆盖的数据已落盘",而 compaction
|
||||||
|
* 只是重排**已经落盘**的 SSTable。修复前 checkpoint 会先 `lsm.flush()` 再
|
||||||
|
* `flushable.flushAll()`,两者都会排空后台维护链 —— 后台 compaction 跑几秒,
|
||||||
|
* 写路径每 1000 次操作就要等它一次(v0.6.1 记录的 "8~11s 悬崖"的另一半)。
|
||||||
|
*
|
||||||
|
* 未实现该方法的调用方退回旧语义(先 flush LSM 再 flushAll),
|
||||||
|
* 因此这个接口对既有测试替身保持兼容。
|
||||||
|
*/
|
||||||
|
flushMemtables?(): Promise<void>;
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -60,9 +72,15 @@ export class CheckpointManager {
|
|||||||
}
|
}
|
||||||
|
|
||||||
async checkpoint(): Promise<void> {
|
async checkpoint(): Promise<void> {
|
||||||
await this.lsm.flush();
|
if (this.flushable && typeof this.flushable.flushMemtables === 'function') {
|
||||||
if (this.flushable) {
|
// v0.8.0(B-6/55):只等 memtable 落盘;compaction 继续在后台跑
|
||||||
await this.flushable.flushAll();
|
await this.flushable.flushMemtables();
|
||||||
|
} else {
|
||||||
|
// 兼容路径(测试替身 / 未实现新接口的调用方):旧语义
|
||||||
|
await this.lsm.flush();
|
||||||
|
if (this.flushable) {
|
||||||
|
await this.flushable.flushAll();
|
||||||
|
}
|
||||||
}
|
}
|
||||||
await this.wal.checkpoint();
|
await this.wal.checkpoint();
|
||||||
this.opCount = 0;
|
this.opCount = 0;
|
||||||
|
|||||||
+187
-14
@@ -20,8 +20,10 @@
|
|||||||
* └──────────┴──────────────┴──────────┘
|
* └──────────┴──────────────┴──────────┘
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
import { DatabaseError } from '../../../constants';
|
||||||
import { WALRecordType, type WALRecord } from '../types';
|
import { WALRecordType, type WALRecord } from '../types';
|
||||||
import { crc32 } from '../crc32';
|
import { crc32 } from '../crc32';
|
||||||
|
import type { WALReadResult } from './segmented_store';
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// WAL 存储接口
|
// WAL 存储接口
|
||||||
@@ -36,6 +38,72 @@ export interface WALStore {
|
|||||||
truncate(): Promise<void>;
|
truncate(): Promise<void>;
|
||||||
/** 检查 WAL 是否存在 */
|
/** 检查 WAL 是否存在 */
|
||||||
exists(): Promise<boolean>;
|
exists(): Promise<boolean>;
|
||||||
|
/**
|
||||||
|
* v0.8.0(可选,分片存储实现):从指定分片起读取,并把空洞如实返回。
|
||||||
|
* 未实现的 store 由 WAL 回退为 `readAll()`(无空洞信息)。
|
||||||
|
*/
|
||||||
|
readAllFrom?(fromSegment: number, fromLsn?: number): Promise<WALReadResult>;
|
||||||
|
/**
|
||||||
|
* v0.8.0(可选,分片存储实现):删除整体已落盘的前缀分片,
|
||||||
|
* 返回仍需保留的最小分片号。未实现时 WAL 只在"全部已落盘"时整体截断。
|
||||||
|
*
|
||||||
|
* @param latestLsn 当前 LSN 高水位(判断"最后一个分片"是否也被完整覆盖)
|
||||||
|
*/
|
||||||
|
truncateBefore?(durableLsn: number, latestLsn?: number): Promise<number>;
|
||||||
|
/**
|
||||||
|
* v0.8.0(可选):整库清空后重置分片编号(仅 `clearAll` 这类"介质整体抹掉、
|
||||||
|
* manifest 也从 0 开始"的场景)。
|
||||||
|
*/
|
||||||
|
reset?(): void;
|
||||||
|
/**
|
||||||
|
* v0.8.0(可选,分片存储实现):**只计算**仍需保留的最小分片号(无副作用)。
|
||||||
|
* 调用方必须先把它提交进 manifest,再调用 `truncateBefore` 删除 —— 顺序反了
|
||||||
|
* 会让恢复无法区分"正常清理过的前缀"与"介质丢了一段记录"。
|
||||||
|
*/
|
||||||
|
planKeepFrom?(durableLsn: number, latestLsn?: number): Promise<number>;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** WAL 恢复的边界选项(v0.8.0) */
|
||||||
|
export interface WALRecoverOptions {
|
||||||
|
/**
|
||||||
|
* 跳过 `lsn <= fromLsn` 的记录。
|
||||||
|
*
|
||||||
|
* 这些记录已由 manifest 证明存在于已提交的 SSTable 中(`wal.startLsn`),
|
||||||
|
* 重放它们不仅浪费启动时间,还会把**已经删除的数据复活**
|
||||||
|
*(旧记录晚于 tombstone 重放时)。
|
||||||
|
*/
|
||||||
|
fromLsn?: number;
|
||||||
|
/** 从该分片号开始读取(manifest 的 `wal.startSegment`) */
|
||||||
|
fromSegment?: number;
|
||||||
|
/**
|
||||||
|
* 允许活跃区间内的分片空洞。默认 false → 抛 `ARIA_WAL_GAP`。
|
||||||
|
*
|
||||||
|
* 为什么默认抛错:空洞意味着"某个已提交事务的记录缺失",静默继续会把
|
||||||
|
* 不完整的状态当成完整状态。引擎显式传 `allowGaps: true` 并把它写进
|
||||||
|
* 恢复报告(用户可见),而不是让调用方悄悄少一批数据。
|
||||||
|
*/
|
||||||
|
allowGaps?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 最近一次恢复的诊断信息 */
|
||||||
|
export interface WALRecoveryInfo {
|
||||||
|
applied: number;
|
||||||
|
skipped: number;
|
||||||
|
maxLsn: number;
|
||||||
|
/** 活跃区间**内部**的分片空洞(这些分片之后的记录不可信,已丢弃) */
|
||||||
|
gaps: number[];
|
||||||
|
/**
|
||||||
|
* v0.8.0(review 修复):起始分片之前缺失的分片号("前缀缺失")。
|
||||||
|
*
|
||||||
|
* 与 `gaps` 区分:前缀缺失可能是**正常代价**(manifest 回退到上一代时,
|
||||||
|
* 它的 `startSegment` 比介质上现存的最小分片更小 —— 那段记录本就被水位跳过),
|
||||||
|
* 因此不能一律当错误;只有内部空洞才意味着记录真的不可信。
|
||||||
|
*/
|
||||||
|
missingPrefix: number[];
|
||||||
|
/** v0.8.0:CRC 校验失败被跳过的记录数(它们承载的写入已丢失) */
|
||||||
|
corruptRecords: number;
|
||||||
|
fromSegment: number;
|
||||||
|
fromLsn: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -50,6 +118,8 @@ export class WAL {
|
|||||||
private syncMode: 'full' | 'batch' | 'none';
|
private syncMode: 'full' | 'batch' | 'none';
|
||||||
/** v0.3.3: 未 checkpoint 的 WAL 累计字节数(full/batch/none 通用) */
|
/** v0.3.3: 未 checkpoint 的 WAL 累计字节数(full/batch/none 通用) */
|
||||||
private bufferedBytes = 0;
|
private bufferedBytes = 0;
|
||||||
|
/** v0.8.0: 最近一次恢复诊断 */
|
||||||
|
private lastRecoveryInfo: WALRecoveryInfo | null = null;
|
||||||
|
|
||||||
constructor(store: WALStore, enabled: boolean = true, syncMode: 'full' | 'batch' | 'none' = 'batch') {
|
constructor(store: WALStore, enabled: boolean = true, syncMode: 'full' | 'batch' | 'none' = 'batch') {
|
||||||
this.store = store;
|
this.store = store;
|
||||||
@@ -137,22 +207,79 @@ export class WAL {
|
|||||||
/** 从 WAL 恢复未提交的事务数据 */
|
/** 从 WAL 恢复未提交的事务数据 */
|
||||||
async recover(
|
async recover(
|
||||||
applyRecord: (record: WALRecord) => void,
|
applyRecord: (record: WALRecord) => void,
|
||||||
|
opts: WALRecoverOptions = {},
|
||||||
): Promise<number> {
|
): Promise<number> {
|
||||||
if (!this.enabled) return 0;
|
if (!this.enabled) return 0;
|
||||||
|
|
||||||
const exists = await this.store.exists();
|
const fromSegment = opts.fromSegment ?? 0;
|
||||||
if (!exists) return 0;
|
const fromLsn = opts.fromLsn ?? 0;
|
||||||
|
let data: Uint8Array;
|
||||||
|
let gaps: number[] = [];
|
||||||
|
let missingPrefix: number[] = [];
|
||||||
|
|
||||||
const data = await this.store.readAll();
|
if (typeof this.store.readAllFrom === 'function') {
|
||||||
if (data.byteLength === 0) return 0;
|
const result = await this.store.readAllFrom(fromSegment, fromLsn);
|
||||||
|
data = result.data;
|
||||||
const records = this.decodeAllRecords(data);
|
gaps = result.gaps;
|
||||||
for (const record of records) {
|
missingPrefix = result.missingPrefix ?? [];
|
||||||
applyRecord(record);
|
} else {
|
||||||
|
const exists = await this.store.exists();
|
||||||
|
if (!exists) return 0;
|
||||||
|
data = await this.store.readAll();
|
||||||
}
|
}
|
||||||
|
|
||||||
this.lsn = records.length > 0 ? records[records.length - 1].lsn : 0;
|
if (gaps.length > 0 && !opts.allowGaps) {
|
||||||
return records.length;
|
throw new DatabaseError(
|
||||||
|
`WAL segment gap detected in live range (missing segment(s): ${gaps.join(', ')}) — ` +
|
||||||
|
'records after the gap cannot be verified; refusing to continue silently',
|
||||||
|
'ARIA_WAL_GAP',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const decoded = data.byteLength === 0 ? { records: [], corrupt: 0 } : this.decodeAllRecords(data);
|
||||||
|
const records = decoded.records;
|
||||||
|
let applied = 0;
|
||||||
|
let skipped = 0;
|
||||||
|
let maxLsn = 0;
|
||||||
|
for (const record of records) {
|
||||||
|
if (record.lsn > maxLsn) maxLsn = record.lsn;
|
||||||
|
// v0.8.0: lsn <= startLsn 的记录已由 manifest 证明落盘,跳过(不再重复重放)
|
||||||
|
if (record.lsn <= fromLsn) {
|
||||||
|
skipped++;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
applyRecord(record);
|
||||||
|
applied++;
|
||||||
|
}
|
||||||
|
|
||||||
|
this.lsn = Math.max(this.lsn, maxLsn);
|
||||||
|
// 磁盘上的全部记录已经读出来了,缓冲区里不再有待落盘记录
|
||||||
|
this.buffer = [];
|
||||||
|
this.bufferedBytes = 0;
|
||||||
|
this.lastRecoveryInfo = {
|
||||||
|
applied, skipped, maxLsn, gaps, missingPrefix,
|
||||||
|
corruptRecords: decoded.corrupt,
|
||||||
|
fromSegment, fromLsn,
|
||||||
|
};
|
||||||
|
return applied;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** v0.8.0: 最近一次恢复诊断(applied/skipped/gaps) */
|
||||||
|
getLastRecoveryInfo(): WALRecoveryInfo | null {
|
||||||
|
return this.lastRecoveryInfo ? { ...this.lastRecoveryInfo, gaps: [...this.lastRecoveryInfo.gaps] } : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** v0.8.0: 当前 LSN 高水位(manifest 记录它以保证重启后 LSN 继续单调) */
|
||||||
|
getLsn(): number {
|
||||||
|
return this.lsn;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0: 从 manifest 的高水位继续 LSN(重启后 LSN 全库单调,永不复用)。
|
||||||
|
* 只允许上调,不允许回退。
|
||||||
|
*/
|
||||||
|
setLsn(lsn: number): void {
|
||||||
|
if (Number.isFinite(lsn) && lsn > this.lsn) this.lsn = lsn;
|
||||||
}
|
}
|
||||||
|
|
||||||
// =======================================================================
|
// =======================================================================
|
||||||
@@ -164,8 +291,50 @@ export class WAL {
|
|||||||
if (!this.enabled) return;
|
if (!this.enabled) return;
|
||||||
await this.flush();
|
await this.flush();
|
||||||
await this.store.truncate();
|
await this.store.truncate();
|
||||||
this.lsn = 0;
|
|
||||||
this.bufferedBytes = 0;
|
this.bufferedBytes = 0;
|
||||||
|
// v0.8.0: **不再把 lsn 归零**。LSN 是 manifest 记录的全库单调水位
|
||||||
|
// (`wal.nextLsn`),归零会让"跳过 lsn <= startLsn"的判定与历史分片冲突:
|
||||||
|
// 同一段 LSN 区间会对应两批完全不同的记录。
|
||||||
|
}
|
||||||
|
|
||||||
|
/** v0.8.0:整库清空后重置分片编号(供 `clearAll` 使用) */
|
||||||
|
reset(): void {
|
||||||
|
if (typeof this.store.reset === 'function') this.store.reset();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0(B-6):查询"提交之后可以删到哪个分片"(无副作用)。
|
||||||
|
* @returns 仍需保留的最小分片号(未实现分片能力的 store 返回 0 = 全部保留)
|
||||||
|
*/
|
||||||
|
async planKeepFromSegment(durableLsn: number): Promise<number> {
|
||||||
|
if (!this.enabled) return 0;
|
||||||
|
if (typeof this.store.planKeepFrom === 'function') {
|
||||||
|
return this.store.planKeepFrom(durableLsn, this.lsn);
|
||||||
|
}
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0(B-6):**按落盘水位**截断 WAL —— manifest 提交之后的清理动作。
|
||||||
|
*
|
||||||
|
* 与 `checkpoint()` 的区别:这里只删除"整段记录都 <= durableLsn"的前缀分片,
|
||||||
|
* 因此可以安全地在**后台 compaction 仍在进行**时调用。
|
||||||
|
*
|
||||||
|
* @returns 仍需保留的最小分片号(调用方应写入 manifest 的 `wal.startSegment`)
|
||||||
|
*/
|
||||||
|
async checkpointBefore(durableLsn: number): Promise<number> {
|
||||||
|
if (!this.enabled) return 0;
|
||||||
|
// 先把缓冲记录落盘:否则"边界分片"的判定会漏掉刚写进缓冲的记录
|
||||||
|
await this.flush();
|
||||||
|
if (typeof this.store.truncateBefore === 'function') {
|
||||||
|
return this.store.truncateBefore(durableLsn, this.lsn);
|
||||||
|
}
|
||||||
|
// 回退(无分片能力的 store):只有确认"全部记录都已落盘"才整体截断
|
||||||
|
if (durableLsn >= this.lsn) {
|
||||||
|
await this.store.truncate();
|
||||||
|
this.bufferedBytes = 0;
|
||||||
|
}
|
||||||
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
// =======================================================================
|
// =======================================================================
|
||||||
@@ -244,8 +413,9 @@ export class WAL {
|
|||||||
return new Uint8Array(buf);
|
return new Uint8Array(buf);
|
||||||
}
|
}
|
||||||
|
|
||||||
private decodeAllRecords(data: Uint8Array): WALRecord[] {
|
private decodeAllRecords(data: Uint8Array): { records: WALRecord[]; corrupt: number } {
|
||||||
const records: WALRecord[] = [];
|
const records: WALRecord[] = [];
|
||||||
|
let corrupt = 0;
|
||||||
const view = new DataView(data.buffer, data.byteOffset, data.byteLength);
|
const view = new DataView(data.buffer, data.byteOffset, data.byteLength);
|
||||||
let offset = 0;
|
let offset = 0;
|
||||||
|
|
||||||
@@ -291,7 +461,10 @@ export class WAL {
|
|||||||
const computedLegacy = this.legacyChecksum(recordBytes);
|
const computedLegacy = this.legacyChecksum(recordBytes);
|
||||||
if ((computedNew >>> 0) !== storedCrc && (computedLegacy >>> 0) !== storedCrc) {
|
if ((computedNew >>> 0) !== storedCrc && (computedLegacy >>> 0) !== storedCrc) {
|
||||||
// CRC 不匹配,跳过此损坏记录(长度字段链完整时后续好记录仍可恢复,
|
// CRC 不匹配,跳过此损坏记录(长度字段链完整时后续好记录仍可恢复,
|
||||||
// 行为由 aria-wal-crc 测试锁定)
|
// 行为由 aria-wal-crc 测试锁定)。
|
||||||
|
// v0.8.0(review 修复):跳过必须**被计数**并上报给恢复方 ——
|
||||||
|
// 否则"少了一条已提交写入"在引擎层完全不可观测(静默丢数据)。
|
||||||
|
corrupt++;
|
||||||
// eslint-disable-next-line no-console
|
// eslint-disable-next-line no-console
|
||||||
console.warn(`[AriaEngine WAL] CRC mismatch at record LSN=${lsn}, skipping`);
|
console.warn(`[AriaEngine WAL] CRC mismatch at record LSN=${lsn}, skipping`);
|
||||||
continue;
|
continue;
|
||||||
@@ -311,6 +484,6 @@ export class WAL {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return records;
|
return { records, corrupt };
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -16,7 +16,9 @@ import type { IStorageBackend } from '../store/backend';
|
|||||||
|
|
||||||
/** 分片文件名:__wal_%06d.bin */
|
/** 分片文件名:__wal_%06d.bin */
|
||||||
export const WAL_SEGMENT_PREFIX = '__wal_';
|
export const WAL_SEGMENT_PREFIX = '__wal_';
|
||||||
const SEGMENT_REGEX = /^__wal_(\d{6})\.bin$/;
|
// v0.8.0:分片号只增不减(见 truncateBefore),因此不能假设一定是 6 位 ——
|
||||||
|
// 超过 999999 之后若仍按 {6} 匹配,分片会突然"消失"(静默丢日志)。
|
||||||
|
const SEGMENT_REGEX = /^__wal_(\d{6,})\.bin$/;
|
||||||
const LEGACY_RECORD_REGEX = /^__wal_(\d+)$/;
|
const LEGACY_RECORD_REGEX = /^__wal_(\d+)$/;
|
||||||
const LEGACY_COUNT_KEY = '__wal_count';
|
const LEGACY_COUNT_KEY = '__wal_count';
|
||||||
|
|
||||||
@@ -34,6 +36,33 @@ export interface WALStore {
|
|||||||
exists(): Promise<boolean>;
|
exists(): Promise<boolean>;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** 读取结果:数据 + 空洞诊断(v0.8.0) */
|
||||||
|
export interface WALReadResult {
|
||||||
|
/** 有效前缀数据(自 fromSegment 起、连续分片拼接) */
|
||||||
|
data: Uint8Array;
|
||||||
|
/** 起始分片号 */
|
||||||
|
fromSegment: number;
|
||||||
|
/** 实际读到的分片号(升序) */
|
||||||
|
segments: number[];
|
||||||
|
/**
|
||||||
|
* 活跃区间**内部**的空洞(缺失的分片号)。非空意味着"这之后的记录不可信" ——
|
||||||
|
* v0.8.0 起由调用方(WAL/引擎)**显式上报**,不再静默丢弃尾部
|
||||||
|
* (修复前:序号不连续 → 直接把空洞之后的分片全部丢掉且没有任何提示)。
|
||||||
|
*/
|
||||||
|
gaps: number[];
|
||||||
|
/**
|
||||||
|
* v0.8.0(review 修复):起始分片**之前**缺失的分片号(前缀缺失)。
|
||||||
|
*
|
||||||
|
* 与 `gaps` 分开的原因:前缀缺失可能是正常代价 —— manifest 回退到上一代时,
|
||||||
|
* 它的 `startSegment` 比介质上现存最小分片更小(那段记录已被水位跳过)。
|
||||||
|
* 因此前缀缺失**不丢弃后缀分片**、也不当错误;只有 `fromLsn === 0`
|
||||||
|
*(从未推进过水位 → 分片应当从 fromSegment 连续存在)才把它计入 `gaps`。
|
||||||
|
*/
|
||||||
|
missingPrefix: number[];
|
||||||
|
/** 最后一条记录之后仍缺失的分片数(尾部截断诊断) */
|
||||||
|
missingTail: number;
|
||||||
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// SegmentedWALStore
|
// SegmentedWALStore
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -44,6 +73,14 @@ export class SegmentedWALStore implements WALStore {
|
|||||||
private currentSegment = 0;
|
private currentSegment = 0;
|
||||||
/** 当前分片字节数(内存跟踪,append 切分片判断) */
|
/** 当前分片字节数(内存跟踪,append 切分片判断) */
|
||||||
private currentSize = 0;
|
private currentSize = 0;
|
||||||
|
/**
|
||||||
|
* v0.8.0:分片 → 该分片**首条记录**的 LSN。
|
||||||
|
*
|
||||||
|
* 用途:manifest 提交后需要知道"哪些分片整体已落盘可以删除"。
|
||||||
|
* 记录格式里 LSN 是每条记录的第 1 个字段,因此取分片前 4 字节即可定位;
|
||||||
|
* 无需改变 WAL 二进制格式(旧库分片同样适用)。
|
||||||
|
*/
|
||||||
|
private segmentFirstLsn = new Map<number, number>();
|
||||||
|
|
||||||
constructor(
|
constructor(
|
||||||
private backend: IStorageBackend,
|
private backend: IStorageBackend,
|
||||||
@@ -64,9 +101,18 @@ export class SegmentedWALStore implements WALStore {
|
|||||||
this.currentSegment++;
|
this.currentSegment++;
|
||||||
this.currentSize = 0;
|
this.currentSize = 0;
|
||||||
}
|
}
|
||||||
const key = this.segmentKey(this.currentSegment);
|
const seq = this.currentSegment;
|
||||||
|
const key = this.segmentKey(seq);
|
||||||
const copy = data.buffer.slice(data.byteOffset, data.byteOffset + data.byteLength) as ArrayBuffer;
|
const copy = data.buffer.slice(data.byteOffset, data.byteOffset + data.byteLength) as ArrayBuffer;
|
||||||
|
|
||||||
|
// 记录该分片首条记录的 LSN(分片内记录按写入顺序追加,首条即最小 LSN)
|
||||||
|
if (!this.segmentFirstLsn.has(seq) && data.byteLength >= 4) {
|
||||||
|
this.segmentFirstLsn.set(
|
||||||
|
seq,
|
||||||
|
new DataView(data.buffer, data.byteOffset, 4).getUint32(0, false),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
if (typeof this.backend.append === 'function') {
|
if (typeof this.backend.append === 'function') {
|
||||||
await this.backend.append!(key, copy);
|
await this.backend.append!(key, copy);
|
||||||
} else {
|
} else {
|
||||||
@@ -84,22 +130,60 @@ export class SegmentedWALStore implements WALStore {
|
|||||||
this.currentSize += data.byteLength;
|
this.currentSize += data.byteLength;
|
||||||
}
|
}
|
||||||
|
|
||||||
async readAll(): Promise<Uint8Array> {
|
/** 列出存储上的分片(升序) */
|
||||||
|
private async listSegments(): Promise<{ seq: number; key: string }[]> {
|
||||||
const keys = await this.backend.listKeys();
|
const keys = await this.backend.listKeys();
|
||||||
|
return keys
|
||||||
// ---- 新格式分片 ----
|
|
||||||
const segments = keys
|
|
||||||
.filter((k) => SEGMENT_REGEX.test(k))
|
.filter((k) => SEGMENT_REGEX.test(k))
|
||||||
.map((k) => ({ seq: Number(k.match(SEGMENT_REGEX)![1]), key: k }))
|
.map((k) => ({ seq: Number(k.match(SEGMENT_REGEX)![1]), key: k }))
|
||||||
.sort((a, b) => a.seq - b.seq);
|
.sort((a, b) => a.seq - b.seq);
|
||||||
|
}
|
||||||
|
|
||||||
// 空洞检测:分片序号必须从 0 严格连续,空洞后的分片整体丢弃
|
/**
|
||||||
let keepCount = 0;
|
* v0.8.0:从 `fromSegment` 开始读取,并把空洞**如实返回**(不再静默丢弃尾部)。
|
||||||
for (let i = 0; i < segments.length; i++) {
|
*
|
||||||
if (segments[i].seq !== i) break;
|
* 为什么必须报告空洞:`truncateBefore` 只会删除**前缀**分片,因此活跃区间内
|
||||||
keepCount = i + 1;
|
* 出现空洞只可能来自介质损坏/外部删除 —— 此时空洞之后的记录无法确认是否属于
|
||||||
|
* 同一个连续历史。修复前 `readAll` 遇到空洞直接丢弃空洞之后的全部记录,
|
||||||
|
* 调用方(引擎恢复)完全无法感知"少了一批已提交事务"。
|
||||||
|
*/
|
||||||
|
async readAllFrom(fromSegment: number = 0, fromLsn: number = 0): Promise<WALReadResult> {
|
||||||
|
const keys = await this.backend.listKeys();
|
||||||
|
const allSegments = await this.listSegments();
|
||||||
|
const segments = allSegments.filter((s) => s.seq >= fromSegment);
|
||||||
|
|
||||||
|
// ---- 前缀缺失(fromSegment .. segments[0].seq - 1)----
|
||||||
|
// v0.8.0(review 修复):以前这里把前缀缺失也算作空洞,于是"回退到上一代
|
||||||
|
// manifest"时(上一代的 startSegment 比现存最小分片更小)会命中
|
||||||
|
// `kept = segments.filter(s => s.seq < firstGap)` = 空 —— **整段活 WAL 被丢掉**。
|
||||||
|
// 正确语义:前缀缺失只影响"本就被水位跳过"的那一段,后缀分片必须照常读取。
|
||||||
|
const missingPrefix: number[] = [];
|
||||||
|
if (segments.length > 0) {
|
||||||
|
for (let missing = fromSegment; missing < segments[0].seq; missing++) missingPrefix.push(missing);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- 内部空洞(segments[i-1] 与 segments[i] 之间)----
|
||||||
|
const gaps: number[] = [];
|
||||||
|
for (let i = 1; i < segments.length; i++) {
|
||||||
|
for (let missing = segments[i - 1].seq + 1; missing < segments[i].seq; missing++) {
|
||||||
|
gaps.push(missing);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// 从未推进过水位(fromLsn === 0)时,分片本应从 fromSegment 连续存在 →
|
||||||
|
// 此时前缀缺失同样是异常(介质丢了一段记录),并入 gaps。
|
||||||
|
if (fromLsn === 0 && missingPrefix.length > 0) {
|
||||||
|
gaps.unshift(...missingPrefix);
|
||||||
|
missingPrefix.length = 0;
|
||||||
|
}
|
||||||
|
gaps.sort((a, b) => a - b);
|
||||||
|
|
||||||
|
// 内部空洞之后的分片整体丢弃(长度字段链无法跨空洞验证);
|
||||||
|
// 前缀缺失**不**丢弃任何后缀分片。
|
||||||
|
let kept = segments;
|
||||||
|
if (gaps.length > 0) {
|
||||||
|
const firstGap = gaps[0];
|
||||||
|
kept = segments.filter((s) => s.seq < firstGap);
|
||||||
}
|
}
|
||||||
const validSegments = segments.slice(0, keepCount);
|
|
||||||
|
|
||||||
// ---- 旧格式兼容:__wal_N 单记录键(迁移前数据) ----
|
// ---- 旧格式兼容:__wal_N 单记录键(迁移前数据) ----
|
||||||
const legacyKeys = keys
|
const legacyKeys = keys
|
||||||
@@ -114,15 +198,23 @@ export class SegmentedWALStore implements WALStore {
|
|||||||
if (raw) parts.push(new Uint8Array(raw));
|
if (raw) parts.push(new Uint8Array(raw));
|
||||||
}
|
}
|
||||||
// 新格式分片在后
|
// 新格式分片在后
|
||||||
for (const { key } of validSegments) {
|
const readSegments: number[] = [];
|
||||||
|
for (const { seq, key } of kept) {
|
||||||
const raw = await this.backend.read(key);
|
const raw = await this.backend.read(key);
|
||||||
if (raw) parts.push(new Uint8Array(raw));
|
if (!raw) continue;
|
||||||
|
const bytes = new Uint8Array(raw);
|
||||||
|
if (!this.segmentFirstLsn.has(seq) && bytes.byteLength >= 4) {
|
||||||
|
this.segmentFirstLsn.set(seq, new DataView(bytes.buffer, bytes.byteOffset, 4).getUint32(0, false));
|
||||||
|
}
|
||||||
|
readSegments.push(seq);
|
||||||
|
parts.push(bytes);
|
||||||
}
|
}
|
||||||
|
|
||||||
// 同步当前分片状态(追加定位)
|
// 同步当前分片状态(追加定位)
|
||||||
if (validSegments.length > 0) {
|
if (segments.length > 0) {
|
||||||
this.currentSegment = validSegments[validSegments.length - 1].seq;
|
const lastSeq = segments[segments.length - 1].seq;
|
||||||
const lastRaw = await this.backend.read(validSegments[validSegments.length - 1].key);
|
this.currentSegment = lastSeq;
|
||||||
|
const lastRaw = await this.backend.read(this.segmentKey(lastSeq));
|
||||||
this.currentSize = lastRaw ? lastRaw.byteLength : 0;
|
this.currentSize = lastRaw ? lastRaw.byteLength : 0;
|
||||||
// 旧格式键存在时(迁移中),下一条记录另起分片,避免与旧键序号冲突
|
// 旧格式键存在时(迁移中),下一条记录另起分片,避免与旧键序号冲突
|
||||||
if (legacyKeys.length > 0) {
|
if (legacyKeys.length > 0) {
|
||||||
@@ -133,13 +225,29 @@ export class SegmentedWALStore implements WALStore {
|
|||||||
// 仅有旧格式:迁移中,新写入从分片 0 开始(checkpoint 会清空旧键)
|
// 仅有旧格式:迁移中,新写入从分片 0 开始(checkpoint 会清空旧键)
|
||||||
this.currentSegment = 0;
|
this.currentSegment = 0;
|
||||||
this.currentSize = 0;
|
this.currentSize = 0;
|
||||||
|
} else if (fromSegment > 0) {
|
||||||
|
// 活跃区间的分片已被清理(例如上一次 truncateBefore 的删除生效):
|
||||||
|
// 新写入从当前分片号继续,避免复用已删除的历史分片号
|
||||||
|
this.currentSegment = fromSegment;
|
||||||
|
this.currentSize = 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
const total = parts.reduce((s, c) => s + c.byteLength, 0);
|
const total = parts.reduce((s, c) => s + c.byteLength, 0);
|
||||||
const combined = new Uint8Array(total);
|
const combined = new Uint8Array(total);
|
||||||
let off = 0;
|
let off = 0;
|
||||||
for (const c of parts) { combined.set(c, off); off += c.byteLength; }
|
for (const c of parts) { combined.set(c, off); off += c.byteLength; }
|
||||||
return combined;
|
return {
|
||||||
|
data: combined,
|
||||||
|
fromSegment,
|
||||||
|
segments: readSegments,
|
||||||
|
gaps,
|
||||||
|
missingPrefix,
|
||||||
|
missingTail: 0,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async readAll(): Promise<Uint8Array> {
|
||||||
|
return (await this.readAllFrom(0)).data;
|
||||||
}
|
}
|
||||||
|
|
||||||
async truncate(): Promise<void> {
|
async truncate(): Promise<void> {
|
||||||
@@ -149,8 +257,151 @@ export class SegmentedWALStore implements WALStore {
|
|||||||
if (walKeys.length > 0) {
|
if (walKeys.length > 0) {
|
||||||
await this.backend.deleteMany(walKeys);
|
await this.backend.deleteMany(walKeys);
|
||||||
}
|
}
|
||||||
|
// 分片号**只降不增**是禁止的(同 truncateBefore 的说明)。这里的准确语义是:
|
||||||
|
// 序号绝不回退,且绝不小于 manifest 记录的水位下限(`open(fromSegment)` 会把它
|
||||||
|
// 抬到 fromSegment);整体清空后允许复用**最后用过的那个号** —— 这是安全的,
|
||||||
|
// 因为记录自带 LSN,`lsn <= startLsn` 的旧世代记录在恢复时一律跳过。
|
||||||
|
//
|
||||||
|
// 反例(修复前的真实缺陷):整段清空后把号重置为 0,而 manifest 的
|
||||||
|
// startSegment 已经是 K>0 → 新记录写进分片 0,恢复时被 `seq >= K` 过滤掉 →
|
||||||
|
// 已确认写入静默消失。因此这里绝不允许回退到 0。
|
||||||
|
const maxSeq = (await this.listSegments()).reduce((max, s) => Math.max(max, s.seq), -1);
|
||||||
|
this.currentSegment = Math.max(this.currentSegment, maxSeq + 1);
|
||||||
|
this.currentSize = 0;
|
||||||
|
this.segmentFirstLsn.clear();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0:**整库清空**后重置分片编号(只有 `clearAll` 这种"介质被整体抹掉、
|
||||||
|
* manifest 也重新从 0 开始"的场景才允许调用)。
|
||||||
|
*/
|
||||||
|
reset(): void {
|
||||||
this.currentSegment = 0;
|
this.currentSegment = 0;
|
||||||
this.currentSize = 0;
|
this.currentSize = 0;
|
||||||
|
this.segmentFirstLsn.clear();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0:删除**整体 LSN 都 <= durableLsn** 的分片(前缀删除)。
|
||||||
|
*
|
||||||
|
* 语义约束(调用方必须保证):`durableLsn` 之前的记录已存在于已提交的
|
||||||
|
* manifest/SSTable 中。返回仍需保留的最小分片号 —— 调用方应把它写进
|
||||||
|
* manifest 的 `wal.startSegment`,这样即便删除只完成了一半(崩溃/介质错误),
|
||||||
|
* 恢复也会忽略那些残留的旧世代分片。
|
||||||
|
*
|
||||||
|
* @param durableLsn 落盘水位(lsn <= 它的记录已确认存在于 SSTable 中)
|
||||||
|
* @param latestLsn 当前 LSN 高水位(调用方 flush 之后的 `wal.getLsn()`):
|
||||||
|
* 用于判断**最后一个分片**是否也已被水位完整覆盖。
|
||||||
|
* WAL 记录里只有"每条记录的 LSN",分片的**末条** LSN 无法
|
||||||
|
* 从分片首字节推出,而"最后一个分片之后没有分片"这一点只有
|
||||||
|
* 调用方知道 —— 缺了它就会出现"分片永远删不掉"(实测:
|
||||||
|
* close 之后 `__wal_000000.bin` 仍在,每次打开都要重读一遍)。
|
||||||
|
*/
|
||||||
|
/**
|
||||||
|
* v0.8.0:**只计算**"还需要保留的最小分片号"(无副作用)。
|
||||||
|
*
|
||||||
|
* 为什么需要"先算后删":调用方必须**先把这个值提交进 manifest**,再删除分片。
|
||||||
|
* 顺序反了(先删后记录)会留下一个窗口:崩溃后 manifest 里的 `startSegment`
|
||||||
|
* 比介质上实际存在的分片更小,恢复时看到"前缀缺失"就无法区分
|
||||||
|
* "正常清理过的前缀"与"介质丢了一段记录"—— 前者无害,后者是数据丢失。
|
||||||
|
*/
|
||||||
|
async planKeepFrom(durableLsn: number, latestLsn?: number): Promise<number> {
|
||||||
|
const segments = await this.listSegments();
|
||||||
|
// v0.8.0(review 修复):介质上没有分片时**返回当前分片号**(= 下一条记录将写入
|
||||||
|
// 的号),而不是 0。返回 0 会让 manifest 的 `startSegment` 回退,与
|
||||||
|
// "分片号只增不减" 的声明冲突(一旦将来前缀缺失被当作异常,就会误判成丢数据)。
|
||||||
|
if (segments.length === 0) return this.currentSegment;
|
||||||
|
|
||||||
|
// 判定每个分片是否"整段已被水位覆盖":
|
||||||
|
// - 非末分片:下一分片的首条记录 LSN <= durableLsn ⇒ 本分片全部记录都 <= 水位
|
||||||
|
// - 末分片:已知当前 LSN 高水位 <= durableLsn ⇒ 本分片全部记录都 <= 水位
|
||||||
|
for (let i = 0; i < segments.length; i++) {
|
||||||
|
const seg = segments[i];
|
||||||
|
const isLast = i === segments.length - 1;
|
||||||
|
const firstLsn = this.segmentFirstLsn.get(seg.seq);
|
||||||
|
// 未知边界:保守地从这里开始保留
|
||||||
|
if (firstLsn === undefined) return seg.seq;
|
||||||
|
const fullyCovered = isLast
|
||||||
|
? (typeof latestLsn === 'number' && latestLsn > 0 && latestLsn <= durableLsn)
|
||||||
|
: (() => {
|
||||||
|
const nextFirst = this.segmentFirstLsn.get(segments[i + 1].seq);
|
||||||
|
return nextFirst !== undefined && nextFirst <= durableLsn;
|
||||||
|
})();
|
||||||
|
if (!fullyCovered) return seg.seq;
|
||||||
|
}
|
||||||
|
// 全部分片都被水位覆盖
|
||||||
|
return segments[segments.length - 1].seq + 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
async truncateBefore(durableLsn: number, latestLsn?: number): Promise<number> {
|
||||||
|
// v0.8.0:旧格式(__wal_N 每条一个 key)同样按水位清理 ——
|
||||||
|
// 它们是"迁移前"的记录,与分片一样只在 lsn <= durableLsn 时才可删。
|
||||||
|
// 修复前这些键只在整个 WAL 被 truncate() 时才清,导致已落盘的旧记录
|
||||||
|
// 永远留在介质上(每次打开都会被 readAll 读出来再按 lsn 跳过)。
|
||||||
|
//
|
||||||
|
// 注意:必须在"没有新格式分片"的早退**之前**处理 —— 只含旧格式的库
|
||||||
|
//(v0.4.4 升级现场)恰恰是这条路径最常见的输入。
|
||||||
|
const legacyKeys = (await this.backend.listKeys()).filter((k) => LEGACY_RECORD_REGEX.test(k));
|
||||||
|
if (legacyKeys.length > 0) {
|
||||||
|
const obsoleteLegacy: string[] = [];
|
||||||
|
for (const key of legacyKeys) {
|
||||||
|
const raw = await this.backend.read(key);
|
||||||
|
if (!raw || raw.byteLength < 4) {
|
||||||
|
obsoleteLegacy.push(key); // 空/残缺:无记录可保留
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const firstLsn = new DataView(raw, 0, 4).getUint32(0, false);
|
||||||
|
if (firstLsn <= durableLsn) obsoleteLegacy.push(key);
|
||||||
|
}
|
||||||
|
if (obsoleteLegacy.length > 0) {
|
||||||
|
try {
|
||||||
|
await this.backend.deleteMany([...obsoleteLegacy, LEGACY_COUNT_KEY]);
|
||||||
|
} catch { /* 同上:残留无害(恢复按 lsn 跳过) */ }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const segments = await this.listSegments();
|
||||||
|
if (segments.length === 0) {
|
||||||
|
// 无分片可删:分片号保持不变(只增不减),返回下一条记录将写入的号
|
||||||
|
this.currentSize = 0;
|
||||||
|
return this.currentSegment;
|
||||||
|
}
|
||||||
|
|
||||||
|
const keepFrom = await this.planKeepFrom(durableLsn, latestLsn);
|
||||||
|
const obsolete = segments.filter((s) => s.seq < keepFrom);
|
||||||
|
if (obsolete.length > 0) {
|
||||||
|
try {
|
||||||
|
await this.backend.deleteMany(obsolete.map((s) => s.key));
|
||||||
|
} catch { /* 删除失败:残留分片由 manifest.startSegment 在恢复时忽略 */ }
|
||||||
|
for (const s of obsolete) this.segmentFirstLsn.delete(s.seq);
|
||||||
|
}
|
||||||
|
|
||||||
|
// 全部删除(含当前分片)→ **分片号继续往后走,绝不重置回 0**。
|
||||||
|
//
|
||||||
|
// 这是 v0.8.0 的一处关键修正:修复前这里是 `currentSegment = 0`。而 manifest
|
||||||
|
// 里的 `startSegment` 是**删除前**就算好并提交的(= 最大分片号 + 1),于是
|
||||||
|
// 重置之后新记录写进分片 0,恢复时从 startSegment 开始读 → 整批新记录被跳过
|
||||||
|
// (实测随机压力用例:删掉的行复活 / 已确认写入丢失,两种方向都出现过)。
|
||||||
|
// 更糟的是"旧世代排在最新记录之后":分片号复用会让同一序号对应两代记录,
|
||||||
|
// 恢复按序号排序时旧 INSERT 会排在一次 DELETE 之后被重放。
|
||||||
|
//
|
||||||
|
// 现在分片号只增不减:删除只回收空间,不复用标识。
|
||||||
|
if (obsolete.length === segments.length) {
|
||||||
|
const maxSeq = segments[segments.length - 1].seq;
|
||||||
|
const leftover = await this.listSegments();
|
||||||
|
if (leftover.length === 0) {
|
||||||
|
this.currentSegment = Math.max(maxSeq + 1, keepFrom);
|
||||||
|
this.currentSize = 0;
|
||||||
|
this.segmentFirstLsn.clear();
|
||||||
|
return keepFrom;
|
||||||
|
}
|
||||||
|
// 删除失败留下残片:它们都在 startSegment 之前(恢复时被忽略),
|
||||||
|
// 新记录必须继续用**更大的**分片号,避免与残片混进同一个文件。
|
||||||
|
this.currentSegment = Math.max(leftover[leftover.length - 1].seq + 1, keepFrom);
|
||||||
|
this.currentSize = 0;
|
||||||
|
return keepFrom;
|
||||||
|
}
|
||||||
|
return keepFrom;
|
||||||
}
|
}
|
||||||
|
|
||||||
async exists(): Promise<boolean> {
|
async exists(): Promise<boolean> {
|
||||||
|
|||||||
@@ -0,0 +1,352 @@
|
|||||||
|
/**
|
||||||
|
* metona-sqlark Engine Change Notifier(v0.8.0)
|
||||||
|
* @module engine/change-notifier
|
||||||
|
*
|
||||||
|
* ============================================================================
|
||||||
|
* 为什么需要它(v0.8.0 审计 A9)
|
||||||
|
* ============================================================================
|
||||||
|
* `db.subscribe(table, fn)` 此前对**本地写入永不触发**:全库只有一处调用 `emit`,
|
||||||
|
* 而那一处在 BroadcastChannel 收到**其它标签页**消息时才执行。于是:
|
||||||
|
* - README:232「订阅表变更」、site/docs.html:667-679 的示例
|
||||||
|
* (`event.type: 'insert' | 'update' | 'delete'`、`event.row`)全部不成立;
|
||||||
|
* - 唯一的实际触发条件是 `multiTabSync: true` 且收到 `external` 事件。
|
||||||
|
*
|
||||||
|
* 更麻烦的是原因:写入路径有**三条**(SQL 语句、Table API、QueryBuilder),各自
|
||||||
|
* 只调用 `broadcastChange(tableName)`,谁都不知道"改了哪些行"。所以此前即便想接线,
|
||||||
|
* 也要在三处分别实现一遍变更描述逻辑 —— 那正是本项目反复出问题的模式
|
||||||
|
* (同一语义多份实现 → 漂移)。
|
||||||
|
*
|
||||||
|
* ============================================================================
|
||||||
|
* 本实现的做法
|
||||||
|
* ============================================================================
|
||||||
|
* 用一个 `IStorageEngine` **装饰器**把变更通知收敛到唯一位置:所有写入都必须经过
|
||||||
|
* 引擎接口,因此在这里拦截一次即可覆盖全部入口(SQL / Table / Builder / 事务内)。
|
||||||
|
*
|
||||||
|
* 事件语义(对文档承诺的兑现):
|
||||||
|
* - INSERT:逐行一个事件,携带该行(`row` 与主键 `key`);
|
||||||
|
* - UPDATE:先按 WHERE 查出**将被修改的行**(快照),写入成功后逐行发事件并
|
||||||
|
* 携带**更新后的行**;查不到(例如 WHERE 无法在写入前求值)时退化为表级事件,
|
||||||
|
* 只带 `count`;
|
||||||
|
* - DELETE:同理,逐行发事件并携带**被删除前的行**;
|
||||||
|
* - CLEAR / DDL:表级事件(`count` 可选)。
|
||||||
|
*
|
||||||
|
* 订阅方回调返回 Promise 时会被 await(保证"写完通知完"的顺序),
|
||||||
|
* 但**订阅方的异常不会影响写入结果** —— 写入已经成功,通知失败只上报 onError。
|
||||||
|
*/
|
||||||
|
|
||||||
|
import type { IStorageEngine } from './interface';
|
||||||
|
import { DatabaseError } from '../constants';
|
||||||
|
import type { QueryPlan } from '../constants';
|
||||||
|
|
||||||
|
/** 变更类型(与 site/docs.html 承诺的 `event.type` 对齐) */
|
||||||
|
export type ChangeType = 'insert' | 'update' | 'delete' | 'clear' | 'ddl' | 'external';
|
||||||
|
|
||||||
|
/** 变更事件 */
|
||||||
|
export interface ChangeEvent {
|
||||||
|
/** 变更类型 */
|
||||||
|
type: ChangeType;
|
||||||
|
/** 表名 */
|
||||||
|
table: string;
|
||||||
|
/** 受影响的行(可用时提供) */
|
||||||
|
row?: Record<string, unknown>;
|
||||||
|
/** 受影响行的主键(可用时提供) */
|
||||||
|
key?: string;
|
||||||
|
/** 受影响行数 */
|
||||||
|
count?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 订阅者回调(可返回 Promise,通知方会 await) */
|
||||||
|
export type ChangeListener = (event: ChangeEvent) => void | Promise<void>;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 变更通知引擎装饰器。
|
||||||
|
*
|
||||||
|
* 只读方法(find/count/findStream/getTableSchema/...)直接透传;
|
||||||
|
* 写入方法在成功后产生事件并同步派发给监听者。
|
||||||
|
*/
|
||||||
|
export class ChangeNotifierEngine implements IStorageEngine {
|
||||||
|
private readonly inner: IStorageEngine;
|
||||||
|
private readonly listeners = new Set<ChangeListener>();
|
||||||
|
private readonly onListenerError: (error: Error) => void;
|
||||||
|
/** 跨标签页广播回调(由 core 注入,避免本模块依赖 BroadcastChannel) */
|
||||||
|
private readonly onBroadcast: (table: string) => void;
|
||||||
|
|
||||||
|
constructor(
|
||||||
|
inner: IStorageEngine,
|
||||||
|
onListenerError: (error: Error) => void = () => {},
|
||||||
|
onBroadcast: (table: string) => void = () => {},
|
||||||
|
) {
|
||||||
|
this.inner = inner;
|
||||||
|
this.onListenerError = onListenerError;
|
||||||
|
this.onBroadcast = onBroadcast;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 暴露内层引擎(需要能力探测或特化逻辑时使用) */
|
||||||
|
getInner(): IStorageEngine {
|
||||||
|
return this.inner;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 注册变更监听(返回退订函数) */
|
||||||
|
addListener(listener: ChangeListener): () => void {
|
||||||
|
this.listeners.add(listener);
|
||||||
|
return () => this.listeners.delete(listener);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 派发事件(供外部事件如跨标签页 `external` 复用同一通道) */
|
||||||
|
async dispatch(event: ChangeEvent): Promise<void> {
|
||||||
|
// 跨标签页广播与本地订阅走同一出口,保证"写入即通知"只有一条路径
|
||||||
|
try {
|
||||||
|
this.onBroadcast(event.table);
|
||||||
|
} catch {
|
||||||
|
// 广播失败不影响本地通知与写入结果
|
||||||
|
}
|
||||||
|
for (const listener of [...this.listeners]) {
|
||||||
|
try {
|
||||||
|
await listener(event);
|
||||||
|
} catch (error) {
|
||||||
|
// 订阅者异常不得影响写入结果(写入已成功),只上报
|
||||||
|
this.onListenerError(error as Error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
// 写入路径:产生事件
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
async insert(tableName: string, rows: Record<string, unknown>[]): Promise<string[]> {
|
||||||
|
const pks = await this.inner.insert(tableName, rows);
|
||||||
|
// 逐行事件;用引擎返回的主键对齐(resolved 行带 default 值,恢复时可对照)
|
||||||
|
for (let i = 0; i < pks.length; i++) {
|
||||||
|
await this.dispatch({
|
||||||
|
type: 'insert',
|
||||||
|
table: tableName,
|
||||||
|
key: pks[i],
|
||||||
|
row: i < rows.length ? rows[i] : undefined,
|
||||||
|
count: 1,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return pks;
|
||||||
|
}
|
||||||
|
|
||||||
|
async update(tableName: string, query: QueryPlan, updates: Record<string, unknown>): Promise<number> {
|
||||||
|
// 写入前快照将被影响的行(用于提供 row 内容)。失败不阻塞写入 —— 快照只是尽力而为。
|
||||||
|
const affected = await this.snapshotAffected(tableName, query);
|
||||||
|
const count = await this.inner.update(tableName, query, updates);
|
||||||
|
if (count > 0 && affected.length > 0) {
|
||||||
|
for (const before of affected) {
|
||||||
|
const after = { ...before, ...stripUndefined(updates) };
|
||||||
|
await this.dispatch({
|
||||||
|
type: 'update',
|
||||||
|
table: tableName,
|
||||||
|
row: after,
|
||||||
|
count: 1,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
} else if (count > 0) {
|
||||||
|
await this.dispatch({ type: 'update', table: tableName, count });
|
||||||
|
}
|
||||||
|
return count;
|
||||||
|
}
|
||||||
|
|
||||||
|
async delete(tableName: string, query: QueryPlan): Promise<number> {
|
||||||
|
const affected = await this.snapshotAffected(tableName, query);
|
||||||
|
const count = await this.inner.delete(tableName, query);
|
||||||
|
if (count > 0 && affected.length > 0) {
|
||||||
|
for (const before of affected) {
|
||||||
|
await this.dispatch({ type: 'delete', table: tableName, row: before, count: 1 });
|
||||||
|
}
|
||||||
|
} else if (count > 0) {
|
||||||
|
await this.dispatch({ type: 'delete', table: tableName, count });
|
||||||
|
}
|
||||||
|
return count;
|
||||||
|
}
|
||||||
|
|
||||||
|
async clear(tableName: string): Promise<void> {
|
||||||
|
await this.inner.clear(tableName);
|
||||||
|
await this.dispatch({ type: 'clear', table: tableName });
|
||||||
|
}
|
||||||
|
|
||||||
|
async createTable(schema: import('../constants').TableSchema): Promise<void> {
|
||||||
|
await this.inner.createTable(schema);
|
||||||
|
await this.dispatch({ type: 'ddl', table: schema.name });
|
||||||
|
}
|
||||||
|
|
||||||
|
async dropTable(tableName: string): Promise<void> {
|
||||||
|
await this.inner.dropTable(tableName);
|
||||||
|
await this.dispatch({ type: 'ddl', table: tableName });
|
||||||
|
}
|
||||||
|
|
||||||
|
async alterTable(
|
||||||
|
tableName: string,
|
||||||
|
action: 'ADD' | 'DROP',
|
||||||
|
column: import('../constants').ColumnDef & { name: string },
|
||||||
|
): Promise<void> {
|
||||||
|
await this.requireCapability('alterTable')(tableName, action, column);
|
||||||
|
await this.dispatch({ type: 'ddl', table: tableName });
|
||||||
|
}
|
||||||
|
|
||||||
|
async createIndex(tableName: string, column: string, unique?: boolean): Promise<void> {
|
||||||
|
await this.requireCapability('createIndex')(tableName, column, unique);
|
||||||
|
await this.dispatch({ type: 'ddl', table: tableName });
|
||||||
|
}
|
||||||
|
|
||||||
|
async dropIndex(tableName: string, column: string, indexName?: string): Promise<void> {
|
||||||
|
await this.requireCapability('dropIndex')(tableName, column, indexName);
|
||||||
|
await this.dispatch({ type: 'ddl', table: tableName });
|
||||||
|
}
|
||||||
|
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
// 只读路径:透传
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
get name(): string { return this.inner.name; }
|
||||||
|
open(dbName: string, version: number): Promise<void> { return this.inner.open(dbName, version); }
|
||||||
|
close(): Promise<void> { return this.inner.close(); }
|
||||||
|
isOpen(): boolean { return this.inner.isOpen(); }
|
||||||
|
hasTable(tableName: string): Promise<boolean> { return this.inner.hasTable(tableName); }
|
||||||
|
getTableNames(): Promise<string[]> { return this.inner.getTableNames(); }
|
||||||
|
getTableSchema(tableName: string): Promise<import('../constants').TableSchema | null> {
|
||||||
|
return this.inner.getTableSchema(tableName);
|
||||||
|
}
|
||||||
|
find(tableName: string, query: QueryPlan): Promise<Record<string, unknown>[]> {
|
||||||
|
return this.inner.find(tableName, query);
|
||||||
|
}
|
||||||
|
count(tableName: string, query?: QueryPlan): Promise<number> {
|
||||||
|
return this.inner.count(tableName, query);
|
||||||
|
}
|
||||||
|
async findStream(
|
||||||
|
tableName: string,
|
||||||
|
query: QueryPlan,
|
||||||
|
onRow: (row: Record<string, unknown>) => void,
|
||||||
|
): Promise<number> {
|
||||||
|
return this.requireCapability('findStream')(tableName, query, onRow);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- 事务:透传(事务内的写入由底层引擎统一处理,事件在语句层产生) ----
|
||||||
|
|
||||||
|
beginTransaction(): Promise<void> { return this.inner.beginTransaction(); }
|
||||||
|
commitTransaction(): Promise<void> { return this.inner.commitTransaction(); }
|
||||||
|
rollbackTransaction(): Promise<void> { return this.inner.rollbackTransaction(); }
|
||||||
|
|
||||||
|
savepoint(name: string): Promise<void> {
|
||||||
|
return this.requireCapability('savepoint')(name);
|
||||||
|
}
|
||||||
|
rollbackToSavepoint(name: string): Promise<void> {
|
||||||
|
return this.requireCapability('rollbackToSavepoint')(name);
|
||||||
|
}
|
||||||
|
releaseSavepoint(name: string): Promise<void> {
|
||||||
|
return this.requireCapability('releaseSavepoint')(name);
|
||||||
|
}
|
||||||
|
backup(): Promise<Record<string, Record<string, unknown>[]>> {
|
||||||
|
return this.requireCapability('backup')();
|
||||||
|
}
|
||||||
|
repair(): Promise<void> {
|
||||||
|
return this.requireCapability('repair')();
|
||||||
|
}
|
||||||
|
clearAll(): Promise<void> {
|
||||||
|
return this.requireCapability('clearAll')();
|
||||||
|
}
|
||||||
|
// ---- 维护能力(可选,AriaEngine 专有;只读/维护语义,不产生变更事件) ----
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0:转发引擎专有的维护方法。
|
||||||
|
*
|
||||||
|
* 为什么必须显式转发:executor 用 `typeof engine.analyzeTable === 'function'`
|
||||||
|
* 做能力探测。装饰器只实现 `IStorageEngine` 声明的成员,于是这些"接口外方法"
|
||||||
|
* 在被包装后会**静默消失** → ANALYZE/REINDEX 报 NOT_SUPPORTED(实测 3 个用例失败)。
|
||||||
|
* 这正是审计指出的"用 typeof/instanceof 做能力探测"的脆弱之处;
|
||||||
|
* 在装饰器里显式补齐是当前最直接的修法。
|
||||||
|
*/
|
||||||
|
analyzeTable(tableName: string): Promise<Record<string, unknown>> {
|
||||||
|
return this.requireOptionalMethod<[string], Record<string, unknown>>('analyzeTable')(tableName);
|
||||||
|
}
|
||||||
|
|
||||||
|
reindexTable(tableName: string): Promise<number> {
|
||||||
|
return this.requireOptionalMethod<[string], number>('reindexTable')(tableName);
|
||||||
|
}
|
||||||
|
|
||||||
|
vacuum(): Promise<{ compactedLevels: number; gcVersions: number }> {
|
||||||
|
return this.requireOptionalMethod<[], { compactedLevels: number; gcVersions: number }>('vacuum')();
|
||||||
|
}
|
||||||
|
|
||||||
|
getMeta(key: string): Promise<string | null> {
|
||||||
|
const method = this.inner.getMeta;
|
||||||
|
if (typeof method !== 'function') return Promise.resolve(null);
|
||||||
|
return method.call(this.inner, key);
|
||||||
|
}
|
||||||
|
setMeta(key: string, value: string): Promise<void> {
|
||||||
|
const method = this.inner.setMeta;
|
||||||
|
if (typeof method !== 'function') return Promise.resolve();
|
||||||
|
return method.call(this.inner, key, value);
|
||||||
|
}
|
||||||
|
|
||||||
|
// -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0: 取一个**可选能力**方法;内层未实现时抛 `NOT_SUPPORTED`。
|
||||||
|
*
|
||||||
|
* 为什么不能直接抛普通 Error:项目对"能力缺失"有明确契约 ——
|
||||||
|
* `NOT_SUPPORTED` 错误码(README「维护语句」、executor 的 savepoint/backup
|
||||||
|
* 分支、以及既有测试都依赖它)。装饰器如果抛原生 Error,调用方按 code 分类
|
||||||
|
* 就会失效(实测让 6 个套件里的 14 个用例失败)。
|
||||||
|
*/
|
||||||
|
private requireCapability<K extends keyof IStorageEngine>(name: K): NonNullable<IStorageEngine[K]> {
|
||||||
|
const method = this.inner[name];
|
||||||
|
if (typeof method !== 'function') {
|
||||||
|
return (() => {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Engine "${this.inner.name}" does not support ${String(name)}`,
|
||||||
|
'NOT_SUPPORTED',
|
||||||
|
);
|
||||||
|
}) as unknown as NonNullable<IStorageEngine[K]>;
|
||||||
|
}
|
||||||
|
return (method as (...args: unknown[]) => unknown).bind(this.inner) as NonNullable<IStorageEngine[K]>;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0: 取**接口之外**的可选方法(如 AriaEngine.analyzeTable / reindexTable / vacuum)。
|
||||||
|
*
|
||||||
|
* 与 requireCapability 的区别:这些方法不在 IStorageEngine 契约里,属于引擎专有能力,
|
||||||
|
* 但 executor 会用 `typeof engine.x === 'function'` 探测它们。装饰器必须显式转发,
|
||||||
|
* 否则能力在被包装后静默消失(实测 ANALYZE/REINDEX 报 NOT_SUPPORTED)。
|
||||||
|
*/
|
||||||
|
private requireOptionalMethod<A extends unknown[], R>(name: string): (...args: A) => Promise<R> {
|
||||||
|
const method = (this.inner as unknown as Record<string, unknown>)[name];
|
||||||
|
if (typeof method !== 'function') {
|
||||||
|
return () => {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Engine "${this.inner.name}" does not support ${name}`,
|
||||||
|
'NOT_SUPPORTED',
|
||||||
|
);
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return (method as (...args: A) => Promise<R>).bind(this.inner);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 尽力而为地取"将被影响的行"快照。
|
||||||
|
*
|
||||||
|
* 用途:让 UPDATE/DELETE 事件能携带行内容(文档承诺 `event.row`)。
|
||||||
|
* 失败时返回空数组 —— 事件退化为表级通知(只带 count),**不影响写入本身**。
|
||||||
|
*/
|
||||||
|
private async snapshotAffected(
|
||||||
|
tableName: string,
|
||||||
|
query: QueryPlan,
|
||||||
|
): Promise<Record<string, unknown>[]> {
|
||||||
|
try {
|
||||||
|
return await this.inner.find(tableName, { table: tableName, where: query.where });
|
||||||
|
} catch {
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 与引擎/executor 一致的 undefined 语义:不更新该列 */
|
||||||
|
function stripUndefined(updates: Record<string, unknown>): Record<string, unknown> {
|
||||||
|
const out: Record<string, unknown> = {};
|
||||||
|
for (const [k, v] of Object.entries(updates)) {
|
||||||
|
if (v !== undefined) out[k] = v;
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
@@ -8,3 +8,9 @@ export { MemoryEngine } from './memory';
|
|||||||
export { KVStoreEngine } from './kvstore_engine';
|
export { KVStoreEngine } from './kvstore_engine';
|
||||||
export { AriaEngine } from './aria/index';
|
export { AriaEngine } from './aria/index';
|
||||||
export type { AriaEngineConfig } from './aria/types';
|
export type { AriaEngineConfig } from './aria/types';
|
||||||
|
// v0.8.0: 补齐后端与 KVStore 的桶导出(此前只能深路径 import)
|
||||||
|
export { KVStore } from './kvstore/index';
|
||||||
|
export { SharedMemoryBackend } from './kvstore/shared_memory_medium';
|
||||||
|
export { OPFSBackend } from './aria/store/opfs_backend';
|
||||||
|
export { MemoryBackend } from './aria/store/backend';
|
||||||
|
export type { IStorageBackend } from './aria/store/backend';
|
||||||
|
|||||||
+44
-1
@@ -5,6 +5,17 @@
|
|||||||
|
|
||||||
import type { QueryPlan, TableSchema } from '../constants';
|
import type { QueryPlan, TableSchema } from '../constants';
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// v0.8.0: 行所有权(row ownership)约定
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
// 行所有权工具(cloneRow / cloneRows)见 `./row_clone` —— v0.8.0 审查修复:
|
||||||
|
// 它们是运行时实现,必须与被描述为"纯类型"的本文件分开,否则会逃过覆盖率统计。
|
||||||
|
// 类型层面的约定(所有引擎必须遵守):
|
||||||
|
// **读出的行是副本,写入接收的行也是副本** —— 引擎不得把内部行对象暴露给
|
||||||
|
// 外部,也不得持有调用方传入的行对象引用。违反者会让调用方一次原地修改就
|
||||||
|
// 改写存储、造成索引与行失配(审计实测:改过的行再也查不出来)。
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// IStorageEngine — 所有存储引擎必须实现的接口
|
// IStorageEngine — 所有存储引擎必须实现的接口
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -40,6 +51,32 @@ export interface IStorageEngine {
|
|||||||
/** 插入行,返回主键值列表 */
|
/** 插入行,返回主键值列表 */
|
||||||
insert(tableName: string, rows: Record<string, unknown>[]): Promise<string[]>;
|
insert(tableName: string, rows: Record<string, unknown>[]): Promise<string[]>;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0(B-1):写入前置校验 —— 未知列、类型、`maxLength` / `min` / `max`、
|
||||||
|
* `required` / 主键非空、`__proto__` 防污染,全部由 `table/validation.ts` 的
|
||||||
|
* **唯一**实现判定。
|
||||||
|
*
|
||||||
|
* 为什么把它放进引擎接口而不是留在 Executor:
|
||||||
|
* - **未知列**必须在引擎边界拦下。`INSERT INTO t (id, nope) VALUES ('1', 2)`
|
||||||
|
* 若在 Executor 拦,QueryBuilder / `db.table().insert()` 等直通路径仍然静默
|
||||||
|
* 丢列(缺陷 A17 的真实形态:executor 按 schema 列序构造行,那个位置没有值,
|
||||||
|
* 于是 `nope` 既不进 schema 也无从校验)。
|
||||||
|
* - **规范化必须在同一处**:`default` 填充与类型检查一旦分家,就会出现
|
||||||
|
* "executor 校验通过、引擎写入时又被改写"这类双份语义。
|
||||||
|
*
|
||||||
|
* 引擎**必须**使用 `compileValidator` 而不是自己实现 —— 此前 Memory 与 Aria
|
||||||
|
* 各写一份,`maxLength` 只在 Aria 生效(A12)。
|
||||||
|
*
|
||||||
|
* @param mode `'insert'`(default 生效、缺列合法)或 `'update'`(仅校验出现的列)。
|
||||||
|
* 默认 `'insert'`,与历史行为一致。
|
||||||
|
* @throws DatabaseError COLUMN_NOT_FOUND | VALIDATION_ERROR | TYPE_ERROR
|
||||||
|
*/
|
||||||
|
validatePayload?(
|
||||||
|
tableName: string,
|
||||||
|
rows: Record<string, unknown>[],
|
||||||
|
mode?: 'insert' | 'update',
|
||||||
|
): Promise<void>;
|
||||||
|
|
||||||
/** 查询行 */
|
/** 查询行 */
|
||||||
find(tableName: string, query: QueryPlan): Promise<Record<string, unknown>[]>;
|
find(tableName: string, query: QueryPlan): Promise<Record<string, unknown>[]>;
|
||||||
|
|
||||||
@@ -93,7 +130,13 @@ export interface IStorageEngine {
|
|||||||
|
|
||||||
// ---- 备份 (可选) ----
|
// ---- 备份 (可选) ----
|
||||||
|
|
||||||
/** 在线备份:导出全库一致性快照 */
|
/**
|
||||||
|
* 在线备份:导出全库数据快照。
|
||||||
|
*
|
||||||
|
* v0.8.0(review 修正文档):返回的是**调用时刻逐表读取的结果**,不构成
|
||||||
|
* "跨表一致性快照" —— 实现没有冻结全局版本(读取期间不会有并发写入时结果
|
||||||
|
* 是一致的,但并发写入下不同表可能落在不同时间点)。
|
||||||
|
*/
|
||||||
backup?(): Promise<Record<string, Record<string, unknown>[]>>;
|
backup?(): Promise<Record<string, Record<string, unknown>[]>>;
|
||||||
|
|
||||||
// ---- 自愈/重置 (可选,v0.4.2-fix) ----
|
// ---- 自愈/重置 (可选,v0.4.2-fix) ----
|
||||||
|
|||||||
+173
-10
@@ -39,6 +39,20 @@ const DEFAULT_CHECKPOINT_THRESHOLD = 16 * 1024 * 1024;
|
|||||||
interface KVStoreMeta {
|
interface KVStoreMeta {
|
||||||
/** 当前日志水位(快照内嵌;无快照时 0) */
|
/** 当前日志水位(快照内嵌;无快照时 0) */
|
||||||
seq: number;
|
seq: number;
|
||||||
|
/**
|
||||||
|
* v0.8.0(B-6):最后一次提交(checkpoint)的实例 id。
|
||||||
|
*
|
||||||
|
* 为什么需要:多个 KVStore 实例可能同时打开同一个库(多标签页、或测试里
|
||||||
|
* 共享介质)。每个实例各有自己的内存索引与 `seq`,而 snapshot/meta 是
|
||||||
|
* **全库共享**的 —— 于是落后实例的一次 checkpoint 会用它的旧索引覆盖介质,
|
||||||
|
* 把其它实例已提交的写入**静默抹掉**。实测(本提交的用例锁定):
|
||||||
|
* A.open → A.put(x) ; B.open(=读到 x) → B.put(y) → B.close()
|
||||||
|
* → A.checkpoint() // A 的索引里没有 y
|
||||||
|
* → 重开:x 在、**y 消失**,且 A.checkpoint() 没有报任何错
|
||||||
|
* 有了 owner 之后,checkpoint 前先比对 meta.owner:不是自己就说明
|
||||||
|
* 介质已被别的实例接管,此时**拒绝提交**并显式报错,而不是覆盖。
|
||||||
|
*/
|
||||||
|
owner?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
function defaultMedium(): IStorageBackend {
|
function defaultMedium(): IStorageBackend {
|
||||||
@@ -67,6 +81,16 @@ export class KVStore {
|
|||||||
private opQueue: Promise<unknown> = Promise.resolve();
|
private opQueue: Promise<unknown> = Promise.resolve();
|
||||||
/** 最近一次后台操作失败(checkpoint 时报告) */
|
/** 最近一次后台操作失败(checkpoint 时报告) */
|
||||||
private lastBackgroundError: unknown = null;
|
private lastBackgroundError: unknown = null;
|
||||||
|
/**
|
||||||
|
* v0.8.0(B-6):本实例的提交所有权标识。
|
||||||
|
*
|
||||||
|
* 每次 `open()` 领取一个新的随机 id 并写入 meta(见 `claimOwnership`)。
|
||||||
|
* checkpoint 前比对 `meta.owner`:不等于自己 → 介质已被更新的实例接管,
|
||||||
|
* 本实例的索引可能落后,**不得**再提交快照(否则会静默抹掉对方的写入)。
|
||||||
|
*/
|
||||||
|
private instanceId = '';
|
||||||
|
/** 是否已被更新的实例接管(进入该状态后所有写入与提交都被拒绝) */
|
||||||
|
private stale = false;
|
||||||
|
|
||||||
constructor(medium?: IStorageBackend, checkpointThreshold: number = DEFAULT_CHECKPOINT_THRESHOLD) {
|
constructor(medium?: IStorageBackend, checkpointThreshold: number = DEFAULT_CHECKPOINT_THRESHOLD) {
|
||||||
this.medium = medium ?? defaultMedium();
|
this.medium = medium ?? defaultMedium();
|
||||||
@@ -85,7 +109,14 @@ export class KVStore {
|
|||||||
async open(dbName: string): Promise<void> {
|
async open(dbName: string): Promise<void> {
|
||||||
if (this.opened) return;
|
if (this.opened) return;
|
||||||
this.dbName = dbName;
|
this.dbName = dbName;
|
||||||
|
// v0.7.4: 打开时同样清理后台错误状态(防 close/reopen 残留)
|
||||||
|
this.lastBackgroundError = null;
|
||||||
|
this.stale = false;
|
||||||
await this.medium.open(dbName);
|
await this.medium.open(dbName);
|
||||||
|
// v0.8.0(B-6):领取提交所有权(写回 meta.owner)。
|
||||||
|
// 打开是唯一"接管"介质的时机;此前的实例之后会在 checkpoint 时发现
|
||||||
|
// owner 已变而拒绝提交,从而不会用陈旧索引覆盖本实例的数据。
|
||||||
|
await this.claimOwnership();
|
||||||
this.index = new Map();
|
this.index = new Map();
|
||||||
this.seq = 0;
|
this.seq = 0;
|
||||||
this.logBytes = 0;
|
this.logBytes = 0;
|
||||||
@@ -131,8 +162,17 @@ export class KVStore {
|
|||||||
this.logBytes = log.byteLength;
|
this.logBytes = log.byteLength;
|
||||||
}
|
}
|
||||||
if (corruptOffsets.length > 0) {
|
if (corruptOffsets.length > 0) {
|
||||||
// 损坏尾部:截断日志(丢弃未确认记录),下次 checkpoint 落盘
|
// v0.8.0(B-6)根治:损坏尾部只能**截断到最后一条有效记录**,
|
||||||
await this.truncateLog();
|
// 绝不能清空整个日志。
|
||||||
|
//
|
||||||
|
// 修复前这里调用 `truncateLog()`(写空文件),于是"尾部一个字节损坏"
|
||||||
|
// 导致**全部已确认写入消失**:
|
||||||
|
// 写 3 条记录 → 第 4 条只写了一半(崩溃)→ 重开
|
||||||
|
// → 前 3 条被上面的重放读到内存,随后被 truncateLog() 从介质上抹掉
|
||||||
|
// → 再重开一次,数据全部为空。
|
||||||
|
// 这是本项目最严重的一类缺陷:把"损坏尾部"放大成"整库丢失"。
|
||||||
|
// 正确做法(与 repair() 一致):保留 [0, validBytes) 前缀。
|
||||||
|
await this.truncateLogTo(this.findValidLogLength(log));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -162,6 +202,11 @@ export class KVStore {
|
|||||||
this.index.clear();
|
this.index.clear();
|
||||||
this.seq = 0;
|
this.seq = 0;
|
||||||
this.logBytes = 0;
|
this.logBytes = 0;
|
||||||
|
// v0.7.4: 清理后台错误状态 —— 此前跨 close/reopen 残留,
|
||||||
|
// 重开后首次 checkpoint 会抛出上一次生命周期的旧错误
|
||||||
|
this.lastBackgroundError = null;
|
||||||
|
this.instanceId = '';
|
||||||
|
this.stale = false;
|
||||||
this.opened = false;
|
this.opened = false;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -255,6 +300,8 @@ export class KVStore {
|
|||||||
/** checkpoint:快照 → meta → 截断日志(时序保证任何崩溃窗口不丢数据) */
|
/** checkpoint:快照 → meta → 截断日志(时序保证任何崩溃窗口不丢数据) */
|
||||||
async checkpoint(): Promise<void> {
|
async checkpoint(): Promise<void> {
|
||||||
await this.enqueue(async () => {
|
await this.enqueue(async () => {
|
||||||
|
// v0.8.0(B-6):提交前确认所有权 —— 拒绝用陈旧索引覆盖介质
|
||||||
|
await this.assertOwnership();
|
||||||
// 报告上次后台失败
|
// 报告上次后台失败
|
||||||
if (this.lastBackgroundError !== null) {
|
if (this.lastBackgroundError !== null) {
|
||||||
const error = this.lastBackgroundError;
|
const error = this.lastBackgroundError;
|
||||||
@@ -266,9 +313,8 @@ export class KVStore {
|
|||||||
// 1. 写快照(COW 原子)
|
// 1. 写快照(COW 原子)
|
||||||
const snapBytes = encodeSnapshot(this.seq, this.index);
|
const snapBytes = encodeSnapshot(this.seq, this.index);
|
||||||
await this.medium.write(SNAPSHOT_KEY, snapBytes.buffer as ArrayBuffer);
|
await this.medium.write(SNAPSHOT_KEY, snapBytes.buffer as ArrayBuffer);
|
||||||
// 2. 写 meta(指向新水位)
|
// 2. 写 meta(指向新水位 + 续期所有权)
|
||||||
const meta: KVStoreMeta = { seq: this.seq };
|
await this.writeMeta();
|
||||||
await this.medium.write(META_KEY, new TextEncoder().encode(JSON.stringify(meta)).buffer);
|
|
||||||
// 3. 截断日志(meta 已更新 → 截断安全)
|
// 3. 截断日志(meta 已更新 → 截断安全)
|
||||||
await this.truncateLog();
|
await this.truncateLog();
|
||||||
});
|
});
|
||||||
@@ -331,6 +377,15 @@ export class KVStore {
|
|||||||
deletes: string[],
|
deletes: string[],
|
||||||
appends: Record<string, ArrayBuffer> = {},
|
appends: Record<string, ArrayBuffer> = {},
|
||||||
): Promise<void> {
|
): Promise<void> {
|
||||||
|
// v0.8.0(B-6):已被接管 → 拒绝写入(否则写入只会进 WAL 而永远无法提交,
|
||||||
|
// 或者在下一次 checkpoint 时把对方的提交覆盖掉)
|
||||||
|
if (this.stale) {
|
||||||
|
throw new DatabaseError(
|
||||||
|
'KVStore instance has been superseded by another instance on the same database;'
|
||||||
|
+ ' re-open the database to continue',
|
||||||
|
'STALE_INSTANCE',
|
||||||
|
);
|
||||||
|
}
|
||||||
this.seq++;
|
this.seq++;
|
||||||
const record = encodeLogRecord(this.seq, puts, deletes, appends);
|
const record = encodeLogRecord(this.seq, puts, deletes, appends);
|
||||||
try {
|
try {
|
||||||
@@ -378,14 +433,41 @@ export class KVStore {
|
|||||||
|
|
||||||
// 自动 checkpoint(日志超阈值)
|
// 自动 checkpoint(日志超阈值)
|
||||||
if (this.checkpointThreshold > 0 && this.logBytes >= this.checkpointThreshold) {
|
if (this.checkpointThreshold > 0 && this.logBytes >= this.checkpointThreshold) {
|
||||||
await this.medium.write(SNAPSHOT_KEY, encodeSnapshot(this.seq, this.index).buffer as ArrayBuffer);
|
await this.autoCheckpoint();
|
||||||
const meta: KVStoreMeta = { seq: this.seq };
|
|
||||||
await this.medium.write(META_KEY, new TextEncoder().encode(JSON.stringify(meta)).buffer);
|
|
||||||
await this.truncateLog();
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 截断日志(清空文件) */
|
/**
|
||||||
|
* v0.8.0(B-6)根治:自动 checkpoint 的失败语义。
|
||||||
|
*
|
||||||
|
* 此前的顺序是"WAL 追加成功 → 更新内存索引 → 自动 checkpoint",而自动
|
||||||
|
* checkpoint **没有 try/catch**:快照/元数据写失败会把异常抛出 `appendRecord`,
|
||||||
|
* 一路冒泡到调用方 —— 但那条写入**已经持久化在 WAL 里了**(WAL 是权威来源,
|
||||||
|
* 崩溃后一定能重放出来)。于是用户看到 `put()` 失败、以为数据没进去,
|
||||||
|
* 实际数据已经落盘 —— 报错与事实相反,属于最有害的一类不一致
|
||||||
|
* (调用方据此重试会写入两次,或据"失败"丢弃业务状态)。
|
||||||
|
*
|
||||||
|
* 现在的语义(与 B-6 止血方案一致):
|
||||||
|
* - 已确认的写入**不得**因为后台失败而报错;
|
||||||
|
* - 失败被记录为 `lastBackgroundError`,由**下一次** `checkpoint()`
|
||||||
|
* 显式报告(那是用户主动要求把数据压实到快照的时机,此时失败是真实问题);
|
||||||
|
* - 内存索引与 WAL 仍然一致(两者都已包含这条写入),不产生半状态。
|
||||||
|
*/
|
||||||
|
private async autoCheckpoint(): Promise<void> {
|
||||||
|
try {
|
||||||
|
await this.assertOwnership();
|
||||||
|
await this.medium.write(SNAPSHOT_KEY, encodeSnapshot(this.seq, this.index).buffer as ArrayBuffer);
|
||||||
|
await this.writeMeta();
|
||||||
|
await this.truncateLog();
|
||||||
|
} catch (error) {
|
||||||
|
// 陈旧实例的自动 checkpoint 失败**不**计入 lastBackgroundError:
|
||||||
|
// 那会把它伪装成"介质故障",而真实原因是本实例已被接管,用户需要的是
|
||||||
|
// 立即的、明确的 STALE_INSTANCE 错误(由 assertOwnership 在显式路径给出)。
|
||||||
|
if (!this.stale) this.lastBackgroundError = error;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 截断日志(清空文件)—— 仅用于 checkpoint 之后:快照已覆盖全部数据 */
|
||||||
private async truncateLog(): Promise<void> {
|
private async truncateLog(): Promise<void> {
|
||||||
try {
|
try {
|
||||||
await this.medium.write(LOG_KEY, new ArrayBuffer(0));
|
await this.medium.write(LOG_KEY, new ArrayBuffer(0));
|
||||||
@@ -393,6 +475,27 @@ export class KVStore {
|
|||||||
this.logBytes = 0;
|
this.logBytes = 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0(B-6):把日志截断到 `keepBytes` 长度(保留有效前缀)。
|
||||||
|
*
|
||||||
|
* 与 `truncateLog()` 的区别:checkpoint 后日志内容已被快照覆盖,可以清空;
|
||||||
|
* 而崩溃恢复时日志里**前面的记录是唯一的数据来源**(快照可能落后很多个
|
||||||
|
* checkpoint),只能丢弃损坏的尾部。两者语义完全不同,因此是两个方法。
|
||||||
|
*/
|
||||||
|
private async truncateLogTo(keepBytes: number): Promise<void> {
|
||||||
|
try {
|
||||||
|
const raw = await this.medium.read(LOG_KEY);
|
||||||
|
if (!raw) return;
|
||||||
|
if (keepBytes >= raw.byteLength) return; // 无需截断(损坏判定与读取之间无变化)
|
||||||
|
const kept = new Uint8Array(raw).subarray(0, keepBytes).slice();
|
||||||
|
await this.medium.write(LOG_KEY, kept.buffer as ArrayBuffer);
|
||||||
|
this.logBytes = keepBytes;
|
||||||
|
} catch {
|
||||||
|
// 截断失败不影响本次恢复的内存状态:日志文件多出的损坏尾部会在
|
||||||
|
// 下次 open 时被同样识别并跳过(解析在损坏处停止),因此不会读到脏数据。
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/** 应用记录条目到内存索引 */
|
/** 应用记录条目到内存索引 */
|
||||||
private applyRecord(entries: { op: KVLogOp; key: string; value: ArrayBuffer }[]): void {
|
private applyRecord(entries: { op: KVLogOp; key: string; value: ArrayBuffer }[]): void {
|
||||||
for (const e of entries) {
|
for (const e of entries) {
|
||||||
@@ -414,6 +517,66 @@ export class KVStore {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// =======================================================================
|
||||||
|
// 提交所有权(v0.8.0 / B-6)
|
||||||
|
// =======================================================================
|
||||||
|
|
||||||
|
/** 生成实例 id(无需密码学强度,只要同期实例间不碰撞) */
|
||||||
|
private newInstanceId(): string {
|
||||||
|
const rand = Math.random().toString(36).slice(2);
|
||||||
|
return `kv-${Date.now().toString(36)}-${rand}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 读取 meta(缺失或损坏时返回 null) */
|
||||||
|
private async readMeta(): Promise<KVStoreMeta | null> {
|
||||||
|
try {
|
||||||
|
const raw = await this.medium.read(META_KEY);
|
||||||
|
if (!raw || raw.byteLength === 0) return null;
|
||||||
|
return JSON.parse(new TextDecoder().decode(raw)) as KVStoreMeta;
|
||||||
|
} catch {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 写入 meta(水位 + 所有权续期) */
|
||||||
|
private async writeMeta(): Promise<void> {
|
||||||
|
const meta: KVStoreMeta = { seq: this.seq, owner: this.instanceId };
|
||||||
|
await this.medium.write(META_KEY, new TextEncoder().encode(JSON.stringify(meta)).buffer);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** open() 时领取所有权:写入本实例 id,使此前打开的实例在提交时发现自己已过期 */
|
||||||
|
private async claimOwnership(): Promise<void> {
|
||||||
|
this.instanceId = this.newInstanceId();
|
||||||
|
await this.writeMeta();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 提交前校验所有权。
|
||||||
|
*
|
||||||
|
* 只有当介质上的 owner 存在、且**不是**本实例时才算过期:
|
||||||
|
* - owner 缺失(旧版本库 / meta 被清理)→ 视为无主,允许提交;
|
||||||
|
* - owner === 自己 → 正常。
|
||||||
|
* 一旦判定过期,本实例进入 `stale` 状态:后续写入与提交都显式报错 ——
|
||||||
|
* 显式失败远好于静默丢数据(这是本 wound 的全部意义)。
|
||||||
|
*/
|
||||||
|
private async assertOwnership(): Promise<void> {
|
||||||
|
const meta = await this.readMeta();
|
||||||
|
if (meta?.owner && this.instanceId && meta.owner !== this.instanceId) {
|
||||||
|
this.stale = true;
|
||||||
|
throw new DatabaseError(
|
||||||
|
'KVStore instance has been superseded by another instance on the same database'
|
||||||
|
+ ' (refusing to commit a stale snapshot, which would discard the newer instance\'s writes);'
|
||||||
|
+ ' re-open the database to continue',
|
||||||
|
'STALE_INSTANCE',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 本实例是否已被更新实例接管(诊断用) */
|
||||||
|
isStale(): boolean {
|
||||||
|
return this.stale;
|
||||||
|
}
|
||||||
|
|
||||||
/** 确定日志中有效字节长度(从 0 开始连续解析到第一条损坏/残缺记录) */
|
/** 确定日志中有效字节长度(从 0 开始连续解析到第一条损坏/残缺记录) */
|
||||||
private findValidLogLength(log: Uint8Array): number {
|
private findValidLogLength(log: Uint8Array): number {
|
||||||
let offset = 0;
|
let offset = 0;
|
||||||
|
|||||||
@@ -7,54 +7,127 @@
|
|||||||
* (模拟"磁盘持久化"语义——重新 open 同名库可读到上次写入的数据)。
|
* (模拟"磁盘持久化"语义——重新 open 同名库可读到上次写入的数据)。
|
||||||
*
|
*
|
||||||
* 仅用于测试与 Node 环境;浏览器使用 OPFS 介质(KVStore 默认自动选择)。
|
* 仅用于测试与 Node 环境;浏览器使用 OPFS 介质(KVStore 默认自动选择)。
|
||||||
|
*
|
||||||
|
* ============================================================================
|
||||||
|
* v0.8.0(B-6):**介质本身必须有"多个实例共享同一份字节"的语义**
|
||||||
|
* ============================================================================
|
||||||
|
* 修复前每个实例各有一个私有 `materialized` 拼接缓存,而缓存只在**自己的**
|
||||||
|
* write/append 时失效。于是实例 A 读过的键会一直被 A 的缓存命中,**看不到**
|
||||||
|
* 实例 B 之后的写入 —— 介质行为退化为"每个实例各有一份快照"。
|
||||||
|
*
|
||||||
|
* 这个缺陷直接损害了本项目的验证能力:所有"两个实例共享介质"的崩溃/多标签页
|
||||||
|
* 用例都在一个错误的介质上运行。实测:
|
||||||
|
* A.open(db) → A 读 meta(缓存 A 的 owner)
|
||||||
|
* B.open(db) → B 写 meta(新 owner)
|
||||||
|
* A.checkpoint() → A 读 meta 仍得到**自己**的 owner
|
||||||
|
* → "陈旧实例拒绝提交"的所有权守卫看起来失效,实际是介质没把 B 的写入
|
||||||
|
* 反映给 A。
|
||||||
|
*
|
||||||
|
* 现在缓存**按库名共享**(挂在注册表条目上),并且失效是"对这一份存储"生效的:
|
||||||
|
* - 任何实例的 write/append/delete 都会清掉该库的共享缓存;
|
||||||
|
* - 存储为空时写回 `null` 哨兵,使"删除后仍读到旧值"不可能发生。
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import type { IStorageBackend } from '../aria/store/backend';
|
import type { IStorageBackend } from '../aria/store/backend';
|
||||||
|
|
||||||
/** 全局注册表:dbName → key → chunk 列表(跨实例共享,模拟持久化) */
|
/** 一份"磁盘":chunk 列表 + 拼接缓存(按库名共享,跨实例可见) */
|
||||||
const registry = new Map<string, Map<string, ArrayBuffer[]>>();
|
interface DbStore {
|
||||||
|
/** key → chunk 列表(append O(1),read 时一次性拼接) */
|
||||||
|
chunks: Map<string, ArrayBuffer[]>;
|
||||||
|
/** 惰性拼接缓存:key → 拼接结果(`null` 表示"该键不存在") */
|
||||||
|
materialized: Map<string, ArrayBuffer | null>;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 全局注册表:dbName(+ 库 id)→ 共享存储(跨实例共享,模拟持久化) */
|
||||||
|
const registry = new Map<string, DbStore>();
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 库 id 计数器:用于 `clearRegistry()` 后让"同名但属于新会话"的库拿到
|
||||||
|
* **新的** DbStore,而不是复用上一会话残留的对象引用。
|
||||||
|
*/
|
||||||
|
let registryEpoch = 0;
|
||||||
|
/** 当前会话里 dbName → 本次会话的 DbStore(换代后失效) */
|
||||||
|
let liveStores = new Map<string, { epoch: number; store: DbStore }>();
|
||||||
|
|
||||||
|
function storeFor(name: string): DbStore {
|
||||||
|
const live = liveStores.get(name);
|
||||||
|
if (live && live.epoch === registryEpoch) return live.store;
|
||||||
|
|
||||||
|
const existing = registry.get(name);
|
||||||
|
if (existing) {
|
||||||
|
// 同一会话内复用(多个 backend 实例共享同一份字节)
|
||||||
|
liveStores.set(name, { epoch: registryEpoch, store: existing });
|
||||||
|
return existing;
|
||||||
|
}
|
||||||
|
const created: DbStore = { chunks: new Map(), materialized: new Map() };
|
||||||
|
registry.set(name, created);
|
||||||
|
liveStores.set(name, { epoch: registryEpoch, store: created });
|
||||||
|
return created;
|
||||||
|
}
|
||||||
|
|
||||||
export class SharedMemoryBackend implements IStorageBackend {
|
export class SharedMemoryBackend implements IStorageBackend {
|
||||||
private dbName = '';
|
private dbName = '';
|
||||||
/** 存储:key → chunk 列表(append O(1),read 时一次性拼接缓存) */
|
private store: DbStore | null = null;
|
||||||
private chunks: Map<string, ArrayBuffer[]> = new Map();
|
|
||||||
/** 惰性拼接缓存(read 后缓存,write/append 失效) */
|
|
||||||
private materialized: Map<string, ArrayBuffer> = new Map();
|
|
||||||
|
|
||||||
/** 清空全局注册表(测试隔离用) */
|
/**
|
||||||
|
* 清空全局注册表(测试隔离用)。
|
||||||
|
*
|
||||||
|
* 同时让"当前会话"失效:后续 open 会为同名库创建全新的存储 ——
|
||||||
|
* 否则 `clearRegistry()` 之后新建的实例可能仍指向上一用例的 DbStore
|
||||||
|
*(注册表被清空但对象引用还活在旧实例里),出现跨用例数据泄漏。
|
||||||
|
*/
|
||||||
static clearRegistry(): void {
|
static clearRegistry(): void {
|
||||||
registry.clear();
|
registry.clear();
|
||||||
|
registryEpoch += 1;
|
||||||
|
liveStores = new Map();
|
||||||
}
|
}
|
||||||
|
|
||||||
async open(name: string): Promise<void> {
|
async open(name: string): Promise<void> {
|
||||||
this.dbName = name;
|
this.dbName = name;
|
||||||
if (!registry.has(name)) {
|
this.store = storeFor(name);
|
||||||
registry.set(name, new Map());
|
|
||||||
}
|
|
||||||
// 从注册表恢复 chunks(持久化语义)
|
|
||||||
this.chunks = registry.get(name)! as unknown as Map<string, ArrayBuffer[]>;
|
|
||||||
this.materialized = new Map();
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/** close 不清除数据(持久化语义:重开同名库数据仍在) */
|
/** close 不清除数据(持久化语义:重开同名库数据仍在) */
|
||||||
async close(): Promise<void> {
|
async close(): Promise<void> {
|
||||||
this.chunks = new Map();
|
this.store = null;
|
||||||
this.materialized = new Map();
|
|
||||||
}
|
}
|
||||||
|
|
||||||
isOpen(): boolean {
|
isOpen(): boolean {
|
||||||
return this.dbName !== '';
|
return this.dbName !== '' && this.store !== null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 取当前共享存储。
|
||||||
|
*
|
||||||
|
* v0.8.0:`close()` 之后**不再**惰性重新绑定到共享存储 —— 那会让
|
||||||
|
* "close 后读不到数据"变成"close 后又读到了"(实测该行为被
|
||||||
|
* `tests/production-abnormal.test.ts` 的 close 契约用例逮到)。
|
||||||
|
*
|
||||||
|
* 正确语义(与既有契约一致,见该用例注释):
|
||||||
|
* - close 后读 → `null`(本实例已不持有数据);
|
||||||
|
* - close 后写/删/清空 → **不抛错**,但作用在游离内存上、**不持久化**
|
||||||
|
* - 跨实例持久化语义不受影响:另一个实例 open 同名库仍能读到 close 前的数据。
|
||||||
|
*/
|
||||||
|
private db(): DbStore {
|
||||||
|
if (!this.store) {
|
||||||
|
// 游离存储:仅供 close 后的调用"安全落地",不进入注册表
|
||||||
|
this.store = { chunks: new Map(), materialized: new Map() };
|
||||||
|
}
|
||||||
|
return this.store;
|
||||||
}
|
}
|
||||||
|
|
||||||
async read(key: string): Promise<ArrayBuffer | null> {
|
async read(key: string): Promise<ArrayBuffer | null> {
|
||||||
if (this.materialized.has(key)) return this.materialized.get(key)!;
|
const db = this.db();
|
||||||
const list = this.chunks.get(key);
|
if (db.materialized.has(key)) return db.materialized.get(key) ?? null;
|
||||||
if (!list || list.length === 0) return null;
|
const list = db.chunks.get(key);
|
||||||
|
if (!list || list.length === 0) {
|
||||||
|
db.materialized.set(key, null);
|
||||||
|
return null;
|
||||||
|
}
|
||||||
if (list.length === 1) {
|
if (list.length === 1) {
|
||||||
this.materialized.set(key, list[0]);
|
db.materialized.set(key, list[0]);
|
||||||
return list[0];
|
return list[0];
|
||||||
}
|
}
|
||||||
const total = list.reduce((s2, c) => s2 + c.byteLength, 0);
|
const total = list.reduce((sum, c) => sum + c.byteLength, 0);
|
||||||
const combined = new Uint8Array(total);
|
const combined = new Uint8Array(total);
|
||||||
let off = 0;
|
let off = 0;
|
||||||
for (const c of list) {
|
for (const c of list) {
|
||||||
@@ -62,56 +135,61 @@ export class SharedMemoryBackend implements IStorageBackend {
|
|||||||
off += c.byteLength;
|
off += c.byteLength;
|
||||||
}
|
}
|
||||||
const buf = combined.buffer as ArrayBuffer;
|
const buf = combined.buffer as ArrayBuffer;
|
||||||
this.materialized.set(key, buf);
|
db.materialized.set(key, buf);
|
||||||
return buf;
|
return buf;
|
||||||
}
|
}
|
||||||
|
|
||||||
async write(key: string, data: ArrayBuffer): Promise<void> {
|
async write(key: string, data: ArrayBuffer): Promise<void> {
|
||||||
this.chunks.set(key, [data]);
|
const db = this.db();
|
||||||
this.materialized.set(key, data);
|
db.chunks.set(key, [data]);
|
||||||
|
db.materialized.set(key, data);
|
||||||
}
|
}
|
||||||
|
|
||||||
async append(key: string, data: ArrayBuffer): Promise<void> {
|
async append(key: string, data: ArrayBuffer): Promise<void> {
|
||||||
// O(1) 追加:只记录 chunk,read 时惰性拼接
|
// O(1) 追加:只记录 chunk,read 时惰性拼接。
|
||||||
const list = this.chunks.get(key);
|
// 缓存按库共享,因此这里清掉的缓存对所有实例都生效 —— 这正是
|
||||||
if (list) {
|
// "另一个实例的写入必须对自己可见"这一语义的实现点。
|
||||||
list.push(data);
|
const db = this.db();
|
||||||
} else {
|
const list = db.chunks.get(key);
|
||||||
this.chunks.set(key, [data]);
|
if (list) list.push(data);
|
||||||
}
|
else db.chunks.set(key, [data]);
|
||||||
this.materialized.delete(key);
|
db.materialized.delete(key);
|
||||||
}
|
}
|
||||||
|
|
||||||
async writeMany(entries: Record<string, ArrayBuffer>): Promise<void> {
|
async writeMany(entries: Record<string, ArrayBuffer>): Promise<void> {
|
||||||
// 同步批量写入 = 原子(JS 单线程,无中间 await 点)
|
// 同步批量写入 = 原子(JS 单线程,无中间 await 点)
|
||||||
|
const db = this.db();
|
||||||
for (const [key, data] of Object.entries(entries)) {
|
for (const [key, data] of Object.entries(entries)) {
|
||||||
this.chunks.set(key, [data]);
|
db.chunks.set(key, [data]);
|
||||||
this.materialized.set(key, data);
|
db.materialized.set(key, data);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
async delete(key: string): Promise<void> {
|
async delete(key: string): Promise<void> {
|
||||||
this.chunks.delete(key);
|
const db = this.db();
|
||||||
this.materialized.delete(key);
|
db.chunks.delete(key);
|
||||||
|
db.materialized.set(key, null);
|
||||||
}
|
}
|
||||||
|
|
||||||
async deleteMany(keys: string[]): Promise<void> {
|
async deleteMany(keys: string[]): Promise<void> {
|
||||||
|
const db = this.db();
|
||||||
for (const key of keys) {
|
for (const key of keys) {
|
||||||
this.chunks.delete(key);
|
db.chunks.delete(key);
|
||||||
this.materialized.delete(key);
|
db.materialized.set(key, null);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
async listKeys(): Promise<string[]> {
|
async listKeys(): Promise<string[]> {
|
||||||
return Array.from(this.chunks.keys());
|
return Array.from(this.db().chunks.keys());
|
||||||
}
|
}
|
||||||
|
|
||||||
async exists(key: string): Promise<boolean> {
|
async exists(key: string): Promise<boolean> {
|
||||||
return this.chunks.has(key);
|
return this.db().chunks.has(key);
|
||||||
}
|
}
|
||||||
|
|
||||||
async clear(): Promise<void> {
|
async clear(): Promise<void> {
|
||||||
this.chunks.clear();
|
const db = this.db();
|
||||||
this.materialized.clear();
|
db.chunks.clear();
|
||||||
|
db.materialized.clear();
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -255,6 +255,22 @@ export class KVStoreEngine implements IStorageEngine {
|
|||||||
|
|
||||||
// ---- CRUD ----
|
// ---- CRUD ----
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0(B-1):写入前置校验 —— 委托给内存引擎(两者共享同一 schema 表)。
|
||||||
|
*
|
||||||
|
* KVStore/Hybrid 的行校验一直"继承"自 MemoryEngine,这正是 A12 的成因:
|
||||||
|
* 三者共用一份**缺 maxLength/min/max** 的实现。现在共享的是
|
||||||
|
* `table/validation.ts` 的规范实现,继承关系不再影响约束覆盖面。
|
||||||
|
*/
|
||||||
|
async validatePayload(
|
||||||
|
tableName: string,
|
||||||
|
rows: Record<string, unknown>[],
|
||||||
|
mode: 'insert' | 'update' = 'insert',
|
||||||
|
): Promise<void> {
|
||||||
|
this.ensureOpen();
|
||||||
|
return this.memory.validatePayload(tableName, rows, mode);
|
||||||
|
}
|
||||||
|
|
||||||
async insert(tableName: string, rows: Record<string, unknown>[]): Promise<string[]> {
|
async insert(tableName: string, rows: Record<string, unknown>[]): Promise<string[]> {
|
||||||
this.ensureOpen();
|
this.ensureOpen();
|
||||||
const pks = await this.memory.insert(tableName, rows);
|
const pks = await this.memory.insert(tableName, rows);
|
||||||
|
|||||||
+233
-38
@@ -6,8 +6,10 @@
|
|||||||
import type { IStorageEngine } from './interface';
|
import type { IStorageEngine } from './interface';
|
||||||
import type { QueryPlan, TableSchema, WhereCondition } from '../constants';
|
import type { QueryPlan, TableSchema, WhereCondition } from '../constants';
|
||||||
import { DatabaseError } from '../constants';
|
import { DatabaseError } from '../constants';
|
||||||
|
import { cloneRow } from './row_clone';
|
||||||
import { matchWhere, applyOrderBy, projectColumns } from '../query/where-matcher';
|
import { matchWhere, applyOrderBy, projectColumns } from '../query/where-matcher';
|
||||||
import { stripUndefinedUpdates } from '../table/schema';
|
import { stripUndefinedUpdates } from '../table/schema';
|
||||||
|
import { compileValidator, type RowValidator } from '../table/validation';
|
||||||
|
|
||||||
export class MemoryEngine implements IStorageEngine {
|
export class MemoryEngine implements IStorageEngine {
|
||||||
readonly name = 'memory';
|
readonly name = 'memory';
|
||||||
@@ -18,6 +20,12 @@ export class MemoryEngine implements IStorageEngine {
|
|||||||
private opened = false;
|
private opened = false;
|
||||||
/** v0.4.2-fix: 库内元数据(迁移版本持久化用) */
|
/** v0.4.2-fix: 库内元数据(迁移版本持久化用) */
|
||||||
private metaStore: Map<string, string> = new Map();
|
private metaStore: Map<string, string> = new Map();
|
||||||
|
/**
|
||||||
|
* v0.7.4: 由 CREATE UNIQUE INDEX 添加的 unique 列(table:col)。
|
||||||
|
* 与建表 UNIQUE 约束区分:DROP INDEX 只允许解除索引来源的 unique,
|
||||||
|
* 建表约束需重建表(对齐 SQLite 语义,此前静默解除且不可恢复)。
|
||||||
|
*/
|
||||||
|
private uniqueIndexCols: Set<string> = new Set();
|
||||||
|
|
||||||
// ---- 事务快照 ----
|
// ---- 事务快照 ----
|
||||||
private snapshot: {
|
private snapshot: {
|
||||||
@@ -82,6 +90,11 @@ export class MemoryEngine implements IStorageEngine {
|
|||||||
async dropTable(tableName: string): Promise<void> {
|
async dropTable(tableName: string): Promise<void> {
|
||||||
this.ensureTable(tableName);
|
this.ensureTable(tableName);
|
||||||
this.schemas.delete(tableName); this.tables.delete(tableName); this.indexes.delete(tableName);
|
this.schemas.delete(tableName); this.tables.delete(tableName); this.indexes.delete(tableName);
|
||||||
|
// v0.7.4: 清理该表的 unique 索引来源标记(重建同名表不残留)
|
||||||
|
const prefix = `${tableName}:`;
|
||||||
|
for (const key of this.uniqueIndexCols) {
|
||||||
|
if (key.startsWith(prefix)) this.uniqueIndexCols.delete(key);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
async hasTable(tableName: string): Promise<boolean> { return this.schemas.has(tableName); }
|
async hasTable(tableName: string): Promise<boolean> { return this.schemas.has(tableName); }
|
||||||
@@ -113,6 +126,49 @@ export class MemoryEngine implements IStorageEngine {
|
|||||||
throw new DatabaseError(`Column "${column.name}" already exists in table "${tableName}"`, 'COLUMN_EXISTS');
|
throw new DatabaseError(`Column "${column.name}" already exists in table "${tableName}"`, 'COLUMN_EXISTS');
|
||||||
}
|
}
|
||||||
schema.columns[column.name] = column;
|
schema.columns[column.name] = column;
|
||||||
|
// v0.8.0 根治:ALTER ADD 必须建立二级索引桶。
|
||||||
|
//
|
||||||
|
// 此前只写 schema.columns 而不建桶,而唯一性预检完全依赖索引桶
|
||||||
|
// (`tableIndexes.get(colName)` 缺失即整段跳过)—— 于是
|
||||||
|
// `ALTER TABLE t ADD COLUMN email STRING UNIQUE` 之后插入重复 email
|
||||||
|
// **不会报错**;close/reopen 时 createTable 依 schema 建桶、回灌第 2 行
|
||||||
|
// 触发 UNIQUE_VIOLATION 而异常被引擎 open 路径吞掉 → **行静默消失**。
|
||||||
|
if (column.index || column.unique) {
|
||||||
|
if (!this.indexes.has(tableName)) this.indexes.set(tableName, new Map());
|
||||||
|
const tableIndexes = this.indexes.get(tableName)!;
|
||||||
|
if (!tableIndexes.has(column.name)) tableIndexes.set(column.name, new Map());
|
||||||
|
// 已存在行:先校验存量唯一性(重复则回滚本次 ALTER),再回填索引桶
|
||||||
|
const colIndex = tableIndexes.get(column.name)!;
|
||||||
|
const table = this.tables.get(tableName)!;
|
||||||
|
const seen = new Set<unknown>();
|
||||||
|
for (const [pk, row] of table) {
|
||||||
|
const value = row[column.name];
|
||||||
|
if (value === null || value === undefined) continue; // null 不受唯一约束
|
||||||
|
if (column.unique && seen.has(value)) {
|
||||||
|
tableIndexes.delete(column.name);
|
||||||
|
delete schema.columns[column.name];
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Duplicate value "${String(value)}" for UNIQUE column "${column.name}" in table "${tableName}"`,
|
||||||
|
'UNIQUE_VIOLATION',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
seen.add(value);
|
||||||
|
let pks = colIndex.get(value);
|
||||||
|
if (!pks) { pks = new Set(); colIndex.set(value, pks); }
|
||||||
|
pks.add(pk);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// v0.8.0(B-1):ADD 列在**已有行**上物化为 NULL。
|
||||||
|
//
|
||||||
|
// 行校验契约(table/validation.ts)保证"行含全部 schema 列",若 ALTER ADD
|
||||||
|
// 不补齐,新列在旧行上就是**键不存在**:内存里 `{id,name}`、而同一行经
|
||||||
|
// 落盘再读回(KVStore/Aria 的恢复路径会走 validateRow)变成
|
||||||
|
// `{id,name,phone:null}` —— 同一行的形状取决于"是否重启过"。
|
||||||
|
// 显式物化后,内存视图与持久化视图一致。
|
||||||
|
const addTable = this.tables.get(tableName)!;
|
||||||
|
for (const row of addTable.values()) {
|
||||||
|
if (!(column.name in row)) row[column.name] = null;
|
||||||
|
}
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
if (!schema.columns[column.name]) {
|
if (!schema.columns[column.name]) {
|
||||||
@@ -173,9 +229,11 @@ export class MemoryEngine implements IStorageEngine {
|
|||||||
|
|
||||||
/** v0.7.3: 按主键取已验证行(KVStoreEngine 持久化 validated 行用,含 default/类型归一) */
|
/** v0.7.3: 按主键取已验证行(KVStoreEngine 持久化 validated 行用,含 default/类型归一) */
|
||||||
getRow(tableName: string, pkValue: string): Record<string, unknown> | null {
|
getRow(tableName: string, pkValue: string): Record<string, unknown> | null {
|
||||||
|
// v0.8.0: 返回副本(调用方用于持久化,不得持有内部引用)
|
||||||
const table = this.tables.get(tableName);
|
const table = this.tables.get(tableName);
|
||||||
if (!table) return null;
|
if (!table) return null;
|
||||||
return table.get(pkValue) ?? null;
|
const row = table.get(pkValue);
|
||||||
|
return row ? cloneRow(row) : null;
|
||||||
}
|
}
|
||||||
|
|
||||||
async find(tableName: string, query: QueryPlan): Promise<Record<string, unknown>[]> {
|
async find(tableName: string, query: QueryPlan): Promise<Record<string, unknown>[]> {
|
||||||
@@ -195,7 +253,9 @@ export class MemoryEngine implements IStorageEngine {
|
|||||||
if (query.columns && query.columns.length > 0 && query.columns[0] !== '*') {
|
if (query.columns && query.columns.length > 0 && query.columns[0] !== '*') {
|
||||||
results = results.map((row) => projectColumns(row, query.columns!));
|
results = results.map((row) => projectColumns(row, query.columns!));
|
||||||
}
|
}
|
||||||
return results;
|
// v0.8.0: 返回副本 —— 此前直接交出内部行对象,调用方原地修改即改写存储
|
||||||
|
// 并让索引与行失配(该行从此查不出来)。见 engine/interface.ts 的行所有权约定。
|
||||||
|
return results.map((row) => cloneRow(row));
|
||||||
}
|
}
|
||||||
|
|
||||||
/** v0.4.0: 流式查询 — 逐行回调(单次迭代,不物化结果数组) */
|
/** v0.4.0: 流式查询 — 逐行回调(单次迭代,不物化结果数组) */
|
||||||
@@ -214,7 +274,7 @@ export class MemoryEngine implements IStorageEngine {
|
|||||||
for (const row of table.values()) {
|
for (const row of table.values()) {
|
||||||
if (hasWhere && !matchWhere(row, query.where!)) continue;
|
if (hasWhere && !matchWhere(row, query.where!)) continue;
|
||||||
if (skipped < offset) { skipped++; continue; }
|
if (skipped < offset) { skipped++; continue; }
|
||||||
onRow(project ? project(row) : row);
|
onRow(project ? project(row) : cloneRow(row));
|
||||||
count++;
|
count++;
|
||||||
if (count >= limit) break;
|
if (count >= limit) break;
|
||||||
}
|
}
|
||||||
@@ -229,11 +289,44 @@ export class MemoryEngine implements IStorageEngine {
|
|||||||
// v0.7.2: undefined 值视为"不更新该列"(保留旧值),null 显式置空
|
// v0.7.2: undefined 值视为"不更新该列"(保留旧值),null 显式置空
|
||||||
const cleanUpdates = stripUndefinedUpdates(updates);
|
const cleanUpdates = stripUndefinedUpdates(updates);
|
||||||
|
|
||||||
|
// v0.8.0(B-3):**不再需要**"检测到未解析标记就抛 NOT_SUPPORTED"的防御。
|
||||||
|
//
|
||||||
|
// 那段防御存在的原因是 QueryBuilder 直通引擎、绕过了 Executor 的子查询解析,
|
||||||
|
// 于是 `$subquery`/`$col`/`$exists` 在引擎层判 UNKNOWN → 静默影响 0 行。
|
||||||
|
// B-3 把 builder 改为"只产出 AST、执行一律经 Executor"之后,写路径上不可能
|
||||||
|
// 再出现未解析标记 —— 把"管线缺失"暴露成用户错误(NOT_SUPPORTED)是错误的
|
||||||
|
// 补救方向:用户没有做错任何事。
|
||||||
|
//
|
||||||
|
// 保留 `containsUnresolvedSubqueries` 的导入会给后来者"这里需要防御"的错觉,
|
||||||
|
// 因此一并移除(见 where-matcher 中该函数仍被 Executor 用于写路径预检)。
|
||||||
|
|
||||||
|
// v0.7.4: 未知列显式报错 —— 此前 SET nonexistent = ... 被静默写入存储行
|
||||||
|
// (validateRow 只遍历 schema 列,脏列残留在行内并随持久化落盘)
|
||||||
|
for (const col of Object.keys(cleanUpdates)) {
|
||||||
|
if (!schema.columns[col]) {
|
||||||
|
throw new DatabaseError(`Column "${col}" does not exist in table "${tableName}"`, 'COLUMN_NOT_FOUND');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// v0.7.2: 语句级原子性 — 两阶段(先全量预检,后执行)。
|
// v0.7.2: 语句级原子性 — 两阶段(先全量预检,后执行)。
|
||||||
// 此前逐行"校验+写入":第 N 行唯一冲突/校验失败抛错时,前 N-1 行已写入
|
// 此前逐行"校验+写入":第 N 行唯一冲突/校验失败抛错时,前 N-1 行已写入
|
||||||
// → 无事务下语句级部分提交(数据半更新且调用方已收到错误)。
|
// → 无事务下语句级部分提交(数据半更新且调用方已收到错误)。
|
||||||
const planned: { pk: string; row: Record<string, unknown>; updated: Record<string, unknown>; newPk: string }[] = [];
|
const planned: { pk: string; row: Record<string, unknown>; updated: Record<string, unknown>; newPk: string }[] = [];
|
||||||
const batchUnique: Map<string, Set<unknown>> = new Map();
|
const batchUnique: Map<string, Set<unknown>> = new Map();
|
||||||
|
/**
|
||||||
|
* v0.8.0 根治:批内新主键互查。
|
||||||
|
*
|
||||||
|
* 此前阶段 1 只用 `table.has(newPk)` 与**语句执行前的表**比对,看不到同一语句内
|
||||||
|
* 其它行即将写入的新主键。于是 `UPDATE t SET id = 'X'`(匹配 2 行)在阶段 2
|
||||||
|
* 逐行 `table.set(newPk, ...)` 相互覆盖 —— 返回 affected=2,表中却只剩 1 行
|
||||||
|
* (静默丢行)。INSERT 路径在 v0.7.3 已做批内 PK Set 互查,UPDATE 漏了。
|
||||||
|
*
|
||||||
|
* 保守拒绝策略:同一语句内两行改到同一新主键必然互相覆盖,直接报错。
|
||||||
|
* 注意这也会拒绝"两行互换主键"(A:x→y, B:y→x)这种最终状态合法的写法 ——
|
||||||
|
* 那属于需要基于最终状态判定的场景,宁可显式报错也不静默丢行
|
||||||
|
* (与既有的"唯一值交换更新保守拒绝"语义一致)。
|
||||||
|
*/
|
||||||
|
const batchNewPks = new Set<string>();
|
||||||
|
|
||||||
// 阶段 1:全量预检(任何一行失败 → 整条语句不执行)
|
// 阶段 1:全量预检(任何一行失败 → 整条语句不执行)
|
||||||
for (const [pk, row] of table) {
|
for (const [pk, row] of table) {
|
||||||
@@ -249,6 +342,16 @@ export class MemoryEngine implements IStorageEngine {
|
|||||||
'DUPLICATE_KEY',
|
'DUPLICATE_KEY',
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
// v0.8.0: 批内互查 —— 同一语句内两行改到同一新主键 → 整体拒绝(不得静默覆盖)
|
||||||
|
if (newPk !== pk) {
|
||||||
|
if (batchNewPks.has(newPk)) {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Duplicate primary key "${newPk}" in table "${tableName}" (multiple rows in the same statement update to the same key)`,
|
||||||
|
'DUPLICATE_KEY',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
batchNewPks.add(newPk);
|
||||||
|
}
|
||||||
planned.push({ pk, row, updated, newPk });
|
planned.push({ pk, row, updated, newPk });
|
||||||
}
|
}
|
||||||
// 阶段 1b:主键变更 RESTRICT 预检(引用表依赖行检查,任何修改前)
|
// 阶段 1b:主键变更 RESTRICT 预检(引用表依赖行检查,任何修改前)
|
||||||
@@ -362,7 +465,13 @@ export class MemoryEngine implements IStorageEngine {
|
|||||||
*/
|
*/
|
||||||
private checkUpdateRestrict(tableName: string, oldPk: string): void {
|
private checkUpdateRestrict(tableName: string, oldPk: string): void {
|
||||||
for (const [refTableName, refSchema] of this.schemas) {
|
for (const [refTableName, refSchema] of this.schemas) {
|
||||||
if (refTableName === tableName) continue;
|
// v0.8.0(A13):**不再跳过自引用外键**(refTableName === tableName)。
|
||||||
|
//
|
||||||
|
// 此前这里 `continue`,于是自引用外键(`parent_id REFERENCES node(id)`)
|
||||||
|
// 在所有级联路径上都被整体跳过:删除只删根、设置不变、预检也不查。
|
||||||
|
// 自引用的处理与普通外键完全相同,唯一需要注意的是遍历时机:
|
||||||
|
// 删除路径必须先递归子树再删父行,且**先收集引用者再处理**
|
||||||
|
//(自引用时遍历的正是同一个 Map,边遍历边删会跳过条目)。
|
||||||
for (const [colName, colDef] of Object.entries(refSchema.columns)) {
|
for (const [colName, colDef] of Object.entries(refSchema.columns)) {
|
||||||
if (!colDef.references || !colDef.onUpdate) continue;
|
if (!colDef.references || !colDef.onUpdate) continue;
|
||||||
const [refTable] = colDef.references.split('.');
|
const [refTable] = colDef.references.split('.');
|
||||||
@@ -401,7 +510,13 @@ export class MemoryEngine implements IStorageEngine {
|
|||||||
*/
|
*/
|
||||||
private async applyUpdateCascade(tableName: string, oldPk: string, newPk: string): Promise<void> {
|
private async applyUpdateCascade(tableName: string, oldPk: string, newPk: string): Promise<void> {
|
||||||
for (const [refTableName, refSchema] of this.schemas) {
|
for (const [refTableName, refSchema] of this.schemas) {
|
||||||
if (refTableName === tableName) continue;
|
// v0.8.0(A13):**不再跳过自引用外键**(refTableName === tableName)。
|
||||||
|
//
|
||||||
|
// 此前这里 `continue`,于是自引用外键(`parent_id REFERENCES node(id)`)
|
||||||
|
// 在所有级联路径上都被整体跳过:删除只删根、设置不变、预检也不查。
|
||||||
|
// 自引用的处理与普通外键完全相同,唯一需要注意的是遍历时机:
|
||||||
|
// 删除路径必须先递归子树再删父行,且**先收集引用者再处理**
|
||||||
|
//(自引用时遍历的正是同一个 Map,边遍历边删会跳过条目)。
|
||||||
for (const [colName, colDef] of Object.entries(refSchema.columns)) {
|
for (const [colName, colDef] of Object.entries(refSchema.columns)) {
|
||||||
if (!colDef.references || !colDef.onUpdate) continue;
|
if (!colDef.references || !colDef.onUpdate) continue;
|
||||||
const [refTable] = colDef.references.split('.');
|
const [refTable] = colDef.references.split('.');
|
||||||
@@ -409,10 +524,20 @@ export class MemoryEngine implements IStorageEngine {
|
|||||||
const refTableData = this.tables.get(refTableName);
|
const refTableData = this.tables.get(refTableName);
|
||||||
if (!refTableData) continue;
|
if (!refTableData) continue;
|
||||||
if (colDef.onUpdate !== 'CASCADE' && colDef.onUpdate !== 'SET NULL') continue;
|
if (colDef.onUpdate !== 'CASCADE' && colDef.onUpdate !== 'SET NULL') continue;
|
||||||
|
// v0.8.0(B-1):级联写入也必须过统一校验。
|
||||||
|
//
|
||||||
|
// 此前这里**直接赋值**绕过校验:`onUpdate: 'CASCADE'` 把新主键写入引用列时,
|
||||||
|
// 若该列有 maxLength / min / max 约束(新主键更长或超出范围),
|
||||||
|
// 约束被静默绕过 —— 与 A12 是同一类"校验只在部分写入路径生效"的问题。
|
||||||
|
// SET NULL 到 required/非空列的检查由 checkUpdateRestrict 在任何修改前完成,
|
||||||
|
// 此处再校验可同时覆盖 maxLength/min/max 这类"具体值相关"的约束。
|
||||||
|
const validator = this.rowValidator(refSchema);
|
||||||
for (const [refPk, refRow] of refTableData) {
|
for (const [refPk, refRow] of refTableData) {
|
||||||
if (String(refRow[colName]) !== oldPk) continue;
|
if (String(refRow[colName]) !== oldPk) continue;
|
||||||
|
const nextValue = colDef.onUpdate === 'CASCADE' ? newPk : null;
|
||||||
|
const { values } = validator.validatePartial({ [colName]: nextValue });
|
||||||
this.removeIndexEntries(refTableName, refRow, refPk);
|
this.removeIndexEntries(refTableName, refRow, refPk);
|
||||||
refRow[colName] = colDef.onUpdate === 'CASCADE' ? newPk : null;
|
refRow[colName] = values[colName];
|
||||||
this.updateIndexes(refTableName, refRow, refPk);
|
this.updateIndexes(refTableName, refRow, refPk);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -421,6 +546,16 @@ export class MemoryEngine implements IStorageEngine {
|
|||||||
|
|
||||||
async delete(tableName: string, query: QueryPlan): Promise<number> {
|
async delete(tableName: string, query: QueryPlan): Promise<number> {
|
||||||
this.ensureTable(tableName);
|
this.ensureTable(tableName);
|
||||||
|
// v0.8.0(B-3):**不再需要**"检测到未解析标记就抛 NOT_SUPPORTED"的防御。
|
||||||
|
//
|
||||||
|
// 那段防御存在的原因是 QueryBuilder 直通引擎、绕过了 Executor 的子查询解析,
|
||||||
|
// 于是 `$subquery`/`$col`/`$exists` 在引擎层判 UNKNOWN → 静默影响 0 行。
|
||||||
|
// B-3 把 builder 改为"只产出 AST、执行一律经 Executor"之后,写路径上不可能
|
||||||
|
// 再出现未解析标记 —— 把"管线缺失"暴露成用户错误(NOT_SUPPORTED)是错误的
|
||||||
|
// 补救方向:用户没有做错任何事。
|
||||||
|
//
|
||||||
|
// 保留 `containsUnresolvedSubqueries` 的导入会给后来者"这里需要防御"的错觉,
|
||||||
|
// 因此一并移除(见 where-matcher 中该函数仍被 Executor 用于写路径预检)。
|
||||||
const table = this.tables.get(tableName)!;
|
const table = this.tables.get(tableName)!;
|
||||||
const toDelete: { pk: string; row: Record<string, unknown> }[] = [];
|
const toDelete: { pk: string; row: Record<string, unknown> }[] = [];
|
||||||
for (const [pk, row] of table) {
|
for (const [pk, row] of table) {
|
||||||
@@ -459,7 +594,13 @@ export class MemoryEngine implements IStorageEngine {
|
|||||||
visited.add(visitKey);
|
visited.add(visitKey);
|
||||||
|
|
||||||
for (const [refTableName, refSchema] of this.schemas) {
|
for (const [refTableName, refSchema] of this.schemas) {
|
||||||
if (refTableName === tableName) continue;
|
// v0.8.0(A13):**不再跳过自引用外键**(refTableName === tableName)。
|
||||||
|
//
|
||||||
|
// 此前这里 `continue`,于是自引用外键(`parent_id REFERENCES node(id)`)
|
||||||
|
// 在所有级联路径上都被整体跳过:删除只删根、设置不变、预检也不查。
|
||||||
|
// 自引用的处理与普通外键完全相同,唯一需要注意的是遍历时机:
|
||||||
|
// 删除路径必须先递归子树再删父行,且**先收集引用者再处理**
|
||||||
|
//(自引用时遍历的正是同一个 Map,边遍历边删会跳过条目)。
|
||||||
for (const [colName, colDef] of Object.entries(refSchema.columns)) {
|
for (const [colName, colDef] of Object.entries(refSchema.columns)) {
|
||||||
if (!colDef.references || !colDef.onDelete) continue;
|
if (!colDef.references || !colDef.onDelete) continue;
|
||||||
const [refTable] = colDef.references.split('.');
|
const [refTable] = colDef.references.split('.');
|
||||||
@@ -551,7 +692,11 @@ export class MemoryEngine implements IStorageEngine {
|
|||||||
throw error;
|
throw error;
|
||||||
}
|
}
|
||||||
colDef.index = true;
|
colDef.index = true;
|
||||||
if (unique) colDef.unique = true;
|
if (unique) {
|
||||||
|
colDef.unique = true;
|
||||||
|
// v0.7.4: 记录唯一约束来源(DROP INDEX 时可解除;建表约束不可)
|
||||||
|
this.uniqueIndexCols.add(`${tableName}:${column}`);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
async dropIndex(tableName: string, column: string, _indexName?: string): Promise<void> {
|
async dropIndex(tableName: string, column: string, _indexName?: string): Promise<void> {
|
||||||
@@ -570,8 +715,20 @@ export class MemoryEngine implements IStorageEngine {
|
|||||||
if (!colDef.index && !colDef.unique) {
|
if (!colDef.index && !colDef.unique) {
|
||||||
throw new DatabaseError(`Index on column "${column}" does not exist in table "${tableName}"`, 'INDEX_NOT_FOUND');
|
throw new DatabaseError(`Index on column "${column}" does not exist in table "${tableName}"`, 'INDEX_NOT_FOUND');
|
||||||
}
|
}
|
||||||
|
// v0.7.4: 建表 UNIQUE 约束不可通过 DROP INDEX 解除 —— 此前 colDef.unique = false
|
||||||
|
// 静默解除约束(后续唯一性检查失效、重复数据入库)。对齐 SQLite 语义:
|
||||||
|
// 约束随建表存在,解除需重建表;仅 CREATE UNIQUE INDEX 添加的约束可随索引删除。
|
||||||
|
const uniqueKey = `${tableName}:${column}`;
|
||||||
|
if (colDef.unique && !this.uniqueIndexCols.has(uniqueKey)) {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Cannot drop index on column "${column}" in table "${tableName}": ` +
|
||||||
|
'UNIQUE constraint defined at table creation must be removed by recreating the table',
|
||||||
|
'NOT_SUPPORTED',
|
||||||
|
);
|
||||||
|
}
|
||||||
colDef.index = false;
|
colDef.index = false;
|
||||||
colDef.unique = false;
|
colDef.unique = false;
|
||||||
|
this.uniqueIndexCols.delete(uniqueKey);
|
||||||
const tableIndexes = this.indexes.get(tableName);
|
const tableIndexes = this.indexes.get(tableName);
|
||||||
if (tableIndexes) tableIndexes.delete(column);
|
if (tableIndexes) tableIndexes.delete(column);
|
||||||
}
|
}
|
||||||
@@ -637,27 +794,44 @@ export class MemoryEngine implements IStorageEngine {
|
|||||||
}
|
}
|
||||||
|
|
||||||
private validateRow(schema: TableSchema, row: Record<string, unknown>): Record<string, unknown> {
|
private validateRow(schema: TableSchema, row: Record<string, unknown>): Record<string, unknown> {
|
||||||
const validated: Record<string, unknown> = {};
|
// v0.8.0(B-1):委托给**唯一**的行校验实现(table/validation.ts)。
|
||||||
for (const [colName, colDef] of Object.entries(schema.columns)) {
|
//
|
||||||
let value = row[colName];
|
// 此前这里是第三份独立实现:只做类型检查,**没有** maxLength / min / max
|
||||||
if (value === undefined && colDef.default !== undefined) value = colDef.default;
|
// 约束(Aria 有)—— 于是同一份 schema、同一条 INSERT 是否报错取决于引擎
|
||||||
if (colDef.required && (value === undefined || value === null)) {
|
//(缺陷 A12)。同时它对未知列静默丢弃(A17)。
|
||||||
throw new DatabaseError(`Column "${colName}" is required in table "${schema.name}"`, 'VALIDATION_ERROR');
|
return this.rowValidator(schema).validateRow(row);
|
||||||
}
|
|
||||||
if (value !== undefined && value !== null) this.checkType(colName, colDef.type, value);
|
|
||||||
if (value !== undefined) validated[colName] = value;
|
|
||||||
}
|
|
||||||
return validated;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
private checkType(colName: string, type: string, value: unknown): void {
|
/**
|
||||||
const jsType = typeof value;
|
* 取该 schema 的行校验器(每次调用重新编译)。
|
||||||
switch (type) {
|
*
|
||||||
case 'string': if (jsType !== 'string') throw new DatabaseError(`Column "${colName}" expects string, got ${jsType}`, 'TYPE_ERROR'); break;
|
* 不缓存在引擎字段上:`alterTable` 会原地修改 schema 对象,
|
||||||
case 'number': if (jsType !== 'number') throw new DatabaseError(`Column "${colName}" expects number, got ${jsType}`, 'TYPE_ERROR'); break;
|
* 长期缓存会继续用过期列定义("加了列却仍被当未知列"这类难查问题)。
|
||||||
case 'boolean': if (jsType !== 'boolean') throw new DatabaseError(`Column "${colName}" expects boolean, got ${jsType}`, 'TYPE_ERROR'); break;
|
* 编译本身只是 `Object.entries` + Set 构造,相对一次 INSERT 的索引维护可忽略。
|
||||||
case 'date': if (jsType !== 'string' || isNaN(Date.parse(value as string))) throw new DatabaseError(`Column "${colName}" expects valid date`, 'TYPE_ERROR'); break;
|
*/
|
||||||
case 'json': if (jsType !== 'object') throw new DatabaseError(`Column "${colName}" expects object/array, got ${jsType}`, 'TYPE_ERROR'); break;
|
private rowValidator(schema: TableSchema): RowValidator {
|
||||||
|
return compileValidator(schema);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0(B-1):写入前置校验(见 `IStorageEngine.validatePayload` 契约)。
|
||||||
|
*
|
||||||
|
* 引擎在 `insert` / `update` 内部**同样**会校验 —— 本方法只是让 Executor 与
|
||||||
|
* QueryBuilder 能在"开始写入之前"拿到同一套判定结果,从而:
|
||||||
|
* - 多行 INSERT 的预检发生在任何副作用之前(错误信息带列名清单);
|
||||||
|
* - 直通路径与 SQL 路径不可能给出不同结论(同一个 `compileValidator`)。
|
||||||
|
*/
|
||||||
|
async validatePayload(
|
||||||
|
tableName: string,
|
||||||
|
rows: Record<string, unknown>[],
|
||||||
|
mode: 'insert' | 'update' = 'insert',
|
||||||
|
): Promise<void> {
|
||||||
|
this.ensureTable(tableName);
|
||||||
|
const schema = this.schemas.get(tableName)!;
|
||||||
|
const validator = this.rowValidator(schema);
|
||||||
|
for (const row of rows) {
|
||||||
|
if (mode === 'update') validator.validatePartial(stripUndefinedUpdates(row));
|
||||||
|
else validator.validateRow(row);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -697,6 +871,14 @@ export class MemoryEngine implements IStorageEngine {
|
|||||||
// colIndex.get(null) 恒 undefined → return [] 短路全表扫描 → 索引列
|
// colIndex.get(null) 恒 undefined → return [] 短路全表扫描 → 索引列
|
||||||
// IS NULL 恒空(对齐 AriaEngine v0.6.2 修复)
|
// IS NULL 恒空(对齐 AriaEngine v0.6.2 修复)
|
||||||
if (targetValue === null || targetValue === undefined) continue;
|
if (targetValue === null || targetValue === undefined) continue;
|
||||||
|
// v0.8.0 根治(与 AriaEngine 同步):**非原始值**(对象/数组)一律不走索引。
|
||||||
|
//
|
||||||
|
// 索引键只存原始值,因此 colIndex.get({...}) 恒 undefined → 下面 `return []`
|
||||||
|
// 会短路全表扫描 → 结果静默为空。真实触发场景正是"未解析的操作数":
|
||||||
|
// { $eq: { $col: 'y' } } ← 列对列比较(t.x = t.y)
|
||||||
|
// { $in: { $subquery: ... } } ← 关联 IN 子查询
|
||||||
|
// 它们本该由 executor 逐行求值,却先在这里被索引路径吞成空集。
|
||||||
|
if (typeof targetValue === 'object') continue;
|
||||||
const colIndex = tableIndexes.get(col);
|
const colIndex = tableIndexes.get(col);
|
||||||
if (colIndex) {
|
if (colIndex) {
|
||||||
const pks = colIndex.get(targetValue);
|
const pks = colIndex.get(targetValue);
|
||||||
@@ -759,7 +941,14 @@ export class MemoryEngine implements IStorageEngine {
|
|||||||
let totalCascade = 0;
|
let totalCascade = 0;
|
||||||
|
|
||||||
for (const [refTableName, refSchema] of this.schemas) {
|
for (const [refTableName, refSchema] of this.schemas) {
|
||||||
if (refTableName === tableName) continue;
|
// v0.8.0(A13):**不再跳过自引用外键**(refTableName === tableName)。
|
||||||
|
//
|
||||||
|
// 此前这里 `continue`,于是 `parent_id REFERENCES node(id)` 这种树形自引用
|
||||||
|
// 完全不做级联。实测(本提交的用例锁定):
|
||||||
|
// INSERT node: root <- a <- b <- c
|
||||||
|
// DELETE root → 只删掉 root,a/b/c 全部残留且 parent_id 指向已删除的行
|
||||||
|
// (且因为父行已删,它们之后**再也无法通过级联清理** —— 永久悬挂)
|
||||||
|
// 这是"删除留下悬挂引用"的静默数据不一致,比报错更糟。
|
||||||
|
|
||||||
for (const [colName, colDef] of Object.entries(refSchema.columns)) {
|
for (const [colName, colDef] of Object.entries(refSchema.columns)) {
|
||||||
if (!colDef.references || !colDef.onDelete) continue;
|
if (!colDef.references || !colDef.onDelete) continue;
|
||||||
@@ -770,16 +959,18 @@ export class MemoryEngine implements IStorageEngine {
|
|||||||
const refTableData = this.tables.get(refTableName);
|
const refTableData = this.tables.get(refTableName);
|
||||||
if (!refTableData) continue;
|
if (!refTableData) continue;
|
||||||
|
|
||||||
// 查找所有引用此主键的行
|
// 查找所有引用此主键的行。
|
||||||
const toDelete: string[] = [];
|
//
|
||||||
|
// v0.8.0(A13):**先完整收集再处理**。自引用场景下 `refTableData`
|
||||||
|
// 与当前遍历的表是同一个 Map,边遍历边删除会跳过条目(Map 迭代器
|
||||||
|
// 对已删除键的行为取决于删除位置)。收集成数组后处理即与迭代解耦。
|
||||||
|
const referrers: string[] = [];
|
||||||
for (const [refPk, refRow] of refTableData) {
|
for (const [refPk, refRow] of refTableData) {
|
||||||
if (String(refRow[colName]) === pkValue) {
|
if (String(refRow[colName]) === pkValue) referrers.push(refPk);
|
||||||
toDelete.push(refPk);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// RESTRICT: 存在引用行时禁止删除
|
// RESTRICT: 存在引用行时禁止删除
|
||||||
if (colDef.onDelete === 'RESTRICT' && toDelete.length > 0) {
|
if (colDef.onDelete === 'RESTRICT' && referrers.length > 0) {
|
||||||
throw new DatabaseError(
|
throw new DatabaseError(
|
||||||
`Cannot delete from "${tableName}": foreign key "${colName}" in "${refTableName}" has dependent rows`,
|
`Cannot delete from "${tableName}": foreign key "${colName}" in "${refTableName}" has dependent rows`,
|
||||||
'FOREIGN_KEY_VIOLATION',
|
'FOREIGN_KEY_VIOLATION',
|
||||||
@@ -788,18 +979,22 @@ export class MemoryEngine implements IStorageEngine {
|
|||||||
|
|
||||||
if (colDef.onDelete === 'CASCADE') {
|
if (colDef.onDelete === 'CASCADE') {
|
||||||
// 递归级联
|
// 递归级联
|
||||||
for (const refPk of toDelete) {
|
for (const refPk of referrers) {
|
||||||
const refRow = refTableData.get(refPk);
|
const refRow = refTableData.get(refPk);
|
||||||
if (refRow) {
|
if (refRow) {
|
||||||
// v0.3.3: 级联删除前清理索引条目
|
// v0.3.3: 级联删除前清理索引条目
|
||||||
this.removeIndexEntries(refTableName, refRow, refPk);
|
this.removeIndexEntries(refTableName, refRow, refPk);
|
||||||
|
// v0.8.0(A13):自引用时**先递归再删自己** —— 子树必须先被清掉,
|
||||||
|
// 否则删掉父行后子行的 parent_id 就再也匹配不上(悬挂)。
|
||||||
totalCascade += await this.cascadeDelete(refTableName, refPk, refRow, visited);
|
totalCascade += await this.cascadeDelete(refTableName, refPk, refRow, visited);
|
||||||
}
|
}
|
||||||
refTableData.delete(refPk);
|
if (refTableData.has(refPk)) {
|
||||||
totalCascade++;
|
refTableData.delete(refPk);
|
||||||
|
totalCascade++;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
} else if (colDef.onDelete === 'SET NULL') {
|
} else if (colDef.onDelete === 'SET NULL') {
|
||||||
for (const refPk of toDelete) {
|
for (const refPk of referrers) {
|
||||||
const refRow = refTableData.get(refPk);
|
const refRow = refTableData.get(refPk);
|
||||||
if (refRow) {
|
if (refRow) {
|
||||||
// v0.6.3-fix: 复用 removeIndexEntries 清理旧值索引 —— 此前手动
|
// v0.6.3-fix: 复用 removeIndexEntries 清理旧值索引 —— 此前手动
|
||||||
|
|||||||
@@ -0,0 +1,65 @@
|
|||||||
|
/**
|
||||||
|
* metona-sqlark — 行所有权(row ownership)工具
|
||||||
|
* @module engine/row_clone
|
||||||
|
*
|
||||||
|
* v0.8.0(审查修复):这三个函数此前放在 `engine/interface.ts`(被文档描述为
|
||||||
|
* "纯类型声明、可执行语句为 0、因此不纳入覆盖率统计")。它们其实是**运行时实现**
|
||||||
|
* 且被 Memory 引擎与 AriaEngine 的读路径调用 —— 放在纯类型文件里等于让真实实现
|
||||||
|
* 代码逃过覆盖率口径(同一类"统计口径虚高"问题正是 G5 门禁要根治的)。
|
||||||
|
* 现在实现搬到本模块,`interface.ts` 回到纯类型。
|
||||||
|
*/
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 深拷贝一行,使调用方**无法通过修改返回值改写存储**。
|
||||||
|
*
|
||||||
|
* 为什么必须做(审计实测):Memory/KVStore/Hybrid 三个引擎此前把内部行对象
|
||||||
|
* **直接**交给调用方:
|
||||||
|
* const rows = await db.query('SELECT * FROM t');
|
||||||
|
* rows[0].tag = 'HACKED'; // 存储被改写
|
||||||
|
* 再查 WHERE tag='HACKED' → 0 行;WHERE tag='x' → 0 行
|
||||||
|
* 即调用方一次无意的原地修改就能让索引与行失配、该行永久查不出来(Aria 因为是
|
||||||
|
* 反序列化路径反而幸免,于是又成了跨引擎行为差异)。
|
||||||
|
*
|
||||||
|
* 约定(所有引擎必须遵守):
|
||||||
|
* **读出的行是副本,写入接收的行也是副本** —— 引擎不得把内部行对象暴露给外部,
|
||||||
|
* 也不得持有调用方传入的行对象引用。
|
||||||
|
*
|
||||||
|
* 实现说明:结构化克隆可用时优先使用;退化路径(`cloneRowFallback`)逐层复制
|
||||||
|
* 普通对象/数组,并额外保留 Date、Uint8Array、ArrayBuffer。
|
||||||
|
*
|
||||||
|
* 前提:存储层写入的行已经过 `validateRow` 的 JSON 安全性检查 —— 例如 `date`
|
||||||
|
* 列要求的是**合法日期字符串**而不是 Date 实例(`table/validation.ts`),因此
|
||||||
|
* 行内永远是 JSON 安全值,"环境提供的 structuredClone 是否保留 Date"不影响
|
||||||
|
* 引擎行为(jsdom 的 structuredClone 是 JSON 化 polyfill,会把 Date 变字符串;
|
||||||
|
* 真实浏览器与 Node 的原生实现会保留)。
|
||||||
|
*/
|
||||||
|
export function cloneRow<T extends Record<string, unknown>>(row: T): T {
|
||||||
|
if (row === null || typeof row !== 'object') return row;
|
||||||
|
if (typeof structuredClone === 'function') {
|
||||||
|
try {
|
||||||
|
return structuredClone(row);
|
||||||
|
} catch {
|
||||||
|
// 含不可克隆值(函数/Proxy)时退化为逐层复制
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return cloneRowFallback(row) as T;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 退化实现:递归复制普通对象与数组(保留 Date) */
|
||||||
|
function cloneRowFallback(value: unknown): unknown {
|
||||||
|
if (value === null || typeof value !== 'object') return value;
|
||||||
|
if (value instanceof Date) return new Date(value.getTime());
|
||||||
|
if (Array.isArray(value)) return value.map((v) => cloneRowFallback(v));
|
||||||
|
if (value instanceof Uint8Array) return new Uint8Array(value);
|
||||||
|
if (value instanceof ArrayBuffer) return value.slice(0);
|
||||||
|
const out: Record<string, unknown> = {};
|
||||||
|
for (const [k, v] of Object.entries(value as Record<string, unknown>)) {
|
||||||
|
out[k] = cloneRowFallback(v);
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 批量深拷贝 */
|
||||||
|
export function cloneRows<T extends Record<string, unknown>>(rows: T[]): T[] {
|
||||||
|
return rows.map((r) => cloneRow(r));
|
||||||
|
}
|
||||||
+23
-1
@@ -208,6 +208,21 @@ export class HybridEngine implements IStorageEngine {
|
|||||||
throw error;
|
throw error;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0(B-1):写入前置校验 —— 委托给内存引擎(与磁盘引擎同 schema)。
|
||||||
|
*
|
||||||
|
* 关键点:**只判定一次**。Hybrid 的 write-through 会把同一批行先写内存再写磁盘,
|
||||||
|
* 两个引擎各自校验会给出同一结论(现在共享同一个 `compileValidator`),
|
||||||
|
* 但由本方法统一前置,可保证多行批量在任何副作用之前整体失败。
|
||||||
|
*/
|
||||||
|
async validatePayload(
|
||||||
|
tableName: string,
|
||||||
|
rows: Record<string, unknown>[],
|
||||||
|
mode: 'insert' | 'update' = 'insert',
|
||||||
|
): Promise<void> {
|
||||||
|
return this.memoryEngine.validatePayload(tableName, rows, mode);
|
||||||
|
}
|
||||||
|
|
||||||
async insert(tableName: string, rows: Record<string, unknown>[]): Promise<string[]> {
|
async insert(tableName: string, rows: Record<string, unknown>[]): Promise<string[]> {
|
||||||
const pks = await this.memoryEngine.insert(tableName, rows);
|
const pks = await this.memoryEngine.insert(tableName, rows);
|
||||||
// write-through: 同步写入磁盘
|
// write-through: 同步写入磁盘
|
||||||
@@ -292,7 +307,14 @@ export class HybridEngine implements IStorageEngine {
|
|||||||
|
|
||||||
async beginTransaction(): Promise<void> {
|
async beginTransaction(): Promise<void> {
|
||||||
await this.memoryEngine.beginTransaction();
|
await this.memoryEngine.beginTransaction();
|
||||||
await this.diskEngine.beginTransaction();
|
// v0.7.4: 磁盘 begin 失败时补偿回滚内存快照 —— 此前内存已 begin、
|
||||||
|
// 磁盘抛错 → 内存快照泄漏(后续所有事务报 TX_ACTIVE)
|
||||||
|
try {
|
||||||
|
await this.diskEngine.beginTransaction();
|
||||||
|
} catch (error) {
|
||||||
|
await this.memoryEngine.rollbackTransaction();
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
async commitTransaction(): Promise<void> {
|
async commitTransaction(): Promise<void> {
|
||||||
|
|||||||
+17
-1
@@ -76,14 +76,30 @@ export const MeSqlark = MetonaSqlark;
|
|||||||
export type { DatabaseConfig, TableSchema, ColumnDef, FieldType, StorageMode, DiskEngine } from './constants';
|
export type { DatabaseConfig, TableSchema, ColumnDef, FieldType, StorageMode, DiskEngine } from './constants';
|
||||||
export type { IStorageEngine } from './engine/interface';
|
export type { IStorageEngine } from './engine/interface';
|
||||||
export type { Statement, SelectStatement, InsertStatement, UpdateStatement, DeleteStatement } from './query/ast';
|
export type { Statement, SelectStatement, InsertStatement, UpdateStatement, DeleteStatement } from './query/ast';
|
||||||
|
// v0.8.0: 补齐公开 API —— 此前 CONTRIBUTING 示例的
|
||||||
|
// `import type { MetonaPlugin } from '@metona-team/metona-sqlark'` 并不成立,
|
||||||
|
// 且 DatabaseError 未导出导致调用方无法 instanceof 判错,只能比 error.code 字符串。
|
||||||
|
export { DatabaseError } from './constants';
|
||||||
|
export type { MetonaPlugin, HookName, WhereCondition, WhereOperator, QueryPlan, OrderBy } from './constants';
|
||||||
|
export { PluginManager } from './plugin/index';
|
||||||
|
export type { HookCallback } from './plugin/index';
|
||||||
|
export { Transaction } from './transaction/index';
|
||||||
|
export { TransactionManager } from './transaction/index';
|
||||||
export { MemoryEngine } from './engine/memory';
|
export { MemoryEngine } from './engine/memory';
|
||||||
export { KVStoreEngine } from './engine/kvstore_engine';
|
export { KVStoreEngine } from './engine/kvstore_engine';
|
||||||
export { AriaEngine } from './engine/aria/index';
|
export { AriaEngine } from './engine/aria/index';
|
||||||
export { HybridEngine } from './hybrid/index';
|
export { HybridEngine } from './hybrid/index';
|
||||||
export { Table } from './table/table';
|
export { Table } from './table/table';
|
||||||
export { parse, parseAll } from './sql/parser';
|
export { parse, parseAll, parseWhereCondition } from './sql/parser';
|
||||||
export { tokenize } from './sql/lexer';
|
export { tokenize } from './sql/lexer';
|
||||||
|
export { bindParameters } from './sql/params';
|
||||||
|
|
||||||
// AriaEngine 类型 & 后端
|
// AriaEngine 类型 & 后端
|
||||||
export type { AriaEngineConfig } from './engine/aria/types';
|
export type { AriaEngineConfig } from './engine/aria/types';
|
||||||
export { OPFSBackend } from './engine/aria/store/opfs_backend';
|
export { OPFSBackend } from './engine/aria/store/opfs_backend';
|
||||||
|
|
||||||
|
// v0.8.0: 迁移工具 —— 此前只在 src/migration/index.ts 定义,主入口未导出,
|
||||||
|
// 于是 package.json 的 "./migration" 子路径**指向的产物里根本没有这个函数**
|
||||||
|
//(`import { migrateFromIndexedDB } from '.../migration'` 会得到 undefined)。
|
||||||
|
export { migrateFromIndexedDB } from './migration/index';
|
||||||
|
export type { MigrationOptions, MigrationResult } from './migration/index';
|
||||||
|
|||||||
Vendored
+63
@@ -0,0 +1,63 @@
|
|||||||
|
/**
|
||||||
|
* metona-sqlark React 集成 —— 公开类型声明(v0.8.0)
|
||||||
|
*
|
||||||
|
* 为什么手写而不是从 `react.ts` 生成:该文件带 `// @ts-nocheck`(hooks 的
|
||||||
|
* 泛型推断需要 React 类型,而 React 是 peer dependency,构建时不一定存在),
|
||||||
|
* 由 rollup-plugin-dts 生成只会得到 `any` —— 对使用者毫无价值。
|
||||||
|
* 这里给出**手写的精确签名**,并随包发布(package.json 的 `./react` 指向它)。
|
||||||
|
*
|
||||||
|
* 注意:`react` 是 peer dependency,本文件不 import 任何 React 类型,
|
||||||
|
* 因此即使使用者的项目里没有装 React 也能通过类型检查(只要不真的调用)。
|
||||||
|
*/
|
||||||
|
import type { MetonaSqlark, DatabaseConfig } from '../constants';
|
||||||
|
|
||||||
|
/** `useQuery` 的返回结构 */
|
||||||
|
export interface UseQueryResult<T = Record<string, unknown>> {
|
||||||
|
/** 查询结果(初次渲染时为空数组) */
|
||||||
|
data: T[];
|
||||||
|
/** 是否正在查询 */
|
||||||
|
loading: boolean;
|
||||||
|
/** 查询错误(成功时为 null) */
|
||||||
|
error: Error | null;
|
||||||
|
/** 重新执行查询 */
|
||||||
|
refresh: () => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** `useTable` 的返回结构 */
|
||||||
|
export interface UseTableResult<T = Record<string, unknown>> {
|
||||||
|
data: T[];
|
||||||
|
loading: boolean;
|
||||||
|
refresh: () => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** `useDatabase` 的返回结构 */
|
||||||
|
export interface UseDatabaseResult {
|
||||||
|
/** 初始化完成的数据库实例;未就绪时为 null */
|
||||||
|
db: MetonaSqlark | null;
|
||||||
|
ready: boolean;
|
||||||
|
error: Error | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 执行 SQL 查询并在结果变化时重渲染。
|
||||||
|
* @param db 已初始化的数据库实例
|
||||||
|
* @param sql SQL 语句
|
||||||
|
* @param deps 依赖数组,变化时重新查询(同 React useEffect 语义)
|
||||||
|
*/
|
||||||
|
export function useQuery<T = Record<string, unknown>>(
|
||||||
|
db: MetonaSqlark,
|
||||||
|
sql: string,
|
||||||
|
deps?: unknown[],
|
||||||
|
): UseQueryResult<T>;
|
||||||
|
|
||||||
|
/** 查询整张表(`SELECT * FROM <table>`),表名会做标识符校验 */
|
||||||
|
export function useTable<T = Record<string, unknown>>(
|
||||||
|
db: MetonaSqlark,
|
||||||
|
tableName: string,
|
||||||
|
): UseTableResult<T>;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 创建并管理数据库实例:组件挂载时 init,卸载时 close。
|
||||||
|
* `config` 变化(按序列化指纹比较)时重建实例。
|
||||||
|
*/
|
||||||
|
export function useDatabase(config: DatabaseConfig): UseDatabaseResult;
|
||||||
Vendored
+56
@@ -0,0 +1,56 @@
|
|||||||
|
/**
|
||||||
|
* metona-sqlark Vue 集成 —— 公开类型声明(v0.8.0)
|
||||||
|
*
|
||||||
|
* 与 `react.d.ts` 同理:`vue.ts` 带 `// @ts-nocheck`,自动生成的声明只有 `any`,
|
||||||
|
* 因此这里手写精确签名并随包发布。`vue` 是 peer dependency,
|
||||||
|
* 本文件用最小结构类型(`Ref`)描述响应式引用,不 import Vue 本身。
|
||||||
|
*/
|
||||||
|
import type { MetonaSqlark, DatabaseConfig } from '../constants';
|
||||||
|
|
||||||
|
/** 最小响应式引用结构(与 Vue 的 `Ref<T>` 兼容,但不依赖 vue 包) */
|
||||||
|
export interface Ref<T> {
|
||||||
|
value: T;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** `useSqlarkQuery` 的返回结构 */
|
||||||
|
export interface UseSqlarkQueryResult<T = Record<string, unknown>> {
|
||||||
|
data: Ref<T[]>;
|
||||||
|
loading: Ref<boolean>;
|
||||||
|
error: Ref<Error | null>;
|
||||||
|
refresh: () => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** `useSqlarkTable` 的返回结构 */
|
||||||
|
export interface UseSqlarkTableResult<T = Record<string, unknown>> {
|
||||||
|
data: Ref<T[]>;
|
||||||
|
loading: Ref<boolean>;
|
||||||
|
refresh: () => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** `useSqlarkDatabase` 的返回结构 */
|
||||||
|
export interface UseSqlarkDatabaseResult {
|
||||||
|
db: Ref<MetonaSqlark | null>;
|
||||||
|
ready: Ref<boolean>;
|
||||||
|
error: Ref<Error | null>;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 执行 SQL 查询并保持结果为响应式引用。
|
||||||
|
* @param db 已初始化的数据库实例
|
||||||
|
* @param sql SQL 语句
|
||||||
|
* @param deps 依赖的响应式引用数组,变化时重新查询
|
||||||
|
*/
|
||||||
|
export function useSqlarkQuery<T = Record<string, unknown>>(
|
||||||
|
db: MetonaSqlark,
|
||||||
|
sql: string,
|
||||||
|
deps?: Ref<unknown>[],
|
||||||
|
): UseSqlarkQueryResult<T>;
|
||||||
|
|
||||||
|
/** 查询整张表(`SELECT * FROM <table>`),表名会做标识符校验 */
|
||||||
|
export function useSqlarkTable<T = Record<string, unknown>>(
|
||||||
|
db: MetonaSqlark,
|
||||||
|
tableName: string,
|
||||||
|
): UseSqlarkTableResult<T>;
|
||||||
|
|
||||||
|
/** 创建并管理数据库实例:`onMounted` 时 init */
|
||||||
|
export function useSqlarkDatabase(config: DatabaseConfig): UseSqlarkDatabaseResult;
|
||||||
@@ -158,6 +158,17 @@ export interface SelectUnionStatement {
|
|||||||
right: SelectStatement | SelectUnionStatement;
|
right: SelectStatement | SelectUnionStatement;
|
||||||
/** UNION ALL 不去重 */
|
/** UNION ALL 不去重 */
|
||||||
all?: boolean;
|
all?: boolean;
|
||||||
|
/**
|
||||||
|
* v0.8.0(A26):复合查询**整体**的 ORDER BY / LIMIT / OFFSET。
|
||||||
|
*
|
||||||
|
* SQL 标准里这三者作用于整个 UNION 结果,而不是最后一个 SELECT。
|
||||||
|
* 此前 AST 没有这三个字段,parser 把它们挂在了 UNION 右侧的 SELECT 上 ——
|
||||||
|
* 于是 `A UNION B ORDER BY id DESC` 只对 B 排序、`... LIMIT 3` 只截断 B
|
||||||
|
* (实测 `SELECT id FROM t UNION SELECT id FROM t LIMIT 3` 返回 4 行)。
|
||||||
|
*/
|
||||||
|
orderBy?: OrderBy[];
|
||||||
|
limit?: number;
|
||||||
|
offset?: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|||||||
+70
-52
@@ -2,11 +2,26 @@
|
|||||||
* metona-sqlark Query Builder — 链式查询构建器
|
* metona-sqlark Query Builder — 链式查询构建器
|
||||||
* @module query/builder
|
* @module query/builder
|
||||||
*
|
*
|
||||||
* 链式调用 → 构建 AST → 执行引擎操作。
|
* ============================================================================
|
||||||
* 支持 JOIN(需要 Executor)。
|
* v0.8.0(B-3):**只产出 AST,执行一律交给 Executor**
|
||||||
|
* ============================================================================
|
||||||
|
* 修复前每个 builder 的 `execute()` 都有自己的执行动作:
|
||||||
|
* - `SelectQueryBuilder`:**无 JOIN 时直接调 `engine.find`**,只有 JOIN 才走
|
||||||
|
* executor;
|
||||||
|
* - `UpdateQueryBuilder` / `DeleteQueryBuilder`:直接调 `engine.update/delete`。
|
||||||
|
*
|
||||||
|
* 于是"同一条语义"在 builder 路径与 SQL 路径上走两条管线,规则各写一份:
|
||||||
|
* - SELECT 的常量列/别名/CASE 投影在 builder 路径不存在
|
||||||
|
* (`db.table('t').select(['t.n'])` 的输出键、`SELECT 1` 之类行为不同);
|
||||||
|
* - `maxRowsPerQuery`、`distinct`、`groupBy`/`having`、`fromSubquery` 等阶段
|
||||||
|
* 在直通路径上完全不经过;
|
||||||
|
* - 未解析的 `$subquery`/`$col` 在直通路径上无人解析 → 引擎判 UNKNOWN →
|
||||||
|
* **静默 0 行**(这正是 v0.7.4 要为此加"显式拒绝"防御的原因)。
|
||||||
|
*
|
||||||
|
* 现在 builder 只负责"拼 AST",执行统一走 `executor.execute(ast)`;
|
||||||
|
* 需要直通时由 executor 内部判断(它本来就是唯一知道"能不能下推"的地方)。
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import type { IStorageEngine } from '../engine/interface';
|
|
||||||
import type { WhereCondition, OrderBy, SortDirection } from '../constants';
|
import type { WhereCondition, OrderBy, SortDirection } from '../constants';
|
||||||
import type { SelectStatement, UpdateStatement, DeleteStatement, JoinType, JoinClause } from './ast';
|
import type { SelectStatement, UpdateStatement, DeleteStatement, JoinType, JoinClause } from './ast';
|
||||||
import { QueryExecutor } from './executor';
|
import { QueryExecutor } from './executor';
|
||||||
@@ -22,13 +37,12 @@ export class SelectQueryBuilder {
|
|||||||
private _offset?: number;
|
private _offset?: number;
|
||||||
private _joins: JoinClause[] = [];
|
private _joins: JoinClause[] = [];
|
||||||
private _alias?: string;
|
private _alias?: string;
|
||||||
private _executor?: QueryExecutor;
|
private _executor: QueryExecutor;
|
||||||
|
|
||||||
constructor(
|
constructor(
|
||||||
private engine: IStorageEngine,
|
|
||||||
private tableName: string,
|
private tableName: string,
|
||||||
|
executor: QueryExecutor,
|
||||||
private _columns: string[] = ['*'],
|
private _columns: string[] = ['*'],
|
||||||
executor?: QueryExecutor,
|
|
||||||
) {
|
) {
|
||||||
this._executor = executor;
|
this._executor = executor;
|
||||||
}
|
}
|
||||||
@@ -93,23 +107,15 @@ export class SelectQueryBuilder {
|
|||||||
return this;
|
return this;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 执行查询 */
|
/**
|
||||||
|
* 执行查询 —— 拼出 AST 后交给 Executor(**唯一**执行管线)。
|
||||||
|
*
|
||||||
|
* 注意不要再为"无 JOIN"加一条 `engine.find` 快路:那条路会绕过
|
||||||
|
* 投影/列校验/LIMIT 下推判定/maxRowsPerQuery,从而与 `db.query()` 给出不同结果
|
||||||
|
* (B-3 修复前的实际状态)。executor 自己会在安全时下推到引擎,不需要 builder 代劳。
|
||||||
|
*/
|
||||||
async execute(): Promise<Record<string, unknown>[]> {
|
async execute(): Promise<Record<string, unknown>[]> {
|
||||||
// 有 JOIN → 通过 Executor 执行
|
return this._executor.execute(this.toAST()) as Promise<Record<string, unknown>[]>;
|
||||||
if (this._joins.length > 0 && this._executor) {
|
|
||||||
const ast = this.toAST();
|
|
||||||
return this._executor.execute(ast) as Promise<Record<string, unknown>[]>;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 无 JOIN → 直接调用引擎
|
|
||||||
return this.engine.find(this.tableName, {
|
|
||||||
table: this.tableName,
|
|
||||||
columns: this._columns,
|
|
||||||
where: this._where,
|
|
||||||
orderBy: this._orderBy.length > 0 ? this._orderBy : undefined,
|
|
||||||
limit: this._limit,
|
|
||||||
offset: this._offset,
|
|
||||||
});
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 获取 AST */
|
/** 获取 AST */
|
||||||
@@ -134,32 +140,47 @@ export class SelectQueryBuilder {
|
|||||||
|
|
||||||
export class UpdateQueryBuilder {
|
export class UpdateQueryBuilder {
|
||||||
private _where: WhereCondition = {};
|
private _where: WhereCondition = {};
|
||||||
private onWrite?: (table: string) => void;
|
|
||||||
/** v0.5.1: CRUD hooks 触发回调 */
|
|
||||||
private onHooks?: (hook: import('../constants').HookName, args: unknown[]) => Promise<void>;
|
|
||||||
|
|
||||||
constructor(
|
constructor(
|
||||||
private engine: IStorageEngine,
|
|
||||||
private tableName: string,
|
private tableName: string,
|
||||||
private _updates: Record<string, unknown>,
|
private _updates: Record<string, unknown>,
|
||||||
onWrite?: (table: string) => void,
|
private executor: QueryExecutor,
|
||||||
onHooks?: (hook: import('../constants').HookName, args: unknown[]) => Promise<void>,
|
/**
|
||||||
) {
|
* TABLE API 的生命周期回调(`beforeUpdate` / `afterUpdate` / onWrite 广播)。
|
||||||
this.onWrite = onWrite;
|
*
|
||||||
this.onHooks = onHooks;
|
* 为什么由 `Table` 注入而不是 builder 自己触发:builder 的职责是**产出 AST**,
|
||||||
}
|
* 它不该知道钩子/广播的存在(否则又会像修复前那样"builder 顺带把写入也做了",
|
||||||
|
* 从而绕过 Executor)。钩子被包裹在**唯一管线**之外,
|
||||||
|
* 顺序与修复前完全一致:before → executor → onWrite → after。
|
||||||
|
*
|
||||||
|
* 回调接收**实际执行的语句**(含 builder 上累积的 where),
|
||||||
|
* 而不是构造 builder 时的空 where —— 后者会让 `beforeUpdate` 的
|
||||||
|
* `query.where` 永远是 `{}`(钩子拿不到过滤条件,等于信息缺失)。
|
||||||
|
*/
|
||||||
|
private hooks?: {
|
||||||
|
before?: (stmt: UpdateStatement) => Promise<void>;
|
||||||
|
after?: (count: number, stmt: UpdateStatement) => Promise<void>;
|
||||||
|
},
|
||||||
|
) {}
|
||||||
|
|
||||||
where(condition: WhereCondition): this {
|
where(condition: WhereCondition): this {
|
||||||
this._where = { ...this._where, ...condition };
|
this._where = { ...this._where, ...condition };
|
||||||
return this;
|
return this;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 执行更新 —— 走 Executor(唯一的写管线)。
|
||||||
|
*
|
||||||
|
* 修复前这里直接调 `engine.update`:`$subquery` / `$col` / `$exists` 无人解析,
|
||||||
|
* 引擎层 matchWhere 判 UNKNOWN → **静默影响 0 行**(返回 0 且无报错)。
|
||||||
|
* 引擎层为此加过"检测未解析标记就抛 NOT_SUPPORTED"的防御 —— 那是把
|
||||||
|
* "管线缺失"暴露成用户错误;正确做法是把请求送进唯一管线。
|
||||||
|
*/
|
||||||
async execute(): Promise<number> {
|
async execute(): Promise<number> {
|
||||||
const query = { table: this.tableName, where: this._where };
|
const stmt = this.toAST();
|
||||||
await this.onHooks?.('beforeUpdate', [query, this._updates]);
|
await this.hooks?.before?.(stmt);
|
||||||
const count = await this.engine.update(this.tableName, query, this._updates);
|
const count = await this.executor.execute(stmt) as number;
|
||||||
this.onWrite?.(this.tableName);
|
await this.hooks?.after?.(count, stmt);
|
||||||
await this.onHooks?.('afterUpdate', [query, this._updates, count]);
|
|
||||||
return count;
|
return count;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -174,31 +195,28 @@ export class UpdateQueryBuilder {
|
|||||||
|
|
||||||
export class DeleteQueryBuilder {
|
export class DeleteQueryBuilder {
|
||||||
private _where: WhereCondition = {};
|
private _where: WhereCondition = {};
|
||||||
private onWrite?: (table: string) => void;
|
|
||||||
/** v0.5.1: CRUD hooks 触发回调 */
|
|
||||||
private onHooks?: (hook: import('../constants').HookName, args: unknown[]) => Promise<void>;
|
|
||||||
|
|
||||||
constructor(
|
constructor(
|
||||||
private engine: IStorageEngine,
|
|
||||||
private tableName: string,
|
private tableName: string,
|
||||||
onWrite?: (table: string) => void,
|
private executor: QueryExecutor,
|
||||||
onHooks?: (hook: import('../constants').HookName, args: unknown[]) => Promise<void>,
|
/** TABLE API 生命周期回调,语义见 `UpdateQueryBuilder` 的说明 */
|
||||||
) {
|
private hooks?: {
|
||||||
this.onWrite = onWrite;
|
before?: (stmt: DeleteStatement) => Promise<void>;
|
||||||
this.onHooks = onHooks;
|
after?: (count: number, stmt: DeleteStatement) => Promise<void>;
|
||||||
}
|
},
|
||||||
|
) {}
|
||||||
|
|
||||||
where(condition: WhereCondition): this {
|
where(condition: WhereCondition): this {
|
||||||
this._where = { ...this._where, ...condition };
|
this._where = { ...this._where, ...condition };
|
||||||
return this;
|
return this;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** 执行删除 —— 走 Executor(同 UpdateQueryBuilder.execute 的理由) */
|
||||||
async execute(): Promise<number> {
|
async execute(): Promise<number> {
|
||||||
const query = { table: this.tableName, where: this._where };
|
const stmt = this.toAST();
|
||||||
await this.onHooks?.('beforeDelete', [query]);
|
await this.hooks?.before?.(stmt);
|
||||||
const count = await this.engine.delete(this.tableName, query);
|
const count = await this.executor.execute(stmt) as number;
|
||||||
this.onWrite?.(this.tableName);
|
await this.hooks?.after?.(count, stmt);
|
||||||
await this.onHooks?.('afterDelete', [query, count]);
|
|
||||||
return count;
|
return count;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,95 @@
|
|||||||
|
/**
|
||||||
|
* metona-sqlark 列引用取值 —— WHERE / 投影 / 聚合 / 表达式**共用**的唯一实现
|
||||||
|
* @module query/column-value
|
||||||
|
*
|
||||||
|
* ============================================================================
|
||||||
|
* 为什么必须只有一个实现(v0.8.0 审计根因 1)
|
||||||
|
* ============================================================================
|
||||||
|
* "从一行里按名字取一列"曾是四处各写一份的实现,规则各不相同:
|
||||||
|
*
|
||||||
|
* | 位置 | 别名前缀 | 后缀回退 | 未知列 |
|
||||||
|
* |---|---|---|---|
|
||||||
|
* | `projectColumns`(where-matcher) | 否 | 唯一后缀 | 静默丢键 |
|
||||||
|
* | `resolveAliasSource`(executor) | **否** | 否 | 静默 undefined |
|
||||||
|
* | 聚合参数(executor) | 否 | 否 | **静默计 0**(A25) |
|
||||||
|
* | `matchWhere` 的 `resolveField` | 否 | 唯一后缀(多个则 UNRESOLVED) | UNRESOLVED |
|
||||||
|
*
|
||||||
|
* 于是 `COUNT(t.n)` 返回 0(A25)、`SELECT d.id FROM (...) AS d` 返回空集(A36)、
|
||||||
|
* CASE 的 THEN 分支引用列名时行为随调用点变化(B-4)。
|
||||||
|
*
|
||||||
|
* 现在四处都调用本模块的 `resolveColumnValue`:
|
||||||
|
* - JOIN 行的键是 `alias.col`,单表行的键是 `col` —— 两种形态都要支持;
|
||||||
|
* - 取不到值时**由调用方**决定是抛错还是返回 UNRESOLVED(`strict` 选项),
|
||||||
|
* 因为"WHERE 里的未解析引用"与"表达式里的未解析引用"需要不同的上层处理。
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { DatabaseError } from '../constants';
|
||||||
|
import { UNRESOLVED } from './sql-compare';
|
||||||
|
|
||||||
|
/** 取值选项 */
|
||||||
|
export interface ResolveOptions {
|
||||||
|
/**
|
||||||
|
* 取不到值时是否抛 `COLUMN_NOT_FOUND`。
|
||||||
|
* - `true`:用于 GROUP BY / 聚合参数 / 投影等"这里必须有一列"的场景;
|
||||||
|
* - `false`:返回 `UNRESOLVED` 哨兵,由调用方决定(WHERE 需要据此判 UNKNOWN,
|
||||||
|
* 表达式需要据此抛错,而"行里确实没有该键"与"该列值为 NULL"必须区分)。
|
||||||
|
*/
|
||||||
|
strict: boolean;
|
||||||
|
/** 错误消息中的位置描述(如 'GROUP BY' / 'aggregate COUNT(n)' / 'SELECT list') */
|
||||||
|
context: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 从行里取一个列引用。
|
||||||
|
*
|
||||||
|
* 解析顺序(与 ORDER BY 的 `stripAlias` 语义一致,保证同一引用在各子句里等价):
|
||||||
|
* 1. 精确命中(行键与引用完全一致,含 `alias.col` 形态);
|
||||||
|
* 2. 剥离别名前缀(`t.n` → `n`);
|
||||||
|
* 3. 唯一后缀匹配(行键是 `t.n` 而引用写作 `n`);
|
||||||
|
* 4. 以上都不中 → 依 `strict` 抛错或返回 `UNRESOLVED`。
|
||||||
|
*
|
||||||
|
* 多个后缀命中视为**歧义**(JOIN 里两表同名列),`strict` 下抛错 ——
|
||||||
|
* 静默取第一个正是"结果取决于表顺序"这类难查问题的来源。
|
||||||
|
*/
|
||||||
|
export function resolveColumnValue(
|
||||||
|
row: Record<string, unknown>,
|
||||||
|
reference: string,
|
||||||
|
opts: ResolveOptions,
|
||||||
|
): unknown {
|
||||||
|
let text = reference.trim();
|
||||||
|
// v0.8.0:分隔标识符(`"col"`)在比较/取值时要脱去引号 ——
|
||||||
|
// 保留引号是为了让投影阶段能区分"名为 1 的列"与"常量 1"(见 parser 说明),
|
||||||
|
// 但比较/取值必须用真实列名。`""` 是引号自身的转义。
|
||||||
|
if (text.length >= 2 && text.startsWith('"') && text.endsWith('"')) {
|
||||||
|
text = text.slice(1, -1).replace(/""/g, '"');
|
||||||
|
}
|
||||||
|
if (text in row) return row[text];
|
||||||
|
|
||||||
|
// 别名前缀(`t.n` → `n`):JOIN 行用 `alias.col` 作键,单表路径的键不带前缀
|
||||||
|
if (text.includes('.')) {
|
||||||
|
const bare = text.split('.').pop()!;
|
||||||
|
if (bare in row) return row[bare];
|
||||||
|
}
|
||||||
|
|
||||||
|
// 唯一后缀匹配:行键 `t.n` 而引用写作 `n`
|
||||||
|
let found: unknown;
|
||||||
|
let hits = 0;
|
||||||
|
for (const key of Object.keys(row)) {
|
||||||
|
if (key.endsWith(`.${text}`)) {
|
||||||
|
found = row[key];
|
||||||
|
hits += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (hits === 1) return found;
|
||||||
|
|
||||||
|
if (opts.strict) {
|
||||||
|
throw new DatabaseError(
|
||||||
|
hits > 1
|
||||||
|
? `Ambiguous column "${text}" in ${opts.context}: present in multiple tables`
|
||||||
|
: `Unknown column "${text}" in ${opts.context}`,
|
||||||
|
'COLUMN_NOT_FOUND',
|
||||||
|
{ column: text },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return UNRESOLVED;
|
||||||
|
}
|
||||||
+1690
-209
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,435 @@
|
|||||||
|
/**
|
||||||
|
* metona-sqlark 表达式解析与求值 —— CASE WHEN 的结构化实现(v0.8.0 / B-4)
|
||||||
|
* @module query/expression
|
||||||
|
*
|
||||||
|
* ============================================================================
|
||||||
|
* 为什么必须换掉原实现(v0.8.0 审计根因 3:字符串化 AST 表达式列)
|
||||||
|
* ============================================================================
|
||||||
|
* 原 `parseCaseExpression` 用**正则**在原始 SQL 文本上切分 WHEN/THEN/ELSE:
|
||||||
|
*
|
||||||
|
* ```ts
|
||||||
|
* /WHEN\s+([\s\S]*?)\s+THEN\s+([\s\S]*?)(?=\s+WHEN\s+|\s+ELSE\s+|\s*$)/gi
|
||||||
|
* ```
|
||||||
|
*
|
||||||
|
* 它不认字符串字面量、不认嵌套结构,于是实测出三类错误结果:
|
||||||
|
*
|
||||||
|
* | 输入 | 实测(修复前) | 应有 |
|
||||||
|
* |---|---|---|
|
||||||
|
* | `CASE WHEN n>10 THEN CASE WHEN n>25 THEN 'huge' ELSE 'big' END ELSE 'small' END` | `"big' END ELSE 'small"` / `null` | `huge`/`big`/`small` |
|
||||||
|
* | `CASE WHEN s='WHEN' THEN 'hit' ELSE 'miss' END` | 依赖切分点,可能错 | `miss/hit/miss` |
|
||||||
|
*
|
||||||
|
* 更严重的是**静默错值**:条件解析失败(如引用不存在的列)时 `cond = null`,
|
||||||
|
* 求值直接跳过该分支 —— 整列变成 ELSE 值,没有任何提示;而同一列名出现在
|
||||||
|
* WHERE 里会正常报错。同一语义两套行为。
|
||||||
|
*
|
||||||
|
* ============================================================================
|
||||||
|
* 本模块的做法
|
||||||
|
* ============================================================================
|
||||||
|
* 1. **复用 `sql/lexer` 的 token 流**(带 `position`),不另写一套词法规则 ——
|
||||||
|
* 字符串里的 `WHEN`/`ELSE`、转义引号、注释都由它正确处理,
|
||||||
|
* 因此"切分点"不再可能落在字面量内部。
|
||||||
|
* 2. **递归下降**解析 CASE(天然支持嵌套),并把每个片段按**源码位置切片**,
|
||||||
|
* 交给既有的 `parseWhereCondition` / 字面量解析器处理 ——
|
||||||
|
* 条件语义与 WHERE 完全同源,不再各写一份。
|
||||||
|
* 3. **解析失败即报错**(`PARSE_ERROR` / `COLUMN_NOT_FOUND`),不静默降级为
|
||||||
|
* 某一个分支的值。
|
||||||
|
* 4. 解析结果**缓存**:`parseCaseExpression` 是纯函数,可安全记忆化;
|
||||||
|
* 聚合与逐行投影会对同一表达式反复求值(N 行 × M 次),缓存把解析开销
|
||||||
|
* 从 O(行数) 降到 O(1)。
|
||||||
|
*
|
||||||
|
* 注意:本模块只负责 **CASE** 表达式。普通列引用/字面量/聚合由
|
||||||
|
* `query/executor` 的 `resolveColumnValue` 与 `parseAggregateExpression`
|
||||||
|
* 处理(B-4 的另一半已由 A25 统一)。
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { tokenize } from '../sql/lexer';
|
||||||
|
import { TokenType } from '../sql/tokens';
|
||||||
|
import { parseWhereCondition } from '../sql/parser';
|
||||||
|
import type { WhereCondition } from '../constants';
|
||||||
|
import { DatabaseError } from '../constants';
|
||||||
|
import { matchWhere } from './where-matcher';
|
||||||
|
import { UNRESOLVED, isUnresolved } from './sql-compare';
|
||||||
|
import { resolveColumnValue } from './column-value';
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// 结构化 CASE 表达式
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/** 单个 `WHEN <条件> THEN <值>` 分支 */
|
||||||
|
export interface CaseWhenBranch {
|
||||||
|
/** 条件的源码片段(已 trim) */
|
||||||
|
conditionText: string;
|
||||||
|
/** 值的源码片段(已 trim) */
|
||||||
|
resultText: string;
|
||||||
|
/**
|
||||||
|
* 条件解析结果。解析在**解析期**完成(失败即抛错),因此这里在成功构造的
|
||||||
|
* 表达式上恒为已解析对象 —— 不再有"cond = null 表示解析失败"的隐式约定。
|
||||||
|
*/
|
||||||
|
condition: WhereCondition;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 结构化 CASE 表达式 */
|
||||||
|
export interface CaseExpression {
|
||||||
|
/** 分支列表(按出现顺序) */
|
||||||
|
branches: CaseWhenBranch[];
|
||||||
|
/** `ELSE <值>` 的源码片段;未写 ELSE 时为 null(求值为 NULL) */
|
||||||
|
elseText: string | null;
|
||||||
|
/** `... END AS alias` 的别名 */
|
||||||
|
alias: string | null;
|
||||||
|
/** 表达式原文(用于输出列名与错误信息) */
|
||||||
|
source: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// 解析
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/** 解析缓存:同一段文本只解析一次(纯函数,可安全记忆化) */
|
||||||
|
const caseParseCache = new Map<string, CaseExpression>();
|
||||||
|
/** 缓存上限(防御性:避免长生命周期进程里无界增长) */
|
||||||
|
const CASE_CACHE_LIMIT = 512;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 解析 CASE WHEN 表达式文本。
|
||||||
|
*
|
||||||
|
* @param expr 形如 `CASE WHEN a > 1 THEN 'x' ELSE 'y' END AS band` 的片段
|
||||||
|
* @returns 结构化表达式;**不是** CASE 表达式时返回 null(调用方据此走其它分支)
|
||||||
|
* @throws DatabaseError 结构不完整(缺 THEN/END)或条件无法解析
|
||||||
|
*/
|
||||||
|
export function parseCaseExpression(expr: string): CaseExpression | null {
|
||||||
|
const text = expr.trim();
|
||||||
|
if (!/^\s*CASE\b/i.test(text)) return null;
|
||||||
|
|
||||||
|
const cached = caseParseCache.get(text);
|
||||||
|
if (cached) return cached;
|
||||||
|
|
||||||
|
const parsed = parseCaseExpressionUncached(text);
|
||||||
|
// 先清理再写入:条目数达到上限时整表清空,避免无界增长。
|
||||||
|
// 用 LRU 会引入额外状态;解析本身是纯函数且调用点集中在少数表达式上,
|
||||||
|
// "清空重建"足够且没有正确性风险。
|
||||||
|
if (caseParseCache.size >= CASE_CACHE_LIMIT) caseParseCache.clear();
|
||||||
|
caseParseCache.set(text, parsed);
|
||||||
|
return parsed;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 清空解析缓存(测试用;schema 变化不影响 CASE 结构,故生产无需清理) */
|
||||||
|
export function clearCaseParseCache(): void {
|
||||||
|
caseParseCache.clear();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 真正的解析实现。
|
||||||
|
*
|
||||||
|
* 用 `tokenize` 得到带位置的 token 流后按源码切片 —— 这样"条件/值片段"与
|
||||||
|
* 原始 SQL 逐字符一致(含引号与转义),可以安全地交给 `parseWhereCondition`。
|
||||||
|
*/
|
||||||
|
function parseCaseExpressionUncached(text: string): CaseExpression {
|
||||||
|
const tokens = tokenize(text);
|
||||||
|
let i = 0;
|
||||||
|
|
||||||
|
const fail = (message: string, position: number): never => {
|
||||||
|
throw new DatabaseError(`${message} (at offset ${position} in "${text}")`, 'PARSE_ERROR');
|
||||||
|
};
|
||||||
|
|
||||||
|
// CASE
|
||||||
|
if (tokens[i]?.type !== TokenType.CASE) fail('Expected CASE', tokens[i]?.position ?? 0);
|
||||||
|
i++;
|
||||||
|
|
||||||
|
const branches: CaseWhenBranch[] = [];
|
||||||
|
let elseText: string | null = null;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 按源码位置取片段。
|
||||||
|
*
|
||||||
|
* 边界必须用"**下一个 token 的起始位置**"作为右开区间:
|
||||||
|
* STRING token 的 `position` 指向**引号之内**(lexer 的 readString 里
|
||||||
|
* `start = position + 1`),用 `position + value.length` 之类算术会把
|
||||||
|
* 结尾引号切掉(实测:`'small'` 被切成 `"small'"`,CASE 全部报
|
||||||
|
* NOT_SUPPORTED —— 这正是本函数必须存在的理由)。
|
||||||
|
* 同理 NUMBER/IDENTIFIER 的 position 也由各自 reader 回推,语义不完全统一;
|
||||||
|
* 只有"下一个 token 的起点"是对所有 token 类型都成立的边界。
|
||||||
|
*/
|
||||||
|
const slice = (startIndex: number, endIndexExclusive: number): string => {
|
||||||
|
const startTok = tokens[startIndex];
|
||||||
|
if (!startTok) return '';
|
||||||
|
const endTok = tokens[endIndexExclusive];
|
||||||
|
const endPos = endTok ? endTok.position : text.length;
|
||||||
|
return text.slice(startTok.position, endPos).trim();
|
||||||
|
};
|
||||||
|
|
||||||
|
while (i < tokens.length) {
|
||||||
|
const tok = tokens[i];
|
||||||
|
|
||||||
|
if (tok.type === TokenType.WHEN) {
|
||||||
|
const condStart = i + 1;
|
||||||
|
// 找与之配对的 THEN:跳过嵌套的括号(CASE 内的子查询/括号表达式)
|
||||||
|
let depth = 0;
|
||||||
|
let thenIndex = -1;
|
||||||
|
for (let j = condStart; j < tokens.length; j++) {
|
||||||
|
const t = tokens[j];
|
||||||
|
if (t.type === TokenType.LPAREN) depth++;
|
||||||
|
else if (t.type === TokenType.RPAREN) depth--;
|
||||||
|
else if (depth === 0 && t.type === TokenType.THEN) { thenIndex = j; break; }
|
||||||
|
else if (depth === 0 && (t.type === TokenType.ELSE || t.type === TokenType.END)) break;
|
||||||
|
}
|
||||||
|
if (thenIndex < 0) fail('CASE WHEN without matching THEN', tok.position);
|
||||||
|
|
||||||
|
const valueStart = thenIndex + 1;
|
||||||
|
// 值的结束点:下一个同级 WHEN / ELSE / END
|
||||||
|
let depth2 = 0;
|
||||||
|
let valueEnd = tokens.length;
|
||||||
|
for (let j = valueStart; j < tokens.length; j++) {
|
||||||
|
const t = tokens[j];
|
||||||
|
if (t.type === TokenType.LPAREN) depth2++;
|
||||||
|
else if (t.type === TokenType.RPAREN) depth2--;
|
||||||
|
else if (depth2 === 0 && (t.type === TokenType.WHEN || t.type === TokenType.ELSE || t.type === TokenType.END)) {
|
||||||
|
valueEnd = j;
|
||||||
|
break;
|
||||||
|
} else if (depth2 === 0 && t.type === TokenType.CASE) {
|
||||||
|
// 嵌套 CASE 作为一个整体:跳到与它配对的 END
|
||||||
|
let nestedDepth = 1;
|
||||||
|
for (let k = j + 1; k < tokens.length; k++) {
|
||||||
|
if (tokens[k].type === TokenType.CASE) nestedDepth++;
|
||||||
|
else if (tokens[k].type === TokenType.END) {
|
||||||
|
nestedDepth--;
|
||||||
|
if (nestedDepth === 0) { j = k; break; }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const conditionText = slice(condStart, thenIndex);
|
||||||
|
const resultText = slice(valueStart, valueEnd);
|
||||||
|
if (!conditionText) fail('CASE WHEN has an empty condition', tok.position);
|
||||||
|
if (!resultText) fail('CASE THEN has an empty result', tokens[valueStart]?.position ?? tok.position);
|
||||||
|
|
||||||
|
// 条件在**解析期**交给与 WHERE 完全相同的解析器 —— 语义同源,
|
||||||
|
// 且"引用不存在的列"之类问题按 WHERE 的口径处理(不再静默跳过分支)。
|
||||||
|
let condition: WhereCondition;
|
||||||
|
try {
|
||||||
|
condition = parseWhereCondition(conditionText);
|
||||||
|
} catch (error) {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Invalid CASE WHEN condition "${conditionText}": ${(error as Error).message}`,
|
||||||
|
'PARSE_ERROR',
|
||||||
|
{ condition: conditionText },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
branches.push({ conditionText, resultText, condition });
|
||||||
|
i = valueEnd;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (tok.type === TokenType.ELSE) {
|
||||||
|
const elseStart = i + 1;
|
||||||
|
let depth = 0;
|
||||||
|
let elseEnd = tokens.length;
|
||||||
|
for (let j = elseStart; j < tokens.length; j++) {
|
||||||
|
const t = tokens[j];
|
||||||
|
if (t.type === TokenType.LPAREN) depth++;
|
||||||
|
else if (t.type === TokenType.RPAREN) depth--;
|
||||||
|
else if (depth === 0 && t.type === TokenType.END) { elseEnd = j; break; }
|
||||||
|
else if (depth === 0 && t.type === TokenType.CASE) {
|
||||||
|
let nestedDepth = 1;
|
||||||
|
for (let k = j + 1; k < tokens.length; k++) {
|
||||||
|
if (tokens[k].type === TokenType.CASE) nestedDepth++;
|
||||||
|
else if (tokens[k].type === TokenType.END) {
|
||||||
|
nestedDepth--;
|
||||||
|
if (nestedDepth === 0) { j = k; break; }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
elseText = slice(elseStart, elseEnd);
|
||||||
|
if (!elseText) fail('CASE ELSE has an empty result', tok.position);
|
||||||
|
i = elseEnd;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (tok.type === TokenType.END) {
|
||||||
|
i++;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
fail(`Unexpected token "${tok.value}" in CASE`, tok.position);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (branches.length === 0) {
|
||||||
|
throw new DatabaseError(`CASE expression has no WHEN branch: "${text}"`, 'PARSE_ERROR');
|
||||||
|
}
|
||||||
|
|
||||||
|
// 可选别名:END AS alias / END alias
|
||||||
|
let alias: string | null = null;
|
||||||
|
const rest = tokens.slice(i).filter((t) => t.type !== TokenType.EOF && t.type !== TokenType.SEMICOLON);
|
||||||
|
if (rest.length > 0) {
|
||||||
|
const first = rest[0];
|
||||||
|
if (first.type === TokenType.AS) {
|
||||||
|
const aliasTok = rest[1];
|
||||||
|
if (aliasTok && (aliasTok.type === TokenType.IDENTIFIER || aliasTok.type === TokenType.QUOTED_IDENTIFIER)) {
|
||||||
|
alias = aliasTok.value;
|
||||||
|
}
|
||||||
|
} else if (first.type === TokenType.IDENTIFIER || first.type === TokenType.QUOTED_IDENTIFIER) {
|
||||||
|
alias = first.value;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return { branches, elseText, alias, source: text };
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// 求值
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 对一行求值 CASE 表达式。
|
||||||
|
*
|
||||||
|
* 求值顺序即声明顺序:第一个条件为 TRUE 的分支胜出;
|
||||||
|
* 无分支命中时取 ELSE(未写 ELSE 则为 NULL)。
|
||||||
|
*
|
||||||
|
* 条件用 `matchWhere`(三值逻辑)判定:UNKNOWN **不**算命中
|
||||||
|
* (与 WHERE 只保留 TRUE 的语义一致)。
|
||||||
|
*/
|
||||||
|
export function evaluateCase(
|
||||||
|
expr: CaseExpression,
|
||||||
|
row: Record<string, unknown>,
|
||||||
|
): unknown {
|
||||||
|
for (const branch of expr.branches) {
|
||||||
|
if (matchWhere(row, branch.condition, { $col: true })) {
|
||||||
|
return evaluateExpressionValue(branch.resultText, row);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return expr.elseText !== null ? evaluateExpressionValue(expr.elseText, row) : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 校验 CASE 条件里引用的列在行源中存在(v0.8.0 / B-4)。
|
||||||
|
*
|
||||||
|
* 为什么必须有:条件用 `matchWhere` 求值时,**引用不存在的列**只会得到
|
||||||
|
* UNKNOWN(三值逻辑的正确行为 —— 引擎层拿不到 schema),于是该分支永不命中,
|
||||||
|
* 整列静默变成 ELSE 值。实测修复前:
|
||||||
|
* `CASE WHEN nope > 1 THEN 'x' ELSE 'y' END` → 每行都是 'y',无任何报错;
|
||||||
|
* 而同一个 `nope` 写在 WHERE 里会正常抛 COLUMN_NOT_FOUND。
|
||||||
|
* 同一语义两套行为,且失败方向是"静默错值"。
|
||||||
|
*
|
||||||
|
* 与 `assertWhereColumnsExist` 的关系:那条路径校验的是 SQL 的 WHERE 子句,
|
||||||
|
* 它拿得到表名与别名;CASE 出现在 SELECT/GROUP BY/HAVING 里,调用点更分散,
|
||||||
|
* 因此这里做**独立的、可复用的**校验,由调用方在有 schema 时调用。
|
||||||
|
*
|
||||||
|
* @param available 该作用域内可见的列名集合(含 `alias.col` 形态;JOIN 时是两表并集)
|
||||||
|
*/
|
||||||
|
export function assertCaseColumnsExist(
|
||||||
|
expr: CaseExpression,
|
||||||
|
available: ReadonlySet<string>,
|
||||||
|
context: string,
|
||||||
|
): void {
|
||||||
|
const missing: string[] = [];
|
||||||
|
const checkRef = (ref: string): void => {
|
||||||
|
const text = ref.trim();
|
||||||
|
if (!text) return;
|
||||||
|
if (available.has(text)) return;
|
||||||
|
const bare = text.includes('.') ? text.split('.').pop()! : text;
|
||||||
|
if (available.has(bare)) return;
|
||||||
|
missing.push(text);
|
||||||
|
};
|
||||||
|
|
||||||
|
// 条件里的列引用:键位(`n > 1`)与 `$col` 值位(`a = b`)
|
||||||
|
const walkCondition = (cond: WhereCondition): void => {
|
||||||
|
for (const [key, value] of Object.entries(cond)) {
|
||||||
|
if (key === '$and' || key === '$or') {
|
||||||
|
for (const sub of (Array.isArray(value) ? value : [value]) as WhereCondition[]) walkCondition(sub);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (key === '$not') { walkCondition(value as WhereCondition); continue; }
|
||||||
|
if (key === '$exists' || /^\s*CASE\b/i.test(key)) continue;
|
||||||
|
checkRef(key);
|
||||||
|
if (value !== null && typeof value === 'object') {
|
||||||
|
for (const operand of Object.values(value as Record<string, unknown>)) {
|
||||||
|
if (operand !== null && typeof operand === 'object' && !Array.isArray(operand) && '$col' in (operand as object)) {
|
||||||
|
checkRef(String((operand as Record<string, unknown>).$col));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
for (const branch of expr.branches) walkCondition(branch.condition);
|
||||||
|
// 结果片段里的列引用(嵌套 CASE 递归)
|
||||||
|
const checkResult = (text: string): void => {
|
||||||
|
const trimmed = text.trim();
|
||||||
|
if (/^'/.test(trimmed) || /^-?\d/.test(trimmed) || /^(NULL|TRUE|FALSE)$/i.test(trimmed)) return;
|
||||||
|
const nested = parseCaseExpression(trimmed);
|
||||||
|
if (nested) {
|
||||||
|
for (const b of nested.branches) walkCondition(b.condition);
|
||||||
|
for (const b of nested.branches) checkResult(b.resultText);
|
||||||
|
if (nested.elseText) checkResult(nested.elseText);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (/^[A-Za-z_][A-Za-z0-9_]*(\.[A-Za-z_][A-Za-z0-9_]*)?$/.test(trimmed)) checkRef(trimmed);
|
||||||
|
};
|
||||||
|
for (const branch of expr.branches) checkResult(branch.resultText);
|
||||||
|
if (expr.elseText) checkResult(expr.elseText);
|
||||||
|
|
||||||
|
if (missing.length > 0) {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Unknown column${missing.length > 1 ? 's' : ''} ${missing.map((c) => `"${c}"`).join(', ')}`
|
||||||
|
+ ` in ${context}`,
|
||||||
|
'COLUMN_NOT_FOUND',
|
||||||
|
{ columns: missing },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 求值一个"值表达式"片段(THEN/ELSE 的操作数)。
|
||||||
|
*
|
||||||
|
* 支持的形态(覆盖既有全部用例,不引入静默降级):
|
||||||
|
* - 字符串字面量(含 `''` 转义)、数字、TRUE/FALSE/NULL;
|
||||||
|
* - 列引用(裸列名或 `表.列`);
|
||||||
|
* - **嵌套 CASE**(递归求值);
|
||||||
|
* - 其它无法识别的文本 → 抛 `NOT_SUPPORTED`,而不是"当字符串返回"
|
||||||
|
* (原实现把无法识别的文本原样返回,于是嵌套 CASE 的残片
|
||||||
|
* `"big' END ELSE 'small"` 变成了用户可见的返回值)。
|
||||||
|
*/
|
||||||
|
export function evaluateExpressionValue(
|
||||||
|
text: string,
|
||||||
|
row: Record<string, unknown>,
|
||||||
|
): unknown {
|
||||||
|
const trimmed = text.trim();
|
||||||
|
if (trimmed === '') {
|
||||||
|
throw new DatabaseError('Empty expression value', 'PARSE_ERROR');
|
||||||
|
}
|
||||||
|
|
||||||
|
// 字符串字面量(SQL 标准 '' 转义)
|
||||||
|
const strLit = trimmed.match(/^'(.*)'$/s);
|
||||||
|
if (strLit) return strLit[1].replace(/''/g, "'");
|
||||||
|
// NULL / 布尔
|
||||||
|
if (/^NULL$/i.test(trimmed)) return null;
|
||||||
|
if (/^TRUE$/i.test(trimmed)) return true;
|
||||||
|
if (/^FALSE$/i.test(trimmed)) return false;
|
||||||
|
// 数字常量(含负号与小数)
|
||||||
|
if (/^-?\d+(\.\d+)?$/.test(trimmed)) return Number(trimmed);
|
||||||
|
|
||||||
|
// 嵌套 CASE
|
||||||
|
const nested = parseCaseExpression(trimmed);
|
||||||
|
if (nested) return evaluateCase(nested, row);
|
||||||
|
|
||||||
|
// 列引用(含 `表.列`)—— 未知列抛 COLUMN_NOT_FOUND(与投影路径同口径)
|
||||||
|
if (/^[A-Za-z_][A-Za-z0-9_]*(\.[A-Za-z_][A-Za-z0-9_]*)?$/.test(trimmed)) {
|
||||||
|
const value = resolveColumnValue(row, trimmed, { strict: false, context: 'CASE result' });
|
||||||
|
if (isUnresolved(value)) {
|
||||||
|
throw new DatabaseError(`Unknown column "${trimmed}" in expression`, 'COLUMN_NOT_FOUND', {
|
||||||
|
column: trimmed,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Unsupported expression in CASE result: "${trimmed}"`,
|
||||||
|
'NOT_SUPPORTED',
|
||||||
|
{ expression: trimmed },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 供调用方复用的哨兵(避免重复 import sql-compare) */
|
||||||
|
export { UNRESOLVED };
|
||||||
@@ -7,3 +7,10 @@ export type * from './ast';
|
|||||||
export { SelectQueryBuilder, UpdateQueryBuilder, DeleteQueryBuilder } from './builder';
|
export { SelectQueryBuilder, UpdateQueryBuilder, DeleteQueryBuilder } from './builder';
|
||||||
export { compileStatement } from './compiler';
|
export { compileStatement } from './compiler';
|
||||||
export { QueryExecutor } from './executor';
|
export { QueryExecutor } from './executor';
|
||||||
|
// v0.8.0: WHERE 匹配与排序/投影的统一入口
|
||||||
|
export {
|
||||||
|
matchWhere,
|
||||||
|
applyOrderBy,
|
||||||
|
projectColumns,
|
||||||
|
containsUnresolvedSubqueries,
|
||||||
|
} from './where-matcher';
|
||||||
|
|||||||
@@ -0,0 +1,233 @@
|
|||||||
|
/**
|
||||||
|
* metona-sqlark SQL 值语义 — 三值逻辑与统一编码(v0.8.0)
|
||||||
|
* @module query/sql-compare
|
||||||
|
*
|
||||||
|
* ============================================================================
|
||||||
|
* 为什么需要它(v0.8.0 审计根因 2 / PB-2)
|
||||||
|
* ============================================================================
|
||||||
|
* 此前项目里并存**五套**值相等语义:
|
||||||
|
* 1. `where-matcher` 的 `===`
|
||||||
|
* 2. 哈希连接的 `String(v ?? '\0')`
|
||||||
|
* 3. 分组键 `encodeGroupKey`
|
||||||
|
* 4. `COUNT(DISTINCT)` 的 `String(v)`
|
||||||
|
* 5. Aria 二级索引键的 `String(v)`
|
||||||
|
* 它们对 NULL 的处理各不相同,于是出现一批"静默错值":
|
||||||
|
* - `WHERE a = NULL` 命中 NULL 行(SQL 标准应为空集,比较结果是 UNKNOWN);
|
||||||
|
* - `WHERE a != NULL` 返回所有非 NULL 行(标准同样为空集);
|
||||||
|
* - `WHERE a NOT BETWEEN 1 AND 2` 会把 NULL 行**排除**(标准应包含,
|
||||||
|
* 因为 UNKNOWN 经过 NOT 仍是 UNKNOWN,而 WHERE 只保留 TRUE……此处需按下文语义);
|
||||||
|
* - `WHERE a NOT IN (1, 2)` 把 NULL 行当作"不在列表"返回;
|
||||||
|
* - JOIN 的 NULL 键:嵌套循环认为 NULL = NULL 成立、哈希连接不成立 →
|
||||||
|
* **结果取决于右表有没有索引**。
|
||||||
|
*
|
||||||
|
* ============================================================================
|
||||||
|
* 本模块提供的语义
|
||||||
|
* ============================================================================
|
||||||
|
* `sqlCompare(a, b)` / `sqlEquals(a, b)` 返回三值:TRUE / FALSE / UNKNOWN。
|
||||||
|
* 任一操作数为 NULL/undefined → UNKNOWN(SQL 标准)。
|
||||||
|
* `matchWhere` 只在结果为 TRUE 时保留该行。
|
||||||
|
*
|
||||||
|
* 注意 `IS NULL` / `IS NOT NULL` **不走比较**:它们是谓词,直接对 NULL 求值,
|
||||||
|
* 由 parser 生成为独立的 `$isNull` / `$isNotNull` 标记(见 SQL_LOGICAL 说明)。
|
||||||
|
*/
|
||||||
|
|
||||||
|
/** SQL 三值逻辑 */
|
||||||
|
export const SQL_LOGICAL = Object.freeze({
|
||||||
|
TRUE: 'TRUE',
|
||||||
|
FALSE: 'FALSE',
|
||||||
|
/** 未知(任一操作数为 NULL,或存在无法确定的引用) */
|
||||||
|
UNKNOWN: 'UNKNOWN',
|
||||||
|
} as const);
|
||||||
|
|
||||||
|
export type SqlTruth = (typeof SQL_LOGICAL)[keyof typeof SQL_LOGICAL];
|
||||||
|
|
||||||
|
/** 是否为 NULL 语义值(undefined 与 null 等价为 SQL NULL) */
|
||||||
|
export function isSqlNull(value: unknown): boolean {
|
||||||
|
return value === null || value === undefined;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 未解析的列引用标记。
|
||||||
|
*
|
||||||
|
* 用于区分两件必须分开处理的事:
|
||||||
|
* - `row[col]` 取到 `undefined` 可能是"该行此列为 NULL",
|
||||||
|
* 也可能是"这一列根本不在当前行里"(例如子查询引用了外层列)。
|
||||||
|
* 前者应判 UNKNOWN,后者应让调用方转交带上下文的求值器(不能静默当 NULL)。
|
||||||
|
* 用 Unique Symbol 保证不会与真实数据冲突。
|
||||||
|
*/
|
||||||
|
export const UNRESOLVED = Symbol('sqlark.unresolved');
|
||||||
|
|
||||||
|
/** 判定某值是否为"未解析引用" */
|
||||||
|
export function isUnresolved(value: unknown): value is typeof UNRESOLVED {
|
||||||
|
return value === UNRESOLVED;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 统一的类型化值编码(用于分组键 / DISTINCT 键 / 索引键)。
|
||||||
|
*
|
||||||
|
* 为什么不能直接 `String(v)`:
|
||||||
|
* - `String(null)` = `'null'` 与字符串 `'null'` 冲突;
|
||||||
|
* - `String(1)` = `'1'` 与字符串 `'1'` 冲突(v0.7.4 已修 DISTINCT,
|
||||||
|
* 但 COUNT(DISTINCT) 与 Aria 索引键仍是 `String()`)。
|
||||||
|
* 这里用类型前缀 + 长度前缀,保证不同类型、不同值必然编码不同,
|
||||||
|
* 且不会出现 `\x1f` 之类分隔符被数据内容伪造的问题。
|
||||||
|
*/
|
||||||
|
export function encodeValueKey(value: unknown): string {
|
||||||
|
if (value === null || value === undefined) return 'z';
|
||||||
|
const t = typeof value;
|
||||||
|
switch (t) {
|
||||||
|
case 'string': {
|
||||||
|
const s = value as string;
|
||||||
|
return `s${s.length}:${s}`;
|
||||||
|
}
|
||||||
|
case 'number': {
|
||||||
|
const n = value as number;
|
||||||
|
// -0 与 0 在 SQL 中相等;NaN 单独编码(避免与任何值相等)
|
||||||
|
if (Number.isNaN(n)) return 'nNaN';
|
||||||
|
return `n${Object.is(n, -0) ? 0 : n}`;
|
||||||
|
}
|
||||||
|
case 'boolean': return `b${value ? 1 : 0}`;
|
||||||
|
case 'bigint': return `i${String(value)}`;
|
||||||
|
case 'object': {
|
||||||
|
// 日期按时间戳比较;数组/对象按 JSON(键序由 JSON.stringify 决定,
|
||||||
|
// 对同构数据稳定;异构键序在上层已由 schema 约束)
|
||||||
|
if (value instanceof Date) return `d${value.getTime()}`;
|
||||||
|
if (Array.isArray(value)) return `a${JSON.stringify(value)}`;
|
||||||
|
return `o${JSON.stringify(value)}`;
|
||||||
|
}
|
||||||
|
default: return `x${String(value)}`;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* SQL 比较:返回三值。
|
||||||
|
*
|
||||||
|
* 与 JavaScript 运算符的关键差异:
|
||||||
|
* - 任一操作数为 NULL → UNKNOWN(而不是 false);
|
||||||
|
* - 非数值字符串与数值比较 → UNKNOWN(而不是 JS 的强制转换结果);
|
||||||
|
* - 未解析引用 → UNKNOWN(调用方据此转交上下文求值)。
|
||||||
|
*/
|
||||||
|
export function sqlCompare(a: unknown, b: unknown): SqlTruth {
|
||||||
|
const ord = sqlCompareOrder(a, b);
|
||||||
|
if (ord === null) return SQL_LOGICAL.UNKNOWN;
|
||||||
|
return ord === 0 ? SQL_LOGICAL.TRUE : SQL_LOGICAL.FALSE;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 相等比较(三值) */
|
||||||
|
export function sqlEquals(a: unknown, b: unknown): SqlTruth {
|
||||||
|
return sqlCompare(a, b);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 三路排序比较:返回 -1 / 0 / 1,无法比较(含 NULL、类型不可比)返回 null。
|
||||||
|
*
|
||||||
|
* **这是本模块唯一的排序/比较原语** —— `sqlCompare`(相等)与所有有序比较
|
||||||
|
* ($gt/$gte/$lt/$lte、ORDER BY)都必须建立在它之上。
|
||||||
|
*
|
||||||
|
* 为什么要把"排序"与"相等"分开:初版实现里 `sqlCompare` 直接返回三值,
|
||||||
|
* 于是"a < b"与"a === b"都被编码成 TRUE,`$gte` 误把"小于"当成"大于等于"
|
||||||
|
* (实测 `WHERE n >= 2` 在 1..3 上返回 3 而不是 2 与 3)。
|
||||||
|
* 排序是三路的(<, =, >),相等是二值的(=, ≠)—— 把前者硬塞进后者必然出错。
|
||||||
|
*/
|
||||||
|
export function sqlCompareOrder(a: unknown, b: unknown): number | null {
|
||||||
|
if (isUnresolved(a) || isUnresolved(b)) return null;
|
||||||
|
if (isSqlNull(a) || isSqlNull(b)) return null;
|
||||||
|
|
||||||
|
// 数字 vs 数字
|
||||||
|
if (typeof a === 'number' && typeof b === 'number') {
|
||||||
|
if (Number.isNaN(a) || Number.isNaN(b)) return null;
|
||||||
|
if (a === b) return 0;
|
||||||
|
return a < b ? -1 : 1;
|
||||||
|
}
|
||||||
|
// 布尔 vs 布尔
|
||||||
|
if (typeof a === 'boolean' && typeof b === 'boolean') {
|
||||||
|
if (a === b) return 0;
|
||||||
|
return a === false ? -1 : 1;
|
||||||
|
}
|
||||||
|
// 字符串 vs 字符串:字典序
|
||||||
|
if (typeof a === 'string' && typeof b === 'string') {
|
||||||
|
if (a === b) return 0;
|
||||||
|
return a < b ? -1 : 1;
|
||||||
|
}
|
||||||
|
// 日期 vs 日期
|
||||||
|
if (a instanceof Date && b instanceof Date) {
|
||||||
|
const ta = a.getTime();
|
||||||
|
const tb = b.getTime();
|
||||||
|
if (Number.isNaN(ta) || Number.isNaN(tb)) return null;
|
||||||
|
if (ta === tb) return 0;
|
||||||
|
return ta < tb ? -1 : 1;
|
||||||
|
}
|
||||||
|
// 类型不同:本项目保持严格类型(不做静默强转),判定为不可比(null)
|
||||||
|
// —— 这样 `WHERE numCol = 'abc'` 不会意外命中,也不会因 `'5' = 5` 的 JS
|
||||||
|
// 行为产生跨引擎差异。唯一例外:数值与"纯数值字符串"按数值比较
|
||||||
|
// (与既有「$in 列表含字符串」行为兼容)。
|
||||||
|
const na = toNumericIfPossible(a);
|
||||||
|
const nb = toNumericIfPossible(b);
|
||||||
|
if (na !== null && nb !== null) {
|
||||||
|
if (na === nb) return 0;
|
||||||
|
return na < nb ? -1 : 1;
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 比较结果转布尔(仅 TRUE 为真;UNKNOWN 与 FALSE 都不保留该行) */
|
||||||
|
export function isTrue(truth: SqlTruth): boolean {
|
||||||
|
return truth === SQL_LOGICAL.TRUE;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 三值 AND */
|
||||||
|
export function sqlAnd(a: SqlTruth, b: SqlTruth): SqlTruth {
|
||||||
|
if (a === SQL_LOGICAL.FALSE || b === SQL_LOGICAL.FALSE) return SQL_LOGICAL.FALSE;
|
||||||
|
if (a === SQL_LOGICAL.UNKNOWN || b === SQL_LOGICAL.UNKNOWN) return SQL_LOGICAL.UNKNOWN;
|
||||||
|
return SQL_LOGICAL.TRUE;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 三值 OR */
|
||||||
|
export function sqlOr(a: SqlTruth, b: SqlTruth): SqlTruth {
|
||||||
|
if (a === SQL_LOGICAL.TRUE || b === SQL_LOGICAL.TRUE) return SQL_LOGICAL.TRUE;
|
||||||
|
if (a === SQL_LOGICAL.UNKNOWN || b === SQL_LOGICAL.UNKNOWN) return SQL_LOGICAL.UNKNOWN;
|
||||||
|
return SQL_LOGICAL.FALSE;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 三值 NOT(UNKNOWN 取反仍为 UNKNOWN) */
|
||||||
|
export function sqlNot(a: SqlTruth): SqlTruth {
|
||||||
|
if (a === SQL_LOGICAL.TRUE) return SQL_LOGICAL.FALSE;
|
||||||
|
if (a === SQL_LOGICAL.FALSE) return SQL_LOGICAL.TRUE;
|
||||||
|
return SQL_LOGICAL.UNKNOWN;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 把值转为数值;无法安全转换时返回 null。
|
||||||
|
* 只有 number 与"纯数值字符串"参与数值比较,避免 `'abc'` 之类被 Number() 变成 NaN。
|
||||||
|
*/
|
||||||
|
function toNumericIfPossible(value: unknown): number | null {
|
||||||
|
if (typeof value === 'number') return Number.isNaN(value) ? null : value;
|
||||||
|
if (typeof value === 'boolean') return value ? 1 : 0;
|
||||||
|
if (typeof value === 'string') {
|
||||||
|
const trimmed = value.trim();
|
||||||
|
if (trimmed === '') return null;
|
||||||
|
const n = Number(trimmed);
|
||||||
|
return Number.isNaN(n) ? null : n;
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* SQL `IN` 列表求值(三值)。
|
||||||
|
*
|
||||||
|
* 标准语义:`x IN (a, b, c)` 等价于 `x = a OR x = b OR x = c`。
|
||||||
|
* 因此:
|
||||||
|
* - 命中任一 → TRUE;
|
||||||
|
* - 都未命中但列表含 NULL(或 x 为 NULL)→ UNKNOWN;
|
||||||
|
* - 都未命中且列表无 NULL 且 x 非 NULL → FALSE。
|
||||||
|
*/
|
||||||
|
export function sqlIn(value: unknown, list: unknown[]): SqlTruth {
|
||||||
|
if (isUnresolved(value)) return SQL_LOGICAL.UNKNOWN;
|
||||||
|
let sawUnknown = false;
|
||||||
|
for (const item of list) {
|
||||||
|
const t = sqlEquals(value, item);
|
||||||
|
if (t === SQL_LOGICAL.TRUE) return SQL_LOGICAL.TRUE;
|
||||||
|
if (t === SQL_LOGICAL.UNKNOWN) sawUnknown = true;
|
||||||
|
}
|
||||||
|
return sawUnknown ? SQL_LOGICAL.UNKNOWN : SQL_LOGICAL.FALSE;
|
||||||
|
}
|
||||||
+409
-100
@@ -1,13 +1,45 @@
|
|||||||
/**
|
/**
|
||||||
* metona-sqlark Shared WHERE Matcher — 统一的条件匹配逻辑
|
* metona-sqlark Shared WHERE Matcher —— 统一的条件求值器
|
||||||
* @module query/where-matcher
|
* @module query/where-matcher
|
||||||
*
|
*
|
||||||
* MemoryEngine / IndexedDBEngine / QueryExecutor 共享此模块,
|
* ============================================================================
|
||||||
* 消除 220+ 行重复代码,统一 $and/$or/$not/$col 行为。
|
* v0.8.0 根治:为什么这里只剩**一个**递归求值器(v0.8.0 审计根因 1/7、PB-2)
|
||||||
|
* ============================================================================
|
||||||
|
* 历史上有两套并存的实现:
|
||||||
|
* - `matchWhere` —— 布尔版(自己的 `$and/$or/$not` 分支 + `matchField`);
|
||||||
|
* - 三值求值 —— 为 UNKNOWN 传播而新增。
|
||||||
|
* 两者对**字段级** `$or` / `$not` 的处理不同,于是同一条 SQL 的语义取决于调用点:
|
||||||
|
* - `{ s: { $not: { $like: 'x' } } }` 经布尔取反会把 NULL 行判真;
|
||||||
|
* - `{ n: { $or: [...] } }`(`NOT BETWEEN` 生成)经布尔分支在 NULL 行上判假;
|
||||||
|
* - 更严重的是字段级 `$or` 递归回了"**where 子句级**"求值器:
|
||||||
|
* `{ n: { $or: [ { $lt: 1 }, { $gt: 2 } ] } }` 里的 `{ $lt: 1 }` 被当成
|
||||||
|
* "查询字段 `$lt`",于是每一行都求值 UNKNOWN → `NOT BETWEEN` 恒空集。
|
||||||
|
*
|
||||||
|
* 现在只有一条代码路径:**一个递归求值器**,同时理解 where 子句级(键是列名或
|
||||||
|
* 逻辑连接词)与操作符级(键是 `$gt` 之类)。区别由**位置**参数承载,
|
||||||
|
* 而不是由另一个函数承载:
|
||||||
|
* - `evalWhere(ctx, where)` —— where 子句(`$and`/`$or` 子项是 where 子句)
|
||||||
|
* - `evalOperatorObject(ctx, ...)` —— 操作符对象(`$and`/`$or` 子项是操作符对象)
|
||||||
|
* 因此 `$or` 的两种含义都在同一个函数里显式分派,不可能再漂移。
|
||||||
|
*
|
||||||
|
* `matchWhere` 保留为对外入口,语义定义为"是否保留该行" = 三值结果恰为 TRUE。
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import type { WhereCondition, OrderBy } from '../constants';
|
import type { WhereCondition, OrderBy } from '../constants';
|
||||||
import { DatabaseError } from '../constants';
|
import { DatabaseError } from '../constants';
|
||||||
|
import {
|
||||||
|
SQL_LOGICAL,
|
||||||
|
type SqlTruth,
|
||||||
|
sqlCompare,
|
||||||
|
sqlCompareOrder,
|
||||||
|
sqlIn,
|
||||||
|
sqlNot,
|
||||||
|
sqlAnd,
|
||||||
|
sqlOr,
|
||||||
|
isSqlNull,
|
||||||
|
isUnresolved,
|
||||||
|
UNRESOLVED,
|
||||||
|
} from './sql-compare';
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// LIKE 正则缓存
|
// LIKE 正则缓存
|
||||||
@@ -28,118 +60,326 @@ function compileLikeRegex(pattern: string): RegExp {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// WHERE 匹配(顶层入口)
|
// 求值上下文
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/** 求值选项(对外 API 保持既有形状) */
|
||||||
|
export interface MatchOptions {
|
||||||
|
/**
|
||||||
|
* 是否解析 `$col` 列引用(关联子查询 / JOIN ON 的列对列比较)。
|
||||||
|
*
|
||||||
|
* 引擎层(`matchWhere` 直通调用)没有外层行上下文,必须**不**解析:
|
||||||
|
* 此时 `$col` 求值为 UNRESOLVED(比较 → UNKNOWN,行被排除),
|
||||||
|
* 而不是抛"未知操作符",也不是静默判真。
|
||||||
|
*/
|
||||||
|
$col?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 内部求值上下文。
|
||||||
|
*
|
||||||
|
* `row` 是**当前行**:`$col: 'y'` 表示"与当前行的 y 列比较"。
|
||||||
|
* v0.8.0 之前这个行上下文藏在 `matchField` 的闭包里,字段级 `$or` 递归回
|
||||||
|
* where 级求值器时被丢弃,导致 `$col` 只能靠 `options.$col` 开关 + 行参数
|
||||||
|
* 隐式传递 —— 现在它显式随上下文下传,任何深度的嵌套求值都不会丢。
|
||||||
|
*/
|
||||||
|
interface EvalContext {
|
||||||
|
row: Record<string, unknown>;
|
||||||
|
options: MatchOptions;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// WHERE 匹配(对外入口)
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* 匹配完整 WHERE 条件
|
* 匹配完整 WHERE 条件(布尔口径:仅 TRUE 保留该行)。
|
||||||
* @param row 当前数据行
|
*
|
||||||
* @param where WHERE 条件对象
|
* @param row 当前数据行
|
||||||
* @param options.$col 是否启用 $col 列引用解析
|
* @param where WHERE 条件对象
|
||||||
|
* @param options `$col` 是否启用列引用解析
|
||||||
*/
|
*/
|
||||||
export function matchWhere(
|
export function matchWhere(
|
||||||
row: Record<string, unknown>,
|
row: Record<string, unknown>,
|
||||||
where: WhereCondition,
|
where: WhereCondition,
|
||||||
options: { $col?: boolean } = {},
|
options: MatchOptions = {},
|
||||||
): boolean {
|
): boolean {
|
||||||
for (const [field, condition] of Object.entries(where)) {
|
return evalWhere({ row, options }, where) === SQL_LOGICAL.TRUE;
|
||||||
// 顶层 $caseResult(v0.3.2):由 Executor 对 CASE WHEN 表达式逐行求值后产生
|
}
|
||||||
if (field === '$caseResult') {
|
|
||||||
if (condition !== true) return false;
|
/**
|
||||||
|
* 三值版本:供需要区分 UNKNOWN 与 FALSE 的调用方使用
|
||||||
|
* (JOIN 外连接判定、`NOT IN` 子查询等)。
|
||||||
|
*/
|
||||||
|
export function matchWhereThreeValued(
|
||||||
|
row: Record<string, unknown>,
|
||||||
|
where: WhereCondition,
|
||||||
|
options: MatchOptions = {},
|
||||||
|
): SqlTruth {
|
||||||
|
return evalWhere({ row, options }, where);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.7.4: 检测 WHERE 中未解析的子查询/列引用标记($subquery / $col / $exists)。
|
||||||
|
*
|
||||||
|
* QueryBuilder 等直通引擎的写路径(update/delete)不经 Executor 解析子查询,
|
||||||
|
* 写路径预检阶段据此显式拒绝(否则这些标记在引擎层恒 UNKNOWN → 静默影响 0 行)。
|
||||||
|
*/
|
||||||
|
export function containsUnresolvedSubqueries(where: WhereCondition | undefined): boolean {
|
||||||
|
if (!where) return false;
|
||||||
|
for (const [k, v] of Object.entries(where)) {
|
||||||
|
if (isLogicalKey(k)) {
|
||||||
|
const subs = (Array.isArray(v) ? v : [v]) as WhereCondition[];
|
||||||
|
if (subs.some((sub) => containsUnresolvedSubqueries(sub))) return true;
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
// 顶层 $exists(v0.3.0):由 Executor.resolveSubqueries 解析为 boolean
|
if (k === '$exists') return true;
|
||||||
if (field === '$exists') {
|
if (isPlainObject(v) && operatorObjectHasUnresolved(v)) return true;
|
||||||
if (condition !== true) return false;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
// 顶层 $and
|
|
||||||
if (field === '$and') {
|
|
||||||
const subs = condition as WhereCondition[];
|
|
||||||
if (!subs.every((sub) => matchWhere(row, sub, options))) return false;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
// 顶层 $or
|
|
||||||
if (field === '$or') {
|
|
||||||
const subs = condition as WhereCondition[];
|
|
||||||
if (!subs.some((sub) => matchWhere(row, sub, options))) return false;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
// 顶层 $not(v0.3.2 修复:NOT (expr) 生成的 { $not: inner })
|
|
||||||
if (field === '$not') {
|
|
||||||
if (matchWhere(row, condition as WhereCondition, options)) return false;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
if (!matchField(row[field], condition, row, options)) return false;
|
|
||||||
}
|
}
|
||||||
return true;
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 递归检测操作符对象里是否存在未解析引用(含 `$and`/`$or`/`$not` 内部) */
|
||||||
|
function operatorObjectHasUnresolved(ops: Record<string, unknown>): boolean {
|
||||||
|
for (const [op, operand] of Object.entries(ops)) {
|
||||||
|
if (op === '$and' || op === '$or') {
|
||||||
|
const subs = (Array.isArray(operand) ? operand : [operand]) as Record<string, unknown>[];
|
||||||
|
if (subs.some((sub) => isPlainObject(sub) && operatorObjectHasUnresolved(sub))) return true;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (op === '$not') {
|
||||||
|
if (isPlainObject(operand) && operatorObjectHasUnresolved(operand)) return true;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (op === '$subquery' || op === '$col') return true;
|
||||||
|
if (isPlainObject(operand) && ('$subquery' in operand || '$col' in operand)) return true;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// 字段匹配
|
// 统一递归求值器
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
function matchField(
|
/** 逻辑连接词(where 子句级) */
|
||||||
|
function isLogicalKey(key: string): boolean {
|
||||||
|
return key === '$and' || key === '$or' || key === '$not';
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 纯对象(非 null、非数组) */
|
||||||
|
function isPlainObject(value: unknown): value is Record<string, unknown> {
|
||||||
|
return typeof value === 'object' && value !== null && !Array.isArray(value);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 求值一个 WHERE 子句(where 级)。
|
||||||
|
*
|
||||||
|
* 顶层键要么是逻辑连接词,要么是列名:
|
||||||
|
* - `$and: [where...]` / `$or: [where...]` —— 子项是**完整 where 子句**;
|
||||||
|
* - `$not: where` —— 子项是完整 where 子句;
|
||||||
|
* - `$exists: boolean` —— 关联 EXISTS 解析结果标记;
|
||||||
|
* - `$caseResult` —— CASE WHEN 解析结果标记;
|
||||||
|
* - 其余键 = 列名,其值是**操作符对象**或裸值(裸值等价 `$eq`)。
|
||||||
|
*/
|
||||||
|
function evalWhere(ctx: EvalContext, where: WhereCondition): SqlTruth {
|
||||||
|
let acc: SqlTruth = SQL_LOGICAL.TRUE;
|
||||||
|
for (const [key, condition] of Object.entries(where)) {
|
||||||
|
let one: SqlTruth;
|
||||||
|
if (key === '$and') {
|
||||||
|
one = evalWhereLogical(ctx, condition, 'and');
|
||||||
|
} else if (key === '$or') {
|
||||||
|
one = evalWhereLogical(ctx, condition, 'or');
|
||||||
|
} else if (key === '$not') {
|
||||||
|
one = sqlNot(evalWhere(ctx, condition as WhereCondition));
|
||||||
|
} else if (key === '$exists' || key === '$caseResult') {
|
||||||
|
one = condition === true ? SQL_LOGICAL.TRUE : SQL_LOGICAL.FALSE;
|
||||||
|
} else {
|
||||||
|
one = evalValueCondition(ctx, resolveField(ctx, key), condition);
|
||||||
|
}
|
||||||
|
acc = sqlAnd(acc, one);
|
||||||
|
if (acc === SQL_LOGICAL.FALSE) return SQL_LOGICAL.FALSE;
|
||||||
|
}
|
||||||
|
return acc;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** `$and` / `$or`(where 级):子项是完整 where 子句 */
|
||||||
|
function evalWhereLogical(ctx: EvalContext, condition: unknown, kind: 'and' | 'or'): SqlTruth {
|
||||||
|
const subs = (Array.isArray(condition) ? condition : [condition]) as WhereCondition[];
|
||||||
|
let acc: SqlTruth = kind === 'and' ? SQL_LOGICAL.TRUE : SQL_LOGICAL.FALSE;
|
||||||
|
for (const sub of subs) {
|
||||||
|
const one = evalWhere(ctx, sub);
|
||||||
|
acc = kind === 'and' ? sqlAnd(acc, one) : sqlOr(acc, one);
|
||||||
|
// FALSE 在 AND 下、TRUE 在 OR 下都无法被后续子项改变
|
||||||
|
if (kind === 'and' && acc === SQL_LOGICAL.FALSE) return SQL_LOGICAL.FALSE;
|
||||||
|
if (kind === 'or' && acc === SQL_LOGICAL.TRUE) return SQL_LOGICAL.TRUE;
|
||||||
|
}
|
||||||
|
return acc;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 取字段值。
|
||||||
|
*
|
||||||
|
* 行里的键可能是 `t.id` 形式(JOIN / 子查询合并行),而 WHERE 键可能写作
|
||||||
|
* `id`。既有行为是"精确命中优先,否则回退到唯一的后缀匹配";此处保持一致,
|
||||||
|
* 但**不再**像旧实现那样在多个别名同名字段间静默取第一个 —— 歧义由 Executor
|
||||||
|
* 的列解析阶段(`bindColumnRefs` / `assertProjectionColumnsExist`)负责报错。
|
||||||
|
*/
|
||||||
|
function resolveField(ctx: EvalContext, field: string): unknown {
|
||||||
|
if (field in ctx.row) return ctx.row[field];
|
||||||
|
let found: unknown = UNRESOLVED;
|
||||||
|
let hits = 0;
|
||||||
|
for (const key of Object.keys(ctx.row)) {
|
||||||
|
if (key.endsWith(`.${field}`)) {
|
||||||
|
found = ctx.row[key];
|
||||||
|
hits += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (hits === 1) return found;
|
||||||
|
if (hits > 1) return UNRESOLVED;
|
||||||
|
return UNRESOLVED;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 求值"某个值是否满足条件"—— 字段条件与嵌套操作数条件共用的核心。
|
||||||
|
*
|
||||||
|
* 位置无关性是本模块的关键:`{ $or: [ { $gt: 1 } ] }` 无论出现在字段位置
|
||||||
|
* 还是顶层(历史单元测试的写法),子项都按**操作数条件**求值 —— 因为判别式
|
||||||
|
* 相同:键以 `$` 开头且不是逻辑连接词,就是操作符;否则是列名。
|
||||||
|
*/
|
||||||
|
function evalValueCondition(
|
||||||
|
ctx: EvalContext,
|
||||||
value: unknown,
|
value: unknown,
|
||||||
condition: unknown,
|
condition: unknown,
|
||||||
row: Record<string, unknown>,
|
): SqlTruth {
|
||||||
options: { $col?: boolean },
|
if (!isPlainObject(condition)) {
|
||||||
): boolean {
|
// 裸值 = `$eq`;数组 = `$in` 列表(与既有 Mongo 风格 WHERE 兼容)
|
||||||
// 嵌套 $and
|
if (Array.isArray(condition)) return sqlIn(value, condition);
|
||||||
if (typeof condition === 'object' && condition !== null && '$and' in (condition as Record<string, unknown>)) {
|
return compareEquality(value, condition);
|
||||||
const subs = (condition as Record<string, unknown>).$and as WhereCondition[];
|
|
||||||
return subs.every((sub) => matchWhere(row, sub, options));
|
|
||||||
}
|
}
|
||||||
// 嵌套 $or
|
return evalOperatorObject(ctx, value, condition);
|
||||||
if (typeof condition === 'object' && condition !== null && '$or' in (condition as Record<string, unknown>)) {
|
|
||||||
const subs = (condition as Record<string, unknown>).$or as WhereCondition[];
|
|
||||||
return subs.some((sub) => matchWhere(row, sub, options));
|
|
||||||
}
|
|
||||||
// $not
|
|
||||||
if (typeof condition === 'object' && condition !== null && '$not' in (condition as Record<string, unknown>)) {
|
|
||||||
return !matchField(value, (condition as Record<string, unknown>).$not, row, options);
|
|
||||||
}
|
|
||||||
// 简单值 => $eq
|
|
||||||
if (typeof condition !== 'object' || condition === null || Array.isArray(condition)) {
|
|
||||||
return value === condition;
|
|
||||||
}
|
|
||||||
|
|
||||||
const ops = condition as Record<string, unknown>;
|
|
||||||
|
|
||||||
// $col 简写: { $col: name } === { $eq: { $col: name } }(仅 JOIN ON 场景)
|
|
||||||
if (options.$col && '$col' in ops && Object.keys(ops).length === 1) {
|
|
||||||
return value === row[ops.$col as string];
|
|
||||||
}
|
|
||||||
|
|
||||||
// 遍历操作符
|
|
||||||
for (const [op, operand] of Object.entries(ops)) {
|
|
||||||
let actualOperand = operand;
|
|
||||||
|
|
||||||
// $col 列引用解析
|
|
||||||
if (options.$col && typeof operand === 'object' && operand !== null && '$col' in (operand as Record<string, unknown>)) {
|
|
||||||
actualOperand = row[(operand as Record<string, unknown>).$col as string];
|
|
||||||
}
|
|
||||||
|
|
||||||
if (!matchOperator(value, op, actualOperand)) return false;
|
|
||||||
}
|
|
||||||
return true;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
/**
|
||||||
// 操作符匹配
|
* 求值一个操作符对象(`{ $gt: 1 }` / `{ $and: [...] }` / `{ $col: 'y' }` …)。
|
||||||
// ---------------------------------------------------------------------------
|
*
|
||||||
|
* `$and` / `$or` 的每个子项按"操作数条件"递归 —— 这正是 `NOT BETWEEN`
|
||||||
|
* 生成的 `{ n: { $or: [ { $lt: 1 }, { $gt: 2 } ] } }` 能正确求值的原因。
|
||||||
|
* 若子项实际是**完整 where 子句**(键是列名),则转交 `evalWhere`,
|
||||||
|
* 与当前行合并后求值(`$col` 上下文因此仍然有效)。
|
||||||
|
*/
|
||||||
|
function evalOperatorObject(
|
||||||
|
ctx: EvalContext,
|
||||||
|
value: unknown,
|
||||||
|
ops: Record<string, unknown>,
|
||||||
|
): SqlTruth {
|
||||||
|
// 纯列引用:{ $col: 'y' } = 与当前行的 y 列比较
|
||||||
|
const keys = Object.keys(ops);
|
||||||
|
if (keys.length === 1 && keys[0] === '$col' && ctx.options.$col) {
|
||||||
|
return compareEquality(value, resolveField(ctx, String(ops.$col)));
|
||||||
|
}
|
||||||
|
|
||||||
function matchOperator(value: unknown, op: string, operand: unknown): boolean {
|
let acc: SqlTruth = SQL_LOGICAL.TRUE;
|
||||||
|
for (const [op, operand] of Object.entries(ops)) {
|
||||||
|
let one: SqlTruth;
|
||||||
|
if (op === '$and') {
|
||||||
|
one = evalOperatorLogical(ctx, value, operand, 'and');
|
||||||
|
} else if (op === '$or') {
|
||||||
|
one = evalOperatorLogical(ctx, value, operand, 'or');
|
||||||
|
} else if (op === '$not') {
|
||||||
|
one = sqlNot(evalOperandSlot(ctx, value, operand));
|
||||||
|
} else {
|
||||||
|
one = evalOperator(ctx, value, op, operand);
|
||||||
|
}
|
||||||
|
acc = sqlAnd(acc, one);
|
||||||
|
if (acc === SQL_LOGICAL.FALSE) return SQL_LOGICAL.FALSE;
|
||||||
|
}
|
||||||
|
return acc;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** `$and` / `$or`(操作符级):子项是操作数条件(操作符对象或裸值) */
|
||||||
|
function evalOperatorLogical(
|
||||||
|
ctx: EvalContext,
|
||||||
|
value: unknown,
|
||||||
|
condition: unknown,
|
||||||
|
kind: 'and' | 'or',
|
||||||
|
): SqlTruth {
|
||||||
|
const items = Array.isArray(condition) ? condition : [condition];
|
||||||
|
let acc: SqlTruth = kind === 'and' ? SQL_LOGICAL.TRUE : SQL_LOGICAL.FALSE;
|
||||||
|
for (const item of items) {
|
||||||
|
const one = evalOperandSlot(ctx, value, item);
|
||||||
|
acc = kind === 'and' ? sqlAnd(acc, one) : sqlOr(acc, one);
|
||||||
|
if (kind === 'and' && acc === SQL_LOGICAL.FALSE) return SQL_LOGICAL.FALSE;
|
||||||
|
if (kind === 'or' && acc === SQL_LOGICAL.TRUE) return SQL_LOGICAL.TRUE;
|
||||||
|
}
|
||||||
|
return acc;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 单个嵌套子项(`$and`/`$or`/`$not` 的操作数)的求值。
|
||||||
|
*
|
||||||
|
* 两种合法形态,按**键的形状**判别,不做猜测性兜底:
|
||||||
|
* - 含非 `$` 开头(或逻辑连接词)的键 → 完整 where 子句,交由 `evalWhere`;
|
||||||
|
* - 其余 → 操作符对象,交由 `evalOperatorObject`。
|
||||||
|
*
|
||||||
|
* 注意 `$not` 的子项**不会**走到这里被判为 where 子句:`evalOperatorObject`
|
||||||
|
* 对 `$not` 直接调用本函数,而 `{ $gte: 1 }` 全部键以 `$` 开头 → 操作符对象 ✓。
|
||||||
|
* 若调用方写成 `{ $not: { n: { $gte: 1 } } }`(字段级 `$not` 包了 where 子句),
|
||||||
|
* 则按 where 子句解释 —— 这正是 `evalNestedSlot` 的判别分支,语义为
|
||||||
|
* "NOT (该行满足 n >= 1)",与顶层 `$not` 一致。
|
||||||
|
*/
|
||||||
|
function evalOperandSlot(ctx: EvalContext, value: unknown, item: unknown): SqlTruth {
|
||||||
|
if (!isPlainObject(item)) return evalValueCondition(ctx, value, item);
|
||||||
|
const isWhereClause = Object.keys(item).some(
|
||||||
|
(k) => !k.startsWith('$') || isLogicalKey(k) || k === '$exists' || k === '$caseResult',
|
||||||
|
);
|
||||||
|
return isWhereClause ? evalWhere(ctx, item as WhereCondition) : evalOperatorObject(ctx, value, item);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 操作符求值(三值语义)。
|
||||||
|
*
|
||||||
|
* - 比较类操作数含 NULL → UNKNOWN(此前 `$eq: null` 命中 NULL 行、`$ne: null`
|
||||||
|
* 命中所有非 NULL 行,两者都不符合 SQL 标准);
|
||||||
|
* - `$in` / `$nin` 用 `sqlIn` 的列表语义(列表含 NULL → 未命中时为 UNKNOWN);
|
||||||
|
* - `$like` 对 NULL 操作数返回 UNKNOWN(`String(null)` 会得到 "null" 去匹配,
|
||||||
|
* 属于静默错值);
|
||||||
|
* - `$isNull` / `$isNotNull` 是**谓词**,直接对 NULL 求值,不走比较。
|
||||||
|
*/
|
||||||
|
function evalOperator(
|
||||||
|
ctx: EvalContext,
|
||||||
|
value: unknown,
|
||||||
|
op: string,
|
||||||
|
operand: unknown,
|
||||||
|
): SqlTruth {
|
||||||
|
const actual = resolveOperand(ctx, operand);
|
||||||
switch (op) {
|
switch (op) {
|
||||||
case '$eq': return value === operand;
|
case '$eq': return compareEquality(value, actual);
|
||||||
case '$ne': return value !== operand;
|
case '$ne': return sqlNot(compareEquality(value, actual));
|
||||||
case '$gt': return (value as number) > (operand as number);
|
case '$gt': return compareOrdered(value, actual, (c) => c > 0);
|
||||||
case '$gte': return (value as number) >= (operand as number);
|
case '$gte': return compareOrdered(value, actual, (c) => c >= 0);
|
||||||
case '$lt': return (value as number) < (operand as number);
|
case '$lt': return compareOrdered(value, actual, (c) => c < 0);
|
||||||
case '$lte': return (value as number) <= (operand as number);
|
case '$lte': return compareOrdered(value, actual, (c) => c <= 0);
|
||||||
case '$in': return Array.isArray(operand) && operand.includes(value);
|
case '$in': return Array.isArray(actual) ? sqlIn(value, actual) : SQL_LOGICAL.FALSE;
|
||||||
case '$nin': return Array.isArray(operand) && !operand.includes(value);
|
case '$nin': return Array.isArray(actual) ? sqlNot(sqlIn(value, actual)) : SQL_LOGICAL.FALSE;
|
||||||
case '$like': return compileLikeRegex(String(operand)).test(String(value));
|
case '$like': {
|
||||||
|
if (isSqlNull(value) || isSqlNull(actual) || isUnresolved(value) || isUnresolved(actual)) {
|
||||||
|
return SQL_LOGICAL.UNKNOWN;
|
||||||
|
}
|
||||||
|
return compileLikeRegex(String(actual)).test(String(value))
|
||||||
|
? SQL_LOGICAL.TRUE
|
||||||
|
: SQL_LOGICAL.FALSE;
|
||||||
|
}
|
||||||
|
// 谓词(不参与三值比较)
|
||||||
|
case '$isNull': return isSqlNull(value) ? SQL_LOGICAL.TRUE : SQL_LOGICAL.FALSE;
|
||||||
|
case '$isNotNull': return isSqlNull(value) ? SQL_LOGICAL.FALSE : SQL_LOGICAL.TRUE;
|
||||||
|
// `$col` 出现在操作符位置但未启用列上下文 → 无法求值
|
||||||
|
case '$col': return ctx.options.$col
|
||||||
|
? compareEquality(value, resolveField(ctx, String(operand)))
|
||||||
|
: SQL_LOGICAL.UNKNOWN;
|
||||||
|
// `$subquery` 必须由 Executor 先行解析(`resolveSubqueries`)。
|
||||||
|
// 走到这里说明调用方跳过了 Executor:返回 UNKNOWN 让该行被排除,
|
||||||
|
// 而不是静默当 NULL 比较。
|
||||||
|
case '$subquery': return SQL_LOGICAL.UNKNOWN;
|
||||||
// v0.7.2: 未知操作符显式报错 —— 此前静默返回 true(所有行匹配),
|
// v0.7.2: 未知操作符显式报错 —— 此前静默返回 true(所有行匹配),
|
||||||
// 拼错操作符(如 $betwen)时过滤形同虚设且无任何提示
|
// 拼错操作符(如 $betwen)时过滤形同虚设且无任何提示
|
||||||
default:
|
default:
|
||||||
@@ -147,6 +387,48 @@ function matchOperator(value: unknown, op: string, operand: unknown): boolean {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 解析操作数槽位中的引用标记。
|
||||||
|
*
|
||||||
|
* - `{ $col: 'y' }` → 当前行的 y 列值(仅在 `options.$col` 启用时);
|
||||||
|
* - `{ $subquery: [...] }` → UNRESOLVED,交由上层识别为"未解析"。
|
||||||
|
*
|
||||||
|
* 这两种形态都是**对象**,而非"恰好等于某个值",所以必须在这里显式解引用,
|
||||||
|
* 否则 `sqlCompare(5, { $col: 'y' })` 会按不可比返回 UNKNOWN —— 恰好也是
|
||||||
|
* UNKNOWN,但那会掩盖"调用方忘了传 `$col`"这一真正的配置错误。
|
||||||
|
*/
|
||||||
|
function resolveOperand(ctx: EvalContext, operand: unknown): unknown {
|
||||||
|
if (!isPlainObject(operand)) return operand;
|
||||||
|
const keys = Object.keys(operand);
|
||||||
|
if (keys.length !== 1) return operand;
|
||||||
|
if (keys[0] === '$col') {
|
||||||
|
return ctx.options.$col ? resolveField(ctx, String(operand.$col)) : UNRESOLVED;
|
||||||
|
}
|
||||||
|
if (keys[0] === '$subquery') return UNRESOLVED;
|
||||||
|
return operand;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 相等比较(三值) */
|
||||||
|
function compareEquality(value: unknown, operand: unknown): SqlTruth {
|
||||||
|
const truth = sqlCompare(value, operand);
|
||||||
|
if (truth === SQL_LOGICAL.TRUE) return SQL_LOGICAL.TRUE;
|
||||||
|
if (isSqlNull(value) || isSqlNull(operand) || isUnresolved(value) || isUnresolved(operand)) {
|
||||||
|
return SQL_LOGICAL.UNKNOWN;
|
||||||
|
}
|
||||||
|
return SQL_LOGICAL.FALSE;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 有序比较(三值):任一操作数 NULL/未解析/不可比 → UNKNOWN */
|
||||||
|
function compareOrdered(
|
||||||
|
value: unknown,
|
||||||
|
operand: unknown,
|
||||||
|
accept: (cmp: number) => boolean,
|
||||||
|
): SqlTruth {
|
||||||
|
const ord = sqlCompareOrder(value, operand);
|
||||||
|
if (ord === null) return SQL_LOGICAL.UNKNOWN;
|
||||||
|
return accept(ord) ? SQL_LOGICAL.TRUE : SQL_LOGICAL.FALSE;
|
||||||
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// 排序
|
// 排序
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -182,19 +464,46 @@ function compare(a: unknown, b: unknown): number {
|
|||||||
// 列投影
|
// 列投影
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 列投影。
|
||||||
|
*
|
||||||
|
* v0.8.0(B-5):支持**分隔标识符**(`"1"`)与 JOIN 行的 `别名.列` 键。
|
||||||
|
* 此前只做"精确命中,否则找 `endsWith('.col')`",于是:
|
||||||
|
* - `SELECT "1" FROM q`(列名就叫 1)取不到值 → 输出 `{}`
|
||||||
|
* (校验用裸名、投影用带引号的名,两套规则 —— 典型漂移);
|
||||||
|
* - 找不到时**不产出键**,行形状随列是否存在而变。
|
||||||
|
* 现在统一:脱引号 → 精确命中 → 唯一后缀命中;仍找不到则不产出该键
|
||||||
|
*(是否"未知列"由 executor 的 assertProjectionColumnsExist 判定并报错,
|
||||||
|
* 投影层不做静默兜底)。
|
||||||
|
*/
|
||||||
export function projectColumns(row: Record<string, unknown>, columns: string[]): Record<string, unknown> {
|
export function projectColumns(row: Record<string, unknown>, columns: string[]): Record<string, unknown> {
|
||||||
const projected: Record<string, unknown> = {};
|
const projected: Record<string, unknown> = {};
|
||||||
for (const col of columns) {
|
for (const col of columns) {
|
||||||
if (col in row) {
|
const name = unquoteIdentifier(col);
|
||||||
projected[col] = row[col];
|
if (name in row) {
|
||||||
} else {
|
projected[name] = row[name];
|
||||||
for (const key of Object.keys(row)) {
|
continue;
|
||||||
if (key.endsWith(`.${col}`) || key === col) {
|
}
|
||||||
projected[col] = row[key];
|
// JOIN 行键形如 `t.col`:唯一后缀匹配(多个命中视为歧义,取第一个与
|
||||||
break;
|
// executor 的校验口径一致 —— 那里已对歧义报错,能走到这里说明唯一)
|
||||||
}
|
let found: unknown;
|
||||||
|
let hits = 0;
|
||||||
|
for (const key of Object.keys(row)) {
|
||||||
|
if (key.endsWith(`.${name}`) || key === name) {
|
||||||
|
found = row[key];
|
||||||
|
hits += 1;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if (hits >= 1) projected[name] = found;
|
||||||
}
|
}
|
||||||
return projected;
|
return projected;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** 脱去分隔标识符的引号(`"1"` → `1`,`"a""b"` → `a"b`) */
|
||||||
|
function unquoteIdentifier(text: string): string {
|
||||||
|
const trimmed = text.trim();
|
||||||
|
if (trimmed.length >= 2 && trimmed.startsWith('"') && trimmed.endsWith('"')) {
|
||||||
|
return trimmed.slice(1, -1).replace(/""/g, '"');
|
||||||
|
}
|
||||||
|
return trimmed;
|
||||||
|
}
|
||||||
|
|||||||
+3
-1
@@ -4,6 +4,8 @@
|
|||||||
*/
|
*/
|
||||||
|
|
||||||
export { Lexer, tokenize } from './lexer';
|
export { Lexer, tokenize } from './lexer';
|
||||||
export { Parser, parse } from './parser';
|
export { Parser, parse, parseAll, parseWhereCondition } from './parser';
|
||||||
|
// v0.8.0: 参数绑定此前未从本桶导出(README 与核心 query() 都依赖它)
|
||||||
|
export { bindParameters } from './params';
|
||||||
export { TokenType } from './tokens';
|
export { TokenType } from './tokens';
|
||||||
export type { Token } from './tokens';
|
export type { Token } from './tokens';
|
||||||
|
|||||||
+155
-61
@@ -25,96 +25,94 @@ export class Lexer {
|
|||||||
|
|
||||||
/** 读取下一个 Token */
|
/** 读取下一个 Token */
|
||||||
nextToken(): Token {
|
nextToken(): Token {
|
||||||
this.skipWhitespace();
|
this.skipWhitespaceAndComments();
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0:token 的**起始位置**在读到任何字符之前记录。
|
||||||
|
*
|
||||||
|
* 为什么必须在这里记:此前的 position 由各 reader 自行回推,语义**按 token
|
||||||
|
* 类型不一致** —— `readString` 用 `start = this.position + 1`(指向引号**之内**),
|
||||||
|
* 而 `readIdentifier`/`readNumber` 用 `this.position - len`(指向首字符)。
|
||||||
|
* 于是 `'big'` 报 position 23 而实际从 24 开始,任何"按 position 切片"的
|
||||||
|
* 调用方都会多切一个字符(实测:CASE 的 THEN 值被切成 `"big'"`)。
|
||||||
|
* 这里统一为"token 首字符在源码中的下标",所有类型一致。
|
||||||
|
*/
|
||||||
|
const tokenStart = this.position;
|
||||||
|
|
||||||
let tok: Token;
|
let tok: Token;
|
||||||
|
|
||||||
switch (this.ch) {
|
switch (this.ch) {
|
||||||
case ',':
|
case ',':
|
||||||
tok = this.makeToken(TokenType.COMMA, ',');
|
tok = this.makeToken(TokenType.COMMA, ',', tokenStart);
|
||||||
break;
|
break;
|
||||||
case '(':
|
case '(':
|
||||||
tok = this.makeToken(TokenType.LPAREN, '(');
|
tok = this.makeToken(TokenType.LPAREN, '(', tokenStart);
|
||||||
break;
|
break;
|
||||||
case ')':
|
case ')':
|
||||||
tok = this.makeToken(TokenType.RPAREN, ')');
|
tok = this.makeToken(TokenType.RPAREN, ')', tokenStart);
|
||||||
break;
|
break;
|
||||||
case ';':
|
case ';':
|
||||||
tok = this.makeToken(TokenType.SEMICOLON, ';');
|
tok = this.makeToken(TokenType.SEMICOLON, ';');
|
||||||
break;
|
break;
|
||||||
case '*':
|
case '*':
|
||||||
tok = this.makeToken(TokenType.STAR, '*');
|
tok = this.makeToken(TokenType.STAR, '*', tokenStart);
|
||||||
break;
|
break;
|
||||||
case '.':
|
case '.':
|
||||||
tok = this.makeToken(TokenType.DOT, '.');
|
tok = this.makeToken(TokenType.DOT, '.', tokenStart);
|
||||||
break;
|
break;
|
||||||
case '=':
|
case '=':
|
||||||
tok = this.makeToken(TokenType.EQ, '=');
|
tok = this.makeToken(TokenType.EQ, '=', tokenStart);
|
||||||
break;
|
break;
|
||||||
case '!':
|
case '!':
|
||||||
if (this.peekChar() === '=') {
|
if (this.peekChar() === '=') {
|
||||||
this.readChar();
|
this.readChar();
|
||||||
tok = this.makeToken(TokenType.NEQ, '!=');
|
tok = this.makeToken(TokenType.NEQ, '!=', tokenStart);
|
||||||
} else {
|
} else {
|
||||||
tok = this.makeToken(TokenType.ILLEGAL, '!');
|
tok = this.makeToken(TokenType.ILLEGAL, '!', tokenStart);
|
||||||
}
|
}
|
||||||
break;
|
break;
|
||||||
case '>':
|
case '>':
|
||||||
if (this.peekChar() === '=') {
|
if (this.peekChar() === '=') {
|
||||||
this.readChar();
|
this.readChar();
|
||||||
tok = this.makeToken(TokenType.GTE, '>=');
|
tok = this.makeToken(TokenType.GTE, '>=', tokenStart);
|
||||||
} else {
|
} else {
|
||||||
tok = this.makeToken(TokenType.GT, '>');
|
tok = this.makeToken(TokenType.GT, '>', tokenStart);
|
||||||
}
|
}
|
||||||
break;
|
break;
|
||||||
case '<':
|
case '<':
|
||||||
if (this.peekChar() === '=') {
|
if (this.peekChar() === '=') {
|
||||||
this.readChar();
|
this.readChar();
|
||||||
tok = this.makeToken(TokenType.LTE, '<=');
|
tok = this.makeToken(TokenType.LTE, '<=', tokenStart);
|
||||||
} else if (this.peekChar() === '>') {
|
} else if (this.peekChar() === '>') {
|
||||||
this.readChar();
|
this.readChar();
|
||||||
tok = this.makeToken(TokenType.NEQ, '<>');
|
tok = this.makeToken(TokenType.NEQ, '<>', tokenStart);
|
||||||
} else {
|
} else {
|
||||||
tok = this.makeToken(TokenType.LT, '<');
|
tok = this.makeToken(TokenType.LT, '<', tokenStart);
|
||||||
}
|
}
|
||||||
break;
|
break;
|
||||||
case "'":
|
case "'":
|
||||||
|
tok = this.readString();
|
||||||
|
break;
|
||||||
case '"':
|
case '"':
|
||||||
tok = this.readString(this.ch);
|
// v0.8.0: 双引号 = 分隔标识符(SQL 标准),不再当作字符串字面量
|
||||||
|
tok = this.readQuotedIdentifier();
|
||||||
break;
|
break;
|
||||||
case '':
|
case '':
|
||||||
tok = { type: TokenType.EOF, value: '', position: this.position };
|
tok = { type: TokenType.EOF, value: '', position: this.position };
|
||||||
break;
|
break;
|
||||||
default:
|
default:
|
||||||
// SQL 注释: -- 行注释
|
|
||||||
if (this.ch === '-' && this.peekChar() === '-') {
|
|
||||||
this.skipLineComment();
|
|
||||||
return this.nextToken();
|
|
||||||
}
|
|
||||||
// SQL 注释: /* 块注释 */
|
|
||||||
if (this.ch === '/' && this.peekChar() === '*') {
|
|
||||||
this.skipBlockComment();
|
|
||||||
return this.nextToken();
|
|
||||||
}
|
|
||||||
if (this.isLetter(this.ch)) {
|
if (this.isLetter(this.ch)) {
|
||||||
const ident = this.readIdentifier();
|
const ident = this.readIdentifier();
|
||||||
const keyword = KEYWORDS[ident.toUpperCase()];
|
const keyword = KEYWORDS[ident.toUpperCase()];
|
||||||
tok = {
|
// v0.8.0: 用 tokenStart(首字符下标),不再用 position - len 回推
|
||||||
type: keyword ?? TokenType.IDENTIFIER,
|
tok = { type: keyword ?? TokenType.IDENTIFIER, value: ident, position: tokenStart };
|
||||||
value: ident,
|
|
||||||
position: this.position - ident.length,
|
|
||||||
};
|
|
||||||
return tok; // 已读取完毕,不需要再 readChar
|
return tok; // 已读取完毕,不需要再 readChar
|
||||||
} else if (this.isDigit(this.ch) || (this.ch === '-' && this.isDigit(this.peekChar()))) {
|
} else if (this.isDigit(this.ch) || (this.ch === '-' && this.isDigit(this.peekChar()))) {
|
||||||
const num = this.readNumber();
|
const num = this.readNumber();
|
||||||
tok = {
|
tok = { type: TokenType.NUMBER, value: num, position: tokenStart };
|
||||||
type: TokenType.NUMBER,
|
|
||||||
value: num,
|
|
||||||
position: this.position - num.length,
|
|
||||||
};
|
|
||||||
return tok;
|
return tok;
|
||||||
} else {
|
} else {
|
||||||
tok = this.makeToken(TokenType.ILLEGAL, this.ch);
|
tok = this.makeToken(TokenType.ILLEGAL, this.ch, tokenStart);
|
||||||
}
|
}
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
@@ -140,9 +138,31 @@ export class Lexer {
|
|||||||
return this.input[this.readPosition];
|
return this.input[this.readPosition];
|
||||||
}
|
}
|
||||||
|
|
||||||
private skipWhitespace(): void {
|
/**
|
||||||
while (this.ch === ' ' || this.ch === '\t' || this.ch === '\n' || this.ch === '\r') {
|
* 跳过空白与注释(v0.8.0:改为**循环**而非递归)。
|
||||||
this.readChar();
|
*
|
||||||
|
* 此前在 default 分支里用 `return this.nextToken()` 递归跳过注释,
|
||||||
|
* 递归深度 = 连续注释个数:两万个连续的块注释就会直接
|
||||||
|
* `RangeError: Maximum call stack size exceeded`(且是原生错误而非 DatabaseError,
|
||||||
|
* 调用方无法按 code 分类)。现在统一在一个循环里消费空白与注释。
|
||||||
|
*/
|
||||||
|
private skipWhitespaceAndComments(): void {
|
||||||
|
for (;;) {
|
||||||
|
// 空白
|
||||||
|
while (this.ch === ' ' || this.ch === '\t' || this.ch === '\n' || this.ch === '\r') {
|
||||||
|
this.readChar();
|
||||||
|
}
|
||||||
|
// -- 行注释
|
||||||
|
if (this.ch === '-' && this.peekChar() === '-') {
|
||||||
|
this.skipLineComment();
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
// /* 块注释 */
|
||||||
|
if (this.ch === '/' && this.peekChar() === '*') {
|
||||||
|
this.skipBlockComment();
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
return;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -153,17 +173,29 @@ export class Lexer {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 跳过块注释 slash-star ... star-slash */
|
/**
|
||||||
|
* 跳过块注释 slash-star ... star-slash
|
||||||
|
*
|
||||||
|
* v0.8.0 根治:未闭合的块注释必须**报错**。
|
||||||
|
* 此前循环到 EOF 就直接返回、不抛错,于是 `DELETE FROM t WHERE id = '4' /*`
|
||||||
|
* 会**照常执行删除**(审计实测真的删掉了 1 行);SQLite 会报
|
||||||
|
* `unterminated /* comment`。任何被截断/拼接的 SQL 都会因此静默改变语义。
|
||||||
|
*/
|
||||||
private skipBlockComment(): void {
|
private skipBlockComment(): void {
|
||||||
this.readChar(); // skip *
|
const start = this.position;
|
||||||
this.readChar(); // move past *
|
this.readChar(); // skip '/'
|
||||||
|
this.readChar(); // skip '*'
|
||||||
while (this.ch !== '' && !(this.ch === '*' && this.peekChar() === '/')) {
|
while (this.ch !== '' && !(this.ch === '*' && this.peekChar() === '/')) {
|
||||||
this.readChar();
|
this.readChar();
|
||||||
}
|
}
|
||||||
if (this.ch !== '') {
|
if (this.ch === '') {
|
||||||
this.readChar(); // skip *
|
throw new DatabaseError(
|
||||||
this.readChar(); // skip /
|
`Unterminated block comment at position ${start}`,
|
||||||
|
'PARSE_ERROR',
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
this.readChar(); // skip '*'
|
||||||
|
this.readChar(); // skip '/'
|
||||||
}
|
}
|
||||||
|
|
||||||
private readIdentifier(): string {
|
private readIdentifier(): string {
|
||||||
@@ -191,29 +223,42 @@ export class Lexer {
|
|||||||
return this.input.slice(start, this.position);
|
return this.input.slice(start, this.position);
|
||||||
}
|
}
|
||||||
|
|
||||||
private readString(quote: string): Token {
|
/**
|
||||||
const start = this.position + 1; // 跳过一个引号
|
* 读取单引号字符串字面量。
|
||||||
|
*
|
||||||
|
* v0.8.0 根治:**移除反斜杠转义**(此前只支持单引号前的反斜杠)。
|
||||||
|
*
|
||||||
|
* 此前只识别「反斜杠 + 单引号」(MySQL 方言的半个实现):双反斜杠不解转义,于是
|
||||||
|
* - 单引号前的反斜杠被静默吞掉;
|
||||||
|
* - 以反斜杠结尾的 Windows 路径会**吞掉闭引号**,抛出一条
|
||||||
|
* 与用户输入无关的 "Unterminated string literal";
|
||||||
|
* - 更严重的是,参数绑定器 params.ts 只把单引号翻倍、不处理反斜杠,
|
||||||
|
* 两份词法规则不一致 —— "参数不可能改变 SQL 结构"这条不变量在文本层已不成立。
|
||||||
|
*
|
||||||
|
* SQL 标准(以及 SQLite / PostgreSQL,本项目对齐的方言)中反斜杠是普通字符,
|
||||||
|
* 反斜杠本身是普通字符、引号靠两个连续单引号表达。移除方言转义后,词法器与绑定器的字符串边界判定
|
||||||
|
* 完全一致,且任意以反斜杠结尾的参数值都能正确绑定。
|
||||||
|
*/
|
||||||
|
private readString(): Token {
|
||||||
|
// v0.8.0: start = **开引号**的位置(此前 +1 指向引号之内,导致所有按
|
||||||
|
// position 切片的调用方都多切一个字符)。未闭合错误消息里用的 start 仍取
|
||||||
|
// 引号之后的位置,便于用户定位到内容起点。
|
||||||
|
const start = this.position;
|
||||||
|
const contentStart = this.position + 1;
|
||||||
this.readChar(); // 跳过开始引号
|
this.readChar(); // 跳过开始引号
|
||||||
let value = '';
|
let value = '';
|
||||||
|
|
||||||
while (this.ch !== '') {
|
while (this.ch !== '') {
|
||||||
if (this.ch === quote) {
|
if (this.ch === "'") {
|
||||||
// v0.3.3: 支持 SQL 标准 '' 转义(两个连续引号 = 一个引号)
|
// SQL 标准 '' 转义(两个连续引号 = 一个引号)
|
||||||
if (this.peekChar() === quote) {
|
if (this.peekChar() === "'") {
|
||||||
value += quote;
|
value += "'";
|
||||||
this.readChar(); // 跳过第二个引号
|
this.readChar(); // 跳过第二个引号
|
||||||
this.readChar();
|
this.readChar();
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
break; // 结束引号(由 nextToken 的 readChar 跳过)
|
break; // 结束引号(由 nextToken 的 readChar 跳过)
|
||||||
}
|
}
|
||||||
// 反斜杠转义(兼容旧语法)
|
|
||||||
if (this.ch === '\\' && this.peekChar() === quote) {
|
|
||||||
this.readChar();
|
|
||||||
value += quote;
|
|
||||||
this.readChar();
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
value += this.ch;
|
value += this.ch;
|
||||||
this.readChar();
|
this.readChar();
|
||||||
}
|
}
|
||||||
@@ -222,7 +267,7 @@ export class Lexer {
|
|||||||
// 上层可解析出错误结果,如 `SELECT 'abc` 被当作合法常量列)
|
// 上层可解析出错误结果,如 `SELECT 'abc` 被当作合法常量列)
|
||||||
if (this.ch === '') {
|
if (this.ch === '') {
|
||||||
throw new DatabaseError(
|
throw new DatabaseError(
|
||||||
`Unterminated string literal at position ${start}`,
|
`Unterminated string literal at position ${contentStart}`,
|
||||||
'PARSE_ERROR',
|
'PARSE_ERROR',
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
@@ -234,6 +279,55 @@ export class Lexer {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0: 读取双引号分隔标识符(SQL 标准 `"column"`)。
|
||||||
|
*
|
||||||
|
* 双引号内以 `""` 表示一个双引号字符(与单引号字符串的 `''` 规则对称)。
|
||||||
|
* 未闭合同样显式报错,与字符串字面量保持一致。
|
||||||
|
*/
|
||||||
|
private readQuotedIdentifier(): Token {
|
||||||
|
// v0.8.0: start = 开引号位置(position 语义统一为 token 首字符)
|
||||||
|
const start = this.position;
|
||||||
|
this.readChar(); // 跳过开始引号
|
||||||
|
let value = '';
|
||||||
|
|
||||||
|
while (this.ch !== '') {
|
||||||
|
if (this.ch === '"') {
|
||||||
|
if (this.peekChar() === '"') {
|
||||||
|
value += '"';
|
||||||
|
this.readChar();
|
||||||
|
this.readChar();
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
value += this.ch;
|
||||||
|
this.readChar();
|
||||||
|
}
|
||||||
|
|
||||||
|
if (this.ch === '') {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Unterminated quoted identifier at position ${start}`,
|
||||||
|
'PARSE_ERROR',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (value.length === 0) {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Empty quoted identifier at position ${start}`,
|
||||||
|
'PARSE_ERROR',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
type: TokenType.QUOTED_IDENTIFIER,
|
||||||
|
value,
|
||||||
|
// v0.8.0: 与 STRING 一致 —— position = **开引号**的位置(按 position 切片
|
||||||
|
// 才能取到完整的分隔标识符文本)
|
||||||
|
position: start,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
private isLetter(ch: string): boolean {
|
private isLetter(ch: string): boolean {
|
||||||
return /[a-zA-Z_]/.test(ch);
|
return /[a-zA-Z_]/.test(ch);
|
||||||
}
|
}
|
||||||
@@ -242,8 +336,8 @@ export class Lexer {
|
|||||||
return /[0-9]/.test(ch);
|
return /[0-9]/.test(ch);
|
||||||
}
|
}
|
||||||
|
|
||||||
private makeToken(type: TokenType, value: string): Token {
|
private makeToken(type: TokenType, value: string, start: number = this.position): Token {
|
||||||
return { type, value, position: this.position };
|
return { type, value, position: start };
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+261
-42
@@ -25,7 +25,7 @@ import type {
|
|||||||
CommitTransactionStatement,
|
CommitTransactionStatement,
|
||||||
ASTColumnDef,
|
ASTColumnDef,
|
||||||
} from '../query/ast';
|
} from '../query/ast';
|
||||||
import type { WhereCondition, OrderBy, SortDirection } from '../constants';
|
import type { WhereCondition, FieldCondition, OrderBy, SortDirection } from '../constants';
|
||||||
import { DatabaseError } from '../constants';
|
import { DatabaseError } from '../constants';
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -354,7 +354,7 @@ export class Parser {
|
|||||||
if (this.curTokenIs(TokenType.GROUP)) {
|
if (this.curTokenIs(TokenType.GROUP)) {
|
||||||
this.nextToken();
|
this.nextToken();
|
||||||
this.expect(TokenType.BY);
|
this.expect(TokenType.BY);
|
||||||
stmt.groupBy = this.parseIdentifierList();
|
stmt.groupBy = this.parseGroupByList();
|
||||||
}
|
}
|
||||||
|
|
||||||
// HAVING(可选)
|
// HAVING(可选)
|
||||||
@@ -401,6 +401,7 @@ export class Parser {
|
|||||||
const right = this.parseSelect();
|
const right = this.parseSelect();
|
||||||
|
|
||||||
const unionStmt: SelectUnionStatement = { type: 'SELECT_UNION', left, right, all: all || undefined };
|
const unionStmt: SelectUnionStatement = { type: 'SELECT_UNION', left, right, all: all || undefined };
|
||||||
|
this.adoptTrailingClauses(unionStmt, right);
|
||||||
// 链式 UNION
|
// 链式 UNION
|
||||||
if (this.curTokenIs(TokenType.UNION)) {
|
if (this.curTokenIs(TokenType.UNION)) {
|
||||||
return this.parseUnionChain(unionStmt);
|
return this.parseUnionChain(unionStmt);
|
||||||
@@ -418,12 +419,46 @@ export class Parser {
|
|||||||
}
|
}
|
||||||
const right = this.parseSelect();
|
const right = this.parseSelect();
|
||||||
const unionStmt: SelectUnionStatement = { type: 'SELECT_UNION', left, right, all: all || undefined };
|
const unionStmt: SelectUnionStatement = { type: 'SELECT_UNION', left, right, all: all || undefined };
|
||||||
|
this.adoptTrailingClauses(unionStmt, right);
|
||||||
if (this.curTokenIs(TokenType.UNION)) {
|
if (this.curTokenIs(TokenType.UNION)) {
|
||||||
return this.parseUnionChain(unionStmt);
|
return this.parseUnionChain(unionStmt);
|
||||||
}
|
}
|
||||||
return unionStmt;
|
return unionStmt;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0(A26):把"最后一个 SELECT 上的 ORDER BY / LIMIT / OFFSET"上移到
|
||||||
|
* 复合查询节点,并把这些子句从该 SELECT 上**移除**。
|
||||||
|
*
|
||||||
|
* 为什么必须"移动"而不是"复制":
|
||||||
|
* - 语法上它们写在最后一个 SELECT 之后,但 SQL 语义作用于整个 UNION
|
||||||
|
* (`A UNION B LIMIT 3` 是"合并去重后取前 3 行",不是"B 取前 3 行");
|
||||||
|
* - 若只复制不移除,LIMIT 会**应用两次** —— 正是 A5/A6 那类"两处都生效"
|
||||||
|
* 缺陷的同一个坑(B 先被截断,再对合并结果截断,结果可能少行)。
|
||||||
|
*
|
||||||
|
* 由于 `parseSelect` 无法预知后面有没有 UNION(它在返回后才知道),
|
||||||
|
* 只能先让它照常解析、发现 UNION 时再回收 —— 这比"预读 UNION"简单且无回溯。
|
||||||
|
*/
|
||||||
|
private adoptTrailingClauses(
|
||||||
|
unionStmt: SelectUnionStatement,
|
||||||
|
right: SelectStatement | SelectUnionStatement,
|
||||||
|
): void {
|
||||||
|
// 链式 UNION 时右侧可能已是 UNION 节点,其尾部子句在创建时已上移
|
||||||
|
if (right.type !== 'SELECT') return;
|
||||||
|
if (right.orderBy) {
|
||||||
|
unionStmt.orderBy = right.orderBy;
|
||||||
|
delete right.orderBy;
|
||||||
|
}
|
||||||
|
if (right.limit !== undefined) {
|
||||||
|
unionStmt.limit = right.limit;
|
||||||
|
delete right.limit;
|
||||||
|
}
|
||||||
|
if (right.offset !== undefined) {
|
||||||
|
unionStmt.offset = right.offset;
|
||||||
|
delete right.offset;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/** 解析 JOIN 子句列表 */
|
/** 解析 JOIN 子句列表 */
|
||||||
private parseJoinClauses(): import('../query/ast').JoinClause[] {
|
private parseJoinClauses(): import('../query/ast').JoinClause[] {
|
||||||
const joins: import('../query/ast').JoinClause[] = [];
|
const joins: import('../query/ast').JoinClause[] = [];
|
||||||
@@ -551,6 +586,26 @@ export class Parser {
|
|||||||
values.push(rowValues);
|
values.push(rowValues);
|
||||||
} while (this.curTokenIs(TokenType.COMMA));
|
} while (this.curTokenIs(TokenType.COMMA));
|
||||||
|
|
||||||
|
// v0.8.0 根治:显式列名时校验每行值的个数与列数一致。
|
||||||
|
//
|
||||||
|
// 此前完全不校验 arity,实测:
|
||||||
|
// INSERT INTO t (id, name) VALUES ('4','z',9) → 多余的 9 被**静默丢弃**
|
||||||
|
// INSERT INTO t VALUES ('3') → 静默写入半行(其余列缺失)
|
||||||
|
// SQLite / MySQL 都会报错。静默丢弃/截断属于"静默数据丢失",
|
||||||
|
// 必须在解析期拦下(此时无需 schema,只要有显式列名即可判断)。
|
||||||
|
//
|
||||||
|
// 未显式给列名时(INSERT INTO t VALUES (...))需要 schema 才能判断个数,
|
||||||
|
// 由 executor 在拿到 schema 后校验(见 validateInsertArity)。
|
||||||
|
if (columns) {
|
||||||
|
for (let i = 0; i < values.length; i++) {
|
||||||
|
if (values[i].length !== columns.length) {
|
||||||
|
throw this.error(
|
||||||
|
`INSERT column/value count mismatch: ${columns.length} column(s) but row ${i + 1} has ${values[i].length} value(s)`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
return {
|
return {
|
||||||
type: 'INSERT',
|
type: 'INSERT',
|
||||||
into: tableName,
|
into: tableName,
|
||||||
@@ -563,13 +618,25 @@ export class Parser {
|
|||||||
// UPDATE
|
// UPDATE
|
||||||
// ===================================================================
|
// ===================================================================
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0: 创建**无原型**对象,用于以用户提供的列名为键的映射。
|
||||||
|
*
|
||||||
|
* 背景:`obj['__proto__'] = v` 在普通对象上会触发原型 setter 而不是新增属性,
|
||||||
|
* 于是 `UPDATE t SET __proto__ = 'x'` 的 sets 变成 `{}` —— 既没写进去、也不会被
|
||||||
|
* v0.7.4 新增的"未知列显式报错"预检看到,表现为"返回成功但什么都没发生"。
|
||||||
|
* 建表路径在 v0.7.1 已用 Object.create(null) 防护,此处补齐其余路径。
|
||||||
|
*/
|
||||||
|
private newColumnMap<T>(): Record<string, T> {
|
||||||
|
return Object.create(null) as Record<string, T>;
|
||||||
|
}
|
||||||
|
|
||||||
private parseUpdate(): UpdateStatement {
|
private parseUpdate(): UpdateStatement {
|
||||||
this.expect(TokenType.UPDATE);
|
this.expect(TokenType.UPDATE);
|
||||||
const tableName = this.expectIdentifier('table name');
|
const tableName = this.expectIdentifier('table name');
|
||||||
this.expect(TokenType.SET);
|
this.expect(TokenType.SET);
|
||||||
|
|
||||||
// SET col=val, ...
|
// SET col=val, ...(v0.8.0: 无原型对象,防 __proto__ 列名静默吞掉赋值)
|
||||||
const sets: Record<string, unknown> = {};
|
const sets: Record<string, unknown> = this.newColumnMap<unknown>();
|
||||||
do {
|
do {
|
||||||
if (this.curTokenIs(TokenType.COMMA)) this.nextToken();
|
if (this.curTokenIs(TokenType.COMMA)) this.nextToken();
|
||||||
const col = this.expectIdentifier('column name');
|
const col = this.expectIdentifier('column name');
|
||||||
@@ -577,7 +644,7 @@ export class Parser {
|
|||||||
sets[col] = this.parseValue();
|
sets[col] = this.parseValue();
|
||||||
} while (this.curTokenIs(TokenType.COMMA));
|
} while (this.curTokenIs(TokenType.COMMA));
|
||||||
|
|
||||||
let where: WhereCondition = {};
|
let where: WhereCondition = this.newColumnMap<unknown>() as WhereCondition;
|
||||||
if (this.curTokenIs(TokenType.WHERE)) {
|
if (this.curTokenIs(TokenType.WHERE)) {
|
||||||
this.nextToken();
|
this.nextToken();
|
||||||
where = this.parseCondition();
|
where = this.parseCondition();
|
||||||
@@ -595,7 +662,7 @@ export class Parser {
|
|||||||
this.expect(TokenType.FROM);
|
this.expect(TokenType.FROM);
|
||||||
const tableName = this.expectIdentifier('table name');
|
const tableName = this.expectIdentifier('table name');
|
||||||
|
|
||||||
let where: WhereCondition = {};
|
let where: WhereCondition = this.newColumnMap<FieldCondition>() as WhereCondition;
|
||||||
if (this.curTokenIs(TokenType.WHERE)) {
|
if (this.curTokenIs(TokenType.WHERE)) {
|
||||||
this.nextToken();
|
this.nextToken();
|
||||||
where = this.parseCondition();
|
where = this.parseCondition();
|
||||||
@@ -776,24 +843,48 @@ export class Parser {
|
|||||||
// 条件表达式
|
// 条件表达式
|
||||||
// ===================================================================
|
// ===================================================================
|
||||||
|
|
||||||
/** condition → simple_cond ((AND|OR) simple_cond)* */
|
/**
|
||||||
|
* condition → or_expr
|
||||||
|
*
|
||||||
|
* v0.8.0 根治:AND 的优先级必须高于 OR(SQL 标准)。
|
||||||
|
*
|
||||||
|
* 此前实现是**纯左折叠**的单层循环:
|
||||||
|
* `a = 1 OR a = 2 AND b = 3` → `(a = 1 OR a = 2) AND b = 3` ← 错
|
||||||
|
* 标准语义应为:
|
||||||
|
* `a = 1 OR a = 2 AND b = 3` → `a = 1 OR (a = 2 AND b = 3)` ← 对
|
||||||
|
*
|
||||||
|
* 影响面:任何"权限条件 OR 业务条件 AND 软删标记"的写法都会静默返回错误行集
|
||||||
|
* (审计实测:4 行表上返回 1 行而非 3 行)。这是本层影响面最大、改动最小的缺陷。
|
||||||
|
*
|
||||||
|
* 现在按标准文法分层:or_expr → and_expr (OR and_expr)*
|
||||||
|
* and_expr → unary (AND unary)*
|
||||||
|
* unary → [NOT] primary
|
||||||
|
* 并且只在**确实有多个操作数**时才包 $and/$or,避免生成 {$and:[x]} 这种冗余节点
|
||||||
|
* (否则 `WHERE a = 1` 的结构会从 `{a:{$eq:1}}` 变成 `{$and:[{a:{$eq:1}}]}`,
|
||||||
|
* 破坏既有 AST 契约与下游引擎的索引下推识别)。
|
||||||
|
*/
|
||||||
private parseCondition(): WhereCondition {
|
private parseCondition(): WhereCondition {
|
||||||
let left = this.parseSimpleCondition();
|
return this.parseOrExpression();
|
||||||
|
}
|
||||||
|
|
||||||
while (this.curTokenIs(TokenType.AND) || this.curTokenIs(TokenType.OR)) {
|
/** or_expr → and_expr (OR and_expr)* */
|
||||||
const isAnd = this.curTokenIs(TokenType.AND);
|
private parseOrExpression(): WhereCondition {
|
||||||
|
const operands: WhereCondition[] = [this.parseAndExpression()];
|
||||||
|
while (this.curTokenIs(TokenType.OR)) {
|
||||||
this.nextToken();
|
this.nextToken();
|
||||||
const right = this.parseSimpleCondition();
|
operands.push(this.parseAndExpression());
|
||||||
|
|
||||||
if (isAnd) {
|
|
||||||
// 合并到 $and
|
|
||||||
left = { $and: [left, right] } as unknown as WhereCondition;
|
|
||||||
} else {
|
|
||||||
left = { $or: [left, right] } as unknown as WhereCondition;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
return operands.length === 1 ? operands[0] : ({ $or: operands } as unknown as WhereCondition);
|
||||||
|
}
|
||||||
|
|
||||||
return left;
|
/** and_expr → simple_cond (AND simple_cond)* */
|
||||||
|
private parseAndExpression(): WhereCondition {
|
||||||
|
const operands: WhereCondition[] = [this.parseSimpleCondition()];
|
||||||
|
while (this.curTokenIs(TokenType.AND)) {
|
||||||
|
this.nextToken();
|
||||||
|
operands.push(this.parseSimpleCondition());
|
||||||
|
}
|
||||||
|
return operands.length === 1 ? operands[0] : ({ $and: operands } as unknown as WhereCondition);
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 公共 WHERE 条件入口(供 CASE WHEN 求值等外部场景,v0.3.1) */
|
/** 公共 WHERE 条件入口(供 CASE WHEN 求值等外部场景,v0.3.1) */
|
||||||
@@ -841,8 +932,12 @@ export class Parser {
|
|||||||
const isNot = this.curTokenIs(TokenType.NOT);
|
const isNot = this.curTokenIs(TokenType.NOT);
|
||||||
if (isNot) this.nextToken();
|
if (isNot) this.nextToken();
|
||||||
this.expect(TokenType.NULL);
|
this.expect(TokenType.NULL);
|
||||||
const result: WhereCondition = {};
|
const result: WhereCondition = this.newColumnMap<FieldCondition>() as WhereCondition;
|
||||||
result[column] = isNot ? { $ne: null } : { $eq: null };
|
// v0.8.0 三值语义:IS NULL / IS NOT NULL 是**谓词**,不是等值比较。
|
||||||
|
// 此前生成为 { $eq: null } / { $ne: null } —— 在 SQL 标准里
|
||||||
|
// `x = NULL` 恒为 UNKNOWN(不保留任何行),而 IS NULL 要保留 NULL 行。
|
||||||
|
// 两者语义不同,必须用不同标记(见 query/sql-compare.ts 的谓词说明)。
|
||||||
|
result[column] = isNot ? { $isNotNull: true } : { $isNull: true };
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -852,7 +947,12 @@ export class Parser {
|
|||||||
const low = this.parseValue();
|
const low = this.parseValue();
|
||||||
this.expect(TokenType.AND);
|
this.expect(TokenType.AND);
|
||||||
const high = this.parseValue();
|
const high = this.parseValue();
|
||||||
const result: WhereCondition = {};
|
// v0.8.0 根治:BETWEEN 必须是**范围**条件。
|
||||||
|
// 此前把同一个对象同时当成"操作符对象"和"操作数"传给 `$eq`
|
||||||
|
// (`{ $eq: { $gte, $lte } }`),matchOperator 的 `$eq` 收到一个对象再去比较,
|
||||||
|
// 结果只对"值恰好等于该对象"的行成立 —— 实测 `WHERE n BETWEEN 1 AND 2`
|
||||||
|
// 在 (1,2,NULL,3) 上只返回 1 行(应 2 行),静默错值。
|
||||||
|
const result: WhereCondition = this.newColumnMap<FieldCondition>() as WhereCondition;
|
||||||
result[column] = { $gte: low, $lte: high };
|
result[column] = { $gte: low, $lte: high };
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
@@ -864,8 +964,19 @@ export class Parser {
|
|||||||
const low = this.parseValue();
|
const low = this.parseValue();
|
||||||
this.expect(TokenType.AND);
|
this.expect(TokenType.AND);
|
||||||
const high = this.parseValue();
|
const high = this.parseValue();
|
||||||
const result: WhereCondition = {};
|
const result: WhereCondition = this.newColumnMap<FieldCondition>() as WhereCondition;
|
||||||
result[column] = { $not: { $gte: low, $lte: high } };
|
// v0.8.0:NOT BETWEEN ≡ (x < low) OR (x > high),直接展开为字段级 `$or`。
|
||||||
|
//
|
||||||
|
// 此前生成 `{ $not: { $gte, $lte } }`。这**曾经**恒为空集:字段级 `$not`
|
||||||
|
// 把内层对象当"单个条件对象",key `$gte`/`$lte` 被当成列名去取
|
||||||
|
// `row['$gte']` → undefined → 整条恒 UNKNOWN → 取反仍 UNKNOWN → 全部排除。
|
||||||
|
//
|
||||||
|
// 求值器已修正(内层按操作符对象解释),但这里仍选择展开为 `$or`:
|
||||||
|
// - `$or` 的三值行为(含 NULL 时 UNKNOWN)是显式可读的;
|
||||||
|
// - 避免依赖"$not 作用于比较"与"$not 作用于谓词"(如 `$isNull`)的差别。
|
||||||
|
// 两条路径都有测试锁定(tests/v080-sql-three-valued.test.ts 与
|
||||||
|
// tests/sql/where-matcher.test.ts 的 `$not` 用例)。
|
||||||
|
result[column] = { $or: [{ $lt: low } as never, { $gt: high } as never] as never };
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -879,13 +990,13 @@ export class Parser {
|
|||||||
if (this.curTokenIs(TokenType.SELECT)) {
|
if (this.curTokenIs(TokenType.SELECT)) {
|
||||||
const subquery = this.parseSelect();
|
const subquery = this.parseSelect();
|
||||||
this.expect(TokenType.RPAREN);
|
this.expect(TokenType.RPAREN);
|
||||||
const result: WhereCondition = {};
|
const result: WhereCondition = this.newColumnMap<FieldCondition>() as WhereCondition;
|
||||||
result[column] = { $nin: { $subquery: subquery } };
|
result[column] = { $nin: { $subquery: subquery } };
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
const values = this.parseValueList();
|
const values = this.parseValueList();
|
||||||
this.expect(TokenType.RPAREN);
|
this.expect(TokenType.RPAREN);
|
||||||
const result: WhereCondition = {};
|
const result: WhereCondition = this.newColumnMap<FieldCondition>() as WhereCondition;
|
||||||
result[column] = { $nin: values };
|
result[column] = { $nin: values };
|
||||||
return result;
|
return result;
|
||||||
} else if (this.peekTokenIs(TokenType.LIKE)) {
|
} else if (this.peekTokenIs(TokenType.LIKE)) {
|
||||||
@@ -893,7 +1004,7 @@ export class Parser {
|
|||||||
this.nextToken(); // skip NOT
|
this.nextToken(); // skip NOT
|
||||||
this.nextToken(); // skip LIKE
|
this.nextToken(); // skip LIKE
|
||||||
const pattern = this.parseValue();
|
const pattern = this.parseValue();
|
||||||
const result: WhereCondition = {};
|
const result: WhereCondition = this.newColumnMap<FieldCondition>() as WhereCondition;
|
||||||
result[column] = { $not: { $like: pattern } };
|
result[column] = { $not: { $like: pattern } };
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
@@ -903,7 +1014,7 @@ export class Parser {
|
|||||||
if (this.curTokenIs(TokenType.LIKE)) {
|
if (this.curTokenIs(TokenType.LIKE)) {
|
||||||
this.nextToken();
|
this.nextToken();
|
||||||
const pattern = this.parseValue();
|
const pattern = this.parseValue();
|
||||||
const result: WhereCondition = {};
|
const result: WhereCondition = this.newColumnMap<FieldCondition>() as WhereCondition;
|
||||||
result[column] = { $like: pattern };
|
result[column] = { $like: pattern };
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
@@ -916,13 +1027,13 @@ export class Parser {
|
|||||||
if (this.curTokenIs(TokenType.SELECT)) {
|
if (this.curTokenIs(TokenType.SELECT)) {
|
||||||
const subquery = this.parseSelect();
|
const subquery = this.parseSelect();
|
||||||
this.expect(TokenType.RPAREN);
|
this.expect(TokenType.RPAREN);
|
||||||
const result: WhereCondition = {};
|
const result: WhereCondition = this.newColumnMap<FieldCondition>() as WhereCondition;
|
||||||
result[column] = { $in: { $subquery: subquery } };
|
result[column] = { $in: { $subquery: subquery } };
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
const values = this.parseValueList();
|
const values = this.parseValueList();
|
||||||
this.expect(TokenType.RPAREN);
|
this.expect(TokenType.RPAREN);
|
||||||
const result: WhereCondition = {};
|
const result: WhereCondition = this.newColumnMap<FieldCondition>() as WhereCondition;
|
||||||
result[column] = { $in: values };
|
result[column] = { $in: values };
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
@@ -933,7 +1044,7 @@ export class Parser {
|
|||||||
this.curTokenIs(TokenType.RPAREN) || this.curTokenIs(TokenType.EOF) ||
|
this.curTokenIs(TokenType.RPAREN) || this.curTokenIs(TokenType.EOF) ||
|
||||||
(this.curToken.type === TokenType.IDENTIFIER && ['THEN', 'END', 'ELSE', 'NULLS', 'LIMIT', 'OFFSET', 'ORDER', 'GROUP', 'HAVING', 'UNION', 'WHERE'].includes(this.curToken.value.toUpperCase()))
|
(this.curToken.type === TokenType.IDENTIFIER && ['THEN', 'END', 'ELSE', 'NULLS', 'LIMIT', 'OFFSET', 'ORDER', 'GROUP', 'HAVING', 'UNION', 'WHERE'].includes(this.curToken.value.toUpperCase()))
|
||||||
) {
|
) {
|
||||||
const result: WhereCondition = {};
|
const result: WhereCondition = this.newColumnMap<FieldCondition>() as WhereCondition;
|
||||||
result[column] = { $eq: true };
|
result[column] = { $eq: true };
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
@@ -946,28 +1057,54 @@ export class Parser {
|
|||||||
this.nextToken(); // skip (
|
this.nextToken(); // skip (
|
||||||
const subquery = this.parseSelect();
|
const subquery = this.parseSelect();
|
||||||
this.expect(TokenType.RPAREN);
|
this.expect(TokenType.RPAREN);
|
||||||
const result: WhereCondition = {};
|
const result: WhereCondition = this.newColumnMap<FieldCondition>() as WhereCondition;
|
||||||
result[column] = { [op]: { $subquery: subquery } };
|
result[column] = { [op]: { $subquery: subquery } };
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
|
|
||||||
// 尝试解析列引用(identifier DOT identifier 格式)
|
// 解析比较运算符右侧的操作数(v0.8.0: 支持列引用)
|
||||||
|
//
|
||||||
|
// 此前只有 `a.b`(带点)形态被当作列引用,裸标识符一律走 parseValue →
|
||||||
|
// `WHERE x = y` 抛 PARSE_ERROR("Expected value, got \"y\"")。于是列对列比较
|
||||||
|
// 必须写成 `WHERE t.x = t.y`,而多表 JOIN 里未限定列恰恰是最自然的写法
|
||||||
|
// (`ON k = k` 同样报错)。
|
||||||
|
//
|
||||||
|
// 判定依据是**位置**而不是猜测:能走到这里说明运算符已被消费,当前 token
|
||||||
|
// 就是右侧操作数。这个位置上除了列名只可能是字面量,而字面量有各自的
|
||||||
|
// token 类型(NUMBER / STRING / TRUE / FALSE / NULL)—— 因此
|
||||||
|
// **IDENTIFIER 在操作数位置必然是列引用**,没有别的可能:
|
||||||
|
// `id = 5` / `id = 'a'` / `active = TRUE` / `x = NULL` → 都不是 IDENTIFIER ✓
|
||||||
|
// `x = y` → y 是 IDENTIFIER → 列引用 ✓
|
||||||
|
//
|
||||||
|
// 拼错的列名(`WHERE id = nmae`)因此变成 **COLUMN_NOT_FOUND**(列不存在)而不是
|
||||||
|
// PARSE_ERROR —— 这是更准确的诊断:问题不是"语法不对",而是"没有这一列"。
|
||||||
let value: unknown;
|
let value: unknown;
|
||||||
if (
|
if (this.curToken.type === TokenType.QUOTED_IDENTIFIER) {
|
||||||
(this.curToken.type === TokenType.IDENTIFIER || this._isKeywordAsIdent()) &&
|
// 分隔标识符("col")在操作数位置同样是列引用
|
||||||
this.peekTokenIs(TokenType.DOT)
|
value = { $col: this.expectColumnReference() };
|
||||||
) {
|
} else if (this.curToken.type === TokenType.IDENTIFIER && !this._isReservedKeywordToken()) {
|
||||||
const colRef = this.parseColumnRef();
|
value = { $col: this.expectColumnReference() };
|
||||||
value = { $col: colRef };
|
|
||||||
} else {
|
} else {
|
||||||
value = this.parseValue();
|
value = this.parseValue();
|
||||||
}
|
}
|
||||||
|
|
||||||
const result: WhereCondition = {};
|
const result: WhereCondition = this.newColumnMap<FieldCondition>() as WhereCondition;
|
||||||
result[column] = { [op]: value };
|
result[column] = { [op]: value };
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0:当前 token 是否"看起来像标识符但其实是关键字"。
|
||||||
|
*
|
||||||
|
* 操作数位置上的 IDENTIFIER 必然是列引用(见上),但 `_isKeywordAsIdent()` 为真
|
||||||
|
* 的那些 token 是**关键字**(它们可能作为列名出现在别处,但不会作为比较的右操作数
|
||||||
|
* 出现)—— 例如 `WHERE a = AND` 之类的非法输入应当继续报 PARSE_ERROR,
|
||||||
|
* 而不是被当成列名再去查表。
|
||||||
|
*/
|
||||||
|
private _isReservedKeywordToken(): boolean {
|
||||||
|
return this.curToken.type !== TokenType.IDENTIFIER && this._isKeywordAsIdent();
|
||||||
|
}
|
||||||
|
|
||||||
/** 解析 EXISTS (SELECT ...) / NOT EXISTS (SELECT ...) */
|
/** 解析 EXISTS (SELECT ...) / NOT EXISTS (SELECT ...) */
|
||||||
private parseExistsCondition(negate: boolean): WhereCondition {
|
private parseExistsCondition(negate: boolean): WhereCondition {
|
||||||
this.expect(TokenType.LPAREN);
|
this.expect(TokenType.LPAREN);
|
||||||
@@ -992,6 +1129,20 @@ export class Parser {
|
|||||||
return this.peekToken.type === type;
|
return this.peekToken.type === type;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 解析一个列引用:`a` 或 `a.b`(v0.8.0 支持未限定形态)。
|
||||||
|
* 与 `parseColumnRef` 的区别:后者还接受 CASE/数字/字符串字面量(SELECT 列表用),
|
||||||
|
* 这里只接受真正的列名。
|
||||||
|
*/
|
||||||
|
private expectColumnReference(): string {
|
||||||
|
const col = this.expectIdentifier('column reference');
|
||||||
|
if (this.curTokenIs(TokenType.DOT)) {
|
||||||
|
this.nextToken();
|
||||||
|
return `${col}.${this.expectIdentifier('column name after "."')}`;
|
||||||
|
}
|
||||||
|
return col;
|
||||||
|
}
|
||||||
|
|
||||||
private parseComparisonOp(): string {
|
private parseComparisonOp(): string {
|
||||||
switch (this.curToken.type) {
|
switch (this.curToken.type) {
|
||||||
case TokenType.EQ: this.nextToken(); return '$eq';
|
case TokenType.EQ: this.nextToken(); return '$eq';
|
||||||
@@ -1042,6 +1193,9 @@ export class Parser {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// 数字常量列:SELECT 1 FROM t(常见于 EXISTS 子查询)
|
// 数字常量列:SELECT 1 FROM t(常见于 EXISTS 子查询)
|
||||||
|
// v0.8.0: 只返回常量文本,别名交给调用方 parseColumnWithAlias 处理
|
||||||
|
//(此前这里直接 return,导致 `SELECT 1 AS one` 的别名被丢弃,
|
||||||
|
// 投影时 row['1'] → undefined → 整行变成 {})。
|
||||||
if (this.curTokenIs(TokenType.NUMBER)) {
|
if (this.curTokenIs(TokenType.NUMBER)) {
|
||||||
const value = this.curToken.value;
|
const value = this.curToken.value;
|
||||||
this.nextToken();
|
this.nextToken();
|
||||||
@@ -1049,10 +1203,11 @@ export class Parser {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// v0.4.0: 字符串常量列:SELECT 'value' FROM t
|
// v0.4.0: 字符串常量列:SELECT 'value' FROM t
|
||||||
|
// v0.8.0: 同样只返回字面量文本,别名由 parseColumnWithAlias 叠加
|
||||||
if (this.curTokenIs(TokenType.STRING)) {
|
if (this.curTokenIs(TokenType.STRING)) {
|
||||||
const value = this.curToken.value;
|
const value = this.curToken.value;
|
||||||
this.nextToken();
|
this.nextToken();
|
||||||
return `'${value}'`;
|
return `'${value.replace(/'/g, "''")}'`;
|
||||||
}
|
}
|
||||||
|
|
||||||
// 聚合函数?
|
// 聚合函数?
|
||||||
@@ -1066,6 +1221,25 @@ export class Parser {
|
|||||||
return this.parseAggregateCall();
|
return this.parseAggregateCall();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0(B-5):分隔标识符必须**保留引号**。
|
||||||
|
*
|
||||||
|
* 此前 `SELECT "1" FROM q`(列名就叫 `1`,建表时用引号声明)被解析成裸 `1`,
|
||||||
|
* 而投影阶段把裸数字当**常量**列 → 返回 `{"1": 1}`(字面量 1),
|
||||||
|
* 而 `SELECT *` 返回正确的 `{"1": "z"}` —— 同一列两种结论(实测)。
|
||||||
|
* 保留引号后,下游能区分"名为 1 的列"与"常量 1"。
|
||||||
|
*/
|
||||||
|
if (this.curTokenIs(TokenType.QUOTED_IDENTIFIER)) {
|
||||||
|
const name = this.curToken.value;
|
||||||
|
this.nextToken();
|
||||||
|
if (this.curTokenIs(TokenType.DOT)) {
|
||||||
|
this.nextToken();
|
||||||
|
const second = this.expectIdentifier('column name after "."');
|
||||||
|
return `"${name.replace(/"/g, '""')}".${second}`;
|
||||||
|
}
|
||||||
|
return `"${name.replace(/"/g, '""')}"`;
|
||||||
|
}
|
||||||
|
|
||||||
const first = this.expectIdentifier('column name');
|
const first = this.expectIdentifier('column name');
|
||||||
if (this.curTokenIs(TokenType.DOT)) {
|
if (this.curTokenIs(TokenType.DOT)) {
|
||||||
this.nextToken();
|
this.nextToken();
|
||||||
@@ -1173,6 +1347,42 @@ export class Parser {
|
|||||||
return vals;
|
return vals;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0(B-5):ORDER BY / GROUP BY 的键可以是**输出列序号**(SQL 标准)。
|
||||||
|
*
|
||||||
|
* `ORDER BY 1` 表示"按第 1 个输出列排序",`GROUP BY 2` 同理 ——
|
||||||
|
* 这在手写 SQL 与 UNION 里非常常用(各分支输出列名可能不同,只能按序号引用)。
|
||||||
|
*
|
||||||
|
* 此前 parser 的 `parseIdentifierWithDot` 只接受标识符,于是 `ORDER BY 1`
|
||||||
|
* 直接 `PARSE_ERROR: Expected identifier, got "1"`(实测)。
|
||||||
|
*
|
||||||
|
* 这里把序号**原样保留为数字字符串**(AST 形状不变),由 executor 在拿到
|
||||||
|
* SELECT 列表后再解析成对应表达式 —— 序号的含义依赖 SELECT 列表,
|
||||||
|
* parser 层无从判断。
|
||||||
|
*/
|
||||||
|
private parseOrderOrGroupKey(): string {
|
||||||
|
if (this.curTokenIs(TokenType.NUMBER)) {
|
||||||
|
const value = this.curToken.value;
|
||||||
|
// 序号必须是正整数(`ORDER BY 0` / `ORDER BY 1.5` 非法)
|
||||||
|
if (!/^\d+$/.test(value) || Number(value) < 1) {
|
||||||
|
throw this.error(`Invalid output column ordinal "${value}" (must be a positive integer)`);
|
||||||
|
}
|
||||||
|
this.nextToken();
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
return this.parseIdentifierWithDot();
|
||||||
|
}
|
||||||
|
|
||||||
|
private parseGroupByList(): string[] {
|
||||||
|
const list: string[] = [];
|
||||||
|
list.push(this.parseOrderOrGroupKey());
|
||||||
|
while (this.curTokenIs(TokenType.COMMA)) {
|
||||||
|
this.nextToken();
|
||||||
|
list.push(this.parseOrderOrGroupKey());
|
||||||
|
}
|
||||||
|
return list;
|
||||||
|
}
|
||||||
|
|
||||||
private parseOrderByList(): OrderBy[] {
|
private parseOrderByList(): OrderBy[] {
|
||||||
const list: OrderBy[] = [];
|
const list: OrderBy[] = [];
|
||||||
list.push(this.parseOrderBy());
|
list.push(this.parseOrderBy());
|
||||||
@@ -1184,7 +1394,7 @@ export class Parser {
|
|||||||
}
|
}
|
||||||
|
|
||||||
private parseOrderBy(): OrderBy {
|
private parseOrderBy(): OrderBy {
|
||||||
const column = this.parseIdentifierWithDot();
|
const column = this.parseOrderOrGroupKey();
|
||||||
let direction: SortDirection = 'asc';
|
let direction: SortDirection = 'asc';
|
||||||
if (this.curTokenIs(TokenType.ASC)) {
|
if (this.curTokenIs(TokenType.ASC)) {
|
||||||
this.nextToken();
|
this.nextToken();
|
||||||
@@ -1260,6 +1470,13 @@ export class Parser {
|
|||||||
}
|
}
|
||||||
|
|
||||||
private expectIdentifier(context: string): string {
|
private expectIdentifier(context: string): string {
|
||||||
|
// v0.8.0: 分隔标识符 "col" 与普通标识符等价(但不参与关键字识别,
|
||||||
|
// 因此可以用它引用保留字列名,如 "order" / "select")
|
||||||
|
if (this.curToken.type === TokenType.QUOTED_IDENTIFIER) {
|
||||||
|
const val = this.curToken.value;
|
||||||
|
this.nextToken();
|
||||||
|
return val;
|
||||||
|
}
|
||||||
if (this.curToken.type === TokenType.IDENTIFIER || this._isKeywordAsIdent()) {
|
if (this.curToken.type === TokenType.IDENTIFIER || this._isKeywordAsIdent()) {
|
||||||
const val = this.curToken.value;
|
const val = this.curToken.value;
|
||||||
this.nextToken();
|
this.nextToken();
|
||||||
@@ -1274,6 +1491,8 @@ export class Parser {
|
|||||||
this.curToken.type !== TokenType.EOF &&
|
this.curToken.type !== TokenType.EOF &&
|
||||||
this.curToken.type !== TokenType.ILLEGAL &&
|
this.curToken.type !== TokenType.ILLEGAL &&
|
||||||
this.curToken.type !== TokenType.STRING &&
|
this.curToken.type !== TokenType.STRING &&
|
||||||
|
// v0.8.0: 分隔标识符由 expectIdentifier 的显式分支处理,不走"关键字当标识符"兜底
|
||||||
|
this.curToken.type !== TokenType.QUOTED_IDENTIFIER &&
|
||||||
this.curToken.type !== TokenType.NUMBER &&
|
this.curToken.type !== TokenType.NUMBER &&
|
||||||
this.curToken.type !== TokenType.COMMA &&
|
this.curToken.type !== TokenType.COMMA &&
|
||||||
this.curToken.type !== TokenType.LPAREN &&
|
this.curToken.type !== TokenType.LPAREN &&
|
||||||
|
|||||||
@@ -94,6 +94,17 @@ export enum TokenType {
|
|||||||
|
|
||||||
// 标识符 & 字面量
|
// 标识符 & 字面量
|
||||||
IDENTIFIER = 'IDENTIFIER',
|
IDENTIFIER = 'IDENTIFIER',
|
||||||
|
/**
|
||||||
|
* v0.8.0: 分隔标识符(双引号包裹,SQL 标准 `"name"`)。
|
||||||
|
*
|
||||||
|
* 此前双引号被当作字符串定界符处理,`SELECT "name" FROM t` 会静默产出一个名为
|
||||||
|
* `'name'` 的**常量列**(行数正确、值全错、无任何报错),且该行为被
|
||||||
|
* tests/sql/lexer.test.ts 钉死为期望。现按 SQL 标准区分:
|
||||||
|
* 'x' → STRING(字符串字面量)
|
||||||
|
* "x" → QUOTED_IDENTIFIER(标识符,用于含特殊字符/保留字/大小写敏感的列名)
|
||||||
|
* 双引号内以 "" 表示一个双引号。
|
||||||
|
*/
|
||||||
|
QUOTED_IDENTIFIER = 'QUOTED_IDENTIFIER',
|
||||||
STRING = 'STRING',
|
STRING = 'STRING',
|
||||||
NUMBER = 'NUMBER',
|
NUMBER = 'NUMBER',
|
||||||
|
|
||||||
|
|||||||
+24
-106
@@ -5,6 +5,17 @@
|
|||||||
|
|
||||||
import type { TableSchema, ColumnDef, FieldType } from '../constants';
|
import type { TableSchema, ColumnDef, FieldType } from '../constants';
|
||||||
import { FIELD_TYPES, DatabaseError } from '../constants';
|
import { FIELD_TYPES, DatabaseError } from '../constants';
|
||||||
|
import { compileValidator } from './validation';
|
||||||
|
|
||||||
|
// v0.8.0(B-1):`checkFieldType` 的实现已迁到 `validation.ts`(唯一校验定义),
|
||||||
|
// 这里重新导出以保持公开 API 不变。
|
||||||
|
//
|
||||||
|
// 为什么必须搬走:'schema.ts ↔ validation.ts' 之前是**循环依赖**
|
||||||
|
//(schema 需要 compileValidator;validation 需要 checkFieldType),
|
||||||
|
// rollup 构建时明确告警 'Circular dependency'。循环依赖在 ESM 下的
|
||||||
|
// 求值顺序不稳定(谁先被 import 谁就先初始化),是难查的运行时陷阱;
|
||||||
|
// 依赖方向必须单向:validation(约束实现)← schema(DDL 工具)。
|
||||||
|
export { checkFieldType } from './validation';
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// Schema 工具
|
// Schema 工具
|
||||||
@@ -83,113 +94,20 @@ export function stripUndefinedUpdates(updates: Record<string, unknown>): Record<
|
|||||||
return clean;
|
return clean;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 校验行数据 */
|
/**
|
||||||
|
* 校验行数据(INSERT 语义:`default` 生效、缺列合法)。
|
||||||
|
*
|
||||||
|
* v0.8.0(B-1):实现已迁移到 `table/validation.ts#compileValidator`。
|
||||||
|
* 这里保留同名导出是因为它是**公开 API**(`table/index.ts` 与 `src/index.ts` 转发),
|
||||||
|
* 且历史调用方(含 tests)依赖它 —— 但实现只有一份:委托过去。
|
||||||
|
*
|
||||||
|
* 为什么必须收敛:修复前本项目有**三份**行校验(memory / aria / 本文件),
|
||||||
|
* 覆盖面不同(memory 那份缺 maxLength / min / max)。于是"约束是否生效"取决于
|
||||||
|
* 引擎选择(缺陷 A12),而"未知列"三份都静默丢弃(A17)。
|
||||||
|
* 现在三处都是同一个 `compileValidator`。
|
||||||
|
*/
|
||||||
export function validateRow(schema: TableSchema, row: Record<string, unknown>): Record<string, unknown> {
|
export function validateRow(schema: TableSchema, row: Record<string, unknown>): Record<string, unknown> {
|
||||||
const validated: Record<string, unknown> = {};
|
return compileValidator(schema).validateRow(row);
|
||||||
|
|
||||||
for (const [colName, colDef] of Object.entries(schema.columns)) {
|
|
||||||
let value = row[colName];
|
|
||||||
|
|
||||||
// 默认值
|
|
||||||
if (value === undefined && colDef.default !== undefined) {
|
|
||||||
value = colDef.default;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 必填检查
|
|
||||||
if (colDef.required && (value === undefined || value === null)) {
|
|
||||||
throw new DatabaseError(
|
|
||||||
`Column "${colName}" is required in table "${schema.name}"`,
|
|
||||||
'VALIDATION_ERROR',
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
// 类型检查
|
|
||||||
if (value !== undefined && value !== null) {
|
|
||||||
checkFieldType(schema.name, colName, colDef.type, value, colDef);
|
|
||||||
}
|
|
||||||
|
|
||||||
if (value !== undefined) {
|
|
||||||
validated[colName] = value;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return validated;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** 检查字段类型(含约束校验) */
|
|
||||||
export function checkFieldType(
|
|
||||||
tableName: string,
|
|
||||||
colName: string,
|
|
||||||
type: FieldType,
|
|
||||||
value: unknown,
|
|
||||||
colDef?: ColumnDef,
|
|
||||||
): void {
|
|
||||||
const jsType = typeof value;
|
|
||||||
|
|
||||||
switch (type) {
|
|
||||||
case 'string':
|
|
||||||
if (jsType !== 'string') {
|
|
||||||
throw new DatabaseError(
|
|
||||||
`Column "${colName}" in table "${tableName}" expects string, got ${jsType}`,
|
|
||||||
'TYPE_ERROR',
|
|
||||||
);
|
|
||||||
}
|
|
||||||
if (colDef?.maxLength !== undefined && (value as string).length > colDef.maxLength) {
|
|
||||||
throw new DatabaseError(
|
|
||||||
`Column "${colName}" in table "${tableName}" exceeds max length ${colDef.maxLength}`,
|
|
||||||
'VALIDATION_ERROR',
|
|
||||||
);
|
|
||||||
}
|
|
||||||
break;
|
|
||||||
|
|
||||||
case 'number':
|
|
||||||
if (jsType !== 'number') {
|
|
||||||
throw new DatabaseError(
|
|
||||||
`Column "${colName}" in table "${tableName}" expects number, got ${jsType}`,
|
|
||||||
'TYPE_ERROR',
|
|
||||||
);
|
|
||||||
}
|
|
||||||
if (colDef?.min !== undefined && (value as number) < colDef.min) {
|
|
||||||
throw new DatabaseError(
|
|
||||||
`Column "${colName}" in table "${tableName}" value ${value} below minimum ${colDef.min}`,
|
|
||||||
'VALIDATION_ERROR',
|
|
||||||
);
|
|
||||||
}
|
|
||||||
if (colDef?.max !== undefined && (value as number) > colDef.max) {
|
|
||||||
throw new DatabaseError(
|
|
||||||
`Column "${colName}" in table "${tableName}" value ${value} above maximum ${colDef.max}`,
|
|
||||||
'VALIDATION_ERROR',
|
|
||||||
);
|
|
||||||
}
|
|
||||||
break;
|
|
||||||
|
|
||||||
case 'boolean':
|
|
||||||
if (jsType !== 'boolean') {
|
|
||||||
throw new DatabaseError(
|
|
||||||
`Column "${colName}" in table "${tableName}" expects boolean, got ${jsType}`,
|
|
||||||
'TYPE_ERROR',
|
|
||||||
);
|
|
||||||
}
|
|
||||||
break;
|
|
||||||
|
|
||||||
case 'date':
|
|
||||||
if (jsType !== 'string' || isNaN(Date.parse(value as string))) {
|
|
||||||
throw new DatabaseError(
|
|
||||||
`Column "${colName}" in table "${tableName}" expects valid date string, got ${typeof value}`,
|
|
||||||
'TYPE_ERROR',
|
|
||||||
);
|
|
||||||
}
|
|
||||||
break;
|
|
||||||
|
|
||||||
case 'json':
|
|
||||||
if (jsType !== 'object') {
|
|
||||||
throw new DatabaseError(
|
|
||||||
`Column "${colName}" in table "${tableName}" expects object/array, got ${jsType}`,
|
|
||||||
'TYPE_ERROR',
|
|
||||||
);
|
|
||||||
}
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 将 AST 列定义转换为 ColumnDef */
|
/** 将 AST 列定义转换为 ColumnDef */
|
||||||
|
|||||||
+46
-3
@@ -70,7 +70,25 @@ export class Table<T = Record<string, unknown>> {
|
|||||||
// ---- 查询 ----
|
// ---- 查询 ----
|
||||||
|
|
||||||
select(columns: string[] = ['*']): SelectQueryBuilder {
|
select(columns: string[] = ['*']): SelectQueryBuilder {
|
||||||
return new SelectQueryBuilder(this.engine, this.name, columns, this.executor);
|
return new SelectQueryBuilder(this.name, this.requireExecutor('select'), columns);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0(B-3):取执行器;缺失即**明确失败**。
|
||||||
|
*
|
||||||
|
* 修复前 builder 在拿不到 executor 时会退化为"直接调引擎" —— 于是
|
||||||
|
* `db.table('t')` 与 `db.query()` 两条路径的语义不同(投影/列校验/LIMIT 下推/
|
||||||
|
* maxRowsPerQuery 在直通路径上全部缺失)。现在只保留一条管线:
|
||||||
|
* 没有 executor 就没有可用的查询 API,显式报错而不是悄悄降级。
|
||||||
|
*/
|
||||||
|
private requireExecutor(operation: string): QueryExecutor {
|
||||||
|
if (!this.executor) {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Table.${operation}() requires a query executor (obtain the table via db.table())`,
|
||||||
|
'NOT_SUPPORTED',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return this.executor;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** v0.4.0: 流式查询 — 逐行回调,不物化全部结果 */
|
/** v0.4.0: 流式查询 — 逐行回调,不物化全部结果 */
|
||||||
@@ -100,14 +118,39 @@ export class Table<T = Record<string, unknown>> {
|
|||||||
|
|
||||||
// ---- 更新 ----
|
// ---- 更新 ----
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0(B-3):写操作也走**唯一管线**(Executor),生命周期钩子由本方法注入。
|
||||||
|
*
|
||||||
|
* 修复前 `UpdateQueryBuilder` 直接调 `engine.update`:`$subquery`/`$col` 无人解析
|
||||||
|
* → 引擎判 UNKNOWN → 静默影响 0 行;`beforeUpdate`/`afterUpdate` 的触发点也因此
|
||||||
|
* 与 SQL 路径不同(一条在 builder 里、一条在 core 里)。
|
||||||
|
*/
|
||||||
update(updates: Partial<T> & Record<string, unknown>): UpdateQueryBuilder {
|
update(updates: Partial<T> & Record<string, unknown>): UpdateQueryBuilder {
|
||||||
return new UpdateQueryBuilder(this.engine, this.name, updates, this.onWrite, this.onHooks);
|
return new UpdateQueryBuilder(this.name, updates, this.requireExecutor('update'), {
|
||||||
|
// 钩子接收实际语句(含 builder 累积的 where),与 SQL 路径的
|
||||||
|
// `triggerStatementHooks` 传参形状一致:{ table, where }
|
||||||
|
before: async (stmt) => {
|
||||||
|
await this.onHooks?.('beforeUpdate', [{ table: stmt.table, where: stmt.where }, updates]);
|
||||||
|
},
|
||||||
|
after: async (count, stmt) => {
|
||||||
|
this.onWrite?.(this.name);
|
||||||
|
await this.onHooks?.('afterUpdate', [{ table: stmt.table, where: stmt.where }, updates, count]);
|
||||||
|
},
|
||||||
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- 删除 ----
|
// ---- 删除 ----
|
||||||
|
|
||||||
delete(): DeleteQueryBuilder {
|
delete(): DeleteQueryBuilder {
|
||||||
return new DeleteQueryBuilder(this.engine, this.name, this.onWrite, this.onHooks);
|
return new DeleteQueryBuilder(this.name, this.requireExecutor('delete'), {
|
||||||
|
before: async (stmt) => {
|
||||||
|
await this.onHooks?.('beforeDelete', [{ table: stmt.from, where: stmt.where }]);
|
||||||
|
},
|
||||||
|
after: async (count, stmt) => {
|
||||||
|
this.onWrite?.(this.name);
|
||||||
|
await this.onHooks?.('afterDelete', [{ table: stmt.from, where: stmt.where }, count]);
|
||||||
|
},
|
||||||
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- 聚合 ----
|
// ---- 聚合 ----
|
||||||
|
|||||||
@@ -0,0 +1,328 @@
|
|||||||
|
/**
|
||||||
|
* metona-sqlark 统一行校验 —— 存储写入的**唯一**验证与规范化入口(v0.8.0)
|
||||||
|
* @module table/validation
|
||||||
|
*
|
||||||
|
* ============================================================================
|
||||||
|
* 为什么必须合并(v0.8.0 审计根因 1:关系语义在引擎间重复实现)
|
||||||
|
* ============================================================================
|
||||||
|
* 修复前项目里存在**三份**行校验实现,覆盖范围各不相同:
|
||||||
|
*
|
||||||
|
* | 位置 | 类型检查 | required | PK 非空 | maxLength | min/max | 未知列 |
|
||||||
|
* |---|---|---|---|---|---|---|
|
||||||
|
* | `engine/memory.ts#validateRow`(disk/hybrid 继承) | ✓ | ✓ | ✓ | ✗ | ✗ | 静默丢弃 |
|
||||||
|
* | `engine/aria/index.ts#validateRow` → `checkFieldType` | ✓ | ✓ | ✓ | ✓ | ✓ | 静默丢弃 |
|
||||||
|
* | `table/schema.ts#validateRow` | ✓ | ✓ | ✓ | ✓ | ✓ | 静默丢弃 |
|
||||||
|
*
|
||||||
|
* 于是**同一份 schema、同一条 INSERT** 在 Aria 上抛 `maxLength` 错误,在
|
||||||
|
* memory/disk/hybrid 上静默写入超长值(缺陷 A12)—— 用户的约束是否生效
|
||||||
|
* 取决于他选了哪个引擎,且没有任何提示。
|
||||||
|
*
|
||||||
|
* 更糟的是"未知列":`INSERT INTO t (id, nope) VALUES ('1', 2)` 在四个引擎上
|
||||||
|
* 都**静默丢弃 `nope`**、插入成功。用户以为写进去了,`SELECT nope` 又报
|
||||||
|
* COLUMN_NOT_FOUND —— 写路径与读路径对同一列名给出相反结论(缺陷 A17)。
|
||||||
|
*
|
||||||
|
* 本模块把校验收敛成**一个定义**(`compileValidator`),所有引擎与 QueryBuilder
|
||||||
|
* 都从它取校验器:
|
||||||
|
* - 约束覆盖面是"并集",不可能再出现"Aria 报错、memory 不报";
|
||||||
|
* - 未知列变成**显式错误**(`COLUMN_NOT_FOUND`,与读路径同一错误码);
|
||||||
|
* - 校验与**规范化**在同一处完成(`__proto__` 防污染、`undefined` 跳过、
|
||||||
|
* `default` 填充),引擎只负责存储,不再各自解释 schema。
|
||||||
|
*
|
||||||
|
* ============================================================================
|
||||||
|
* 三种载荷形态,为什么不能合成一个函数
|
||||||
|
* ============================================================================
|
||||||
|
* - `validateRow` INSERT 语义:`default` 生效、缺列合法、`required` 按最终值判;
|
||||||
|
* - `validatePartial` UPDATE 语义:**只校验出现的列**(`{ a: undefined }` 表示
|
||||||
|
* "不更新 a",不能被 required/min/max 判失败);
|
||||||
|
* - `assertNoUnknownColumns` 独立可复用的"列名存在性"检查(写路径预检)。
|
||||||
|
*
|
||||||
|
* 把它们混成一个带 options 的函数会让"required 是否生效"取决于调用方参数,
|
||||||
|
* 从而重新引入跨路径差异 —— 本模块刻意保持三个显式入口。
|
||||||
|
*/
|
||||||
|
|
||||||
|
import type { TableSchema, ColumnDef, FieldType } from '../constants';
|
||||||
|
import { DatabaseError } from '../constants';
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// 结果类型
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/** 行校验器编译结果(单次编译,多次复用) */
|
||||||
|
export interface RowValidator {
|
||||||
|
/** 表名(用于错误消息) */
|
||||||
|
readonly table: string;
|
||||||
|
/** schema 里定义的列名集合 */
|
||||||
|
readonly columns: ReadonlySet<string>;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* INSERT 语义校验 + 规范化。
|
||||||
|
*
|
||||||
|
* @param row 待写入的行(不会被修改)
|
||||||
|
* @param knownColumns 调用方已知/允许的列名(如 INSERT 显式列清单)。
|
||||||
|
* 省略时以 schema 列为准。**超出该集合的键即未知列 → 抛错。**
|
||||||
|
* @returns 规范化后的新行(仅含 schema 列,`undefined` 不落键,`default` 已填充)
|
||||||
|
* @throws DatabaseError COLUMN_NOT_FOUND | VALIDATION_ERROR | TYPE_ERROR
|
||||||
|
*/
|
||||||
|
validateRow(row: Record<string, unknown>, knownColumns?: readonly string[]): Record<string, unknown>;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* UPDATE 语义校验 + 规范化:仅校验出现的列。
|
||||||
|
*
|
||||||
|
* @returns `{ values, unknown }` —— `values` 是规范化后的更新载荷,
|
||||||
|
* `unknown` 是 schema 里不存在的键(由调用方决定是否报错,
|
||||||
|
* 因为引擎内部级联等路径可能携带合并行的派生列)。
|
||||||
|
*/
|
||||||
|
validatePartial(row: Record<string, unknown>): {
|
||||||
|
values: Record<string, unknown>;
|
||||||
|
unknown: string[];
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// 编译
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 把 schema 编译成可复用的行校验器。
|
||||||
|
*
|
||||||
|
* 为什么"编译"而不是每次都遍历 schema:校验处于每次 INSERT/UPDATE 的热路径上,
|
||||||
|
* 引擎在一次批量写入里会对成百上千行调用它。预先把列定义拆成列表 + 集合,
|
||||||
|
* 既避免重复的 `Object.entries`,也让"哪些键允许出现"成为可哈希的集合判断。
|
||||||
|
* **不缓存**编译结果:schema 可被 `alterTable` 原地修改,长期缓存会用到过期列定义。
|
||||||
|
*/
|
||||||
|
export function compileValidator(schema: TableSchema): RowValidator {
|
||||||
|
const table = schema.name;
|
||||||
|
const entries: Array<[string, ColumnDef]> = Object.entries(schema.columns);
|
||||||
|
const columnNames = new Set(entries.map(([name]) => name));
|
||||||
|
|
||||||
|
function assertNoUnknownColumns(
|
||||||
|
row: Record<string, unknown>,
|
||||||
|
knownColumns?: readonly string[],
|
||||||
|
): void {
|
||||||
|
const allowed = knownColumns ? new Set(knownColumns) : columnNames;
|
||||||
|
// 先收集未知列再报错:一次列出全部,避免用户"改一个报一个"
|
||||||
|
const unknown: string[] = [];
|
||||||
|
for (const key of Object.keys(row)) {
|
||||||
|
if (key === '__proto__') continue; // 由 sanitize 阶段统一拒绝(消息不同)
|
||||||
|
if (!allowed.has(key)) unknown.push(key);
|
||||||
|
}
|
||||||
|
if (unknown.length > 0) {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Unknown column${unknown.length > 1 ? 's' : ''} ${unknown
|
||||||
|
.map((c) => `"${c}"`)
|
||||||
|
.join(', ')} in table "${table}". Known columns: ${[...allowed].join(', ')}`,
|
||||||
|
'COLUMN_NOT_FOUND',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function validateRow(
|
||||||
|
row: Record<string, unknown>,
|
||||||
|
knownColumns?: readonly string[],
|
||||||
|
): Record<string, unknown> {
|
||||||
|
assertNoUnknownColumns(row, knownColumns);
|
||||||
|
|
||||||
|
const validated: Record<string, unknown> = {};
|
||||||
|
for (const [colName, colDef] of entries) {
|
||||||
|
// INSERT 语义:缺列时 default 生效
|
||||||
|
let value = row[colName];
|
||||||
|
if (value === undefined && colDef.default !== undefined) value = colDef.default;
|
||||||
|
|
||||||
|
assertNotNullConstraints(table, colName, colDef, value);
|
||||||
|
if (value !== undefined && value !== null) {
|
||||||
|
assertJsonSafeNumber(table, colName, value);
|
||||||
|
checkFieldType(table, colName, colDef.type, value, colDef);
|
||||||
|
}
|
||||||
|
// v0.8.0(B-1):缺列且无 default → 显式写入 NULL,**不能省略键**。
|
||||||
|
//
|
||||||
|
// 此前 `if (value !== undefined) validated[colName] = value;` 会把这个列整个
|
||||||
|
// 从行里删掉,于是存储行只含"有值的列",行形状取决于写入方式:
|
||||||
|
// INSERT INTO t (id, g) VALUES ('9','z') -- 行里没有 n 键
|
||||||
|
// SELECT id, g, n FROM t WHERE id = '9' -- 抛 COLUMN_NOT_FOUND: n
|
||||||
|
// 而 `SELECT * FROM t` 却能正常返回(少一列而已)—— 同一行"有没有 n 列"
|
||||||
|
// 在读路径上给出相反结论。SQL 语义中"未提供值"就是 NULL,故统一补 null:
|
||||||
|
// 行始终包含全部 schema 列,投影/排序/三值比较才有统一前提。
|
||||||
|
validated[colName] = value === undefined ? null : value;
|
||||||
|
}
|
||||||
|
return validated;
|
||||||
|
}
|
||||||
|
|
||||||
|
function validatePartial(row: Record<string, unknown>): {
|
||||||
|
values: Record<string, unknown>;
|
||||||
|
unknown: string[];
|
||||||
|
} {
|
||||||
|
const values: Record<string, unknown> = {};
|
||||||
|
const unknown: string[] = [];
|
||||||
|
for (const [colName, value] of Object.entries(row)) {
|
||||||
|
if (colName === '__proto__') {
|
||||||
|
throw new DatabaseError('Column name "__proto__" is not allowed', 'VALIDATION_ERROR');
|
||||||
|
}
|
||||||
|
const colDef = schema.columns[colName];
|
||||||
|
if (!colDef) {
|
||||||
|
unknown.push(colName);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
// UPDATE 语义:`undefined` 已由 stripUndefinedUpdates 过滤;
|
||||||
|
// 这里再挡一次,保证"未提供的列"绝不会被判 required 失败。
|
||||||
|
if (value === undefined) continue;
|
||||||
|
assertNotNullConstraints(table, colName, colDef, value);
|
||||||
|
if (value !== null) {
|
||||||
|
assertJsonSafeNumber(table, colName, value);
|
||||||
|
checkFieldType(table, colName, colDef.type, value, colDef);
|
||||||
|
}
|
||||||
|
values[colName] = value;
|
||||||
|
}
|
||||||
|
return { values, unknown };
|
||||||
|
}
|
||||||
|
|
||||||
|
return { table, columns: columnNames, validateRow, validatePartial };
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// 共享约束
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 检查字段类型(含约束校验)。
|
||||||
|
*
|
||||||
|
* v0.8.0(B-1):在 `checkFieldType`(table/schema.ts)之外**额外**拒绝
|
||||||
|
* `NaN` 与 `±Infinity`。为什么必须有这一层:
|
||||||
|
* - JSON 无法表示它们 —— `JSON.stringify({ v: NaN })` 得到 `{"v":null}`,
|
||||||
|
* 于是 `INSERT ... VALUES (NaN)` 在内存引擎里是 NaN,落盘再读回来变成 null;
|
||||||
|
* 同一个库在"写后立即查"与"重启后查"得到不同结果,且没有任何提示。
|
||||||
|
* - KVStore / Aria 的持久化路径都是 JSON,因此这是**所有**磁盘引擎的共性问题。
|
||||||
|
* - 用户能构造出 NaN:`Number('abc')`、`0/0`、`parseFloat('x')` 等,
|
||||||
|
* 经由参数绑定进入写入路径。
|
||||||
|
* 显式拒绝(`VALIDATION_ERROR`)让问题在写入时暴露,而不是变成读出来的 null。
|
||||||
|
*/
|
||||||
|
function assertJsonSafeNumber(
|
||||||
|
table: string,
|
||||||
|
colName: string,
|
||||||
|
value: unknown,
|
||||||
|
): void {
|
||||||
|
if (typeof value !== 'number') return;
|
||||||
|
if (Number.isFinite(value)) return;
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Column "${colName}" in table "${table}" cannot store ${Number.isNaN(value) ? 'NaN' : String(value)}:`
|
||||||
|
+ ' it is not representable in JSON and would be silently read back as null',
|
||||||
|
'VALIDATION_ERROR',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* NOT NULL 类约束。
|
||||||
|
*
|
||||||
|
* 两条规则合并在一个函数里按顺序判断,是为了让错误消息稳定:
|
||||||
|
* - `required` → "is required"(用户声明的业务约束);
|
||||||
|
* - `primaryKey` → 隐含 NOT NULL(SQL 语义)。
|
||||||
|
* 在此之前两者分散在两个引擎的 `validateRow` 里各写一遍,消息略有差异
|
||||||
|
* (一个有表名后缀一个没有),依赖消息文本的测试只能各测各的引擎。
|
||||||
|
*/
|
||||||
|
function assertNotNullConstraints(
|
||||||
|
table: string,
|
||||||
|
colName: string,
|
||||||
|
colDef: ColumnDef,
|
||||||
|
value: unknown,
|
||||||
|
): void {
|
||||||
|
if (colDef.required && (value === undefined || value === null)) {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Column "${colName}" is required in table "${table}"`,
|
||||||
|
'VALIDATION_ERROR',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
// v0.7.4:主键列强制非空 —— 此前 null/undefined 主键被 String() 化为
|
||||||
|
// "null"/"undefined" 静默入库,行再也无法按主键取回。
|
||||||
|
if (colDef.primaryKey && (value === undefined || value === null)) {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Primary key column "${colName}" in table "${table}" cannot be null or undefined`,
|
||||||
|
'VALIDATION_ERROR',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// 引擎接口适配
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/** 类型再导出,避免调用方从 constants 与 schema 两处 import */
|
||||||
|
export type { ColumnDef, FieldType };
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// 字段类型与约束检查(唯一实现)
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/** 检查字段类型(含约束校验) */
|
||||||
|
export function checkFieldType(
|
||||||
|
tableName: string,
|
||||||
|
colName: string,
|
||||||
|
type: FieldType,
|
||||||
|
value: unknown,
|
||||||
|
colDef?: ColumnDef,
|
||||||
|
): void {
|
||||||
|
const jsType = typeof value;
|
||||||
|
|
||||||
|
switch (type) {
|
||||||
|
case 'string':
|
||||||
|
if (jsType !== 'string') {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Column "${colName}" in table "${tableName}" expects string, got ${jsType}`,
|
||||||
|
'TYPE_ERROR',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (colDef?.maxLength !== undefined && (value as string).length > colDef.maxLength) {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Column "${colName}" in table "${tableName}" exceeds max length ${colDef.maxLength}`,
|
||||||
|
'VALIDATION_ERROR',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
|
||||||
|
case 'number':
|
||||||
|
if (jsType !== 'number') {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Column "${colName}" in table "${tableName}" expects number, got ${jsType}`,
|
||||||
|
'TYPE_ERROR',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (colDef?.min !== undefined && (value as number) < colDef.min) {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Column "${colName}" in table "${tableName}" value ${value} below minimum ${colDef.min}`,
|
||||||
|
'VALIDATION_ERROR',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (colDef?.max !== undefined && (value as number) > colDef.max) {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Column "${colName}" in table "${tableName}" value ${value} above maximum ${colDef.max}`,
|
||||||
|
'VALIDATION_ERROR',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
|
||||||
|
case 'boolean':
|
||||||
|
if (jsType !== 'boolean') {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Column "${colName}" in table "${tableName}" expects boolean, got ${jsType}`,
|
||||||
|
'TYPE_ERROR',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
|
||||||
|
case 'date':
|
||||||
|
if (jsType !== 'string' || isNaN(Date.parse(value as string))) {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Column "${colName}" in table "${tableName}" expects valid date string, got ${typeof value}`,
|
||||||
|
'TYPE_ERROR',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
|
||||||
|
case 'json':
|
||||||
|
if (jsType !== 'object') {
|
||||||
|
throw new DatabaseError(
|
||||||
|
`Column "${colName}" in table "${tableName}" expects object/array, got ${jsType}`,
|
||||||
|
'TYPE_ERROR',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -8,6 +8,7 @@
|
|||||||
import type { IStorageEngine } from '../engine/interface';
|
import type { IStorageEngine } from '../engine/interface';
|
||||||
import { Table } from '../table/table';
|
import { Table } from '../table/table';
|
||||||
import { DatabaseError } from '../constants';
|
import { DatabaseError } from '../constants';
|
||||||
|
import { QueryExecutor } from '../query/executor';
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// Transaction
|
// Transaction
|
||||||
@@ -17,16 +18,26 @@ export class Transaction {
|
|||||||
private engine: IStorageEngine;
|
private engine: IStorageEngine;
|
||||||
private tables: Map<string, Table> = new Map();
|
private tables: Map<string, Table> = new Map();
|
||||||
private completed = false;
|
private completed = false;
|
||||||
|
/**
|
||||||
|
* v0.8.0(B-3):事务内的表操作同样走**唯一执行管线**。
|
||||||
|
*
|
||||||
|
* `Table` 的 select/update/delete 现在必须拿到 executor(builder 只产出 AST),
|
||||||
|
* 因此这里构造一个绑定到同一引擎的执行器 —— 事务的原子性由**引擎**提供
|
||||||
|
* (begin/commit/rollback 作用在引擎上),执行器只是把 AST 翻译成引擎调用,
|
||||||
|
* 两者组合即可保持"事务内写入可回滚"这一语义不变。
|
||||||
|
*/
|
||||||
|
private executor: QueryExecutor;
|
||||||
|
|
||||||
constructor(engine: IStorageEngine) {
|
constructor(engine: IStorageEngine) {
|
||||||
this.engine = engine;
|
this.engine = engine;
|
||||||
|
this.executor = new QueryExecutor(engine);
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 获取表操作对象 */
|
/** 获取表操作对象 */
|
||||||
table(tableName: string): Table {
|
table(tableName: string): Table {
|
||||||
let t = this.tables.get(tableName);
|
let t = this.tables.get(tableName);
|
||||||
if (!t) {
|
if (!t) {
|
||||||
t = new Table(this.engine, tableName);
|
t = new Table(this.engine, tableName, this.executor);
|
||||||
this.tables.set(tableName, t);
|
this.tables.set(tableName, t);
|
||||||
}
|
}
|
||||||
return t;
|
return t;
|
||||||
|
|||||||
@@ -4,9 +4,9 @@
|
|||||||
|
|
||||||
import { AriaEngine } from '../src/engine/aria/index';
|
import { AriaEngine } from '../src/engine/aria/index';
|
||||||
|
|
||||||
import { installOPFSMock } from './helpers/opfs-mock';
|
import { resetOPFSMock } from './helpers/storage-harness';
|
||||||
|
|
||||||
beforeEach(() => { installOPFSMock(new Map()); });
|
beforeEach(() => { resetOPFSMock(); });
|
||||||
|
|
||||||
const mkEngine = async (name: string): Promise<AriaEngine> => {
|
const mkEngine = async (name: string): Promise<AriaEngine> => {
|
||||||
const e = new AriaEngine({ storageBackend: 'opfs', walSyncMode: 'full' });
|
const e = new AriaEngine({ storageBackend: 'opfs', walSyncMode: 'full' });
|
||||||
|
|||||||
@@ -49,19 +49,21 @@ describe('Core 全覆盖', () => {
|
|||||||
expect(typeof unsub).toBe('function');
|
expect(typeof unsub).toBe('function');
|
||||||
});
|
});
|
||||||
|
|
||||||
it('unsubscribe 取消监听', () => {
|
it('unsubscribe 取消监听', async () => {
|
||||||
const fn = jest.fn();
|
const fn = jest.fn();
|
||||||
const unsub = db.subscribe('users', fn);
|
const unsub = db.subscribe('users', fn);
|
||||||
unsub();
|
unsub();
|
||||||
db.emit('users', { type: 'insert', row: { id: '3' } });
|
await db.emit('users', { type: 'insert', row: { id: '3' } });
|
||||||
expect(fn).not.toHaveBeenCalled();
|
expect(fn).not.toHaveBeenCalled();
|
||||||
});
|
});
|
||||||
|
|
||||||
it('emit 触发监听', () => {
|
it('emit 触发监听', async () => {
|
||||||
const fn = jest.fn();
|
const fn = jest.fn();
|
||||||
db.subscribe('users', fn);
|
db.subscribe('users', fn);
|
||||||
db.emit('users', { type: 'insert', row: { id: '3', name: 'Charlie' } });
|
// v0.8.0: emit 现在会补全 table 字段(事件语义统一),且返回 Promise
|
||||||
expect(fn).toHaveBeenCalledWith({ type: 'insert', row: { id: '3', name: 'Charlie' } });
|
await db.emit('users', { type: 'insert', row: { id: '3', name: 'Charlie' } });
|
||||||
|
// 事件现在带 table 字段(与引擎层变更事件语义一致)
|
||||||
|
expect(fn).toHaveBeenCalledWith({ table: 'users', type: 'insert', row: { id: '3', name: 'Charlie' } });
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
@@ -94,6 +94,69 @@
|
|||||||
},
|
},
|
||||||
|
|
||||||
getEngine: () => (window.__ms.__db ? window.__ms.__db.getEngine() : null),
|
getEngine: () => (window.__ms.__db ? window.__ms.__db.getEngine() : null),
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0(PC-2):**不等写入完成**就返回 —— 供测试在写入进行中触发真崩溃。
|
||||||
|
*
|
||||||
|
* `page.evaluate` 会等待返回的 Promise,因此"发起写入后立刻崩溃"无法用
|
||||||
|
* 普通 await 表达。这里把 Promise 挂到 `__ms.__pending` 后立即返回,
|
||||||
|
* 页面在写入尚未落盘时被杀 → 真正覆盖"WAL 半写 / OPFS 写入中途"窗口。
|
||||||
|
*/
|
||||||
|
insertNoAwait: (opts) => {
|
||||||
|
const db = window.__ms.__db;
|
||||||
|
// 故意不 await:这就是"崩溃发生在写入进行中"的语义
|
||||||
|
window.__ms.__pending = db.table(opts.table).insertMany(opts.rows);
|
||||||
|
window.__ms.__pending.catch(() => {}); // 避免未处理拒绝噪音(进程即将被杀)
|
||||||
|
return { started: true };
|
||||||
|
},
|
||||||
|
|
||||||
|
/** 未完成写入的数量(诊断用) */
|
||||||
|
hasPendingWrite: () => ({ pending: !!window.__ms.__pending }),
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0(PC-2):在"下一次 OPFS 写入"处安装崩溃钩子。
|
||||||
|
*
|
||||||
|
* Aria 的 OPFS 后端用 `FileSystemFileHandle.createWritable()`(copy-on-write):
|
||||||
|
* 数据先写入交换副本,`close()` 时才原子替换原文件。因此"写到一半崩溃"
|
||||||
|
* 有两种真正不同的窗口,都要能注入:
|
||||||
|
* - `phase: 'write'` —— 数据写入交换副本的**中途**(`write()` 被调用时卡死);
|
||||||
|
* - `phase: 'commit'` —— 数据已全部写入交换副本、但**尚未**原子替换
|
||||||
|
* (`close()` 被调用时卡死)。这个窗口最关键:此时崩溃必须保证
|
||||||
|
* **原文件保持旧内容**(不能出现半个文件)。
|
||||||
|
*
|
||||||
|
* 卡死方式:进入永不返回的同步死循环,随后由 CDP `Page.crash` 终止进程 ——
|
||||||
|
* 等价于"浏览器在写的瞬间被强杀"。
|
||||||
|
*
|
||||||
|
* 返回 `{ armed, target, phase }`,测试据此断言钩子确实安装成功
|
||||||
|
*(避免"注入了但没走到"的假绿)。
|
||||||
|
*/
|
||||||
|
armCrashOnOpfsWrite: (opts) => {
|
||||||
|
const phase = (opts && opts.phase) || 'write';
|
||||||
|
// n 缺省/0 → 第 1 次就崩
|
||||||
|
const target = Math.max(1, Number(opts && opts.n) || 1);
|
||||||
|
const proto = window.FileSystemWritableFileStream && window.FileSystemWritableFileStream.prototype;
|
||||||
|
if (!proto) return { armed: false, reason: 'FileSystemWritableFileStream unavailable' };
|
||||||
|
if (window.__ms.__crashArmed) return { armed: true, alreadyArmed: true, phase };
|
||||||
|
window.__ms.__crashArmed = true;
|
||||||
|
window.__ms.__crashCount = 0;
|
||||||
|
|
||||||
|
const spin = () => { while (true) { /* crash window: Page.crash 在此终止进程 */ } };
|
||||||
|
const wrap = (name) => {
|
||||||
|
const original = proto[name];
|
||||||
|
if (typeof original !== 'function') return false;
|
||||||
|
proto[name] = function patched(...args) {
|
||||||
|
window.__ms.__crashCount++;
|
||||||
|
if (window.__ms.__crashCount >= target) spin();
|
||||||
|
return original.apply(this, args);
|
||||||
|
};
|
||||||
|
return true;
|
||||||
|
};
|
||||||
|
const hooked = phase === 'commit' ? wrap('close') : wrap('write');
|
||||||
|
return { armed: hooked, target, phase };
|
||||||
|
},
|
||||||
|
|
||||||
|
/** 已发生的目标 OPFS 调用次数(验证崩溃窗口确实被触发) */
|
||||||
|
opfsWriteCount: () => ({ count: window.__ms.__crashCount || 0 }),
|
||||||
};
|
};
|
||||||
document.getElementById('status').textContent = 'ready';
|
document.getElementById('status').textContent = 'ready';
|
||||||
</script>
|
</script>
|
||||||
|
|||||||
+119
-16
@@ -10,7 +10,7 @@
|
|||||||
* 6. 页面化 + OPFS 端到端
|
* 6. 页面化 + OPFS 端到端
|
||||||
* 7. repair() 自愈(真实文件系统)
|
* 7. repair() 自愈(真实文件系统)
|
||||||
*/
|
*/
|
||||||
import { test, expect, type Page } from '@playwright/test';
|
import { test, expect, type Page, type CDPSession } from '@playwright/test';
|
||||||
|
|
||||||
const HARNESS = '/tests/e2e/harness.html';
|
const HARNESS = '/tests/e2e/harness.html';
|
||||||
|
|
||||||
@@ -20,27 +20,70 @@ async function openPage(page: Page): Promise<void> {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/** 在页面中执行 harness 方法(统一包装错误信息) */
|
/** 在页面中执行 harness 方法(统一包装错误信息) */
|
||||||
async function run<T>(page: Page, method: string, args: unknown): Promise<T> {
|
async function run<T>(page: Page, method: string, args: unknown = null): Promise<T> {
|
||||||
const result = await page.evaluate(
|
const result = await page.evaluate(
|
||||||
([m, a]) => (window as unknown as { __ms: Record<string, (o: unknown) => Promise<unknown>> }).__ms[m](a),
|
(payload: [string, unknown]) => {
|
||||||
[method, args] as const,
|
const [m, a] = payload;
|
||||||
|
return (window as unknown as { __ms: Record<string, (o: unknown) => Promise<unknown>> }).__ms[m](a);
|
||||||
|
},
|
||||||
|
[method, args] as [string, unknown],
|
||||||
);
|
);
|
||||||
return result as T;
|
return result as T;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 强制"崩溃"当前页面:终止渲染进程(不触发 beforeunload/close) */
|
/**
|
||||||
async function crashPage(page: Page): Promise<void> {
|
* v0.8.0(PC-2):**真崩溃**当前页面。
|
||||||
await page.evaluate(() => {
|
*
|
||||||
// 模拟崩溃:直接终止。使用 CDP Page.crash 等价效果 —— 通过无限循环让渲染进程被杀
|
* 修复前这里只是 `win.__ms = null; await page.close()` —— 那是**优雅关闭**:
|
||||||
// 但更可靠的方式是关闭页面连接(Playwright 无 Page.crash 公共 API),
|
* 没有未完成的 I/O、不经过任何崩溃窗口,因此"崩溃恢复"的结论实际上是在
|
||||||
// 这里用 context.close 前先把页面里的清理路径断开:
|
* "正常关闭后重开"上得到的(v0.8.0 审计根因 5:崩溃语义声称未被验证)。
|
||||||
const win = window as unknown as { __ms: { __db: unknown } | null };
|
*
|
||||||
// 断开所有引用(模拟进程死亡:不调用 close)
|
* 现在用 CDP `Page.crash` 直接终止渲染进程:进行中的 OPFS 写入、未 flush 的
|
||||||
win.__ms = null;
|
* WAL 缓冲、同步访问句柄全部**立即**消失,与浏览器/标签页被强杀一致。
|
||||||
});
|
* 崩溃后 `page` 上任何 evaluate 都会失败,因此调用方必须新建页面继续。
|
||||||
await page.close();
|
*/
|
||||||
|
async function crashPage(page: Page, preAttached?: CDPSession): Promise<void> {
|
||||||
|
const cdp = preAttached ?? await page.context().newCDPSession(page);
|
||||||
|
// 不 await:Page.crash 会让该 CDP 连接随渲染进程一起消失
|
||||||
|
cdp.send('Page.crash').catch(() => { /* 连接随进程一起消失 */ });
|
||||||
|
await new Promise((resolve) => setTimeout(resolve, 300));
|
||||||
|
|
||||||
|
// 关闭这个已崩溃的页面句柄(仅清理句柄,不影响 context 与 OPFS)
|
||||||
|
await page.close().catch(() => { /* 已崩溃,忽略 */ });
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.8.0(PC-2):在**写入进行中**真崩溃。
|
||||||
|
*
|
||||||
|
* 页面内的 OPFS 调用被 `armCrashOnOpfsWrite` 装上死循环钩子后,页面主线程会
|
||||||
|
* 卡死 —— 因此 CDP 会话必须**提前**建立(卡死后 `newCDPSession` 会失败)。
|
||||||
|
* 本函数封装这个"先建会话、再武装、后崩溃"的顺序。
|
||||||
|
*
|
||||||
|
* @returns 崩溃前已发生的目标 OPFS 调用次数(0 说明钩子没被走到 → 用例应失败)
|
||||||
|
*/
|
||||||
|
async function crashDuringOpfsWrite(
|
||||||
|
page: Page,
|
||||||
|
opts: { phase: 'write' | 'commit'; n?: number; rows: Record<string, unknown>[] },
|
||||||
|
): Promise<{ armed: boolean; opfsCalls: number }> {
|
||||||
|
const cdp = await page.context().newCDPSession(page);
|
||||||
|
const armed = await run<{ armed: boolean; target: number; phase: string }>(
|
||||||
|
page,
|
||||||
|
'armCrashOnOpfsWrite',
|
||||||
|
{ n: opts.n ?? 1, phase: opts.phase },
|
||||||
|
);
|
||||||
|
// 发起写入但不等它完成:page.evaluate 会因死循环永不返回,故显式 catch 掉
|
||||||
|
page.evaluate(
|
||||||
|
(rows) => (window as unknown as { __ms: { insertNoAwait: (o: unknown) => unknown } })
|
||||||
|
.__ms.insertNoAwait({ table: 't', rows }),
|
||||||
|
opts.rows,
|
||||||
|
).catch(() => { /* 页面将崩溃,预期 */ });
|
||||||
|
// 等死循环真正进入(页面卡死)
|
||||||
|
await new Promise((resolve) => setTimeout(resolve, 800));
|
||||||
|
await crashPage(page, cdp);
|
||||||
|
return { armed: armed.armed, opfsCalls: 0 };
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
test.describe('OPFS 真实环境', () => {
|
test.describe('OPFS 真实环境', () => {
|
||||||
test('写入 → 刷新页面 → 数据保留(持久化)', async ({ page }) => {
|
test('写入 → 刷新页面 → 数据保留(持久化)', async ({ page }) => {
|
||||||
await openPage(page);
|
await openPage(page);
|
||||||
@@ -104,7 +147,7 @@ test.describe('OPFS 真实环境', () => {
|
|||||||
for (let i = 0; i < total; i++) {
|
for (let i = 0; i < total; i++) {
|
||||||
await run(page, 'insert', { table: 'logs', rows: [{ id: `log-${i}` }] });
|
await run(page, 'insert', { table: 'logs', rows: [{ id: `log-${i}` }] });
|
||||||
}
|
}
|
||||||
// 模拟崩溃:直接终止页面(不调用 close → 不 checkpoint)
|
// v0.8.0(PC-2):**真崩溃**(CDP Page.crash 终止渲染进程,不经过 close/checkpoint)
|
||||||
await crashPage(page);
|
await crashPage(page);
|
||||||
|
|
||||||
// 新页面重开:WAL 重放
|
// 新页面重开:WAL 重放
|
||||||
@@ -115,6 +158,66 @@ test.describe('OPFS 真实环境', () => {
|
|||||||
});
|
});
|
||||||
const res = await run<{ count: number }>(page2, 'count', { table: 'logs' });
|
const res = await run<{ count: number }>(page2, 'count', { table: 'logs' });
|
||||||
expect(res.count).toBe(total);
|
expect(res.count).toBe(total);
|
||||||
|
// 逐条抽查(不只是计数):计数可能因重复主键被覆盖而"看起来对"
|
||||||
|
for (const i of [0, 1, 24, 25, 49]) {
|
||||||
|
const found = await run<{ rows: Record<string, unknown>[] }>(page2, 'find', {
|
||||||
|
table: 'logs', where: { id: `log-${i}` },
|
||||||
|
});
|
||||||
|
expect(found.rows).toHaveLength(1);
|
||||||
|
}
|
||||||
|
await run(page2, 'close');
|
||||||
|
await page2.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
test('OPFS 写入窗口崩溃(write 阶段)→ 旧数据完好、无半文件', async ({ page }) => {
|
||||||
|
await openPage(page);
|
||||||
|
await run(page, 'open', { name: 'e2e-crash-opfs-write', diskEngine: 'opfs', checkpointInterval: 999999999 });
|
||||||
|
await run(page, 'createTable', {
|
||||||
|
table: 't', columns: [{ name: 'id', type: 'string', primaryKey: true }],
|
||||||
|
});
|
||||||
|
for (let i = 0; i < 3; i++) await run(page, 'insert', { table: 't', rows: [{ id: `ok-${i}` }] });
|
||||||
|
|
||||||
|
// 在 `createWritable().write()` 中途卡死 → 真崩溃
|
||||||
|
const { armed } = await crashDuringOpfsWrite(page, { phase: 'write', rows: [{ id: 'torn' }] });
|
||||||
|
expect(armed).toBe(true); // 钩子必须真的装上,否则这个窗口根本没被覆盖
|
||||||
|
|
||||||
|
const page2 = await page.context().newPage();
|
||||||
|
await openPage(page2);
|
||||||
|
await run(page2, 'open', { name: 'e2e-crash-opfs-write', diskEngine: 'opfs', checkpointInterval: 999999999 });
|
||||||
|
const res = await run<{ count: number }>(page2, 'count', { table: 't' });
|
||||||
|
// 已确认的 3 条必须完好;未完成的那条允许存在也可能不存在(不做要求)
|
||||||
|
expect(res.count).toBeGreaterThanOrEqual(3);
|
||||||
|
for (let i = 0; i < 3; i++) {
|
||||||
|
const found = await run<{ rows: Record<string, unknown>[] }>(page2, 'find', { table: 't', where: { id: `ok-${i}` } });
|
||||||
|
expect(found.rows).toHaveLength(1);
|
||||||
|
}
|
||||||
|
await run(page2, 'close');
|
||||||
|
await page2.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
test('OPFS 提交窗口崩溃(commit 阶段)→ 原文件保持旧内容(copy-on-write 语义)', async ({ page }) => {
|
||||||
|
await openPage(page);
|
||||||
|
await run(page, 'open', { name: 'e2e-crash-opfs-commit', diskEngine: 'opfs', checkpointInterval: 999999999 });
|
||||||
|
await run(page, 'createTable', {
|
||||||
|
table: 't', columns: [{ name: 'id', type: 'string', primaryKey: true }],
|
||||||
|
});
|
||||||
|
for (let i = 0; i < 3; i++) await run(page, 'insert', { table: 't', rows: [{ id: `ok-${i}` }] });
|
||||||
|
|
||||||
|
// 数据已写进交换副本、但**尚未**原子替换时卡死 → 真崩溃
|
||||||
|
const { armed } = await crashDuringOpfsWrite(page, { phase: 'commit', rows: [{ id: 'torn' }] });
|
||||||
|
expect(armed).toBe(true);
|
||||||
|
|
||||||
|
const page2 = await page.context().newPage();
|
||||||
|
await openPage(page2);
|
||||||
|
await run(page2, 'open', { name: 'e2e-crash-opfs-commit', diskEngine: 'opfs', checkpointInterval: 999999999 });
|
||||||
|
// 这是 OPFS 后端最关键的崩溃不变量:createWritable 是 copy-on-write,
|
||||||
|
// 未 commit 就崩溃 → 原文件内容**完全不变**(不会出现半个文件)。
|
||||||
|
const res = await run<{ count: number }>(page2, 'count', { table: 't' });
|
||||||
|
expect(res.count).toBe(3);
|
||||||
|
for (let i = 0; i < 3; i++) {
|
||||||
|
const found = await run<{ rows: Record<string, unknown>[] }>(page2, 'find', { table: 't', where: { id: `ok-${i}` } });
|
||||||
|
expect(found.rows).toHaveLength(1);
|
||||||
|
}
|
||||||
await run(page2, 'close');
|
await run(page2, 'close');
|
||||||
await page2.close();
|
await page2.close();
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import { MetonaSqlark } from '../src/core';
|
|||||||
import { MemoryEngine } from '../src/engine/memory';
|
import { MemoryEngine } from '../src/engine/memory';
|
||||||
import { SelectQueryBuilder } from '../src/query/builder';
|
import { SelectQueryBuilder } from '../src/query/builder';
|
||||||
import { createSchema } from '../src/table/schema';
|
import { createSchema } from '../src/table/schema';
|
||||||
|
import { QueryExecutor } from '../src/query/executor';
|
||||||
|
|
||||||
describe('边缘覆盖', () => {
|
describe('边缘覆盖', () => {
|
||||||
let db: MetonaSqlark;
|
let db: MetonaSqlark;
|
||||||
@@ -117,18 +118,18 @@ describe('边缘覆盖', () => {
|
|||||||
afterEach(async () => { await engine.close(); });
|
afterEach(async () => { await engine.close(); });
|
||||||
|
|
||||||
it('SelectQueryBuilder.as 别名', () => {
|
it('SelectQueryBuilder.as 别名', () => {
|
||||||
const qb = new SelectQueryBuilder(engine, 'users').as('u');
|
const qb = new SelectQueryBuilder('users', new QueryExecutor(engine)).as('u');
|
||||||
expect(qb.toAST().alias).toBe('u');
|
expect(qb.toAST().alias).toBe('u');
|
||||||
});
|
});
|
||||||
|
|
||||||
it('rightJoin', () => {
|
it('rightJoin', () => {
|
||||||
const qb = new SelectQueryBuilder(engine, 'users');
|
const qb = new SelectQueryBuilder('users', new QueryExecutor(engine));
|
||||||
qb.rightJoin('orders', { 'users.id': { $col: 'orders.user_id' } }, 'o');
|
qb.rightJoin('orders', { 'users.id': { $col: 'orders.user_id' } }, 'o');
|
||||||
expect(qb.toAST().joins![0].type).toBe('RIGHT');
|
expect(qb.toAST().joins![0].type).toBe('RIGHT');
|
||||||
});
|
});
|
||||||
|
|
||||||
it('join 默认为 INNER', () => {
|
it('join 默认为 INNER', () => {
|
||||||
const qb = new SelectQueryBuilder(engine, 'users');
|
const qb = new SelectQueryBuilder('users', new QueryExecutor(engine));
|
||||||
qb.join('orders', { 'users.id': { $col: 'orders.user_id' } });
|
qb.join('orders', { 'users.id': { $col: 'orders.user_id' } });
|
||||||
expect(qb.toAST().joins![0].type).toBe('INNER');
|
expect(qb.toAST().joins![0].type).toBe('INNER');
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -8,11 +8,10 @@ import { WAL, type WALStore } from '../../src/engine/aria/wal/log';
|
|||||||
import { WALRecordType } from '../../src/engine/aria/types';
|
import { WALRecordType } from '../../src/engine/aria/types';
|
||||||
import { BufferPool } from '../../src/engine/aria/buffer/pool';
|
import { BufferPool } from '../../src/engine/aria/buffer/pool';
|
||||||
import { MemoryBackend } from '../../src/engine/aria/store/backend';
|
import { MemoryBackend } from '../../src/engine/aria/store/backend';
|
||||||
import { createSchema } from '../../src/table/schema';
|
|
||||||
|
|
||||||
import { installOPFSMock } from '../helpers/opfs-mock';
|
import { resetOPFSMock } from '../helpers/storage-harness';
|
||||||
|
|
||||||
beforeEach(() => { installOPFSMock(new Map()); });
|
beforeEach(() => { resetOPFSMock(); });
|
||||||
|
|
||||||
describe('AriaEngine — Bloom + WAL + BufferPool', () => {
|
describe('AriaEngine — Bloom + WAL + BufferPool', () => {
|
||||||
// ---- BloomFilter 完整测试 ----
|
// ---- BloomFilter 完整测试 ----
|
||||||
|
|||||||
@@ -5,9 +5,9 @@ import { AriaEngine } from '../../src/engine/aria/index';
|
|||||||
import { createSchema } from '../../src/table/schema';
|
import { createSchema } from '../../src/table/schema';
|
||||||
import { MemoryBackend } from '../../src/engine/aria/store/backend';
|
import { MemoryBackend } from '../../src/engine/aria/store/backend';
|
||||||
|
|
||||||
import { installOPFSMock } from '../helpers/opfs-mock';
|
import { resetOPFSMock } from '../helpers/storage-harness';
|
||||||
|
|
||||||
beforeEach(() => { installOPFSMock(new Map()); });
|
beforeEach(() => { resetOPFSMock(); });
|
||||||
|
|
||||||
describe('AriaEngine — 批量扩展测试', () => {
|
describe('AriaEngine — 批量扩展测试', () => {
|
||||||
let engine: AriaEngine;
|
let engine: AriaEngine;
|
||||||
|
|||||||
@@ -98,8 +98,8 @@ describe('AriaEngine — EvictionManager', () => {
|
|||||||
}
|
}
|
||||||
|
|
||||||
it('access 更新 LRU', async () => {
|
it('access 更新 LRU', async () => {
|
||||||
let evicted = -1;
|
let _evicted = -1;
|
||||||
const em = new EvictionManager(3, async (p) => { evicted = p.pageId; });
|
const em = new EvictionManager(3, async (p) => { _evicted = p.pageId; });
|
||||||
const p = makePage(1);
|
const p = makePage(1);
|
||||||
em.add(p);
|
em.add(p);
|
||||||
em.access(p);
|
em.access(p);
|
||||||
|
|||||||
@@ -10,9 +10,9 @@
|
|||||||
import { AriaEngine } from '../../src/engine/aria/index';
|
import { AriaEngine } from '../../src/engine/aria/index';
|
||||||
import { createSchema } from '../../src/table/schema';
|
import { createSchema } from '../../src/table/schema';
|
||||||
|
|
||||||
import { installOPFSMock } from '../helpers/opfs-mock';
|
import { resetOPFSMock } from '../helpers/storage-harness';
|
||||||
|
|
||||||
beforeEach(() => { installOPFSMock(new Map()); });
|
beforeEach(() => { resetOPFSMock(); });
|
||||||
|
|
||||||
/** 构造小缓存 + 小 MemTable 阈值的引擎,快速产生多个 SSTable */
|
/** 构造小缓存 + 小 MemTable 阈值的引擎,快速产生多个 SSTable */
|
||||||
function createSmallCacheEngine(bufferPoolPages = 2) {
|
function createSmallCacheEngine(bufferPoolPages = 2) {
|
||||||
@@ -33,6 +33,27 @@ function makeRows(count: number): Record<string, unknown>[] {
|
|||||||
return rows;
|
return rows;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 按 schema 列裁剪行(v0.8.0 B-1 连带修正)。
|
||||||
|
*
|
||||||
|
* 此前 `makeRows` 无条件返回 `{id, name, age}`,而部分用例的表只有 `{id, name}` ——
|
||||||
|
* 多余列被引擎**静默丢弃**,测试因此"通过"。B-1 把未知列变成
|
||||||
|
* COLUMN_NOT_FOUND 后这些看起来无关的用例暴露出来。
|
||||||
|
*
|
||||||
|
* 这里的修法是让夹具**忠实反映表结构**(而不是放宽校验):测试本就不该依赖
|
||||||
|
* "写了不存在的列也不报错"这一行为。
|
||||||
|
*/
|
||||||
|
function rowsFor(schema: ReturnType<typeof createSchema>, count: number): Record<string, unknown>[] {
|
||||||
|
const allowed = Object.keys(schema.columns);
|
||||||
|
return makeRows(count).map((row) => {
|
||||||
|
const picked: Record<string, unknown> = {};
|
||||||
|
for (const key of allowed) {
|
||||||
|
if (key in row) picked[key] = row[key];
|
||||||
|
}
|
||||||
|
return picked;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
describe('AriaEngine SSTable 缓存内存上限', () => {
|
describe('AriaEngine SSTable 缓存内存上限', () => {
|
||||||
test('缓存大小受 cacheLimitBytes 约束', async () => {
|
test('缓存大小受 cacheLimitBytes 约束', async () => {
|
||||||
const engine = createSmallCacheEngine(2); // 2 * 4096 = 8KB 上限
|
const engine = createSmallCacheEngine(2); // 2 * 4096 = 8KB 上限
|
||||||
@@ -48,6 +69,7 @@ describe('AriaEngine SSTable 缓存内存上限', () => {
|
|||||||
flush(): Promise<void>;
|
flush(): Promise<void>;
|
||||||
getCacheSize(): number;
|
getCacheSize(): number;
|
||||||
getCacheLimit(): number;
|
getCacheLimit(): number;
|
||||||
|
getOversizedCount(): number;
|
||||||
getStats(): { sstableCount: number };
|
getStats(): { sstableCount: number };
|
||||||
};
|
};
|
||||||
// v0.6.1-perf: insert 不再隐式排空后台链(逐行 prefetchKeys 已移除),
|
// v0.6.1-perf: insert 不再隐式排空后台链(逐行 prefetchKeys 已移除),
|
||||||
@@ -57,44 +79,88 @@ describe('AriaEngine SSTable 缓存内存上限', () => {
|
|||||||
// 300 行 / 2KB 阈值 → 应产生多个 SSTable
|
// 300 行 / 2KB 阈值 → 应产生多个 SSTable
|
||||||
expect(stats.sstableCount).toBeGreaterThan(1);
|
expect(stats.sstableCount).toBeGreaterThan(1);
|
||||||
|
|
||||||
// 多轮查询后缓存仍受上限约束
|
// v0.8.0 契约修正:内存上限只约束**可驱逐条目**。
|
||||||
|
//
|
||||||
|
// 单个 SSTable 大于整个缓存上限时,它必须常驻:一旦驱逐,
|
||||||
|
// `loadSSTableReader` 未命中就会让调用方 `continue` 跳过整个文件 ——
|
||||||
|
// 那是静默丢数据(审计实测:300 行只能查回 59 行)。
|
||||||
|
// 因此这里断言的是"数据完整"这一真正重要的不变量,而不是一个
|
||||||
|
// 在极小缓存下无法成立的字节上限(上限 = cacheLimit + 单个最大 SSTable)。
|
||||||
for (let round = 0; round < 5; round++) {
|
for (let round = 0; round < 5; round++) {
|
||||||
const rows = await engine.find('users', { table: 'users', where: { age: 25 } });
|
const rows = await engine.find('users', { table: 'users', where: { age: 25 } });
|
||||||
expect(rows.length).toBe(10);
|
expect(rows.length).toBe(10);
|
||||||
expect(lsm.cacheSize).toBeLessThanOrEqual(lsm.cacheLimitBytes);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// 若所有 SSTable 都能装进上限,则缓存大小必须受上限约束
|
||||||
|
const oversizedPinned = lsm.getOversizedCount();
|
||||||
|
if (oversizedPinned === 0) {
|
||||||
|
expect(lsm.getCacheSize()).toBeLessThanOrEqual(lsm.getCacheLimit());
|
||||||
|
}
|
||||||
|
|
||||||
|
await engine.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
test('超大 SSTable 常驻缓存(驱逐会导致读取静默跳过整个文件)', async () => {
|
||||||
|
const engine = createSmallCacheEngine(1); // 4KB 上限,单个 SSTable 必然超过
|
||||||
|
await engine.open('cache-oversized-pin', 1);
|
||||||
|
const schema = createSchema('users', {
|
||||||
|
id: { type: 'string', primaryKey: true },
|
||||||
|
name: { type: 'string' },
|
||||||
|
});
|
||||||
|
await engine.createTable(schema);
|
||||||
|
// v0.8.0(B-1):夹具按 schema 裁剪(此前 makeRows 多带的 age 列被静默丢弃)
|
||||||
|
await engine.insert('users', rowsFor(schema, 300));
|
||||||
|
const lsm = (engine as any).lsm as {
|
||||||
|
flush(): Promise<void>;
|
||||||
|
getCacheSize(): number;
|
||||||
|
getCacheLimit(): number;
|
||||||
|
getOversizedCount(): number;
|
||||||
|
trimCache(): void;
|
||||||
|
getStats(): { sstableCount: number };
|
||||||
|
};
|
||||||
|
await lsm.flush();
|
||||||
|
expect(lsm.getStats().sstableCount).toBeGreaterThan(0);
|
||||||
|
|
||||||
|
// 强制裁剪后,超大文件仍必须可读(数据完整)
|
||||||
|
lsm.trimCache();
|
||||||
|
expect(lsm.getOversizedCount()).toBeGreaterThan(0);
|
||||||
|
const all = await engine.find('users', { table: 'users' });
|
||||||
|
expect(all.length).toBe(300);
|
||||||
await engine.close();
|
await engine.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
test('缓存驱逐后全表扫描仍返回完整数据(prefetch 兜底)', async () => {
|
test('缓存驱逐后全表扫描仍返回完整数据(prefetch 兜底)', async () => {
|
||||||
const engine = createSmallCacheEngine(1); // 4KB 上限,必然触发驱逐
|
const engine = createSmallCacheEngine(1); // 4KB 上限,必然触发驱逐
|
||||||
await engine.open('cache-evict-fullscan', 1);
|
await engine.open('cache-evict-fullscan', 1);
|
||||||
await engine.createTable(createSchema('users', {
|
const schema = createSchema('users', {
|
||||||
id: { type: 'string', primaryKey: true },
|
id: { type: 'string', primaryKey: true },
|
||||||
name: { type: 'string' },
|
name: { type: 'string' },
|
||||||
}));
|
});
|
||||||
|
await engine.createTable(schema);
|
||||||
|
|
||||||
const rows = makeRows(300);
|
// v0.8.0(B-1):夹具按 schema 裁剪(此前 makeRows 多带的 age 列被静默丢弃)
|
||||||
|
const rows = rowsFor(schema, 300);
|
||||||
await engine.insert('users', rows);
|
await engine.insert('users', rows);
|
||||||
|
|
||||||
|
// v0.8.0: 这是最关键的数据完整性断言 —— 缓存上限极小(4KB)而 SSTable 更大时,
|
||||||
|
// 读取路径必须仍然返回**全部** 300 行(此前会静默少数据)。
|
||||||
const all = await engine.find('users', { table: 'users' });
|
const all = await engine.find('users', { table: 'users' });
|
||||||
expect(all.length).toBe(300);
|
expect(all.length).toBe(300);
|
||||||
|
|
||||||
const lsm = (engine as any).lsm;
|
|
||||||
expect(lsm.cacheSize).toBeLessThanOrEqual(lsm.cacheLimitBytes);
|
|
||||||
await engine.close();
|
await engine.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
test('缓存驱逐后 PK 等值查询仍正确(prefetchKeys 兜底)', async () => {
|
test('缓存驱逐后 PK 等值查询仍正确(prefetchKeys 兜底)', async () => {
|
||||||
const engine = createSmallCacheEngine(1);
|
const engine = createSmallCacheEngine(1);
|
||||||
await engine.open('cache-evict-pk', 1);
|
await engine.open('cache-evict-pk', 1);
|
||||||
await engine.createTable(createSchema('users', {
|
const schema = createSchema('users', {
|
||||||
id: { type: 'string', primaryKey: true },
|
id: { type: 'string', primaryKey: true },
|
||||||
name: { type: 'string' },
|
name: { type: 'string' },
|
||||||
}));
|
});
|
||||||
|
await engine.createTable(schema);
|
||||||
|
|
||||||
const rows = makeRows(300);
|
// v0.8.0(B-1):夹具按 schema 裁剪(此前 makeRows 多带的 age 列被静默丢弃)
|
||||||
|
const rows = rowsFor(schema, 300);
|
||||||
await engine.insert('users', rows);
|
await engine.insert('users', rows);
|
||||||
|
|
||||||
// 分散查询多个 PK,每次都会经历 驱逐+重新加载
|
// 分散查询多个 PK,每次都会经历 驱逐+重新加载
|
||||||
@@ -164,16 +230,18 @@ describe('AriaEngine SSTable 缓存内存上限', () => {
|
|||||||
test('写入路径不突破缓存上限(flush 后立即裁剪)', async () => {
|
test('写入路径不突破缓存上限(flush 后立即裁剪)', async () => {
|
||||||
const engine = createSmallCacheEngine(2);
|
const engine = createSmallCacheEngine(2);
|
||||||
await engine.open('cache-write-bound', 1);
|
await engine.open('cache-write-bound', 1);
|
||||||
await engine.createTable(createSchema('users', {
|
const schema = createSchema('users', {
|
||||||
id: { type: 'string', primaryKey: true },
|
id: { type: 'string', primaryKey: true },
|
||||||
name: { type: 'string' },
|
name: { type: 'string' },
|
||||||
}));
|
});
|
||||||
|
await engine.createTable(schema);
|
||||||
|
|
||||||
// 分批写入,每批都触发多次 flush
|
// 分批写入,每批都触发多次 flush
|
||||||
for (let batch = 0; batch < 10; batch++) {
|
for (let batch = 0; batch < 10; batch++) {
|
||||||
await engine.insert('users', makeRows(30).map((r, i) => ({ ...r, id: `b${batch}_u${i}` })));
|
// v0.8.0(B-1):夹具按 schema 裁剪(此前 makeRows 的 age 列被静默丢弃)
|
||||||
|
await engine.insert('users', rowsFor(schema, 30).map((r, i) => ({ ...r, id: `b${batch}_u${i}` })));
|
||||||
const lsm = (engine as any).lsm;
|
const lsm = (engine as any).lsm;
|
||||||
expect(lsm.cacheSize).toBeLessThanOrEqual(lsm.cacheLimitBytes);
|
expect(lsm.getCacheSize()).toBeLessThanOrEqual(lsm.getCacheLimit());
|
||||||
}
|
}
|
||||||
|
|
||||||
const all = await engine.find('users', { table: 'users' });
|
const all = await engine.find('users', { table: 'users' });
|
||||||
@@ -239,12 +307,14 @@ describe('AriaEngine SSTable 缓存内存上限', () => {
|
|||||||
test('回归:删除后 tombstone 跨 flush 仍生效(不残留旧数据)', async () => {
|
test('回归:删除后 tombstone 跨 flush 仍生效(不残留旧数据)', async () => {
|
||||||
const engine = createSmallCacheEngine(4);
|
const engine = createSmallCacheEngine(4);
|
||||||
await engine.open('regression-tombstone', 1);
|
await engine.open('regression-tombstone', 1);
|
||||||
await engine.createTable(createSchema('users', {
|
const schema = createSchema('users', {
|
||||||
id: { type: 'string', primaryKey: true },
|
id: { type: 'string', primaryKey: true },
|
||||||
age: { type: 'number', index: true },
|
age: { type: 'number', index: true },
|
||||||
}));
|
});
|
||||||
|
await engine.createTable(schema);
|
||||||
|
|
||||||
await engine.insert('users', makeRows(120));
|
// v0.8.0(B-1):夹具按 schema 裁剪(此前 makeRows 的 name 列被静默丢弃)
|
||||||
|
await engine.insert('users', rowsFor(schema, 120));
|
||||||
|
|
||||||
// 分批删除,触发多次 flush
|
// 分批删除,触发多次 flush
|
||||||
for (let batch = 0; batch < 4; batch++) {
|
for (let batch = 0; batch < 4; batch++) {
|
||||||
|
|||||||
@@ -9,9 +9,9 @@
|
|||||||
import { AriaEngine } from '../../src/engine/aria/index';
|
import { AriaEngine } from '../../src/engine/aria/index';
|
||||||
import { createSchema } from '../../src/table/schema';
|
import { createSchema } from '../../src/table/schema';
|
||||||
|
|
||||||
import { installOPFSMock } from '../helpers/opfs-mock';
|
import { resetOPFSMock, readManifestNamespace } from '../helpers/storage-harness';
|
||||||
|
|
||||||
beforeEach(() => { installOPFSMock(new Map()); });
|
beforeEach(() => { resetOPFSMock(); });
|
||||||
|
|
||||||
let idbCounter = 0;
|
let idbCounter = 0;
|
||||||
function uniqueDB(): string {
|
function uniqueDB(): string {
|
||||||
@@ -41,12 +41,15 @@ async function listSSTKeys(engine: AriaEngine): Promise<string[]> {
|
|||||||
return keys.filter((k) => k.startsWith('pg_'));
|
return keys.filter((k) => k.startsWith('pg_'));
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 读取主 LSM 的 SSTable meta 列表 */
|
/**
|
||||||
|
* 读取主 LSM 的 SSTable meta 列表(v0.8.0 B-6:改读 manifest)。
|
||||||
|
*
|
||||||
|
* 元数据不再是独立的裸 JSON:它随 manifest 原子提交(带 CRC + 世代号)。
|
||||||
|
* 旧布局下 `JSON.parse` 失败会被当成"没有文件",即元数据损坏 = 静默空库。
|
||||||
|
*/
|
||||||
async function listSSTMetas(engine: AriaEngine): Promise<{ id: number; pageIds?: number[] }[]> {
|
async function listSSTMetas(engine: AriaEngine): Promise<{ id: number; pageIds?: number[] }[]> {
|
||||||
const backend = (engine as any).backend;
|
const backend = (engine as any).backend;
|
||||||
const raw = await backend.read('__aria_lsm_meta');
|
return readManifestNamespace(backend, 'main');
|
||||||
if (!raw) return [];
|
|
||||||
return JSON.parse(new TextDecoder().decode(raw)) as { id: number; pageIds?: number[] }[];
|
|
||||||
}
|
}
|
||||||
|
|
||||||
describe('AriaEngine — SSTable CRC 损坏检测(集成)', () => {
|
describe('AriaEngine — SSTable CRC 损坏检测(集成)', () => {
|
||||||
|
|||||||
@@ -0,0 +1,204 @@
|
|||||||
|
/**
|
||||||
|
* v0.8.0 回归套件 —— A38 页面化压缩 + A39 compressLZ4 复杂度
|
||||||
|
* ============================================================================
|
||||||
|
* A38 **`compression` 在页面化路径上被静默忽略**
|
||||||
|
* `AriaEngine` 的默认配置是 `pageStorage` 自动(OPFS 后端下为 true),
|
||||||
|
* 而压缩只写在"整 value 存一个 backend value"的旧分支里,页面化分支
|
||||||
|
* **提前 return** —— 于是 `compression: true` 在默认配置下完全没有效果,
|
||||||
|
* 且没有任何提示。本套件的断言方式是**结构性**的:
|
||||||
|
* 开启压缩后 `SSTableMeta.totalSize`(实际落盘字节数)必须显著变小 ——
|
||||||
|
* 它与实现细节无关,只反映"数据真的被压缩了"。
|
||||||
|
*
|
||||||
|
* * 为什么之前没被发现:既有测试只断言"压缩后能读回来"(往返正确),
|
||||||
|
* 而"根本没压缩"同样能正确读回 —— 断言太弱,测不出"功能是否生效"。
|
||||||
|
*
|
||||||
|
* A39 **`compressLZ4` 匹配搜索是 O(n²)**
|
||||||
|
* 旧实现逐字节向前扫描最多 65535 个候选位置,每个位置再逐字节比较。
|
||||||
|
* 在低压缩率数据(伪随机)上退化为二次复杂度:实测 60KB 输入耗时 **2.3 秒**
|
||||||
|
* (新实现 6ms,约 390×)。SSTable 页/日志段正好是几百 KB 到几 MB,
|
||||||
|
* 因此这是普通写入路径上的真实卡顿,不是极端场景。
|
||||||
|
*
|
||||||
|
* 本套件同时锁定"输出格式未变":新实现与保留的线性参考实现在同一输入上
|
||||||
|
* 必须产出**可互相解压**的流(压缩率可略有差异,因为新实现能找到不同但
|
||||||
|
* 同样合法的匹配)。
|
||||||
|
*/
|
||||||
|
import { AriaEngine } from '../../src/engine/aria/index';
|
||||||
|
import { compressLZ4, compressLZ4LinearReference, decompressLZ4 } from '../../src/engine/aria/compression/lz4';
|
||||||
|
import { resetOPFSMock } from '../helpers/storage-harness';
|
||||||
|
|
||||||
|
beforeEach(() => { resetOPFSMock(); });
|
||||||
|
|
||||||
|
const SCHEMA = () => ({
|
||||||
|
name: 't',
|
||||||
|
columns: {
|
||||||
|
id: { type: 'string' as const, primaryKey: true },
|
||||||
|
blob: { type: 'string' as const },
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
/** 高度可压缩的载荷 */
|
||||||
|
const COMPRESSIBLE = 'x'.repeat(500) + 'y'.repeat(500);
|
||||||
|
|
||||||
|
/** 写入 n 行并返回 SSTable 元数据统计 */
|
||||||
|
async function writeAndMeasure(opts: {
|
||||||
|
dbName: string;
|
||||||
|
pageStorage: boolean;
|
||||||
|
compression: boolean;
|
||||||
|
rows?: number;
|
||||||
|
}): Promise<{ storedBytes: number; sstableCount: number; rowCount: number; roundTripOk: boolean }> {
|
||||||
|
const rows = opts.rows ?? 60;
|
||||||
|
const engine = new AriaEngine({
|
||||||
|
storageBackend: 'opfs',
|
||||||
|
pageStorage: opts.pageStorage,
|
||||||
|
compression: opts.compression,
|
||||||
|
memtableSizeThreshold: 2048,
|
||||||
|
checkpointInterval: 100_000_000,
|
||||||
|
} as never);
|
||||||
|
await engine.open(opts.dbName, 1);
|
||||||
|
await engine.createTable(SCHEMA() as never);
|
||||||
|
for (let i = 0; i < rows; i++) await engine.insert('t', [{ id: `k${i}`, blob: COMPRESSIBLE }]);
|
||||||
|
|
||||||
|
const lsm = (engine as unknown as {
|
||||||
|
lsm: { flush(): Promise<void>; sstableStore: { listMeta(): Promise<Array<{ totalSize: number }>> } };
|
||||||
|
}).lsm;
|
||||||
|
// v0.8.0(B-6):**测量前显式 flush**。
|
||||||
|
//
|
||||||
|
// 此前这里直接读 meta 求和,于是"落盘字节数"取决于测量瞬间有多少数据恰好在
|
||||||
|
// SSTable 里 —— 而 manifest 提交(单一提交点)让每次 flush 多一次原子提交,
|
||||||
|
// 后台 flush 的进度随之变化,两次实验的"已落盘比例"不再相同,比值就变成在
|
||||||
|
// 测时序而不是测压缩(实测:未 flush 时 off=19040/on=5130,flush 后
|
||||||
|
// off=58570/on=10096 —— 后者才是同一份数据的真实压缩率)。
|
||||||
|
await lsm.flush();
|
||||||
|
const metas = await lsm.sstableStore.listMeta();
|
||||||
|
const storedBytes = metas.reduce((sum, m) => sum + (m.totalSize ?? 0), 0);
|
||||||
|
|
||||||
|
const read = await engine.find('t', { table: 't' });
|
||||||
|
const roundTripOk = read.length === rows && read.every((r) => r.blob === COMPRESSIBLE);
|
||||||
|
await engine.close();
|
||||||
|
return { storedBytes, sstableCount: metas.length, rowCount: read.length, roundTripOk };
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('[v0.8.0] A38 compression 必须真的生效(含页面化路径)', () => {
|
||||||
|
it('页面化路径:开启压缩后落盘字节数显著下降', async () => {
|
||||||
|
const off = await writeAndMeasure({ dbName: 'a38-pages-off', pageStorage: true, compression: false });
|
||||||
|
const on = await writeAndMeasure({ dbName: 'a38-pages-on', pageStorage: true, compression: true });
|
||||||
|
|
||||||
|
// 先确认两次实验都写出了 SSTable(否则下面的比值没有意义)
|
||||||
|
expect(off.sstableCount).toBeGreaterThan(0);
|
||||||
|
expect(on.sstableCount).toBeGreaterThan(0);
|
||||||
|
// 数据完整性不受压缩影响
|
||||||
|
expect(off.roundTripOk).toBe(true);
|
||||||
|
expect(on.roundTripOk).toBe(true);
|
||||||
|
|
||||||
|
// 核心断言:修复前 compression 在页面化路径上被忽略 → 两者字节数相同
|
||||||
|
expect(on.storedBytes).toBeLessThan(off.storedBytes);
|
||||||
|
// 高可压缩内容应有的量级(10 倍是保守下界,实测远高于此)
|
||||||
|
expect(off.storedBytes).toBeGreaterThan(on.storedBytes * 5);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('整 value 路径:压缩同样生效(两条路径行为一致)', async () => {
|
||||||
|
const off = await writeAndMeasure({ dbName: 'a38-whole-off', pageStorage: false, compression: false });
|
||||||
|
const on = await writeAndMeasure({ dbName: 'a38-whole-on', pageStorage: false, compression: true });
|
||||||
|
expect(off.roundTripOk).toBe(true);
|
||||||
|
expect(on.roundTripOk).toBe(true);
|
||||||
|
expect(off.storedBytes).toBeGreaterThan(on.storedBytes * 5);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('页面化 + 压缩:重开后数据与索引完整', async () => {
|
||||||
|
const engine = new AriaEngine({
|
||||||
|
storageBackend: 'opfs', pageStorage: true, compression: true,
|
||||||
|
memtableSizeThreshold: 2048, checkpointInterval: 100_000_000,
|
||||||
|
} as never);
|
||||||
|
await engine.open('a38-reopen', 1);
|
||||||
|
await engine.createTable(SCHEMA() as never);
|
||||||
|
for (let i = 0; i < 40; i++) await engine.insert('t', [{ id: `k${i}`, blob: COMPRESSIBLE }]);
|
||||||
|
// 不 close(模拟崩溃后重开,覆盖压缩数据的恢复路径)
|
||||||
|
const engine2 = new AriaEngine({
|
||||||
|
storageBackend: 'opfs', pageStorage: true, compression: true,
|
||||||
|
memtableSizeThreshold: 2048, checkpointInterval: 100_000_000,
|
||||||
|
} as never);
|
||||||
|
await engine2.open('a38-reopen', 1);
|
||||||
|
const rows = await engine2.find('t', { table: 't' });
|
||||||
|
expect(rows).toHaveLength(40);
|
||||||
|
expect(rows.every((r) => r.blob === COMPRESSIBLE)).toBe(true);
|
||||||
|
// 单行精确读取(走 SSTable 数据块解析,而不只是全表扫描)
|
||||||
|
const one = await engine2.find('t', { table: 't', where: { id: 'k7' } });
|
||||||
|
expect(one).toHaveLength(1);
|
||||||
|
expect(one[0].blob).toBe(COMPRESSIBLE);
|
||||||
|
await engine.close().catch(() => { /* 已崩溃语义,忽略 */ });
|
||||||
|
await engine2.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('压缩关闭时数据同样完整(回归护栏)', async () => {
|
||||||
|
const result = await writeAndMeasure({ dbName: 'a38-nocomp', pageStorage: true, compression: false });
|
||||||
|
expect(result.roundTripOk).toBe(true);
|
||||||
|
expect(result.rowCount).toBe(60);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// A39:compressLZ4 复杂度与格式兼容
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/** 确定性伪随机(避免测试本身依赖随机源) */
|
||||||
|
function makeRandom(length: number, seed = 42): Uint8Array {
|
||||||
|
let s = seed >>> 0;
|
||||||
|
const out = new Uint8Array(length);
|
||||||
|
for (let i = 0; i < length; i++) {
|
||||||
|
s = (Math.imul(s, 1664525) + 1013904223) >>> 0;
|
||||||
|
out[i] = (s >>> 24) & 0xFF;
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('[v0.8.0] A39 compressLZ4:复杂度与格式兼容', () => {
|
||||||
|
const cases: Array<[string, Uint8Array]> = [
|
||||||
|
['空输入', new Uint8Array(0)],
|
||||||
|
['单字节', new Uint8Array([42])],
|
||||||
|
['全同字节', new Uint8Array(5000).fill(7)],
|
||||||
|
['周期序列', new Uint8Array(Array.from({ length: 40_000 }, (_, i) => i % 7))],
|
||||||
|
['伪随机 8KB', makeRandom(8_000)],
|
||||||
|
['伪随机 60KB', makeRandom(60_000)],
|
||||||
|
];
|
||||||
|
|
||||||
|
it.each(cases)('%s:往返一致且与线性参考实现可互解', (_label, data) => {
|
||||||
|
const fast = compressLZ4(data);
|
||||||
|
const slow = compressLZ4LinearReference(data);
|
||||||
|
|
||||||
|
// 新实现自身往返
|
||||||
|
const fromFast = decompressLZ4(fast);
|
||||||
|
expect(fromFast.length).toBe(data.length);
|
||||||
|
expect(Array.from(fromFast)).toEqual(Array.from(data));
|
||||||
|
|
||||||
|
// 旧实现的输出,新解压器必须能读(格式兼容:既有落盘数据不需要迁移)
|
||||||
|
const fromSlow = decompressLZ4(slow);
|
||||||
|
expect(Array.from(fromSlow)).toEqual(Array.from(data));
|
||||||
|
|
||||||
|
// 新实现的输出,旧解压器(同一份代码,此处仅作对称性验证)也必须能读
|
||||||
|
expect(Array.from(decompressLZ4(fast))).toEqual(Array.from(data));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('低压缩率数据不再退化(60KB 伪随机在 1 秒内完成)', () => {
|
||||||
|
const data = makeRandom(60_000);
|
||||||
|
const start = Date.now();
|
||||||
|
const compressed = compressLZ4(data);
|
||||||
|
const elapsed = Date.now() - start;
|
||||||
|
|
||||||
|
// 修复前该输入耗时约 2.3 秒(逐位置线性扫描 → 二次复杂度)。
|
||||||
|
// 阈值取 1 秒:即使 CI 慢 10 倍也仍能通过,而回退到旧实现必然失败。
|
||||||
|
expect(elapsed).toBeLessThan(1000);
|
||||||
|
expect(decompressLZ4(compressed).length).toBe(data.length);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('大输入(1MB 可压缩内容)在合理时间内完成', () => {
|
||||||
|
const data = new Uint8Array(1_000_000);
|
||||||
|
for (let i = 0; i < data.length; i++) data[i] = i % 251;
|
||||||
|
const start = Date.now();
|
||||||
|
const compressed = compressLZ4(data);
|
||||||
|
const elapsed = Date.now() - start;
|
||||||
|
expect(elapsed).toBeLessThan(3000);
|
||||||
|
// 周期性内容应被显著压缩
|
||||||
|
expect(compressed.length).toBeLessThan(data.length);
|
||||||
|
expect(decompressLZ4(compressed).length).toBe(data.length);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -5,9 +5,9 @@
|
|||||||
import { AriaEngine } from '../../src/engine/aria/index';
|
import { AriaEngine } from '../../src/engine/aria/index';
|
||||||
import { createSchema } from '../../src/table/schema';
|
import { createSchema } from '../../src/table/schema';
|
||||||
|
|
||||||
import { installOPFSMock } from '../helpers/opfs-mock';
|
import { resetOPFSMock } from '../helpers/storage-harness';
|
||||||
|
|
||||||
beforeEach(() => { installOPFSMock(new Map()); });
|
beforeEach(() => { resetOPFSMock(); });
|
||||||
|
|
||||||
describe('AriaEngine — 扩展边缘测试', () => {
|
describe('AriaEngine — 扩展边缘测试', () => {
|
||||||
let engine: AriaEngine;
|
let engine: AriaEngine;
|
||||||
@@ -69,7 +69,7 @@ describe('AriaEngine — 扩展边缘测试', () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
it('多列 ORDER BY', async () => {
|
it('多列 ORDER BY', async () => {
|
||||||
const rows = await engine.find('users', { table: 'users', orderBy: [{ column: 'age', direction: 'asc' }, { column: 'score', direction: 'desc' }] });
|
const rows = await engine.find('users', { table: 'users', orderBy: [{ column: 'age', direction: 'asc' }, { column: 'score', direction: 'desc' }] }) as Array<{ age: number }>;
|
||||||
expect(rows[0].age).toBeLessThanOrEqual(rows[4].age);
|
expect(rows[0].age).toBeLessThanOrEqual(rows[4].age);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
@@ -15,9 +15,10 @@ import { EncryptedBackend } from '../../src/engine/aria/store/encrypted_backend'
|
|||||||
import { MemoryBackend } from '../../src/engine/aria/store/backend';
|
import { MemoryBackend } from '../../src/engine/aria/store/backend';
|
||||||
import { createSchema } from '../../src/table/schema';
|
import { createSchema } from '../../src/table/schema';
|
||||||
|
|
||||||
import { installOPFSMock } from '../helpers/opfs-mock';
|
import { resetOPFSMock } from '../helpers/storage-harness';
|
||||||
|
import { decode as decodeBytes } from '../helpers/assertions';
|
||||||
|
|
||||||
beforeEach(() => { installOPFSMock(new Map()); });
|
beforeEach(() => { resetOPFSMock(); });
|
||||||
|
|
||||||
let idbCounter = 0;
|
let idbCounter = 0;
|
||||||
function uniqueDB(): string {
|
function uniqueDB(): string {
|
||||||
@@ -48,11 +49,11 @@ describe('AriaEngine — EncryptedBackend 单元', () => {
|
|||||||
await backend.write('k1', payload);
|
await backend.write('k1', payload);
|
||||||
const back = await backend.read('k1');
|
const back = await backend.read('k1');
|
||||||
expect(back).not.toBeNull();
|
expect(back).not.toBeNull();
|
||||||
expect(new TextDecoder().decode(back)).toBe('hello encrypted world');
|
expect(decodeBytes(back)).toBe('hello encrypted world');
|
||||||
|
|
||||||
// 底层是密文(非明文)
|
// 底层是密文(非明文)
|
||||||
const raw = await inner.read('k1');
|
const raw = await inner.read('k1');
|
||||||
const rawStr = new TextDecoder().decode(raw);
|
const rawStr = decodeBytes(raw, 'ciphertext');
|
||||||
expect(rawStr).not.toContain('hello encrypted world');
|
expect(rawStr).not.toContain('hello encrypted world');
|
||||||
|
|
||||||
await backend.close();
|
await backend.close();
|
||||||
@@ -65,8 +66,8 @@ describe('AriaEngine — EncryptedBackend 单元', () => {
|
|||||||
|
|
||||||
const enc = (s: string) => new TextEncoder().encode(s).buffer;
|
const enc = (s: string) => new TextEncoder().encode(s).buffer;
|
||||||
await backend.writeMany({ a: enc('AAA'), b: enc('BBB'), c: enc('CCC') });
|
await backend.writeMany({ a: enc('AAA'), b: enc('BBB'), c: enc('CCC') });
|
||||||
expect(new TextDecoder().decode(await backend.read('a'))).toBe('AAA');
|
expect(decodeBytes(await backend.read('a'))).toBe('AAA');
|
||||||
expect(new TextDecoder().decode(await backend.read('c'))).toBe('CCC');
|
expect(decodeBytes(await backend.read('c'))).toBe('CCC');
|
||||||
|
|
||||||
await backend.deleteMany(['a', 'c']);
|
await backend.deleteMany(['a', 'c']);
|
||||||
expect(await backend.exists('a')).toBe(false);
|
expect(await backend.exists('a')).toBe(false);
|
||||||
@@ -94,8 +95,8 @@ describe('AriaEngine — EncryptedBackend 单元', () => {
|
|||||||
expect((rawX as ArrayBuffer).byteLength).toBe((rawY as ArrayBuffer).byteLength);
|
expect((rawX as ArrayBuffer).byteLength).toBe((rawY as ArrayBuffer).byteLength);
|
||||||
|
|
||||||
// 但解密一致
|
// 但解密一致
|
||||||
expect(new TextDecoder().decode(await backend.read('x'))).toBe('same plaintext');
|
expect(decodeBytes(await backend.read('x'))).toBe('same plaintext');
|
||||||
expect(new TextDecoder().decode(await backend.read('y'))).toBe('same plaintext');
|
expect(decodeBytes(await backend.read('y'))).toBe('same plaintext');
|
||||||
await backend.close();
|
await backend.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -115,7 +116,7 @@ describe('AriaEngine — EncryptedBackend 单元', () => {
|
|||||||
const backend2 = new EncryptedBackend(inner, 'pw');
|
const backend2 = new EncryptedBackend(inner, 'pw');
|
||||||
await backend2.open('enc-unit-4');
|
await backend2.open('enc-unit-4');
|
||||||
await backend2.write('k2', new TextEncoder().encode('v2').buffer);
|
await backend2.write('k2', new TextEncoder().encode('v2').buffer);
|
||||||
expect(new TextDecoder().decode(await backend2.read('k2'))).toBe('v2');
|
expect(decodeBytes(await backend2.read('k2'))).toBe('v2');
|
||||||
|
|
||||||
// 换密码被拒(keymeta 与旧密码绑定)——注意:MemoryBackend close 清空 store,
|
// 换密码被拒(keymeta 与旧密码绑定)——注意:MemoryBackend close 清空 store,
|
||||||
// 因此在 close 前验证(backend2 仍持有 inner)
|
// 因此在 close 前验证(backend2 仍持有 inner)
|
||||||
|
|||||||
@@ -4,9 +4,9 @@
|
|||||||
import { AriaEngine } from '../../src/engine/aria/index';
|
import { AriaEngine } from '../../src/engine/aria/index';
|
||||||
import { createSchema } from '../../src/table/schema';
|
import { createSchema } from '../../src/table/schema';
|
||||||
|
|
||||||
import { installOPFSMock } from '../helpers/opfs-mock';
|
import { resetOPFSMock } from '../helpers/storage-harness';
|
||||||
|
|
||||||
beforeEach(() => { installOPFSMock(new Map()); });
|
beforeEach(() => { resetOPFSMock(); });
|
||||||
|
|
||||||
describe('AriaEngine — 补充测试', () => {
|
describe('AriaEngine — 补充测试', () => {
|
||||||
let engine: AriaEngine;
|
let engine: AriaEngine;
|
||||||
|
|||||||
@@ -7,9 +7,9 @@ import { createSchema } from '../../src/table/schema';
|
|||||||
import { MemoryBackend } from '../../src/engine/aria/store/backend';
|
import { MemoryBackend } from '../../src/engine/aria/store/backend';
|
||||||
import { checkFieldType } from '../../src/table/schema';
|
import { checkFieldType } from '../../src/table/schema';
|
||||||
|
|
||||||
import { installOPFSMock } from '../helpers/opfs-mock';
|
import { resetOPFSMock } from '../helpers/storage-harness';
|
||||||
|
|
||||||
beforeEach(() => { installOPFSMock(new Map()); });
|
beforeEach(() => { resetOPFSMock(); });
|
||||||
|
|
||||||
describe('AriaEngine — 最终扩展测试', () => {
|
describe('AriaEngine — 最终扩展测试', () => {
|
||||||
let engine: AriaEngine;
|
let engine: AriaEngine;
|
||||||
|
|||||||
@@ -8,14 +8,14 @@
|
|||||||
*/
|
*/
|
||||||
import { AriaEngine } from '../../src/engine/aria/index';
|
import { AriaEngine } from '../../src/engine/aria/index';
|
||||||
import { createSchema } from '../../src/table/schema';
|
import { createSchema } from '../../src/table/schema';
|
||||||
import { installOPFSMock } from '../helpers/opfs-mock';
|
import { resetOPFSMock } from '../helpers/storage-harness';
|
||||||
|
|
||||||
let counter = 0;
|
let counter = 0;
|
||||||
function uniqueDB(): string {
|
function uniqueDB(): string {
|
||||||
return `idxrace-${Date.now()}-${++counter}-${Math.random().toString(36).slice(2, 6)}`;
|
return `idxrace-${Date.now()}-${++counter}-${Math.random().toString(36).slice(2, 6)}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
beforeEach(() => { installOPFSMock(new Map()); });
|
beforeEach(() => { resetOPFSMock(); });
|
||||||
|
|
||||||
describe('AriaEngine — 二级索引完整性(P0 回归)', () => {
|
describe('AriaEngine — 二级索引完整性(P0 回归)', () => {
|
||||||
it('5 万行写入:索引查询与主表一致(修复前丢 106~771 条)', async () => {
|
it('5 万行写入:索引查询与主表一致(修复前丢 106~771 条)', async () => {
|
||||||
|
|||||||
@@ -5,9 +5,9 @@
|
|||||||
import { AriaEngine } from '../../src/engine/aria/index';
|
import { AriaEngine } from '../../src/engine/aria/index';
|
||||||
import { createSchema } from '../../src/table/schema';
|
import { createSchema } from '../../src/table/schema';
|
||||||
|
|
||||||
import { installOPFSMock } from '../helpers/opfs-mock';
|
import { resetOPFSMock } from '../helpers/storage-harness';
|
||||||
|
|
||||||
beforeEach(() => { installOPFSMock(new Map()); });
|
beforeEach(() => { resetOPFSMock(); });
|
||||||
|
|
||||||
describe('AriaEngine — 二级索引查询', () => {
|
describe('AriaEngine — 二级索引查询', () => {
|
||||||
let engine: AriaEngine;
|
let engine: AriaEngine;
|
||||||
|
|||||||
@@ -275,7 +275,7 @@ describe('MetonaSqlark + diskEngine: kv(高层 API)', () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
// 事务回滚
|
// 事务回滚
|
||||||
await expect(db.transaction(async (trx) => {
|
await expect(db.transaction(async (trx: { table: (n: string) => any }) => {
|
||||||
await trx.table('users').insert({ id: '1' });
|
await trx.table('users').insert({ id: '1' });
|
||||||
throw new Error('boom');
|
throw new Error('boom');
|
||||||
})).rejects.toThrow('boom');
|
})).rejects.toThrow('boom');
|
||||||
|
|||||||
@@ -12,9 +12,9 @@ import { AriaEngine } from '../../src/engine/aria/index';
|
|||||||
import { DatabaseLock, lockName } from '../../src/engine/aria/locks';
|
import { DatabaseLock, lockName } from '../../src/engine/aria/locks';
|
||||||
import { createSchema } from '../../src/table/schema';
|
import { createSchema } from '../../src/table/schema';
|
||||||
|
|
||||||
import { installOPFSMock } from '../helpers/opfs-mock';
|
import { resetOPFSMock } from '../helpers/storage-harness';
|
||||||
|
|
||||||
beforeEach(() => { installOPFSMock(new Map()); });
|
beforeEach(() => { resetOPFSMock(); });
|
||||||
|
|
||||||
let idbCounter = 0;
|
let idbCounter = 0;
|
||||||
function uniqueDB(): string {
|
function uniqueDB(): string {
|
||||||
|
|||||||
@@ -7,9 +7,10 @@ import { createSchema } from '../../src/table/schema';
|
|||||||
import { QueryExecutor } from '../../src/query/executor';
|
import { QueryExecutor } from '../../src/query/executor';
|
||||||
import { parse } from '../../src/sql/parser';
|
import { parse } from '../../src/sql/parser';
|
||||||
|
|
||||||
import { installOPFSMock } from '../helpers/opfs-mock';
|
import { resetOPFSMock } from '../helpers/storage-harness';
|
||||||
|
import { object } from '../helpers/assertions';
|
||||||
|
|
||||||
beforeEach(() => { installOPFSMock(new Map()); });
|
beforeEach(() => { resetOPFSMock(); });
|
||||||
|
|
||||||
describe('AriaEngine — ANALYZE/VACUUM/REINDEX + EXPLAIN', () => {
|
describe('AriaEngine — ANALYZE/VACUUM/REINDEX + EXPLAIN', () => {
|
||||||
let engine: AriaEngine;
|
let engine: AriaEngine;
|
||||||
@@ -78,7 +79,9 @@ describe('AriaEngine — ANALYZE/VACUUM/REINDEX + EXPLAIN', () => {
|
|||||||
const executor = new QueryExecutor(engine);
|
const executor = new QueryExecutor(engine);
|
||||||
const selectStmt = parse('SELECT * FROM users WHERE id = \'1\'');
|
const selectStmt = parse('SELECT * FROM users WHERE id = \'1\'');
|
||||||
const explainStmt: any = { type: 'EXPLAIN', query: selectStmt };
|
const explainStmt: any = { type: 'EXPLAIN', query: selectStmt };
|
||||||
const plan = await executor.execute(explainStmt);
|
const plan = object<{ type: string; table: string; usingIndex?: unknown; actualTimeMs: number }>(
|
||||||
|
await executor.execute(explainStmt),
|
||||||
|
);
|
||||||
expect(plan.type).toBe('SELECT');
|
expect(plan.type).toBe('SELECT');
|
||||||
expect(plan.table).toBe('users');
|
expect(plan.table).toBe('users');
|
||||||
expect(plan.usingIndex).toBeDefined();
|
expect(plan.usingIndex).toBeDefined();
|
||||||
|
|||||||
@@ -9,7 +9,7 @@
|
|||||||
import { AriaEngine } from '../../src/engine/aria/index';
|
import { AriaEngine } from '../../src/engine/aria/index';
|
||||||
import { createSchema } from '../../src/table/schema';
|
import { createSchema } from '../../src/table/schema';
|
||||||
import { SharedMemoryBackend } from '../../src/engine/kvstore/shared_memory_medium';
|
import { SharedMemoryBackend } from '../../src/engine/kvstore/shared_memory_medium';
|
||||||
import { installOPFSMock } from '../helpers/opfs-mock';
|
import { resetOPFSMock } from '../helpers/storage-harness';
|
||||||
|
|
||||||
let counter = 0;
|
let counter = 0;
|
||||||
function uniqueDB(): string {
|
function uniqueDB(): string {
|
||||||
@@ -56,7 +56,7 @@ async function reopen(dbName: string, config: Record<string, unknown>): Promise<
|
|||||||
|
|
||||||
beforeEach(() => {
|
beforeEach(() => {
|
||||||
SharedMemoryBackend.clearRegistry();
|
SharedMemoryBackend.clearRegistry();
|
||||||
installOPFSMock(new Map());
|
resetOPFSMock();
|
||||||
});
|
});
|
||||||
|
|
||||||
describe('生产矩阵审计 — 后端 × 核心功能', () => {
|
describe('生产矩阵审计 — 后端 × 核心功能', () => {
|
||||||
|
|||||||
@@ -5,9 +5,9 @@
|
|||||||
import { AriaEngine } from '../../src/engine/aria/index';
|
import { AriaEngine } from '../../src/engine/aria/index';
|
||||||
import { createSchema } from '../../src/table/schema';
|
import { createSchema } from '../../src/table/schema';
|
||||||
|
|
||||||
import { installOPFSMock } from '../helpers/opfs-mock';
|
import { resetOPFSMock } from '../helpers/storage-harness';
|
||||||
|
|
||||||
beforeEach(() => { installOPFSMock(new Map()); });
|
beforeEach(() => { resetOPFSMock(); });
|
||||||
|
|
||||||
describe('AriaEngine — MVCC 事务 + Savepoint', () => {
|
describe('AriaEngine — MVCC 事务 + Savepoint', () => {
|
||||||
let engine: AriaEngine;
|
let engine: AriaEngine;
|
||||||
|
|||||||
@@ -10,82 +10,28 @@
|
|||||||
* 6. open 清理崩溃残留临时文件(.crswap/.tmp)
|
* 6. open 清理崩溃残留临时文件(.crswap/.tmp)
|
||||||
* 7. writeMany/deleteMany 语义
|
* 7. writeMany/deleteMany 语义
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
// ===================================================================
|
||||||
|
// v0.8.0: 删除本文件内的第三份 OPFS mock —— 改用共享的 storage-harness。
|
||||||
|
//
|
||||||
|
// 原实现的问题(见 v0.8.0 迭代工作流 C-1):
|
||||||
|
// - 与 tests/helpers/opfs-mock.ts 重复(两份都在测同一件事,语义各自漂移);
|
||||||
|
// - `close()` 是空函数、写入立即可见 → 无法表达"提交前崩溃";
|
||||||
|
// - 定义了 `writeCalls` 记录但**从未被任何断言使用**(死代码);
|
||||||
|
// - `entry.content.subarray ? entry.content : entry.content` 是恒等表达式(无意义分支)。
|
||||||
|
//
|
||||||
|
// 共享 harness 提供:真实提交语义(close 才可见)、读返回副本、
|
||||||
|
// keepExistingData:false 截断、字节级故障注入、真崩溃模拟。
|
||||||
|
// ===================================================================
|
||||||
import { OPFSBackend } from '../../src/engine/aria/store/opfs_backend';
|
import { OPFSBackend } from '../../src/engine/aria/store/opfs_backend';
|
||||||
|
import { resetOPFSMock } from '../helpers/storage-harness';
|
||||||
// ===================================================================
|
import { decode as decodeBytes } from '../helpers/assertions';
|
||||||
// 真实语义 OPFS mock:记录文件内容、支持 keepExistingData+position
|
|
||||||
// ===================================================================
|
|
||||||
interface MockFile {
|
|
||||||
content: ArrayBuffer;
|
|
||||||
writeCalls: { position?: number; data: ArrayBuffer; keepExistingData?: boolean }[];
|
|
||||||
}
|
|
||||||
|
|
||||||
function createOPFSMock() {
|
|
||||||
const files = new Map<string, MockFile>();
|
|
||||||
|
|
||||||
const getFileHandle = async (name: string, opts?: { create?: boolean }) => {
|
|
||||||
if (!files.has(name)) {
|
|
||||||
if (!opts?.create) throw new Error(`NotFoundError: ${name}`);
|
|
||||||
files.set(name, { content: new ArrayBuffer(0), writeCalls: [] });
|
|
||||||
}
|
|
||||||
const entry = files.get(name)!;
|
|
||||||
return {
|
|
||||||
getFile: async () => ({ size: entry.content.byteLength, arrayBuffer: async () => entry.content }),
|
|
||||||
createWritable: async (wOpts?: { keepExistingData?: boolean }) => {
|
|
||||||
const w: {
|
|
||||||
write: (arg: ArrayBuffer | { type: string; position: number; data: ArrayBuffer }) => Promise<void>;
|
|
||||||
close: () => Promise<void>;
|
|
||||||
} = {
|
|
||||||
write: async (arg) => {
|
|
||||||
const keepExisting = wOpts?.keepExistingData ?? false;
|
|
||||||
const isChunk = typeof arg !== 'object' || !('type' in (arg as object)) || (arg as { type?: string }).type === undefined
|
|
||||||
? { data: arg as ArrayBuffer, position: keepExisting ? entry.content.byteLength : 0 }
|
|
||||||
: { data: (arg as { data: ArrayBuffer }).data, position: (arg as { position: number }).position };
|
|
||||||
entry.writeCalls.push({ position: isChunk.position, data: isChunk.data, keepExistingData: keepExisting });
|
|
||||||
const merged = new Uint8Array(isChunk.position + isChunk.data.byteLength);
|
|
||||||
if (keepExisting || isChunk.position > 0) {
|
|
||||||
merged.set(new Uint8Array(entry.content.subarray ? entry.content : entry.content), 0);
|
|
||||||
}
|
|
||||||
merged.set(new Uint8Array(isChunk.data), isChunk.position);
|
|
||||||
entry.content = merged.buffer;
|
|
||||||
},
|
|
||||||
close: async () => { /* no-op */ },
|
|
||||||
};
|
|
||||||
return w;
|
|
||||||
},
|
|
||||||
};
|
|
||||||
};
|
|
||||||
|
|
||||||
const dir = {
|
|
||||||
getFileHandle,
|
|
||||||
entries: async function* () {
|
|
||||||
for (const [name] of files) yield [name];
|
|
||||||
},
|
|
||||||
removeEntry: async (name: string) => {
|
|
||||||
files.delete(name);
|
|
||||||
},
|
|
||||||
};
|
|
||||||
|
|
||||||
Object.defineProperty(globalThis, 'navigator', {
|
|
||||||
value: {
|
|
||||||
storage: {
|
|
||||||
getDirectory: async () => ({
|
|
||||||
getDirectoryHandle: async (_name: string, _opts?: unknown) => dir,
|
|
||||||
}),
|
|
||||||
},
|
|
||||||
},
|
|
||||||
configurable: true,
|
|
||||||
writable: true,
|
|
||||||
});
|
|
||||||
|
|
||||||
return { files, dir };
|
|
||||||
}
|
|
||||||
|
|
||||||
const enc = (s: string) => new Uint8Array(new TextEncoder().encode(s)).buffer;
|
const enc = (s: string) => new Uint8Array(new TextEncoder().encode(s)).buffer;
|
||||||
|
|
||||||
describe('AriaEngine — OPFSBackend v2', () => {
|
describe('AriaEngine — OPFSBackend v2', () => {
|
||||||
beforeEach(() => {
|
beforeEach(() => {
|
||||||
createOPFSMock();
|
resetOPFSMock();
|
||||||
});
|
});
|
||||||
|
|
||||||
it('read/write/exists/delete 基本语义', async () => {
|
it('read/write/exists/delete 基本语义', async () => {
|
||||||
@@ -95,7 +41,7 @@ describe('AriaEngine — OPFSBackend v2', () => {
|
|||||||
|
|
||||||
await backend.write('k1', enc('hello'));
|
await backend.write('k1', enc('hello'));
|
||||||
expect(await backend.exists('k1')).toBe(true);
|
expect(await backend.exists('k1')).toBe(true);
|
||||||
expect(new TextDecoder().decode(await backend.read('k1'))).toBe('hello');
|
expect(decodeBytes(await backend.read('k1'))).toBe('hello');
|
||||||
expect(await backend.read('missing')).toBeNull();
|
expect(await backend.read('missing')).toBeNull();
|
||||||
|
|
||||||
await backend.delete('k1');
|
await backend.delete('k1');
|
||||||
@@ -111,7 +57,7 @@ describe('AriaEngine — OPFSBackend v2', () => {
|
|||||||
await backend.append('wal', enc('BBB'));
|
await backend.append('wal', enc('BBB'));
|
||||||
await backend.append('wal', enc('CCC'));
|
await backend.append('wal', enc('CCC'));
|
||||||
const all = await backend.read('wal');
|
const all = await backend.read('wal');
|
||||||
expect(new TextDecoder().decode(all)).toBe('AAABBBCCC');
|
expect(decodeBytes(all)).toBe('AAABBBCCC');
|
||||||
await backend.close();
|
await backend.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -120,7 +66,7 @@ describe('AriaEngine — OPFSBackend v2', () => {
|
|||||||
await backend.open('opfs-test-3');
|
await backend.open('opfs-test-3');
|
||||||
await backend.write('f', enc('OLD-CONTENT'));
|
await backend.write('f', enc('OLD-CONTENT'));
|
||||||
await backend.write('f', enc('NEW'));
|
await backend.write('f', enc('NEW'));
|
||||||
expect(new TextDecoder().decode(await backend.read('f'))).toBe('NEW');
|
expect(decodeBytes(await backend.read('f'))).toBe('NEW');
|
||||||
await backend.close();
|
await backend.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -132,7 +78,7 @@ describe('AriaEngine — OPFSBackend v2', () => {
|
|||||||
const p2 = backend.append('log', enc('B'));
|
const p2 = backend.append('log', enc('B'));
|
||||||
const p3 = backend.append('log', enc('C'));
|
const p3 = backend.append('log', enc('C'));
|
||||||
await Promise.all([p1, p2, p3]);
|
await Promise.all([p1, p2, p3]);
|
||||||
expect(new TextDecoder().decode(await backend.read('log'))).toBe('ABC');
|
expect(decodeBytes(await backend.read('log'))).toBe('ABC');
|
||||||
await backend.close();
|
await backend.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -157,7 +103,7 @@ describe('AriaEngine — OPFSBackend v2', () => {
|
|||||||
await expect(origWrite('boom', enc('x'))).rejects.toThrow('Injected write failure');
|
await expect(origWrite('boom', enc('x'))).rejects.toThrow('Injected write failure');
|
||||||
// 队列链恢复:后续写成功
|
// 队列链恢复:后续写成功
|
||||||
await backend.write('ok', enc('fine'));
|
await backend.write('ok', enc('fine'));
|
||||||
expect(new TextDecoder().decode(await backend.read('ok'))).toBe('fine');
|
expect(decodeBytes(await backend.read('ok'))).toBe('fine');
|
||||||
await backend.close();
|
await backend.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -189,7 +135,7 @@ describe('AriaEngine — OPFSBackend v2', () => {
|
|||||||
expect(keys).toEqual(['data']);
|
expect(keys).toEqual(['data']);
|
||||||
expect(keys.some((k) => k.endsWith('.crswap') || k.endsWith('.tmp'))).toBe(false);
|
expect(keys.some((k) => k.endsWith('.crswap') || k.endsWith('.tmp'))).toBe(false);
|
||||||
// 正常数据不受影响
|
// 正常数据不受影响
|
||||||
expect(new TextDecoder().decode(await backend2.read('data'))).toBe('real');
|
expect(decodeBytes(await backend2.read('data'))).toBe('real');
|
||||||
await backend2.close();
|
await backend2.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -197,8 +143,8 @@ describe('AriaEngine — OPFSBackend v2', () => {
|
|||||||
const backend = new OPFSBackend();
|
const backend = new OPFSBackend();
|
||||||
await backend.open('opfs-test-8');
|
await backend.open('opfs-test-8');
|
||||||
await backend.writeMany({ a: enc('AAA'), b: enc('BBB') });
|
await backend.writeMany({ a: enc('AAA'), b: enc('BBB') });
|
||||||
expect(new TextDecoder().decode(await backend.read('a'))).toBe('AAA');
|
expect(decodeBytes(await backend.read('a'))).toBe('AAA');
|
||||||
expect(new TextDecoder().decode(await backend.read('b'))).toBe('BBB');
|
expect(decodeBytes(await backend.read('b'))).toBe('BBB');
|
||||||
await backend.deleteMany(['a']);
|
await backend.deleteMany(['a']);
|
||||||
expect(await backend.exists('a')).toBe(false);
|
expect(await backend.exists('a')).toBe(false);
|
||||||
expect(await backend.exists('b')).toBe(true);
|
expect(await backend.exists('b')).toBe(true);
|
||||||
|
|||||||
@@ -0,0 +1,97 @@
|
|||||||
|
/**
|
||||||
|
* OPFS 不可用时的错误必须"可操作"
|
||||||
|
*
|
||||||
|
* 现场复现(v0.8.0 回归 review 发现):用 file:// 直接打开站点演示页 → 切到 Aria
|
||||||
|
* 引擎 → `navigator.storage.getDirectory()` 抛 SecurityError(Chromium 认为本地
|
||||||
|
* 文件不适合 Web 应用访问)→ 引擎把它包成 `ARIA_OPEN_ERROR`,**cause 没往上带**,
|
||||||
|
* 用户只看到 "Failed to open AriaEngine database" —— 既不知道是环境问题,
|
||||||
|
* 也不知道该怎么办(实测:`isSecureContext` 仍是 true、API 也**存在**,
|
||||||
|
* 只有真正调用才会发现;所以不能靠"API 是否存在"提前判断)。
|
||||||
|
*/
|
||||||
|
import { describe, it, expect, afterEach } from '@jest/globals';
|
||||||
|
import { OPFSBackend } from '../../src/engine/aria/store/opfs_backend';
|
||||||
|
|
||||||
|
const realNavigator = globalThis.navigator;
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
Object.defineProperty(globalThis, 'navigator', { value: realNavigator, configurable: true });
|
||||||
|
});
|
||||||
|
|
||||||
|
function stubStorage(storage: unknown): void {
|
||||||
|
Object.defineProperty(globalThis, 'navigator', {
|
||||||
|
value: { ...(realNavigator as object), storage },
|
||||||
|
configurable: true,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('OPFSBackend.open — 环境不可用时给可操作的错误', () => {
|
||||||
|
it('navigator.storage 缺失(老浏览器 / 非安全上下文)→ ARIA_OPFS_UNAVAILABLE', async () => {
|
||||||
|
stubStorage(undefined);
|
||||||
|
const backend = new OPFSBackend();
|
||||||
|
await expect(backend.open('demo')).rejects.toMatchObject({ code: 'ARIA_OPFS_UNAVAILABLE' });
|
||||||
|
try {
|
||||||
|
await backend.open('demo');
|
||||||
|
} catch (error) {
|
||||||
|
const message = (error as Error).message;
|
||||||
|
expect(message).toContain('navigator.storage.getDirectory is missing');
|
||||||
|
expect(message).toContain('http(s)'); // 必须告诉用户怎么修
|
||||||
|
expect(message).toContain('memory');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it('getDirectory 抛 SecurityError(file:// 页面)→ 带原始 cause + 可操作建议', async () => {
|
||||||
|
const security = new Error('It was determined that certain files are unsafe for access within a Web application');
|
||||||
|
security.name = 'SecurityError';
|
||||||
|
stubStorage({ getDirectory: async () => { throw security; } });
|
||||||
|
|
||||||
|
const backend = new OPFSBackend();
|
||||||
|
const error = await backend.open('demo').then(() => null, (e: unknown) => e) as
|
||||||
|
(Error & { code?: string; cause?: unknown }) | null;
|
||||||
|
expect(error).not.toBeNull();
|
||||||
|
expect(error!.code).toBe('ARIA_OPFS_UNAVAILABLE');
|
||||||
|
expect(error!.cause).toBe(security); // 原始错误必须保留(cause 非枚举,直接断言)
|
||||||
|
expect(error!.message).toContain('SecurityError');
|
||||||
|
expect(error!.message).toContain('file://'); // 点明最可能的成因
|
||||||
|
expect(error!.message).toContain('http(s)');
|
||||||
|
expect(error!.message).toContain('memory');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('正常环境照旧可用(正控:不能把好环境也拒了)', async () => {
|
||||||
|
const files = new Map<string, unknown>();
|
||||||
|
const dirHandle = {
|
||||||
|
async getFileHandle(name: string) {
|
||||||
|
return { async createWritable() { return { async write() {}, async close() { files.set(name, 1); } }; } };
|
||||||
|
},
|
||||||
|
async getDirectoryHandle() { return dirHandle; },
|
||||||
|
async removeEntry(name: string) { files.delete(name); },
|
||||||
|
async *entries() { /* 无残留文件 */ },
|
||||||
|
};
|
||||||
|
stubStorage({ getDirectory: async () => dirHandle });
|
||||||
|
const backend = new OPFSBackend();
|
||||||
|
await backend.open('demo');
|
||||||
|
expect(backend.isOpen()).toBe(true);
|
||||||
|
await backend.close();
|
||||||
|
expect(backend.isOpen()).toBe(false);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('DatabaseError — 根因必须可通过标准 cause 取到', () => {
|
||||||
|
it('第三个参数同时进 details 与 Error.cause(两者语义都成立)', async () => {
|
||||||
|
const { DatabaseError } = await import('../../src/constants');
|
||||||
|
const root = new Error('underlying failure');
|
||||||
|
const err = new DatabaseError('wrapper message', 'WRAP_CODE', root);
|
||||||
|
expect(err.code).toBe('WRAP_CODE');
|
||||||
|
expect(err.details).toBe(root); // 既有契约(文档示例打印 err.details)
|
||||||
|
expect(err.cause).toBe(root); // 标准错误链(v0.8.0 修复前恒为 undefined)
|
||||||
|
expect(err.message).toBe('wrapper message');
|
||||||
|
expect(err.name).toBe('DatabaseError');
|
||||||
|
expect(err instanceof Error).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('没有底层错误时不制造 cause(保持 undefined)', async () => {
|
||||||
|
const { DatabaseError } = await import('../../src/constants');
|
||||||
|
const err = new DatabaseError('plain', 'PLAIN_CODE');
|
||||||
|
expect(err.details).toBeUndefined();
|
||||||
|
expect(err.cause).toBeUndefined();
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -20,7 +20,8 @@ import { PAGE_SIZE } from '../../src/engine/aria/types';
|
|||||||
// ===================================================================
|
// ===================================================================
|
||||||
// OPFS mock(共享工具)
|
// OPFS mock(共享工具)
|
||||||
// ===================================================================
|
// ===================================================================
|
||||||
import { installOPFSMock } from '../helpers/opfs-mock';
|
import { resetOPFSMock } from '../helpers/storage-harness';
|
||||||
|
import { decode as decodeBytes } from '../helpers/assertions';
|
||||||
|
|
||||||
const SCHEMA = () => createSchema('users', {
|
const SCHEMA = () => createSchema('users', {
|
||||||
id: { type: 'string', primaryKey: true },
|
id: { type: 'string', primaryKey: true },
|
||||||
@@ -47,7 +48,7 @@ describe('AriaEngine — PageSSTableStore 单元', () => {
|
|||||||
|
|
||||||
const back = await store.load(1, pageIds, data.byteLength);
|
const back = await store.load(1, pageIds, data.byteLength);
|
||||||
expect(back).not.toBeNull();
|
expect(back).not.toBeNull();
|
||||||
expect(new TextDecoder().decode(back)).toBe('hello page store');
|
expect(decodeBytes(back)).toBe('hello page store');
|
||||||
|
|
||||||
// 页面已落盘(backend 有 pg_ 键)
|
// 页面已落盘(backend 有 pg_ 键)
|
||||||
expect(await backend.exists('pg_1')).toBe(true);
|
expect(await backend.exists('pg_1')).toBe(true);
|
||||||
@@ -130,7 +131,7 @@ describe('AriaEngine — PageSSTableStore 单元', () => {
|
|||||||
// ===================================================================
|
// ===================================================================
|
||||||
describe('AriaEngine — 页面化 SSTable 集成(OPFS)', () => {
|
describe('AriaEngine — 页面化 SSTable 集成(OPFS)', () => {
|
||||||
it('写入 → flush → close → reopen 数据完整(页面模式默认启用)', async () => {
|
it('写入 → flush → close → reopen 数据完整(页面模式默认启用)', async () => {
|
||||||
const files = installOPFSMock(new Map());
|
resetOPFSMock();
|
||||||
const engine = new AriaEngine({
|
const engine = new AriaEngine({
|
||||||
storageBackend: 'opfs',
|
storageBackend: 'opfs',
|
||||||
memtableSizeThreshold: 64 * 1024 * 1024,
|
memtableSizeThreshold: 64 * 1024 * 1024,
|
||||||
@@ -173,7 +174,7 @@ describe('AriaEngine — 页面化 SSTable 集成(OPFS)', () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
it('多级 compaction 后页面化数据仍完整', async () => {
|
it('多级 compaction 后页面化数据仍完整', async () => {
|
||||||
const files = installOPFSMock(new Map());
|
resetOPFSMock();
|
||||||
const engine = new AriaEngine({
|
const engine = new AriaEngine({
|
||||||
storageBackend: 'opfs',
|
storageBackend: 'opfs',
|
||||||
memtableSizeThreshold: 16 * 1024,
|
memtableSizeThreshold: 16 * 1024,
|
||||||
@@ -204,7 +205,7 @@ describe('AriaEngine — 页面化 SSTable 集成(OPFS)', () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
it('页面损坏(篡改 pg_ 文件)→ 打开自愈清理,其余数据可读', async () => {
|
it('页面损坏(篡改 pg_ 文件)→ 打开自愈清理,其余数据可读', async () => {
|
||||||
const files = installOPFSMock(new Map());
|
resetOPFSMock();
|
||||||
const engine = new AriaEngine({
|
const engine = new AriaEngine({
|
||||||
storageBackend: 'opfs',
|
storageBackend: 'opfs',
|
||||||
memtableSizeThreshold: 64 * 1024 * 1024,
|
memtableSizeThreshold: 64 * 1024 * 1024,
|
||||||
@@ -245,7 +246,7 @@ describe('AriaEngine — 页面化 SSTable 集成(OPFS)', () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
it('页面模式与非页面模式混合兼容(pageIds 缺失 → 整 value 读取)', async () => {
|
it('页面模式与非页面模式混合兼容(pageIds 缺失 → 整 value 读取)', async () => {
|
||||||
const { files } = installOPFSMock(new Map());
|
const opfs = resetOPFSMock();
|
||||||
// 阶段 1:非页面模式写入(pageStorage: false → 整 value SSTable)
|
// 阶段 1:非页面模式写入(pageStorage: false → 整 value SSTable)
|
||||||
const engine = new AriaEngine({
|
const engine = new AriaEngine({
|
||||||
storageBackend: 'opfs',
|
storageBackend: 'opfs',
|
||||||
@@ -259,7 +260,9 @@ describe('AriaEngine — 页面化 SSTable 集成(OPFS)', () => {
|
|||||||
await (engine as any).lsm.flush();
|
await (engine as any).lsm.flush();
|
||||||
await engine.close();
|
await engine.close();
|
||||||
|
|
||||||
const backendKeys1 = Array.from(files.keys());
|
// v0.8.0:mock 现在是"真实目录语义"(文件在 `${dbName}/` 下),
|
||||||
|
// 因此要用与生产代码相同的目录视图查询,而不是根目录列表。
|
||||||
|
const backendKeys1 = await opfs.dir('page-db-4').listKeys();
|
||||||
expect(backendKeys1.some((k) => k.startsWith('sst_'))).toBe(true);
|
expect(backendKeys1.some((k) => k.startsWith('sst_'))).toBe(true);
|
||||||
|
|
||||||
// 阶段 2:页面模式打开(默认 opfs → 启用),读旧数据 + 写新数据
|
// 阶段 2:页面模式打开(默认 opfs → 启用),读旧数据 + 写新数据
|
||||||
@@ -285,7 +288,7 @@ describe('AriaEngine — 页面化 SSTable 集成(OPFS)', () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
it('页面化 + 加密 + 压缩组合:往返完整', async () => {
|
it('页面化 + 加密 + 压缩组合:往返完整', async () => {
|
||||||
const files = installOPFSMock(new Map());
|
resetOPFSMock();
|
||||||
const engine = new AriaEngine({
|
const engine = new AriaEngine({
|
||||||
storageBackend: 'opfs',
|
storageBackend: 'opfs',
|
||||||
compression: true,
|
compression: true,
|
||||||
|
|||||||
@@ -14,7 +14,7 @@
|
|||||||
import { AriaEngine } from '../../src/engine/aria/index';
|
import { AriaEngine } from '../../src/engine/aria/index';
|
||||||
import { createSchema } from '../../src/table/schema';
|
import { createSchema } from '../../src/table/schema';
|
||||||
import { SharedMemoryBackend } from '../../src/engine/kvstore/shared_memory_medium';
|
import { SharedMemoryBackend } from '../../src/engine/kvstore/shared_memory_medium';
|
||||||
import { installOPFSMock } from '../helpers/opfs-mock';
|
import { resetOPFSMock } from '../helpers/storage-harness';
|
||||||
|
|
||||||
let counter = 0;
|
let counter = 0;
|
||||||
function uniqueDB(): string {
|
function uniqueDB(): string {
|
||||||
@@ -30,7 +30,7 @@ const SCHEMA = () => createSchema('big', {
|
|||||||
|
|
||||||
beforeEach(() => {
|
beforeEach(() => {
|
||||||
SharedMemoryBackend.clearRegistry();
|
SharedMemoryBackend.clearRegistry();
|
||||||
installOPFSMock(new Map());
|
resetOPFSMock();
|
||||||
});
|
});
|
||||||
|
|
||||||
describe('AriaEngine — 生产负载验证', () => {
|
describe('AriaEngine — 生产负载验证', () => {
|
||||||
@@ -317,6 +317,7 @@ describe('AriaEngine — 生产负载验证', () => {
|
|||||||
// 修复后本机 ~12.5s。CI(debian runner 慢 2~3 倍、重型套件串行)下
|
// 修复后本机 ~12.5s。CI(debian runner 慢 2~3 倍、重型套件串行)下
|
||||||
// 健康耗时约 30~80s;护栏放宽到 240s —— 仍能拦截性能悬崖回归(353s >> 240s),
|
// 健康耗时约 30~80s;护栏放宽到 240s —— 仍能拦截性能悬崖回归(353s >> 240s),
|
||||||
// 不误报健康慢环境。
|
// 不误报健康慢环境。
|
||||||
|
// eslint-disable-next-line no-console -- 性能护栏需要输出实测耗时
|
||||||
console.log(`10万行 kv 插入耗时: ${insertMs}ms`);
|
console.log(`10万行 kv 插入耗时: ${insertMs}ms`);
|
||||||
expect(insertMs).toBeLessThan(240000);
|
expect(insertMs).toBeLessThan(240000);
|
||||||
expect(await engine.count('big')).toBe(TOTAL);
|
expect(await engine.count('big')).toBe(TOTAL);
|
||||||
@@ -364,6 +365,7 @@ describe('AriaEngine — 生产负载验证', () => {
|
|||||||
}
|
}
|
||||||
const insertMs = Date.now() - t0;
|
const insertMs = Date.now() - t0;
|
||||||
// 同上:CI 慢环境护栏放宽(本机 ~25s;悬崖回归仍会被拦截)
|
// 同上:CI 慢环境护栏放宽(本机 ~25s;悬崖回归仍会被拦截)
|
||||||
|
// eslint-disable-next-line no-console -- 性能护栏需要输出实测耗时
|
||||||
console.log(`10万行 opfs 插入耗时: ${insertMs}ms`);
|
console.log(`10万行 opfs 插入耗时: ${insertMs}ms`);
|
||||||
expect(insertMs).toBeLessThan(300000);
|
expect(insertMs).toBeLessThan(300000);
|
||||||
expect(await engine.count('big')).toBe(TOTAL);
|
expect(await engine.count('big')).toBe(TOTAL);
|
||||||
|
|||||||
@@ -11,9 +11,9 @@
|
|||||||
*/
|
*/
|
||||||
import { AriaEngine } from '../../src/engine/aria/index';
|
import { AriaEngine } from '../../src/engine/aria/index';
|
||||||
import { createSchema } from '../../src/table/schema';
|
import { createSchema } from '../../src/table/schema';
|
||||||
import { installOPFSMock } from '../helpers/opfs-mock';
|
import { resetOPFSMock } from '../helpers/storage-harness';
|
||||||
|
|
||||||
beforeEach(() => { installOPFSMock(new Map()); });
|
beforeEach(() => { resetOPFSMock(); });
|
||||||
|
|
||||||
let idbCounter = 0;
|
let idbCounter = 0;
|
||||||
function uniqueDB(): string {
|
function uniqueDB(): string {
|
||||||
@@ -65,20 +65,23 @@ describe('AriaEngine — repair 自愈增强', () => {
|
|||||||
|
|
||||||
it('清理 OPFS 残留临时文件(.crswap/.tmp)', async () => {
|
it('清理 OPFS 残留临时文件(.crswap/.tmp)', async () => {
|
||||||
// 用 OPFS mock 后端验证 cleanupStaleFiles 被调用
|
// 用 OPFS mock 后端验证 cleanupStaleFiles 被调用
|
||||||
const { files, dir } = installOPFSMock(new Map());
|
const opfs = resetOPFSMock();
|
||||||
const engine = new AriaEngine({ storageBackend: 'opfs', checkpointInterval: 100000 });
|
const engine = new AriaEngine({ storageBackend: 'opfs', checkpointInterval: 100000 });
|
||||||
await engine.open('repair-opfs-1', 1);
|
await engine.open('repair-opfs-1', 1);
|
||||||
await engine.createTable(SCHEMA());
|
await engine.createTable(SCHEMA());
|
||||||
await engine.insert('items', [{ id: 'a', val: 1, tag: 'x' }]);
|
await engine.insert('items', [{ id: 'a', val: 1, tag: 'x' }]);
|
||||||
|
|
||||||
// 制造残留
|
// 制造残留(模拟 createWritable 中断留下的临时文件)。
|
||||||
await dir.getFileHandle('junk.crswap', { create: true });
|
// v0.8.0:mock 现在是真实目录语义,残留必须落在**该库的目录**里 ——
|
||||||
await dir.getFileHandle('junk2.tmp', { create: true });
|
// repair 的 cleanupStaleFiles 扫的是库目录(root 上的文件不属于任何库)。
|
||||||
expect(Array.from(files.keys()).some((k) => k.endsWith('.crswap'))).toBe(true);
|
const dir = opfs.dir('repair-opfs-1');
|
||||||
|
await dir.createFile('junk.crswap');
|
||||||
|
await dir.createFile('junk2.tmp');
|
||||||
|
expect((await dir.listKeys()).some((k) => k.endsWith('.crswap'))).toBe(true);
|
||||||
|
|
||||||
await (engine as any).repair();
|
await (engine as any).repair();
|
||||||
|
|
||||||
const after = Array.from(files.keys());
|
const after = await dir.listKeys();
|
||||||
expect(after.some((k) => k.endsWith('.crswap'))).toBe(false);
|
expect(after.some((k) => k.endsWith('.crswap'))).toBe(false);
|
||||||
expect(after.some((k) => k.endsWith('.tmp'))).toBe(false);
|
expect(after.some((k) => k.endsWith('.tmp'))).toBe(false);
|
||||||
expect(await engine.count('items')).toBe(1);
|
expect(await engine.count('items')).toBe(1);
|
||||||
@@ -193,7 +196,7 @@ describe('AriaEngine — 随机操作压力 + 模拟崩溃', () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
it('随机操作 + 页面化 + 模拟崩溃 → 重开验证', async () => {
|
it('随机操作 + 页面化 + 模拟崩溃 → 重开验证', async () => {
|
||||||
const files = installOPFSMock(new Map());
|
resetOPFSMock();
|
||||||
const dbName = `rand-opfs-${Date.now()}-${Math.random().toString(36).slice(2, 6)}`;
|
const dbName = `rand-opfs-${Date.now()}-${Math.random().toString(36).slice(2, 6)}`;
|
||||||
const engine = new AriaEngine({
|
const engine = new AriaEngine({
|
||||||
storageBackend: 'opfs',
|
storageBackend: 'opfs',
|
||||||
|
|||||||
@@ -14,9 +14,9 @@ import { MemoryBackend } from '../../src/engine/aria/store/backend';
|
|||||||
import { AriaEngine } from '../../src/engine/aria/index';
|
import { AriaEngine } from '../../src/engine/aria/index';
|
||||||
import { createSchema } from '../../src/table/schema';
|
import { createSchema } from '../../src/table/schema';
|
||||||
|
|
||||||
import { installOPFSMock } from '../helpers/opfs-mock';
|
import { resetOPFSMock } from '../helpers/storage-harness';
|
||||||
|
|
||||||
beforeEach(() => { installOPFSMock(new Map()); });
|
beforeEach(() => { resetOPFSMock(); });
|
||||||
|
|
||||||
let idbCounter = 0;
|
let idbCounter = 0;
|
||||||
function uniqueDB(): string {
|
function uniqueDB(): string {
|
||||||
@@ -25,6 +25,15 @@ function uniqueDB(): string {
|
|||||||
|
|
||||||
const enc = (s: string) => new Uint8Array(new TextEncoder().encode(s));
|
const enc = (s: string) => new Uint8Array(new TextEncoder().encode(s));
|
||||||
|
|
||||||
|
/** 构造"首 4 字节为大端 LSN"的记录(分片的水位判定依赖它) */
|
||||||
|
function rec(lsn: number, pad = ''): Uint8Array {
|
||||||
|
const filler = enc(pad.padEnd(4, '.'));
|
||||||
|
const out = new Uint8Array(4 + filler.byteLength);
|
||||||
|
new DataView(out.buffer).setUint32(0, lsn, false);
|
||||||
|
out.set(filler, 4);
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
describe('AriaEngine — SegmentedWALStore 单元', () => {
|
describe('AriaEngine — SegmentedWALStore 单元', () => {
|
||||||
it('append → readAll 往返,多批次字节顺序一致', async () => {
|
it('append → readAll 往返,多批次字节顺序一致', async () => {
|
||||||
const backend = new MemoryBackend();
|
const backend = new MemoryBackend();
|
||||||
@@ -62,7 +71,7 @@ describe('AriaEngine — SegmentedWALStore 单元', () => {
|
|||||||
await backend.close();
|
await backend.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
it('空洞检测:分片序号不连续 → 空洞后的分片整体丢弃', async () => {
|
it('尾部分片缺失(无法与更后分片比较)→ 已读到的分片仍然返回', async () => {
|
||||||
const backend = new MemoryBackend();
|
const backend = new MemoryBackend();
|
||||||
await backend.open('seg-unit-3');
|
await backend.open('seg-unit-3');
|
||||||
const store = new SegmentedWALStore(backend, 16);
|
const store = new SegmentedWALStore(backend, 16);
|
||||||
@@ -75,21 +84,120 @@ describe('AriaEngine — SegmentedWALStore 单元', () => {
|
|||||||
await backend.delete('__wal_000001.bin');
|
await backend.delete('__wal_000001.bin');
|
||||||
|
|
||||||
const all = await store.readAll();
|
const all = await store.readAll();
|
||||||
// 空洞在分片 1 → 1 及之后丢弃,只保留分片 0
|
// 介质上只剩分片 0:没有更后的分片作为参照,缺失的"尾部"无法被识别成空洞
|
||||||
|
//(这是介质信息本身的限制 —— 尾部丢失只能靠 manifest.nextLsn 之外的证据发现)
|
||||||
expect(new TextDecoder().decode(all)).toBe('SEG0-SEG0-S');
|
expect(new TextDecoder().decode(all)).toBe('SEG0-SEG0-S');
|
||||||
await backend.close();
|
await backend.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
it('空洞在开头(分片 0 缺失)→ 全部丢弃(保守截断)', async () => {
|
it('前缀缺失:水位未推进(fromLsn=0)→ 全部丢弃;水位已推进 → 后缀照常读取', async () => {
|
||||||
const backend = new MemoryBackend();
|
const backend = new MemoryBackend();
|
||||||
await backend.open('seg-unit-4');
|
await backend.open('seg-unit-4');
|
||||||
const store = new SegmentedWALStore(backend, 16);
|
const store = new SegmentedWALStore(backend, 3); // 2 字节记录 → 每条一个分片
|
||||||
await store.append(enc('X0'));
|
await store.append(enc('X0'));
|
||||||
await store.append(enc('X1'));
|
await store.append(enc('X1'));
|
||||||
await backend.delete('__wal_000000.bin');
|
await backend.delete('__wal_000000.bin');
|
||||||
|
|
||||||
|
// 从未推进过水位:分片本应从 0 连续存在 → 前缀缺失是异常,保守丢弃
|
||||||
const all = await store.readAll();
|
const all = await store.readAll();
|
||||||
expect(all.byteLength).toBe(0);
|
expect(all.byteLength).toBe(0);
|
||||||
|
|
||||||
|
// 水位已推进(例如回退到上一代 manifest,startSegment/startLsn 指向更早的位置):
|
||||||
|
// 前缀那一段本就被水位跳过,**不能**因此丢弃后面的活分片
|
||||||
|
const fromLater = await store.readAllFrom(0, 1);
|
||||||
|
expect(new TextDecoder().decode(fromLater.data)).toBe('X1');
|
||||||
|
expect(fromLater.missingPrefix).toEqual([0]);
|
||||||
|
expect(fromLater.gaps).toEqual([]);
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('内部空洞:空洞之后的分片整体丢弃,并把空洞号上报', async () => {
|
||||||
|
const backend = new MemoryBackend();
|
||||||
|
await backend.open('seg-unit-4b');
|
||||||
|
const store = new SegmentedWALStore(backend, 8); // 8 字节记录 → 每条一个分片
|
||||||
|
await store.append(rec(1, 'aaaa')); // 分片 0
|
||||||
|
await store.append(rec(2, 'bbbb')); // 分片 1
|
||||||
|
await store.append(rec(3, 'cccc')); // 分片 2
|
||||||
|
expect((await store.readAllFrom(0, 0)).segments).toEqual([0, 1, 2]);
|
||||||
|
await backend.delete('__wal_000001.bin'); // 内部空洞
|
||||||
|
|
||||||
|
const result = await store.readAllFrom(0, 0);
|
||||||
|
expect(result.segments).toEqual([0]); // 只读了空洞之前的分片
|
||||||
|
expect(result.gaps).toEqual([1]); // 显式上报(不再静默)
|
||||||
|
expect(result.data.byteLength).toBe(8); // 只保留空洞之前的分片
|
||||||
|
expect(new DataView(result.data.buffer, result.data.byteOffset).getUint32(0, false)).toBe(1);
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('整体清空后分片号绝不回退(可以复用最后用过的号,但绝不能回到 0)', async () => {
|
||||||
|
const backend = new MemoryBackend();
|
||||||
|
await backend.open('seg-unit-4c');
|
||||||
|
const store = new SegmentedWALStore(backend, 8); // 8 字节记录 → 每条一个分片
|
||||||
|
await store.append(rec(1, 'aaaa')); // 分片 0
|
||||||
|
await store.append(rec(2, 'bbbb')); // 分片 1
|
||||||
|
await store.append(rec(3, 'cccc')); // 分片 2
|
||||||
|
expect((await store.readAllFrom(0, 0)).segments).toEqual([0, 1, 2]);
|
||||||
|
await store.truncate();
|
||||||
|
expect((await backend.listKeys()).length).toBe(0); // 介质上确实没有分片了
|
||||||
|
|
||||||
|
await store.append(rec(4, 'dddd'));
|
||||||
|
const walKeys = (await backend.listKeys()).filter((k) => k.startsWith('__wal_'));
|
||||||
|
expect(walKeys).toHaveLength(1);
|
||||||
|
const newSeq = Number(walKeys[0].match(/^__wal_(\d{6,})\.bin$/)![1]);
|
||||||
|
// 关键不变量:不回退到 0(回退 + manifest.startSegment>0 = 新记录被恢复过滤掉)
|
||||||
|
expect(newSeq).toBeGreaterThanOrEqual(2);
|
||||||
|
expect(walKeys).not.toContain('__wal_000000.bin');
|
||||||
|
// 清空后从新分片号读取仍能读回新记录(该契约不能被破坏)
|
||||||
|
const after = await store.readAllFrom(newSeq, 0);
|
||||||
|
expect(after.data.byteLength).toBe(8);
|
||||||
|
expect(new DataView(after.data.buffer, after.data.byteOffset).getUint32(0, false)).toBe(4);
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('写入永不落到 manifest 水位下限之下(startSegment=K → 新分片号 >= K)', async () => {
|
||||||
|
const backend = new MemoryBackend();
|
||||||
|
await backend.open('seg-unit-4e');
|
||||||
|
const store = new SegmentedWALStore(backend, 8);
|
||||||
|
await store.append(rec(1, 'aaaa')); // 分片 0(首条 LSN=1)
|
||||||
|
await store.append(rec(2, 'bbbb')); // 分片 1(首条 LSN=2)
|
||||||
|
await store.append(rec(3, 'cccc')); // 分片 2(首条 LSN=3)
|
||||||
|
|
||||||
|
// 模拟 checkpoint:先算水位下限并**提交进 manifest**,再删除前缀分片
|
||||||
|
const keepFrom = await store.planKeepFrom(3, 3); // 全部记录都已落盘 → 全部可删
|
||||||
|
expect(keepFrom).toBe(3);
|
||||||
|
await store.truncateBefore(3, 3);
|
||||||
|
expect((await backend.listKeys()).filter((k) => k.startsWith('__wal_'))).toEqual([]);
|
||||||
|
|
||||||
|
// 之后的新写入必须落在 >= keepFrom 的分片里(否则重开时被 `seq >= startSegment` 过滤 → 静默丢失)
|
||||||
|
await store.append(rec(4, 'dddd'));
|
||||||
|
const seqs = (await backend.listKeys())
|
||||||
|
.filter((k) => k.startsWith('__wal_'))
|
||||||
|
.map((k) => Number(k.match(/^__wal_(\d{6,})\.bin$/)![1]));
|
||||||
|
expect(seqs.length).toBeGreaterThan(0);
|
||||||
|
expect(Math.min(...seqs)).toBeGreaterThanOrEqual(keepFrom);
|
||||||
|
// 而且从 manifest 的水位下限开始读,这条新记录必须读得到(双向验证)
|
||||||
|
const reread = await store.readAllFrom(keepFrom, 3);
|
||||||
|
expect(reread.data.byteLength).toBe(8);
|
||||||
|
expect(new DataView(reread.data.buffer, reread.data.byteOffset).getUint32(0, false)).toBe(4);
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('活跃区间已被清理(fromSegment > 0,介质上无分片):新写入从 fromSegment 继续', async () => {
|
||||||
|
const backend = new MemoryBackend();
|
||||||
|
await backend.open('seg-unit-4d');
|
||||||
|
const store = new SegmentedWALStore(backend, 8);
|
||||||
|
await store.append(enc('AAAA'));
|
||||||
|
await store.append(enc('BBBB')); // 分片 0、1
|
||||||
|
// 介质上分片被全部清理,manifest 说 startSegment = 5
|
||||||
|
const later = await store.readAllFrom(5, 100);
|
||||||
|
// 一个分片都没有:无法判断缺了哪些号 → 不谎报空洞,也不丢已有数据
|
||||||
|
expect(later.data.byteLength).toBe(0);
|
||||||
|
expect(later.segments).toEqual([]);
|
||||||
|
expect(later.gaps).toEqual([]);
|
||||||
|
|
||||||
|
await store.append(enc('EEEE'));
|
||||||
|
const keys = await backend.listKeys();
|
||||||
|
expect(keys.some((k) => k === '__wal_000005.bin')).toBe(true); // 从 fromSegment 继续
|
||||||
await backend.close();
|
await backend.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user