MetonaTeam/MetonaSqlark
2
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

release: v0.5.1 — 存储后端生产级硬化（CRC-32/全库加密/WAL分片/页面化存储/多标签页锁/e2e）+ 深度审查修复（假实现接线/死代码清理）
CI / test (18.x) (push) Successful in 10m10s
Details
CI / test (20.x) (push) Successful in 10m10s
Details
CI / test (22.x) (push) Successful in 10m6s
Details
CI / e2e (push) Successful in 9m51s
Details
CI / test (24.x) (push) Successful in 10m28s
Details

Browse Source
This commit is contained in:
thzxx
2026-08-10 12:07:00 +08:00
parent cff98b0903
commit 334067d89e
88 changed files with 15713 additions and 10626 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitea/workflows/ci.yml
+22
View File
@@ -40,3 +40,25 @@ jobs:
- name: Build
run: npm run build
e2e:
runs-on: debian-latest
# 与 MetonaEditor 一致：Playwright 官方镜像（自带 Chromium 与全部系统依赖），
# 避免 runner 宿主（Debian 11）不被 Playwright 1.62 支持的问题
container: mcr.microsoft.com/playwright:v1.62.1-noble
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- name: Use Node.js
uses: actions/setup-node@v4
with:
node-version: 22.x
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Build
run: npm run build
- name: Verify Playwright browsers
run: npx playwright install chromium
- name: Run E2E smoke tests
run: npm run test:e2e
.gitignore
+4
View File
@@ -7,3 +7,7 @@ coverage/
# CatPaw / .zcode 工作区文件（本地 AI 会话计划，勿提交）
.zcode/
# Playwright e2e 产物
test-results/
playwright-report/
CHANGELOG.md
+109
View File
@@ -2,6 +2,115 @@
All notable changes to MetonaSqlark will be documented in this file.
## [0.5.1] - 2026-08-10
### 深度审查：消除假实现/半成品/死代码
> 对全部源码做系统性审查，修复"宣称存在但从未接线"的假实现、补齐无入口的半成品、
> 删除无引用的死代码。
### Added（假实现接线）
- **CRUD 生命周期钩子真实接线** — 此前 `beforeInsert`/`afterInsert`/`beforeUpdate`/`afterUpdate`/`beforeDelete`/`afterDelete`
六个钩子仅存在于类型与文档宣称（"14 种钩子"），从未被任何路径触发。现已在两条路径真实触发：
- Table API：`table().insert/insertMany/update().execute()/delete().execute()`
- SQL：`db.query("INSERT/UPDATE/DELETE ...")`
参数与文档一致（beforeInsert: rows[]、afterUpdate: query, updates, count 等）；事务路径不触发（Table 无 hooks 回调，与现状一致）
- **EXPLAIN SQL 入口** — 此前 executor 有 EXPLAIN 分支但 parser 无法解析（无 token），只能手工构造 AST 测试。
补齐 `EXPLAIN <statement>` 语法（tokens/lexer/parser），任意引擎可用
- **ANALYZE / REINDEX / VACUUM SQL 入口** — 此前仅有 AriaEngine 方法（`(engine as any)` 才能调用），README 宣称能力但用户无法触发。
补齐 `ANALYZE [TABLE] name` / `REINDEX [TABLE] name` / `VACUUM` 语法；不支持的引擎抛 `NOT_SUPPORTED`
- **SAVEPOINT SQL 入口** — 补齐 `SAVEPOINT name` / `ROLLBACK TO [SAVEPOINT] name` / `RELEASE [SAVEPOINT] name`；
`ROLLBACK [TRANSACTION]` 保持原语义
- **db.backup() 公共方法** — 引擎级 backup（MVCC 一致性视图）此前无公共入口；Aria 走引擎级，其余引擎回退 exportAll
### Removed（死代码）
- **`src/utils.ts` 整文件** — generateId/escapeHTML/deepMerge/debounce/throttle/isBrowser 在 src 内零引用
（v0.0.1 遗留，仅测试锁定）；连同 tests/utils.test.ts 删除
- **`estimateQueryCost`（未接线的"查询优化器"）** — 实现与测试存在但从未接入查询路径（find 用 tryIndexLookup 硬编码），删除消除"优化器存在但没用"假象
- **MVCC 读侧全套** — readVersion/getLatestCommittedVersions/clearTable/getActiveTxnCount/isActive/getActiveWriteKeys：
引擎事务读取走 txnSnapshot（MVCC 仅作 undo log），读侧实现从未执行。删除并改写相关测试为验证版本链清理/GC 行为
- **统计/辅助死方法** — EvictionManager.getAllPages/getSize/popLRU/getCapacity、BufferPool.getCachedPageCount/getDirtyPageCount/getCapacity/newPage、
WAL.isEnabled/getLSN、CheckpointManager.forceCheckpoint/setInterval/getOpCount、LSM.isInitialized/getCacheSize/getCacheLimit、
SSTableReader.getMeta/getIndexBlockCount/getStoredChecksum、BloomFilter.getBitSize/getInsertedCount、MemTable.contains、
FileManager.allocateTableRootPage/createEmptyPage、OPFSEngine.loadTableIntoMemory、QueryExecutor.setMaxRowsPerQuery、
ConnectionManager.getRefCount
- **类型/常量死代码** — STORAGE_MODES/DISK_ENGINES、ColumnEncoding/DEFAULT_BLOCK_SIZE/DataBlock、CallbackEntrySource、StatementType、AggregateExpression、SubqueryExpression
- **Crypto 全局兼容层** — initCrypto/encryptPage/decryptPage/closeCrypto/isCryptoEnabled（v0.5.0 后引擎走 EncryptedBackend 实例，全局单例无消费者）；删除过时的"全局加密落盘"测试（aria-encryption.test.ts 已覆盖等价路径）
### Changed
- 测试 1022 → **1010**（63 套件；删除 12 个死代码测试 + 新增 22 个 hooks/维护语句测试）；覆盖率 86.68% → 待发布后统计
- README 补维护语句表（EXPLAIN/ANALYZE/REINDEX/VACUUM/SAVEPOINT SQL 入口）与 `db.backup()` API
- 14 个生命周期钩子现在全部真实触发（site/docs.html 与 CONTRIBUTING 宣称变为事实）
---
## [0.5.0] - 2026-08-10
### 里程碑：存储后端生产级硬化（自研 OPFS 后端全面落地）
> 本次版本对 AriaEngine 存储层做全链路生产级改造：修复"宣称有校验和/加密/页面式存储但实际未实现"的失实实现，
> WAL 重构为分片文件（消除多文件原子性与文件爆炸问题）、SSTable 页面化物理存储（BufferPool/FileManager 真正接入）、
> 多标签页独占锁、repair 自愈增强，并引入真实 Chromium e2e 验证。
### Added
- **全库 AES-256-GCM 加密（真实接线）** — `AriaEngineConfig.encryption.password`：
- `EncryptedBackend` 装饰器在 backend 层透明加解密（WAL/SSTable/Schema/元数据全覆盖，每个 value 独立随机 IV + GCM 认证）
- PBKDF2-SHA256（100000 迭代）派生密钥，salt + verifier 持久化于 `__aria_keymeta`（密码错误 → `ARIA_DECRYPT_ERROR`）
- 加密开关一致性检测：明文旧库 + 加密配置 → `ARIA_ENCRYPT_CONFIG_ERROR`；加密库 + 无密码 → `ARIA_ENCRYPT_REQUIRED`
- clearAll 保留密钥元数据（库身份保留，密码不变）
- **WAL 分片存储重构** — `SegmentedWALStore`（`__wal_%06d.bin`）：
- 取代"每条记录一个 key"旧结构：分片阈值 4MB 自动切换，文件数量可控
- WAL 序号（LSN）内嵌记录字节流，移除 `__wal_count` 键 → append 单文件原子写（createWritable COW）
- 空洞检测：分片序号不连续 → 空洞之后保守丢弃（仅可能来自 truncate 部分完成，数据已落盘，无害）
- 旧格式 `__wal_N` + `__wal_count` 兼容：读取迁移重放，checkpoint 统一清空
- **OPFS 后端 v2** —
- `append()` 真追加：`createWritable({ keepExistingData })` + seek，O(chunk) 无需读旧内容
- 写队列健壮性：单次任务失败不中断队列链（错误返回给调用方），close 等待队列排空
- open 自动清理崩溃残留临时文件（`.crswap`/`.tmp`）
- 原子性文档化：单文件 createWritable 为 copy-on-write 原子替换
- **SSTable 页面化物理存储（BufferPool/FileManager 真正接入）** —
- `PageSSTableStore`：SSTable 切分为 4KB 页面，经 FileManager 分配 pageId、BufferPool LRU 缓存
- `save` 语义 = 已持久化（逐页 flushPage 落盘后才返回，WAL checkpoint 截断安全）
- 页面 ID 列表经 `SSTableMeta.pageIds` 持久化；旧 meta（无 pageIds）自动走整 value 读取（兼容）
- OPFS 后端默认启用（`pageStorage` 配置可覆盖）；FileManager 批量分配 `allocatePageIds`（一次 meta 持久化）
- 读取缺失页面返回 null（此前返回空页会静默掩盖页面丢失，自愈路径依赖 null 触发清理）
- **多标签页独占锁** — Web Locks API（Chrome 69+ / Firefox 96+ / Safari 15.4+）：
- `ifAvailable` 模式：第二个标签页打开同一库 → 抛 `ARIA_LOCKED`（不排队挂起）
- 打开失败自动释放锁（不泄漏）；close 等待锁真正归还
- 不支持 Web Locks 的环境降级为无锁 + 告警（如实文档化）
- **repair() 自愈增强** — 孤儿页面清理（meta 未引用的 `pg_` 文件）、OPFS 残留临时文件清理、WAL 空洞截断落地
- **LZ4 格式 v2** — 压缩流前置 4 字节原始大小头：解压不再依赖外部估算（高压缩率数据下 `buf.length*2` 估算不足会截断），损坏头显式抛错
- **SSTable 真实 CRC-32 校验和** — footer checksum 字段从写死 0 升级为整文件标准 CRC32（checksum=0 识别旧版文件跳过校验，兼容旧库）；打开校验/compaction 加载/运行期预加载三处均校验，损坏文件自动清理自愈
- **WAL 标准 CRC32** — 记录校验从弱滚动校验升级为标准 CRC32；双算法探测（新算法失败再试旧算法）兼容旧 WAL 记录
- **配置透传** — `DatabaseConfig.aria`：`MetonaSqlark.create()` 可配置 `walSyncMode`/`checkpointInterval`/`encryption`/`pageStorage`/`compression` 等 AriaEngine 专属参数
- **Playwright e2e（真实 Chromium + OPFS）** — 7 用例：持久化（刷新保留）、关闭重开、崩溃恢复（WAL 重放）、多标签页锁、加密密码往返、页面化端到端 + 索引查询、repair 自愈；`npm run test:e2e`
### Fixed
- **加密此前从未接线（严重失实）** — crypto 模块全局单例从未被引擎调用、salt 不持久化重启无法解密；README 宣称的 AES-GCM 实际不可用。改为 EncryptedBackend 装饰器 + keymeta 持久化 + 密码验证
- **SSTable checksum 写死 0（严重失实）** — 数据腐坏（bit flip）完全无法检测。升级为真实 CRC-32，损坏文件打开自愈
- **OPFS writeMany 无原子性（P0）** — WAL 记录 + count 多键写入非原子，记录空洞导致后续记录字节错位解析失败。分片重构后 WAL append 单文件原子写 + 空洞检测截断根治
- **OPFS 高频写入文件爆炸（P1）** — 每条 WAL 记录一个文件，8 万条记录 = 8 万次 createWritable。分片存储 + 真追加根治
- **LZ4 高压缩率解压截断（P1）** — `buf.length * 2` 估算原始大小，压缩率 >50% 时截断（SSTable 魔数在文件末尾被丢 → 整库数据丢失）。格式 v2 头部自描述根治
- **OPFS 写队列失败永久中断（P1）** — 队列任务抛错后链 rejected，后续所有写挂起。单次失败不中断链
- **OPFS close 不等挂起写（P1）** — close 立即置空目录句柄，挂起写静默丢失。close 等待队列排空
- **Web Locks 方法解构 this 绑定丢失（e2e 发现）** — `navigator.locks.request` 解构后调用抛 Illegal invocation。改用 `.bind(lockManager)` 保持 this
- **页面化 + 崩溃窗口数据丢失（压力测试发现）** — 修复前页面化路径在模拟崩溃测试中丢失已确认写入；根治后 500 随机操作 + 崩溃重放零丢失
### Changed
- `AriaEngineConfig` 新增 `encryption` / `pageStorage`；`DatabaseConfig` 新增 `aria` 透传
- 页面层精简：删除从未接入主路径的 slot/tuple 模块（假"页面式存储"实现），仅保留生产代码实际使用的 `initPageHeader`/`createPage`；架构宣称如实更新（4KB 页面承载 SSTable 字节切片，非行级 Slotted Page）
- LZ4 压缩格式不向后兼容（v0.2.6 已声明 compression 数据需重新导入，本次格式 v2 同理）
- 测试 958 → **1022**（62 套件）；行覆盖率 84.2% → **86.68%**；新增 6 套件：crc32 / aria-checksum / aria-wal-crc / aria-wal-segment / aria-opfs-backend / aria-page-store / aria-encryption / aria-locks / aria-repair-hardening / aria-page-core
- 新增 Playwright e2e 基建（`npm run test:e2e`，7 用例真实 Chromium）
---
## [0.4.4] - 2026-08-09
### Fixed
CONTRIBUTING.md
+3 -1
View File
@@ -73,7 +73,9 @@ src/
├── plugin/ # Plugin system (14 lifecycle hooks)
└── integrations/ # React & Vue hooks
tests/ # Test suite (837+ test cases, 44 test suites)
tests/ # Test suite (1022 test cases, 62 suites + 7 e2e)
tests/helpers/ # 共享测试工具（OPFS mock 等）
tests/e2e/ # Playwright e2e（真实 Chromium + OPFS）
site/ # Documentation site (index / docs / demo)
```
README.md
+83 -50
View File
@@ -1,21 +1,22 @@
# MetonaSqlark
<p align="center">
<img src="https://img.shields.io/badge/version-0.4.4-blue?style=flat-square" alt="version">
<img src="https://img.shields.io/badge/version-0.5.0-blue?style=flat-square" alt="version">
<img src="https://img.shields.io/badge/license-MIT-green?style=flat-square" alt="license">
<img src="https://img.shields.io/badge/coverage-84.2%25-brightgreen?style=flat-square" alt="coverage">
<img src="https://img.shields.io/badge/tests-958%20passed-success?style=flat-square" alt="tests">
<img src="https://img.shields.io/badge/coverage-86.7%25-brightgreen?style=flat-square" alt="coverage">
<img src="https://img.shields.io/badge/tests-1022%20passed-success?style=flat-square" alt="tests">
</p>
> 基于 TypeScript 的**前端关系型数据库**，支持完整 SQL 查询、Query Builder 链式 API、与 **AriaEngine 自研页面式存储引擎**。
> 基于 TypeScript 的**前端关系型数据库**，支持完整 SQL 查询、Query Builder 链式 API、与 **AriaEngine 自研存储引擎**。
---
## ✨ 特性
- 🚀 **AriaEngine 自研存储引擎** — LSM-Tree 页面式存储，4KB Slotted Page、WAL 崩溃恢复（full 模式真正同步）、LZ4 压缩
- 💾 **OPFS 自研存储后端** — 纯浏览器文件系统，零 IndexedDB 依赖，二进制页面文件，schema 持久化（空表/索引跨重启完整保留）
- 🔒 **生产级数据安全** — WAL 原子写入 + CRC 完整性校验、`RESTRICT` 外键约束、崩溃恢复自愈（`repair()` 无需删库重建）、SQL 注入防护
- 🚀 **AriaEngine 自研存储引擎** — LSM-Tree + **SSTable 4KB 页面化物理存储**（BufferPool LRU 缓存 + FileManager 页面管理，v0.5.0 真实接入）、WAL 分片文件（真追加 + 空洞检测）、LZ4 压缩
- 💾 **OPFS 自研存储后端** — 纯浏览器文件系统，零 IndexedDB 依赖，二进制页面文件，schema 持久化，单文件 COW 原子写，崩溃残留自动清理
- 🔒 **生产级数据安全** — WAL 原子写入 + 标准 CRC32 完整性校验（SSTable 整文件校验 + WAL 记录校验）、**全库 AES-256-GCM 透明加密**（PBKDF2 密钥派生 + salt 持久化 + 密码验证，v0.5.0 真实接线）、`RESTRICT` 外键约束、崩溃恢复自愈
- 🔐 **多标签页独占锁** — Web Locks API，第二个标签页打开同一库抛 `ARIA_LOCKED`（v0.5.0）
- 🛡 **输入校验全覆盖** — `maxLength`/`min`/`max` 约束、类型检查、必填验证
- 💾 **多引擎架构** — Memory / IndexedDB / OPFS / Hybrid(write-through) / Aria 五种模式
- 📝 **完整 SQL 支持** — SELECT/JOIN/子查询/GROUP BY/HAVING/ORDER BY/LIMIT/BETWEEN/IF NOT EXISTS/ALTER TABLE/TRUNCATE TABLE/UNION/INSERT...SELECT/事务语句/CREATE INDEX/EXISTS（v0.3.0）+ CASE WHEN/哈希连接/组提交（v0.3.1）+ 多标签页同步（v0.3.2）
@@ -24,13 +25,13 @@
- 🔗 **Query Builder API** — 链式 `.select().where().orderBy().limit().execute()`
- 🔄 **事务回滚** — Memory/IndexedDB/Hybrid/Aria 四引擎事务原子性，自动回滚，MVCC 版本链接入读写路径
- 🔗 **外键级联** — ON DELETE + ON UPDATE（CASCADE / SET NULL / RESTRICT）全引擎支持，支持更新主键（v0.4.2）
- 🛡 **崩溃恢复自愈** — 残缺 SSTable 打开自动跳过、`db.repair()` 自愈、`db.clearAll()` 重置、迁移版本持久化（v0.4.2）
- 🧵 **关闭时序与后台任务加固** — 后台 flush/compaction 串行入队（close 排空后才关闭存储，失败显式报告 `ARIA_BACKGROUND_ERROR`）、预加载等待链稳定（compaction 竞态修复）、事务提交先落 WAL 再合并快照（v0.4.3）
- 📏 **SSTable 编码修复** — 块大小按 UTF-8 字节精确计算（大段中文内容不再因缓冲区低估崩溃）、长度字段 u32（>64KB value 不截断）、v1/v2 双格式兼容（旧库数据不丢）（v0.4.4）
- 🛡 **崩溃恢复自愈** — 残缺 SSTable 打开自动跳过、`db.repair()` 自愈（含孤儿页面清理与残留清理）、`db.clearAll()` 重置、迁移版本持久化（v0.4.2/v0.5.0）
- 🧵 **关闭时序与后台任务加固** — 后台 flush/compaction 串行入队（close 排空后才关闭存储，失败显式报告 `ARIA_BACKGROUND_ERROR`）
- 📏 **SSTable 编码修复** — 块大小按 UTF-8 字节精确计算、长度字段 u32、v1/v2 双格式兼容（v0.4.4）
- 🌲 **RB-Tree 完整实现** — 标准红黑树插入+删除修复，O(log n) 保证
- ⚡ **性能优化** — SSTableReader 二分查找统一、IndexedDB 索引利用、crypto 实例化避免全局状态
- 🌐 **浏览器兼容** — Chrome 80+ / Firefox 80+ / Safari 14+ / Edge 80+ / Node.js 16+
- 🧪 **958 测试 · 84.2% 覆盖率** — 52 套件，生产级质量保证
- 🧪 **1022 测试 · 86.7% 覆盖率** — 62 套件 + 7 个 Playwright 真实 Chromium e2e，生产级质量保证
---
@@ -217,6 +218,7 @@ await db2.disconnect(); // 引用计数 -1
| `maxRowsPerQuery` | `number` | `0` | 查询结果行数上限（0=不限制）✅ v0.2.5 生效 |
| `debug` | `boolean` | `false` | 调试模式，输出详细日志 🆕 |
| `onError` | `(error) => void` | — | 全局错误回调 🆕 |
| `aria` | `AriaEngineConfig` | — | AriaEngine 专属配置透传：`walSyncMode`/`checkpointInterval`/`encryption`/`pageStorage`/`compression` 等 🆕 v0.5.0 |
### ColumnDef 列定义
@@ -254,9 +256,24 @@ await db2.disconnect(); // 引用计数 -1
| `db.clearAll()` | 清空全部数据与表结构（保留库本身，实例可继续使用）🆕 v0.4.2 |
| `db.exportTable(name)` / `db.exportAll()` | 导出数据 JSON |
| `db.importTable(name, data)` | 导入数据 |
| `db.backup()` | 在线备份：全库一致性快照（Aria 引擎级备份 / 其余引擎回退 exportAll）🆕 v0.5.1 |
| `db.addMigration(v, fn)` / `db.migrateTo(v)` | 数据迁移（版本持久化到库内，重启不重跑）🆕 v0.4.2 |
| `db.subscribe(table, fn)` | 订阅表变更 |
| `db.on(hook, fn)` | 注册钩子 （14 种） |
| `db.on(hook, fn)` | 注册钩子 （14 种，全部真实触发）🆕 v0.5.1 |
### 维护语句（v0.5.1 SQL 入口）
| 语句 | 说明 | 引擎支持 |
|------|------|----------|
| `EXPLAIN SELECT ...` | 输出查询计划（type/table/where/estimatedRows/actualTimeMs） | 全部 |
| `ANALYZE [TABLE] name` | 收集表统计信息（行数/平均行大小/索引深度/列基数） | Aria |
| `REINDEX [TABLE] name` | 重建表二级索引 | Aria |
| `VACUUM` | 压缩 LSM + 清理 MVCC 碎片 | Aria |
| `SAVEPOINT name` | 创建嵌套事务保存点 | Aria |
| `ROLLBACK TO [SAVEPOINT] name` | 回滚到保存点 | Aria |
| `RELEASE [SAVEPOINT] name` | 释放保存点 | Aria |
> 不支持的引擎执行维护语句抛 `NOT_SUPPORTED`。
### 连接池（v0.1.13）
@@ -289,10 +306,12 @@ const { data, loading, refresh } = useSqlarkQuery(db, 'SELECT * FROM users');
| **事务回滚** | ✅ 快照 | ✅ 原子flush | ✅ 快照 | ✅ 双引擎 | ✅ MVCC |
| **二级索引** | ✅ Hash | ✅ Hash | ✅ Hash（重启恢复） | ✅ Hash | ✅ LSM（重启恢复） |
| **查询性能** | ⚡ O(1) PK | 🟡 O(1) PK | 🟡 O(1) PK | ⚡ O(1) PK | ⚡ O(log n) |
| **数据上限** | 内存限制 | ~2GB(IDB限制) | ~磁盘可用（整表重写，≤1000行/表为宜） | ~2GB(IDB) | 内存限制 |
| **浏览器** | 全部 | 全部 | Chrome/Edge 102+ | 全部 | 全部 |
| **适用场景** | 缓存/测试 | 标准持久化 | Chromium专有 | 速度+持久化 | 大规模/分析 |
| **测试覆盖** | 30+ | 30+ | 15 | 15+ | 200+ |
| **数据上限** | 内存限制 | ~2GB(IDB限制) | ~磁盘可用（页面化后大表可行） | ~2GB(IDB) | 内存限制 |
| **浏览器** | 全部 | 全部 | Chrome/Edge 102+ / Firefox 111+ / Safari 15.2+ | 全部 | 全部 |
| **多标签页** | — | ✅ | ❌（Web Locks 独占锁，v0.5.0） | ❌ | ✅ 锁保护 |
| **全库加密** | — | — | — | — | ✅ AES-GCM（v0.5.0） |
| **适用场景** | 缓存/测试 | 标准持久化 | Chromium+ 持久化 | 速度+持久化 | 大规模/分析 |
| **测试覆盖** | 30+ | 30+ | 15+e2e | 15+ | 400+ |
### Memory 模式
- **环境**: 所有浏览器、Node.js
@@ -307,10 +326,10 @@ const { data, loading, refresh } = useSqlarkQuery(db, 'SELECT * FROM users');
- **适用**: 标准前端数据库持久化场景
### Disk (OPFS) 模式
- **环境**: **仅限** Chrome 102+ / Edge 102+（Origin Private File System）
- **限制**: Firefox/Safari 不支持 OPFS API；每次写入重写整表 JSON 文件（大表性能差，不建议 >1000 行）
- **环境**: Chrome 102+ / Edge 102+ / Firefox 111+ / Safari 15.2+（Origin Private File System）
- **限制**: 每次写入重写整表 JSON 文件（大表性能差，不建议 >1000 行）；多标签页并发写无保护（Hybrid 模式）
- **能力**: 完整 CRUD、重启自动加载数据（空表/索引/schema 完整保留，v0.4.2）、事务回滚、并发写安全（内存快照一致）
- **适用**: Chromium 独占场景、小数据集持久化
- **适用**: 小数据集持久化（大表请用 Aria + OPFS 页面化存储）
### Hybrid 模式
- **环境**: 所有浏览器
@@ -320,8 +339,8 @@ const { data, loading, refresh } = useSqlarkQuery(db, 'SELECT * FROM users');
### Aria 模式
- **环境**: 所有浏览器（后端可选 IndexedDB / OPFS / Memory）
- **限制**: Memory 后端重启丢失；IndexedDB 后端受配额限制；OPFS 后端仅 Chromium
- **能力**: LSM-Tree 存储引擎、二级索引（跨重启恢复）、MVCC 事务、WAL 原子写入崩溃恢复（残缺 SSTable 打开自动跳过）、ON UPDATE/DELETE 外键级联、Bloom Filter、AES-GCM 加密、Savepoint、EXPLAIN、ANALYZE、REINDEX、VACUUM、`repair()` 自愈
- **限制**: Memory 后端重启丢失；IndexedDB 后端受配额限制；OPFS 后端需 OPFS 支持（Chrome 102+ / Firefox 111+ / Safari 15.2+）；同一库仅允许一个标签页打开（Web Locks 独占锁，第二个标签页抛 `ARIA_LOCKED`）
- **能力**: LSM-Tree 存储引擎、SSTable 4KB 页面化物理存储（OPFS 后端默认启用，BufferPool LRU 缓存）、二级索引（跨重启恢复）、MVCC 事务、WAL 分片文件原子写入崩溃恢复（真追加 + 空洞检测）、ON UPDATE/DELETE 外键级联、Bloom Filter、**全库 AES-GCM 透明加密**（v0.5.0）、LZ4 压缩、Savepoint、EXPLAIN、ANALYZE、REINDEX、VACUUM、`repair()` 自愈（含孤儿页面清理）
- **适用**: 大规模数据分析、需要自研引擎可控性的高级场景
---
@@ -331,11 +350,15 @@ const { data, loading, refresh } = useSqlarkQuery(db, 'SELECT * FROM users');
AriaEngine 是内置的页面式存储引擎，对标 SQLite 的设计理念：
```typescript
// 激活 AriaEngine
// 激活 AriaEngine（全库加密 + OPFS 页面化存储，v0.5.0）
const db = await MetonaSqlark.create({
name: 'my-app',
mode: 'aria', // 🆕 自研引擎模式
diskEngine: 'indexeddb', // 'indexeddb' | 'opfs' | 'memory'
diskEngine: 'opfs', // 'indexeddb' | 'opfs' | 'memory'
aria: { // 🆕 AriaEngine 专属配置透传（v0.5.0）
walSyncMode: 'full', // WAL 同步模式
encryption: { password: 'my-password' }, // 全库 AES-GCM 加密
},
});
// 与现有 API 完全兼容
@@ -351,36 +374,39 @@ const rows = await db.query('SELECT * FROM users');
```
┌──────────────────────────────────────────┐
│ AriaEngine v0.2.5 │
│ AriaEngine v0.5.0 │
│ (implements IStorageEngine) │
├──────────────────────────────────────────┤
│ LSM-Tree │ Buffer Pool │ WAL │
│ MemTable │ LRU (256pp) │ Recovery │
│ +SSTable │ +FileManager │ +CRC │
│ MemTable │ LRU (256pp) │ Segmented │
│ +SSTable │ +FileManager │ +CRC-32 │
│ (4KB Page) │ │ +空洞检测 │
├──────────────────────────────────────────┤
│ MVCC │ Bloom Filter │ LZ4 │
│ Snapshot │ FNV-1a+Murmur│ Compress │
│ +Savepoint│ +Serialize │ │
│ MVCC │ Bloom Filter │ LZ4 v2 │
│ Snapshot │ FNV-1a+Murmur│ +大小头 │
│ +Savepoint│ +Serialize │ │
├──────────────────────────────────────────┤
│ AES-GCM │ 二级索引 │ ANALYZE │
│ Encrypt │ Per-Column │ EXPLAIN │
│ EncryptedBackend │ 二级索引 │ ANALYZE │
│ AES-GCM 全库 │ Per-Column│ EXPLAIN │
├──────────────────────────────────────────┤
│ Storage Backend (IDB / OPFS / Memory) │
│ Web Locks 独占锁 │ OPFS Backend (v2) │
│ (多标签页) │ append/COW原子/清理 │
└──────────────────────────────────────────┘
```
| 特性 | 说明 |
|------|------|
| **LSM-Tree** | MemTable (红黑树) → SSTable 多级索引，异步 Compaction（从存储兜底加载，不依赖缓存），写背压 |
| **WAL** | Write-Ahead Log 二进制格式，CRC 校验，记录与计数单事务原子写入，full/batch/none 三种模式（full 模式真正同步 ✅ v0.2.5），16MB 阈值自动 checkpoint（活跃事务期间不截断 ✅ v0.4.2） |
| **崩溃恢复** | 打开时完整性校验（残缺 SSTable 自动跳过并清理）、WAL 按 key 扫描恢复（不丢记录）、恢复后自动重建二级索引 ✅ v0.4.2 |
| **页面化物理存储** | OPFS 后端下 SSTable 存为 4KB 页面（`pageStorage` 自动启用）：FileManager 分配 pageId、BufferPool LRU 缓存（256 页 ≈ 1MB 可控内存）、save 即落盘（WAL checkpoint 截断安全）、`SSTableMeta.pageIds` 持久化（旧整 value 数据兼容）v0.5.0 |
| **WAL** | 分片文件 `__wal_%06d.bin`（4MB 阈值切换）+ 真追加（createWritable keepExistingData）；标准 CRC32 记录校验；full/batch/none 三种模式（full 真正同步）；空洞检测截断；16MB 阈值自动 checkpoint（活跃事务期间不截断）v0.5.0 |
| **崩溃恢复** | 打开时完整性校验（残缺 SSTable 自动跳过并清理 + 整文件 CRC-32 校验）、WAL 按 key 扫描恢复（不丢记录）、恢复后自动重建二级索引 |
| **全库加密** | `encryption.password` 配置 → EncryptedBackend 透明加解密（WAL/SSTable/Schema/元数据全密文）；PBKDF2-SHA256 密钥派生 + salt/verifier 持久化；密码错误/数据篡改 → `ARIA_DECRYPT_ERROR`；明文库/加密库开关一致性检测 v0.5.0 |
| **MVCC** | 版本链 + 快照隔离，事务读写不互斥，提交/回滚按事务写入 key 精准清理，自动 GC |
| **Buffer Pool** | SSTable 缓存 LRU 上限（`bufferPoolPages` × `pageSize`，默认 256 页 ≈ 1MB 可控内存）✅ v0.2.6 生效，查询前异步预加载兜底，缓存驱逐不丢数据 |
| **Bloom Filter** | FNV-1a + Murmur 双哈希，SSTable footer 序列化，查询时 probe |
| **二级索引** | 每列独立 LSM Tree，支持 $eq/$in/$gt/$lt 范围扫描，跨重启自动恢复，WAL 恢复后自动重建 ✅ v0.4.2 |
| **AES-GCM** | PBKDF2 密钥派生 + AES-256-GCM 页面级加密，CryptoManager 实例化 ✅ v0.2.5 |
| **Compaction** | 异步 Leveled Compaction，Level 0 > 8 触发同步背压，compactLevel public 接口 ✅ v0.2.5 |
| **OPFS Backend** | 纯浏览器文件系统，Promise 队列串行写，零外部依赖 |
| **二级索引** | 每列独立 LSM Tree，支持 $eq/$in/$gt/$lt 范围扫描，跨重启自动恢复，WAL 恢复后自动重建 |
| **多标签页锁** | Web Locks API 库级独占锁（`ifAvailable` 不排队）：第二个标签页抛 `ARIA_LOCKED`；不支持的环境降级无锁并告警 v0.5.0 |
| **Compaction** | 异步 Leveled Compaction，Level 0 > 8 触发同步背压，compactLevel public 接口 |
| **OPFS Backend** | 零外部依赖纯浏览器文件系统：单文件 COW 原子写、`append` 真追加、写队列失败不中断、close 排空、崩溃残留自动清理 v0.5.0 |
---
@@ -391,6 +417,7 @@ npm install # 安装依赖
npm run dev # 开发模式（localhost:3001）
npm run build # 生产构建（生成 dist/）
npm test # 运行测试
npm run test:e2e # Playwright e2e（真实 Chromium + OPFS，需先 build）
npm run lint # 代码检查
npm run typecheck # 类型检查
```
@@ -401,23 +428,29 @@ npm run typecheck # 类型检查
| 指标 | 数值 |
|------|------|
| 测试用例 | 958 |
| 测试套件 | 51 |
| 行覆盖率 | 84.2% |
| 测试用例 | 1022 |
| 测试套件 | 62（+7 Playwright e2e） |
| 行覆盖率 | 86.7% |
| SQL 关键字 | 36 |
| 存储引擎 | 5（Memory / IndexedDB / OPFS / Hybrid / **Aria**） |
### 🌐 浏览器兼容性
| 浏览器 | 最低版本 | Memory | IndexedDB | OPFS | Aria |
|--------|----------|--------|-----------|------|------|
| Chrome | 80+ | ✅ | ✅ | ✅ (102+) | ✅ |
| Firefox | 80+ | ✅ | ✅ | ❌ | ✅ |
| Safari | 14+ | ✅ | ✅ | ❌ | ✅ |
| Edge | 80+ | ✅ | ✅ | ✅ (102+) | ✅ |
| Node.js | 16+ | ✅ | ✅ (fake-idb) | ❌ | ✅ |
| 浏览器 | 最低版本 | Memory | IndexedDB | OPFS | Web Locks | Aria |
|--------|----------|--------|-----------|------|-----------|------|
| Chrome | 80+ | ✅ | ✅ | ✅ (102+) | ✅ (69+) | ✅ |
| Firefox | 80+ | ✅ | ✅ | ✅ (111+) | ✅ (96+) | ✅ |
| Safari | 14+ | ✅ | ✅ | ✅ (15.2+) | ✅ (15.4+) | ✅ |
| Edge | 80+ | ✅ | ✅ | ✅ (102+) | ✅ (79+) | ✅ |
| Node.js | 16+ | ✅ | ✅ (fake-idb) | ❌ (测试用 mock) | ❌ | ✅ |
> **注意**: OPFS 模式仅限 Chromium 内核浏览器 (Chrome/Edge 102+)，Firefox/Safari 请使用 `diskEngine: 'indexeddb'`。
> **OPFS 支持说明（v0.4.5 更新）**: Chromium（Chrome/Edge 102+）、Firefox 111+、Safari 15.2+
> 均已支持基础 OPFS API（`createWritable` 原子写）。OPFS 无跨文件事务，
> AriaEngine 以 WAL 分片单文件原子写 + 空洞检测截断保证崩溃一致性。
>
> **多标签页保护（v0.4.5）**: AriaEngine 打开库时通过 Web Locks API 获取库级独占锁，
> 第二个标签页打开同一库会抛 `ARIA_LOCKED`。Web Locks 不可用的环境降级为无锁并告警
> （仅理论上，现代浏览器均支持）。
---
dist/metona-sqlark.cjs
Vendored
+1262 -522
View File
File diff suppressed because it is too large Load Diff
dist/metona-sqlark.cjs.map
Vendored
+1 -1
View File
File diff suppressed because one or more lines are too long
dist/metona-sqlark.d.ts
Vendored
+145 -62
View File
@@ -1,3 +1,44 @@
interface AriaEngineConfig {
/** 页面大小（默认 4096） */
pageSize?: number;
/** Buffer Pool 页面数量（默认 256） */
bufferPoolPages?: number;
/** MemTable 刷盘阈值（默认 4MB） */
memtableSizeThreshold?: number;
/** LSM 层级之间的容量倍数（默认 10） */
levelSizeMultiplier?: number;
/** Bloom Filter 每 key 位数（默认 10） */
bloomFilterBitsPerKey?: number;
/** 是否启用 WAL（默认 true） */
walEnabled?: boolean;
/** WAL 同步模式 */
walSyncMode?: 'full' | 'batch' | 'none';
/** Checkpoint 间隔（操作数，默认 1000） */
checkpointInterval?: number;
/** 是否启用页面压缩（默认 false） */
compression?: boolean;
/** 存储后端 */
storageBackend?: 'indexeddb' | 'opfs' | 'memory';
/** WAL 大小阈值（字节，超过则强制 checkpoint，默认 16MB） */
walSizeThreshold?: number;
/** 最大内存预算（MB，默认 64） */
maxMemoryMB?: number;
/**
* v0.4.5: 全库 AES-256-GCM 加密（backend 层透明加解密，WAL/SSTable/Schema/元数据全覆盖）。
* 密钥由 PBKDF2（salt 持久化于库内 __aria_keymeta）派生，重启用同一密码即可解密。
*/
encryption?: {
/** 加密密码 */
password: string;
};
/**
* v0.4.5: SSTable 页面化存储（4KB 页面 + BufferPool/FileManager 管理，LRU 缓存）。
* 默认：storageBackend === 'opfs' 时自动启用（大文件随机读/写放大优化）；
* 显式 false 强制关闭（整 value 存储，兼容旧行为）。
*/
pageStorage?: boolean;
}
/**
* metona-sqlark Constants — 类型定义 / 默认配置 / 枚举
* @module constants
@@ -64,6 +105,11 @@ interface DatabaseConfig {
debug?: boolean;
/** 多标签页同步（v0.3.2）：BroadcastChannel 广播表变更，其他标签页自动刷新 */
multiTabSync?: boolean;
/**
* v0.4.5: AriaEngine 专属配置（mode='aria' 时透传）：
* walSyncMode / checkpointInterval / encryption / pageStorage / compression 等
*/
aria?: AriaEngineConfig;
}
/** Where 条件操作符 */
type WhereOperator = '$eq' | '$ne' | '$gt' | '$gte' | '$lt' | '$lte' | '$in' | '$nin' | '$like' | '$and' | '$or' | '$not';
@@ -118,7 +164,7 @@ interface MetonaPlugin {
/** 销毁 */
destroy(): void;
}
declare const VERSION = "0.4.4";
declare const VERSION = "0.5.1";
/**
* metona-sqlark Plugin — 插件系统
@@ -350,7 +396,28 @@ interface CommitTransactionStatement {
interface RollbackTransactionStatement {
type: 'ROLLBACK';
}
type Statement = SelectStatement | SelectUnionStatement | ExplainStatement | InsertStatement | UpdateStatement | DeleteStatement | CreateTableStatement | DropTableStatement | AlterTableStatement | TruncateTableStatement | CreateIndexStatement | DropIndexStatement | BeginTransactionStatement | CommitTransactionStatement | RollbackTransactionStatement;
/** SAVEPOINT name / ROLLBACK TO SAVEPOINT name / RELEASE SAVEPOINT name */
interface SavepointStatement {
type: 'SAVEPOINT';
name: string;
/** SAVE = 创建；ROLLBACK = 回滚到；RELEASE = 释放 */
action: 'SAVE' | 'ROLLBACK' | 'RELEASE';
}
/** ANALYZE TABLE name — 收集表统计信息 */
interface AnalyzeStatement {
type: 'ANALYZE';
table: string;
}
/** REINDEX TABLE name — 重建表二级索引 */
interface ReindexStatement {
type: 'REINDEX';
table: string;
}
/** VACUUM — 压缩 LSM + 清理碎片 */
interface VacuumStatement {
type: 'VACUUM';
}
type Statement = SelectStatement | SelectUnionStatement | ExplainStatement | InsertStatement | UpdateStatement | DeleteStatement | CreateTableStatement | DropTableStatement | AlterTableStatement | TruncateTableStatement | CreateIndexStatement | DropIndexStatement | BeginTransactionStatement | CommitTransactionStatement | RollbackTransactionStatement | SavepointStatement | AnalyzeStatement | ReindexStatement | VacuumStatement;
/**
* metona-sqlark Query Executor — AST 执行器
@@ -363,8 +430,6 @@ declare class QueryExecutor {
private engine;
private maxRowsPerQuery;
constructor(engine: IStorageEngine, maxRowsPerQuery?: number);
/** 设置查询结果行数上限 */
setMaxRowsPerQuery(max: number): void;
execute(stmt: Statement): Promise<unknown>;
/** 递归执行 UNION / UNION ALL，返回合并结果 */
private executeSelectUnion;
@@ -406,6 +471,14 @@ declare class QueryExecutor {
private executeBegin;
private executeCommit;
private executeRollback;
/** SAVEPOINT name / ROLLBACK TO SAVEPOINT name / RELEASE SAVEPOINT name */
private executeSavepoint;
/** ANALYZE TABLE name — 收集表统计信息 */
private executeAnalyze;
/** REINDEX TABLE name — 重建表二级索引 */
private executeReindex;
/** VACUUM — 压缩 LSM + 清理碎片 */
private executeVacuum;
/** 列列表是否包含 CASE WHEN 表达式 */
private hasCaseColumn;
/**
@@ -456,14 +529,6 @@ declare class QueryExecutor {
private resolveOperatorSubqueries;
}
/**
* metona-sqlark Query Builder — 链式查询构建器
* @module query/builder
*
* 链式调用 → 构建 AST → 执行引擎操作。
* 支持 JOIN（需要 Executor）。
*/
declare class SelectQueryBuilder {
private engine;
private tableName;
@@ -508,7 +573,9 @@ declare class UpdateQueryBuilder {
private _updates;
private _where;
private onWrite?;
constructor(engine: IStorageEngine, tableName: string, _updates: Record<string, unknown>, onWrite?: (table: string) => void);
/** v0.5.1: CRUD hooks 触发回调 */
private onHooks?;
constructor(engine: IStorageEngine, tableName: string, _updates: Record<string, unknown>, onWrite?: (table: string) => void, onHooks?: (hook: HookName, args: unknown[]) => Promise<void>);
where(condition: WhereCondition): this;
execute(): Promise<number>;
toAST(): UpdateStatement;
@@ -518,17 +585,14 @@ declare class DeleteQueryBuilder {
private tableName;
private _where;
private onWrite?;
constructor(engine: IStorageEngine, tableName: string, onWrite?: (table: string) => void);
/** v0.5.1: CRUD hooks 触发回调 */
private onHooks?;
constructor(engine: IStorageEngine, tableName: string, onWrite?: (table: string) => void, onHooks?: (hook: HookName, args: unknown[]) => Promise<void>);
where(condition: WhereCondition): this;
execute(): Promise<number>;
toAST(): DeleteStatement;
}
/**
* metona-sqlark Table — 表操作 API
* @module table/table
*/
declare class Table<T = Record<string, unknown>> {
readonly name: string;
private engine;
@@ -536,7 +600,9 @@ declare class Table<T = Record<string, unknown>> {
private executor;
/** 写入回调（多标签页广播，v0.3.2） */
private onWrite?;
constructor(engine: IStorageEngine, tableName: string, executor?: QueryExecutor, onWrite?: (table: string) => void);
/** v0.5.1: CRUD 生命周期钩子触发回调（beforeInsert/afterInsert/... 真实接线） */
private onHooks?;
constructor(engine: IStorageEngine, tableName: string, executor?: QueryExecutor, onWrite?: (table: string) => void, onHooks?: (hook: HookName, args: unknown[]) => Promise<void>);
getSchema(): Promise<TableSchema>;
insert(row: T & Record<string, unknown>): Promise<string>;
insertMany(rows: (T & Record<string, unknown>)[]): Promise<string[]>;
@@ -637,6 +703,11 @@ declare class MetonaSqlark {
importTable(tableName: string, data: Record<string, unknown>[]): Promise<string[]>;
/** 导出整个数据库为 JSON */
exportAll(): Promise<Record<string, Record<string, unknown>[]>>;
/**
* v0.5.1: 在线备份 — 导出全库一致性快照。
* Aria 引擎走引擎级 backup()（MVCC 一致性视图）；其余引擎回退 exportAll()。
*/
backup(): Promise<Record<string, Record<string, unknown>[]>>;
private listeners;
/** 订阅表变更 */
subscribe(tableName: string, callback: (event: {
@@ -654,6 +725,12 @@ declare class MetonaSqlark {
broadcastChange(tableName: string): void;
/** 写语句对应的表名（多标签页广播用） */
private writeStatementTable;
/**
* v0.5.1: SQL 写语句触发 CRUD 生命周期钩子。
* INSERT/UPDATE/DELETE 分别触发 beforeInsert/afterInsert、beforeUpdate/afterUpdate、
* beforeDelete/afterDelete（参数与 Table API 路径一致）。
*/
private triggerStatementHooks;
private migrations;
/** 注册迁移 */
addMigration(version: number, up: (db: MetonaSqlark) => Promise<void>): void;
@@ -901,35 +978,6 @@ declare class OPFSEngine implements IStorageEngine {
* 空表不再消失、索引标记/主键/约束完整；无 schema 记录的旧库从数据推断（兼容）。
*/
private loadExistingTables;
/** 从 OPFS 加载表数据到内存缓存 */
loadTableIntoMemory(tableName: string, schema: TableSchema): Promise<void>;
}
interface AriaEngineConfig {
/** 页面大小（默认 4096） */
pageSize?: number;
/** Buffer Pool 页面数量（默认 256） */
bufferPoolPages?: number;
/** MemTable 刷盘阈值（默认 4MB） */
memtableSizeThreshold?: number;
/** LSM 层级之间的容量倍数（默认 10） */
levelSizeMultiplier?: number;
/** Bloom Filter 每 key 位数（默认 10） */
bloomFilterBitsPerKey?: number;
/** 是否启用 WAL（默认 true） */
walEnabled?: boolean;
/** WAL 同步模式 */
walSyncMode?: 'full' | 'batch' | 'none';
/** Checkpoint 间隔（操作数，默认 1000） */
checkpointInterval?: number;
/** 是否启用页面压缩（默认 false） */
compression?: boolean;
/** 存储后端 */
storageBackend?: 'indexeddb' | 'opfs' | 'memory';
/** WAL 大小阈值（字节，超过则强制 checkpoint，默认 16MB） */
walSizeThreshold?: number;
/** 最大内存预算（MB，默认 64） */
maxMemoryMB?: number;
}
declare class AriaEngine implements IStorageEngine {
@@ -941,6 +989,9 @@ declare class AriaEngine implements IStorageEngine {
private backend;
private opened;
private dbName;
private fileManager;
private bufferPool;
private dbLock;
private schemas;
private tablePKs;
private opCounter;
@@ -949,7 +1000,6 @@ declare class AriaEngine implements IStorageEngine {
private currentTxnId;
private txnSnapshot;
private gcCounter;
private bufferPool;
constructor(config?: AriaEngineConfig);
open(dbName: string, _version: number): Promise<void>;
/** open 内部实现（错误包装在 open 外层） */
@@ -957,9 +1007,15 @@ declare class AriaEngine implements IStorageEngine {
close(): Promise<void>;
/**
* v0.4.2-fix: 崩溃恢复/自愈 — 校验并移除损坏 SSTable、截断 WAL、重建二级索引。
* v0.4.5 增强：清理 OPFS 残留临时文件、清理孤儿页面（meta 未引用的 pg_ 文件）。
* 应用层检测到异常后调用，无需删库重建。
*/
repair(): Promise<void>;
/**
* v0.4.5: 清理孤儿页面 — 扫描全部 pg_* 文件，未被任何 LSM 命名空间 meta 引用的删除。
* 孤儿页面来自：崩溃中断的 compaction/删除流程（旧 SSTable 页面残留）。
*/
private cleanupOrphanPages;
/**
* v0.4.1: 重置数据库 — 清空全部数据与表结构（演示页刷新/重新初始化用）。
* 清空存储后端、LSM、WAL、MVCC 与二级索引，后续可继续使用本实例。
@@ -1044,6 +1100,8 @@ declare class AriaEngine implements IStorageEngine {
* - id 序列独立（避免 v0.2.4 共享 id 空间导致的文件互相覆盖）
*/
private createSSTableStore;
/** v0.4.5: 是否启用页面化物理存储（默认 OPFS 后端启用，显式配置可覆盖） */
private isPageStorage;
private applyWALRecord;
/**
* v0.3.3: DROP_TABLE 恢复 — 删除 schema 并清除主 LSM 中该表的所有残留数据。
@@ -1084,21 +1142,10 @@ declare class AriaEngine implements IStorageEngine {
compactedLevels: number;
gcVersions: number;
}>;
/**
* 查询优化器：估算各索引成本，选择最优方案
*/
estimateQueryCost(tableName: string, query: QueryPlan): {
strategy: string;
estimatedRows: number;
};
private ensureOpen;
/** v0.4.2-fix: Aria 事务中 DDL 显式拒绝（结构变更无法通过行快照回滚） */
private ensureNoDDLInTransaction;
private ensureTable;
/** Get the number of WAL records stored */
private getWALCount;
/** Set the number of WAL records stored */
private setWALCount;
}
declare class HybridEngine implements IStorageEngine {
@@ -1234,6 +1281,13 @@ declare enum TokenType {
THEN = "THEN",
ELSE = "ELSE",
END = "END",
EXPLAIN = "EXPLAIN",
ANALYZE = "ANALYZE",
REINDEX = "REINDEX",
VACUUM = "VACUUM",
SAVEPOINT = "SAVEPOINT",
RELEASE = "RELEASE",
TO = "TO",
IDENTIFIER = "IDENTIFIER",
STRING = "STRING",
NUMBER = "NUMBER",
@@ -1286,6 +1340,13 @@ interface IStorageBackend {
read(key: string): Promise<ArrayBuffer | null>;
/** 写入数据块 */
write(key: string, data: ArrayBuffer): Promise<void>;
/**
* 追加写入（v0.4.5 WAL 分片用，可选）：
* - OPFS 后端实现真追加（createWritable keepExistingData + seek，O(chunk)）
* - 未实现的后端由调用方回退 read+write（EncryptedBackend 包装时整体重写保正确性）
* 语义：在 key 现有内容末尾追加 data；key 不存在时等同 write。
*/
append?(key: string, data: ArrayBuffer): Promise<void>;
/**
* 批量原子写入（v0.4.2-fix）：多个 key 在单个底层事务中提交，
* 中断时整体回滚，不留半写状态。WAL count 与记录同事务保证一致性。
@@ -1312,7 +1373,17 @@ interface IStorageBackend {
* 零外部依赖，纯浏览器文件系统 API。
* 每个 key 对应 OPFS 目录下的一个二进制文件。
*
* 浏览器要求：Chrome 102+ / Edge 102+
* 原子性与一致性保证（v0.4.5 固化）：
* - 单文件 write/append：createWritable 为 copy-on-write —— close 前崩溃旧文件保持不变，
* close 后原子替换（单文件写入原子）
* - 多文件 writeMany/deleteMany：OPFS 无跨文件事务，串行逐个落盘；调用方（WAL 分片）
* 已改为单文件语义，多键操作仅用于一次性的 schema/meta 写入
* - 所有写操作串行队列化（同源同进程顺序一致）；单次任务失败不中断队列链，
* 错误如实返回给该次调用的调用方
* - close() 等待写队列排空后再释放目录句柄（杜绝 close 后挂起写丢失/读旧数据）
* - open() 自动清理崩溃残留临时文件（Chromium createWritable 的 .crswap 等）
*
* 浏览器要求：Chrome 102+ / Edge 102+（Safari 15.2+ / Firefox 111+ 支持基础 OPFS）
*/
declare class OPFSBackend implements IStorageBackend {
@@ -1323,8 +1394,20 @@ declare class OPFSBackend implements IStorageBackend {
open(name: string): Promise<void>;
close(): Promise<void>;
isOpen(): boolean;
/** 清理崩溃残留的临时文件（open 时自动调用，repair 也可调用） */
cleanupStaleFiles(): Promise<void>;
read(key: string): Promise<ArrayBuffer | null>;
/**
* 单文件原子写：createWritable 为 copy-on-write，close 后原子替换；
* 写入期间崩溃 → 旧文件保持（原子性由浏览器 OPFS 实现保证）。
*/
write(key: string, data: ArrayBuffer): Promise<void>;
/**
* v0.4.5: 真追加写 — createWritable(keepExistingData) + seek 到文件末尾。
* 单文件 COW 原子（close 前崩溃旧文件保持），无需读旧内容即实现 O(chunk) 追加
* （WAL 分片高频写入用）。
*/
append(key: string, data: ArrayBuffer): Promise<void>;
/** v0.4.2-fix: 批量写入 — 串行队列内逐个落盘（OPFS 无跨文件事务，顺序保证一致） */
writeMany(entries: Record<string, ArrayBuffer>): Promise<void>;
delete(key: string): Promise<void>;
dist/metona-sqlark.esm.js
Vendored
+1262 -522
View File
File diff suppressed because it is too large Load Diff
dist/metona-sqlark.esm.js.map
Vendored
+1 -1
View File
File diff suppressed because one or more lines are too long
dist/metona-sqlark.js
Vendored
+1262 -522
View File
File diff suppressed because it is too large Load Diff
dist/metona-sqlark.js.map
Vendored
+1 -1
View File
File diff suppressed because one or more lines are too long
dist/metona-sqlark.min.js
Vendored
+1 -1
View File
File diff suppressed because one or more lines are too long
jest.config.cjs
+6
View File
@@ -8,6 +8,12 @@ module.exports = {
'/node_modules/(?!(@rollup)/)',
],
moduleFileExtensions: ['ts', 'js', 'json'],
// Playwright e2e（tests/e2e/）与共享测试工具（tests/helpers/）不属于 jest 单元测试
testPathIgnorePatterns: [
'/node_modules/',
'/tests/e2e/',
'/tests/helpers/',
],
collectCoverageFrom: [
'src/**/*.ts',
'!src/index.ts',
package-lock.json
Generated
+67 -2
View File
@@ -1,18 +1,19 @@
{
"name": "@metona-team/metona-sqlark",
"version": "0.4.1",
"version": "0.4.4",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "@metona-team/metona-sqlark",
"version": "0.4.1",
"version": "0.4.4",
"license": "MIT",
"devDependencies": {
"@babel/core": "^7.22.0",
"@babel/plugin-transform-modules-commonjs": "^7.22.0",
"@babel/preset-env": "^7.22.0",
"@babel/preset-typescript": "^7.22.0",
"@playwright/test": "^1.62.1",
"@rollup/plugin-commonjs": "^25.0.0",
"@rollup/plugin-node-resolve": "^15.0.0",
"@rollup/plugin-terser": "^0.4.0",
@@ -25,6 +26,7 @@
"fake-indexeddb": "^6.2.5",
"jest": "^29.5.0",
"jest-environment-jsdom": "^29.5.0",
"playwright": "^1.62.1",
"prettier": "^2.8.0",
"rollup": "^3.20.0",
"rollup-plugin-dts": "^5.3.0",
@@ -2543,6 +2545,22 @@
"node": ">= 8"
}
},
"node_modules/@playwright/test": {
"version": "1.62.1",
"resolved": "https://registry.npmmirror.com/@playwright/test/-/test-1.62.1.tgz",
"integrity": "sha512-DTcUc8qii+cpHvtOwggMtBRMjKZHXYWdw8syRYu2vtzuq4Wxphqq4NfCs5Zt44L6mA8rfDfj+PHnxFc/FeK6mQ==",
"dev": true,
"license": "Apache-2.0",
"dependencies": {
"playwright": "1.62.1"
},
"bin": {
"playwright": "cli.js"
},
"engines": {
"node": ">=20"
}
},
"node_modules/@rollup/plugin-commonjs": {
"version": "25.0.8",
"resolved": "https://registry.npmmirror.com/@rollup/plugin-commonjs/-/plugin-commonjs-25.0.8.tgz",
@@ -6750,6 +6768,53 @@
"node": ">=8"
}
},
"node_modules/playwright": {
"version": "1.62.1",
"resolved": "https://registry.npmmirror.com/playwright/-/playwright-1.62.1.tgz",
"integrity": "sha512-0M+L3LAD8/nm554LOla9Ayx0j0tmFZ0FBcoQ7F1VuVHpM/XpiC8RcDzBQB8W5+hA8L22THxELzeF+2WcUzvcLg==",
"dev": true,
"license": "Apache-2.0",
"dependencies": {
"playwright-core": "1.62.1"
},
"bin": {
"playwright": "cli.js"
},
"engines": {
"node": ">=20"
},
"optionalDependencies": {
"fsevents": "2.3.2"
}
},
"node_modules/playwright-core": {
"version": "1.62.1",
"resolved": "https://registry.npmmirror.com/playwright-core/-/playwright-core-1.62.1.tgz",
"integrity": "sha512-wPYSwEBJY9GHraISXqyqtx0na0LpO3XEX7jNDhntbex7tzUS7kLnZsOlFruFJB4Hi/rhDMjXGqHewDZ68nYZVw==",
"dev": true,
"license": "Apache-2.0",
"bin": {
"playwright-core": "cli.js"
},
"engines": {
"node": ">=20"
}
},
"node_modules/playwright/node_modules/fsevents": {
"version": "2.3.2",
"resolved": "https://registry.npmmirror.com/fsevents/-/fsevents-2.3.2.tgz",
"integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==",
"dev": true,
"hasInstallScript": true,
"license": "MIT",
"optional": true,
"os": [
"darwin"
],
"engines": {
"node": "^8.16.0 || ^10.6.0 || >=11.0.0"
}
},
"node_modules/prelude-ls": {
"version": "1.2.1",
"resolved": "https://registry.npmmirror.com/prelude-ls/-/prelude-ls-1.2.1.tgz",
package.json
+4 -1
View File
@@ -1,6 +1,6 @@
{
"name": "@metona-team/metona-sqlark",
"version": "0.4.4",
"version": "0.5.1",
"description": "Frontend SQL database with in-memory and disk dual-mode storage",
"type": "module",
"main": "dist/metona-sqlark.cjs",
@@ -35,6 +35,7 @@
"test": "jest",
"test:coverage": "jest --coverage",
"test:watch": "jest --watch",
"test:e2e": "playwright test",
"lint": "eslint \"src/**/*.ts\"",
"lint:fix": "eslint \"src/**/*.ts\" --fix",
"format": "prettier --write \"src/**/*.ts\"",
@@ -66,6 +67,7 @@
"@babel/plugin-transform-modules-commonjs": "^7.22.0",
"@babel/preset-env": "^7.22.0",
"@babel/preset-typescript": "^7.22.0",
"@playwright/test": "^1.62.1",
"@rollup/plugin-commonjs": "^25.0.0",
"@rollup/plugin-node-resolve": "^15.0.0",
"@rollup/plugin-terser": "^0.4.0",
@@ -78,6 +80,7 @@
"fake-indexeddb": "^6.2.5",
"jest": "^29.5.0",
"jest-environment-jsdom": "^29.5.0",
"playwright": "^1.62.1",
"prettier": "^2.8.0",
"rollup": "^3.20.0",
"rollup-plugin-dts": "^5.3.0",
playwright.config.ts
+24
View File
@@ -0,0 +1,24 @@
/**
* Playwright E2E 配置 — 真实 Chromium 环境验证 OPFS 存储后端
*/
import { defineConfig } from '@playwright/test';
export default defineConfig({
testDir: './tests/e2e',
timeout: 60_000,
fullyParallel: false,
workers: 1, // OPFS 有锁/配额，串行执行避免互相干扰
retries: 1,
reporter: [['list']],
use: {
baseURL: 'http://127.0.0.1:3344',
trace: 'retain-on-failure',
},
webServer: {
// Node 静态服务（与 MetonaEditor 一致，不依赖 python3）
command: 'node tests/e2e/server.cjs 3344',
url: 'http://127.0.0.1:3344',
reuseExistingServer: !process.env.CI,
timeout: 30_000,
},
});
site/benchmark.html
+61 -33
View File
@@ -79,14 +79,14 @@
<div class="container">
<div class="hero">
<h1>⚡ 性能基准测试</h1>
<p>在浏览器中实测 <code>Memory</code> 与 <code>Aria</code>（LSM-Tree 自研引擎）两种模式的真实性能。
数据量分三档（1K / 10K / 50K 行），测量 <code>INSERT</code> / 主键查询 / 索引查询 / <code>UPDATE</code> / <code>DELETE</code> 的吞吐（ops/sec）。
<strong>测试结果取决于你的设备与浏览器</strong>，仅供参考。</p>
<p>在浏览器中实测 <code>Memory</code> 与 <code>Aria</code>（LSM-Tree 自研引擎）在纯内存 / IndexedDB / OPFS（页面化 4KB 存储）三种存储后端下的真实性能。
数据量分档（1K / 10K 行），测量 <code>INSERT</code> / 主键查询 / 索引查询 / <code>UPDATE</code> / <code>DELETE</code> 的吞吐（ops/sec）。
<strong>测试结果取决于你的设备与浏览器</strong>，仅供参考。磁盘后端结果反映真实持久化成本（WAL full 同步模式）。</p>
</div>
<div class="controls">
<button class="btn btn-run" id="run-btn" onclick="runAll()">▶ 运行完整基准</button>
<button class="btn btn-small" onclick="runAll(true)">⚡ 快速模式（1K/10K）</button>
<button class="btn btn-small" onclick="runAll(true)">⚡ 快速模式（仅 1K）</button>
<button class="btn btn-small" onclick="window.location.reload()">↻ 重置</button>
<span class="status" id="status"></span>
</div>
@@ -99,7 +99,9 @@
<thead><tr><th>引擎</th><th>数据量</th><th>INSERT</th><th>主键查询</th><th>索引查询</th><th>UPDATE</th><th>DELETE</th></tr></thead>
<tbody></tbody>
</table>
<p class="note">注：Memory 模式为纯内存 Map 存储；Aria 模式为 LSM-Tree + MemTable 存储（<code>storageBackend: 'memory'</code>，无磁盘 I/O）。
<p class="note">注：Memory = 纯内存 Map 引擎；Aria-memory = LSM-Tree（内存后端，无磁盘 I/O）；
Aria-IndexedDB = LSM-Tree（IDB 后端，整 value SSTable）；Aria-OPFS = LSM-Tree（OPFS 后端，<strong>4KB 页面化存储</strong> + BufferPool LRU 缓存 + WAL 分片真追加）。
磁盘后端使用 <code>walSyncMode: 'full'</code>（每条写入同步落盘，最安全的崩溃一致性）。
INSERT 为单行批量写入（含 WAL 记录）；UPDATE/DELETE 按主键条件执行。</p>
</div>
</div>
@@ -107,8 +109,16 @@
<script src="../dist/metona-sqlark.js"></script>
<script>
const M = window.MetonaSqlark;
const SIZES = [1000, 10000, 50000];
const SIZES_FAST = [1000, 10000];
const SIZES = [1000, 10000];
const SIZES_FAST = [1000];
// [label, mode, diskEngine]
const ENGINES = [
['memory', 'memory', 'memory'],
['aria-memory', 'aria', 'memory'],
['aria-indexeddb', 'aria', 'indexeddb'],
['aria-opfs', 'aria', 'opfs'],
];
let cancelled = false;
@@ -128,13 +138,25 @@ function makeRows(n) {
return rows;
}
async function createDb(mode) {
const db = new M.MetonaSqlark({
name: 'bench-' + mode + '-' + Date.now(),
mode,
diskEngine: 'memory',
async function createDb(mode, diskEngine) {
if (mode === 'memory') {
const db = new M.MetonaSqlark({ name: 'bench-mem-' + Date.now(), mode: 'memory' });
await db.init();
await db.defineTable('bench', {
id: { type: 'string', primaryKey: true },
name: { type: 'string' },
age: { type: 'number', index: true },
city: { type: 'string' },
});
return db;
}
// aria 模式：磁盘后端走真实持久化（WAL full 同步）
const db = await M.create({
name: 'bench-aria-' + diskEngine + '-' + Date.now(),
mode: 'aria',
diskEngine,
aria: { walSyncMode: 'full', checkpointInterval: 100000, memtableSizeThreshold: 64 * 1024 * 1024 },
});
await db.init();
await db.defineTable('bench', {
id: { type: 'string', primaryKey: true },
name: { type: 'string' },
@@ -150,39 +172,39 @@ async function timeIt(fn) {
return performance.now() - start;
}
async function runBench(mode, size) {
const db = await createDb(mode);
async function runBench(label, mode, diskEngine, size) {
const db = await createDb(mode, diskEngine);
const rows = makeRows(size);
const result = { mode, size, insert: 0, pk: 0, idx: 0, update: 0, del: 0 };
const result = { mode: label, size, insert: 0, pk: 0, idx: 0, update: 0, del: 0 };
try {
// INSERT（批量）
let ms = await timeIt(() => db.table('bench').insertMany(rows));
result.insert = Math.round(size / (ms / 1000));
// 主键查询 × 500
// 主键查询 × 300
ms = await timeIt(async () => {
for (let i = 0; i < 500; i++) await db.query(`SELECT * FROM bench WHERE id = 'k${(i * 37) % size}'`);
for (let i = 0; i < 300; i++) await db.query(`SELECT * FROM bench WHERE id = 'k${(i * 37) % size}'`);
});
result.pk = Math.round(500 / (ms / 1000));
result.pk = Math.round(300 / (ms / 1000));
// 索引查询 × 500
// 索引查询 × 300
ms = await timeIt(async () => {
for (let i = 0; i < 500; i++) await db.query(`SELECT * FROM bench WHERE age = ${18 + ((i * 17) % 60)}`);
for (let i = 0; i < 300; i++) await db.query(`SELECT * FROM bench WHERE age = ${18 + ((i * 17) % 60)}`);
});
result.idx = Math.round(500 / (ms / 1000));
result.idx = Math.round(300 / (ms / 1000));
// UPDATE × 200（按主键）
// UPDATE × 150（按主键）
ms = await timeIt(async () => {
for (let i = 0; i < 200; i++) await db.query(`UPDATE bench SET name = 'X${i}' WHERE id = 'k${(i * 53) % size}'`);
for (let i = 0; i < 150; i++) await db.query(`UPDATE bench SET name = 'X${i}' WHERE id = 'k${(i * 53) % size}'`);
});
result.update = Math.round(200 / (ms / 1000));
result.update = Math.round(150 / (ms / 1000));
// DELETE × 200（按主键）
// DELETE × 150（按主键）
ms = await timeIt(async () => {
for (let i = 0; i < 200; i++) await db.query(`DELETE FROM bench WHERE id = 'k${(i * 71) % size}'`);
for (let i = 0; i < 150; i++) await db.query(`DELETE FROM bench WHERE id = 'k${(i * 71) % size}'`);
});
result.del = Math.round(200 / (ms / 1000));
result.del = Math.round(150 / (ms / 1000));
} finally {
await db.close();
}
@@ -199,16 +221,16 @@ async function runAll(quick) {
document.getElementById('summary').innerHTML = '';
const results = [];
for (const mode of ['memory', 'aria']) {
for (const [label, mode, diskEngine] of ENGINES) {
for (const size of sizes) {
setStatus(`正在测试 ${mode === 'memory' ? 'Memory' : 'Aria'} 引擎 · ${size.toLocaleString()} 行…`, true);
setStatus(`正在测试 ${label} · ${size.toLocaleString()} 行…`, true);
// 让 UI 刷新
await new Promise((r) => setTimeout(r, 30));
try {
const r = await runBench(mode, size);
const r = await runBench(label, mode, diskEngine, size);
results.push(r);
} catch (e) {
setStatus(`✗ ${mode}/${size} 失败: ${e.message}`, false);
setStatus(`✗ ${label}/${size} 失败: ${e.message}`, false);
console.error(e);
}
}
@@ -239,13 +261,19 @@ function renderResults(results) {
`;
// 表格（按引擎分组）
const GROUP_LABEL = {
'memory': '🚀 Memory 引擎',
'aria-memory': '🌲 Aria 引擎（内存后端）',
'aria-indexeddb': '🌲 Aria 引擎（IndexedDB 后端）',
'aria-opfs': '🌲 Aria 引擎（OPFS · 4KB 页面化）',
};
let lastMode = '';
for (const r of results) {
if (r.mode !== lastMode) {
lastMode = r.mode;
const row = document.createElement('tr');
row.className = 'group-header';
row.innerHTML = `<td colspan="6">${r.mode === 'memory' ? '🚀 Memory 引擎' : '🌲 Aria 引擎（LSM-Tree）'}</td>`;
row.innerHTML = `<td colspan="6">${GROUP_LABEL[r.mode] || r.mode}</td>`;
tbody.appendChild(row);
}
const row = document.createElement('tr');
src/connection-manager.ts
-6
View File
@@ -106,12 +106,6 @@ class ConnectionManager {
return Array.from(this.connections.keys());
}
/**
* 获取连接的引用计数
*/
getRefCount(dbName: string): number {
return this.refCount.get(dbName) ?? 0;
}
}
// ---------------------------------------------------------------------------
src/constants.ts
+217 -217
View File
@@ -1,217 +1,217 @@
/**
* metona-sqlark Constants — 类型定义 / 默认配置 / 枚举
* @module constants
*/
// ---------------------------------------------------------------------------
// 存储模式
// ---------------------------------------------------------------------------
/** 存储模式 */
export type StorageMode = 'memory' | 'disk' | 'hybrid' | 'aria';
/** 磁盘引擎类型 */
export type DiskEngine = 'indexeddb' | 'opfs';
/** 所有存储模式 */
export const STORAGE_MODES: StorageMode[] = ['memory', 'disk', 'hybrid', 'aria'];
/** 所有磁盘引擎 */
export const DISK_ENGINES: DiskEngine[] = ['indexeddb', 'opfs'];
// ---------------------------------------------------------------------------
// 字段类型
// ---------------------------------------------------------------------------
/** 字段数据类型 */
export type FieldType = 'string' | 'number' | 'boolean' | 'date' | 'json';
/** 所有字段类型 */
export const FIELD_TYPES: FieldType[] = ['string', 'number', 'boolean', 'date', 'json'];
// ---------------------------------------------------------------------------
// 列定义 & 表结构
// ---------------------------------------------------------------------------
/** 列定义 */
export interface ColumnDef {
/** 字段类型 */
type: FieldType;
/** 是否主键 */
primaryKey?: boolean;
/** 是否必填 */
required?: boolean;
/** 是否唯一 */
unique?: boolean;
/** 默认值 */
default?: unknown;
/** 是否创建索引 */
index?: boolean;
/** 外键引用: 'table.column' */
references?: string;
/** 删除级联: 'CASCADE' | 'SET NULL' | 'RESTRICT' */
onDelete?: 'CASCADE' | 'SET NULL' | 'RESTRICT';
/** 更新级联: 'CASCADE' | 'SET NULL' | 'RESTRICT' */
onUpdate?: 'CASCADE' | 'SET NULL' | 'RESTRICT';
/** 字符串最大长度 */
maxLength?: number;
/** 数字最小值 */
min?: number;
/** 数字最大值 */
max?: number;
}
/** 表结构定义 */
export interface TableSchema {
/** 表名 */
name: string;
/** 列定义映射 */
columns: Record<string, ColumnDef>;
}
// ---------------------------------------------------------------------------
// 数据库配置
// ---------------------------------------------------------------------------
/** 数据库配置 */
export interface DatabaseConfig {
/** 数据库名称 */
name: string;
/** 存储模式 */
mode?: StorageMode;
/** 磁盘引擎（仅 mode='disk'|'hybrid' 时生效） */
diskEngine?: DiskEngine;
/** 版本号 */
version?: number;
/** 插件列表 */
plugins?: MetonaPlugin[];
/** 数据库就绪回调 */
onReady?: (db: unknown) => void;
/** 错误回调 */
onError?: (error: Error) => void;
/** 查询结果行数上限（默认 0，0 表示不限制） */
maxRowsPerQuery?: number;
/** 调试模式（启用后输出详细操作日志） */
debug?: boolean;
/** 多标签页同步（v0.3.2）：BroadcastChannel 广播表变更，其他标签页自动刷新 */
multiTabSync?: boolean;
}
/** 数据库默认配置 */
export const DB_DEFAULTS: Readonly<Required<Omit<DatabaseConfig, 'plugins' | 'onReady' | 'onError'>>> = Object.freeze({
name: 'metona-sqlark',
mode: 'hybrid' as const,
diskEngine: 'indexeddb' as const,
version: 1,
maxRowsPerQuery: 0, // 0 = 不限制
debug: false,
multiTabSync: false,
});
// ---------------------------------------------------------------------------
// Where 操作符
// ---------------------------------------------------------------------------
/** Where 条件操作符 */
export type WhereOperator = '$eq' | '$ne' | '$gt' | '$gte' | '$lt' | '$lte' | '$in' | '$nin' | '$like' | '$and' | '$or' | '$not';
/** 简单条件值：直接相等 */
export type SimpleCondition = unknown;
/** 操作符条件 */
export type OperatorCondition = Partial<Record<WhereOperator, unknown>>;
/** 字段条件：简单值 | 操作符对象 */
export type FieldCondition = SimpleCondition | OperatorCondition;
/** Where 条件对象 */
export type WhereCondition = Record<string, FieldCondition>;
// ---------------------------------------------------------------------------
// 排序
// ---------------------------------------------------------------------------
/** 排序方向 */
export type SortDirection = 'asc' | 'desc';
/** 排序定义 */
export interface OrderBy {
/** 列名 */
column: string;
/** 排序方向 */
direction: SortDirection;
/** v0.4.0: NULL 值排序位置（first 排最前 / last 排最后，默认同引擎行为） */
nulls?: 'first' | 'last';
}
// ---------------------------------------------------------------------------
// 查询计划（引擎层使用）
// ---------------------------------------------------------------------------
/** 查询计划 — 由 Executor 编译 AST 后生成 */
export interface QueryPlan {
/** 表名 */
table: string;
/** 要返回的列（undefined = 全部，['*'] = 全部） */
columns?: string[];
/** 过滤条件 */
where?: WhereCondition;
/** 排序 */
orderBy?: OrderBy[];
/** 限制条数 */
limit?: number;
/** 偏移量 */
offset?: number;
}
// ---------------------------------------------------------------------------
// 插件接口
// ---------------------------------------------------------------------------
/** 钩子名称 */
export type HookName =
| 'beforeCreateTable' | 'afterCreateTable'
| 'beforeDropTable' | 'afterDropTable'
| 'beforeInsert' | 'afterInsert'
| 'beforeUpdate' | 'afterUpdate'
| 'beforeDelete' | 'afterDelete'
| 'beforeQuery' | 'afterQuery'
| 'beforeTransaction' | 'afterTransaction';
/** 插件定义 */
export interface MetonaPlugin {
/** 插件名称 */
name: string;
/** 插件版本 */
version: string;
/** 描述 */
description?: string;
/** 优先级，越大越先执行 */
priority?: number;
/** 安装 */
install(db: unknown): void;
/** 销毁 */
destroy(): void;
}
// ---------------------------------------------------------------------------
// 错误类型
// ---------------------------------------------------------------------------
/** 数据库错误 */
export class DatabaseError extends Error {
constructor(
message: string,
public code: string,
public details?: unknown,
) {
super(message);
this.name = 'DatabaseError';
}
}
// ---------------------------------------------------------------------------
// 版本
// ---------------------------------------------------------------------------
export const VERSION = '0.4.4';
/**
* metona-sqlark Constants — 类型定义 / 默认配置 / 枚举
* @module constants
*/
// ---------------------------------------------------------------------------
// 存储模式
// ---------------------------------------------------------------------------
/** 存储模式 */
export type StorageMode = 'memory' | 'disk' | 'hybrid' | 'aria';
/** 磁盘引擎类型 */
export type DiskEngine = 'indexeddb' | 'opfs';
// ---------------------------------------------------------------------------
// 字段类型
// ---------------------------------------------------------------------------
/** 字段数据类型 */
export type FieldType = 'string' | 'number' | 'boolean' | 'date' | 'json';
/** 所有字段类型 */
export const FIELD_TYPES: FieldType[] = ['string', 'number', 'boolean', 'date', 'json'];
// ---------------------------------------------------------------------------
// 列定义 & 表结构
// ---------------------------------------------------------------------------
/** 列定义 */
export interface ColumnDef {
/** 字段类型 */
type: FieldType;
/** 是否主键 */
primaryKey?: boolean;
/** 是否必填 */
required?: boolean;
/** 是否唯一 */
unique?: boolean;
/** 默认值 */
default?: unknown;
/** 是否创建索引 */
index?: boolean;
/** 外键引用: 'table.column' */
references?: string;
/** 删除级联: 'CASCADE' | 'SET NULL' | 'RESTRICT' */
onDelete?: 'CASCADE' | 'SET NULL' | 'RESTRICT';
/** 更新级联: 'CASCADE' | 'SET NULL' | 'RESTRICT' */
onUpdate?: 'CASCADE' | 'SET NULL' | 'RESTRICT';
/** 字符串最大长度 */
maxLength?: number;
/** 数字最小值 */
min?: number;
/** 数字最大值 */
max?: number;
}
/** 表结构定义 */
export interface TableSchema {
/** 表名 */
name: string;
/** 列定义映射 */
columns: Record<string, ColumnDef>;
}
// ---------------------------------------------------------------------------
// 数据库配置
// ---------------------------------------------------------------------------
/** 数据库配置 */
export interface DatabaseConfig {
/** 数据库名称 */
name: string;
/** 存储模式 */
mode?: StorageMode;
/** 磁盘引擎（仅 mode='disk'|'hybrid' 时生效） */
diskEngine?: DiskEngine;
/** 版本号 */
version?: number;
/** 插件列表 */
plugins?: MetonaPlugin[];
/** 数据库就绪回调 */
onReady?: (db: unknown) => void;
/** 错误回调 */
onError?: (error: Error) => void;
/** 查询结果行数上限（默认 0，0 表示不限制） */
maxRowsPerQuery?: number;
/** 调试模式（启用后输出详细操作日志） */
debug?: boolean;
/** 多标签页同步（v0.3.2）：BroadcastChannel 广播表变更，其他标签页自动刷新 */
multiTabSync?: boolean;
/**
* v0.4.5: AriaEngine 专属配置（mode='aria' 时透传）：
* walSyncMode / checkpointInterval / encryption / pageStorage / compression 等
*/
aria?: import('./engine/aria/types').AriaEngineConfig;
}
/** 数据库默认配置 */
export const DB_DEFAULTS: Readonly<Required<Omit<DatabaseConfig, 'plugins' | 'onReady' | 'onError' | 'aria'>>> & { aria: undefined } = Object.freeze({
name: 'metona-sqlark',
mode: 'hybrid' as const,
diskEngine: 'indexeddb' as const,
version: 1,
maxRowsPerQuery: 0, // 0 = 不限制
debug: false,
multiTabSync: false,
aria: undefined,
});
// ---------------------------------------------------------------------------
// Where 操作符
// ---------------------------------------------------------------------------
/** Where 条件操作符 */
export type WhereOperator = '$eq' | '$ne' | '$gt' | '$gte' | '$lt' | '$lte' | '$in' | '$nin' | '$like' | '$and' | '$or' | '$not';
/** 简单条件值：直接相等 */
export type SimpleCondition = unknown;
/** 操作符条件 */
export type OperatorCondition = Partial<Record<WhereOperator, unknown>>;
/** 字段条件：简单值 | 操作符对象 */
export type FieldCondition = SimpleCondition | OperatorCondition;
/** Where 条件对象 */
export type WhereCondition = Record<string, FieldCondition>;
// ---------------------------------------------------------------------------
// 排序
// ---------------------------------------------------------------------------
/** 排序方向 */
export type SortDirection = 'asc' | 'desc';
/** 排序定义 */
export interface OrderBy {
/** 列名 */
column: string;
/** 排序方向 */
direction: SortDirection;
/** v0.4.0: NULL 值排序位置（first 排最前 / last 排最后，默认同引擎行为） */
nulls?: 'first' | 'last';
}
// ---------------------------------------------------------------------------
// 查询计划（引擎层使用）
// ---------------------------------------------------------------------------
/** 查询计划 — 由 Executor 编译 AST 后生成 */
export interface QueryPlan {
/** 表名 */
table: string;
/** 要返回的列（undefined = 全部，['*'] = 全部） */
columns?: string[];
/** 过滤条件 */
where?: WhereCondition;
/** 排序 */
orderBy?: OrderBy[];
/** 限制条数 */
limit?: number;
/** 偏移量 */
offset?: number;
}
// ---------------------------------------------------------------------------
// 插件接口
// ---------------------------------------------------------------------------
/** 钩子名称 */
export type HookName =
| 'beforeCreateTable' | 'afterCreateTable'
| 'beforeDropTable' | 'afterDropTable'
| 'beforeInsert' | 'afterInsert'
| 'beforeUpdate' | 'afterUpdate'
| 'beforeDelete' | 'afterDelete'
| 'beforeQuery' | 'afterQuery'
| 'beforeTransaction' | 'afterTransaction';
/** 插件定义 */
export interface MetonaPlugin {
/** 插件名称 */
name: string;
/** 插件版本 */
version: string;
/** 描述 */
description?: string;
/** 优先级，越大越先执行 */
priority?: number;
/** 安装 */
install(db: unknown): void;
/** 销毁 */
destroy(): void;
}
// ---------------------------------------------------------------------------
// 错误类型
// ---------------------------------------------------------------------------
/** 数据库错误 */
export class DatabaseError extends Error {
constructor(
message: string,
public code: string,
public details?: unknown,
) {
super(message);
this.name = 'DatabaseError';
}
}
// ---------------------------------------------------------------------------
// 版本
// ---------------------------------------------------------------------------
export const VERSION = '0.5.1';
src/core.ts
+65 -2
View File
@@ -152,7 +152,14 @@ export class MetonaSqlark {
let t = this.tableCache.get(name);
if (!t) {
// v0.3.2: 表操作写入后广播变更（多标签页同步）
t = new Table(this.engine, name, this.executor, (tableName) => this.broadcastChange(tableName));
// v0.5.1: CRUD 生命周期钩子真实接线（beforeInsert/afterInsert/...）
t = new Table(
this.engine,
name,
this.executor,
(tableName) => this.broadcastChange(tableName),
(hook, args) => this.pluginManager.trigger(hook, ...args),
);
this.tableCache.set(name, t);
}
return t;
@@ -192,7 +199,10 @@ export class MetonaSqlark {
// v0.3.0: 支持分号分隔的多语句，逐条顺序执行，返回最后一条的结果
const statements: Statement[] = parseAll(sql);
for (const stmt of statements) {
// v0.5.1: SQL 写语句触发 CRUD 生命周期钩子（与 Table API 路径一致）
await this.triggerStatementHooks(stmt, 'before');
result = await this.executor.execute(stmt);
await this.triggerStatementHooks(stmt, 'after', result);
// v0.3.2: 写语句广播表变更（多标签页同步）
const table = this.writeStatementTable(stmt);
if (table) this.broadcastChange(table);
@@ -344,6 +354,18 @@ export class MetonaSqlark {
return result;
}
/**
* v0.5.1: 在线备份 — 导出全库一致性快照。
* Aria 引擎走引擎级 backup()（MVCC 一致性视图）；其余引擎回退 exportAll()。
*/
async backup(): Promise<Record<string, Record<string, unknown>[]>> {
this.ensureReady();
if (typeof this.engine.backup === 'function') {
return this.engine.backup();
}
return this.exportAll();
}
// ---- 发布订阅 ----
private listeners: Map<string, Set<(data: unknown) => void>> = new Map();
@@ -393,6 +415,43 @@ export class MetonaSqlark {
}
}
/**
* v0.5.1: SQL 写语句触发 CRUD 生命周期钩子。
* INSERT/UPDATE/DELETE 分别触发 beforeInsert/afterInsert、beforeUpdate/afterUpdate、
* beforeDelete/afterDelete（参数与 Table API 路径一致）。
*/
private async triggerStatementHooks(stmt: Statement, phase: 'before' | 'after', result?: unknown): Promise<void> {
switch (stmt.type) {
case 'INSERT': {
const rows: Record<string, unknown>[] = (stmt.values ?? []).map((vals: unknown[]) => {
const row: Record<string, unknown> = {};
const cols = stmt.columns ?? [];
for (let i = 0; i < vals.length; i++) {
row[cols[i] ?? String(i)] = vals[i];
}
return row;
});
if (phase === 'before') await this.pluginManager.trigger('beforeInsert', rows);
else await this.pluginManager.trigger('afterInsert', rows, result as string[]);
break;
}
case 'UPDATE': {
const query = { table: stmt.table, where: stmt.where };
if (phase === 'before') await this.pluginManager.trigger('beforeUpdate', query, stmt.sets);
else await this.pluginManager.trigger('afterUpdate', query, stmt.sets, result as number);
break;
}
case 'DELETE': {
const query = { table: stmt.from, where: stmt.where };
if (phase === 'before') await this.pluginManager.trigger('beforeDelete', query);
else await this.pluginManager.trigger('afterDelete', query, result as number);
break;
}
default:
break;
}
}
// ---- 迁移 ----
private migrations: Map<number, (db: MetonaSqlark) => Promise<void>> = new Map();
@@ -497,7 +556,11 @@ export class MetonaSqlark {
case 'disk':
return diskEngine === 'opfs' ? new OPFSEngine() : new IndexedDBEngine();
case 'aria':
return new AriaEngine({ storageBackend: diskEngine === 'opfs' ? 'opfs' : 'indexeddb' });
// v0.4.5: 透传 AriaEngine 专属配置（walSyncMode/checkpointInterval/encryption/pageStorage 等）
return new AriaEngine({
storageBackend: diskEngine === 'opfs' ? 'opfs' : 'indexeddb',
...(this.config.aria ?? {}),
});
case 'hybrid':
return new HybridEngine(diskEngine);
default:
src/engine/aria/buffer/eviction.ts
+177 -206
View File
@@ -1,206 +1,177 @@
/**
* AriaEngine Buffer Pool Eviction — LRU 驱逐策略
* @module engine/aria/buffer/eviction
*/
import type { PageHandle } from '../types';
// ---------------------------------------------------------------------------
// LRU 双向链表
// ---------------------------------------------------------------------------
/**
* LRU 链表管理器 — 双向链表，头部是 most recently used，尾部是 least recently used。
*/
export class LRUList {
private head: PageHandle | null = null;
private tail: PageHandle | null = null;
private _size = 0;
get size(): number {
return this._size;
}
/** 将页面移到链表头部。如果是新页面则插入，已存在则移动。 */
moveToHead(page: PageHandle): void {
// 如果已经在头部，无需操作
if (this.head === page) return;
// 检测是否在链表中
const inList = page.prev !== null || page.next !== null || this.head === page || this.tail === page;
if (inList) {
// 先从当前位置移除
this.detach(page);
} else {
this._size++;
}
// 插入头部
page.prev = null;
page.next = this.head;
if (this.head) {
this.head.prev = page;
}
this.head = page;
if (!this.tail) {
this.tail = page;
}
}
/** 从链表中移除页面 */
remove(page: PageHandle): void {
const inList = page.prev !== null || page.next !== null || this.head === page || this.tail === page;
if (!inList) return;
this.detach(page);
this._size = Math.max(0, this._size - 1);
}
/** 内部：只调整指针，不修改 _size */
private detach(page: PageHandle): void {
if (page.prev) {
page.prev.next = page.next;
} else if (this.head === page) {
this.head = page.next;
}
if (page.next) {
page.next.prev = page.prev;
} else if (this.tail === page) {
this.tail = page.prev;
}
page.prev = null;
page.next = null;
}
/** 获取 LRU 尾部（最久未使用的页面） */
getLRU(): PageHandle | null {
return this.tail;
}
/** 弹出 LRU 尾部 */
popLRU(): PageHandle | null {
const lru = this.tail;
if (lru) {
this.remove(lru);
}
return lru;
}
/** 清空链表 */
clear(): void {
this.head = null;
this.tail = null;
this._size = 0;
}
/** 获取所有页面（用于迭代） */
getAllPages(): PageHandle[] {
const pages: PageHandle[] = [];
let current = this.head;
while (current) {
pages.push(current);
current = current.next;
}
return pages;
}
}
// ---------------------------------------------------------------------------
// Eviction 策略
// ---------------------------------------------------------------------------
export type EvictionCallback = (page: PageHandle) => Promise<void>;
/**
* 驱逐管理器 — 当 Buffer Pool 满时驱逐页面。
*/
export class EvictionManager {
private lru: LRUList;
private onEvict: EvictionCallback;
private capacity: number;
constructor(capacity: number, onEvict: EvictionCallback) {
this.lru = new LRUList();
this.capacity = capacity;
this.onEvict = onEvict;
}
/** 访问页面，更新 LRU */
access(page: PageHandle): void {
page.lastAccess = Date.now();
this.lru.moveToHead(page);
}
/** 添加新页面到池中 */
add(page: PageHandle): void {
this.access(page);
}
/** 移除指定页面 */
remove(page: PageHandle): void {
this.lru.remove(page);
}
/**
* 驱逐页面直到池中有足够空间。
* 只驱逐未 pin 的干净页面（dirty=false）。
* 如果没有干净页面可驱逐，尝试刷脏页。
*/
async evictIfNeeded(count: number): Promise<number> {
let evicted = 0;
while (this.lru.size + count > this.capacity && this.lru.size > 0) {
// 找到可驱逐的页面
const victim = this.findEvictionCandidate();
if (!victim) break;
// 脏页先刷盘
if (victim.dirty) {
await this.onEvict(victim);
victim.dirty = false;
}
this.lru.remove(victim);
evicted++;
}
return evicted;
}
/** 查找驱逐候选（优先干净页面，然后最久未用的脏页） */
private findEvictionCandidate(): PageHandle | null {
// 先从尾部找未 pin 的干净页面
let current = this.lru.getLRU();
while (current) {
if (current.pins === 0 && !current.dirty) return current;
current = current.prev;
}
// 没有干净页，找未 pin 的脏页
current = this.lru.getLRU();
while (current) {
if (current.pins === 0) return current;
current = current.prev;
}
return null;
}
/** 获取当前大小 */
getSize(): number {
return this.lru.size;
}
/** 获取容量 */
getCapacity(): number {
return this.capacity;
}
/** 清空 */
clear(): void {
this.lru.clear();
}
}
/**
* AriaEngine Buffer Pool Eviction — LRU 驱逐策略
* @module engine/aria/buffer/eviction
*/
import type { PageHandle } from '../types';
// ---------------------------------------------------------------------------
// LRU 双向链表
// ---------------------------------------------------------------------------
/**
* LRU 链表管理器 — 双向链表，头部是 most recently used，尾部是 least recently used。
*/
export class LRUList {
private head: PageHandle | null = null;
private tail: PageHandle | null = null;
private _size = 0;
get size(): number {
return this._size;
}
/** 将页面移到链表头部。如果是新页面则插入，已存在则移动。 */
moveToHead(page: PageHandle): void {
// 如果已经在头部，无需操作
if (this.head === page) return;
// 检测是否在链表中
const inList = page.prev !== null || page.next !== null || this.head === page || this.tail === page;
if (inList) {
// 先从当前位置移除
this.detach(page);
} else {
this._size++;
}
// 插入头部
page.prev = null;
page.next = this.head;
if (this.head) {
this.head.prev = page;
}
this.head = page;
if (!this.tail) {
this.tail = page;
}
}
/** 从链表中移除页面 */
remove(page: PageHandle): void {
const inList = page.prev !== null || page.next !== null || this.head === page || this.tail === page;
if (!inList) return;
this.detach(page);
this._size = Math.max(0, this._size - 1);
}
/** 内部：只调整指针，不修改 _size */
private detach(page: PageHandle): void {
if (page.prev) {
page.prev.next = page.next;
} else if (this.head === page) {
this.head = page.next;
}
if (page.next) {
page.next.prev = page.prev;
} else if (this.tail === page) {
this.tail = page.prev;
}
page.prev = null;
page.next = null;
}
/** 清空链表 */
clear(): void {
this.head = null;
this.tail = null;
this._size = 0;
}
/** 获取 LRU 尾部（最久未使用的页面） */
getLRU(): PageHandle | null {
return this.tail;
}
}
// ---------------------------------------------------------------------------
// Eviction 策略
// ---------------------------------------------------------------------------
export type EvictionCallback = (page: PageHandle) => Promise<void>;
/**
* 驱逐管理器 — 当 Buffer Pool 满时驱逐页面。
*/
export class EvictionManager {
private lru: LRUList;
private onEvict: EvictionCallback;
private capacity: number;
constructor(capacity: number, onEvict: EvictionCallback) {
this.lru = new LRUList();
this.capacity = capacity;
this.onEvict = onEvict;
}
/** 访问页面，更新 LRU */
access(page: PageHandle): void {
page.lastAccess = Date.now();
this.lru.moveToHead(page);
}
/** 添加新页面到池中 */
add(page: PageHandle): void {
this.access(page);
}
/** 移除指定页面 */
remove(page: PageHandle): void {
this.lru.remove(page);
}
/**
* 驱逐页面直到池中有足够空间。
* 只驱逐未 pin 的干净页面（dirty=false）。
* 如果没有干净页面可驱逐，尝试刷脏页。
*/
async evictIfNeeded(count: number): Promise<number> {
let evicted = 0;
while (this.lru.size + count > this.capacity && this.lru.size > 0) {
// 找到可驱逐的页面
const victim = this.findEvictionCandidate();
if (!victim) break;
// 脏页先刷盘
if (victim.dirty) {
await this.onEvict(victim);
victim.dirty = false;
}
this.lru.remove(victim);
evicted++;
}
return evicted;
}
/** 查找驱逐候选（优先干净页面，然后最久未用的脏页） */
private findEvictionCandidate(): PageHandle | null {
// 先从尾部找未 pin 的干净页面
let current = this.lru.getLRU();
while (current) {
if (current.pins === 0 && !current.dirty) return current;
current = current.prev;
}
// 没有干净页，找未 pin 的脏页
current = this.lru.getLRU();
while (current) {
if (current.pins === 0) return current;
current = current.prev;
}
return null;
}
/** 清空 */
clear(): void {
this.lru.clear();
}
}
src/engine/aria/buffer/pool.ts
+177 -185
View File
@@ -1,185 +1,177 @@
/**
* AriaEngine Buffer Pool — 页面缓存池
* @module engine/aria/buffer/pool
*
* 管理固定数量页面的 LRU 缓存，减少对底层储存的访问。
*/
import type { PageHandle } from '../types';
import { PageType, DEFAULT_BUFFER_POOL_PAGES } from '../types';
import { createPage } from '../page/format';
import { EvictionManager } from './eviction';
// ---------------------------------------------------------------------------
// Page Read / Write 回调
// ---------------------------------------------------------------------------
export interface PageIO {
/** 从存储后端读取页面 */
readPage(pageId: number): Promise<ArrayBuffer | null>;
/** 将页面写入存储后端 */
writePage(pageId: number, data: ArrayBuffer): Promise<void>;
/** 分配新页面 ID */
allocatePageId(): Promise<number>;
/** 释放页面 ID */
freePageId(pageId: number): Promise<void>;
}
// ---------------------------------------------------------------------------
// Buffer Pool
// ---------------------------------------------------------------------------
export class BufferPool {
private pages: Map<number, PageHandle> = new Map();
private eviction: EvictionManager;
private pageIO: PageIO;
private nextPageId = 0;
constructor(pageIO: PageIO, capacity: number = DEFAULT_BUFFER_POOL_PAGES) {
this.pageIO = pageIO;
this.eviction = new EvictionManager(capacity, async (page) => {
if (page.dirty) {
await this.pageIO.writePage(page.pageId, page.data);
page.dirty = false;
}
});
}
// -----------------------------------------------------------------------
// 页面获取
// -----------------------------------------------------------------------
/**
* 获取页面（必要时从磁盘读取）。
* 返回 pin 的页面，使用完成后必须调用 unpin()。
*/
async getPage(pageId: number): Promise<PageHandle | null> {
// 已在池中
let page = this.pages.get(pageId);
if (page) {
this.eviction.access(page);
page.pins++;
return page;
}
// 需要从磁盘加载
const buffer = await this.pageIO.readPage(pageId);
if (!buffer) return null;
// 确保有空间
await this.eviction.evictIfNeeded(1);
const type = new DataView(buffer).getUint8(4) as PageType;
page = {
pageId,
type,
data: buffer,
dirty: false,
pins: 1,
prev: null,
next: null,
lastAccess: Date.now(),
};
this.pages.set(pageId, page);
this.eviction.add(page);
return page;
}
/**
* 创建新页面。
*/
async newPage(type: PageType = PageType.DATA): Promise<PageHandle> {
const pageId = await this.pageIO.allocatePageId();
await this.eviction.evictIfNeeded(1);
const page = createPage(pageId, type);
page.pins = 1;
this.pages.set(pageId, page);
this.eviction.add(page);
return page;
}
/**
* 释放页面的 pin。
*/
unpin(page: PageHandle): void {
if (page.pins > 0) {
page.pins--;
}
}
/**
* 标记页面为脏（需要写回）。
*/
markDirty(page: PageHandle): void {
page.dirty = true;
}
/**
* 将脏页面刷新到磁盘。
*/
async flushPage(pageId: number): Promise<void> {
const page = this.pages.get(pageId);
if (page && page.dirty) {
await this.pageIO.writePage(pageId, page.data);
page.dirty = false;
}
}
/**
* 刷新所有脏页面。
*/
async flushAll(): Promise<void> {
for (const [, page] of this.pages) {
if (page.dirty) {
await this.pageIO.writePage(page.pageId, page.data);
page.dirty = false;
}
}
}
/**
* 从缓存中删除指定页面（不刷盘）。
*/
removePage(pageId: number): void {
const page = this.pages.get(pageId);
if (page) {
this.eviction.remove(page);
this.pages.delete(pageId);
}
}
/**
* 清空缓存池（先刷脏页）。
*/
async clear(): Promise<void> {
await this.flushAll();
this.pages.clear();
this.eviction.clear();
}
// -----------------------------------------------------------------------
// 统计
// -----------------------------------------------------------------------
/** 获取当前缓存页面数 */
getCachedPageCount(): number {
return this.pages.size;
}
/** 获取缓存容量 */
getCapacity(): number {
return this.eviction.getCapacity();
}
/** 获取脏页面数 */
getDirtyPageCount(): number {
let count = 0;
for (const [, page] of this.pages) {
if (page.dirty) count++;
}
return count;
}
}
/**
* AriaEngine Buffer Pool — 页面缓存池
* @module engine/aria/buffer/pool
*
* 管理固定数量页面的 LRU 缓存，减少对底层储存的访问。
*/
import type { PageHandle } from '../types';
import { PageType, DEFAULT_BUFFER_POOL_PAGES } from '../types';
import { createPage } from '../page/format';
import { EvictionManager } from './eviction';
// ---------------------------------------------------------------------------
// Page Read / Write 回调
// ---------------------------------------------------------------------------
export interface PageIO {
/** 从存储后端读取页面 */
readPage(pageId: number): Promise<ArrayBuffer | null>;
/** 将页面写入存储后端 */
writePage(pageId: number, data: ArrayBuffer): Promise<void>;
/** 分配新页面 ID */
allocatePageId(): Promise<number>;
/** v0.4.5: 批量分配页面 ID（一次 meta 持久化） */
allocatePageIds?(count: number): Promise<number[]>;
/** 释放页面 ID */
freePageId(pageId: number): Promise<void>;
}
// ---------------------------------------------------------------------------
// Buffer Pool
// ---------------------------------------------------------------------------
export class BufferPool {
private pages: Map<number, PageHandle> = new Map();
private eviction: EvictionManager;
private pageIO: PageIO;
private nextPageId = 0;
constructor(pageIO: PageIO, capacity: number = DEFAULT_BUFFER_POOL_PAGES) {
this.pageIO = pageIO;
this.eviction = new EvictionManager(capacity, async (page) => {
if (page.dirty) {
await this.pageIO.writePage(page.pageId, page.data);
page.dirty = false;
}
});
}
// -----------------------------------------------------------------------
// 页面获取
// -----------------------------------------------------------------------
/**
* 获取页面（必要时从磁盘读取）。
* 返回 pin 的页面，使用完成后必须调用 unpin()。
*/
async getPage(pageId: number): Promise<PageHandle | null> {
// 已在池中
let page = this.pages.get(pageId);
if (page) {
this.eviction.access(page);
page.pins++;
return page;
}
// 需要从磁盘加载
const buffer = await this.pageIO.readPage(pageId);
if (!buffer) return null;
// 确保有空间
await this.eviction.evictIfNeeded(1);
const type = new DataView(buffer).getUint8(4) as PageType;
page = {
pageId,
type,
data: buffer,
dirty: false,
pins: 1,
prev: null,
next: null,
lastAccess: Date.now(),
};
this.pages.set(pageId, page);
this.eviction.add(page);
return page;
}
/**
* v0.4.5: 批量创建新页面（一次页面 ID 分配，页面化 SSTable 保存用）。
* 返回的页面均 pin 且 dirty=false（调用方写入后需 markDirty + flushPage）。
*/
async newPages(count: number, type: PageType = PageType.DATA): Promise<PageHandle[]> {
if (count <= 0) return [];
let pageIds: number[];
if (typeof this.pageIO.allocatePageIds === 'function') {
pageIds = await this.pageIO.allocatePageIds!(count);
} else {
pageIds = [];
for (let i = 0; i < count; i++) pageIds.push(await this.pageIO.allocatePageId());
}
await this.eviction.evictIfNeeded(count);
const handles: PageHandle[] = [];
for (const pageId of pageIds) {
const page = createPage(pageId, type);
page.pins = 1;
this.pages.set(pageId, page);
this.eviction.add(page);
handles.push(page);
}
return handles;
}
/**
* 释放页面的 pin。
*/
unpin(page: PageHandle): void {
if (page.pins > 0) {
page.pins--;
}
}
/**
* 标记页面为脏（需要写回）。
*/
markDirty(page: PageHandle): void {
page.dirty = true;
}
/**
* 将脏页面刷新到磁盘。
*/
async flushPage(pageId: number): Promise<void> {
const page = this.pages.get(pageId);
if (page && page.dirty) {
await this.pageIO.writePage(pageId, page.data);
page.dirty = false;
}
}
/**
* 刷新所有脏页面。
*/
async flushAll(): Promise<void> {
for (const [, page] of this.pages) {
if (page.dirty) {
await this.pageIO.writePage(page.pageId, page.data);
page.dirty = false;
}
}
}
/**
* 从缓存中删除指定页面（不刷盘）。
*/
removePage(pageId: number): void {
const page = this.pages.get(pageId);
if (page) {
this.eviction.remove(page);
this.pages.delete(pageId);
}
}
/**
* 清空缓存池（先刷脏页）。
*/
async clear(): Promise<void> {
await this.flushAll();
this.pages.clear();
this.eviction.clear();
}
}
src/engine/aria/compression/lz4.ts
+36 -17
View File
@@ -2,10 +2,9 @@
* AriaEngine LZ4 Compression — 简化 LZ4 压缩/解压
* @module engine/aria/compression/lz4
*
* v0.2.6: 修复往返一致性
* - 匹配长度截断到 19 字节（matchField 上限 15 + MIN_MATCH），长匹配分段输出
* - 组合 token 的 matchField ∈ [1,15]；matchField=0 且 lo=0 表示末尾纯字面量（无 offset）
* - 消除"matchField=0 的组合 token 与纯字面量 token 歧义"
* v0.4.5 格式 v2：压缩流前增加 4 字节原始大小头（LE u32），
* 解压不再依赖外部估算（高压缩率数据下 buf.length*2 估算不足会截断）。
* 旧版压缩数据（无头）视为损坏（compression 选项自 v0.2.6 起已声明不向后兼容）。
*
* Token 格式（1 字节）:
* hi 4bit = litLen (0-15)
@@ -17,10 +16,16 @@
const MIN_MATCH = 4;
const MAX_MATCH = MIN_MATCH + 15; // 19，匹配长度上限
/** 原始大小头字节数 */
const HEADER_SIZE = 4;
export function compressLZ4(input: Uint8Array): Uint8Array {
// 空输入直接返回（无 token 可输出）
if (input.byteLength === 0) return input;
// 空输入：仅头部（原始大小 0）
if (input.byteLength === 0) {
const empty = new Uint8Array(HEADER_SIZE);
new DataView(empty.buffer).setUint32(0, 0, true);
return empty;
}
// 最坏情况：纯字面量分块输出 len/15 个 token + 末尾 token
// 上限：len + ceil(len/15) + 8（组合 token 的 offset 开销已包含在内）
@@ -73,24 +78,38 @@ export function compressLZ4(input: Uint8Array): Uint8Array {
litStart += chunk;
}
// 始终输出压缩流（即使比原数据略大）。
// 注意：不能返回原样 input —— 解压端无法区分"压缩流"与"原始数据"，
// 原样返回会导致解压器将原始字节误解析为 token（v0.2.6 修复）
return out.slice(0, di);
// v0.4.5: 前置原始大小头，解压端自描述
const stream = out.slice(0, di);
const combined = new Uint8Array(HEADER_SIZE + stream.byteLength);
new DataView(combined.buffer).setUint32(0, input.byteLength, true);
combined.set(stream, HEADER_SIZE);
return combined;
}
export function decompressLZ4(input: Uint8Array, originalSize: number): Uint8Array {
export function decompressLZ4(input: Uint8Array, _originalSize?: number): Uint8Array {
if (input.byteLength < HEADER_SIZE) {
throw new Error('LZ4 stream too short: missing header');
}
const view = new DataView(input.buffer, input.byteOffset, input.byteLength);
const originalSize = view.getUint32(0, true);
if (originalSize === 0 && input.byteLength === HEADER_SIZE) {
return new Uint8Array(0); // 空输入
}
if (originalSize <= 0 || originalSize > 0x3fffffff) {
throw new Error('Invalid LZ4 header: bad original size');
}
const stream = input.subarray(HEADER_SIZE);
const out = new Uint8Array(originalSize);
let si = 0, di = 0;
while (si < input.byteLength && di < originalSize) {
const token = input[si++];
while (si < stream.byteLength && di < originalSize) {
const token = stream[si++];
const litLen = (token >> 4) & 0x0F;
const matchField = token & 0x0F;
// 复制字面量
for (let i = 0; i < litLen && si < input.byteLength && di < originalSize; i++) {
out[di++] = input[si++];
for (let i = 0; i < litLen && si < stream.byteLength && di < originalSize; i++) {
out[di++] = stream[si++];
}
// matchField=0：纯字面量 token（无 offset 无匹配）。
@@ -98,8 +117,8 @@ export function decompressLZ4(input: Uint8Array, originalSize: number): Uint8Arr
if (matchField === 0) continue;
// 组合 token：读取 offset + 复制匹配（可能自重叠）
if (si + 1 >= input.byteLength) break;
const offset = input[si++] | (input[si++] << 8);
if (si + 1 >= stream.byteLength) break;
const offset = stream[si++] | (stream[si++] << 8);
const matchLen = matchField + MIN_MATCH;
for (let i = 0; i < matchLen && di < originalSize; i++) {
out[di] = out[di - offset];
src/engine/aria/crc32.ts
+58
View File
@@ -0,0 +1,58 @@
/**
* AriaEngine CRC32 — 标准 CRC-32（IEEE 802.3，多项式 0xEDB88320）
* @module engine/aria/crc32
*
* 查表法实现。
* 分段计算约定：
* crc32(head + tail) === crc32Finalize(crc32Update(crc32Update(0xFFFFFFFF, head), tail))
* === crc32(tail, crc32(head))
* 用于 SSTable 文件校验和与 WAL 记录完整性校验。
*/
/** CRC-32 查找表（0xEDB88320 反射多项式） */
const CRC32_TABLE = (() => {
const table = new Uint32Array(256);
for (let i = 0; i < 256; i++) {
let c = i;
for (let k = 0; k < 8; k++) {
c = c & 1 ? 0xedb88320 ^ (c >>> 1) : c >>> 1;
}
table[i] = c >>> 0;
}
return table;
})();
/** CRC-32 初始累加器状态 */
export const CRC32_INIT = 0xffffffff;
/**
* 更新内部 CRC 累加状态（供分段计算使用，接收中间状态返回中间状态）。
*/
export function crc32Update(state: number, data: Uint8Array): number {
let crc = state >>> 0;
for (let i = 0; i < data.byteLength; i++) {
crc = (CRC32_TABLE[(crc ^ data[i]) & 0xff] ^ (crc >>> 8)) >>> 0;
}
return crc >>> 0;
}
/** 将中间状态转换为最终校验和（终止计算） */
export function crc32Finalize(state: number): number {
return (state ^ CRC32_INIT) >>> 0;
}
/**
* 计算标准 CRC-32 校验和。
* @param data 输入字节
* @param seed 前序片段计算的最终校验和（首段传 0 或不传）
* @returns 32 位无符号校验和
*/
export function crc32(data: Uint8Array, seed = 0): number {
const state = crc32Update(seed ^ CRC32_INIT, data);
return crc32Finalize(state);
}
/** 分段计算便捷函数：crc32Continue(prevFinal, chunk) === crc32(chunk, prevFinal) */
export function crc32Continue(prev: number, data: Uint8Array): number {
return crc32(data, prev);
}
src/engine/aria/crypto.ts
+6 -32
View File
@@ -33,18 +33,20 @@ export class CryptoManager {
return actualSalt as Uint8Array;
}
async encryptPage(data: ArrayBuffer): Promise<{ iv: Uint8Array; data: ArrayBuffer }> {
async encryptPage(data: ArrayBuffer | Uint8Array): Promise<{ iv: Uint8Array; data: ArrayBuffer }> {
if (!this.cryptoKey) throw new Error('Crypto not initialized');
const iv = crypto.getRandomValues(new Uint8Array(IV_LENGTH)) as any;
// 传 TypedArray 视图而非裸 ArrayBuffer：SubtleCrypto 通过 ArrayBuffer.isView 检查，
// 对跨 realm / 跨 vm 环境的 ArrayBuffer 兼容（Node 18/20 的 webcrypto 对裸 ArrayBuffer 检查严格）
const ciphertext = await crypto.subtle.encrypt({ name: ALGO, iv } as any, this.cryptoKey, new Uint8Array(data));
const plain = (data instanceof Uint8Array ? data : new Uint8Array(data)) as BufferSource;
const ciphertext = await crypto.subtle.encrypt({ name: ALGO, iv } as any, this.cryptoKey, plain);
return { iv: iv as Uint8Array, data: ciphertext };
}
async decryptPage(iv: Uint8Array, data: ArrayBuffer): Promise<ArrayBuffer> {
async decryptPage(iv: Uint8Array, data: ArrayBuffer | Uint8Array): Promise<ArrayBuffer> {
if (!this.cryptoKey) throw new Error('Crypto not initialized');
return crypto.subtle.decrypt({ name: ALGO, iv } as any, this.cryptoKey, new Uint8Array(data));
const ciphertext = (data instanceof Uint8Array ? data : new Uint8Array(data)) as BufferSource;
return crypto.subtle.decrypt({ name: ALGO, iv } as any, this.cryptoKey, ciphertext);
}
close(): void {
@@ -53,31 +55,3 @@ export class CryptoManager {
}
}
// ---------------------------------------------------------------------------
// 全局兼容层（旧代码仍可使用全局函数）
// ---------------------------------------------------------------------------
const globalCrypto = new CryptoManager();
/** @deprecated 使用 CryptoManager 实例代替 */
export async function initCrypto(password: string, salt?: Uint8Array): Promise<Uint8Array> {
return globalCrypto.init(password, salt);
}
/** @deprecated 使用 CryptoManager 实例代替 */
export function isCryptoEnabled(): boolean { return globalCrypto.enabled; }
/** @deprecated 使用 CryptoManager 实例代替 */
export async function encryptPage(data: ArrayBuffer): Promise<{ iv: Uint8Array; data: ArrayBuffer }> {
return globalCrypto.encryptPage(data);
}
/** @deprecated 使用 CryptoManager 实例代替 */
export async function decryptPage(iv: Uint8Array, data: ArrayBuffer): Promise<ArrayBuffer> {
return globalCrypto.decryptPage(iv, data);
}
/** @deprecated 使用 CryptoManager 实例代替 */
export function closeCrypto(): void {
globalCrypto.close();
}
src/engine/aria/index.ts
+1848 -1827
View File
File diff suppressed because it is too large Load Diff
src/engine/aria/index/bloom.ts
+113 -123
View File
@@ -1,123 +1,113 @@
/**
* AriaEngine Bloom Filter — 快速判定 key 是否可能存在
* @module engine/aria/index/bloom
*
* 使用双哈希函数 + Kirsch-Mitzenmacher 优化生成 k 个哈希值。
*/
import { DEFAULT_BLOOM_BITS_PER_KEY } from '../types';
// ---------------------------------------------------------------------------
// BloomFilter
// ---------------------------------------------------------------------------
export class BloomFilter {
private bits: Uint8Array;
private numHashes: number;
private _inserted = 0;
/**
* @param numKeys 预期插入的 key 数量
* @param bitsPerKey 每个 key 的位数（默认 10，误报率约 1%）
*/
constructor(numKeys: number, bitsPerKey: number = DEFAULT_BLOOM_BITS_PER_KEY) {
// ceil(numKeys * bitsPerKey / 8)，最少 64 位
const numBits = Math.max(64, numKeys * bitsPerKey);
const numBytes = Math.ceil(numBits / 8);
this.bits = new Uint8Array(numBytes);
// k = bitsPerKey * ln(2) ≈ bitsPerKey * 0.69
this.numHashes = Math.max(1, Math.floor(bitsPerKey * 0.69));
}
/** 从现有数据恢复 */
static fromData(data: Uint8Array, numHashes: number): BloomFilter {
const bf = new BloomFilter(1); // dummy
bf.bits = data;
bf.numHashes = numHashes;
return bf;
}
/** 插入 key */
insert(key: string): void {
const hashes = this.getHashes(key);
for (const h of hashes) {
const byteIdx = Math.floor(h / 8);
const bitIdx = h % 8;
this.bits[byteIdx] |= (1 << bitIdx);
}
this._inserted++;
}
/** 检查 key 可能存在（false positive 可能，false negative 不可能） */
mayContain(key: string): boolean {
const hashes = this.getHashes(key);
for (const h of hashes) {
const byteIdx = Math.floor(h / 8);
const bitIdx = h % 8;
if ((this.bits[byteIdx] & (1 << bitIdx)) === 0) {
return false; // 确定不存在
}
}
return true; // 可能存在
}
/** 获取序列化数据 */
serialize(): Uint8Array {
return this.bits;
}
/** bit 数组大小 */
getBitSize(): number {
return this.bits.byteLength * 8;
}
/** 已插入 key 数量 */
getInsertedCount(): number {
return this._inserted;
}
/** hash 函数数量 */
getHashCount(): number {
return this.numHashes;
}
// -----------------------------------------------------------------------
// 哈希
// -----------------------------------------------------------------------
private getHashes(key: string): number[] {
const bits = this.bits.byteLength * 8;
const h1 = this.fnv1a(key);
const h2 = this.murmurSimple(key);
const hashes: number[] = [];
for (let i = 0; i < this.numHashes; i++) {
// Kirsch-Mitzenmacher: h_i = h1 + i * h2
const h = Math.abs((h1 + i * h2) % bits);
hashes.push(h);
}
return hashes;
}
/** FNV-1a 哈希 */
private fnv1a(str: string): number {
let hash = 0x811c9dc5;
for (let i = 0; i < str.length; i++) {
hash ^= str.charCodeAt(i);
hash = (hash * 0x01000193) >>> 0;
}
return hash;
}
/** 简化的 Murmur-like 哈希 */
private murmurSimple(str: string): number {
let hash = 0;
for (let i = 0; i < str.length; i++) {
const ch = str.charCodeAt(i);
hash = ((hash << 5) - hash + ch) | 0;
hash = (hash ^ (hash >>> 16)) >>> 0;
}
return Math.abs(hash);
}
}
/**
* AriaEngine Bloom Filter — 快速判定 key 是否可能存在
* @module engine/aria/index/bloom
*
* 使用双哈希函数 + Kirsch-Mitzenmacher 优化生成 k 个哈希值。
*/
import { DEFAULT_BLOOM_BITS_PER_KEY } from '../types';
// ---------------------------------------------------------------------------
// BloomFilter
// ---------------------------------------------------------------------------
export class BloomFilter {
private bits: Uint8Array;
private numHashes: number;
private _inserted = 0;
/**
* @param numKeys 预期插入的 key 数量
* @param bitsPerKey 每个 key 的位数（默认 10，误报率约 1%）
*/
constructor(numKeys: number, bitsPerKey: number = DEFAULT_BLOOM_BITS_PER_KEY) {
// ceil(numKeys * bitsPerKey / 8)，最少 64 位
const numBits = Math.max(64, numKeys * bitsPerKey);
const numBytes = Math.ceil(numBits / 8);
this.bits = new Uint8Array(numBytes);
// k = bitsPerKey * ln(2) ≈ bitsPerKey * 0.69
this.numHashes = Math.max(1, Math.floor(bitsPerKey * 0.69));
}
/** 从现有数据恢复 */
static fromData(data: Uint8Array, numHashes: number): BloomFilter {
const bf = new BloomFilter(1); // dummy
bf.bits = data;
bf.numHashes = numHashes;
return bf;
}
/** 插入 key */
insert(key: string): void {
const hashes = this.getHashes(key);
for (const h of hashes) {
const byteIdx = Math.floor(h / 8);
const bitIdx = h % 8;
this.bits[byteIdx] |= (1 << bitIdx);
}
this._inserted++;
}
/** 检查 key 可能存在（false positive 可能，false negative 不可能） */
mayContain(key: string): boolean {
const hashes = this.getHashes(key);
for (const h of hashes) {
const byteIdx = Math.floor(h / 8);
const bitIdx = h % 8;
if ((this.bits[byteIdx] & (1 << bitIdx)) === 0) {
return false; // 确定不存在
}
}
return true; // 可能存在
}
/** 获取序列化数据 */
serialize(): Uint8Array {
return this.bits;
}
/** hash 函数数量 */
getHashCount(): number {
return this.numHashes;
}
// -----------------------------------------------------------------------
// 哈希
// -----------------------------------------------------------------------
private getHashes(key: string): number[] {
const bits = this.bits.byteLength * 8;
const h1 = this.fnv1a(key);
const h2 = this.murmurSimple(key);
const hashes: number[] = [];
for (let i = 0; i < this.numHashes; i++) {
// Kirsch-Mitzenmacher: h_i = h1 + i * h2
const h = Math.abs((h1 + i * h2) % bits);
hashes.push(h);
}
return hashes;
}
/** FNV-1a 哈希 */
private fnv1a(str: string): number {
let hash = 0x811c9dc5;
for (let i = 0; i < str.length; i++) {
hash ^= str.charCodeAt(i);
hash = (hash * 0x01000193) >>> 0;
}
return hash;
}
/** 简化的 Murmur-like 哈希 */
private murmurSimple(str: string): number {
let hash = 0;
for (let i = 0; i < str.length; i++) {
const ch = str.charCodeAt(i);
hash = ((hash << 5) - hash + ch) | 0;
hash = (hash ^ (hash >>> 16)) >>> 0;
}
return Math.abs(hash);
}
}
src/engine/aria/index/lsm.ts
+768 -748
View File
File diff suppressed because it is too large Load Diff
src/engine/aria/index/memtable.ts
+462 -467
View File
@@ -1,467 +1,462 @@
/**
* AriaEngine MemTable — 基于红黑树的内存表
* @module engine/aria/index/memtable
*
* 写操作先进入 MemTable，达到阈值后冻结并 flush 成 SSTable。
*/
// ---------------------------------------------------------------------------
// RB-Tree Node
// ---------------------------------------------------------------------------
enum Color { RED, BLACK }
class RBNode<K, V> {
key: K;
value: V;
color: Color = Color.RED;
left: RBNode<K, V> | null = null;
right: RBNode<K, V> | null = null;
parent: RBNode<K, V> | null = null;
constructor(key: K, value: V) {
this.key = key;
this.value = value;
}
}
// ---------------------------------------------------------------------------
// Red-Black Tree
// ---------------------------------------------------------------------------
class RedBlackTree<K, V> {
private root: RBNode<K, V> | null = null;
private _size = 0;
get size(): number { return this._size; }
// ---- 插入 ----
insert(key: K, value: V): void {
const node = new RBNode(key, value);
if (!this.root) {
this.root = node;
node.color = Color.BLACK;
this._size++;
return;
}
let parent: RBNode<K, V> | null = null;
let current: RBNode<K, V> | null = this.root;
while (current) {
parent = current;
if (key < current.key) {
current = current.left;
} else if (key > current.key) {
current = current.right;
} else {
// 更新已存在的 key
current.value = value;
return;
}
}
node.parent = parent;
if (key < parent!.key) {
parent!.left = node;
} else {
parent!.right = node;
}
this._size++;
this.fixInsert(node);
}
// ---- 查找 ----
find(key: K): V | null {
let current = this.root;
while (current) {
if (key < current.key) {
current = current.left;
} else if (key > current.key) {
current = current.right;
} else {
return current.value;
}
}
return null;
}
// ---- 删除 ----
delete(key: K): boolean {
// 简化实现：标记删除（实际改为找到并调整树）
const node = this.findNode(key);
if (!node) return false;
this.deleteNode(node);
this._size--;
return true;
}
// ---- 遍历 ----
/** 中序遍历（有序） */
inorder(callback: (key: K, value: V) => void): void {
this._inorder(this.root, callback);
}
/** 范围遍历 */
rangeScan(
startKey: K,
endKey: K,
callback: (key: K, value: V) => void,
): void {
this._rangeScan(this.root, startKey, endKey, callback);
}
/** 获取所有条目 */
getAllEntries(): [K, V][] {
const entries: [K, V][] = [];
this.inorder((k, v) => entries.push([k, v]));
return entries;
}
/** 清空 */
clear(): void {
this.root = null;
this._size = 0;
}
// ---- 内部方法 ----
private findNode(key: K): RBNode<K, V> | null {
let current = this.root;
while (current) {
if (key < current.key) {
current = current.left;
} else if (key > current.key) {
current = current.right;
} else {
return current;
}
}
return null;
}
private deleteNode(node: RBNode<K, V>): void {
// 简化：用左子树最大或右子树最小替换
// 完整实现较复杂，这里采用简化策略
if (!node.left && !node.right) {
this.transplant(node, null);
if (node.color === Color.BLACK) this.fixDelete(null, node.parent);
} else if (!node.left) {
this.transplant(node, node.right);
if (node.color === Color.BLACK) this.fixDelete(node.right, node.right!.parent);
} else if (!node.right) {
this.transplant(node, node.left);
if (node.color === Color.BLACK) this.fixDelete(node.left, node.left!.parent);
} else {
const successor = this.minimum(node.right);
if (successor!.parent !== node) {
this.transplant(successor!, successor!.right);
successor!.right = node.right;
successor!.right!.parent = successor;
}
this.transplant(node, successor);
successor!.left = node.left;
successor!.left!.parent = successor;
const origColor = successor!.color;
successor!.color = node.color;
if (origColor === Color.BLACK) this.fixDelete(successor!.right, successor!.right?.parent ?? null);
}
}
private transplant(u: RBNode<K, V> | null, v: RBNode<K, V> | null): void {
if (!u!.parent) {
this.root = v;
} else if (u === u!.parent.left) {
u!.parent.left = v;
} else {
u!.parent.right = v;
}
if (v) v.parent = u!.parent;
}
private minimum(node: RBNode<K, V>): RBNode<K, V> {
while (node.left) node = node.left;
return node;
}
private fixInsert(node: RBNode<K, V>): void {
while (node.parent && node.parent.color === Color.RED) {
const parent = node.parent;
const grandparent = parent.parent;
if (!grandparent) break;
if (parent === grandparent.left) {
const uncle = grandparent.right;
if (uncle && uncle.color === Color.RED) {
parent.color = Color.BLACK;
uncle.color = Color.BLACK;
grandparent.color = Color.RED;
node = grandparent;
} else {
if (node === parent.right) {
node = parent;
this.rotateLeft(node);
}
if (node.parent) node.parent.color = Color.BLACK;
if (node.parent?.parent) node.parent.parent.color = Color.RED;
if (node.parent?.parent) this.rotateRight(node.parent.parent);
}
} else {
const uncle = grandparent.left;
if (uncle && uncle.color === Color.RED) {
parent.color = Color.BLACK;
uncle.color = Color.BLACK;
grandparent.color = Color.RED;
node = grandparent;
} else {
if (node === parent.left) {
node = parent;
this.rotateRight(node);
}
if (node.parent) node.parent.color = Color.BLACK;
if (node.parent?.parent) node.parent.parent.color = Color.RED;
if (node.parent?.parent) this.rotateLeft(node.parent.parent);
}
}
}
if (this.root) this.root.color = Color.BLACK;
}
private fixDelete(x: RBNode<K, V> | null, parent: RBNode<K, V> | null): void {
// 标准 RB-Tree 删除修复（修复"双黑"问题）
let node = x;
let nodeParent = parent;
while ((!node || node.color === Color.BLACK) && node !== this.root) {
if (!nodeParent) break;
if (node === nodeParent.left) {
let sibling = nodeParent.right;
if (!sibling) break;
// Case 1: 兄弟是红色
if (sibling.color === Color.RED) {
sibling.color = Color.BLACK;
nodeParent.color = Color.RED;
this.rotateLeft(nodeParent);
sibling = nodeParent.right;
if (!sibling) break;
}
// Case 2: 兄弟的两个子节点都是黑色
const sibLeft = sibling.left;
const sibRight = sibling.right;
if ((!sibLeft || sibLeft.color === Color.BLACK) &&
(!sibRight || sibRight.color === Color.BLACK)) {
sibling.color = Color.RED;
node = nodeParent;
nodeParent = node.parent;
} else {
// Case 3: 兄弟右子黑色（左子红色）
if (!sibRight || sibRight.color === Color.BLACK) {
if (sibLeft) sibLeft.color = Color.BLACK;
sibling.color = Color.RED;
this.rotateRight(sibling);
sibling = nodeParent.right;
if (!sibling) break;
}
// Case 4: 兄弟右子红色
sibling.color = nodeParent.color;
nodeParent.color = Color.BLACK;
if (sibling.right) sibling.right.color = Color.BLACK;
this.rotateLeft(nodeParent);
node = this.root;
}
} else {
// 镜像：node 是父节点的右子
let sibling = nodeParent.left;
if (!sibling) break;
if (sibling.color === Color.RED) {
sibling.color = Color.BLACK;
nodeParent.color = Color.RED;
this.rotateRight(nodeParent);
sibling = nodeParent.left;
if (!sibling) break;
}
const sibLeft = sibling.left;
const sibRight = sibling.right;
if ((!sibLeft || sibLeft.color === Color.BLACK) &&
(!sibRight || sibRight.color === Color.BLACK)) {
sibling.color = Color.RED;
node = nodeParent;
nodeParent = node.parent;
} else {
if (!sibLeft || sibLeft.color === Color.BLACK) {
if (sibRight) sibRight.color = Color.BLACK;
sibling.color = Color.RED;
this.rotateLeft(sibling);
sibling = nodeParent.left;
if (!sibling) break;
}
sibling.color = nodeParent.color;
nodeParent.color = Color.BLACK;
if (sibling.left) sibling.left.color = Color.BLACK;
this.rotateRight(nodeParent);
node = this.root;
}
}
}
if (node) node.color = Color.BLACK;
}
private rotateLeft(x: RBNode<K, V>): void {
const y = x.right;
if (!y) return;
x.right = y.left;
if (y.left) y.left.parent = x;
y.parent = x.parent;
if (!x.parent) {
this.root = y;
} else if (x === x.parent.left) {
x.parent.left = y;
} else {
x.parent.right = y;
}
y.left = x;
x.parent = y;
}
private rotateRight(x: RBNode<K, V>): void {
const y = x.left;
if (!y) return;
x.left = y.right;
if (y.right) y.right.parent = x;
y.parent = x.parent;
if (!x.parent) {
this.root = y;
} else if (x === x.parent.right) {
x.parent.right = y;
} else {
x.parent.left = y;
}
y.right = x;
x.parent = y;
}
private _inorder(node: RBNode<K, V> | null, cb: (k: K, v: V) => void): void {
if (!node) return;
this._inorder(node.left, cb);
cb(node.key, node.value);
this._inorder(node.right, cb);
}
private _rangeScan(
node: RBNode<K, V> | null,
start: K,
end: K,
cb: (k: K, v: V) => void,
): void {
if (!node) return;
if (node.key > start) this._rangeScan(node.left, start, end, cb);
if (node.key >= start && node.key <= end) cb(node.key, node.value);
if (node.key < end) this._rangeScan(node.right, start, end, cb);
}
}
// ---------------------------------------------------------------------------
// MemTable
// ---------------------------------------------------------------------------
export class MemTable {
private tree: RedBlackTree<string, Record<string, unknown>>;
private _estimatedSize = 0;
private maxSize: number;
constructor(maxSize: number = 4 * 1024 * 1024) {
this.tree = new RedBlackTree();
this.maxSize = maxSize;
}
/** 插入或更新 */
put(key: string, value: Record<string, unknown>): void {
const oldSize = this.estimateEntrySize(key, this.tree.find(key));
const newSize = this.estimateEntrySize(key, value);
this.tree.insert(key, value);
this._estimatedSize += newSize - oldSize;
}
/** 获取 */
get(key: string): Record<string, unknown> | null {
return this.tree.find(key);
}
/** 删除 */
delete(key: string): boolean {
const oldVal = this.tree.find(key);
if (oldVal) {
this._estimatedSize -= this.estimateEntrySize(key, oldVal);
}
return this.tree.delete(key);
}
/** 是否应刷盘 */
shouldFlush(): boolean {
return this._estimatedSize >= this.maxSize;
}
/** 获取所有有序条目 */
getAllEntries(): [string, Record<string, unknown>][] {
return this.tree.getAllEntries();
}
/** 范围扫描 */
rangeScan(
startKey: string,
endKey: string,
): [string, Record<string, unknown>][] {
const entries: [string, Record<string, unknown>][] = [];
this.tree.rangeScan(startKey, endKey, (k, v) => entries.push([k, v]));
return entries;
}
/** 条目数 */
getEntryCount(): number {
return this.tree.size;
}
/** 估计大小（字节） */
getEstimatedSize(): number {
return this._estimatedSize;
}
/** 清空 */
clear(): void {
this.tree.clear();
this._estimatedSize = 0;
}
/** 检查 key 是否存在 */
contains(key: string): boolean {
return this.tree.find(key) !== null;
}
// -----------------------------------------------------------------------
// 内部
// -----------------------------------------------------------------------
private estimateEntrySize(key: string, value: Record<string, unknown> | null): number {
if (!value) return 0;
let size = key.length * 2; // UTF-16
for (const entry of Object.entries(value)) {
size += entry[0].length * 2;
const v = entry[1];
if (typeof v === 'string') size += v.length * 2;
else if (typeof v === 'number') size += 8;
else if (typeof v === 'boolean') size += 1;
else if (v === null || v === undefined) size += 1;
else size += 16; // rough estimate
}
return size;
}
}
/**
* AriaEngine MemTable — 基于红黑树的内存表
* @module engine/aria/index/memtable
*
* 写操作先进入 MemTable，达到阈值后冻结并 flush 成 SSTable。
*/
// ---------------------------------------------------------------------------
// RB-Tree Node
// ---------------------------------------------------------------------------
enum Color { RED, BLACK }
class RBNode<K, V> {
key: K;
value: V;
color: Color = Color.RED;
left: RBNode<K, V> | null = null;
right: RBNode<K, V> | null = null;
parent: RBNode<K, V> | null = null;
constructor(key: K, value: V) {
this.key = key;
this.value = value;
}
}
// ---------------------------------------------------------------------------
// Red-Black Tree
// ---------------------------------------------------------------------------
class RedBlackTree<K, V> {
private root: RBNode<K, V> | null = null;
private _size = 0;
get size(): number { return this._size; }
// ---- 插入 ----
insert(key: K, value: V): void {
const node = new RBNode(key, value);
if (!this.root) {
this.root = node;
node.color = Color.BLACK;
this._size++;
return;
}
let parent: RBNode<K, V> | null = null;
let current: RBNode<K, V> | null = this.root;
while (current) {
parent = current;
if (key < current.key) {
current = current.left;
} else if (key > current.key) {
current = current.right;
} else {
// 更新已存在的 key
current.value = value;
return;
}
}
node.parent = parent;
if (key < parent!.key) {
parent!.left = node;
} else {
parent!.right = node;
}
this._size++;
this.fixInsert(node);
}
// ---- 查找 ----
find(key: K): V | null {
let current = this.root;
while (current) {
if (key < current.key) {
current = current.left;
} else if (key > current.key) {
current = current.right;
} else {
return current.value;
}
}
return null;
}
// ---- 删除 ----
delete(key: K): boolean {
// 简化实现：标记删除（实际改为找到并调整树）
const node = this.findNode(key);
if (!node) return false;
this.deleteNode(node);
this._size--;
return true;
}
// ---- 遍历 ----
/** 中序遍历（有序） */
inorder(callback: (key: K, value: V) => void): void {
this._inorder(this.root, callback);
}
/** 范围遍历 */
rangeScan(
startKey: K,
endKey: K,
callback: (key: K, value: V) => void,
): void {
this._rangeScan(this.root, startKey, endKey, callback);
}
/** 获取所有条目 */
getAllEntries(): [K, V][] {
const entries: [K, V][] = [];
this.inorder((k, v) => entries.push([k, v]));
return entries;
}
/** 清空 */
clear(): void {
this.root = null;
this._size = 0;
}
// ---- 内部方法 ----
private findNode(key: K): RBNode<K, V> | null {
let current = this.root;
while (current) {
if (key < current.key) {
current = current.left;
} else if (key > current.key) {
current = current.right;
} else {
return current;
}
}
return null;
}
private deleteNode(node: RBNode<K, V>): void {
// 简化：用左子树最大或右子树最小替换
// 完整实现较复杂，这里采用简化策略
if (!node.left && !node.right) {
this.transplant(node, null);
if (node.color === Color.BLACK) this.fixDelete(null, node.parent);
} else if (!node.left) {
this.transplant(node, node.right);
if (node.color === Color.BLACK) this.fixDelete(node.right, node.right!.parent);
} else if (!node.right) {
this.transplant(node, node.left);
if (node.color === Color.BLACK) this.fixDelete(node.left, node.left!.parent);
} else {
const successor = this.minimum(node.right);
if (successor!.parent !== node) {
this.transplant(successor!, successor!.right);
successor!.right = node.right;
successor!.right!.parent = successor;
}
this.transplant(node, successor);
successor!.left = node.left;
successor!.left!.parent = successor;
const origColor = successor!.color;
successor!.color = node.color;
if (origColor === Color.BLACK) this.fixDelete(successor!.right, successor!.right?.parent ?? null);
}
}
private transplant(u: RBNode<K, V> | null, v: RBNode<K, V> | null): void {
if (!u!.parent) {
this.root = v;
} else if (u === u!.parent.left) {
u!.parent.left = v;
} else {
u!.parent.right = v;
}
if (v) v.parent = u!.parent;
}
private minimum(node: RBNode<K, V>): RBNode<K, V> {
while (node.left) node = node.left;
return node;
}
private fixInsert(node: RBNode<K, V>): void {
while (node.parent && node.parent.color === Color.RED) {
const parent = node.parent;
const grandparent = parent.parent;
if (!grandparent) break;
if (parent === grandparent.left) {
const uncle = grandparent.right;
if (uncle && uncle.color === Color.RED) {
parent.color = Color.BLACK;
uncle.color = Color.BLACK;
grandparent.color = Color.RED;
node = grandparent;
} else {
if (node === parent.right) {
node = parent;
this.rotateLeft(node);
}
if (node.parent) node.parent.color = Color.BLACK;
if (node.parent?.parent) node.parent.parent.color = Color.RED;
if (node.parent?.parent) this.rotateRight(node.parent.parent);
}
} else {
const uncle = grandparent.left;
if (uncle && uncle.color === Color.RED) {
parent.color = Color.BLACK;
uncle.color = Color.BLACK;
grandparent.color = Color.RED;
node = grandparent;
} else {
if (node === parent.left) {
node = parent;
this.rotateRight(node);
}
if (node.parent) node.parent.color = Color.BLACK;
if (node.parent?.parent) node.parent.parent.color = Color.RED;
if (node.parent?.parent) this.rotateLeft(node.parent.parent);
}
}
}
if (this.root) this.root.color = Color.BLACK;
}
private fixDelete(x: RBNode<K, V> | null, parent: RBNode<K, V> | null): void {
// 标准 RB-Tree 删除修复（修复"双黑"问题）
let node = x;
let nodeParent = parent;
while ((!node || node.color === Color.BLACK) && node !== this.root) {
if (!nodeParent) break;
if (node === nodeParent.left) {
let sibling = nodeParent.right;
if (!sibling) break;
// Case 1: 兄弟是红色
if (sibling.color === Color.RED) {
sibling.color = Color.BLACK;
nodeParent.color = Color.RED;
this.rotateLeft(nodeParent);
sibling = nodeParent.right;
if (!sibling) break;
}
// Case 2: 兄弟的两个子节点都是黑色
const sibLeft = sibling.left;
const sibRight = sibling.right;
if ((!sibLeft || sibLeft.color === Color.BLACK) &&
(!sibRight || sibRight.color === Color.BLACK)) {
sibling.color = Color.RED;
node = nodeParent;
nodeParent = node.parent;
} else {
// Case 3: 兄弟右子黑色（左子红色）
if (!sibRight || sibRight.color === Color.BLACK) {
if (sibLeft) sibLeft.color = Color.BLACK;
sibling.color = Color.RED;
this.rotateRight(sibling);
sibling = nodeParent.right;
if (!sibling) break;
}
// Case 4: 兄弟右子红色
sibling.color = nodeParent.color;
nodeParent.color = Color.BLACK;
if (sibling.right) sibling.right.color = Color.BLACK;
this.rotateLeft(nodeParent);
node = this.root;
}
} else {
// 镜像：node 是父节点的右子
let sibling = nodeParent.left;
if (!sibling) break;
if (sibling.color === Color.RED) {
sibling.color = Color.BLACK;
nodeParent.color = Color.RED;
this.rotateRight(nodeParent);
sibling = nodeParent.left;
if (!sibling) break;
}
const sibLeft = sibling.left;
const sibRight = sibling.right;
if ((!sibLeft || sibLeft.color === Color.BLACK) &&
(!sibRight || sibRight.color === Color.BLACK)) {
sibling.color = Color.RED;
node = nodeParent;
nodeParent = node.parent;
} else {
if (!sibLeft || sibLeft.color === Color.BLACK) {
if (sibRight) sibRight.color = Color.BLACK;
sibling.color = Color.RED;
this.rotateLeft(sibling);
sibling = nodeParent.left;
if (!sibling) break;
}
sibling.color = nodeParent.color;
nodeParent.color = Color.BLACK;
if (sibling.left) sibling.left.color = Color.BLACK;
this.rotateRight(nodeParent);
node = this.root;
}
}
}
if (node) node.color = Color.BLACK;
}
private rotateLeft(x: RBNode<K, V>): void {
const y = x.right;
if (!y) return;
x.right = y.left;
if (y.left) y.left.parent = x;
y.parent = x.parent;
if (!x.parent) {
this.root = y;
} else if (x === x.parent.left) {
x.parent.left = y;
} else {
x.parent.right = y;
}
y.left = x;
x.parent = y;
}
private rotateRight(x: RBNode<K, V>): void {
const y = x.left;
if (!y) return;
x.left = y.right;
if (y.right) y.right.parent = x;
y.parent = x.parent;
if (!x.parent) {
this.root = y;
} else if (x === x.parent.right) {
x.parent.right = y;
} else {
x.parent.left = y;
}
y.right = x;
x.parent = y;
}
private _inorder(node: RBNode<K, V> | null, cb: (k: K, v: V) => void): void {
if (!node) return;
this._inorder(node.left, cb);
cb(node.key, node.value);
this._inorder(node.right, cb);
}
private _rangeScan(
node: RBNode<K, V> | null,
start: K,
end: K,
cb: (k: K, v: V) => void,
): void {
if (!node) return;
if (node.key > start) this._rangeScan(node.left, start, end, cb);
if (node.key >= start && node.key <= end) cb(node.key, node.value);
if (node.key < end) this._rangeScan(node.right, start, end, cb);
}
}
// ---------------------------------------------------------------------------
// MemTable
// ---------------------------------------------------------------------------
export class MemTable {
private tree: RedBlackTree<string, Record<string, unknown>>;
private _estimatedSize = 0;
private maxSize: number;
constructor(maxSize: number = 4 * 1024 * 1024) {
this.tree = new RedBlackTree();
this.maxSize = maxSize;
}
/** 插入或更新 */
put(key: string, value: Record<string, unknown>): void {
const oldSize = this.estimateEntrySize(key, this.tree.find(key));
const newSize = this.estimateEntrySize(key, value);
this.tree.insert(key, value);
this._estimatedSize += newSize - oldSize;
}
/** 获取 */
get(key: string): Record<string, unknown> | null {
return this.tree.find(key);
}
/** 删除 */
delete(key: string): boolean {
const oldVal = this.tree.find(key);
if (oldVal) {
this._estimatedSize -= this.estimateEntrySize(key, oldVal);
}
return this.tree.delete(key);
}
/** 是否应刷盘 */
shouldFlush(): boolean {
return this._estimatedSize >= this.maxSize;
}
/** 获取所有有序条目 */
getAllEntries(): [string, Record<string, unknown>][] {
return this.tree.getAllEntries();
}
/** 范围扫描 */
rangeScan(
startKey: string,
endKey: string,
): [string, Record<string, unknown>][] {
const entries: [string, Record<string, unknown>][] = [];
this.tree.rangeScan(startKey, endKey, (k, v) => entries.push([k, v]));
return entries;
}
/** 条目数 */
getEntryCount(): number {
return this.tree.size;
}
/** 估计大小（字节） */
getEstimatedSize(): number {
return this._estimatedSize;
}
/** 清空 */
clear(): void {
this.tree.clear();
this._estimatedSize = 0;
}
// -----------------------------------------------------------------------
// 内部
// -----------------------------------------------------------------------
private estimateEntrySize(key: string, value: Record<string, unknown> | null): number {
if (!value) return 0;
let size = key.length * 2; // UTF-16
for (const entry of Object.entries(value)) {
size += entry[0].length * 2;
const v = entry[1];
if (typeof v === 'string') size += v.length * 2;
else if (typeof v === 'number') size += 8;
else if (typeof v === 'boolean') size += 1;
else if (v === null || v === undefined) size += 1;
else size += 16; // rough estimate
}
return size;
}
}
src/engine/aria/index/merge_iterator.ts
+169 -192
View File
@@ -1,192 +1,169 @@
/**
* AriaEngine Merge Iterator — 多路归并迭代器
* @module engine/aria/index/merge_iterator
*
* 对多个有序 SSTable 或 MemTable 的结果进行归并去重（保留最新值）。
*/
// ---------------------------------------------------------------------------
// MergeIterator
// ---------------------------------------------------------------------------
export interface EntrySource {
/** 获取下一个条目，无更多时返回 null */
next(): [string, Record<string, unknown>] | null;
/** 重置迭代器 */
reset(): void;
}
/** 数组数据源的迭代器 */
export class ArrayEntrySource implements EntrySource {
private entries: [string, Record<string, unknown>][];
private index = 0;
constructor(entries: [string, Record<string, unknown>][]) {
this.entries = entries;
}
next(): [string, Record<string, unknown>] | null {
if (this.index >= this.entries.length) return null;
return this.entries[this.index++];
}
reset(): void {
this.index = 0;
}
}
/** 回调数据源的迭代器 */
export class CallbackEntrySource implements EntrySource {
private items: [string, Record<string, unknown>][] = [];
private index = 0;
private consumed = false;
/**
* @param producer 产生所有条目的回调
*/
constructor(producer: (cb: (key: string, value: Record<string, unknown>) => void) => void) {
producer((key, value) => this.items.push([key, value]));
}
next(): [string, Record<string, unknown>] | null {
if (this.index >= this.items.length) return null;
return this.items[this.index++];
}
reset(): void {
this.index = 0;
}
}
// ---------------------------------------------------------------------------
// Heap 节点（用于多路归并）
// ---------------------------------------------------------------------------
interface HeapNode {
key: string;
value: Record<string, unknown>;
sourceIndex: number;
}
/** 最小堆 */
class MinHeap {
private heap: HeapNode[] = [];
push(node: HeapNode): void {
this.heap.push(node);
this.bubbleUp(this.heap.length - 1);
}
pop(): HeapNode | null {
if (this.heap.length === 0) return null;
if (this.heap.length === 1) return this.heap.pop()!;
const result = this.heap[0];
this.heap[0] = this.heap.pop()!;
this.bubbleDown(0);
return result;
}
peek(): HeapNode | null {
return this.heap.length > 0 ? this.heap[0] : null;
}
get size(): number {
return this.heap.length;
}
private bubbleUp(idx: number): void {
while (idx > 0) {
const parent = Math.floor((idx - 1) / 2);
if (this.heap[idx].key >= this.heap[parent].key) break;
[this.heap[idx], this.heap[parent]] = [this.heap[parent], this.heap[idx]];
idx = parent;
}
}
private bubbleDown(idx: number): void {
const n = this.heap.length;
while (true) {
let smallest = idx;
const left = 2 * idx + 1;
const right = 2 * idx + 2;
if (left < n && this.heap[left].key < this.heap[smallest].key) smallest = left;
if (right < n && this.heap[right].key < this.heap[smallest].key) smallest = right;
if (smallest === idx) break;
[this.heap[idx], this.heap[smallest]] = [this.heap[smallest], this.heap[idx]];
idx = smallest;
}
}
}
// ---------------------------------------------------------------------------
// MergeIterator
// ---------------------------------------------------------------------------
/**
* 对多个有序数据源进行归并，重复 key 保留最新（后出现的）。
* 数据源按新鲜度排序：越新的数据源在下标越小（如 MemTable 在 SSTable 之前）。
*/
export class MergeIterator {
private sources: EntrySource[];
private heap: MinHeap;
constructor() {
this.sources = [];
this.heap = new MinHeap();
}
/** 添加数据源 */
addSource(source: EntrySource): void {
this.sources.push(source);
this.seedFromSource(this.sources.length - 1);
}
/** 获取下一个归并后的条目 */
next(): [string, Record<string, unknown>] | null {
if (this.heap.size === 0) return null;
const first = this.heap.pop()!;
const key = first.key;
let best = first;
// 刷新 first 来源的下一个值
this.seedFromSource(first.sourceIndex);
// 跳过重复 key：在多个来源中保留 sourceIndex 最小（最新）的条目
while (this.heap.peek() && this.heap.peek()!.key === key) {
const dup = this.heap.pop()!;
this.seedFromSource(dup.sourceIndex);
if (dup.sourceIndex < best.sourceIndex) {
best = dup;
}
}
return [best.key, best.value];
}
/** 耗尽管道，返回所有归并结果 */
drain(): [string, Record<string, unknown>][] {
const result: [string, Record<string, unknown>][] = [];
let entry = this.next();
while (entry) {
result.push(entry);
entry = this.next();
}
return result;
}
private seedFromSource(sourceIndex: number): void {
const entry = this.sources[sourceIndex].next();
if (entry) {
this.heap.push({
key: entry[0],
value: entry[1],
sourceIndex,
});
}
}
}
/**
* AriaEngine Merge Iterator — 多路归并迭代器
* @module engine/aria/index/merge_iterator
*
* 对多个有序 SSTable 或 MemTable 的结果进行归并去重（保留最新值）。
*/
// ---------------------------------------------------------------------------
// MergeIterator
// ---------------------------------------------------------------------------
export interface EntrySource {
/** 获取下一个条目，无更多时返回 null */
next(): [string, Record<string, unknown>] | null;
/** 重置迭代器 */
reset(): void;
}
/** 数组数据源的迭代器 */
export class ArrayEntrySource implements EntrySource {
private entries: [string, Record<string, unknown>][];
private index = 0;
constructor(entries: [string, Record<string, unknown>][]) {
this.entries = entries;
}
next(): [string, Record<string, unknown>] | null {
if (this.index >= this.entries.length) return null;
return this.entries[this.index++];
}
reset(): void {
this.index = 0;
}
}
// ---------------------------------------------------------------------------
// Heap 节点（用于多路归并）
// ---------------------------------------------------------------------------
interface HeapNode {
key: string;
value: Record<string, unknown>;
sourceIndex: number;
}
/** 最小堆 */
class MinHeap {
private heap: HeapNode[] = [];
push(node: HeapNode): void {
this.heap.push(node);
this.bubbleUp(this.heap.length - 1);
}
pop(): HeapNode | null {
if (this.heap.length === 0) return null;
if (this.heap.length === 1) return this.heap.pop()!;
const result = this.heap[0];
this.heap[0] = this.heap.pop()!;
this.bubbleDown(0);
return result;
}
peek(): HeapNode | null {
return this.heap.length > 0 ? this.heap[0] : null;
}
get size(): number {
return this.heap.length;
}
private bubbleUp(idx: number): void {
while (idx > 0) {
const parent = Math.floor((idx - 1) / 2);
if (this.heap[idx].key >= this.heap[parent].key) break;
[this.heap[idx], this.heap[parent]] = [this.heap[parent], this.heap[idx]];
idx = parent;
}
}
private bubbleDown(idx: number): void {
const n = this.heap.length;
while (true) {
let smallest = idx;
const left = 2 * idx + 1;
const right = 2 * idx + 2;
if (left < n && this.heap[left].key < this.heap[smallest].key) smallest = left;
if (right < n && this.heap[right].key < this.heap[smallest].key) smallest = right;
if (smallest === idx) break;
[this.heap[idx], this.heap[smallest]] = [this.heap[smallest], this.heap[idx]];
idx = smallest;
}
}
}
// ---------------------------------------------------------------------------
// MergeIterator
// ---------------------------------------------------------------------------
/**
* 对多个有序数据源进行归并，重复 key 保留最新（后出现的）。
* 数据源按新鲜度排序：越新的数据源在下标越小（如 MemTable 在 SSTable 之前）。
*/
export class MergeIterator {
private sources: EntrySource[];
private heap: MinHeap;
constructor() {
this.sources = [];
this.heap = new MinHeap();
}
/** 添加数据源 */
addSource(source: EntrySource): void {
this.sources.push(source);
this.seedFromSource(this.sources.length - 1);
}
/** 获取下一个归并后的条目 */
next(): [string, Record<string, unknown>] | null {
if (this.heap.size === 0) return null;
const first = this.heap.pop()!;
const key = first.key;
let best = first;
// 刷新 first 来源的下一个值
this.seedFromSource(first.sourceIndex);
// 跳过重复 key：在多个来源中保留 sourceIndex 最小（最新）的条目
while (this.heap.peek() && this.heap.peek()!.key === key) {
const dup = this.heap.pop()!;
this.seedFromSource(dup.sourceIndex);
if (dup.sourceIndex < best.sourceIndex) {
best = dup;
}
}
return [best.key, best.value];
}
/** 耗尽管道，返回所有归并结果 */
drain(): [string, Record<string, unknown>][] {
const result: [string, Record<string, unknown>][] = [];
let entry = this.next();
while (entry) {
result.push(entry);
entry = this.next();
}
return result;
}
private seedFromSource(sourceIndex: number): void {
const entry = this.sources[sourceIndex].next();
if (entry) {
this.heap.push({
key: entry[0],
value: entry[1],
sourceIndex,
});
}
}
}
src/engine/aria/index/sstable.ts
+15 -10
View File
@@ -8,6 +8,7 @@
import type { IndexEntry, SSTableMeta } from '../types';
import { BloomFilter } from './bloom';
import { crc32 } from '../crc32';
import { SSTABLE_MAGIC_V1, SSTABLE_MAGIC_V2 } from './sstable_builder';
// ---------------------------------------------------------------------------
@@ -23,6 +24,8 @@ export class SSTableReader {
private bloomFilter: BloomFilter | null = null;
/** 格式版本：1 = u16 长度字段（旧），2 = u32 长度字段（v0.4.4） */
private format: 1 | 2 = 2;
/** footer 中存储的 checksum（0 = 旧版无校验文件） */
private storedChecksum = 0;
constructor(data: Uint8Array, meta: SSTableMeta) {
this.data = data;
@@ -31,6 +34,17 @@ export class SSTableReader {
this.parseFooter();
}
/**
* 校验整文件 CRC-32（覆盖除 checksum 字段外的全部字节）。
* checksum === 0 表示旧版文件（v1 / v0.4.4 及更早的 v2），跳过校验返回 true（兼容）。
*/
verifyChecksum(): boolean {
if (this.storedChecksum === 0) return true;
if (this.data.byteLength < 4) return false;
const computed = crc32(this.data.subarray(0, this.data.byteLength - 4));
return computed === this.storedChecksum;
}
/** 长度字段宽度：v2 = 4 字节 u32，v1 = 2 字节 u16 */
private lenFieldSize(): number {
return this.format === 2 ? 4 : 2;
@@ -170,16 +184,6 @@ export class SSTableReader {
}
}
/** 获取元数据 */
getMeta(): SSTableMeta {
return this.meta;
}
/** 获取索引条目数 */
getIndexBlockCount(): number {
return this.indexEntries.length;
}
// -----------------------------------------------------------------------
// 内部
// -----------------------------------------------------------------------
@@ -207,6 +211,7 @@ export class SSTableReader {
const bloomSize = this.view.getUint32(footerOffset + 12, false);
const bloomHashCount = this.view.getUint32(footerOffset + 16, false);
this.entryCount = this.view.getUint32(footerOffset + 20, false);
this.storedChecksum = this.view.getUint32(footerOffset + 28, false);
// v0.4.1-fix: 完整性校验 — 索引块必须完全落在文件内，否则视为残缺文件跳过
if (indexOffset + 4 > this.data.byteLength || indexOffset + indexSize > this.data.byteLength) {
src/engine/aria/index/sstable_builder.ts
+10 -1
View File
@@ -34,6 +34,7 @@
*/
import { BloomFilter } from './bloom';
import { crc32 } from '../crc32';
import type { IndexEntry } from '../types';
/** v1 格式魔数（"SSTB"，u16 长度字段，兼容旧文件读取） */
@@ -137,7 +138,15 @@ export class SSTableBuilder {
view.setUint32(footerOffset + 16, bloomFilter.getHashCount(), false);
view.setUint32(footerOffset + 20, this.entries.length, false);
view.setUint32(footerOffset + 24, SSTABLE_MAGIC_V2, false);
view.setUint32(footerOffset + 28, 0, false); // checksum (simplified: 0)
// checksum 字段先写 0，全部字节就绪后计算整文件 CRC32 再回填
view.setUint32(footerOffset + 28, 0, false);
// v0.4.5: 真实 CRC-32 校验和 — 覆盖除自身（最后 4 字节）外的全部内容。
// checksum 永远非 0（计算结果为 0 时用 1 代替），读取端以 0 识别旧版无校验文件
const all = new Uint8Array(buf);
let checksum = crc32(all.subarray(0, all.byteLength - 4));
if (checksum === 0) checksum = 1;
view.setUint32(footerOffset + 28, checksum, false);
return {
sstableData: new Uint8Array(buf),
src/engine/aria/locks.ts
+92
View File
@@ -0,0 +1,92 @@
/**
* AriaEngine Database Lock — 多标签页独占锁（Web Locks API）
* @module engine/aria/locks
*
* v0.4.5: OPFS 等无事务后端缺乏多标签页并发协调，多个标签页同时打开同一库
* 会导致写竞态与数据损坏。用 Web Locks API（Chrome 69+ / Firefox 96+ / Safari 15.4+）
* 获取库级排他锁：
* - ifAvailable 模式：锁被其他标签页持有 → 立即抛 ARIA_LOCKED（不排队挂起）
* - 持锁期间回调挂起，close() 时释放
* - 浏览器不支持 navigator.locks → 返回 false（如实降级：无并发保护，文档注明）
*/
import { DatabaseError } from '../../constants';
/** Web Locks 锁名（库级排他） */
export function lockName(dbName: string): string {
return `metona-sqlark:${dbName}`;
}
export class DatabaseLock {
private acquired = false;
private supported = false;
private releaseResolve: (() => void) | null = null;
private releasePromise: Promise<void> | null = null;
/**
* 尝试获取独占锁。
* @returns true = 已持锁；false = 环境不支持 Web Locks（无并发保护，调用方可警告）
* @throws ARIA_LOCKED 锁被其他标签页持有
*/
async acquire(dbName: string): Promise<boolean> {
const nav = (globalThis as { navigator?: unknown }).navigator as {
locks?: { request: (name: string, opts: unknown, cb: (lock: { name: string } | null) => Promise<void> | void) => Promise<void> };
} | undefined;
const lockManager = nav?.locks;
if (!lockManager || typeof lockManager.request !== 'function') {
this.supported = false;
return false;
}
this.supported = true;
await new Promise<void>((resolve, reject) => {
// 注意：必须直接调用（不能解构 request —— LockManager 方法依赖 this 绑定）
const request = lockManager.request.bind(lockManager);
request(
lockName(dbName),
{ ifAvailable: true, mode: 'exclusive' },
async (lock) => {
if (!lock) {
reject(new DatabaseError(
`Database "${dbName}" is already open in another tab (locked)`,
'ARIA_LOCKED',
));
return;
}
this.acquired = true;
const releasePromise = new Promise<void>((res) => { this.releaseResolve = res; });
this.releasePromise = releasePromise;
// 锁已获取：acquire 返回（open 流程继续）
resolve();
// 回调挂起：保持锁直到 release() 触发
await releasePromise;
},
);
});
return true;
}
/** 释放锁（等待回调真正结束，保证锁已归还） */
async release(): Promise<void> {
if (!this.acquired) return;
if (this.releaseResolve) {
const res = this.releaseResolve;
const p = this.releasePromise!;
this.releaseResolve = null;
this.releasePromise = null;
res();
try { await p; } catch { /* 释放过程异常不阻塞 */ }
}
this.acquired = false;
}
/** 是否已持锁 */
isAcquired(): boolean {
return this.acquired;
}
/** 环境是否支持 Web Locks */
isSupported(): boolean {
return this.supported;
}
}
src/engine/aria/page/format.ts
+28 -164
View File
@@ -1,164 +1,28 @@
/**
* AriaEngine Page Format — 页面格式整合层
* @module engine/aria/page/format
*
* 组合 Header / Slot / Tuple 操作，提供统一的页面管理接口。
*/
import { PAGE_SIZE, PageType, type PageHandle } from '../types';
import {
initPageHeader,
getSlotCount,
} from './header';
import { allocateSlot, freeSlot, readSlotData } from './slot';
import { encodeTuple, decodeTuple } from './tuple';
// ---------------------------------------------------------------------------
// 页面创建
// ---------------------------------------------------------------------------
/** 创建一个新的空页面 */
export function createPage(pageId: number, type: PageType): PageHandle {
const data = new ArrayBuffer(PAGE_SIZE);
initPageHeader(data, pageId, type);
return {
pageId,
type,
data,
dirty: true,
pins: 0,
prev: null,
next: null,
lastAccess: Date.now(),
};
}
/** 从 ArrayBuffer 恢复页面句柄 */
export function pageFromBuffer(
pageId: number,
buffer: ArrayBuffer,
): PageHandle {
return {
pageId,
type: new DataView(buffer).getUint8(4) as PageType,
data: buffer,
dirty: false,
pins: 0,
prev: null,
next: null,
lastAccess: Date.now(),
};
}
// ---------------------------------------------------------------------------
// 行操作（页面级）
// ---------------------------------------------------------------------------
/**
* 插入一行到页面，返回 slot 索引，空间不足返回 -1。
*/
export function pageInsertRow(
page: PageHandle,
row: Record<string, unknown>,
columnOrder: string[],
columnTypes: Record<string, string>,
): number {
const encoded = encodeTuple(row, columnOrder, columnTypes);
const slotIdx = allocateSlot(page.data, encoded);
if (slotIdx >= 0) {
page.dirty = true;
page.lastAccess = Date.now();
}
return slotIdx;
}
/**
* 读取指定 slot 的行数据并解码。
*/
export function pageReadRow(
page: PageHandle,
slotIndex: number,
columnOrder: string[],
columnTypes: Record<string, string>,
): Record<string, unknown> | null {
const slotData = readSlotData(page.data, slotIndex);
if (!slotData) return null;
return decodeTuple(slotData, columnOrder, columnTypes);
}
/**
* 读取页面中的所有行。
*/
export function pageReadAllRows(
page: PageHandle,
columnOrder: string[],
columnTypes: Record<string, string>,
): Record<string, unknown>[] {
const rows: Record<string, unknown>[] = [];
const slotCount = getSlotCount(page.data);
for (let i = 0; i < slotCount; i++) {
const row = pageReadRow(page, i, columnOrder, columnTypes);
if (row) rows.push(row);
}
return rows;
}
/**
* 标记 slot 为已删除。
*/
export function pageDeleteRow(page: PageHandle, slotIndex: number): void {
freeSlot(page.data, slotIndex);
page.dirty = true;
page.lastAccess = Date.now();
}
/**
* 重写指定 slot 的行数据。
*/
export function pageUpdateRow(
page: PageHandle,
slotIndex: number,
row: Record<string, unknown>,
columnOrder: string[],
columnTypes: Record<string, string>,
): void {
// 先标记旧 slot 为删除
pageDeleteRow(page, slotIndex);
// 分配新 slot，可能会在不同位置
pageInsertRow(page, row, columnOrder, columnTypes);
// 注意：调用者需要自行维护 slot index → pk 的映射
}
// ---------------------------------------------------------------------------
// 校验和
// ---------------------------------------------------------------------------
/** 简单 CRC32（使用预先计算的查找表简化） */
export function computeChecksum(data: ArrayBuffer): number {
const view = new Uint8Array(data);
let hash = 0;
for (let i = 0; i < view.byteLength; i++) {
hash = ((hash << 5) - hash + view[i]) | 0;
}
return hash >>> 0;
}
/** 更新页面的校验和字段 */
export function updateChecksum(page: PageHandle): void {
// 先清零校验和字段
const view = new DataView(page.data);
view.setUint32(11, 0, false);
// 计算校验和
const cksum = computeChecksum(page.data);
view.setUint32(11, cksum, false);
}
/** 验证页面校验和 */
export function verifyChecksum(page: PageHandle): boolean {
const stored = new DataView(page.data).getUint32(11, false);
// 临时清零
new DataView(page.data).setUint32(11, 0, false);
const computed = computeChecksum(page.data);
new DataView(page.data).setUint32(11, stored, false);
return stored === computed;
}
/**
* AriaEngine Page Format — 页面创建
* @module engine/aria/page/format
*
* v0.4.5: 仅保留生产代码实际使用的页面创建逻辑。
* （Slot Directory / Tuple 编解码曾为行级页面存储设计，但从未接入 LSM 主路径，
* 属"宣称页面式但未实现"的半成品，已删除 —— SSTable 以 4KB 页面承载，
* 页面内容为原始字节切片，由 PageSSTableStore 管理）
*/
import { PAGE_SIZE, PageType, type PageHandle } from '../types';
import { initPageHeader } from './header';
/** 创建一个新的空页面 */
export function createPage(pageId: number, type: PageType): PageHandle {
const data = new ArrayBuffer(PAGE_SIZE);
initPageHeader(data, pageId, type);
return {
pageId,
type,
data,
dirty: true,
pins: 0,
prev: null,
next: null,
lastAccess: Date.now(),
};
}
src/engine/aria/page/header.ts
+37 -97
View File
@@ -1,97 +1,37 @@
/**
* AriaEngine Page Header — 页面头部编解码
* @module engine/aria/page/header
*/
import { PAGE_HEADER_SIZE, PageType, type PageHeader } from '../types';
// ---------------------------------------------------------------------------
// 编码
// ---------------------------------------------------------------------------
/**
* 将 PageHeader 编码写入 ArrayBuffer 的前 16 字节。
* 布局（大端序）:
* [0-3] page_id u32
* [4] type u8
* [5-6] free_start u16
* [7-8] free_end u16
* [9-10] slot_count u16
* [11-14] checksum u32
* [15] reserved u8
*/
export function encodePageHeader(header: PageHeader, buf: ArrayBuffer): void {
const view = new DataView(buf);
view.setUint32(0, header.pageId, false);
view.setUint8(4, header.type);
view.setUint16(5, header.freeStart, false);
view.setUint16(7, header.freeEnd, false);
view.setUint16(9, header.slotCount, false);
view.setUint32(11, header.checksum, false);
view.setUint8(15, 0); // reserved
}
/**
* 从 ArrayBuffer 解码 PageHeader。
*/
export function decodePageHeader(buf: ArrayBuffer): PageHeader {
const view = new DataView(buf);
return {
pageId: view.getUint32(0, false),
type: view.getUint8(4) as PageType,
freeStart: view.getUint16(5, false),
freeEnd: view.getUint16(7, false),
slotCount: view.getUint16(9, false),
checksum: view.getUint32(11, false),
};
}
/**
* 初始化新页面的 Header。
*/
export function initPageHeader(
buf: ArrayBuffer,
pageId: number,
type: PageType,
): void {
const view = new DataView(buf);
view.setUint32(0, pageId, false);
view.setUint8(4, type);
view.setUint16(5, PAGE_HEADER_SIZE, false); // freeStart = header 之后
view.setUint16(7, buf.byteLength, false); // freeEnd = 页面末尾
view.setUint16(9, 0, false); // slotCount = 0
view.setUint32(11, 0, false); // checksum = 0
view.setUint8(15, 0);
}
// ---------------------------------------------------------------------------
// 从 Buffer 中提取 Header 字段的辅助函数
// ---------------------------------------------------------------------------
export function getPageType(buf: ArrayBuffer): PageType {
return new DataView(buf).getUint8(4) as PageType;
}
export function getPageId(buf: ArrayBuffer): number {
return new DataView(buf).getUint32(0, false);
}
export function getSlotCount(buf: ArrayBuffer): number {
return new DataView(buf).getUint16(9, false);
}
export function getFreeStart(buf: ArrayBuffer): number {
return new DataView(buf).getUint16(5, false);
}
export function setFreeStart(buf: ArrayBuffer, val: number): void {
new DataView(buf).setUint16(5, val, false);
}
export function setFreeEnd(buf: ArrayBuffer, val: number): void {
new DataView(buf).setUint16(7, val, false);
}
export function setSlotCount(buf: ArrayBuffer, val: number): void {
new DataView(buf).setUint16(9, val, false);
}
/**
* AriaEngine Page Header — 页面头初始化
* @module engine/aria/page/header
*
* v0.4.5: 仅保留生产代码实际使用的 initPageHeader。
* 页面头部布局（大端序）:
* [0-3] page_id u32
* [4] type u8
* [5-6] free_start u16
* [7-8] free_end u16
* [9-10] slot_count u16
* [11-14] checksum u32
* [15] reserved u8
*
* 注：free_start/free_end/slot_count/checksum 字段为历史行级页面格式遗留，
* 页面化 SSTable 使用页面原始字节区（跳过头部），字段保留以维持 16 字节头部对齐。
*/
import { PAGE_HEADER_SIZE } from '../types';
/**
* 初始化新页面的 Header。
*/
export function initPageHeader(
buf: ArrayBuffer,
pageId: number,
type: number,
): void {
const view = new DataView(buf);
view.setUint32(0, pageId, false);
view.setUint8(4, type);
view.setUint16(5, PAGE_HEADER_SIZE, false); // freeStart = header 之后
view.setUint16(7, buf.byteLength, false); // freeEnd = 页面末尾
view.setUint16(9, 0, false); // slotCount = 0
view.setUint32(11, 0, false); // checksum = 0
view.setUint8(15, 0);
}
src/engine/aria/page/slot.ts
-184
View File
@@ -1,184 +0,0 @@
/**
* AriaEngine Slot Directory — 页面内行槽位管理
* @module engine/aria/page/slot
*
* Slot 目录从页面 Header 之后向下增长，每条记录 4 字节。
*/
import {
PAGE_HEADER_SIZE,
SLOT_ENTRY_SIZE,
PAGE_SIZE,
type SlotEntry,
} from '../types';
// ---------------------------------------------------------------------------
// Slot 读写
// ---------------------------------------------------------------------------
/**
* 读取 slot 号对应的 SlotEntry。
*/
export function getSlotEntry(buf: ArrayBuffer, slotIndex: number): SlotEntry {
const offset = PAGE_HEADER_SIZE + slotIndex * SLOT_ENTRY_SIZE;
const view = new DataView(buf);
return {
offset: view.getUint16(offset, false),
length: view.getUint16(offset + 2, false),
};
}
/**
* 写入 slot 号对应的 SlotEntry。
*/
export function setSlotEntry(
buf: ArrayBuffer,
slotIndex: number,
entry: SlotEntry,
): void {
const offset = PAGE_HEADER_SIZE + slotIndex * SLOT_ENTRY_SIZE;
const view = new DataView(buf);
view.setUint16(offset, entry.offset, false);
view.setUint16(offset + 2, entry.length, false);
}
/**
* 获取页面中所有 slot 条目。
*/
export function getAllSlots(
buf: ArrayBuffer,
slotCount: number,
): SlotEntry[] {
const entries: SlotEntry[] = [];
for (let i = 0; i < slotCount; i++) {
entries.push(getSlotEntry(buf, i));
}
return entries;
}
// ---------------------------------------------------------------------------
// Slot 空间计算
// ---------------------------------------------------------------------------
/** 获取 slot 目录占用的总字节数 */
export function getSlotDirectorySize(slotCount: number): number {
return slotCount * SLOT_ENTRY_SIZE;
}
/** 获取可用空闲空间（字节） */
export function getFreeSpace(buf: ArrayBuffer): number {
const view = new DataView(buf);
const freeStart = view.getUint16(5, false); // slot 区之后
const freeEnd = view.getUint16(7, false); // 数据区之前
return freeEnd - freeStart;
}
/** 检查是否有足够空间存放长度为 len 的行 */
export function hasEnoughSpace(buf: ArrayBuffer, len: number): boolean {
const slotCount = new DataView(buf).getUint16(9, false);
const neededSlotSize = (slotCount + 1) * SLOT_ENTRY_SIZE;
const freeStart = PAGE_HEADER_SIZE + neededSlotSize;
const freeEnd = new DataView(buf).getUint16(7, false);
return freeEnd - freeStart >= len;
}
// ---------------------------------------------------------------------------
// 插入 / 删除 slot
// ---------------------------------------------------------------------------
/**
* 在页面中分配一个 slot 并写入行数据。
* 返回分配的 slot 索引，失败返回 -1。
*/
export function allocateSlot(
buf: ArrayBuffer,
rowData: Uint8Array,
): number {
const view = new DataView(buf);
const slotCount = view.getUint16(9, false);
const neededSlotSpace = (slotCount + 1) * SLOT_ENTRY_SIZE;
const freeStart = PAGE_HEADER_SIZE + neededSlotSpace;
const freeEnd = view.getUint16(7, false);
if (freeEnd - freeStart < rowData.byteLength) {
return -1; // 空间不足
}
// 将数据放入页面底部
const dataOffset = freeEnd - rowData.byteLength;
const dest = new Uint8Array(buf, dataOffset, rowData.byteLength);
dest.set(rowData);
// 写入 slot 条目
setSlotEntry(buf, slotCount, { offset: dataOffset, length: rowData.byteLength });
// 更新 header
view.setUint16(5, freeStart, false); // freeStart
view.setUint16(7, dataOffset, false); // freeEnd
view.setUint16(9, slotCount + 1, false); // slotCount
return slotCount;
}
/**
* 从页面中删除指定 slot 的数据（标记为无效，offset 置 0）。
* 注：简化实现，不做 slot 压缩。
*/
export function freeSlot(buf: ArrayBuffer, slotIndex: number): void {
setSlotEntry(buf, slotIndex, { offset: 0, length: 0 });
}
/**
* 压缩页面槽位：移除已删除 slot，整理碎片空间。
* 将有效数据紧凑排列，释放空洞。
*/
export function compactSlots(buf: ArrayBuffer): number {
const view = new DataView(buf);
const slotCount = view.getUint16(9, false);
if (slotCount === 0) return 0;
// 收集有效 slot（offset>0 的）
const validSlots: { index: number; offset: number; length: number; data: Uint8Array }[] = [];
for (let i = 0; i < slotCount; i++) {
const entry = getSlotEntry(buf, i);
if (entry.offset > 0 && entry.length > 0) {
const data = new Uint8Array(buf, entry.offset, entry.length);
validSlots.push({ index: i, offset: entry.offset, length: entry.length, data: new Uint8Array(data) });
}
}
if (validSlots.length === slotCount) return 0; // 无碎片
// 从页面底部重新紧凑排列
let dataEnd = PAGE_SIZE;
const newSlots: { offset: number; length: number }[] = [];
for (let i = validSlots.length - 1; i >= 0; i--) {
const s = validSlots[i];
dataEnd -= s.length;
new Uint8Array(buf).set(s.data, dataEnd);
newSlots.unshift({ offset: dataEnd, length: s.length });
}
// 重写 slot directory
view.setUint16(9, validSlots.length, false); // slotCount
view.setUint16(7, dataEnd, false); // freeEnd
for (let i = 0; i < validSlots.length; i++) {
setSlotEntry(buf, i, newSlots[i]);
}
// 清除剩余 slot 条目
for (let i = validSlots.length; i < slotCount; i++) {
setSlotEntry(buf, i, { offset: 0, length: 0 });
}
return slotCount - validSlots.length; // 回收的 slot 数量
}
/**
* 读取指定 slot 的行数据。
*/
export function readSlotData(buf: ArrayBuffer, slotIndex: number): Uint8Array | null {
const entry = getSlotEntry(buf, slotIndex);
if (entry.offset === 0 || entry.length === 0) return null;
return new Uint8Array(buf, entry.offset, entry.length);
}
src/engine/aria/page/tuple.ts
-251
View File
@@ -1,251 +0,0 @@
/**
* AriaEngine Tuple Codec — 行数据的二进制编解码
* @module engine/aria/page/tuple
*
* 将 Record<string, unknown> 编码为紧凑的二进制格式。
*
* 格式:
* [null bitmap: ceil(colCount/8) bytes]
* [column 1 data]
* [column 2 data]
* ...
*
* 每列:
* type tag (u8) + data
* - STRING: [len: u16][UTF-8 bytes]
* - NUMBER: [f64: 8 bytes]
* - BOOLEAN: [u8: 1 byte]
* - DATE: [f64: 8 bytes] (epoch ms)
* - JSON: [len: u16][UTF-8 bytes]
* - NULL: (no data, just the tag)
*/
import { ColumnEncoding } from '../types';
// ---------------------------------------------------------------------------
// 编码
// ---------------------------------------------------------------------------
/**
* 将行数据编码为二进制字节数组。
* @param row 行数据
* @param columnOrder 列名顺序列表（决定编码顺序）
* @param columnTypes 列名 → FieldType 映射
*/
export function encodeTuple(
row: Record<string, unknown>,
columnOrder: string[],
columnTypes: Record<string, string>,
): Uint8Array {
// 先计算总大小
let size = 0;
const nullBitmapBytes = Math.ceil(columnOrder.length / 8);
size += nullBitmapBytes;
// 预计算每列编码后的字节
const colData: (Uint8Array | null)[] = [];
for (let i = 0; i < columnOrder.length; i++) {
const col = columnOrder[i];
const val = row[col];
const encoded = encodeColumn(val, columnTypes[col] ?? 'string');
colData.push(encoded);
if (encoded) {
size += 1 + encoded.byteLength; // tag + data
} else {
size += 1; // just the NULL tag
}
}
const buf = new Uint8Array(size);
const view = new DataView(buf.buffer, buf.byteOffset, buf.byteLength);
let offset = 0;
// Null bitmap
const nullBitmap = new Uint8Array(nullBitmapBytes);
for (let i = 0; i < columnOrder.length; i++) {
if (colData[i] === null) {
nullBitmap[Math.floor(i / 8)] |= (1 << (i % 8));
}
}
buf.set(nullBitmap, offset);
offset += nullBitmapBytes;
// Column data
for (let i = 0; i < columnOrder.length; i++) {
const encoded = colData[i];
if (encoded === null) {
view.setUint8(offset, ColumnEncoding.NULL);
offset += 1;
} else {
const tag = getEncodingTag(columnTypes[columnOrder[i]] ?? 'string');
view.setUint8(offset, tag);
offset += 1;
buf.set(encoded, offset);
offset += encoded.byteLength;
}
}
return buf;
}
/**
* 编码单个列的值。
*/
function encodeColumn(value: unknown, fieldType: string): Uint8Array | null {
if (value === null || value === undefined) return null;
switch (fieldType) {
case 'string': {
const str = String(value);
const encoder = new TextEncoder();
const bytes = encoder.encode(str);
const buf = new Uint8Array(2 + bytes.byteLength);
new DataView(buf.buffer).setUint16(0, bytes.byteLength, false);
buf.set(bytes, 2);
return buf;
}
case 'number': {
const buf = new ArrayBuffer(8);
new DataView(buf).setFloat64(0, Number(value), false);
return new Uint8Array(buf);
}
case 'boolean': {
return new Uint8Array([value ? 1 : 0]);
}
case 'date': {
const ts = value instanceof Date ? value.getTime() : new Date(String(value)).getTime();
const buf = new ArrayBuffer(8);
new DataView(buf).setFloat64(0, ts, false);
return new Uint8Array(buf);
}
case 'json': {
const str = JSON.stringify(value);
const encoder = new TextEncoder();
const bytes = encoder.encode(str);
const buf = new Uint8Array(2 + bytes.byteLength);
new DataView(buf.buffer).setUint16(0, bytes.byteLength, false);
buf.set(bytes, 2);
return buf;
}
default:
return null;
}
}
// ---------------------------------------------------------------------------
// 解码
// ---------------------------------------------------------------------------
/**
* 从二进制字节数组解码行数据。
* @returns 行数据，如果格式错误返回 null
*/
export function decodeTuple(
bytes: Uint8Array,
columnOrder: string[],
columnTypes: Record<string, string>,
): Record<string, unknown> | null {
try {
const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
let offset = 0;
const nullBitmapBytes = Math.ceil(columnOrder.length / 8);
if (offset + nullBitmapBytes > bytes.byteLength) return null;
offset += nullBitmapBytes;
const row: Record<string, unknown> = {};
for (let i = 0; i < columnOrder.length; i++) {
if (offset >= bytes.byteLength) break;
const tag = view.getUint8(offset);
offset += 1;
if (tag === ColumnEncoding.NULL) {
row[columnOrder[i]] = null;
continue;
}
const col = columnOrder[i];
const fType = columnTypes[col] ?? 'string';
const result = decodeColumnValue(bytes, offset, tag, fType);
if (result === null) return null;
row[col] = result.value;
offset = result.nextOffset;
}
return row;
} catch {
return null;
}
}
function decodeColumnValue(
bytes: Uint8Array,
offset: number,
tag: number,
_fieldType: string,
): { value: unknown; nextOffset: number } | null {
const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
switch (tag) {
case ColumnEncoding.STRING:
case ColumnEncoding.JSON: {
if (offset + 2 > bytes.byteLength) return null;
const len = view.getUint16(offset, false);
offset += 2;
if (offset + len > bytes.byteLength) return null;
const decoder = new TextDecoder();
const str = decoder.decode(bytes.slice(offset, offset + len));
return {
value: tag === ColumnEncoding.JSON ? JSON.parse(str) : str,
nextOffset: offset + len,
};
}
case ColumnEncoding.NUMBER: {
if (offset + 8 > bytes.byteLength) return null;
const val = view.getFloat64(offset, false);
return { value: val, nextOffset: offset + 8 };
}
case ColumnEncoding.BOOLEAN: {
if (offset >= bytes.byteLength) return null;
return { value: view.getUint8(offset) !== 0, nextOffset: offset + 1 };
}
case ColumnEncoding.DATE: {
if (offset + 8 > bytes.byteLength) return null;
const ts = view.getFloat64(offset, false);
return { value: new Date(ts).toISOString(), nextOffset: offset + 8 };
}
default:
return null;
}
}
// ---------------------------------------------------------------------------
// 辅助
// ---------------------------------------------------------------------------
function getEncodingTag(fieldType: string): ColumnEncoding {
switch (fieldType) {
case 'string': return ColumnEncoding.STRING;
case 'number': return ColumnEncoding.NUMBER;
case 'boolean': return ColumnEncoding.BOOLEAN;
case 'date': return ColumnEncoding.DATE;
case 'json': return ColumnEncoding.JSON;
default: return ColumnEncoding.NULL;
}
}
/** 获取列类型到编码标签的映射 */
export function getColumnEncodingMap(
columnOrder: string[],
columnTypes: Record<string, string>,
): Map<string, ColumnEncoding> {
const map = new Map<string, ColumnEncoding>();
for (const col of columnOrder) {
map.set(col, getEncodingTag(columnTypes[col] ?? 'string'));
}
return map;
}
src/engine/aria/store/backend.ts
+7
View File
@@ -23,6 +23,13 @@ export interface IStorageBackend {
read(key: string): Promise<ArrayBuffer | null>;
/** 写入数据块 */
write(key: string, data: ArrayBuffer): Promise<void>;
/**
* 追加写入（v0.4.5 WAL 分片用，可选）：
* - OPFS 后端实现真追加（createWritable keepExistingData + seek，O(chunk)）
* - 未实现的后端由调用方回退 read+write（EncryptedBackend 包装时整体重写保正确性）
* 语义：在 key 现有内容末尾追加 data；key 不存在时等同 write。
*/
append?(key: string, data: ArrayBuffer): Promise<void>;
/**
* 批量原子写入（v0.4.2-fix）：多个 key 在单个底层事务中提交，
* 中断时整体回滚，不留半写状态。WAL count 与记录同事务保证一致性。
src/engine/aria/store/encrypted_backend.ts
+233
View File
@@ -0,0 +1,233 @@
/**
* AriaEngine Encrypted Backend — 全库透明 AES-256-GCM 加密后端
* @module engine/aria/store/encrypted_backend
*
* 装饰器模式包装底层 IStorageBackend：
* - 写入时加密（每个 value 独立随机 IV：格式 [12B IV][AES-GCM ciphertext]）
* - 读取时解密（GCM 认证标签同时保证完整性）
* - WAL / SSTable / Schema / 元数据 全部密文存储（除密钥元数据本身）
*
* 密钥管理：
* - PBKDF2-SHA256（100000 迭代）从密码派生 AES-256-GCM 密钥
* - `__aria_keymeta` 明文保存 { salt, verifier }：
* - salt：PBKDF2 盐（重启后用同一密码重新派生密钥）
* - verifier：对固定明文加密的密文（打开时解密验证密码正确性）
* - 密码错误 → GCM 认证失败 → 抛 ARIA_DECRYPT_ERROR
*
* 依赖浏览器/Node 的 WebCrypto（jest.setup.js 已提供 polyfill）。
*/
import { DatabaseError } from '../../../constants';
import type { IStorageBackend } from './backend';
import { CryptoManager } from '../crypto';
const IV_LENGTH = 12;
const KEYMETA_KEY = '__aria_keymeta';
/** verifier 固定明文（无数据泄露风险） */
const VERIFIER_PLAIN = 'metona-sqlark-encryption-verifier';
// ---------------------------------------------------------------------------
// Base64 工具（浏览器 btoa/atob 与 Node Buffer 双环境）
// ---------------------------------------------------------------------------
function bytesToBase64(bytes: Uint8Array): string {
if (typeof Buffer !== 'undefined') {
return Buffer.from(bytes).toString('base64');
}
let bin = '';
for (let i = 0; i < bytes.byteLength; i++) bin += String.fromCharCode(bytes[i]);
return btoa(bin);
}
function base64ToBytes(b64: string): Uint8Array {
if (typeof Buffer !== 'undefined') {
return new Uint8Array(Buffer.from(b64, 'base64'));
}
const bin = atob(b64);
const bytes = new Uint8Array(bin.length);
for (let i = 0; i < bin.length; i++) bytes[i] = bin.charCodeAt(i);
return bytes;
}
// ---------------------------------------------------------------------------
// EncryptedBackend
// ---------------------------------------------------------------------------
export class EncryptedBackend implements IStorageBackend {
private crypto: CryptoManager = new CryptoManager();
private opened = false;
constructor(
private inner: IStorageBackend,
private password: string,
) {
if (!password || password.length === 0) {
throw new DatabaseError('Encryption password must not be empty', 'ARIA_ENCRYPT_CONFIG_ERROR');
}
}
/** 底层后端（测试/调试用） */
getInner(): IStorageBackend {
return this.inner;
}
/** 加密是否已初始化（打开并验证/创建密钥后为 true） */
isCryptoReady(): boolean {
return this.crypto.enabled;
}
async open(name: string): Promise<void> {
if (!this.inner.isOpen()) {
await this.inner.open(name);
}
const raw = await this.inner.read(KEYMETA_KEY);
if (raw) {
// 已有密钥元数据：用持久化 salt 重新派生并验证密码
let meta: { salt?: string; verifier?: string };
try {
meta = JSON.parse(new TextDecoder().decode(raw)) as { salt?: string; verifier?: string };
} catch {
throw new DatabaseError('Corrupted encryption key metadata', 'ARIA_DECRYPT_ERROR');
}
if (!meta.salt || !meta.verifier) {
throw new DatabaseError('Corrupted encryption key metadata', 'ARIA_DECRYPT_ERROR');
}
const salt = base64ToBytes(meta.salt);
await this.crypto.init(this.password, salt);
const verifierBytes = base64ToBytes(meta.verifier);
if (verifierBytes.byteLength <= IV_LENGTH) {
throw new DatabaseError('Corrupted encryption key metadata', 'ARIA_DECRYPT_ERROR');
}
const iv = verifierBytes.subarray(0, IV_LENGTH);
const ciphertext = verifierBytes.subarray(IV_LENGTH);
try {
await this.crypto.decryptPage(iv, ciphertext);
} catch {
// GCM 认证失败：密码错误（或密钥元数据被篡改）
throw new DatabaseError(
'Decryption failed: wrong password or corrupted key metadata',
'ARIA_DECRYPT_ERROR',
);
}
} else {
// 无 keymeta：若库中已存在其他数据 → 明文旧库或 keymeta 丢失，拒绝以加密模式打开
const keys = await this.inner.listKeys();
if (keys.some((k) => k !== KEYMETA_KEY)) {
throw new DatabaseError(
'Cannot open with encryption: existing database has no key metadata ' +
'(database was created without encryption, or key metadata was lost)',
'ARIA_ENCRYPT_CONFIG_ERROR',
);
}
// 新库：生成随机 salt + 派生密钥 + 写入 verifier
const salt = await this.crypto.init(this.password);
const enc = await this.crypto.encryptPage(new TextEncoder().encode(VERIFIER_PLAIN).buffer);
const verifier = new Uint8Array(IV_LENGTH + enc.data.byteLength);
verifier.set(enc.iv, 0);
verifier.set(new Uint8Array(enc.data), IV_LENGTH);
const keymeta = JSON.stringify({
salt: bytesToBase64(salt),
verifier: bytesToBase64(verifier),
});
await this.inner.write(KEYMETA_KEY, new TextEncoder().encode(keymeta).buffer);
}
this.opened = true;
}
async close(): Promise<void> {
await this.inner.close();
this.crypto.close();
this.opened = false;
}
isOpen(): boolean {
return this.opened;
}
async read(key: string): Promise<ArrayBuffer | null> {
this.ensureReady();
const raw = await this.inner.read(key);
if (raw === null) return null;
return this.decrypt(raw);
}
async write(key: string, data: ArrayBuffer): Promise<void> {
this.ensureReady();
await this.inner.write(key, await this.encrypt(data));
}
async writeMany(entries: Record<string, ArrayBuffer>): Promise<void> {
this.ensureReady();
const encrypted: Record<string, ArrayBuffer> = {};
for (const [key, data] of Object.entries(entries)) {
encrypted[key] = await this.encrypt(data);
}
await this.inner.writeMany(encrypted);
}
async delete(key: string): Promise<void> {
this.ensureReady();
await this.inner.delete(key);
}
async deleteMany(keys: string[]): Promise<void> {
this.ensureReady();
await this.inner.deleteMany(keys);
}
async listKeys(): Promise<string[]> {
this.ensureReady();
return this.inner.listKeys();
}
async exists(key: string): Promise<boolean> {
this.ensureReady();
return this.inner.exists(key);
}
async clear(): Promise<void> {
// 清空全部数据但保留密钥元数据 —— clearAll 语义：库本身保留，密码不失效
const keys = await this.inner.listKeys();
const toDelete = keys.filter((k) => k !== KEYMETA_KEY);
if (toDelete.length > 0) {
await this.inner.deleteMany(toDelete);
}
}
// -----------------------------------------------------------------------
// 内部
// -----------------------------------------------------------------------
private ensureReady(): void {
if (!this.opened) throw new DatabaseError('EncryptedBackend not opened', 'ARIA_DB_NOT_OPEN');
if (!this.crypto.enabled) throw new DatabaseError('EncryptedBackend key not initialized', 'ARIA_DECRYPT_ERROR');
}
/** 加密单块数据：[IV(12)][ciphertext] */
private async encrypt(data: ArrayBuffer): Promise<ArrayBuffer> {
const enc = await this.crypto.encryptPage(data);
const out = new ArrayBuffer(IV_LENGTH + enc.data.byteLength);
const outBytes = new Uint8Array(out);
outBytes.set(enc.iv, 0);
outBytes.set(new Uint8Array(enc.data), IV_LENGTH);
return out;
}
/** 解密单块数据（GCM 认证失败抛错） */
private async decrypt(data: ArrayBuffer): Promise<ArrayBuffer> {
const bytes = new Uint8Array(data);
if (bytes.byteLength <= IV_LENGTH) {
throw new DatabaseError('Corrupted encrypted data block (too short)', 'ARIA_DECRYPT_ERROR');
}
const iv = bytes.subarray(0, IV_LENGTH);
const ciphertext = bytes.subarray(IV_LENGTH);
try {
return await this.crypto.decryptPage(iv, ciphertext);
} catch (error) {
if (error instanceof DatabaseError) throw error;
throw new DatabaseError('Decryption failed (data corrupted or wrong key)', 'ARIA_DECRYPT_ERROR', error);
}
}
}
src/engine/aria/store/file_manager.ts
+104 -111
View File
@@ -1,111 +1,104 @@
/**
* AriaEngine File Manager — 页面文件管理 + PageIO 实现
* @module engine/aria/store/file_manager
*
* 负责管理页面文件的生命周期：分配/释放页面 ID，读写页面。
*/
import type { IStorageBackend } from './backend';
import type { PageIO } from '../buffer/pool';
import { PAGE_SIZE, PageType } from '../types';
import { initPageHeader } from '../page/header';
// ---------------------------------------------------------------------------
// FileManager (implements PageIO)
// ---------------------------------------------------------------------------
export class FileManager implements PageIO {
private backend: IStorageBackend;
private nextPageId = 0;
private metaLoaded = false;
private dbName = '';
constructor(backend: IStorageBackend) {
this.backend = backend;
}
/** 初始化：从存储中读取元数据 */
async init(dbName: string): Promise<void> {
this.dbName = dbName;
const meta = await this.backend.read('__aria_meta');
if (meta && meta instanceof ArrayBuffer && meta.byteLength >= 4) {
const view = new DataView(meta);
this.nextPageId = view.getUint32(0, false);
} else {
this.nextPageId = 1;
await this.saveMeta();
}
this.metaLoaded = true;
}
// ---- PageIO ----
async readPage(pageId: number): Promise<ArrayBuffer | null> {
const key = `pg_${pageId}`;
const data = await this.backend.read(key);
if (!data) {
// 第一次访问：创建新页面
return this.createEmptyPage(pageId, PageType.DATA);
}
// 确保大小正确
if (data.byteLength < PAGE_SIZE) {
const padded = new ArrayBuffer(PAGE_SIZE);
new Uint8Array(padded).set(new Uint8Array(data));
return padded;
}
return data;
}
async writePage(pageId: number, data: ArrayBuffer): Promise<void> {
const key = `pg_${pageId}`;
await this.backend.write(key, data);
}
async allocatePageId(): Promise<number> {
const id = this.nextPageId++;
await this.saveMeta();
return id;
}
async freePageId(_pageId: number): Promise<void> {
// 简化实现：不回收 pageId
const key = `pg_${_pageId}`;
await this.backend.delete(key);
}
// ---- 表页面分配 ----
/**
* 分配一个新的表元数据页面。
*/
async allocateTableRootPage(): Promise<number> {
const pageId = await this.allocatePageId();
const data = new ArrayBuffer(PAGE_SIZE);
initPageHeader(data, pageId, PageType.META);
await this.writePage(pageId, data);
return pageId;
}
// ---- 辅助 ----
private async saveMeta(): Promise<void> {
const buf = new ArrayBuffer(8);
new DataView(buf).setUint32(0, this.nextPageId, false);
await this.backend.write('__aria_meta', buf);
}
private createEmptyPage(pageId: number, type: PageType): ArrayBuffer {
const buf = new ArrayBuffer(PAGE_SIZE);
initPageHeader(buf, pageId, type);
return buf;
}
/** 清空所有数据 */
async clearAll(): Promise<void> {
await this.backend.clear();
this.nextPageId = 1;
await this.saveMeta();
}
}
/**
* AriaEngine File Manager — 页面文件管理 + PageIO 实现
* @module engine/aria/store/file_manager
*
* 负责管理页面文件的生命周期：分配/释放页面 ID，读写页面。
*/
import type { IStorageBackend } from './backend';
import type { PageIO } from '../buffer/pool';
import { PAGE_SIZE, PageType } from '../types';
import { initPageHeader } from '../page/header';
// ---------------------------------------------------------------------------
// FileManager (implements PageIO)
// ---------------------------------------------------------------------------
export class FileManager implements PageIO {
private backend: IStorageBackend;
private nextPageId = 0;
private metaLoaded = false;
private dbName = '';
constructor(backend: IStorageBackend) {
this.backend = backend;
}
/** 初始化：从存储中读取元数据 */
async init(dbName: string): Promise<void> {
this.dbName = dbName;
const meta = await this.backend.read('__aria_meta');
if (meta && meta instanceof ArrayBuffer && meta.byteLength >= 4) {
const view = new DataView(meta);
this.nextPageId = view.getUint32(0, false);
} else {
this.nextPageId = 1;
await this.saveMeta();
}
this.metaLoaded = true;
}
// ---- PageIO ----
async readPage(pageId: number): Promise<ArrayBuffer | null> {
const key = `pg_${pageId}`;
const data = await this.backend.read(key);
if (!data) {
// v0.4.5: 页面总是先分配（allocatePageId 持久化）后写入 —— 读取缺失页面视为损坏
// （此前返回空页面会静默掩盖页面丢失，页面化 SSTable 依赖 null 触发自愈清理）
return null;
}
// 确保大小正确
if (data.byteLength < PAGE_SIZE) {
const padded = new ArrayBuffer(PAGE_SIZE);
new Uint8Array(padded).set(new Uint8Array(data));
return padded;
}
return data;
}
async writePage(pageId: number, data: ArrayBuffer): Promise<void> {
const key = `pg_${pageId}`;
await this.backend.write(key, data);
}
async allocatePageId(): Promise<number> {
const id = this.nextPageId++;
await this.saveMeta();
return id;
}
/** v0.4.5: 批量分配页面 ID（一次 meta 持久化，避免页面化 SSTable 保存时逐页写 meta） */
async allocatePageIds(count: number): Promise<number[]> {
if (count <= 0) return [];
const ids: number[] = [];
const start = this.nextPageId;
this.nextPageId += count;
for (let i = 0; i < count; i++) ids.push(start + i);
await this.saveMeta();
return ids;
}
async freePageId(_pageId: number): Promise<void> {
// 简化实现：不回收 pageId
const key = `pg_${_pageId}`;
await this.backend.delete(key);
}
// ---- 辅助 ----
private async saveMeta(): Promise<void> {
const buf = new ArrayBuffer(8);
new DataView(buf).setUint32(0, this.nextPageId, false);
await this.backend.write('__aria_meta', buf);
}
/** 清空所有数据 */
async clearAll(): Promise<void> {
await this.backend.clear();
this.nextPageId = 1;
await this.saveMeta();
}
}
src/engine/aria/store/opfs_backend.ts
+72 -9
View File
@@ -5,10 +5,23 @@
* 零外部依赖，纯浏览器文件系统 API。
* 每个 key 对应 OPFS 目录下的一个二进制文件。
*
* 浏览器要求：Chrome 102+ / Edge 102+
* 原子性与一致性保证（v0.4.5 固化）：
* - 单文件 write/append：createWritable 为 copy-on-write —— close 前崩溃旧文件保持不变，
* close 后原子替换（单文件写入原子）
* - 多文件 writeMany/deleteMany：OPFS 无跨文件事务，串行逐个落盘；调用方（WAL 分片）
* 已改为单文件语义，多键操作仅用于一次性的 schema/meta 写入
* - 所有写操作串行队列化（同源同进程顺序一致）；单次任务失败不中断队列链，
* 错误如实返回给该次调用的调用方
* - close() 等待写队列排空后再释放目录句柄（杜绝 close 后挂起写丢失/读旧数据）
* - open() 自动清理崩溃残留临时文件（Chromium createWritable 的 .crswap 等）
*
* 浏览器要求：Chrome 102+ / Edge 102+（Safari 15.2+ / Firefox 111+ 支持基础 OPFS）
*/
import type { IStorageBackend } from './backend';
/** 崩溃残留临时文件后缀（createWritable 底层实现可能留下） */
const STALE_SUFFIXES = ['.crswap', '.tmp'];
export class OPFSBackend implements IStorageBackend {
private root: FileSystemDirectoryHandle | null = null;
private dbDir: FileSystemDirectoryHandle | null = null;
@@ -19,9 +32,15 @@ export class OPFSBackend implements IStorageBackend {
this.dbName = name;
this.root = await navigator.storage.getDirectory();
this.dbDir = await this.root.getDirectoryHandle(name, { create: true });
// v0.4.5: 清理崩溃残留临时文件（不阻塞打开）
await this.cleanupStaleFiles();
}
async close(): Promise<void> {
// v0.4.5-fix: 等待所有挂起写完成（否则 close 后挂起写被静默丢弃/读旧数据）
try {
await this.writeQueue;
} catch { /* 写失败已返回给调用方 */ }
this.dbDir = null;
this.root = null;
}
@@ -30,6 +49,23 @@ export class OPFSBackend implements IStorageBackend {
return this.dbDir !== null;
}
/** 清理崩溃残留的临时文件（open 时自动调用，repair 也可调用） */
async cleanupStaleFiles(): Promise<void> {
if (!this.dbDir) return;
try {
const dir = this.dbDir as any;
const stale: string[] = [];
for await (const [name] of dir.entries()) {
if (STALE_SUFFIXES.some((s) => name.endsWith(s))) {
stale.push(name);
}
}
for (const name of stale) {
try { await this.dbDir!.removeEntry(name); } catch { /* ignore */ }
}
} catch { /* 清理失败不阻塞 */ }
}
async read(key: string): Promise<ArrayBuffer | null> {
if (!this.dbDir) return null;
try {
@@ -41,21 +77,45 @@ export class OPFSBackend implements IStorageBackend {
}
}
/**
* 单文件原子写：createWritable 为 copy-on-write，close 后原子替换；
* 写入期间崩溃 → 旧文件保持（原子性由浏览器 OPFS 实现保证）。
*/
async write(key: string, data: ArrayBuffer): Promise<void> {
if (!this.dbDir) return;
this.writeQueue = this.writeQueue.then(async () => {
const run = this.writeQueue.then(async () => {
const fh = await this.dbDir!.getFileHandle(key, { create: true });
const writable = await fh.createWritable();
await writable.write(data);
await writable.close();
});
return this.writeQueue;
// v0.4.5-fix: 单次任务失败不中断队列链（错误仍返回给本次调用方）
this.writeQueue = run.then(() => undefined, () => undefined);
return run;
}
/**
* v0.4.5: 真追加写 — createWritable(keepExistingData) + seek 到文件末尾。
* 单文件 COW 原子（close 前崩溃旧文件保持），无需读旧内容即实现 O(chunk) 追加
* （WAL 分片高频写入用）。
*/
async append(key: string, data: ArrayBuffer): Promise<void> {
if (!this.dbDir) return;
const run = this.writeQueue.then(async () => {
const fh = await this.dbDir!.getFileHandle(key, { create: true });
const existing = await fh.getFile();
const writable = await fh.createWritable({ keepExistingData: true });
await writable.write({ type: 'write', position: existing.size, data });
await writable.close();
});
this.writeQueue = run.then(() => undefined, () => undefined);
return run;
}
/** v0.4.2-fix: 批量写入 — 串行队列内逐个落盘（OPFS 无跨文件事务，顺序保证一致） */
async writeMany(entries: Record<string, ArrayBuffer>): Promise<void> {
if (!this.dbDir) return;
this.writeQueue = this.writeQueue.then(async () => {
const run = this.writeQueue.then(async () => {
for (const [key, data] of Object.entries(entries)) {
const fh = await this.dbDir!.getFileHandle(key, { create: true });
const writable = await fh.createWritable();
@@ -63,26 +123,29 @@ export class OPFSBackend implements IStorageBackend {
await writable.close();
}
});
return this.writeQueue;
this.writeQueue = run.then(() => undefined, () => undefined);
return run;
}
async delete(key: string): Promise<void> {
if (!this.dbDir) return;
this.writeQueue = this.writeQueue.then(async () => {
const run = this.writeQueue.then(async () => {
try { await this.dbDir!.removeEntry(key); } catch { /* ignore */ }
});
return this.writeQueue;
this.writeQueue = run.then(() => undefined, () => undefined);
return run;
}
/** v0.4.2-fix: 批量删除 — 串行队列内逐个删除 */
async deleteMany(keys: string[]): Promise<void> {
if (!this.dbDir) return;
this.writeQueue = this.writeQueue.then(async () => {
const run = this.writeQueue.then(async () => {
for (const key of keys) {
try { await this.dbDir!.removeEntry(key); } catch { /* ignore */ }
}
});
return this.writeQueue;
this.writeQueue = run.then(() => undefined, () => undefined);
return run;
}
async listKeys(): Promise<string[]> {
src/engine/aria/store/page_sstable_store.ts
+95
View File
@@ -0,0 +1,95 @@
/**
* AriaEngine Page SSTable Store — SSTable 页面化物理存储
* @module engine/aria/store/page_sstable_store
*
* v0.4.5: 让 BufferPool/FileManager 真正接入 LSM 读写路径。
* SSTable 不再整体存为一个 backend value，而是切分为 4KB 页面：
* - 页面由 FileManager 分配 pageId，经 BufferPool 缓存（LRU 驱逐，脏页写回）
* - save 语义 = 数据已落盘：页面写入后逐个 flushPage（await 底层写）才返回，
* 保证 WAL checkpoint（截断）前 SSTable 数据真实持久化
* - 页面 ID 列表经 SSTableMeta.pageIds 持久化；旧 meta（无 pageIds）走整 value 读取
*
* 与 LSM 的 sstableCache（整文件 LRU）双层缓存并存：
* - BufferPool 缓存物理页面（跨 SSTable 共享、受 bufferPoolPages 上限约束）
* - LSM 缓存解析后的整文件字节（查询热点复用）
*/
import type { FileManager } from './file_manager';
import type { BufferPool } from '../buffer/pool';
import type { PageHandle } from '../types';
import { PAGE_SIZE, PageType } from '../types';
export class PageSSTableStore {
/** SSTable id → 页面 ID 列表（save 时记录，saveMeta 时注入 meta） */
private pageIds = new Map<number, number[]>();
constructor(
private fileManager: FileManager,
private bufferPool: BufferPool,
) {}
/** 保存数据：切页 → 写入 BufferPool → 逐页落盘 → 记录 pageIds */
async save(id: number, data: Uint8Array): Promise<void> {
const pageCount = Math.max(1, Math.ceil(data.byteLength / PAGE_SIZE));
const handles: PageHandle[] = await this.bufferPool.newPages(pageCount, PageType.DATA);
const ids: number[] = [];
for (let i = 0; i < pageCount; i++) {
const page = handles[i];
ids.push(page.pageId);
const dest = new Uint8Array(page.data);
dest.fill(0); // 清空（最后一页可能不满）
const slice = data.subarray(i * PAGE_SIZE, Math.min((i + 1) * PAGE_SIZE, data.byteLength));
dest.set(slice, 0);
page.dirty = true;
// save 语义 = 已持久化：立即落盘（WAL checkpoint 截断依赖此保证）
await this.bufferPool.flushPage(page.pageId);
this.bufferPool.unpin(page);
}
this.pageIds.set(id, ids);
}
/** 获取指定 SSTable 的页面 ID 列表（saveMeta 注入用） */
getPageIds(id: number): number[] | undefined {
return this.pageIds.get(id);
}
/**
* 按页面 ID 列表读取并拼接为完整字节流。
* @param totalSize SSTable 真实大小（meta 持久化）——最后一页可能有 0 填充，按真实大小截断
* @returns 缺失页面/读取失败返回 null（调用方视为损坏并清理）
*/
async load(id: number, pageIds: number[], totalSize: number): Promise<Uint8Array | null> {
if (pageIds.length === 0) return null;
const chunks: Uint8Array[] = [];
for (const pageId of pageIds) {
const page = await this.bufferPool.getPage(pageId);
if (!page) return null;
// 立即复制（后续驱逐安全）
chunks.push(new Uint8Array(page.data));
this.bufferPool.unpin(page);
}
const total = chunks.reduce((s, c) => s + c.byteLength, 0);
const out = new Uint8Array(Math.min(total, totalSize));
let off = 0;
for (const c of chunks) {
const take = Math.min(c.byteLength, out.byteLength - off);
if (take <= 0) break;
out.set(c.subarray(0, take), off);
off += take;
}
this.pageIds.delete(id);
return out;
}
/** 释放页面（删除物理页面文件 + 移出 BufferPool） */
async delete(id: number, pageIds: number[]): Promise<void> {
for (const pageId of pageIds) {
this.bufferPool.removePage(pageId);
try {
await this.fileManager.freePageId(pageId);
} catch { /* 清理失败不阻塞 */ }
}
this.pageIds.delete(id);
}
}
src/engine/aria/transaction/mvcc.ts
+181 -284
View File
@@ -1,284 +1,181 @@
/**
* AriaEngine MVCC — 多版本并发控制
* @module engine/aria/transaction/mvcc
*
* 实现快照隔离 (Snapshot Isolation)。
* 每个事务看到数据库在事务开始时的快照。
*/
import type { RowVersion, TxnEntry } from '../types';
import { TransactionState } from '../types';
// ---------------------------------------------------------------------------
// MVCCManager
// ---------------------------------------------------------------------------
export class MVCCManager {
/** 所有行版本的存储：tableName.key → 版本链 */
private versionStore: Map<string, RowVersion[]> = new Map();
/** 活跃事务表：txnId → TxnEntry */
private activeTxns: Map<number, TxnEntry> = new Map();
/** 事务 ID 计数器 */
private nextTxnId = 1;
/** 全局提交序列号（用于可见性判断） */
private globalCommitLsn = 0;
/**
* v0.4.2-fix: 每个事务写入的 tableKey 集合 —
* commit/rollback 只遍历本事务写过的 key，避免全库版本链扫描（大表事务 O(N) → O(写入数)）
*/
private txnWriteKeys: Map<number, Set<string>> = new Map();
// =======================================================================
// 事务管理
// =======================================================================
/** 开始一个事务，返回事务 ID */
beginTransaction(): number {
const txnId = this.nextTxnId++;
this.activeTxns.set(txnId, {
txnId,
state: TransactionState.ACTIVE,
snapshotLsn: this.globalCommitLsn,
startTime: Date.now(),
});
this.txnWriteKeys.set(txnId, new Set());
return txnId;
}
/** 提交事务 */
commitTransaction(txnId: number): void {
const txn = this.activeTxns.get(txnId);
if (!txn) throw new Error(`Transaction ${txnId} not found`);
txn.state = TransactionState.COMMITTED;
this.globalCommitLsn++;
// v0.4.2-fix: 仅标记本事务写入的版本（此前遍历全库 versionStore）
const writeKeys = this.txnWriteKeys.get(txnId);
if (writeKeys) {
for (const tableKey of writeKeys) {
const versions = this.versionStore.get(tableKey);
if (!versions) continue;
for (const version of versions) {
if (version.txnId === txnId) {
version.committed = true;
}
}
}
}
// 清理已提交事务的记录
this.activeTxns.delete(txnId);
this.txnWriteKeys.delete(txnId);
}
/** 回滚事务 */
rollbackTransaction(txnId: number): void {
const txn = this.activeTxns.get(txnId);
if (!txn) throw new Error(`Transaction ${txnId} not found`);
txn.state = TransactionState.ABORTED;
// v0.4.2-fix: 仅移除本事务写入的版本（此前遍历全库 versionStore）
const writeKeys = this.txnWriteKeys.get(txnId);
if (writeKeys) {
for (const tableKey of writeKeys) {
const versions = this.versionStore.get(tableKey);
if (!versions) continue;
const filtered = versions.filter((v) => v.txnId !== txnId);
if (filtered.length === 0) {
this.versionStore.delete(tableKey);
} else {
this.versionStore.set(tableKey, filtered);
}
}
}
this.activeTxns.delete(txnId);
this.txnWriteKeys.delete(txnId);
}
/** 检查事务是否活跃 */
isActive(txnId: number): boolean {
const txn = this.activeTxns.get(txnId);
return txn !== undefined && txn.state === TransactionState.ACTIVE;
}
// =======================================================================
// 版本读写
// =======================================================================
/**
* 写入一行（创建新版本）。
*/
writeVersion(
tableName: string,
key: string,
data: Record<string, unknown>,
txnId: number,
): void {
const tableKey = `${tableName}.${key}`;
const versions = this.versionStore.get(tableKey) ?? [];
const newVersion: RowVersion = {
txnId,
data,
prevVersion: versions.length > 0 ? versions[versions.length - 1] : null,
committed: false,
};
versions.push(newVersion);
this.versionStore.set(tableKey, versions);
// v0.4.2-fix: 记录本事务写过的 key（commit/rollback 精准清理）
this.txnWriteKeys.get(txnId)?.add(tableKey);
}
/**
* 读取一行（对指定事务可见的最新版本）。
*/
readVersion(
tableName: string,
key: string,
txnId: number,
): Record<string, unknown> | null {
const txn = this.activeTxns.get(txnId);
if (!txn) return null;
const tableKey = `${tableName}.${key}`;
const versions = this.versionStore.get(tableKey);
if (!versions || versions.length === 0) return null;
// 从最新版本向前遍历
for (let i = versions.length - 1; i >= 0; i--) {
const version = versions[i];
// 1. 如果是当前事务写入的（未提交），可见
if (version.txnId === txnId) {
return version.data;
}
// 2. 如果是已提交的版本，且在快照 LSN 之前提交，可见
if (version.committed) {
// 简化：所有已提交版本都可见
return version.data;
}
// 3. 其他事务的未提交版本，不可见，继续找更早的版本
}
return null;
}
/**
* 删除一行（创建墓碑版本）。
*/
deleteVersion(tableName: string, key: string, txnId: number): void {
this.writeVersion(tableName, key, { __mvcc_tombstone: true } as unknown as Record<string, unknown>, txnId);
}
/**
* 获取所有行的最新已提交版本（用于非事务读取）。
*/
getLatestCommittedVersions(
tableName: string,
): Record<string, Record<string, unknown>> {
const result: Record<string, Record<string, unknown>> = {};
for (const [tableKey, versions] of this.versionStore) {
if (!tableKey.startsWith(`${tableName}.`)) continue;
const key = tableKey.slice(tableName.length + 1);
for (let i = versions.length - 1; i >= 0; i--) {
const version = versions[i];
if (version.committed && !(version.data as unknown as Record<string, unknown>).__mvcc_tombstone) {
result[key] = version.data;
break;
}
}
}
return result;
}
/**
* v0.3.3: 丢弃指定事务的所有版本记录，但保留事务登记（Savepoint 回滚用）。
* 快照数据由调用方（引擎 txnSnapshot）负责恢复。
* v0.4.2-fix: 仅遍历本事务写过的 key（此前全库扫描）。
*/
discardVersions(txnId: number): void {
const writeKeys = this.txnWriteKeys.get(txnId);
if (!writeKeys) return;
for (const tableKey of writeKeys) {
const versions = this.versionStore.get(tableKey);
if (!versions) continue;
const filtered = versions.filter((v) => v.txnId !== txnId);
if (filtered.length === 0) {
this.versionStore.delete(tableKey);
} else {
this.versionStore.set(tableKey, filtered);
}
}
}
/**
* 清理过旧版本（GC）。
* 保留每个 key 的最新 N 个已提交版本。
*/
gc(maxVersionsPerKey: number = 100): void {
for (const [tableKey, versions] of this.versionStore) {
if (versions.length <= maxVersionsPerKey) continue;
// 保留最新的 maxVersionsPerKey 个版本
const pruned = versions.slice(versions.length - maxVersionsPerKey);
this.versionStore.set(tableKey, pruned);
}
}
/**
* 获取所有未提交事务中的 key 列表。
*/
getActiveWriteKeys(tableName: string, txnId: number): Set<string> {
const keys = new Set<string>();
const prefix = `${tableName}.`;
for (const [tableKey, versions] of this.versionStore) {
if (!tableKey.startsWith(prefix)) continue;
const latestVersion = versions[versions.length - 1];
if (latestVersion.txnId === txnId && !latestVersion.committed) {
keys.add(tableKey.slice(prefix.length));
}
}
return keys;
}
/**
* 清理指定表的所有版本。
*/
clearTable(tableName: string): void {
const prefix = `${tableName}.`;
for (const [tableKey] of this.versionStore) {
if (tableKey.startsWith(prefix)) {
this.versionStore.delete(tableKey);
}
}
}
/**
* 获取活跃事务数。
*/
getActiveTxnCount(): number {
return this.activeTxns.size;
}
/**
* 获取全局 LSN。
*/
getGlobalLSN(): number {
return this.globalCommitLsn;
}
}
/**
* AriaEngine MVCC — 多版本并发控制
* @module engine/aria/transaction/mvcc
*
* 实现快照隔离 (Snapshot Isolation)。
* 每个事务看到数据库在事务开始时的快照。
*/
import type { RowVersion, TxnEntry } from '../types';
import { TransactionState } from '../types';
// ---------------------------------------------------------------------------
// MVCCManager
// ---------------------------------------------------------------------------
export class MVCCManager {
/** 所有行版本的存储：tableName.key → 版本链 */
private versionStore: Map<string, RowVersion[]> = new Map();
/** 活跃事务表：txnId → TxnEntry */
private activeTxns: Map<number, TxnEntry> = new Map();
/** 事务 ID 计数器 */
private nextTxnId = 1;
/** 全局提交序列号（用于可见性判断） */
private globalCommitLsn = 0;
/**
* v0.4.2-fix: 每个事务写入的 tableKey 集合 —
* commit/rollback 只遍历本事务写过的 key，避免全库版本链扫描（大表事务 O(N) → O(写入数)）
*/
private txnWriteKeys: Map<number, Set<string>> = new Map();
// =======================================================================
// 事务管理
// =======================================================================
/** 开始一个事务，返回事务 ID */
beginTransaction(): number {
const txnId = this.nextTxnId++;
this.activeTxns.set(txnId, {
txnId,
state: TransactionState.ACTIVE,
snapshotLsn: this.globalCommitLsn,
startTime: Date.now(),
});
this.txnWriteKeys.set(txnId, new Set());
return txnId;
}
/** 提交事务 */
commitTransaction(txnId: number): void {
const txn = this.activeTxns.get(txnId);
if (!txn) throw new Error(`Transaction ${txnId} not found`);
txn.state = TransactionState.COMMITTED;
this.globalCommitLsn++;
// v0.4.2-fix: 仅标记本事务写入的版本（此前遍历全库 versionStore）
const writeKeys = this.txnWriteKeys.get(txnId);
if (writeKeys) {
for (const tableKey of writeKeys) {
const versions = this.versionStore.get(tableKey);
if (!versions) continue;
for (const version of versions) {
if (version.txnId === txnId) {
version.committed = true;
}
}
}
}
// 清理已提交事务的记录
this.activeTxns.delete(txnId);
this.txnWriteKeys.delete(txnId);
}
/** 回滚事务 */
rollbackTransaction(txnId: number): void {
const txn = this.activeTxns.get(txnId);
if (!txn) throw new Error(`Transaction ${txnId} not found`);
txn.state = TransactionState.ABORTED;
// v0.4.2-fix: 仅移除本事务写入的版本（此前遍历全库 versionStore）
const writeKeys = this.txnWriteKeys.get(txnId);
if (writeKeys) {
for (const tableKey of writeKeys) {
const versions = this.versionStore.get(tableKey);
if (!versions) continue;
const filtered = versions.filter((v) => v.txnId !== txnId);
if (filtered.length === 0) {
this.versionStore.delete(tableKey);
} else {
this.versionStore.set(tableKey, filtered);
}
}
}
this.activeTxns.delete(txnId);
this.txnWriteKeys.delete(txnId);
}
// =======================================================================
// 版本读写
// =======================================================================
/**
* 写入一行（创建新版本）。
*/
writeVersion(
tableName: string,
key: string,
data: Record<string, unknown>,
txnId: number,
): void {
const tableKey = `${tableName}.${key}`;
const versions = this.versionStore.get(tableKey) ?? [];
const newVersion: RowVersion = {
txnId,
data,
prevVersion: versions.length > 0 ? versions[versions.length - 1] : null,
committed: false,
};
versions.push(newVersion);
this.versionStore.set(tableKey, versions);
// v0.4.2-fix: 记录本事务写过的 key（commit/rollback 精准清理）
this.txnWriteKeys.get(txnId)?.add(tableKey);
}
/**
* 删除一行（创建墓碑版本）。
*/
deleteVersion(tableName: string, key: string, txnId: number): void {
this.writeVersion(tableName, key, { __mvcc_tombstone: true } as unknown as Record<string, unknown>, txnId);
}
/**
* v0.3.3: 丢弃指定事务的所有版本记录，但保留事务登记（Savepoint 回滚用）。
* 快照数据由调用方（引擎 txnSnapshot）负责恢复。
* v0.4.2-fix: 仅遍历本事务写过的 key（此前全库扫描）。
*/
discardVersions(txnId: number): void {
const writeKeys = this.txnWriteKeys.get(txnId);
if (!writeKeys) return;
for (const tableKey of writeKeys) {
const versions = this.versionStore.get(tableKey);
if (!versions) continue;
const filtered = versions.filter((v) => v.txnId !== txnId);
if (filtered.length === 0) {
this.versionStore.delete(tableKey);
} else {
this.versionStore.set(tableKey, filtered);
}
}
}
/**
* 清理过旧版本（GC）。
* 保留每个 key 的最新 N 个已提交版本。
*/
gc(maxVersionsPerKey: number = 100): void {
for (const [tableKey, versions] of this.versionStore) {
if (versions.length <= maxVersionsPerKey) continue;
// 保留最新的 maxVersionsPerKey 个版本
const pruned = versions.slice(versions.length - maxVersionsPerKey);
this.versionStore.set(tableKey, pruned);
}
}
/**
* 获取全局 LSN。
*/
getGlobalLSN(): number {
return this.globalCommitLsn;
}
}
src/engine/aria/types.ts
+249 -302
View File
@@ -1,302 +1,249 @@
/**
* AriaEngine Types — 内部类型定义
* @module engine/aria/types
*
* 页面式存储引擎的所有内部枚举、接口和常量。
*/
// =============================================================================
// 页面常量
// =============================================================================
/** 页面大小：4KB */
export const PAGE_SIZE = 4096;
/** 页面头大小：16 字节 */
export const PAGE_HEADER_SIZE = 16;
/** 每个 Slot 目录项大小：4 字节 (offset: u16 + len: u16) */
export const SLOT_ENTRY_SIZE = 4;
/** 页面数据区起始偏移（头部之后） */
export const PAGE_DATA_START = PAGE_HEADER_SIZE;
/** 无效页面 ID */
export const INVALID_PAGE_ID = 0xFFFFFFFF;
// =============================================================================
// 页面类型
// =============================================================================
export enum PageType {
/** 数据页面：存储行数据 */
DATA = 1,
/** 索引页面：存储索引节点 */
INDEX = 2,
/** 溢出页面：存储大字段 */
OVERFLOW = 3,
/** 元数据页面：存储表/库元信息 */
META = 4,
}
// =============================================================================
// 页面头部（16 字节）
// =============================================================================
export interface PageHeader {
/** 页面 ID（全局唯一） */
pageId: number;
/** 页面类型 */
type: PageType;
/** 空闲空间起始偏移（slot 区结束位置） */
freeStart: number;
/** 数据区结束偏移（从页面底部向上增长） */
freeEnd: number;
/** 当前 slot 数量 */
slotCount: number;
/** CRC32 校验和 */
checksum: number;
}
// =============================================================================
// Slot 目录项（4 字节）
// =============================================================================
export interface SlotEntry {
/** 行数据在页面内的偏移 */
offset: number;
/** 行数据长度（字节） */
length: number;
}
// =============================================================================
// 页面句柄（Buffer Pool 中的页面）
// =============================================================================
export interface PageHandle {
/** 页面 ID */
pageId: number;
/** 页面类型 */
type: PageType;
/** 页面数据缓冲区（4KB ArrayBuffer） */
data: ArrayBuffer;
/** 是否被修改（脏页） */
dirty: boolean;
/** 引用计数（pin count） */
pins: number;
/** LRU 链表前驱 */
prev: PageHandle | null;
/** LRU 链表后继 */
next: PageHandle | null;
/** 最后访问时间戳 */
lastAccess: number;
}
// =============================================================================
// 行编解码
// =============================================================================
/** 行/元组的二进制表示 */
export interface SerializedTuple {
/** 序列化后的字节数组 */
bytes: Uint8Array;
/** 该行中 null 列的位图 */
nullBitmap: Uint8Array;
}
/** 列类型（内部二进制编码用） */
export enum ColumnEncoding {
STRING = 1,
NUMBER = 2,
BOOLEAN = 3,
DATE = 4,
JSON = 5,
NULL = 6,
}
// =============================================================================
// LSM-Tree
// =============================================================================
/** MemTable 最大大小（默认 4MB） */
export const DEFAULT_MEMTABLE_SIZE = 4 * 1024 * 1024;
/** SSTable 中每个 Data Block 的默认大小 */
export const DEFAULT_BLOCK_SIZE = 4096;
/** Bloom Filter 每 key 的默认位数 */
export const DEFAULT_BLOOM_BITS_PER_KEY = 10;
/** SSTable 最大层级 */
export const MAX_LSM_LEVELS = 7;
/** 每层之间的大小倍数 */
export const DEFAULT_LEVEL_SIZE_MULTIPLIER = 10;
/** SSTable 元数据 */
export interface SSTableMeta {
/** SSTable 文件 ID */
id: number;
/** 所在层级 */
level: number;
/** 最小 key */
minKey: string;
/** 最大 key */
maxKey: string;
/** 数据块数量 */
blockCount: number;
/** 总大小（字节） */
totalSize: number;
/** Bloom Filter 序列化数据 */
bloomData: Uint8Array | null;
}
/** SSTable 内部的 Data Block */
export interface DataBlock {
/** 该块内的 key-value 条目数 */
entryCount: number;
/** 该块数据区 */
data: Uint8Array;
/** 该块起始 key */
startKey: string;
/** 该块结束 key */
endKey: string;
}
/** 索引块条目：key → data block offset */
export interface IndexEntry {
/** 到此 block 的最后一个 key */
key: string;
/** data block 在 SSTable 文件中的偏移 */
blockOffset: number;
/** data block 大小 */
blockSize: number;
}
// =============================================================================
// WAL (Write-Ahead Log)
// =============================================================================
/** WAL 记录类型 */
export enum WALRecordType {
INSERT = 1,
UPDATE = 2,
DELETE = 3,
BEGIN = 4,
COMMIT = 5,
ROLLBACK = 6,
CREATE_TABLE = 7,
DROP_TABLE = 8,
}
/** 单条 WAL 记录 */
export interface WALRecord {
/** 日志序列号 */
lsn: number;
/** 记录类型 */
type: WALRecordType;
/** 事务 ID */
txnId: number;
/** 表名 */
tableName: string;
/** 主键值 */
key: string;
/** 操作数据（INSERT/UPDATE 时有效） */
data?: Record<string, unknown>;
/** 校验和 */
checksum: number;
}
// =============================================================================
// MVCC
// =============================================================================
/** 事务隔离级别 */
export enum IsolationLevel {
READ_COMMITTED = 1,
SNAPSHOT = 2,
}
/** 事务状态 */
export enum TransactionState {
ACTIVE = 1,
COMMITTED = 2,
ABORTED = 3,
}
/** 行版本 */
export interface RowVersion {
/** 事务 ID（创建此版本的事务） */
txnId: number;
/** 版本数据 */
data: Record<string, unknown>;
/** 指向上一版本的指针（undo 链） */
prevVersion: RowVersion | null;
/** 该版本是否已提交 */
committed: boolean;
}
/** 活跃事务表项 */
export interface TxnEntry {
/** 事务 ID */
txnId: number;
/** 事务状态 */
state: TransactionState;
/** 快照序列号（用于可见性判断） */
snapshotLsn: number;
/** 事务开始时间 */
startTime: number;
}
// =============================================================================
// Buffer Pool
// =============================================================================
/** Buffer Pool 默认容量：256 页 ≈ 1MB */
export const DEFAULT_BUFFER_POOL_PAGES = 256;
// =============================================================================
// AriaEngine 配置
// =============================================================================
export interface AriaEngineConfig {
/** 页面大小（默认 4096） */
pageSize?: number;
/** Buffer Pool 页面数量（默认 256） */
bufferPoolPages?: number;
/** MemTable 刷盘阈值（默认 4MB） */
memtableSizeThreshold?: number;
/** LSM 层级之间的容量倍数（默认 10） */
levelSizeMultiplier?: number;
/** Bloom Filter 每 key 位数（默认 10） */
bloomFilterBitsPerKey?: number;
/** 是否启用 WAL（默认 true） */
walEnabled?: boolean;
/** WAL 同步模式 */
walSyncMode?: 'full' | 'batch' | 'none';
/** Checkpoint 间隔（操作数，默认 1000） */
checkpointInterval?: number;
/** 是否启用页面压缩（默认 false） */
compression?: boolean;
/** 存储后端 */
storageBackend?: 'indexeddb' | 'opfs' | 'memory';
/** WAL 大小阈值（字节，超过则强制 checkpoint，默认 16MB） */
walSizeThreshold?: number;
/** 最大内存预算（MB，默认 64） */
maxMemoryMB?: number;
}
export const DEFAULT_ARIA_CONFIG: Required<AriaEngineConfig> = {
pageSize: PAGE_SIZE,
bufferPoolPages: DEFAULT_BUFFER_POOL_PAGES,
memtableSizeThreshold: DEFAULT_MEMTABLE_SIZE,
levelSizeMultiplier: DEFAULT_LEVEL_SIZE_MULTIPLIER,
bloomFilterBitsPerKey: DEFAULT_BLOOM_BITS_PER_KEY,
walEnabled: true,
walSyncMode: 'full',
checkpointInterval: 1000,
compression: false,
storageBackend: 'indexeddb',
walSizeThreshold: 16 * 1024 * 1024, // 16MB
maxMemoryMB: 64,
};
/**
* AriaEngine Types — 内部类型定义
* @module engine/aria/types
*
* 页面式存储引擎的所有内部枚举、接口和常量。
*/
// =============================================================================
// 页面常量
// =============================================================================
/** 页面大小：4KB */
export const PAGE_SIZE = 4096;
/** 页面头大小：16 字节 */
export const PAGE_HEADER_SIZE = 16;
// =============================================================================
// 页面类型
// =============================================================================
export enum PageType {
/** 数据页面：存储 SSTable 字节切片 */
DATA = 1,
/** 索引页面：存储索引节点 */
INDEX = 2,
/** 溢出页面：存储大字段 */
OVERFLOW = 3,
/** 元数据页面：存储表/库元信息 */
META = 4,
}
// =============================================================================
// 页面句柄（Buffer Pool 中的页面）
// =============================================================================
export interface PageHandle {
/** 页面 ID */
pageId: number;
/** 页面类型 */
type: PageType;
/** 页面数据缓冲区（4KB ArrayBuffer） */
data: ArrayBuffer;
/** 是否被修改（脏页） */
dirty: boolean;
/** 引用计数（pin count） */
pins: number;
/** LRU 链表前驱 */
prev: PageHandle | null;
/** LRU 链表后继 */
next: PageHandle | null;
/** 最后访问时间戳 */
lastAccess: number;
}
// =============================================================================
// 列类型（内部二进制编码用）
// =============================================================================
// =============================================================================
// LSM-Tree
// =============================================================================
/** MemTable 最大大小（默认 4MB） */
export const DEFAULT_MEMTABLE_SIZE = 4 * 1024 * 1024;
/** Bloom Filter 每 key 的默认位数 */
export const DEFAULT_BLOOM_BITS_PER_KEY = 10;
/** SSTable 最大层级 */
export const MAX_LSM_LEVELS = 7;
/** 每层之间的大小倍数 */
export const DEFAULT_LEVEL_SIZE_MULTIPLIER = 10;
/** SSTable 元数据 */
export interface SSTableMeta {
/** SSTable 文件 ID */
id: number;
/** 所在层级 */
level: number;
/** 最小 key */
minKey: string;
/** 最大 key */
maxKey: string;
/** 数据块数量 */
blockCount: number;
/** 总大小（字节） */
totalSize: number;
/** Bloom Filter 序列化数据 */
bloomData: Uint8Array | null;
/**
* v0.4.5: 页面化存储（OPFS 后端）— SSTable 的物理页面 ID 列表。
* 存在 → 数据以 4KB 页面存储（BufferPool/FileManager 管理）；
* 缺失 → 整 value 存储（旧格式，兼容读取）。
*/
pageIds?: number[];
}
/** 索引块条目：key → data block offset */
export interface IndexEntry {
/** 到此 block 的最后一个 key */
key: string;
/** data block 在 SSTable 文件中的偏移 */
blockOffset: number;
/** data block 大小 */
blockSize: number;
}
// =============================================================================
// WAL (Write-Ahead Log)
// =============================================================================
/** WAL 记录类型 */
export enum WALRecordType {
INSERT = 1,
UPDATE = 2,
DELETE = 3,
BEGIN = 4,
COMMIT = 5,
ROLLBACK = 6,
CREATE_TABLE = 7,
DROP_TABLE = 8,
}
/** 单条 WAL 记录 */
export interface WALRecord {
/** 日志序列号 */
lsn: number;
/** 记录类型 */
type: WALRecordType;
/** 事务 ID */
txnId: number;
/** 表名 */
tableName: string;
/** 主键值 */
key: string;
/** 操作数据（INSERT/UPDATE 时有效） */
data?: Record<string, unknown>;
/** 校验和 */
checksum: number;
}
// =============================================================================
// MVCC
// =============================================================================
/** 事务状态 */
export enum TransactionState {
ACTIVE = 1,
COMMITTED = 2,
ABORTED = 3,
}
/** 行版本 */
export interface RowVersion {
/** 事务 ID（创建此版本的事务） */
txnId: number;
/** 版本数据 */
data: Record<string, unknown>;
/** 指向上一版本的指针（undo 链） */
prevVersion: RowVersion | null;
/** 该版本是否已提交 */
committed: boolean;
}
/** 活跃事务表项 */
export interface TxnEntry {
/** 事务 ID */
txnId: number;
/** 事务状态 */
state: TransactionState;
/** 快照序列号（用于可见性判断） */
snapshotLsn: number;
/** 事务开始时间 */
startTime: number;
}
// =============================================================================
// Buffer Pool
// =============================================================================
/** Buffer Pool 默认容量：256 页 ≈ 1MB */
export const DEFAULT_BUFFER_POOL_PAGES = 256;
// =============================================================================
// AriaEngine 配置
// =============================================================================
export interface AriaEngineConfig {
/** 页面大小（默认 4096） */
pageSize?: number;
/** Buffer Pool 页面数量（默认 256） */
bufferPoolPages?: number;
/** MemTable 刷盘阈值（默认 4MB） */
memtableSizeThreshold?: number;
/** LSM 层级之间的容量倍数（默认 10） */
levelSizeMultiplier?: number;
/** Bloom Filter 每 key 位数（默认 10） */
bloomFilterBitsPerKey?: number;
/** 是否启用 WAL（默认 true） */
walEnabled?: boolean;
/** WAL 同步模式 */
walSyncMode?: 'full' | 'batch' | 'none';
/** Checkpoint 间隔（操作数，默认 1000） */
checkpointInterval?: number;
/** 是否启用页面压缩（默认 false） */
compression?: boolean;
/** 存储后端 */
storageBackend?: 'indexeddb' | 'opfs' | 'memory';
/** WAL 大小阈值（字节，超过则强制 checkpoint，默认 16MB） */
walSizeThreshold?: number;
/** 最大内存预算（MB，默认 64） */
maxMemoryMB?: number;
/**
* v0.4.5: 全库 AES-256-GCM 加密（backend 层透明加解密，WAL/SSTable/Schema/元数据全覆盖）。
* 密钥由 PBKDF2（salt 持久化于库内 __aria_keymeta）派生，重启用同一密码即可解密。
*/
encryption?: {
/** 加密密码 */
password: string;
};
/**
* v0.4.5: SSTable 页面化存储（4KB 页面 + BufferPool/FileManager 管理，LRU 缓存）。
* 默认：storageBackend === 'opfs' 时自动启用（大文件随机读/写放大优化）；
* 显式 false 强制关闭（整 value 存储，兼容旧行为）。
*/
pageStorage?: boolean;
}
export const DEFAULT_ARIA_CONFIG: Required<Omit<AriaEngineConfig, 'encryption' | 'pageStorage'>> & {
encryption: undefined;
pageStorage: undefined;
} = {
pageSize: PAGE_SIZE,
bufferPoolPages: DEFAULT_BUFFER_POOL_PAGES,
memtableSizeThreshold: DEFAULT_MEMTABLE_SIZE,
levelSizeMultiplier: DEFAULT_LEVEL_SIZE_MULTIPLIER,
bloomFilterBitsPerKey: DEFAULT_BLOOM_BITS_PER_KEY,
walEnabled: true,
walSyncMode: 'full',
checkpointInterval: 1000,
compression: false,
storageBackend: 'indexeddb',
walSizeThreshold: 16 * 1024 * 1024, // 16MB
maxMemoryMB: 64,
encryption: undefined,
pageStorage: undefined,
};
src/engine/aria/wal/checkpoint.ts
+71 -82
View File
@@ -1,82 +1,71 @@
/**
* AriaEngine Checkpoint — 检查点机制
* @module engine/aria/wal/checkpoint
*/
import type { LSM } from '../index/lsm';
import type { WAL } from './log';
// ---------------------------------------------------------------------------
// 简化的 flush 接口
// ---------------------------------------------------------------------------
export interface Flushable {
flushAll(): Promise<void>;
}
// ---------------------------------------------------------------------------
// CheckpointManager
// ---------------------------------------------------------------------------
export class CheckpointManager {
private lsm: LSM;
private wal: WAL;
private flushable: Flushable | null;
private interval: number;
private opCount = 0;
private walSizeThreshold: number;
constructor(
lsm: LSM,
wal: WAL,
flushable: Flushable | null = null,
interval: number = 1000,
walSizeThreshold: number = 16 * 1024 * 1024,
) {
this.lsm = lsm;
this.wal = wal;
this.flushable = flushable;
this.interval = interval;
this.walSizeThreshold = walSizeThreshold;
}
async tick(): Promise<void> {
this.opCount++;
// 检查操作计数或 WAL 大小是否超阈值
if (this.opCount >= this.interval || this.getWALEstimatedSize() >= this.walSizeThreshold) {
await this.checkpoint();
}
}
/** 估算 WAL 大小（优先真实字节数，回退到缓冲计数估算） */
private getWALEstimatedSize(): number {
const wal = this.wal as unknown as { getBufferedBytes?: () => number; getBufferedCount?: () => number };
if (typeof wal.getBufferedBytes === 'function') {
const bytes = wal.getBufferedBytes();
if (bytes > 0) return bytes;
}
const count = typeof wal.getBufferedCount === 'function' ? wal.getBufferedCount() : 0;
return count * 200;
}
async checkpoint(): Promise<void> {
await this.lsm.flush();
if (this.flushable) {
await this.flushable.flushAll();
}
await this.wal.checkpoint();
this.opCount = 0;
}
async forceCheckpoint(): Promise<void> {
await this.checkpoint();
}
setInterval(ops: number): void {
this.interval = ops;
}
getOpCount(): number {
return this.opCount;
}
}
/**
* AriaEngine Checkpoint — 检查点机制
* @module engine/aria/wal/checkpoint
*/
import type { LSM } from '../index/lsm';
import type { WAL } from './log';
// ---------------------------------------------------------------------------
// 简化的 flush 接口
// ---------------------------------------------------------------------------
export interface Flushable {
flushAll(): Promise<void>;
}
// ---------------------------------------------------------------------------
// CheckpointManager
// ---------------------------------------------------------------------------
export class CheckpointManager {
private lsm: LSM;
private wal: WAL;
private flushable: Flushable | null;
private interval: number;
private opCount = 0;
private walSizeThreshold: number;
constructor(
lsm: LSM,
wal: WAL,
flushable: Flushable | null = null,
interval: number = 1000,
walSizeThreshold: number = 16 * 1024 * 1024,
) {
this.lsm = lsm;
this.wal = wal;
this.flushable = flushable;
this.interval = interval;
this.walSizeThreshold = walSizeThreshold;
}
async tick(): Promise<void> {
this.opCount++;
// 检查操作计数或 WAL 大小是否超阈值
if (this.opCount >= this.interval || this.getWALEstimatedSize() >= this.walSizeThreshold) {
await this.checkpoint();
}
}
/** 估算 WAL 大小（优先真实字节数，回退到缓冲计数估算） */
private getWALEstimatedSize(): number {
const wal = this.wal as unknown as { getBufferedBytes?: () => number; getBufferedCount?: () => number };
if (typeof wal.getBufferedBytes === 'function') {
const bytes = wal.getBufferedBytes();
if (bytes > 0) return bytes;
}
const count = typeof wal.getBufferedCount === 'function' ? wal.getBufferedCount() : 0;
return count * 200;
}
async checkpoint(): Promise<void> {
await this.lsm.flush();
if (this.flushable) {
await this.flushable.flushAll();
}
await this.wal.checkpoint();
this.opCount = 0;
}
}
src/engine/aria/wal/log.ts
+322 -325
View File
@@ -1,325 +1,322 @@
/**
* AriaEngine WAL — Write-Ahead Log
* @module engine/aria/wal/log
*
* 崩溃恢复前的写操作持久化日志。
*
* WAL 文件格式:
* ┌──────────┬──────────────┬──────────┐
* │ Record 1│ Record 2 │ ... │
* │ 4B LSN │ │ │
* │ 1B type │ │ │
* │ 4B txnId│ │ │
* │ 2B tblLen│ │ │
* │ N table│ │ │
* │ 2B keyLen│ │ │
* │ N key │ │ │
* │ 4B jsonLen│ │ │
* │ N json │ │ │
* │ 4B CRC │ │ │
* └──────────┴──────────────┴──────────┘
*/
import { WALRecordType, type WALRecord } from '../types';
// ---------------------------------------------------------------------------
// WAL 存储接口
// ---------------------------------------------------------------------------
export interface WALStore {
/** 追加 WAL 记录 */
append(data: Uint8Array): Promise<void>;
/** 读取所有 WAL 记录 */
readAll(): Promise<Uint8Array>;
/** 截断 WAL（checkpoint 后清理） */
truncate(): Promise<void>;
/** 检查 WAL 是否存在 */
exists(): Promise<boolean>;
}
// ---------------------------------------------------------------------------
// WAL
// ---------------------------------------------------------------------------
export class WAL {
private lsn = 0;
private store: WALStore;
private enabled: boolean;
private buffer: Uint8Array[] = [];
private syncMode: 'full' | 'batch' | 'none';
/** v0.3.3: 未 checkpoint 的 WAL 累计字节数（full/batch/none 通用） */
private bufferedBytes = 0;
constructor(store: WALStore, enabled: boolean = true, syncMode: 'full' | 'batch' | 'none' = 'batch') {
this.store = store;
this.enabled = enabled;
this.syncMode = syncMode;
}
// =======================================================================
// 写入
// =======================================================================
/** 追加一条 WAL 记录（full 模式同步等待写入完成） */
async append(record: Omit<WALRecord, 'lsn' | 'checksum'>): Promise<void> {
if (!this.enabled) return;
this.lsn++;
const fullRecord: WALRecord = {
...record,
lsn: this.lsn,
checksum: 0, // 稍后计算
};
const bytes = this.encodeRecord(fullRecord);
if (this.syncMode === 'full') {
try {
await this.store.append(bytes);
this.bufferedBytes += bytes.byteLength;
} catch {
// eslint-disable-next-line no-console
console.warn('[AriaEngine WAL] Failed to append record');
}
} else if (this.syncMode === 'batch') {
this.buffer.push(bytes);
this.bufferedBytes += bytes.byteLength;
}
// 'none' mode: 不写 WAL
}
/** 批量追加多条 WAL 记录（组提交：合并为一次底层写入，v0.3.1） */
async appendBatch(records: Omit<WALRecord, 'lsn' | 'checksum'>[]): Promise<void> {
if (!this.enabled || records.length === 0) return;
const chunks: Uint8Array[] = [];
for (const record of records) {
this.lsn++;
chunks.push(this.encodeRecord({ ...record, lsn: this.lsn, checksum: 0 }));
}
const combined = this.mergeChunks(chunks);
if (this.syncMode === 'full') {
try {
await this.store.append(combined);
this.bufferedBytes += combined.byteLength;
} catch {
// eslint-disable-next-line no-console
console.warn('[AriaEngine WAL] Failed to append batch record');
}
} else if (this.syncMode === 'batch') {
this.buffer.push(combined);
this.bufferedBytes += combined.byteLength;
}
// 'none' mode: 不写 WAL
}
/** 批量刷新缓冲的 WAL 记录 */
async flush(): Promise<void> {
if (!this.enabled || this.buffer.length === 0) return;
const combined = this.mergeChunks(this.buffer);
await this.store.append(combined);
this.buffer = [];
}
/** 合并多个字节块为一个连续缓冲区 */
private mergeChunks(chunks: Uint8Array[]): Uint8Array {
if (chunks.length === 1) return chunks[0];
const totalLen = chunks.reduce((sum, b) => sum + b.byteLength, 0);
const combined = new Uint8Array(totalLen);
let offset = 0;
for (const buf of chunks) {
combined.set(buf, offset);
offset += buf.byteLength;
}
return combined;
}
// =======================================================================
// 恢复
// =======================================================================
/** 从 WAL 恢复未提交的事务数据 */
async recover(
applyRecord: (record: WALRecord) => void,
): Promise<number> {
if (!this.enabled) return 0;
const exists = await this.store.exists();
if (!exists) return 0;
const data = await this.store.readAll();
if (data.byteLength === 0) return 0;
const records = this.decodeAllRecords(data);
for (const record of records) {
applyRecord(record);
}
this.lsn = records.length > 0 ? records[records.length - 1].lsn : 0;
return records.length;
}
// =======================================================================
// Checkpoint
// =======================================================================
/** Checkpoint 后清空 WAL */
async checkpoint(): Promise<void> {
if (!this.enabled) return;
await this.flush();
await this.store.truncate();
this.lsn = 0;
this.bufferedBytes = 0;
}
// =======================================================================
// 统计
// =======================================================================
isEnabled(): boolean {
return this.enabled;
}
getLSN(): number {
return this.lsn;
}
getBufferedCount(): number {
return this.buffer.length;
}
/** v0.3.3: 未 checkpoint 的 WAL 累计字节数（full/batch/none 通用） */
getBufferedBytes(): number {
return this.bufferedBytes;
}
// -----------------------------------------------------------------------
// 编解码
// -----------------------------------------------------------------------
private encodeRecord(record: WALRecord): Uint8Array {
const encoder = new TextEncoder();
const tableBytes = encoder.encode(record.tableName);
const keyBytes = encoder.encode(record.key);
const jsonStr = record.data ? JSON.stringify(record.data) : '';
const jsonBytes = encoder.encode(jsonStr);
const size =
4 + // LSN
1 + // type
4 + // txnId
2 + tableBytes.length + // table
2 + keyBytes.length + // key
4 + jsonBytes.length + // json
4; // CRC
const buf = new ArrayBuffer(size);
const view = new DataView(buf);
let offset = 0;
view.setUint32(offset, record.lsn, false);
offset += 4;
view.setUint8(offset, record.type);
offset += 1;
view.setUint32(offset, record.txnId, false);
offset += 4;
view.setUint16(offset, tableBytes.length, false);
offset += 2;
new Uint8Array(buf).set(tableBytes, offset);
offset += tableBytes.length;
view.setUint16(offset, keyBytes.length, false);
offset += 2;
new Uint8Array(buf).set(keyBytes, offset);
offset += keyBytes.length;
view.setUint32(offset, jsonBytes.length, false);
offset += 4;
new Uint8Array(buf).set(jsonBytes, offset);
offset += jsonBytes.length;
// 简单 CRC
let crc = 0;
const u8 = new Uint8Array(buf, 0, offset);
for (let i = 0; i < u8.length; i++) {
crc = ((crc << 5) - crc + u8[i]) | 0;
}
view.setUint32(offset, crc >>> 0, false);
return new Uint8Array(buf);
}
private decodeAllRecords(data: Uint8Array): WALRecord[] {
const records: WALRecord[] = [];
const view = new DataView(data.buffer, data.byteOffset, data.byteLength);
let offset = 0;
while (offset + 15 <= data.byteLength) {
try {
const recordStart = offset;
const lsn = view.getUint32(offset, false);
offset += 4;
const type = view.getUint8(offset) as WALRecordType;
offset += 1;
const txnId = view.getUint32(offset, false);
offset += 4;
const tableLen = view.getUint16(offset, false);
offset += 2;
if (offset + tableLen > data.byteLength) break;
const tableName = new TextDecoder().decode(data.slice(offset, offset + tableLen));
offset += tableLen;
const keyLen = view.getUint16(offset, false);
offset += 2;
if (offset + keyLen > data.byteLength) break;
const key = new TextDecoder().decode(data.slice(offset, offset + keyLen));
offset += keyLen;
const jsonLen = view.getUint32(offset, false);
offset += 4;
if (offset + jsonLen > data.byteLength) break;
let recordData: Record<string, unknown> | undefined;
if (jsonLen > 0) {
const json = new TextDecoder().decode(data.slice(offset, offset + jsonLen));
try {
recordData = JSON.parse(json);
} catch { /* ok */ }
}
offset += jsonLen;
// 验证 CRC（跨记录数据计算，不含 CRC 自身）
const storedCrc = view.getUint32(offset, false);
offset += 4;
const recordBytes = data.slice(recordStart, offset - 4);
let computedCrc = 0;
for (let i = 0; i < recordBytes.length; i++) {
computedCrc = ((computedCrc << 5) - computedCrc + recordBytes[i]) | 0;
}
if ((computedCrc >>> 0) !== storedCrc) {
// CRC 不匹配，跳过此损坏记录
// eslint-disable-next-line no-console
console.warn(`[AriaEngine WAL] CRC mismatch at record LSN=${lsn}, skipping`);
continue;
}
records.push({
lsn,
type,
txnId,
tableName,
key,
data: recordData,
checksum: storedCrc,
});
} catch {
break;
}
}
return records;
}
}
/**
* AriaEngine WAL — Write-Ahead Log
* @module engine/aria/wal/log
*
* 崩溃恢复前的写操作持久化日志。
*
* WAL 文件格式:
* ┌──────────┬──────────────┬──────────┐
* │ Record 1│ Record 2 │ ... │
* │ 4B LSN │ │ │
* │ 1B type │ │ │
* │ 4B txnId│ │ │
* │ 2B tblLen│ │ │
* │ N table│ │ │
* │ 2B keyLen│ │ │
* │ N key │ │ │
* │ 4B jsonLen│ │ │
* │ N json │ │ │
* │ 4B CRC │ │ │
* └──────────┴──────────────┴──────────┘
*/
import { WALRecordType, type WALRecord } from '../types';
import { crc32 } from '../crc32';
// ---------------------------------------------------------------------------
// WAL 存储接口
// ---------------------------------------------------------------------------
export interface WALStore {
/** 追加 WAL 记录 */
append(data: Uint8Array): Promise<void>;
/** 读取所有 WAL 记录 */
readAll(): Promise<Uint8Array>;
/** 截断 WAL（checkpoint 后清理） */
truncate(): Promise<void>;
/** 检查 WAL 是否存在 */
exists(): Promise<boolean>;
}
// ---------------------------------------------------------------------------
// WAL
// ---------------------------------------------------------------------------
export class WAL {
private lsn = 0;
private store: WALStore;
private enabled: boolean;
private buffer: Uint8Array[] = [];
private syncMode: 'full' | 'batch' | 'none';
/** v0.3.3: 未 checkpoint 的 WAL 累计字节数（full/batch/none 通用） */
private bufferedBytes = 0;
constructor(store: WALStore, enabled: boolean = true, syncMode: 'full' | 'batch' | 'none' = 'batch') {
this.store = store;
this.enabled = enabled;
this.syncMode = syncMode;
}
// =======================================================================
// 写入
// =======================================================================
/** 追加一条 WAL 记录（full 模式同步等待写入完成） */
async append(record: Omit<WALRecord, 'lsn' | 'checksum'>): Promise<void> {
if (!this.enabled) return;
this.lsn++;
const fullRecord: WALRecord = {
...record,
lsn: this.lsn,
checksum: 0, // 稍后计算
};
const bytes = this.encodeRecord(fullRecord);
if (this.syncMode === 'full') {
try {
await this.store.append(bytes);
this.bufferedBytes += bytes.byteLength;
} catch {
// eslint-disable-next-line no-console
console.warn('[AriaEngine WAL] Failed to append record');
}
} else if (this.syncMode === 'batch') {
this.buffer.push(bytes);
this.bufferedBytes += bytes.byteLength;
}
// 'none' mode: 不写 WAL
}
/** 批量追加多条 WAL 记录（组提交：合并为一次底层写入，v0.3.1） */
async appendBatch(records: Omit<WALRecord, 'lsn' | 'checksum'>[]): Promise<void> {
if (!this.enabled || records.length === 0) return;
const chunks: Uint8Array[] = [];
for (const record of records) {
this.lsn++;
chunks.push(this.encodeRecord({ ...record, lsn: this.lsn, checksum: 0 }));
}
const combined = this.mergeChunks(chunks);
if (this.syncMode === 'full') {
try {
await this.store.append(combined);
this.bufferedBytes += combined.byteLength;
} catch {
// eslint-disable-next-line no-console
console.warn('[AriaEngine WAL] Failed to append batch record');
}
} else if (this.syncMode === 'batch') {
this.buffer.push(combined);
this.bufferedBytes += combined.byteLength;
}
// 'none' mode: 不写 WAL
}
/** 批量刷新缓冲的 WAL 记录 */
async flush(): Promise<void> {
if (!this.enabled || this.buffer.length === 0) return;
const combined = this.mergeChunks(this.buffer);
await this.store.append(combined);
this.buffer = [];
}
/** 合并多个字节块为一个连续缓冲区 */
private mergeChunks(chunks: Uint8Array[]): Uint8Array {
if (chunks.length === 1) return chunks[0];
const totalLen = chunks.reduce((sum, b) => sum + b.byteLength, 0);
const combined = new Uint8Array(totalLen);
let offset = 0;
for (const buf of chunks) {
combined.set(buf, offset);
offset += buf.byteLength;
}
return combined;
}
// =======================================================================
// 恢复
// =======================================================================
/** 从 WAL 恢复未提交的事务数据 */
async recover(
applyRecord: (record: WALRecord) => void,
): Promise<number> {
if (!this.enabled) return 0;
const exists = await this.store.exists();
if (!exists) return 0;
const data = await this.store.readAll();
if (data.byteLength === 0) return 0;
const records = this.decodeAllRecords(data);
for (const record of records) {
applyRecord(record);
}
this.lsn = records.length > 0 ? records[records.length - 1].lsn : 0;
return records.length;
}
// =======================================================================
// Checkpoint
// =======================================================================
/** Checkpoint 后清空 WAL */
async checkpoint(): Promise<void> {
if (!this.enabled) return;
await this.flush();
await this.store.truncate();
this.lsn = 0;
this.bufferedBytes = 0;
}
// =======================================================================
// 统计
// =======================================================================
getBufferedCount(): number {
return this.buffer.length;
}
/** v0.3.3: 未 checkpoint 的 WAL 累计字节数（full/batch/none 通用） */
getBufferedBytes(): number {
return this.bufferedBytes;
}
// -----------------------------------------------------------------------
// 编解码
// -----------------------------------------------------------------------
/** 旧版弱滚动校验（v0.4.4 及更早写入的 WAL 记录使用，双算法探测兼容） */
private legacyChecksum(data: Uint8Array): number {
let crc = 0;
for (let i = 0; i < data.length; i++) {
crc = ((crc << 5) - crc + data[i]) | 0;
}
return crc >>> 0;
}
private encodeRecord(record: WALRecord): Uint8Array {
const encoder = new TextEncoder();
const tableBytes = encoder.encode(record.tableName);
const keyBytes = encoder.encode(record.key);
const jsonStr = record.data ? JSON.stringify(record.data) : '';
const jsonBytes = encoder.encode(jsonStr);
const size =
4 + // LSN
1 + // type
4 + // txnId
2 + tableBytes.length + // table
2 + keyBytes.length + // key
4 + jsonBytes.length + // json
4; // CRC
const buf = new ArrayBuffer(size);
const view = new DataView(buf);
let offset = 0;
view.setUint32(offset, record.lsn, false);
offset += 4;
view.setUint8(offset, record.type);
offset += 1;
view.setUint32(offset, record.txnId, false);
offset += 4;
view.setUint16(offset, tableBytes.length, false);
offset += 2;
new Uint8Array(buf).set(tableBytes, offset);
offset += tableBytes.length;
view.setUint16(offset, keyBytes.length, false);
offset += 2;
new Uint8Array(buf).set(keyBytes, offset);
offset += keyBytes.length;
view.setUint32(offset, jsonBytes.length, false);
offset += 4;
new Uint8Array(buf).set(jsonBytes, offset);
offset += jsonBytes.length;
// v0.4.5: 标准 CRC-32 校验（此前为弱滚动校验，误检率更高）
const u8 = new Uint8Array(buf, 0, offset);
const crc = crc32(u8);
view.setUint32(offset, crc, false);
return new Uint8Array(buf);
}
private decodeAllRecords(data: Uint8Array): WALRecord[] {
const records: WALRecord[] = [];
const view = new DataView(data.buffer, data.byteOffset, data.byteLength);
let offset = 0;
while (offset + 15 <= data.byteLength) {
try {
const recordStart = offset;
const lsn = view.getUint32(offset, false);
offset += 4;
const type = view.getUint8(offset) as WALRecordType;
offset += 1;
const txnId = view.getUint32(offset, false);
offset += 4;
const tableLen = view.getUint16(offset, false);
offset += 2;
if (offset + tableLen > data.byteLength) break;
const tableName = new TextDecoder().decode(data.slice(offset, offset + tableLen));
offset += tableLen;
const keyLen = view.getUint16(offset, false);
offset += 2;
if (offset + keyLen > data.byteLength) break;
const key = new TextDecoder().decode(data.slice(offset, offset + keyLen));
offset += keyLen;
const jsonLen = view.getUint32(offset, false);
offset += 4;
if (offset + jsonLen > data.byteLength) break;
let recordData: Record<string, unknown> | undefined;
if (jsonLen > 0) {
const json = new TextDecoder().decode(data.slice(offset, offset + jsonLen));
try {
recordData = JSON.parse(json);
} catch { /* ok */ }
}
offset += jsonLen;
// 验证 CRC：先标准 CRC-32，失败再尝试旧版弱滚动校验（兼容旧库 WAL 记录）
const storedCrc = view.getUint32(offset, false);
offset += 4;
const recordBytes = data.slice(recordStart, offset - 4);
const computedNew = crc32(recordBytes);
const computedLegacy = this.legacyChecksum(recordBytes);
if ((computedNew >>> 0) !== storedCrc && (computedLegacy >>> 0) !== storedCrc) {
// CRC 不匹配，跳过此损坏记录
// eslint-disable-next-line no-console
console.warn(`[AriaEngine WAL] CRC mismatch at record LSN=${lsn}, skipping`);
continue;
}
records.push({
lsn,
type,
txnId,
tableName,
key,
data: recordData,
checksum: storedCrc,
});
} catch {
break;
}
}
return records;
}
}
src/engine/aria/wal/segmented_store.ts
+161
View File
@@ -0,0 +1,161 @@
/**
* AriaEngine Segmented WAL Store — 分片式 WAL 持久化存储
* @module engine/aria/wal/segmented_store
*
* v0.4.5: 取代"每条记录一个 key"的旧结构：
* - 分片文件 `__wal_%06d.bin`，达到阈值（默认 4MB）切新分片 → 文件数量可控
* - append 追加写当前分片（后端支持真追加则 O(chunk)，否则回退 read+write）
* - WAL 序号（LSN）内嵌于记录字节流，无需独立 count 键 → append 单文件原子写
* - readAll 检测分片序号空洞：序号不连续 → 丢弃空洞之后的分片（保守截断，
* 空洞仅可能来自 truncate 部分完成——此时数据已落盘，丢弃无害）
* - 兼容旧格式 `__wal_N`（每条记录一个键）+ `__wal_count`：读取时迁移重放，
* checkpoint 时一并清空
*/
import type { IStorageBackend } from '../store/backend';
/** 分片文件名：__wal_%06d.bin */
export const WAL_SEGMENT_PREFIX = '__wal_';
const SEGMENT_REGEX = /^__wal_(\d{6})\.bin$/;
const LEGACY_RECORD_REGEX = /^__wal_(\d+)$/;
const LEGACY_COUNT_KEY = '__wal_count';
/** 默认分片阈值 */
export const DEFAULT_WAL_SEGMENT_SIZE = 4 * 1024 * 1024;
// ---------------------------------------------------------------------------
// WAL 存储接口（与 WAL 类解耦）
// ---------------------------------------------------------------------------
export interface WALStore {
append(data: Uint8Array): Promise<void>;
readAll(): Promise<Uint8Array>;
truncate(): Promise<void>;
exists(): Promise<boolean>;
}
// ---------------------------------------------------------------------------
// SegmentedWALStore
// ---------------------------------------------------------------------------
export class SegmentedWALStore implements WALStore {
private segmentSize: number;
/** 当前分片序号（append 定位） */
private currentSegment = 0;
/** 当前分片字节数（内存跟踪，append 切分片判断） */
private currentSize = 0;
constructor(
private backend: IStorageBackend,
segmentSize: number = DEFAULT_WAL_SEGMENT_SIZE,
) {
this.segmentSize = segmentSize;
}
/** 分片 key 生成 */
private segmentKey(seq: number): string {
return `${WAL_SEGMENT_PREFIX}${String(seq).padStart(6, '0')}.bin`;
}
async append(data: Uint8Array): Promise<void> {
if (data.byteLength === 0) return;
if (this.currentSize + data.byteLength > this.segmentSize) {
// 当前分片放不下 → 切新分片
this.currentSegment++;
this.currentSize = 0;
}
const key = this.segmentKey(this.currentSegment);
const copy = data.buffer.slice(data.byteOffset, data.byteOffset + data.byteLength) as ArrayBuffer;
if (typeof this.backend.append === 'function') {
await this.backend.append!(key, copy);
} else {
// 回退：读旧 + 拼接 + 写（单文件原子写）
const existing = await this.backend.read(key);
if (existing) {
const combined = new ArrayBuffer(existing.byteLength + copy.byteLength);
new Uint8Array(combined).set(new Uint8Array(existing), 0);
new Uint8Array(combined).set(new Uint8Array(copy), existing.byteLength);
await this.backend.write(key, combined);
} else {
await this.backend.write(key, copy);
}
}
this.currentSize += data.byteLength;
}
async readAll(): Promise<Uint8Array> {
const keys = await this.backend.listKeys();
// ---- 新格式分片 ----
const segments = keys
.filter((k) => SEGMENT_REGEX.test(k))
.map((k) => ({ seq: Number(k.match(SEGMENT_REGEX)![1]), key: k }))
.sort((a, b) => a.seq - b.seq);
// 空洞检测：分片序号必须从 0 严格连续，空洞后的分片整体丢弃
let keepCount = 0;
for (let i = 0; i < segments.length; i++) {
if (segments[i].seq !== i) break;
keepCount = i + 1;
}
const validSegments = segments.slice(0, keepCount);
// ---- 旧格式兼容：__wal_N 单记录键（迁移前数据） ----
const legacyKeys = keys
.filter((k) => LEGACY_RECORD_REGEX.test(k))
.map((k) => ({ seq: Number(k.match(LEGACY_RECORD_REGEX)![1]), key: k }))
.sort((a, b) => a.seq - b.seq);
const parts: Uint8Array[] = [];
// 旧格式在前（它们是最早的记录）
for (const { key } of legacyKeys) {
const raw = await this.backend.read(key);
if (raw) parts.push(new Uint8Array(raw));
}
// 新格式分片在后
for (const { key } of validSegments) {
const raw = await this.backend.read(key);
if (raw) parts.push(new Uint8Array(raw));
}
// 同步当前分片状态（追加定位）
if (validSegments.length > 0) {
this.currentSegment = validSegments[validSegments.length - 1].seq;
const lastRaw = await this.backend.read(validSegments[validSegments.length - 1].key);
this.currentSize = lastRaw ? lastRaw.byteLength : 0;
// 旧格式键存在时（迁移中），下一条记录另起分片，避免与旧键序号冲突
if (legacyKeys.length > 0) {
this.currentSegment++;
this.currentSize = 0;
}
} else if (legacyKeys.length > 0) {
// 仅有旧格式：迁移中，新写入从分片 0 开始（checkpoint 会清空旧键）
this.currentSegment = 0;
this.currentSize = 0;
}
const total = parts.reduce((s, c) => s + c.byteLength, 0);
const combined = new Uint8Array(total);
let off = 0;
for (const c of parts) { combined.set(c, off); off += c.byteLength; }
return combined;
}
async truncate(): Promise<void> {
const keys = await this.backend.listKeys();
const walKeys = keys.filter((k) =>
k.startsWith(WAL_SEGMENT_PREFIX) || k === LEGACY_COUNT_KEY);
if (walKeys.length > 0) {
await this.backend.deleteMany(walKeys);
}
this.currentSegment = 0;
this.currentSize = 0;
}
async exists(): Promise<boolean> {
const keys = await this.backend.listKeys();
return keys.some((k) =>
k.startsWith(WAL_SEGMENT_PREFIX) || k === LEGACY_COUNT_KEY);
}
}
src/engine/opfs.ts
-12
View File
@@ -360,16 +360,4 @@ export class OPFSEngine implements IStorageEngine {
}
}
}
/** 从 OPFS 加载表数据到内存缓存 */
async loadTableIntoMemory(tableName: string, schema: TableSchema): Promise<void> {
await this.memoryCache.createTable(schema);
const rows = await this.readTableData(tableName);
if (rows.length > 0) {
// 直接用 Map 设置绕过 insert 校验
for (const row of rows) {
await this.memoryCache.insert(tableName, [row]);
}
}
}
}
src/query/ast.ts
+34 -39
View File
@@ -12,23 +12,6 @@ import type { WhereCondition, OrderBy } from '../constants';
// AST 语句类型枚举
// ---------------------------------------------------------------------------
export type StatementType =
| 'SELECT'
| 'SELECT_UNION'
| 'EXPLAIN'
| 'INSERT'
| 'UPDATE'
| 'DELETE'
| 'CREATE_TABLE'
| 'DROP_TABLE'
| 'ALTER_TABLE'
| 'TRUNCATE_TABLE'
| 'CREATE_INDEX'
| 'DROP_INDEX'
| 'BEGIN'
| 'COMMIT'
| 'ROLLBACK';
// ---------------------------------------------------------------------------
// 列引用
// ---------------------------------------------------------------------------
@@ -51,31 +34,10 @@ export interface JoinClause {
on: WhereCondition;
}
// ---------------------------------------------------------------------------
// 聚合函数
// ---------------------------------------------------------------------------
/** 聚合函数类型 */
export type AggregateFunc = 'COUNT' | 'SUM' | 'AVG' | 'MIN' | 'MAX';
/** 聚合表达式 */
export interface AggregateExpression {
type: 'AGGREGATE';
func: AggregateFunc;
column: string; // '*' for COUNT(*)
alias?: string;
}
// ---------------------------------------------------------------------------
// 子查询
// ---------------------------------------------------------------------------
/** 子查询表达式 */
export interface SubqueryExpression {
type: 'SUBQUERY';
statement: SelectStatement | SelectUnionStatement;
}
// ---------------------------------------------------------------------------
// DDL: CREATE TABLE
// ---------------------------------------------------------------------------
@@ -255,6 +217,35 @@ export interface RollbackTransactionStatement {
type: 'ROLLBACK';
}
// ---------------------------------------------------------------------------
// TCL: 维护语句（v0.5.1）
// ---------------------------------------------------------------------------
/** SAVEPOINT name / ROLLBACK TO SAVEPOINT name / RELEASE SAVEPOINT name */
export interface SavepointStatement {
type: 'SAVEPOINT';
name: string;
/** SAVE = 创建；ROLLBACK = 回滚到；RELEASE = 释放 */
action: 'SAVE' | 'ROLLBACK' | 'RELEASE';
}
/** ANALYZE TABLE name — 收集表统计信息 */
export interface AnalyzeStatement {
type: 'ANALYZE';
table: string;
}
/** REINDEX TABLE name — 重建表二级索引 */
export interface ReindexStatement {
type: 'REINDEX';
table: string;
}
/** VACUUM — 压缩 LSM + 清理碎片 */
export interface VacuumStatement {
type: 'VACUUM';
}
// ---------------------------------------------------------------------------
// AST 联合类型
// ---------------------------------------------------------------------------
@@ -274,4 +265,8 @@ export type Statement =
| DropIndexStatement
| BeginTransactionStatement
| CommitTransactionStatement
| RollbackTransactionStatement;
| RollbackTransactionStatement
| SavepointStatement
| AnalyzeStatement
| ReindexStatement
| VacuumStatement;
src/query/builder.ts
+16 -2
View File
@@ -135,14 +135,18 @@ export class SelectQueryBuilder {
export class UpdateQueryBuilder {
private _where: WhereCondition = {};
private onWrite?: (table: string) => void;
/** v0.5.1: CRUD hooks 触发回调 */
private onHooks?: (hook: import('../constants').HookName, args: unknown[]) => Promise<void>;
constructor(
private engine: IStorageEngine,
private tableName: string,
private _updates: Record<string, unknown>,
onWrite?: (table: string) => void,
onHooks?: (hook: import('../constants').HookName, args: unknown[]) => Promise<void>,
) {
this.onWrite = onWrite;
this.onHooks = onHooks;
}
where(condition: WhereCondition): this {
@@ -151,8 +155,11 @@ export class UpdateQueryBuilder {
}
async execute(): Promise<number> {
const count = await this.engine.update(this.tableName, { table: this.tableName, where: this._where }, this._updates);
const query = { table: this.tableName, where: this._where };
await this.onHooks?.('beforeUpdate', [query, this._updates]);
const count = await this.engine.update(this.tableName, query, this._updates);
this.onWrite?.(this.tableName);
await this.onHooks?.('afterUpdate', [query, this._updates, count]);
return count;
}
@@ -168,13 +175,17 @@ export class UpdateQueryBuilder {
export class DeleteQueryBuilder {
private _where: WhereCondition = {};
private onWrite?: (table: string) => void;
/** v0.5.1: CRUD hooks 触发回调 */
private onHooks?: (hook: import('../constants').HookName, args: unknown[]) => Promise<void>;
constructor(
private engine: IStorageEngine,
private tableName: string,
onWrite?: (table: string) => void,
onHooks?: (hook: import('../constants').HookName, args: unknown[]) => Promise<void>,
) {
this.onWrite = onWrite;
this.onHooks = onHooks;
}
where(condition: WhereCondition): this {
@@ -183,8 +194,11 @@ export class DeleteQueryBuilder {
}
async execute(): Promise<number> {
const count = await this.engine.delete(this.tableName, { table: this.tableName, where: this._where });
const query = { table: this.tableName, where: this._where };
await this.onHooks?.('beforeDelete', [query]);
const count = await this.engine.delete(this.tableName, query);
this.onWrite?.(this.tableName);
await this.onHooks?.('afterDelete', [query, count]);
return count;
}
src/query/executor.ts
+1315 -1244
View File
File diff suppressed because it is too large Load Diff
src/sql/parser.ts
+77 -1
View File
@@ -72,6 +72,19 @@ export class Parser {
return this.parseCommit();
case TokenType.ROLLBACK:
return this.parseRollback();
// v0.5.1: 维护语句入口
case TokenType.EXPLAIN:
return this.parseExplain();
case TokenType.ANALYZE:
return this.parseAnalyze();
case TokenType.REINDEX:
return this.parseReindex();
case TokenType.VACUUM:
return this.parseVacuum();
case TokenType.SAVEPOINT:
return this.parseSavepoint();
case TokenType.RELEASE:
return this.parseSavepoint();
default:
throw this.error(`Unexpected token "${this.curToken.value}"`);
}
@@ -95,6 +108,57 @@ export class Parser {
return statements;
}
// ---- 维护语句（v0.5.1） ----
/** EXPLAIN <statement> — 输出查询计划 */
private parseExplain(): import('../query/ast').ExplainStatement {
this.expect(TokenType.EXPLAIN);
if (this.curTokenIs(TokenType.EXPLAIN)) {
throw this.error('Nested EXPLAIN is not allowed');
}
const query = this.parseStatement();
return { type: 'EXPLAIN', query };
}
/** ANALYZE [TABLE] name — 收集表统计信息 */
private parseAnalyze(): import('../query/ast').AnalyzeStatement {
this.expect(TokenType.ANALYZE);
if (this._isKeywordAsIdent() && this.curToken.value.toUpperCase() === 'TABLE') {
this.nextToken();
}
return { type: 'ANALYZE', table: this.expectIdentifier('table name') };
}
/** REINDEX [TABLE] name — 重建表二级索引 */
private parseReindex(): import('../query/ast').ReindexStatement {
this.expect(TokenType.REINDEX);
if (this._isKeywordAsIdent() && this.curToken.value.toUpperCase() === 'TABLE') {
this.nextToken();
}
return { type: 'REINDEX', table: this.expectIdentifier('table name') };
}
/** VACUUM — 压缩 LSM + 清理碎片 */
private parseVacuum(): import('../query/ast').VacuumStatement {
this.expect(TokenType.VACUUM);
return { type: 'VACUUM' };
}
/** SAVEPOINT name | RELEASE [SAVEPOINT] name */
private parseSavepoint(): import('../query/ast').SavepointStatement {
let action: 'SAVE' | 'ROLLBACK' | 'RELEASE';
if (this.curTokenIs(TokenType.RELEASE)) {
action = 'RELEASE';
this.nextToken();
} else {
action = 'SAVE';
this.expect(TokenType.SAVEPOINT);
}
// 可选 SAVEPOINT 关键字（RELEASE SAVEPOINT name）
if (this.curTokenIs(TokenType.SAVEPOINT)) this.nextToken();
return { type: 'SAVEPOINT', name: this.expectIdentifier('savepoint name'), action };
}
// ---- 事务语句 ----
private parseBegin(): BeginTransactionStatement {
@@ -114,10 +178,22 @@ export class Parser {
return { type: 'COMMIT' };
}
private parseRollback(): RollbackTransactionStatement {
/** ROLLBACK [TRANSACTION] | ROLLBACK TO [SAVEPOINT] name（v0.5.1） */
private parseRollback(): Statement {
this.expect(TokenType.ROLLBACK);
if (this._isKeywordAsIdent() && this.curToken.value.toUpperCase() === 'TRANSACTION') {
this.nextToken();
return { type: 'ROLLBACK' };
}
// ROLLBACK TO [SAVEPOINT] name
if (this.curTokenIs(TokenType.TO) ||
(this._isKeywordAsIdent() && this.curToken.value.toUpperCase() === 'TO')) {
this.nextToken();
if (this.curTokenIs(TokenType.SAVEPOINT) ||
(this._isKeywordAsIdent() && this.curToken.value.toUpperCase() === 'SAVEPOINT')) {
this.nextToken();
}
return { type: 'SAVEPOINT', name: this.expectIdentifier('savepoint name'), action: 'ROLLBACK' };
}
return { type: 'ROLLBACK' };
}
src/sql/tokens.ts
+18
View File
@@ -83,6 +83,15 @@ export enum TokenType {
ELSE = 'ELSE',
END = 'END',
// v0.5.1: 维护语句（EXPLAIN / ANALYZE / REINDEX / VACUUM / SAVEPOINT）
EXPLAIN = 'EXPLAIN',
ANALYZE = 'ANALYZE',
REINDEX = 'REINDEX',
VACUUM = 'VACUUM',
SAVEPOINT = 'SAVEPOINT',
RELEASE = 'RELEASE',
TO = 'TO',
// 标识符 & 字面量
IDENTIFIER = 'IDENTIFIER',
STRING = 'STRING',
@@ -195,4 +204,13 @@ export const KEYWORDS: Record<string, TokenType> = {
'THEN': TokenType.THEN,
'ELSE': TokenType.ELSE,
'END': TokenType.END,
// v0.5.1
'EXPLAIN': TokenType.EXPLAIN,
'ANALYZE': TokenType.ANALYZE,
'REINDEX': TokenType.REINDEX,
'VACUUM': TokenType.VACUUM,
'SAVEPOINT': TokenType.SAVEPOINT,
'RELEASE': TokenType.RELEASE,
'TO': TokenType.TO,
};
src/table/table.ts
+17 -3
View File
@@ -20,12 +20,21 @@ export class Table<T = Record<string, unknown>> {
private executor: QueryExecutor | undefined;
/** 写入回调（多标签页广播，v0.3.2） */
private onWrite?: (table: string) => void;
/** v0.5.1: CRUD 生命周期钩子触发回调（beforeInsert/afterInsert/... 真实接线） */
private onHooks?: (hook: import('../constants').HookName, args: unknown[]) => Promise<void>;
constructor(engine: IStorageEngine, tableName: string, executor?: QueryExecutor, onWrite?: (table: string) => void) {
constructor(
engine: IStorageEngine,
tableName: string,
executor?: QueryExecutor,
onWrite?: (table: string) => void,
onHooks?: (hook: import('../constants').HookName, args: unknown[]) => Promise<void>,
) {
this.engine = engine;
this.name = tableName;
this.executor = executor;
this.onWrite = onWrite;
this.onHooks = onHooks;
}
// ---- Schema ----
@@ -42,14 +51,19 @@ export class Table<T = Record<string, unknown>> {
// ---- 插入 ----
async insert(row: T & Record<string, unknown>): Promise<string> {
// beforeInsert 参数约定：rows[]（单行时也是数组）
await this.onHooks?.('beforeInsert', [[row]]);
const pks = await this.engine.insert(this.name, [row as Record<string, unknown>]);
this.onWrite?.(this.name);
await this.onHooks?.('afterInsert', [[row], pks]);
return pks[0];
}
async insertMany(rows: (T & Record<string, unknown>)[]): Promise<string[]> {
await this.onHooks?.('beforeInsert', [rows]);
const pks = await this.engine.insert(this.name, rows as Record<string, unknown>[]);
this.onWrite?.(this.name);
await this.onHooks?.('afterInsert', [rows, pks]);
return pks;
}
@@ -87,13 +101,13 @@ export class Table<T = Record<string, unknown>> {
// ---- 更新 ----
update(updates: Partial<T> & Record<string, unknown>): UpdateQueryBuilder {
return new UpdateQueryBuilder(this.engine, this.name, updates, this.onWrite);
return new UpdateQueryBuilder(this.engine, this.name, updates, this.onWrite, this.onHooks);
}
// ---- 删除 ----
delete(): DeleteQueryBuilder {
return new DeleteQueryBuilder(this.engine, this.name, this.onWrite);
return new DeleteQueryBuilder(this.engine, this.name, this.onWrite, this.onHooks);
}
// ---- 聚合 ----
src/utils.ts
-80
View File
@@ -1,80 +0,0 @@
/**
* metona-sqlark Utils — 工具函数
* @module utils
*/
/** 生成唯一 ID */
export const generateId = (): string => {
return 'id-' + Date.now().toString(36) + '-' + Math.random().toString(36).slice(2, 8);
};
/** HTML 转义 */
export const escapeHTML = (s: unknown): string => {
const str = String(s == null ? '' : s);
if (typeof document === 'undefined') {
return str
.replace(/&/g, '&amp;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;')
.replace(/"/g, '&quot;')
.replace(/'/g, '&#039;');
}
const div = document.createElement('div');
div.textContent = str;
return div.innerHTML;
};
/** 防抖 */
export function debounce<T extends (...args: any[]) => void>(
func: T,
wait: number,
immediate = false,
): (...args: Parameters<T>) => void {
let timeout: ReturnType<typeof setTimeout> | null;
return function (this: any, ...args: Parameters<T>) {
const context = this;
const later = () => {
timeout = null;
if (!immediate) func.apply(context, args);
};
const callNow = immediate && !timeout;
if (timeout) clearTimeout(timeout);
timeout = setTimeout(later, wait);
if (callNow) func.apply(context, args);
};
}
/** 节流 */
export function throttle<T extends (...args: any[]) => void>(
func: T,
limit: number,
): (...args: Parameters<T>) => void {
let inThrottle = false;
return function (this: any, ...args: Parameters<T>) {
if (!inThrottle) {
func.apply(this, args);
inThrottle = true;
setTimeout(() => { inThrottle = false; }, limit);
}
};
}
/** 深度合并 */
export const deepMerge = <T extends Record<string, unknown>>(target: T, source: Partial<T>): T => {
const output: Record<string, unknown> = { ...target };
for (const key of Object.keys(source as Record<string, unknown>)) {
const sv = (source as Record<string, unknown>)[key];
const tv = (target as Record<string, unknown>)[key];
if (sv instanceof Object && key in target && tv instanceof Object) {
output[key] = deepMerge(tv as Record<string, unknown>, sv as Record<string, unknown>);
} else {
output[key] = source[key];
}
}
return output as T;
};
/** 浏览器环境检测 */
export const isBrowser = (): boolean => {
return typeof window !== 'undefined' && typeof document !== 'undefined';
};
tests/e2e/harness.html
+97
View File
@@ -0,0 +1,97 @@
<!DOCTYPE html>
<html lang="zh-CN">
<head>
<meta charset="UTF-8">
<title>MetonaSqlark E2E Harness</title>
</head>
<body>
<h1>MetonaSqlark E2E Harness</h1>
<div id="status">loading</div>
<script src="/dist/metona-sqlark.js"></script>
<script>
// E2E 驱动器：暴露全局 API，测试通过 page.evaluate 调用
// 使用高层 API（MetonaSqlark.create），保证 e2e 覆盖真实入口路径
window.__ms = {
version: () => (window.MetonaSqlark ? window.MetonaSqlark.VERSION : null),
/**
* 创建并打开 Aria 库。
* opts: {
* name, diskEngine ('opfs'|'indexeddb'|'memory'),
* walSyncMode, checkpointInterval,
* encryptionPassword, memtableSizeThreshold
* }
*/
open: async (opts) => {
const db = await window.MetonaSqlark.create({
name: opts.name,
mode: 'aria',
diskEngine: opts.diskEngine || 'opfs',
aria: {
walSyncMode: opts.walSyncMode || 'full',
checkpointInterval: opts.checkpointInterval || 100000,
memtableSizeThreshold: opts.memtableSizeThreshold || 64 * 1024 * 1024,
...(opts.encryptionPassword ? { encryption: { password: opts.encryptionPassword } } : {}),
},
});
window.__ms.__db = db;
return { ok: true, engineName: db.getEngine().name };
},
createTable: async (opts) => {
const db = window.__ms.__db;
const columns = {};
for (const col of opts.columns) {
columns[col.name] = { type: col.type, primaryKey: !!col.primaryKey, index: !!col.index };
}
await db.defineTable(opts.table, columns);
return { ok: true };
},
insert: async (opts) => {
const db = window.__ms.__db;
const pks = await db.table(opts.table).insertMany(opts.rows);
return { pks };
},
count: async (opts) => {
const db = window.__ms.__db;
return { count: await db.table(opts.table).count() };
},
find: async (opts) => {
const db = window.__ms.__db;
const rows = await db.table(opts.table).select().where(opts.where || {}).execute();
return { rows };
},
query: async (opts) => {
const db = window.__ms.__db;
return { rows: await db.query(opts.sql) };
},
repair: async () => {
const db = window.__ms.__db;
await db.repair();
return { ok: true };
},
clearAll: async () => {
const db = window.__ms.__db;
await db.clearAll();
return { ok: true };
},
close: async () => {
const db = window.__ms.__db;
await db.close();
window.__ms.__db = null;
return { ok: true };
},
getEngine: () => (window.__ms.__db ? window.__ms.__db.getEngine() : null),
};
document.getElementById('status').textContent = 'ready';
</script>
</body>
</html>
tests/e2e/opfs.spec.ts
+261
View File
@@ -0,0 +1,261 @@
/**
* MetonaSqlark E2E — 真实 Chromium + OPFS 环境验证
*
* 覆盖：
* 1. OPFS 持久化：写入 → 刷新页面 → 数据保留（跨页面生命周期）
* 2. 优雅关闭后重开：数据完整
* 3. 崩溃恢复：写入后强制终止（不 close）→ 重开 → WAL 重放数据完整
* 4. 多标签页锁：页 A 持锁 → 页 B 打开同一库 → ARIA_LOCKED
* 5. 加密库密码往返（真实 WebCrypto）
* 6. 页面化 + OPFS 端到端
* 7. repair() 自愈（真实文件系统）
*/
import { test, expect, type Page } from '@playwright/test';
const HARNESS = '/tests/e2e/harness.html';
async function openPage(page: Page): Promise<void> {
await page.goto(HARNESS);
await expect(page.locator('#status')).toHaveText('ready');
}
/** 在页面中执行 harness 方法（统一包装错误信息） */
async function run<T>(page: Page, method: string, args: unknown): Promise<T> {
const result = await page.evaluate(
([m, a]) => (window as unknown as { __ms: Record<string, (o: unknown) => Promise<unknown>> }).__ms[m](a),
[method, args] as const,
);
return result as T;
}
/** 强制"崩溃"当前页面：终止渲染进程（不触发 beforeunload/close） */
async function crashPage(page: Page): Promise<void> {
await page.evaluate(() => {
// 模拟崩溃：直接终止。使用 CDP Page.crash 等价效果 —— 通过无限循环让渲染进程被杀
// 但更可靠的方式是关闭页面连接（Playwright 无 Page.crash 公共 API），
// 这里用 context.close 前先把页面里的清理路径断开：
const win = window as unknown as { __ms: { __db: unknown } | null };
// 断开所有引用（模拟进程死亡：不调用 close）
win.__ms = null;
});
await page.close();
}
test.describe('OPFS 真实环境', () => {
test('写入 → 刷新页面 → 数据保留（持久化）', async ({ page }) => {
await openPage(page);
await run(page, 'open', { name: 'e2e-persist', diskEngine: 'opfs' });
await run(page, 'createTable', {
table: 'users',
columns: [
{ name: 'id', type: 'string', primaryKey: true },
{ name: 'name', type: 'string' },
{ name: 'age', type: 'number' },
],
});
await run(page, 'insert', {
table: 'users',
rows: [
{ id: '1', name: 'Alice', age: 30 },
{ id: '2', name: 'Bob', age: 25 },
],
});
expect(await run<{ count: number }>(page, 'count', { table: 'users' })).toEqual({ count: 2 });
// 刷新页面（完全重新加载，OPFS 数据必须保留）
await page.reload();
await expect(page.locator('#status')).toHaveText('ready');
await run(page, 'open', { name: 'e2e-persist', diskEngine: 'opfs' });
const res = await run<{ count: number }>(page, 'count', { table: 'users' });
expect(res.count).toBe(2);
const found = await run<{ rows: Record<string, unknown>[] }>(page, 'find', {
table: 'users', where: { id: '1' },
});
expect(found.rows[0].name).toBe('Alice');
await run(page, 'close');
});
test('优雅关闭后重开：数据完整', async ({ page }) => {
await openPage(page);
await run(page, 'open', { name: 'e2e-close-reopen', diskEngine: 'opfs' });
await run(page, 'createTable', {
table: 't', columns: [{ name: 'id', type: 'string', primaryKey: true }],
});
for (let i = 0; i < 20; i++) {
await run(page, 'insert', { table: 't', rows: [{ id: `r-${i}` }] });
}
await run(page, 'close');
await run(page, 'open', { name: 'e2e-close-reopen', diskEngine: 'opfs' });
expect(await run<{ count: number }>(page, 'count', { table: 't' })).toEqual({ count: 20 });
await run(page, 'close');
});
test('崩溃恢复：写入后强制终止 → 重开 WAL 重放数据完整', async ({ page }) => {
await openPage(page);
// 不 checkpoint（interval 极大），全部写入留在 WAL
await run(page, 'open', {
name: 'e2e-crash', diskEngine: 'opfs', checkpointInterval: 999999999,
});
await run(page, 'createTable', {
table: 'logs', columns: [{ name: 'id', type: 'string', primaryKey: true }],
});
const total = 50;
for (let i = 0; i < total; i++) {
await run(page, 'insert', { table: 'logs', rows: [{ id: `log-${i}` }] });
}
// 模拟崩溃：直接终止页面（不调用 close → 不 checkpoint）
await crashPage(page);
// 新页面重开：WAL 重放
const page2 = await page.context().newPage();
await openPage(page2);
await run(page2, 'open', {
name: 'e2e-crash', diskEngine: 'opfs', checkpointInterval: 999999999,
});
const res = await run<{ count: number }>(page2, 'count', { table: 'logs' });
expect(res.count).toBe(total);
await run(page2, 'close');
await page2.close();
});
test('多标签页锁：第二个标签页打开同一库 → ARIA_LOCKED', async ({ page }) => {
await openPage(page);
await run(page, 'open', { name: 'e2e-lock', diskEngine: 'opfs' });
await run(page, 'createTable', {
table: 't', columns: [{ name: 'id', type: 'string', primaryKey: true }],
});
// 第二个标签页尝试打开同一库
const page2 = await page.context().newPage();
await openPage(page2);
const err = await page2.evaluate(async () => {
try {
await (window as unknown as { __ms: { open: (o: unknown) => Promise<unknown> } }).__ms.open(
{ name: 'e2e-lock', diskEngine: 'opfs' },
);
return { ok: true };
} catch (e) {
return { ok: false, code: (e as { code?: string }).code, msg: (e as Error).message };
}
});
expect(err.ok).toBe(false);
expect((err as { code?: string }).code).toBe('ARIA_LOCKED');
// 第一个标签页关闭后 → 第二标签页可打开
await run(page, 'close');
const retry = await page2.evaluate(async () => {
const ms = (window as unknown as { __ms: { open: (o: unknown) => Promise<unknown>; close: () => Promise<unknown> } }).__ms;
await ms.open({ name: 'e2e-lock', diskEngine: 'opfs' });
return { ok: true };
});
expect(retry.ok).toBe(true);
await run(page2, 'close');
await page2.close();
});
test('加密库：密码往返（真实 WebCrypto AES-GCM）', async ({ page }) => {
await openPage(page);
await run(page, 'open', { name: 'e2e-enc', diskEngine: 'opfs', encryptionPassword: 'correct-horse' });
await run(page, 'createTable', {
table: 'secrets',
columns: [
{ name: 'id', type: 'string', primaryKey: true },
{ name: 'payload', type: 'string' },
],
});
await run(page, 'insert', {
table: 'secrets',
rows: [{ id: 's1', payload: 'top-secret-content' }],
});
await run(page, 'close');
// 错误密码 → 打开失败
const page2 = await page.context().newPage();
await openPage(page2);
const err = await page2.evaluate(async () => {
try {
await (window as unknown as { __ms: { open: (o: unknown) => Promise<unknown> } }).__ms.open(
{ name: 'e2e-enc', diskEngine: 'opfs', encryptionPassword: 'wrong-password' },
);
return { ok: true };
} catch (e) {
return { ok: false, code: (e as { code?: string }).code };
}
});
expect((err as { code?: string }).code).toBe('ARIA_DECRYPT_ERROR');
// 正确密码 → 数据完整
await run(page2, 'open', { name: 'e2e-enc', diskEngine: 'opfs', encryptionPassword: 'correct-horse' });
const found = await run<{ rows: Record<string, unknown>[] }>(page2, 'find', {
table: 'secrets', where: { id: 's1' },
});
expect(found.rows[0].payload).toBe('top-secret-content');
await run(page2, 'close');
await page2.close();
});
test('OPFS 页面化端到端：大数据量 + 二级索引查询', async ({ page }) => {
await openPage(page);
await run(page, 'open', { name: 'e2e-paged', diskEngine: 'opfs' });
await run(page, 'createTable', {
table: 'docs',
columns: [
{ name: 'id', type: 'string', primaryKey: true },
{ name: 'tag', type: 'string', index: true },
{ name: 'body', type: 'string' },
],
});
const total = 100;
for (let batch = 0; batch < 5; batch++) {
const rows = [];
for (let i = 0; i < 20; i++) {
const idx = batch * 20 + i;
rows.push({ id: `d-${idx}`, tag: idx % 3 === 0 ? 'red' : 'blue', body: '页面化端到端内容'.repeat(30) });
}
await run(page, 'insert', { table: 'docs', rows });
}
expect(await run<{ count: number }>(page, 'count', { table: 'docs' })).toEqual({ count: total });
// 二级索引查询（真实浏览器中走 LSM 索引）
const reds = await run<{ rows: Record<string, unknown>[] }>(page, 'find', {
table: 'docs', where: { tag: 'red' },
});
expect(reds.rows.length).toBeGreaterThan(0);
// SQL 查询
const sql = await run<{ rows: Record<string, unknown>[] }>(page, 'query', {
sql: 'SELECT COUNT(*) AS n FROM docs WHERE tag = \'red\'',
});
expect(Number(sql.rows[0].n)).toBe(reds.rows.length);
await run(page, 'close');
});
test('repair() 自愈（真实文件系统）', async ({ page }) => {
await openPage(page);
await run(page, 'open', { name: 'e2e-repair', diskEngine: 'opfs' });
await run(page, 'createTable', {
table: 't',
columns: [
{ name: 'id', type: 'string', primaryKey: true },
{ name: 'v', type: 'string' },
],
});
await run(page, 'insert', { table: 't', rows: [{ id: 'a', v: 'keep-me' }] });
// 通过 SQL 触发 checkpoint + 手动 repair
await run(page, 'repair');
expect(await run<{ count: number }>(page, 'count', { table: 't' })).toEqual({ count: 1 });
// clearAll 后重建表（表结构随 clearAll 清空）
await run(page, 'clearAll');
await run(page, 'createTable', {
table: 't',
columns: [
{ name: 'id', type: 'string', primaryKey: true },
{ name: 'v', type: 'string' },
],
});
expect(await run<{ count: number }>(page, 'count', { table: 't' })).toEqual({ count: 0 });
await run(page, 'close');
});
});
tests/e2e/server.cjs
+39
View File
@@ -0,0 +1,39 @@
#!/usr/bin/env node
/**
* MetonaSqlark E2E 静态文件服务器（Playwright webServer 用）
* 与 MetonaEditor 的 e2e 服务模式保持一致：Node http 静态服务，不依赖 python3。
* 用法: node tests/e2e/server.cjs [port]
*/
const http = require('http');
const fs = require('fs');
const path = require('path');
const ROOT = path.resolve(__dirname, '..', '..');
const PORT = Number(process.argv[2] || process.env.E2E_PORT || 3344);
const MIME = {
'.html': 'text/html; charset=utf-8',
'.js': 'text/javascript',
'.mjs': 'text/javascript',
'.cjs': 'text/javascript',
'.css': 'text/css',
'.json': 'application/json',
'.svg': 'image/svg+xml',
'.png': 'image/png',
};
const server = http.createServer((req, res) => {
let p = decodeURIComponent((req.url || '/').split('?')[0]);
// 根路径指向 e2e harness（playwright webServer 探测要求 2xx/3xx）
if (p === '/') p = '/tests/e2e/harness.html';
const file = path.resolve(ROOT, '.' + p);
if (!file.startsWith(ROOT) || !fs.existsSync(file) || !fs.statSync(file).isFile()) {
res.writeHead(404); res.end('404 Not Found'); return;
}
res.writeHead(200, { 'Content-Type': MIME[path.extname(file)] || 'text/plain' });
fs.createReadStream(file).pipe(res);
});
server.listen(PORT, '127.0.0.1', () => {
console.log(`E2E server · http://127.0.0.1:${PORT}/`);
});
tests/engine/aria-advanced.test.ts
+9 -8
View File
@@ -113,10 +113,10 @@ describe('AriaEngine — Bloom + WAL + BufferPool', () => {
freePageId: async (_id: number) => {},
};
const pool = new BufferPool(pageIO, 4);
const page = await pool.newPage();
expect(page.pageId).toBeGreaterThan(0);
expect(page.pins).toBe(1);
pool.unpin(page);
const pages = await pool.newPages(1);
expect(pages[0].pageId).toBeGreaterThan(0);
expect(pages[0].pins).toBe(1);
pool.unpin(pages[0]);
await backend.close();
});
@@ -130,11 +130,12 @@ describe('AriaEngine — Bloom + WAL + BufferPool', () => {
freePageId: async (_id: number) => {},
};
const pool = new BufferPool(pageIO, 4);
const page = await pool.newPage();
pool.markDirty(page);
pool.unpin(page);
const pages = await pool.newPages(1);
pool.markDirty(pages[0]);
pool.unpin(pages[0]);
await pool.flushAll();
expect(pool.getDirtyPageCount()).toBe(0);
const buf = await backend.read('pg_1');
expect(buf).not.toBeNull();
await backend.close();
});
});
tests/engine/aria-buffer.test.ts
+240 -273
View File
@@ -1,273 +1,240 @@
/**
* AriaEngine Buffer Pool + Eviction 单元测试
*/
import { BufferPool, type PageIO } from '../../src/engine/aria/buffer/pool';
import { LRUList, EvictionManager } from '../../src/engine/aria/buffer/eviction';
import { PageType, PAGE_SIZE } from '../../src/engine/aria/types';
import { initPageHeader } from '../../src/engine/aria/page/header';
// ===================================================================
// LRUList
// ===================================================================
describe('AriaEngine — LRUList', () => {
function makePage(id: number) {
const data = new ArrayBuffer(PAGE_SIZE);
initPageHeader(data, id, PageType.DATA);
return { pageId: id, type: PageType.DATA, data, dirty: false, pins: 0, prev: null, next: null, lastAccess: Date.now() };
}
it('moveToHead — 单元素', () => {
const list = new LRUList();
const p = makePage(1);
list.moveToHead(p);
expect(list.size).toBe(1);
});
it('moveToHead — 多元素保持 MRU 顺序', () => {
const list = new LRUList();
const a = makePage(1);
const b = makePage(2);
const c = makePage(3);
list.moveToHead(a);
list.moveToHead(b);
list.moveToHead(c);
expect(list.size).toBe(3);
// c 是最新的
});
it('getLRU 返回最久未使用', () => {
const list = new LRUList();
const a = makePage(1);
const b = makePage(2);
list.moveToHead(a);
list.moveToHead(b);
expect(list.getLRU()!.pageId).toBe(1);
});
it('popLRU 移除并返回最久未使用', () => {
const list = new LRUList();
const a = makePage(1);
const b = makePage(2);
list.moveToHead(a);
list.moveToHead(b);
const popped = list.popLRU();
expect(popped!.pageId).toBe(1);
expect(list.size).toBe(1);
});
it('remove — 从中间移除', () => {
const list = new LRUList();
const a = makePage(1);
const b = makePage(2);
const c = makePage(3);
list.moveToHead(a);
list.moveToHead(b);
list.moveToHead(c);
list.remove(b);
expect(list.size).toBe(2);
});
it('clear 清空', () => {
const list = new LRUList();
list.moveToHead(makePage(1));
list.moveToHead(makePage(2));
list.clear();
expect(list.size).toBe(0);
});
it('getAllPages 返回所有页面', () => {
const list = new LRUList();
list.moveToHead(makePage(1));
list.moveToHead(makePage(2));
expect(list.getAllPages()).toHaveLength(2);
});
it('空 LRU getLRU 返回 null', () => {
const list = new LRUList();
expect(list.getLRU()).toBeNull();
});
it('空 LRU popLRU 返回 null', () => {
const list = new LRUList();
expect(list.popLRU()).toBeNull();
});
it('moveToHead 同元素不移重复', () => {
const list = new LRUList();
const p = makePage(1);
list.moveToHead(p);
list.moveToHead(p);
list.moveToHead(p);
expect(list.size).toBe(1);
});
});
// ===================================================================
// EvictionManager
// ===================================================================
describe('AriaEngine — EvictionManager', () => {
function makePage(id: number, dirty = false) {
const data = new ArrayBuffer(PAGE_SIZE);
initPageHeader(data, id, PageType.DATA);
return { pageId: id, type: PageType.DATA, data, dirty, pins: 0, prev: null, next: null, lastAccess: Date.now() };
}
it('access 更新 LRU', async () => {
let evicted = -1;
const em = new EvictionManager(3, async (p) => { evicted = p.pageId; });
const p = makePage(1);
em.add(p);
em.access(p);
expect(em.getSize()).toBe(1);
});
it('add 不超过容量不触发驱逐', async () => {
const evictedPages: number[] = [];
const em = new EvictionManager(4, async (p) => { evictedPages.push(p.pageId); });
em.add(makePage(1));
em.add(makePage(2));
em.add(makePage(3));
await em.evictIfNeeded(1);
expect(evictedPages).toHaveLength(0);
expect(em.getSize()).toBe(3);
});
it('evictIfNeeded 超容量触发驱逐', async () => {
const evictedPages: number[] = [];
const em = new EvictionManager(2, async (p) => { evictedPages.push(p.pageId); });
em.add(makePage(1));
em.add(makePage(2));
em.add(makePage(3)); // 超容量
await em.evictIfNeeded(0);
// 驱逐后 size 应 <= 2
expect(em.getSize()).toBeLessThanOrEqual(2);
});
it('remove 减少 size', () => {
const em = new EvictionManager(4, async () => {});
const p = makePage(1);
em.add(p);
em.add(makePage(2));
em.remove(p);
expect(em.getSize()).toBe(1);
});
it('getCapacity 返回配置容量', () => {
const em = new EvictionManager(128, async () => {});
expect(em.getCapacity()).toBe(128);
});
it('clear 清空', () => {
const em = new EvictionManager(4, async () => {});
em.add(makePage(1));
em.add(makePage(2));
em.clear();
expect(em.getSize()).toBe(0);
});
it('脏页驱逐前调用 onEvict 回调', async () => {
let flushed = 0;
const em = new EvictionManager(2, async (_p) => { flushed++; });
em.add(makePage(1, true)); // dirty page
em.add(makePage(2));
await em.evictIfNeeded(1); // need space → evict page 1
expect(flushed).toBeGreaterThanOrEqual(0); // May or may not evict
});
});
// ===================================================================
// BufferPool (with Mock PageIO)
// ===================================================================
describe('AriaEngine — BufferPool', () => {
class MockPageIO implements PageIO {
store = new Map<number, ArrayBuffer>();
nextId = 1;
reads = 0;
writes = 0;
async readPage(pageId: number) { this.reads++; return this.store.get(pageId) ?? null; }
async writePage(pageId: number, data: ArrayBuffer) { this.writes++; this.store.set(pageId, data); }
async allocatePageId() { return this.nextId++; }
async freePageId(_pageId: number) {}
}
it('newPage 创建页面并 pin', async () => {
const io = new MockPageIO();
const pool = new BufferPool(io, 4);
const page = await pool.newPage();
expect(page.pageId).toBe(1);
expect(page.pins).toBe(1);
expect(page.type).toBe(PageType.DATA);
pool.unpin(page);
});
it('getPage — 池中已存在则 pin++', async () => {
const io = new MockPageIO();
const pool = new BufferPool(io, 4);
const p1 = await pool.newPage();
pool.unpin(p1);
const p2 = await pool.getPage(p1.pageId);
expect(p2!.pageId).toBe(p1.pageId);
expect(p2!.pins).toBe(1);
pool.unpin(p2!);
});
it('markDirty + flushPage 写回', async () => {
const io = new MockPageIO();
const pool = new BufferPool(io, 4);
const page = await pool.newPage();
new Uint8Array(page.data)[100] = 42;
pool.markDirty(page);
pool.unpin(page);
await pool.flushPage(page.pageId);
expect(io.writes).toBeGreaterThan(0);
});
it('flushAll 刷新所有脏页', async () => {
const io = new MockPageIO();
const pool = new BufferPool(io, 8);
const p1 = await pool.newPage();
const p2 = await pool.newPage();
pool.markDirty(p1);
pool.markDirty(p2);
pool.unpin(p1);
pool.unpin(p2);
await pool.flushAll();
expect(io.writes).toBe(2);
});
it('getCachedPageCount 返回缓存数', async () => {
const io = new MockPageIO();
const pool = new BufferPool(io, 4);
await pool.newPage();
await pool.newPage();
expect(pool.getCachedPageCount()).toBe(2);
});
it('getDirtyPageCount 返回脏页数', async () => {
const io = new MockPageIO();
const pool = new BufferPool(io, 4);
const p = await pool.newPage();
pool.markDirty(p);
pool.unpin(p);
expect(pool.getDirtyPageCount()).toBe(1);
});
it('getCapacity 返回容量', async () => {
const io = new MockPageIO();
const pool = new BufferPool(io, 64);
expect(pool.getCapacity()).toBe(64);
});
it('removePage 移除缓存', async () => {
const io = new MockPageIO();
const pool = new BufferPool(io, 4);
const p = await pool.newPage();
pool.unpin(p);
pool.removePage(p.pageId);
expect(pool.getCachedPageCount()).toBe(0);
});
});
/**
* AriaEngine Buffer Pool + Eviction 单元测试
*/
import { BufferPool, type PageIO } from '../../src/engine/aria/buffer/pool';
import { LRUList, EvictionManager } from '../../src/engine/aria/buffer/eviction';
import { PageType, PAGE_SIZE } from '../../src/engine/aria/types';
import { initPageHeader } from '../../src/engine/aria/page/header';
// ===================================================================
// LRUList
// ===================================================================
describe('AriaEngine — LRUList', () => {
function makePage(id: number) {
const data = new ArrayBuffer(PAGE_SIZE);
initPageHeader(data, id, PageType.DATA);
return { pageId: id, type: PageType.DATA, data, dirty: false, pins: 0, prev: null, next: null, lastAccess: Date.now() };
}
it('moveToHead — 单元素', () => {
const list = new LRUList();
const p = makePage(1);
list.moveToHead(p);
expect(list.size).toBe(1);
});
it('moveToHead — 多元素保持 MRU 顺序', () => {
const list = new LRUList();
const a = makePage(1);
const b = makePage(2);
const c = makePage(3);
list.moveToHead(a);
list.moveToHead(b);
list.moveToHead(c);
expect(list.size).toBe(3);
// c 是最新的
});
it('getLRU 返回最久未使用', () => {
const list = new LRUList();
const a = makePage(1);
const b = makePage(2);
list.moveToHead(a);
list.moveToHead(b);
expect(list.getLRU()!.pageId).toBe(1);
});
it('remove — 从中间移除', () => {
const list = new LRUList();
const a = makePage(1);
const b = makePage(2);
const c = makePage(3);
list.moveToHead(a);
list.moveToHead(b);
list.moveToHead(c);
list.remove(b);
expect(list.size).toBe(2);
});
it('clear 清空', () => {
const list = new LRUList();
list.moveToHead(makePage(1));
list.moveToHead(makePage(2));
list.clear();
expect(list.size).toBe(0);
});
it('remove 从尾部移除', () => {
const list = new LRUList();
const a = makePage(1);
list.moveToHead(a);
list.remove(a);
expect(list.size).toBe(0);
});
it('空 LRU getLRU 返回 null', () => {
const list = new LRUList();
expect(list.getLRU()).toBeNull();
});
it('moveToHead 同元素不移重复', () => {
const list = new LRUList();
const p = makePage(1);
list.moveToHead(p);
list.moveToHead(p);
list.moveToHead(p);
expect(list.size).toBe(1);
});
});
// ===================================================================
// EvictionManager
// ===================================================================
describe('AriaEngine — EvictionManager', () => {
function makePage(id: number, dirty = false) {
const data = new ArrayBuffer(PAGE_SIZE);
initPageHeader(data, id, PageType.DATA);
return { pageId: id, type: PageType.DATA, data, dirty, pins: 0, prev: null, next: null, lastAccess: Date.now() };
}
it('access 更新 LRU', async () => {
let evicted = -1;
const em = new EvictionManager(3, async (p) => { evicted = p.pageId; });
const p = makePage(1);
em.add(p);
em.access(p);
expect((em as unknown as { lru: { size: number } }).lru.size).toBe(1);
});
it('add 不超过容量不触发驱逐', async () => {
const evictedPages: number[] = [];
const em = new EvictionManager(4, async (p) => { evictedPages.push(p.pageId); });
em.add(makePage(1));
em.add(makePage(2));
em.add(makePage(3));
await em.evictIfNeeded(1);
expect(evictedPages).toHaveLength(0);
expect((em as unknown as { lru: { size: number } }).lru.size).toBe(3);
});
it('evictIfNeeded 超容量触发驱逐', async () => {
const evictedPages: number[] = [];
const em = new EvictionManager(2, async (p) => { evictedPages.push(p.pageId); });
em.add(makePage(1));
em.add(makePage(2));
em.add(makePage(3)); // 超容量
await em.evictIfNeeded(0);
// 驱逐后 size 应 <= 2
expect((em as unknown as { lru: { size: number } }).lru.size).toBeLessThanOrEqual(2);
});
it('remove 减少 size', () => {
const em = new EvictionManager(4, async () => {});
const p = makePage(1);
em.add(p);
em.add(makePage(2));
em.remove(p);
expect((em as unknown as { lru: { size: number } }).lru.size).toBe(1);
});
it('clear 清空', () => {
const em = new EvictionManager(4, async () => {});
em.add(makePage(1));
em.add(makePage(2));
em.clear();
expect((em as unknown as { lru: { size: number } }).lru.size).toBe(0);
});
it('脏页驱逐前调用 onEvict 回调', async () => {
let flushed = 0;
const em = new EvictionManager(2, async (_p) => { flushed++; });
em.add(makePage(1, true)); // dirty page
em.add(makePage(2));
await em.evictIfNeeded(1); // need space → evict page 1
expect(flushed).toBeGreaterThanOrEqual(0); // May or may not evict
});
});
// ===================================================================
// BufferPool (with Mock PageIO)
// ===================================================================
describe('AriaEngine — BufferPool', () => {
class MockPageIO implements PageIO {
store = new Map<number, ArrayBuffer>();
nextId = 1;
reads = 0;
writes = 0;
async readPage(pageId: number) { this.reads++; return this.store.get(pageId) ?? null; }
async writePage(pageId: number, data: ArrayBuffer) { this.writes++; this.store.set(pageId, data); }
async allocatePageId() { return this.nextId++; }
async freePageId(_pageId: number) {}
}
it('newPage 创建页面并 pin', async () => {
const io = new MockPageIO();
const pool = new BufferPool(io, 4);
const page = (await pool.newPages(1))[0];
expect(page.pageId).toBe(1);
expect(page.pins).toBe(1);
expect(page.type).toBe(PageType.DATA);
pool.unpin(page);
});
it('getPage — 池中已存在则 pin++', async () => {
const io = new MockPageIO();
const pool = new BufferPool(io, 4);
const p1 = (await pool.newPages(1))[0];
pool.unpin(p1);
const p2 = await pool.getPage(p1.pageId);
expect(p2!.pageId).toBe(p1.pageId);
expect(p2!.pins).toBe(1);
pool.unpin(p2!);
});
it('markDirty + flushPage 写回', async () => {
const io = new MockPageIO();
const pool = new BufferPool(io, 4);
const page = (await pool.newPages(1))[0];
new Uint8Array(page.data)[100] = 42;
pool.markDirty(page);
pool.unpin(page);
await pool.flushPage(page.pageId);
expect(io.writes).toBeGreaterThan(0);
});
it('flushAll 刷新所有脏页', async () => {
const io = new MockPageIO();
const pool = new BufferPool(io, 8);
const p1 = (await pool.newPages(1))[0];
const p2 = (await pool.newPages(1))[0];
pool.markDirty(p1);
pool.markDirty(p2);
pool.unpin(p1);
pool.unpin(p2);
await pool.flushAll();
expect(io.writes).toBe(2);
});
it('newPages 批量创建页面并 pin', async () => {
const io = new MockPageIO();
const pool = new BufferPool(io, 8);
const pages = await pool.newPages(3);
expect(pages.map((p) => p.pageId)).toEqual([1, 2, 3]);
expect(pages.every((p) => p.pins === 1)).toBe(true);
expect(pages.every((p) => p.type === PageType.DATA)).toBe(true);
for (const p of pages) pool.unpin(p);
});
it('removePage 移除缓存', async () => {
const io = new MockPageIO();
const pool = new BufferPool(io, 4);
const p = (await pool.newPages(1))[0];
pool.unpin(p);
pool.removePage(p.pageId);
expect((pool as unknown as { pages: Map<number, unknown> }).pages.size).toBe(0);
});
});
tests/engine/aria-cache.test.ts
+3 -3
View File
@@ -53,7 +53,7 @@ describe('AriaEngine SSTable 缓存内存上限', () => {
for (let round = 0; round < 5; round++) {
const rows = await engine.find('users', { table: 'users', where: { age: 25 } });
expect(rows.length).toBe(10);
expect(lsm.getCacheSize()).toBeLessThanOrEqual(lsm.getCacheLimit());
expect(lsm.cacheSize).toBeLessThanOrEqual(lsm.cacheLimitBytes);
}
await engine.close();
@@ -74,7 +74,7 @@ describe('AriaEngine SSTable 缓存内存上限', () => {
expect(all.length).toBe(300);
const lsm = (engine as any).lsm;
expect(lsm.getCacheSize()).toBeLessThanOrEqual(lsm.getCacheLimit());
expect(lsm.cacheSize).toBeLessThanOrEqual(lsm.cacheLimitBytes);
await engine.close();
});
@@ -165,7 +165,7 @@ describe('AriaEngine SSTable 缓存内存上限', () => {
for (let batch = 0; batch < 10; batch++) {
await engine.insert('users', makeRows(30).map((r, i) => ({ ...r, id: `b${batch}_u${i}` })));
const lsm = (engine as any).lsm;
expect(lsm.getCacheSize()).toBeLessThanOrEqual(lsm.getCacheLimit());
expect(lsm.cacheSize).toBeLessThanOrEqual(lsm.cacheLimitBytes);
}
const all = await engine.find('users', { table: 'users' });
tests/engine/aria-checksum.test.ts
+181
View File
@@ -0,0 +1,181 @@
/**
* AriaEngine — SSTable CRC-32 损坏检测集成测试
*
* 覆盖：
* 1. 打开时整文件 CRC 校验失败 → 损坏 SSTable 被清理（自愈），打开不阻塞
* 2. 运行期预加载 CRC 校验失败 → 损坏文件不缓存并被清理
* 3. 旧版无校验文件（checksum=0）在 LSM 层仍正常加载
*/
import { AriaEngine } from '../../src/engine/aria/index';
import { createSchema } from '../../src/table/schema';
import 'fake-indexeddb/auto';
let idbCounter = 0;
function uniqueDB(): string {
return `crc-${Date.now()}-${++idbCounter}-${Math.random().toString(36).slice(2, 8)}`;
}
const SCHEMA = () => createSchema('users', {
id: { type: 'string', primaryKey: true },
name: { type: 'string' },
age: { type: 'number' },
});
/** 篡改 MemoryBackend 中指定 key 的一个字节 */
async function corruptKey(engine: AriaEngine, key: string, byteOffset: number): Promise<void> {
const backend = (engine as any).backend;
const raw = await backend.read(key);
expect(raw).not.toBeNull();
const buf = new Uint8Array(raw as ArrayBuffer);
buf[Math.min(byteOffset, buf.length - 1)] ^= 0xff;
await backend.write(key, buf.buffer as ArrayBuffer);
}
/** 列出 backend 中所有 sst_ 前缀 key */
async function listSSTKeys(engine: AriaEngine): Promise<string[]> {
const backend = (engine as any).backend;
const keys = await backend.listKeys();
return (keys as string[]).filter((k) => k.startsWith('sst_'));
}
describe('AriaEngine — SSTable CRC 损坏检测（集成）', () => {
it('打开时 CRC 损坏的 SSTable 被清理，其余数据可读', async () => {
const engine = new AriaEngine({ storageBackend: 'indexeddb', memtableSizeThreshold: 64 * 1024 * 1024 });
const dbName = uniqueDB();
await engine.open(dbName, 1);
await engine.createTable(SCHEMA());
// 写入 200 行（触发多次 flush → 多个 SSTable）
for (let batch = 0; batch < 3; batch++) {
const rows = [] as Record<string, unknown>[];
for (let i = 0; i < 50; i++) {
rows.push({ id: `u-${batch * 50 + i}`, name: `User${batch * 50 + i}`, age: batch * 50 + i });
}
await engine.insert('users', rows);
await (engine as any).lsm.flush();
}
await engine.close();
// 重新打开，篡改第一个 SSTable 文件的数据区
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', memtableSizeThreshold: 64 * 1024 * 1024 });
await engine2.open(dbName, 1);
const sstKeys = await listSSTKeys(engine2);
expect(sstKeys.length).toBeGreaterThanOrEqual(2);
await corruptKey(engine2, sstKeys[0], 64);
// 再次打开：损坏文件应被跳过并清理，打开不抛错
await engine2.close();
const engine3 = new AriaEngine({ storageBackend: 'indexeddb', memtableSizeThreshold: 64 * 1024 * 1024 });
await engine3.open(dbName, 1);
// 剩余未损坏文件的数据应可查询
const remaining = await engine3.find('users', { table: 'users' });
expect(remaining.length).toBeGreaterThan(0);
expect(remaining.length).toBeLessThan(200);
// 损坏文件已被清理（meta 移除）
const afterKeys = await listSSTKeys(engine3);
expect(afterKeys).not.toContain(sstKeys[0]);
const lsm = (engine3 as any).lsm;
const metas = await lsm.sstableStore.listMeta();
expect(metas.some((m: { id: number }) => m.id === Number(sstKeys[0].replace('sst_', '')))).toBe(false);
await engine3.close();
});
it('打开时损坏全部 SSTable → 库仍可打开，数据为空但不崩溃', async () => {
const engine = new AriaEngine({ storageBackend: 'indexeddb', memtableSizeThreshold: 64 * 1024 * 1024 });
const dbName = uniqueDB();
await engine.open(dbName, 1);
await engine.createTable(SCHEMA());
for (let batch = 0; batch < 2; batch++) {
const rows = [] as Record<string, unknown>[];
for (let i = 0; i < 50; i++) {
rows.push({ id: `u-${batch * 50 + i}`, name: `User${batch * 50 + i}`, age: batch * 50 + i });
}
await engine.insert('users', rows);
await (engine as any).lsm.flush();
}
await engine.close();
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', memtableSizeThreshold: 64 * 1024 * 1024 });
await engine2.open(dbName, 1);
for (const key of await listSSTKeys(engine2)) {
await corruptKey(engine2, key, 16);
}
await engine2.close();
const engine3 = new AriaEngine({ storageBackend: 'indexeddb', memtableSizeThreshold: 64 * 1024 * 1024 });
// 不应抛 ARIA_OPEN_ERROR
await engine3.open(dbName, 1);
const rows3 = await engine3.find('users', { table: 'users' });
expect(rows3.length).toBe(0);
await engine3.close();
});
it('运行期 CRC 损坏 → 预加载不缓存损坏文件并清理（自愈）', async () => {
const engine = new AriaEngine({ storageBackend: 'indexeddb', memtableSizeThreshold: 64 * 1024 * 1024 });
const dbName = uniqueDB();
await engine.open(dbName, 1);
await engine.createTable(SCHEMA());
for (let batch = 0; batch < 2; batch++) {
const rows = [] as Record<string, unknown>[];
for (let i = 0; i < 50; i++) {
rows.push({ id: `u-${batch * 50 + i}`, name: `User${batch * 50 + i}`, age: batch * 50 + i });
}
await engine.insert('users', rows);
await (engine as any).lsm.flush();
}
// 运行期：直接篡改后端中的某个 SSTable 文件（不触发打开校验路径）
const sstKeys = await listSSTKeys(engine);
expect(sstKeys.length).toBeGreaterThanOrEqual(2);
const victim = sstKeys[sstKeys.length - 1]; // 篡改最新（memtable 已 flush 后的文件）
await corruptKey(engine, victim, 100);
// 缓存里已有该文件（flush 时缓存）→ 先清缓存模拟运行期重载
const lsm = (engine as any).lsm;
lsm.sstableCache.delete(Number(victim.replace('sst_', '')));
// 查询触发 prefetchRange → preloadSSTable 发现 CRC 失败 → 清理 + 不缓存
const remaining = await engine.find('users', { table: 'users' });
expect(remaining.length).toBeLessThan(100);
expect(lsm.sstableCache.has(Number(victim.replace('sst_', '')))).toBe(false);
const metas = await lsm.sstableStore.listMeta();
expect(metas.some((m: { id: number }) => m.id === Number(victim.replace('sst_', '')))).toBe(false);
await engine.close();
});
it('旧版无校验文件（checksum=0）在 LSM 中正常加载', async () => {
const engine = new AriaEngine({ storageBackend: 'indexeddb', memtableSizeThreshold: 64 * 1024 * 1024 });
const dbName = uniqueDB();
await engine.open(dbName, 1);
await engine.createTable(SCHEMA());
await engine.insert('users', [
{ id: 'a', name: 'Alice', age: 30 },
{ id: 'b', name: 'Bob', age: 25 },
]);
await (engine as any).lsm.flush();
await engine.close();
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', memtableSizeThreshold: 64 * 1024 * 1024 });
await engine2.open(dbName, 1);
const sstKeys = await listSSTKeys(engine2);
expect(sstKeys.length).toBe(1);
const backend = (engine2 as any).backend;
const raw = await backend.read(sstKeys[0]);
const buf = new Uint8Array(raw as ArrayBuffer);
new DataView(buf.buffer).setUint32(buf.byteLength - 4, 0, false); // checksum 清零 → 旧版
await backend.write(sstKeys[0], buf.buffer as ArrayBuffer);
await engine2.close();
// 重开：checksum=0 跳过校验，数据完整可读
const engine3 = new AriaEngine({ storageBackend: 'indexeddb', memtableSizeThreshold: 64 * 1024 * 1024 });
await engine3.open(dbName, 1);
const rows3 = await engine3.find('users', { table: 'users' });
expect(rows3.length).toBe(2);
expect(rows3.map((r) => r.id).sort()).toEqual(['a', 'b']);
await engine3.close();
});
});
tests/engine/aria-compress.test.ts
+250 -218
View File
@@ -1,218 +1,250 @@
/**
* AriaEngine LZ4 压缩 + LSM Merge Iterator 单元测试
* 注：LZ4 为简化演示实现（默认 compression:false），测试聚焦于「不卡死」
*/
import { compressLZ4, decompressLZ4 } from '../../src/engine/aria/compression/lz4';
import { MergeIterator, ArrayEntrySource } from '../../src/engine/aria/index/merge_iterator';
// ===================================================================
// LZ4 压缩 — 安全烟雾测试（不卡死）
// ===================================================================
describe('AriaEngine — LZ4 Compression', () => {
it('短于 4 字节时压缩为纯字面量 token 且往返一致', () => {
const input = new Uint8Array([1, 2]);
const compressed = compressLZ4(input);
// token(lo=0) + 2 字节字面量
expect(compressed.byteLength).toBe(3);
expect(compressed[0]).toBe(0x20); // litLen=2, matchField=0
const restored = decompressLZ4(compressed, 2);
expect(Array.from(restored)).toEqual([1, 2]);
});
it('简单文本压缩不抛出异常且产生输出', () => {
const input = new TextEncoder().encode('hello world hello world hello world');
const compressed = compressLZ4(input);
expect(compressed).toBeInstanceOf(Uint8Array);
expect(compressed.byteLength).toBeGreaterThan(0);
});
it('重复数据有压缩效果', () => {
const pattern = 'ABCD';
const repeated = pattern.repeat(100);
const input = new TextEncoder().encode(repeated);
const compressed = compressLZ4(input);
expect(compressed.byteLength).toBeLessThan(input.byteLength);
});
it('随机不可压缩数据不卡死', () => {
const input = new Uint8Array(256);
for (let i = 0; i < 256; i++) input[i] = Math.floor(Math.random() * 256);
const compressed = compressLZ4(input);
expect(compressed).toBeInstanceOf(Uint8Array);
expect(compressed.byteLength).toBeGreaterThan(0);
});
it('长文本压缩不卡死', () => {
const input = new TextEncoder().encode('The quick brown fox jumps over the lazy dog. '.repeat(10));
const compressed = compressLZ4(input);
expect(compressed).toBeTruthy();
expect(compressed.byteLength).toBeGreaterThan(0);
});
it('多种长度输入均不卡死', () => {
for (const size of [10, 50, 100, 200, 500]) {
const input = new Uint8Array(size);
for (let i = 0; i < size; i++) input[i] = i % 256;
const compressed = compressLZ4(input);
expect(compressed.byteLength).toBeLessThanOrEqual(input.byteLength + 16);
}
});
it('解压不抛出异常', () => {
const input = new TextEncoder().encode('test data for decompression smoke test');
const compressed = compressLZ4(input);
expect(() => decompressLZ4(compressed, input.byteLength)).not.toThrow();
});
// ---- v0.2.6 补强：压缩 → 解压 往返一致性 ----
it('往返一致：重复模式数据', () => {
const input = new TextEncoder().encode('ABCD'.repeat(100));
const compressed = compressLZ4(input);
const restored = decompressLZ4(compressed, input.byteLength);
expect(Array.from(restored)).toEqual(Array.from(input));
});
it('往返一致：自然文本数据', () => {
const input = new TextEncoder().encode(
'The quick brown fox jumps over the lazy dog. '.repeat(10),
);
const compressed = compressLZ4(input);
const restored = decompressLZ4(compressed, input.byteLength);
expect(Array.from(restored)).toEqual(Array.from(input));
});
it('往返一致：随机不可压缩数据', () => {
const input = new Uint8Array(512);
for (let i = 0; i < 512; i++) input[i] = Math.floor(Math.random() * 256);
const compressed = compressLZ4(input);
const restored = decompressLZ4(compressed, input.byteLength);
expect(Array.from(restored)).toEqual(Array.from(input));
});
it('往返一致：多种长度与字节模式', () => {
for (const size of [4, 5, 15, 16, 17, 50, 100, 300, 1000]) {
const input = new Uint8Array(size);
for (let i = 0; i < size; i++) input[i] = i % 7 === 0 ? i % 256 : 0x41;
const compressed = compressLZ4(input);
const restored = decompressLZ4(compressed, input.byteLength);
expect(Array.from(restored)).toEqual(Array.from(input));
}
});
it('往返一致：恰好 15 字节字面量边界', () => {
// 字面量长度恰好 15（token 上限）时不应丢字节
const input = new Uint8Array(15);
for (let i = 0; i < 15; i++) input[i] = i;
const compressed = compressLZ4(input);
const restored = decompressLZ4(compressed, input.byteLength);
expect(Array.from(restored)).toEqual(Array.from(input));
});
it('往返一致：超过 15 字节的连续匹配', () => {
const input = new TextEncoder().encode('X'.repeat(200) + 'Y' + 'X'.repeat(60));
const compressed = compressLZ4(input);
const restored = decompressLZ4(compressed, input.byteLength);
expect(Array.from(restored)).toEqual(Array.from(input));
});
});
// ===================================================================
// MergeIterator — 归并迭代器单元测试
// ===================================================================
describe('AriaEngine — MergeIterator', () => {
it('单数据源归并', () => {
const mi = new MergeIterator();
mi.addSource(new ArrayEntrySource([
['a', { v: 1 }],
['b', { v: 2 }],
['c', { v: 3 }],
]));
const result = mi.drain();
expect(result).toHaveLength(3);
expect(result.map(([k]) => k)).toEqual(['a', 'b', 'c']);
});
it('多数据源归并去重（保留最新）', () => {
const mi = new MergeIterator();
mi.addSource(new ArrayEntrySource([['a', { v: 'new' }], ['c', { v: 3 }]]));
mi.addSource(new ArrayEntrySource([['a', { v: 'old' }], ['b', { v: 2 }]]));
const result = mi.drain();
expect(result).toHaveLength(3);
expect(result[0][0]).toBe('a');
expect(result[0][1].v).toBe('new');
expect(result[1][0]).toBe('b');
expect(result[2][0]).toBe('c');
});
// ---- v0.2.6 回归：同 key 多来源时保留 sourceIndex 最小（最新）的条目 ----
it('回归：同 key 出现在多个来源时返回 sourceIndex 最小（最新来源）的值', () => {
const mi = new MergeIterator();
// 语义：sourceIndex 越小越新（memtable=0 < immutable=1 < sstable=2+）
mi.addSource(new ArrayEntrySource([['a', { v: 'source0' }]])); // 最新来源
mi.addSource(new ArrayEntrySource([['a', { v: 'source1' }]]));
mi.addSource(new ArrayEntrySource([['a', { v: 'source2' }]])); // 最旧来源
const result = mi.drain();
expect(result).toHaveLength(1);
// 取的是 sourceIndex 最小（最新来源）的条目，而非堆序决定的任意条目
expect(result[0][1].v).toBe('source0');
});
it('回归：最新来源的值位于中间 sourceIndex 时仍取 sourceIndex 最小者', () => {
const mi = new MergeIterator();
mi.addSource(new ArrayEntrySource([['a', { v: 'middle' }]])); // index 0 = 最新来源
mi.addSource(new ArrayEntrySource([['a', { v: 'newest' }]])); // index 1
mi.addSource(new ArrayEntrySource([['a', { v: 'oldest' }]])); // index 2
const result = mi.drain();
expect(result).toHaveLength(1);
expect(result[0][1].v).toBe('middle'); // index 0 的条目胜出
});
it('回归：多个同 key 来源 + 其他独立 key 混合', () => {
const mi = new MergeIterator();
mi.addSource(new ArrayEntrySource([['a', { v: 1 }], ['b', { v: 10 }]]));
mi.addSource(new ArrayEntrySource([['a', { v: 2 }]]));
mi.addSource(new ArrayEntrySource([['a', { v: 3 }], ['c', { v: 30 }]]));
const result = mi.drain();
expect(result.map(([k]) => k)).toEqual(['a', 'b', 'c']);
expect(result[0][1].v).toBe(1); // sourceIndex 0 = 最新
expect(result[1][1].v).toBe(10);
expect(result[2][1].v).toBe(30);
});
it('空数据源归并', () => {
const mi = new MergeIterator();
mi.addSource(new ArrayEntrySource([]));
const result = mi.drain();
expect(result).toHaveLength(0);
});
it('大数量归并', () => {
const mi = new MergeIterator();
for (let s = 0; s < 5; s++) {
const entries: [string, Record<string, unknown>][] = [];
for (let i = 0; i < 100; i++) {
entries.push([`src${s}-key-${String(i).padStart(3, '0')}`, { src: s, idx: i }]);
}
mi.addSource(new ArrayEntrySource(entries));
}
const result = mi.drain();
// 5 sources × 100 unique keys = 500 total (keys are unique per source)
expect(result).toHaveLength(500);
});
it('ArrayEntrySource — 迭代器用完返回 null', () => {
const src = new ArrayEntrySource([['k', { v: 1 }]]);
expect(src.next()).not.toBeNull();
expect(src.next()).toBeNull();
expect(src.next()).toBeNull();
});
it('ArrayEntrySource — reset 重置', () => {
const src = new ArrayEntrySource([['k1', { v: 1 }], ['k2', { v: 2 }]]);
src.next();
src.reset();
const val = src.next();
expect(val![0]).toBe('k1');
});
});
/**
* AriaEngine LZ4 压缩 + LSM Merge Iterator 单元测试
* 注：LZ4 为简化演示实现（默认 compression:false），测试聚焦于「不卡死」
*/
import { compressLZ4, decompressLZ4 } from '../../src/engine/aria/compression/lz4';
import { MergeIterator, ArrayEntrySource } from '../../src/engine/aria/index/merge_iterator';
// ===================================================================
// LZ4 压缩 — 安全烟雾测试（不卡死）
// ===================================================================
describe('AriaEngine — LZ4 Compression', () => {
it('短于 4 字节时压缩为纯字面量 token 且往返一致（v0.4.5 带 4 字节原始大小头）', () => {
const input = new Uint8Array([1, 2]);
const compressed = compressLZ4(input);
// HEADER(4) + token(lo=0) + 2 字节字面量
expect(compressed.byteLength).toBe(7);
// 头部 = 原始大小 2（LE）
expect(new DataView(compressed.buffer).getUint32(0, true)).toBe(2);
expect(compressed[4]).toBe(0x20); // litLen=2, matchField=0
const restored = decompressLZ4(compressed, 2);
expect(Array.from(restored)).toEqual([1, 2]);
});
it('简单文本压缩不抛出异常且产生输出', () => {
const input = new TextEncoder().encode('hello world hello world hello world');
const compressed = compressLZ4(input);
expect(compressed).toBeInstanceOf(Uint8Array);
expect(compressed.byteLength).toBeGreaterThan(0);
});
it('重复数据有压缩效果', () => {
const pattern = 'ABCD';
const repeated = pattern.repeat(100);
const input = new TextEncoder().encode(repeated);
const compressed = compressLZ4(input);
expect(compressed.byteLength).toBeLessThan(input.byteLength);
});
it('随机不可压缩数据不卡死', () => {
const input = new Uint8Array(256);
for (let i = 0; i < 256; i++) input[i] = Math.floor(Math.random() * 256);
const compressed = compressLZ4(input);
expect(compressed).toBeInstanceOf(Uint8Array);
expect(compressed.byteLength).toBeGreaterThan(0);
});
it('长文本压缩不卡死', () => {
const input = new TextEncoder().encode('The quick brown fox jumps over the lazy dog. '.repeat(10));
const compressed = compressLZ4(input);
expect(compressed).toBeTruthy();
expect(compressed.byteLength).toBeGreaterThan(0);
});
it('多种长度输入均不卡死', () => {
for (const size of [10, 50, 100, 200, 500]) {
const input = new Uint8Array(size);
for (let i = 0; i < size; i++) input[i] = i % 256;
const compressed = compressLZ4(input);
// 头部 +4 字节开销
expect(compressed.byteLength).toBeLessThanOrEqual(input.byteLength + 4 + 16);
}
});
it('解压不抛出异常', () => {
const input = new TextEncoder().encode('test data for decompression smoke test');
const compressed = compressLZ4(input);
expect(() => decompressLZ4(compressed, input.byteLength)).not.toThrow();
});
// ---- v0.2.6 补强：压缩 → 解压 往返一致性 ----
it('往返一致：重复模式数据', () => {
const input = new TextEncoder().encode('ABCD'.repeat(100));
const compressed = compressLZ4(input);
const restored = decompressLZ4(compressed, input.byteLength);
expect(Array.from(restored)).toEqual(Array.from(input));
});
it('往返一致：自然文本数据', () => {
const input = new TextEncoder().encode(
'The quick brown fox jumps over the lazy dog. '.repeat(10),
);
const compressed = compressLZ4(input);
const restored = decompressLZ4(compressed, input.byteLength);
expect(Array.from(restored)).toEqual(Array.from(input));
});
it('往返一致：随机不可压缩数据', () => {
const input = new Uint8Array(512);
for (let i = 0; i < 512; i++) input[i] = Math.floor(Math.random() * 256);
const compressed = compressLZ4(input);
const restored = decompressLZ4(compressed, input.byteLength);
expect(Array.from(restored)).toEqual(Array.from(input));
});
it('往返一致：多种长度与字节模式', () => {
for (const size of [4, 5, 15, 16, 17, 50, 100, 300, 1000]) {
const input = new Uint8Array(size);
for (let i = 0; i < size; i++) input[i] = i % 7 === 0 ? i % 256 : 0x41;
const compressed = compressLZ4(input);
const restored = decompressLZ4(compressed, input.byteLength);
expect(Array.from(restored)).toEqual(Array.from(input));
}
});
it('往返一致：恰好 15 字节字面量边界', () => {
// 字面量长度恰好 15（token 上限）时不应丢字节
const input = new Uint8Array(15);
for (let i = 0; i < 15; i++) input[i] = i;
const compressed = compressLZ4(input);
const restored = decompressLZ4(compressed, input.byteLength);
expect(Array.from(restored)).toEqual(Array.from(input));
});
it('往返一致：超过 15 字节的连续匹配', () => {
const input = new TextEncoder().encode('X'.repeat(200) + 'Y' + 'X'.repeat(60));
const compressed = compressLZ4(input);
const restored = decompressLZ4(compressed, input.byteLength);
expect(Array.from(restored)).toEqual(Array.from(input));
});
it('往返一致：空输入', () => {
const compressed = compressLZ4(new Uint8Array(0));
expect(compressed.byteLength).toBe(4); // 仅头部
expect(new DataView(compressed.buffer).getUint32(0, true)).toBe(0);
const restored = decompressLZ4(compressed);
expect(restored.byteLength).toBe(0);
});
it('往返一致：头部自描述，无需外部原始大小（高压缩率回归）', () => {
// 中文重复内容 → 极高压缩率（此前 buf.length*2 估算不足导致解压截断）
const text = '这是需要加密的敏感内容'.repeat(500);
const input = new TextEncoder().encode(text);
const compressed = compressLZ4(input);
// 压缩率验证：远小于原始
expect(compressed.byteLength).toBeLessThan(input.byteLength / 2);
// 不传 originalSize：从头部恢复
const restored = decompressLZ4(compressed);
expect(Array.from(restored)).toEqual(Array.from(input));
});
it('损坏头（原始大小非法）→ 抛错而非静默截断', () => {
const bad = new Uint8Array([0xff, 0xff, 0xff, 0xff, 0x20, 0x01, 0x02]);
expect(() => decompressLZ4(bad)).toThrow('Invalid LZ4 header');
});
it('过短输入（无头）→ 抛错', () => {
expect(() => decompressLZ4(new Uint8Array([1, 2, 3]))).toThrow('missing header');
});
});
// ===================================================================
// MergeIterator — 归并迭代器单元测试
// ===================================================================
describe('AriaEngine — MergeIterator', () => {
it('单数据源归并', () => {
const mi = new MergeIterator();
mi.addSource(new ArrayEntrySource([
['a', { v: 1 }],
['b', { v: 2 }],
['c', { v: 3 }],
]));
const result = mi.drain();
expect(result).toHaveLength(3);
expect(result.map(([k]) => k)).toEqual(['a', 'b', 'c']);
});
it('多数据源归并去重（保留最新）', () => {
const mi = new MergeIterator();
mi.addSource(new ArrayEntrySource([['a', { v: 'new' }], ['c', { v: 3 }]]));
mi.addSource(new ArrayEntrySource([['a', { v: 'old' }], ['b', { v: 2 }]]));
const result = mi.drain();
expect(result).toHaveLength(3);
expect(result[0][0]).toBe('a');
expect(result[0][1].v).toBe('new');
expect(result[1][0]).toBe('b');
expect(result[2][0]).toBe('c');
});
// ---- v0.2.6 回归：同 key 多来源时保留 sourceIndex 最小（最新）的条目 ----
it('回归：同 key 出现在多个来源时返回 sourceIndex 最小（最新来源）的值', () => {
const mi = new MergeIterator();
// 语义：sourceIndex 越小越新（memtable=0 < immutable=1 < sstable=2+）
mi.addSource(new ArrayEntrySource([['a', { v: 'source0' }]])); // 最新来源
mi.addSource(new ArrayEntrySource([['a', { v: 'source1' }]]));
mi.addSource(new ArrayEntrySource([['a', { v: 'source2' }]])); // 最旧来源
const result = mi.drain();
expect(result).toHaveLength(1);
// 取的是 sourceIndex 最小（最新来源）的条目，而非堆序决定的任意条目
expect(result[0][1].v).toBe('source0');
});
it('回归：最新来源的值位于中间 sourceIndex 时仍取 sourceIndex 最小者', () => {
const mi = new MergeIterator();
mi.addSource(new ArrayEntrySource([['a', { v: 'middle' }]])); // index 0 = 最新来源
mi.addSource(new ArrayEntrySource([['a', { v: 'newest' }]])); // index 1
mi.addSource(new ArrayEntrySource([['a', { v: 'oldest' }]])); // index 2
const result = mi.drain();
expect(result).toHaveLength(1);
expect(result[0][1].v).toBe('middle'); // index 0 的条目胜出
});
it('回归：多个同 key 来源 + 其他独立 key 混合', () => {
const mi = new MergeIterator();
mi.addSource(new ArrayEntrySource([['a', { v: 1 }], ['b', { v: 10 }]]));
mi.addSource(new ArrayEntrySource([['a', { v: 2 }]]));
mi.addSource(new ArrayEntrySource([['a', { v: 3 }], ['c', { v: 30 }]]));
const result = mi.drain();
expect(result.map(([k]) => k)).toEqual(['a', 'b', 'c']);
expect(result[0][1].v).toBe(1); // sourceIndex 0 = 最新
expect(result[1][1].v).toBe(10);
expect(result[2][1].v).toBe(30);
});
it('空数据源归并', () => {
const mi = new MergeIterator();
mi.addSource(new ArrayEntrySource([]));
const result = mi.drain();
expect(result).toHaveLength(0);
});
it('大数量归并', () => {
const mi = new MergeIterator();
for (let s = 0; s < 5; s++) {
const entries: [string, Record<string, unknown>][] = [];
for (let i = 0; i < 100; i++) {
entries.push([`src${s}-key-${String(i).padStart(3, '0')}`, { src: s, idx: i }]);
}
mi.addSource(new ArrayEntrySource(entries));
}
const result = mi.drain();
// 5 sources × 100 unique keys = 500 total (keys are unique per source)
expect(result).toHaveLength(500);
});
it('ArrayEntrySource — 迭代器用完返回 null', () => {
const src = new ArrayEntrySource([['k', { v: 1 }]]);
expect(src.next()).not.toBeNull();
expect(src.next()).toBeNull();
expect(src.next()).toBeNull();
});
it('ArrayEntrySource — reset 重置', () => {
const src = new ArrayEntrySource([['k1', { v: 1 }], ['k2', { v: 2 }]]);
src.next();
src.reset();
const val = src.next();
expect(val![0]).toBe('k1');
});
});
tests/engine/aria-crypto.test.ts
+1 -60
View File
@@ -5,13 +5,7 @@
* v0.2.6 补强：此前仅验证实例化，现在验证真实的加解密往返一致性。
*/
import 'fake-indexeddb/auto';
import {
CryptoManager,
initCrypto,
encryptPage,
decryptPage,
closeCrypto,
} from '../../src/engine/aria/crypto';
import { CryptoManager } from '../../src/engine/aria/crypto';
function toBytes(data: ArrayBuffer): number[] {
return Array.from(new Uint8Array(data));
@@ -89,15 +83,6 @@ describe('AriaEngine — CryptoManager', () => {
cm2.close();
});
test('全局兼容层往返一致', async () => {
await initCrypto('global-password');
const original = new TextEncoder().encode('global compat layer').buffer;
const { iv, data } = await encryptPage(original);
const decrypted = await decryptPage(iv, data);
expect(toBytes(decrypted)).toEqual(toBytes(original));
closeCrypto();
});
test('大块数据（接近页面大小）往返一致', async () => {
const cm = new CryptoManager();
await cm.init('page-size-test');
@@ -111,47 +96,3 @@ describe('AriaEngine — CryptoManager', () => {
});
});
// ===================================================================
// 真实存储路径：SSTable 加密 → 持久化 → 重载解密（v0.3.2 CI 修复）
// ===================================================================
describe('AriaEngine — SSTable 加密存储真实路径', () => {
test('加密落盘后重载可完整解密', async () => {
// eslint-disable-next-line @typescript-eslint/no-var-requires
const { AriaEngine } = require('../../src/engine/aria/index');
// eslint-disable-next-line @typescript-eslint/no-var-requires
const { createSchema } = require('../../src/table/schema');
const dbName = `enc-store-${Date.now()}`;
await initCrypto('engine-store-password');
const engine = new AriaEngine({
storageBackend: 'indexeddb',
walSyncMode: 'none',
} as any);
await engine.open(dbName, 1);
await engine.createTable(createSchema('t', {
id: { type: 'string', primaryKey: true },
v: { type: 'number' },
}));
await engine.insert('t', [{ id: '1', v: 42 }, { id: '2', v: 99 }]);
// 强制刷盘：SSTable 经加密保存（encryptPage 真实路径）
await (engine as any).lsm.flush();
await engine.close();
// 重载：SSTable 解密恢复
const engine2 = new AriaEngine({
storageBackend: 'indexeddb',
walSyncMode: 'none',
} as any);
await engine2.open(dbName, 1);
const rows = await engine2.find('t', { table: 't' });
expect(rows).toHaveLength(2);
const byId = Object.fromEntries(rows.map((r: any) => [r.id, r.v]));
expect(byId['1']).toBe(42);
expect(byId['2']).toBe(99);
await engine2.close();
closeCrypto();
});
});
tests/engine/aria-encryption.test.ts
+349
View File
@@ -0,0 +1,349 @@
/**
* AriaEngine — 全库 AES-256-GCM 加密测试
*
* 覆盖：
* 1. EncryptedBackend 单元：读写往返 / writeMany / delete / clear / 密文落盘验证
* 2. AriaEngine 集成：密码往返（open → 写入 → close → 同密码重开数据完整）
* 3. 错误密码 → ARIA_DECRYPT_ERROR
* 4. 明文旧库 + 加密配置 → ARIA_ENCRYPT_CONFIG_ERROR
* 5. 加密库 + 无密码 → ARIA_ENCRYPT_REQUIRED
* 6. clearAll 后可换新密码重建
* 7. 元数据（迁移版本）加密往返
*/
import { AriaEngine } from '../../src/engine/aria/index';
import { EncryptedBackend } from '../../src/engine/aria/store/encrypted_backend';
import { MemoryBackend } from '../../src/engine/aria/store/backend';
import { createSchema } from '../../src/table/schema';
import 'fake-indexeddb/auto';
let idbCounter = 0;
function uniqueDB(): string {
return `enc-${Date.now()}-${++idbCounter}-${Math.random().toString(36).slice(2, 8)}`;
}
const SCHEMA = () => createSchema('users', {
id: { type: 'string', primaryKey: true },
name: { type: 'string' },
secret: { type: 'string' },
});
// ===================================================================
// EncryptedBackend 单元
// ===================================================================
describe('AriaEngine — EncryptedBackend 单元', () => {
it('open 创建 keymeta，读写往返完整', async () => {
const inner = new MemoryBackend();
const backend = new EncryptedBackend(inner, 's3cret-password');
await backend.open('enc-unit-1');
// keymeta 已创建
const keymeta = await inner.read('__aria_keymeta');
expect(keymeta).not.toBeNull();
// 读写往返
const payload = new TextEncoder().encode('hello encrypted world').buffer;
await backend.write('k1', payload);
const back = await backend.read('k1');
expect(back).not.toBeNull();
expect(new TextDecoder().decode(back)).toBe('hello encrypted world');
// 底层是密文（非明文）
const raw = await inner.read('k1');
const rawStr = new TextDecoder().decode(raw);
expect(rawStr).not.toContain('hello encrypted world');
await backend.close();
});
it('writeMany / deleteMany 往返', async () => {
const inner = new MemoryBackend();
const backend = new EncryptedBackend(inner, 'pw');
await backend.open('enc-unit-2');
const enc = (s: string) => new TextEncoder().encode(s).buffer;
await backend.writeMany({ a: enc('AAA'), b: enc('BBB'), c: enc('CCC') });
expect(new TextDecoder().decode(await backend.read('a'))).toBe('AAA');
expect(new TextDecoder().decode(await backend.read('c'))).toBe('CCC');
await backend.deleteMany(['a', 'c']);
expect(await backend.exists('a')).toBe(false);
expect(await backend.exists('b')).toBe(true);
await backend.delete('b');
expect(await backend.exists('b')).toBe(false);
expect(await backend.listKeys()).toEqual(['__aria_keymeta']);
await backend.close();
});
it('每个 value 独立随机 IV（相同明文两次加密密文不同）', async () => {
const inner = new MemoryBackend();
const backend = new EncryptedBackend(inner, 'pw');
await backend.open('enc-unit-3');
const payload = new TextEncoder().encode('same plaintext').buffer;
await backend.write('x', payload);
await backend.write('y', payload);
const rawX = await inner.read('x');
const rawY = await inner.read('y');
// Jest toEqual 对 ArrayBuffer 不比较内容 → 字节级比较
expect(new Uint8Array(rawX as ArrayBuffer)).not.toEqual(new Uint8Array(rawY as ArrayBuffer));
// 但 IV 前 12 字节后的长度一致（密文长度 = 明文 + GCM tag 16B）
expect((rawX as ArrayBuffer).byteLength).toBe((rawY as ArrayBuffer).byteLength);
// 但解密一致
expect(new TextDecoder().decode(await backend.read('x'))).toBe('same plaintext');
expect(new TextDecoder().decode(await backend.read('y'))).toBe('same plaintext');
await backend.close();
});
it('clear 清空全部数据但保留密钥元数据（库身份保留）', async () => {
const inner = new MemoryBackend();
const backend = new EncryptedBackend(inner, 'pw');
await backend.open('enc-unit-4');
await backend.write('k', new TextEncoder().encode('v').buffer);
expect((await backend.listKeys()).length).toBe(2);
await backend.clear();
// 数据清空，keymeta 保留
expect(await backend.listKeys()).toEqual(['__aria_keymeta']);
expect(await backend.exists('k')).toBe(false);
await backend.close();
// 重新打开：keymeta 仍在，同密码可用
const backend2 = new EncryptedBackend(inner, 'pw');
await backend2.open('enc-unit-4');
await backend2.write('k2', new TextEncoder().encode('v2').buffer);
expect(new TextDecoder().decode(await backend2.read('k2'))).toBe('v2');
// 换密码被拒（keymeta 与旧密码绑定）——注意：MemoryBackend close 清空 store，
// 因此在 close 前验证（backend2 仍持有 inner）
const backend3 = new EncryptedBackend(inner, 'other-pw');
await expect(backend3.open('enc-unit-4')).rejects.toMatchObject({ code: 'ARIA_DECRYPT_ERROR' });
await backend2.close();
});
it('明文旧库（有数据无 keymeta）+ 加密 → ARIA_ENCRYPT_CONFIG_ERROR', async () => {
const inner = new MemoryBackend();
await inner.open('enc-unit-5');
await inner.write('__aria_schemas', new TextEncoder().encode('{}').buffer);
const backend = new EncryptedBackend(inner, 'pw');
await expect(backend.open('enc-unit-5')).rejects.toMatchObject({ code: 'ARIA_ENCRYPT_CONFIG_ERROR' });
});
it('空密码配置直接抛错', () => {
const inner = new MemoryBackend();
try {
new EncryptedBackend(inner, '');
fail('should throw');
} catch (error) {
expect((error as { code?: string }).code).toBe('ARIA_ENCRYPT_CONFIG_ERROR');
}
});
it('getInner / isCryptoReady 访问器', async () => {
const inner = new MemoryBackend();
const backend = new EncryptedBackend(inner, 'pw');
await backend.open('enc-unit-6');
expect(backend.getInner()).toBe(inner);
expect(backend.isCryptoReady()).toBe(true);
await backend.close();
expect(backend.isCryptoReady()).toBe(false);
});
it('密文块被篡改 → 读取抛 ARIA_DECRYPT_ERROR（GCM 认证失败）', async () => {
const inner = new MemoryBackend();
const backend = new EncryptedBackend(inner, 'pw');
await backend.open('enc-unit-7');
await backend.write('k', new TextEncoder().encode('integrity-check').buffer);
// 篡改密文一个字节（绕过 IV 区）
const raw = (await inner.read('k')) as ArrayBuffer;
const buf = new Uint8Array(raw);
buf[16] ^= 0xff;
await inner.write('k', buf.buffer as ArrayBuffer);
await expect(backend.read('k')).rejects.toMatchObject({ code: 'ARIA_DECRYPT_ERROR' });
await backend.close();
});
it('过短密文块 → 抛 ARIA_DECRYPT_ERROR', async () => {
const inner = new MemoryBackend();
const backend = new EncryptedBackend(inner, 'pw');
await backend.open('enc-unit-8');
await inner.write('short', new Uint8Array(4).buffer); // 不足 IV(12) + tag
await expect(backend.read('short')).rejects.toMatchObject({ code: 'ARIA_DECRYPT_ERROR' });
await backend.close();
});
});
// ===================================================================
// AriaEngine 集成 — 加密往返
// ===================================================================
describe('AriaEngine — 全库加密（集成）', () => {
it('加密库：写入 → close → 同密码重开数据完整', async () => {
const dbName = uniqueDB();
const engine = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'master-pass' } });
await engine.open(dbName, 1);
await engine.createTable(SCHEMA());
await engine.insert('users', [
{ id: '1', name: 'Alice', secret: 'top-secret-a' },
{ id: '2', name: 'Bob', secret: 'top-secret-b' },
]);
await engine.setMeta('__metona_version', '3');
await engine.close();
// 重开（同密码）
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'master-pass' } });
await engine2.open(dbName, 1);
const rows = await engine2.find('users', { table: 'users' });
expect(rows).toHaveLength(2);
const byId = Object.fromEntries(rows.map((r) => [r.id, r]));
expect(byId['1'].secret).toBe('top-secret-a');
expect(byId['2'].secret).toBe('top-secret-b');
expect(await engine2.getMeta('__metona_version')).toBe('3');
await engine2.close();
});
it('加密库：错误密码重开 → ARIA_DECRYPT_ERROR', async () => {
const dbName = uniqueDB();
const engine = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'right-pass' } });
await engine.open(dbName, 1);
await engine.createTable(SCHEMA());
await engine.insert('users', [{ id: '1', name: 'A', secret: 'x' }]);
await engine.close();
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'wrong-pass' } });
await expect(engine2.open(dbName, 1)).rejects.toMatchObject({ code: 'ARIA_DECRYPT_ERROR' });
});
it('加密库：无密码打开 → ARIA_ENCRYPT_REQUIRED', async () => {
const dbName = uniqueDB();
const engine = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'pw' } });
await engine.open(dbName, 1);
await engine.createTable(SCHEMA());
await engine.close();
const engine2 = new AriaEngine({ storageBackend: 'indexeddb' });
await expect(engine2.open(dbName, 1)).rejects.toMatchObject({ code: 'ARIA_ENCRYPT_REQUIRED' });
});
it('明文库：加密配置打开 → ARIA_ENCRYPT_CONFIG_ERROR', async () => {
const dbName = uniqueDB();
const engine = new AriaEngine({ storageBackend: 'indexeddb' });
await engine.open(dbName, 1);
await engine.createTable(SCHEMA());
await engine.insert('users', [{ id: '1', name: 'A', secret: 'plain' }]);
await engine.close();
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'pw' } });
await expect(engine2.open(dbName, 1)).rejects.toMatchObject({ code: 'ARIA_ENCRYPT_CONFIG_ERROR' });
});
it('加密库：底层存储全部为密文（SSTable/Schema/WAL 均不可见明文）', async () => {
const dbName = uniqueDB();
const engine = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'pw' } });
await engine.open(dbName, 1);
await engine.createTable(SCHEMA());
await engine.insert('users', [{ id: '1', name: 'Alice', secret: 'needle-in-cipher' }]);
await (engine as any).lsm.flush();
// 强制落盘后 WAL 仍有记录（checkpoint 截断前）→ 全量扫描检查
const backend = (engine as any).backend as EncryptedBackend;
const inner = backend.getInner();
const keys = await inner.listKeys();
expect(keys.length).toBeGreaterThan(0);
// 任何一个底层 key 的内容都不包含明文数据（需解密才可见）
for (const key of keys) {
if (key === '__aria_keymeta') continue;
const raw = await inner.read(key);
const str = new TextDecoder().decode(raw as ArrayBuffer);
expect(str).not.toContain('needle-in-cipher');
expect(str).not.toContain('Alice');
expect(str).not.toContain('users');
}
await engine.close();
});
it('加密 + 压缩组合：往返完整（压缩层先压，加密层后加密）', async () => {
const dbName = uniqueDB();
const engine = new AriaEngine({
storageBackend: 'indexeddb',
compression: true,
encryption: { password: 'pw-compress' },
memtableSizeThreshold: 64 * 1024 * 1024,
});
await engine.open(dbName, 1);
await engine.createTable(SCHEMA());
const rows = [] as Record<string, unknown>[];
for (let i = 0; i < 200; i++) {
rows.push({ id: `u-${i}`, name: `User${i}`, secret: '这是需要加密的敏感内容'.repeat(5) });
}
await engine.insert('users', rows);
await (engine as any).lsm.flush();
await engine.close();
const engine2 = new AriaEngine({
storageBackend: 'indexeddb',
compression: true,
encryption: { password: 'pw-compress' },
});
await engine2.open(dbName, 1);
expect(await engine2.count('users')).toBe(200);
const one = await engine2.find('users', { table: 'users', where: { id: 'u-42' } });
expect(one[0].secret).toBe('这是需要加密的敏感内容'.repeat(5));
await engine2.close();
});
it('加密库：clearAll 保留密钥（库身份），同密码重开重建，新密码被拒', async () => {
const dbName = uniqueDB();
const engine = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'old-pw' } });
await engine.open(dbName, 1);
await engine.createTable(SCHEMA());
await engine.insert('users', [{ id: '1', name: 'A', secret: 'x' }]);
await engine.clearAll();
await engine.close();
// clearAll 保留 keymeta → 换新密码会被拒绝
const engineWrong = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'new-pw' } });
await expect(engineWrong.open(dbName, 1)).rejects.toMatchObject({ code: 'ARIA_DECRYPT_ERROR' });
// 同密码重开可重建
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'old-pw' } });
await engine2.open(dbName, 1);
await engine2.createTable(SCHEMA());
await engine2.insert('users', [{ id: '1', name: 'A', secret: 'new' }]);
expect(await engine2.count('users')).toBe(1);
await engine2.close();
});
it('加密库：WAL 崩溃恢复路径可用（含加密 WAL）', async () => {
const dbName = uniqueDB();
const engine = new AriaEngine({
storageBackend: 'indexeddb',
encryption: { password: 'pw' },
walSyncMode: 'full',
checkpointInterval: 100000,
});
await engine.open(dbName, 1);
await engine.createTable(SCHEMA());
// 写入后不 flush 不 checkpoint，直接"崩溃"（close 会 checkpoint，这里模拟崩溃：直接断 backend）
for (let i = 0; i < 10; i++) {
await engine.insert('users', [{ id: `w-${i}`, name: `W${i}`, secret: `s${i}` }]);
}
// 模拟崩溃：不 close，直接断开（真实崩溃时 WAL 留在存储中）
await (engine as any).backend.close();
(engine as any).opened = false;
// 重开：WAL 重放（加密 WAL 解密后解析）
const engine2 = new AriaEngine({
storageBackend: 'indexeddb',
encryption: { password: 'pw' },
walSyncMode: 'full',
checkpointInterval: 100000,
});
await engine2.open(dbName, 1);
const rows = await engine2.find('users', { table: 'users' });
expect(rows).toHaveLength(10);
await engine2.close();
});
});
tests/engine/aria-index.test.ts
+198 -216
View File
@@ -1,216 +1,198 @@
/**
* AriaEngine Bloom Filter + MemTable 单元测试
*/
import { BloomFilter } from '../../src/engine/aria/index/bloom';
import { MemTable } from '../../src/engine/aria/index/memtable';
// ===================================================================
// BloomFilter
// ===================================================================
describe('AriaEngine — BloomFilter', () => {
it('插入后 mayContain 返回 true', () => {
const bf = new BloomFilter(100);
bf.insert('hello');
expect(bf.mayContain('hello')).toBe(true);
});
it('未插入的 key mayContain 返回 false', () => {
const bf = new BloomFilter(100);
bf.insert('hello');
expect(bf.mayContain('world')).toBe(false);
});
it('批量插入后所有 key 都判定存在', () => {
const bf = new BloomFilter(500);
const keys: string[] = [];
for (let i = 0; i < 200; i++) {
const k = `key-${i}`;
keys.push(k);
bf.insert(k);
}
for (const k of keys) {
expect(bf.mayContain(k)).toBe(true);
}
});
it('False positive 率可控', () => {
const n = 500;
const bf = new BloomFilter(n, 10);
for (let i = 0; i < n; i++) {
bf.insert(`present-${i}`);
}
let fp = 0;
for (let i = 0; i < 500; i++) {
if (bf.mayContain(`absent-${i}`)) fp++;
}
expect(fp).toBeLessThan(25);
});
it('getBitSize 返回正确位数', () => {
const bf = new BloomFilter(100, 10);
expect(bf.getBitSize()).toBeGreaterThanOrEqual(64);
});
it('getInsertedCount 追踪插入数', () => {
const bf = new BloomFilter(100);
bf.insert('a');
bf.insert('b');
bf.insert('c');
expect(bf.getInsertedCount()).toBe(3);
});
it('getHashCount 返回哈希函数数量', () => {
const bf = new BloomFilter(1000, 10);
expect(bf.getHashCount()).toBeGreaterThan(0);
});
it('serialize + fromData 往返', () => {
const bf1 = new BloomFilter(100);
bf1.insert('a');
bf1.insert('b');
const data = bf1.serialize();
const bf2 = BloomFilter.fromData(data, bf1.getHashCount());
expect(bf2.mayContain('a')).toBe(true);
expect(bf2.mayContain('b')).toBe(true);
expect(bf2.mayContain('c')).toBe(false);
});
it('空过滤器 mayContain 返回 false', () => {
const bf = new BloomFilter(100);
expect(bf.mayContain('anything')).toBe(false);
});
it('最少 1 个哈希函数', () => {
const bf = new BloomFilter(10, 1);
expect(bf.getHashCount()).toBeGreaterThanOrEqual(1);
});
});
// ===================================================================
// MemTable
// ===================================================================
describe('AriaEngine — MemTable', () => {
let mt: MemTable;
beforeEach(() => {
mt = new MemTable(4 * 1024 * 1024);
});
it('put + get 往返', () => {
mt.put('key1', { name: 'Alice', age: 30 });
const val = mt.get('key1');
expect(val).not.toBeNull();
expect(val!.name).toBe('Alice');
});
it('get — 不存在的 key 返回 null', () => {
expect(mt.get('nonexistent')).toBeNull();
});
it('put 更新已存在的 key', () => {
mt.put('k', { v: 1 });
mt.put('k', { v: 2 });
expect(mt.get('k')!.v).toBe(2);
});
it('delete 删除成功', () => {
mt.put('k', { v: 1 });
expect(mt.delete('k')).toBe(true);
expect(mt.get('k')).toBeNull();
});
it('delete — 不存在的 key 返回 false', () => {
expect(mt.delete('ghost')).toBe(false);
});
it('getAllEntries 返回所有条目（有序）', () => {
mt.put('c', { v: 3 });
mt.put('a', { v: 1 });
mt.put('b', { v: 2 });
const entries = mt.getAllEntries();
expect(entries).toHaveLength(3);
expect(entries[0][0]).toBe('a');
expect(entries[1][0]).toBe('b');
expect(entries[2][0]).toBe('c');
});
it('rangeScan — 范围查询', () => {
mt.put('a', { v: 1 });
mt.put('b', { v: 2 });
mt.put('c', { v: 3 });
mt.put('d', { v: 4 });
const results = mt.rangeScan('b', 'c');
expect(results).toHaveLength(2);
expect(results[0][0]).toBe('b');
expect(results[1][0]).toBe('c');
});
it('rangeScan — 空结果', () => {
mt.put('a', { v: 1 });
const results = mt.rangeScan('z', 'zz');
expect(results).toHaveLength(0);
});
it('getEntryCount 正确计数', () => {
expect(mt.getEntryCount()).toBe(0);
mt.put('a', { v: 1 });
mt.put('b', { v: 2 });
expect(mt.getEntryCount()).toBe(2);
mt.delete('a');
expect(mt.getEntryCount()).toBe(1);
});
it('contains 检查存在性', () => {
mt.put('x', { v: 1 });
expect(mt.contains('x')).toBe(true);
expect(mt.contains('y')).toBe(false);
});
it('shouldFlush — 未达阈值返回 false', () => {
expect(mt.shouldFlush()).toBe(false);
});
it('clear 清空所有数据', () => {
mt.put('a', { v: 1 });
mt.put('b', { v: 2 });
mt.clear();
expect(mt.getEntryCount()).toBe(0);
expect(mt.get('a')).toBeNull();
});
it('getEstimatedSize 返回合理估计值', () => {
expect(mt.getEstimatedSize()).toBe(0);
mt.put('hello', { name: 'world', count: 42 });
expect(mt.getEstimatedSize()).toBeGreaterThan(0);
});
it('大量数据插入保持有序', () => {
const count = 100;
for (let i = count - 1; i >= 0; i--) {
mt.put(`key-${String(i).padStart(3, '0')}`, { idx: i });
}
const entries = mt.getAllEntries();
expect(entries).toHaveLength(count);
for (let i = 0; i < count; i++) {
expect(entries[i][1].idx).toBe(i);
}
});
it('删除后重新插入', () => {
mt.put('k', { v: 1 });
mt.delete('k');
mt.put('k', { v: 2 });
expect(mt.get('k')!.v).toBe(2);
});
it('范围扫描包含边界', () => {
mt.put('aa', { v: 1 });
mt.put('ab', { v: 2 });
mt.put('ac', { v: 3 });
const results = mt.rangeScan('aa', 'ab');
expect(results).toHaveLength(2);
expect(results[0][0]).toBe('aa');
expect(results[1][0]).toBe('ab');
});
});
/**
* AriaEngine Bloom Filter + MemTable 单元测试
*/
import { BloomFilter } from '../../src/engine/aria/index/bloom';
import { MemTable } from '../../src/engine/aria/index/memtable';
// ===================================================================
// BloomFilter
// ===================================================================
describe('AriaEngine — BloomFilter', () => {
it('插入后 mayContain 返回 true', () => {
const bf = new BloomFilter(100);
bf.insert('hello');
expect(bf.mayContain('hello')).toBe(true);
});
it('未插入的 key mayContain 返回 false', () => {
const bf = new BloomFilter(100);
bf.insert('hello');
expect(bf.mayContain('world')).toBe(false);
});
it('批量插入后所有 key 都判定存在', () => {
const bf = new BloomFilter(500);
const keys: string[] = [];
for (let i = 0; i < 200; i++) {
const k = `key-${i}`;
keys.push(k);
bf.insert(k);
}
for (const k of keys) {
expect(bf.mayContain(k)).toBe(true);
}
});
it('False positive 率可控', () => {
const n = 500;
const bf = new BloomFilter(n, 10);
for (let i = 0; i < n; i++) {
bf.insert(`present-${i}`);
}
let fp = 0;
for (let i = 0; i < 500; i++) {
if (bf.mayContain(`absent-${i}`)) fp++;
}
expect(fp).toBeLessThan(25);
});
it('getHashCount 返回哈希函数数量', () => {
const bf = new BloomFilter(1000, 10);
expect(bf.getHashCount()).toBeGreaterThan(0);
});
it('serialize + fromData 往返', () => {
const bf1 = new BloomFilter(100);
bf1.insert('a');
bf1.insert('b');
const data = bf1.serialize();
const bf2 = BloomFilter.fromData(data, bf1.getHashCount());
expect(bf2.mayContain('a')).toBe(true);
expect(bf2.mayContain('b')).toBe(true);
expect(bf2.mayContain('c')).toBe(false);
});
it('空过滤器 mayContain 返回 false', () => {
const bf = new BloomFilter(100);
expect(bf.mayContain('anything')).toBe(false);
});
it('最少 1 个哈希函数', () => {
const bf = new BloomFilter(10, 1);
expect(bf.getHashCount()).toBeGreaterThanOrEqual(1);
});
});
// ===================================================================
// MemTable
// ===================================================================
describe('AriaEngine — MemTable', () => {
let mt: MemTable;
beforeEach(() => {
mt = new MemTable(4 * 1024 * 1024);
});
it('put + get 往返', () => {
mt.put('key1', { name: 'Alice', age: 30 });
const val = mt.get('key1');
expect(val).not.toBeNull();
expect(val!.name).toBe('Alice');
});
it('get — 不存在的 key 返回 null', () => {
expect(mt.get('nonexistent')).toBeNull();
});
it('put 更新已存在的 key', () => {
mt.put('k', { v: 1 });
mt.put('k', { v: 2 });
expect(mt.get('k')!.v).toBe(2);
});
it('delete 删除成功', () => {
mt.put('k', { v: 1 });
expect(mt.delete('k')).toBe(true);
expect(mt.get('k')).toBeNull();
});
it('delete — 不存在的 key 返回 false', () => {
expect(mt.delete('ghost')).toBe(false);
});
it('getAllEntries 返回所有条目（有序）', () => {
mt.put('c', { v: 3 });
mt.put('a', { v: 1 });
mt.put('b', { v: 2 });
const entries = mt.getAllEntries();
expect(entries).toHaveLength(3);
expect(entries[0][0]).toBe('a');
expect(entries[1][0]).toBe('b');
expect(entries[2][0]).toBe('c');
});
it('rangeScan — 范围查询', () => {
mt.put('a', { v: 1 });
mt.put('b', { v: 2 });
mt.put('c', { v: 3 });
mt.put('d', { v: 4 });
const results = mt.rangeScan('b', 'c');
expect(results).toHaveLength(2);
expect(results[0][0]).toBe('b');
expect(results[1][0]).toBe('c');
});
it('rangeScan — 空结果', () => {
mt.put('a', { v: 1 });
const results = mt.rangeScan('z', 'zz');
expect(results).toHaveLength(0);
});
it('getEntryCount 正确计数', () => {
expect(mt.getEntryCount()).toBe(0);
mt.put('a', { v: 1 });
mt.put('b', { v: 2 });
expect(mt.getEntryCount()).toBe(2);
mt.delete('a');
expect(mt.getEntryCount()).toBe(1);
});
it('shouldFlush — 未达阈值返回 false', () => {
expect(mt.shouldFlush()).toBe(false);
});
it('clear 清空所有数据', () => {
mt.put('a', { v: 1 });
mt.put('b', { v: 2 });
mt.clear();
expect(mt.getEntryCount()).toBe(0);
expect(mt.get('a')).toBeNull();
});
it('getEstimatedSize 返回合理估计值', () => {
expect(mt.getEstimatedSize()).toBe(0);
mt.put('hello', { name: 'world', count: 42 });
expect(mt.getEstimatedSize()).toBeGreaterThan(0);
});
it('大量数据插入保持有序', () => {
const count = 100;
for (let i = count - 1; i >= 0; i--) {
mt.put(`key-${String(i).padStart(3, '0')}`, { idx: i });
}
const entries = mt.getAllEntries();
expect(entries).toHaveLength(count);
for (let i = 0; i < count; i++) {
expect(entries[i][1].idx).toBe(i);
}
});
it('删除后重新插入', () => {
mt.put('k', { v: 1 });
mt.delete('k');
mt.put('k', { v: 2 });
expect(mt.get('k')!.v).toBe(2);
});
it('范围扫描包含边界', () => {
mt.put('aa', { v: 1 });
mt.put('ab', { v: 2 });
mt.put('ac', { v: 3 });
const results = mt.rangeScan('aa', 'ab');
expect(results).toHaveLength(2);
expect(results[0][0]).toBe('aa');
expect(results[1][0]).toBe('ab');
});
});
tests/engine/aria-locks.test.ts
+170
View File
@@ -0,0 +1,170 @@
/**
* AriaEngine — 多标签页独占锁（Web Locks）测试
*
* 覆盖：
* 1. 第一个引擎持锁 → 第二个引擎打开抛 ARIA_LOCKED
* 2. 关闭后锁释放 → 可再次打开
* 3. 环境不支持 Web Locks → 降级打开（返回 false 语义 + 警告）
* 4. open 失败（如加密密码错误）→ 锁释放（其他标签页可打开）
* 5. 锁名按库隔离（不同库不互斥）
*/
import { AriaEngine } from '../../src/engine/aria/index';
import { DatabaseLock, lockName } from '../../src/engine/aria/locks';
import { createSchema } from '../../src/table/schema';
import 'fake-indexeddb/auto';
let idbCounter = 0;
function uniqueDB(): string {
return `lock-${Date.now()}-${++idbCounter}-${Math.random().toString(36).slice(2, 8)}`;
}
// ===================================================================
// Web Locks mock：真实语义（ifAvailable / 回调结束锁自动释放）
// ===================================================================
let lockHolders: Map<string, true>;
function installLocksMock() {
lockHolders = new Map();
Object.defineProperty(globalThis, 'navigator', {
value: {
locks: {
request: async (_name: string, opts: { ifAvailable?: boolean }, cb: (lock: { name: string } | null) => Promise<void> | void) => {
const name = _name;
if (lockHolders.has(name)) {
if (opts?.ifAvailable) {
await cb(null);
return;
}
throw new Error('unreachable: non-ifAvailable request not used');
}
lockHolders.set(name, true);
try {
// 真实 Web Locks 语义：回调结束 → 锁自动释放
await cb({ name });
} finally {
lockHolders.delete(name);
}
},
},
},
configurable: true,
writable: true,
});
}
function removeLocksMock() {
Object.defineProperty(globalThis, 'navigator', {
value: {},
configurable: true,
writable: true,
});
}
// ===================================================================
// DatabaseLock 单元
// ===================================================================
describe('AriaEngine — DatabaseLock 单元', () => {
beforeEach(installLocksMock);
it('acquire 持锁 → 二次 acquire 抛 ARIA_LOCKED → release 后恢复', async () => {
const lock1 = new DatabaseLock();
expect(await lock1.acquire('db-a')).toBe(true);
expect(lock1.isAcquired()).toBe(true);
expect(lock1.isSupported()).toBe(true);
expect(lockHolders.has(lockName('db-a'))).toBe(true);
const lock2 = new DatabaseLock();
await expect(lock2.acquire('db-a')).rejects.toMatchObject({ code: 'ARIA_LOCKED' });
expect(lock2.isAcquired()).toBe(false);
await lock1.release();
expect(lock1.isAcquired()).toBe(false);
expect(lockHolders.has(lockName('db-a'))).toBe(false);
const lock3 = new DatabaseLock();
expect(await lock3.acquire('db-a')).toBe(true);
await lock3.release();
});
it('不同库名互不阻塞', async () => {
const l1 = new DatabaseLock();
const l2 = new DatabaseLock();
expect(await l1.acquire('db-x')).toBe(true);
expect(await l2.acquire('db-y')).toBe(true);
await l1.release();
await l2.release();
});
it('无 Web Locks 环境 → acquire 返回 false（降级）', async () => {
removeLocksMock();
const lock = new DatabaseLock();
expect(await lock.acquire('db-z')).toBe(false);
expect(lock.isSupported()).toBe(false);
expect(lock.isAcquired()).toBe(false);
});
it('重复 release 幂等', async () => {
const lock = new DatabaseLock();
await lock.acquire('db-r');
await lock.release();
await lock.release();
expect(lock.isAcquired()).toBe(false);
});
});
// ===================================================================
// AriaEngine 集成 — 多标签页锁
// ===================================================================
describe('AriaEngine — 多标签页锁（集成）', () => {
beforeEach(installLocksMock);
it('第二个标签页打开同一库 → ARIA_LOCKED', async () => {
const dbName = uniqueDB();
const engine1 = new AriaEngine({ storageBackend: 'indexeddb' });
await engine1.open(dbName, 1);
await engine1.createTable(createSchema('t', { id: { type: 'string', primaryKey: true } }));
const engine2 = new AriaEngine({ storageBackend: 'indexeddb' });
await expect(engine2.open(dbName, 1)).rejects.toMatchObject({ code: 'ARIA_LOCKED' });
// engine2 未持锁、未 opened
expect((engine2 as any).opened).toBe(false);
await engine1.close();
// 释放后可打开
const engine3 = new AriaEngine({ storageBackend: 'indexeddb' });
await engine3.open(dbName, 1);
expect(await engine3.count('t')).toBe(0);
await engine3.close();
});
it('open 失败（错误密码）→ 锁释放，其他标签页可打开', async () => {
const dbName = uniqueDB();
const engine1 = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'pw' } });
await engine1.open(dbName, 1);
await engine1.createTable(createSchema('t', { id: { type: 'string', primaryKey: true } }));
await engine1.close();
// 错误密码 → open 失败
const engineBad = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'wrong' } });
await expect(engineBad.open(dbName, 1)).rejects.toMatchObject({ code: 'ARIA_DECRYPT_ERROR' });
// 失败后锁已释放（未泄漏）
expect(lockHolders.has(lockName(dbName))).toBe(false);
// 正确密码可打开
const engineOk = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'pw' } });
await engineOk.open(dbName, 1);
await engineOk.close();
});
it('无 Web Locks 环境 → 降级打开（不抛错）', async () => {
removeLocksMock();
const dbName = uniqueDB();
const engine = new AriaEngine({ storageBackend: 'indexeddb' });
await engine.open(dbName, 1);
await engine.createTable(createSchema('t', { id: { type: 'string', primaryKey: true } }));
await engine.insert('t', [{ id: '1' }]);
expect(await engine.count('t')).toBe(1);
await engine.close();
});
});
tests/engine/aria-maintenance.test.ts
-17
View File
@@ -81,23 +81,6 @@ describe('AriaEngine — ANALYZE/VACUUM/REINDEX + EXPLAIN', () => {
expect(plan.actualTimeMs).toBeGreaterThanOrEqual(0);
});
// ---- 查询优化器 ----
it('estimateQueryCost PK 等值 → pk_lookup', () => {
const cost = (engine as any).estimateQueryCost('users', { table: 'users', where: { id: '1' } });
expect(cost.strategy).toBe('pk_lookup');
expect(cost.estimatedRows).toBe(1);
});
it('estimateQueryCost 索引等值 → index_eq', () => {
const cost = (engine as any).estimateQueryCost('users', { table: 'users', where: { name: 'Alice' } });
expect(cost.strategy).toBe('index_eq:name');
});
it('estimateQueryCost 无索引 → full_scan', () => {
const cost = (engine as any).estimateQueryCost('users', { table: 'users', where: { age: { $gt: 20 } } });
expect(cost.strategy).toMatch(/index_range|full_scan/);
});
// ---- 在线备份 ----
it('backup 导出全库一致性快照', async () => {
const backup = await (engine as any).backup();
tests/engine/aria-mvcc-savepoint.test.ts
+10 -23
View File
@@ -27,31 +27,20 @@ describe('AriaEngine — MVCC 事务 + Savepoint', () => {
expect(id1).not.toBe(id2);
});
it('commit 后事务标记为非活跃', () => {
it('commit 后版本标记已提交', () => {
const txnId = (engine as any).mvcc.beginTransaction();
(engine as any).mvcc.writeVersion('users', '1', { name: 'Alice' }, txnId);
(engine as any).mvcc.commitTransaction(txnId);
expect((engine as any).mvcc.isActive(txnId)).toBe(false);
const store = (engine as any).mvcc.versionStore;
expect(store.get('users.1')[0].committed).toBe(true);
});
it('rollback 后事务标记为非活跃', () => {
it('rollback 移除本事务版本', () => {
const txnId = (engine as any).mvcc.beginTransaction();
(engine as any).mvcc.writeVersion('users', '1', { name: 'Alice' }, txnId);
(engine as any).mvcc.rollbackTransaction(txnId);
expect((engine as any).mvcc.isActive(txnId)).toBe(false);
});
it('事务中写入版本可读', () => {
const txnId = (engine as any).mvcc.beginTransaction();
(engine as any).mvcc.writeVersion('users', '1', { name: 'Alice', balance: 100 }, txnId);
const val = (engine as any).mvcc.readVersion('users', '1', txnId);
expect(val).not.toBeNull();
expect(val!.name).toBe('Alice');
});
it('未提交版本对其他事务不可见', () => {
const txn1 = (engine as any).mvcc.beginTransaction();
(engine as any).mvcc.writeVersion('users', '1', { name: 'Alice' }, txn1);
const txn2 = (engine as any).mvcc.beginTransaction();
expect((engine as any).mvcc.readVersion('users', '1', txn2)).toBeNull();
const store = (engine as any).mvcc.versionStore;
expect(store.has('users.1')).toBe(false);
});
it('引擎层事务 commit 后数据可见', async () => {
@@ -69,16 +58,14 @@ describe('AriaEngine — MVCC 事务 + Savepoint', () => {
expect(await engine.count('users')).toBe(1);
});
it('GC 清理过旧版本', () => {
it('GC 清理过旧版本（保留最新 N 个）', () => {
for (let i = 0; i < 200; i++) {
const txnId = (engine as any).mvcc.beginTransaction();
(engine as any).mvcc.writeVersion('users', '1', { ver: i }, txnId);
(engine as any).mvcc.commitTransaction(txnId);
}
(engine as any).mvcc.gc(50);
const txnId = (engine as any).mvcc.beginTransaction();
expect((engine as any).mvcc.readVersion('users', '1', txnId)).not.toBeNull();
(engine as any).mvcc.commitTransaction(txnId);
expect((engine as any).mvcc.versionStore.get('users.1').length).toBe(50);
});
// ---- Savepoint ----
tests/engine/aria-opfs-backend.test.ts
+228
View File
@@ -0,0 +1,228 @@
/**
* AriaEngine — OPFSBackend v2 测试
*
* 覆盖：
* 1. read/write/append/exists/delete/clear 基本语义
* 2. append 追加语义（keepExistingData + position）与字节顺序
* 3. 写队列串行化（并发写顺序一致）
* 4. 单次任务失败不中断队列链（后续写正常）
* 5. close 等待挂起写完成
* 6. open 清理崩溃残留临时文件（.crswap/.tmp）
* 7. writeMany/deleteMany 语义
*/
import { OPFSBackend } from '../../src/engine/aria/store/opfs_backend';
// ===================================================================
// 真实语义 OPFS mock：记录文件内容、支持 keepExistingData+position
// ===================================================================
interface MockFile {
content: ArrayBuffer;
writeCalls: { position?: number; data: ArrayBuffer; keepExistingData?: boolean }[];
}
function createOPFSMock() {
const files = new Map<string, MockFile>();
const getFileHandle = async (name: string, opts?: { create?: boolean }) => {
if (!files.has(name)) {
if (!opts?.create) throw new Error(`NotFoundError: ${name}`);
files.set(name, { content: new ArrayBuffer(0), writeCalls: [] });
}
const entry = files.get(name)!;
return {
getFile: async () => ({ size: entry.content.byteLength, arrayBuffer: async () => entry.content }),
createWritable: async (wOpts?: { keepExistingData?: boolean }) => {
const w: {
write: (arg: ArrayBuffer | { type: string; position: number; data: ArrayBuffer }) => Promise<void>;
close: () => Promise<void>;
} = {
write: async (arg) => {
const keepExisting = wOpts?.keepExistingData ?? false;
const isChunk = typeof arg !== 'object' || !('type' in (arg as object)) || (arg as { type?: string }).type === undefined
? { data: arg as ArrayBuffer, position: keepExisting ? entry.content.byteLength : 0 }
: { data: (arg as { data: ArrayBuffer }).data, position: (arg as { position: number }).position };
entry.writeCalls.push({ position: isChunk.position, data: isChunk.data, keepExistingData: keepExisting });
const merged = new Uint8Array(isChunk.position + isChunk.data.byteLength);
if (keepExisting || isChunk.position > 0) {
merged.set(new Uint8Array(entry.content.subarray ? entry.content : entry.content), 0);
}
merged.set(new Uint8Array(isChunk.data), isChunk.position);
entry.content = merged.buffer;
},
close: async () => { /* no-op */ },
};
return w;
},
};
};
const dir = {
getFileHandle,
entries: async function* () {
for (const [name] of files) yield [name];
},
removeEntry: async (name: string) => {
files.delete(name);
},
};
Object.defineProperty(globalThis, 'navigator', {
value: {
storage: {
getDirectory: async () => ({
getDirectoryHandle: async (_name: string, _opts?: unknown) => dir,
}),
},
},
configurable: true,
writable: true,
});
return { files, dir };
}
const enc = (s: string) => new Uint8Array(new TextEncoder().encode(s)).buffer;
describe('AriaEngine — OPFSBackend v2', () => {
beforeEach(() => {
createOPFSMock();
});
it('read/write/exists/delete 基本语义', async () => {
const backend = new OPFSBackend();
await backend.open('opfs-test-1');
expect(backend.isOpen()).toBe(true);
await backend.write('k1', enc('hello'));
expect(await backend.exists('k1')).toBe(true);
expect(new TextDecoder().decode(await backend.read('k1'))).toBe('hello');
expect(await backend.read('missing')).toBeNull();
await backend.delete('k1');
expect(await backend.exists('k1')).toBe(false);
await backend.close();
expect(backend.isOpen()).toBe(false);
});
it('append 追加语义：字节顺序与位置正确', async () => {
const backend = new OPFSBackend();
await backend.open('opfs-test-2');
await backend.append('wal', enc('AAA'));
await backend.append('wal', enc('BBB'));
await backend.append('wal', enc('CCC'));
const all = await backend.read('wal');
expect(new TextDecoder().decode(all)).toBe('AAABBBCCC');
await backend.close();
});
it('write 覆盖语义（COW 原子替换）', async () => {
const backend = new OPFSBackend();
await backend.open('opfs-test-3');
await backend.write('f', enc('OLD-CONTENT'));
await backend.write('f', enc('NEW'));
expect(new TextDecoder().decode(await backend.read('f'))).toBe('NEW');
await backend.close();
});
it('并发写串行化：写入顺序与调用顺序一致', async () => {
const backend = new OPFSBackend();
await backend.open('opfs-test-4');
// 同时发起多个写（不 await 中间结果）
const p1 = backend.append('log', enc('A'));
const p2 = backend.append('log', enc('B'));
const p3 = backend.append('log', enc('C'));
await Promise.all([p1, p2, p3]);
expect(new TextDecoder().decode(await backend.read('log'))).toBe('ABC');
await backend.close();
});
it('单次写失败不中断队列链，后续写正常', async () => {
const backend = new OPFSBackend();
await backend.open('opfs-test-5');
// 注入失败：让 write 抛错一次
let failNext = true;
const origWrite = backend.write.bind(backend);
const mockDir = (backend as any).dbDir;
const origGetFileHandle = mockDir.getFileHandle.bind(mockDir);
mockDir.getFileHandle = async (name: string, opts?: { create?: boolean }) => {
if (failNext && name === 'boom') {
failNext = false;
throw new Error('Injected write failure');
}
return origGetFileHandle(name, opts);
};
(backend as any).dbDir = mockDir;
await expect(origWrite('boom', enc('x'))).rejects.toThrow('Injected write failure');
// 队列链恢复：后续写成功
await backend.write('ok', enc('fine'));
expect(new TextDecoder().decode(await backend.read('ok'))).toBe('fine');
await backend.close();
});
it('close 等待挂起写完成（挂起写不丢失）', async () => {
const backend = new OPFSBackend();
await backend.open('opfs-test-6');
// 发起一个写但不 await，立即 close
const pending = backend.append('late', enc('LATE-DATA'));
await backend.close();
await pending; // close 已等队列 → 写已完成
expect(pending).resolves.toBeUndefined();
});
it('open 清理崩溃残留临时文件（.crswap/.tmp）', async () => {
const backend = new OPFSBackend();
await backend.open('opfs-test-7');
await backend.write('data', enc('real'));
// 手工制造残留
const mockDir = (backend as any).dbDir;
await mockDir.getFileHandle('data.crswap', { create: true });
await mockDir.getFileHandle('data.tmp', { create: true });
await backend.close();
// 重开：open 自动清理
const backend2 = new OPFSBackend();
await backend2.open('opfs-test-7');
const keys = await backend2.listKeys();
expect(keys).toEqual(['data']);
expect(keys.some((k) => k.endsWith('.crswap') || k.endsWith('.tmp'))).toBe(false);
// 正常数据不受影响
expect(new TextDecoder().decode(await backend2.read('data'))).toBe('real');
await backend2.close();
});
it('writeMany / deleteMany 语义', async () => {
const backend = new OPFSBackend();
await backend.open('opfs-test-8');
await backend.writeMany({ a: enc('AAA'), b: enc('BBB') });
expect(new TextDecoder().decode(await backend.read('a'))).toBe('AAA');
expect(new TextDecoder().decode(await backend.read('b'))).toBe('BBB');
await backend.deleteMany(['a']);
expect(await backend.exists('a')).toBe(false);
expect(await backend.exists('b')).toBe(true);
await backend.clear();
expect(await backend.listKeys()).toEqual([]);
await backend.close();
});
it('close 后所有操作安全返回（dbDir 为 null 不抛错）', async () => {
const backend = new OPFSBackend();
await backend.open('opfs-test-9');
await backend.write('k', enc('v'));
await backend.close();
expect(backend.isOpen()).toBe(false);
expect(await backend.read('k')).toBeNull();
await backend.write('late', enc('x')); // 静默忽略
await backend.append('late2', enc('x'));
await backend.writeMany({ a: enc('1') });
await backend.delete('k');
await backend.deleteMany(['a']);
expect(await backend.listKeys()).toEqual([]);
expect(await backend.exists('k')).toBe(false);
await backend.clear();
await backend.cleanupStaleFiles();
});
});
tests/engine/aria-page-core.test.ts
+62
View File
@@ -0,0 +1,62 @@
/**
* AriaEngine Page — 页面核心逻辑测试
*
* v0.4.5: 页面层精简后仅保留生产代码实际使用的函数：
* - initPageHeader（FileManager 创建空页面）
* - createPage（BufferPool 分配新页面）
* - PAGE_HEADER_SIZE / PAGE_SIZE 常量
* 页面内容为 SSTable 原始字节切片（PageSSTableStore 管理），无行级 slot 结构。
*/
import { createPage } from '../../src/engine/aria/page/format';
import { initPageHeader } from '../../src/engine/aria/page/header';
import { PAGE_SIZE, PAGE_HEADER_SIZE, PageType } from '../../src/engine/aria/types';
describe('AriaEngine — Page 核心', () => {
it('initPageHeader 初始化 4KB 页面头部字段', () => {
const buf = new ArrayBuffer(PAGE_SIZE);
initPageHeader(buf, 42, PageType.DATA);
const view = new DataView(buf);
expect(view.getUint32(0, false)).toBe(42); // page_id
expect(view.getUint8(4)).toBe(PageType.DATA); // type
expect(view.getUint16(5, false)).toBe(PAGE_HEADER_SIZE); // free_start
expect(view.getUint16(7, false)).toBe(PAGE_SIZE); // free_end
expect(view.getUint16(9, false)).toBe(0); // slot_count
expect(view.getUint8(15)).toBe(0); // reserved
});
it('initPageHeader 支持不同页面类型', () => {
const buf = new ArrayBuffer(PAGE_SIZE);
initPageHeader(buf, 1, PageType.META);
expect(new DataView(buf).getUint8(4)).toBe(PageType.META);
});
it('createPage 创建脏页面（pins=0，初始化头部）', () => {
const page = createPage(7, PageType.DATA);
expect(page.pageId).toBe(7);
expect(page.type).toBe(PageType.DATA);
expect(page.data.byteLength).toBe(PAGE_SIZE);
expect(page.dirty).toBe(true);
expect(page.pins).toBe(0);
// 头部已初始化
expect(new DataView(page.data).getUint32(0, false)).toBe(7);
expect(new DataView(page.data).getUint8(4)).toBe(PageType.DATA);
});
it('页面数据区从头部之后开始（跳过 16 字节头）', () => {
const page = createPage(1, PageType.DATA);
// 头部之后写入数据，头部字段不受影响
const view = new Uint8Array(page.data);
view[PAGE_HEADER_SIZE] = 0xAB;
view[PAGE_HEADER_SIZE + 1] = 0xCD;
expect(view[PAGE_HEADER_SIZE]).toBe(0xAB);
expect(view[PAGE_HEADER_SIZE + 1]).toBe(0xCD);
// 头部字段保持
expect(new DataView(page.data).getUint32(0, false)).toBe(1);
});
it('createPage 的 dirty 标记在写入后保持（PageSSTableStore 依赖）', () => {
const page = createPage(2, PageType.DATA);
new Uint8Array(page.data)[100] = 0x01;
expect(page.dirty).toBe(true);
});
});
tests/engine/aria-page-store.test.ts
+318
View File
@@ -0,0 +1,318 @@
/**
* AriaEngine — 页面化 SSTable 存储测试
*
* 覆盖：
* 1. PageSSTableStore 单元：save/load 往返（页边界、非整数页）、delete 释放页面
* 2. AriaEngine + OPFS（页面模式）：写入 → flush → close → reopen 数据完整
* 3. 大 SSTable 多页 + pageIds 注入 meta 验证
* 4. 页面损坏（CRC 校验失败）→ 打开自愈清理
* 5. 页面模式与非页面模式混合（pageIds 兼容：新页面写 + 旧整 value 读）
* 6. 页面化 + 加密 + 压缩组合
*/
import { AriaEngine } from '../../src/engine/aria/index';
import { PageSSTableStore } from '../../src/engine/aria/store/page_sstable_store';
import { FileManager } from '../../src/engine/aria/store/file_manager';
import { BufferPool } from '../../src/engine/aria/buffer/pool';
import { MemoryBackend } from '../../src/engine/aria/store/backend';
import { createSchema } from '../../src/table/schema';
import { PAGE_SIZE } from '../../src/engine/aria/types';
// ===================================================================
// OPFS mock（共享工具）
// ===================================================================
import { installOPFSMock } from '../helpers/opfs-mock';
const SCHEMA = () => createSchema('users', {
id: { type: 'string', primaryKey: true },
name: { type: 'string' },
note: { type: 'string' },
});
// ===================================================================
// PageSSTableStore 单元
// ===================================================================
describe('AriaEngine — PageSSTableStore 单元', () => {
it('save/load 往返：单页数据', async () => {
const backend = new MemoryBackend();
await backend.open('pg-unit-1');
const fm = new FileManager(backend);
await fm.init('pg-unit-1');
const bp = new BufferPool(fm, 64);
const store = new PageSSTableStore(fm, bp);
const data = new Uint8Array(new TextEncoder().encode('hello page store'));
await store.save(1, data);
const pageIds = store.getPageIds(1)!;
expect(pageIds.length).toBe(1);
const back = await store.load(1, pageIds, data.byteLength);
expect(back).not.toBeNull();
expect(new TextDecoder().decode(back)).toBe('hello page store');
// 页面已落盘（backend 有 pg_ 键）
expect(await backend.exists('pg_1')).toBe(true);
await store.delete(1, pageIds);
expect(await backend.exists('pg_1')).toBe(false);
await backend.close();
});
it('save/load 往返：多页数据（跨 4KB 边界）', async () => {
const backend = new MemoryBackend();
await backend.open('pg-unit-2');
const fm = new FileManager(backend);
await fm.init('pg-unit-2');
const bp = new BufferPool(fm, 64);
const store = new PageSSTableStore(fm, bp);
// 3 页：4KB * 2 + 1KB
const size = PAGE_SIZE * 2 + 1024;
const data = new Uint8Array(size);
let x = 42;
for (let i = 0; i < size; i++) {
x = (x * 31 + 7) & 0xffff;
data[i] = x & 0xff;
}
await store.save(7, data);
const pageIds = store.getPageIds(7)!;
expect(pageIds.length).toBe(3);
const back = await store.load(7, pageIds, size);
expect(back).not.toBeNull();
expect(back!.byteLength).toBe(size);
expect(back).toEqual(data);
await backend.close();
});
it('load 使用 totalSize 截断最后一页的 0 填充', async () => {
const backend = new MemoryBackend();
await backend.open('pg-unit-3');
const fm = new FileManager(backend);
await fm.init('pg-unit-3');
const bp = new BufferPool(fm, 64);
const store = new PageSSTableStore(fm, bp);
// 非整数页：4096 + 100 字节
const real = new Uint8Array(PAGE_SIZE + 100);
for (let i = 0; i < real.length; i++) real[i] = i & 0xff;
await store.save(9, real);
const pageIds = store.getPageIds(9)!;
expect(pageIds.length).toBe(2);
const back = await store.load(9, pageIds, real.length);
expect(back!.byteLength).toBe(real.length);
expect(back).toEqual(real);
await backend.close();
});
it('页面缺失（磁盘文件丢失 + 缓存清空）→ load 返回 null', async () => {
const backend = new MemoryBackend();
await backend.open('pg-unit-4');
const fm = new FileManager(backend);
await fm.init('pg-unit-4');
const bp = new BufferPool(fm, 64);
const store = new PageSSTableStore(fm, bp);
await store.save(5, new Uint8Array(100));
const pageIds = store.getPageIds(5)!;
// 模拟崩溃重开：磁盘文件丢失 + 页面缓存清空
await backend.delete('pg_' + pageIds[0]);
bp.removePage(pageIds[0]);
const back = await store.load(5, pageIds, 100);
expect(back).toBeNull();
await backend.close();
});
});
// ===================================================================
// AriaEngine + OPFS 页面模式集成
// ===================================================================
describe('AriaEngine — 页面化 SSTable 集成（OPFS）', () => {
it('写入 → flush → close → reopen 数据完整（页面模式默认启用）', async () => {
const files = installOPFSMock(new Map());
const engine = new AriaEngine({
storageBackend: 'opfs',
memtableSizeThreshold: 64 * 1024 * 1024,
checkpointInterval: 100000,
});
await engine.open('page-db-1', 1);
await engine.createTable(SCHEMA());
// 多行 + 大 note → 超过 4KB → 多页
const rows = [] as Record<string, unknown>[];
for (let i = 0; i < 30; i++) {
rows.push({ id: `u-${i}`, name: `User${i}`, note: 'page-store-note-'.repeat(40) });
}
await engine.insert('users', rows);
await (engine as any).lsm.flush();
// 页面化落盘：backend 有 pg_ 键，meta 含 pageIds
const backend = (engine as any).backend;
const keys = await backend.listKeys();
const pgKeys = keys.filter((k: string) => k.startsWith('pg_'));
const sstKeys = keys.filter((k: string) => k.startsWith('sst_'));
expect(pgKeys.length).toBeGreaterThan(0);
expect(sstKeys.length).toBe(0); // 页面模式下不写整 value SSTable
const metas = await (engine as any).lsm.sstableStore.listMeta();
expect(metas.length).toBeGreaterThan(0);
expect(metas[0].pageIds).toBeDefined();
expect(metas[0].pageIds.length).toBeGreaterThan(0);
await engine.close();
// reopen：数据完整
const engine2 = new AriaEngine({ storageBackend: 'opfs', checkpointInterval: 100000 });
await engine2.open('page-db-1', 1);
const found = await engine2.find('users', { table: 'users' });
expect(found).toHaveLength(30);
expect(found[0].note).toBe('page-store-note-'.repeat(40));
expect(await engine2.count('users')).toBe(30);
await engine2.close();
});
it('多级 compaction 后页面化数据仍完整', async () => {
const files = installOPFSMock(new Map());
const engine = new AriaEngine({
storageBackend: 'opfs',
memtableSizeThreshold: 16 * 1024,
checkpointInterval: 100000,
});
await engine.open('page-db-2', 1);
await engine.createTable(SCHEMA());
// 大量小批量写入 → 多次 flush → 触发 compaction
for (let batch = 0; batch < 8; batch++) {
const rows = [] as Record<string, unknown>[];
for (let i = 0; i < 20; i++) {
rows.push({ id: `b${batch}-u${i}`, name: `U${i}`, note: 'x'.repeat(200) });
}
await engine.insert('users', rows);
await (engine as any).lsm.flush();
}
// 强制压缩
await (engine as any).lsm.compactLevel(0);
await engine.close();
const engine2 = new AriaEngine({ storageBackend: 'opfs', checkpointInterval: 100000 });
await engine2.open('page-db-2', 1);
expect(await engine2.count('users')).toBe(160);
const one = await engine2.find('users', { table: 'users', where: { id: 'b3-u10' } });
expect(one).toHaveLength(1);
await engine2.close();
});
it('页面损坏（篡改 pg_ 文件）→ 打开自愈清理，其余数据可读', async () => {
const files = installOPFSMock(new Map());
const engine = new AriaEngine({
storageBackend: 'opfs',
memtableSizeThreshold: 64 * 1024 * 1024,
checkpointInterval: 100000,
});
await engine.open('page-db-3', 1);
await engine.createTable(SCHEMA());
for (let batch = 0; batch < 2; batch++) {
const rows = [] as Record<string, unknown>[];
for (let i = 0; i < 20; i++) {
rows.push({ id: `b${batch}-u${i}`, name: `U${i}`, note: 'n'.repeat(100) });
}
await engine.insert('users', rows);
await (engine as any).lsm.flush();
}
await engine.close();
// 篡改第一个 SSTable 的第一页
const engine2 = new AriaEngine({ storageBackend: 'opfs', checkpointInterval: 100000 });
await engine2.open('page-db-3', 1);
const metas = await (engine2 as any).lsm.sstableStore.listMeta();
expect(metas.length).toBeGreaterThanOrEqual(2);
const victim = metas[0];
const backend = (engine2 as any).backend;
const pageRaw = await backend.read(`pg_${victim.pageIds[0]}`);
const corrupted = new Uint8Array(pageRaw);
corrupted[100] ^= 0xff;
await backend.write(`pg_${victim.pageIds[0]}`, corrupted.buffer as ArrayBuffer);
await engine2.close();
// 重开：损坏页面的 SSTable 被清理，其余数据可读
const engine3 = new AriaEngine({ storageBackend: 'opfs', checkpointInterval: 100000 });
await engine3.open('page-db-3', 1);
const rows3 = await engine3.find('users', { table: 'users' });
expect(rows3.length).toBeGreaterThan(0);
expect(rows3.length).toBeLessThan(40);
await engine3.close();
});
it('页面模式与非页面模式混合兼容（pageIds 缺失 → 整 value 读取）', async () => {
const { files } = installOPFSMock(new Map());
// 阶段 1：非页面模式写入（pageStorage: false → 整 value SSTable）
const engine = new AriaEngine({
storageBackend: 'opfs',
pageStorage: false,
memtableSizeThreshold: 64 * 1024 * 1024,
checkpointInterval: 100000,
});
await engine.open('page-db-4', 1);
await engine.createTable(SCHEMA());
await engine.insert('users', [{ id: 'legacy-1', name: 'Old', note: 'whole-value' }]);
await (engine as any).lsm.flush();
await engine.close();
const backendKeys1 = Array.from(files.keys());
expect(backendKeys1.some((k) => k.startsWith('sst_'))).toBe(true);
// 阶段 2：页面模式打开（默认 opfs → 启用），读旧数据 + 写新数据
const engine2 = new AriaEngine({ storageBackend: 'opfs', checkpointInterval: 100000 });
await engine2.open('page-db-4', 1);
expect(await engine2.count('users')).toBe(1);
const legacy = await engine2.find('users', { table: 'users', where: { id: 'legacy-1' } });
expect(legacy[0].note).toBe('whole-value');
// 新写入 → 页面化
await engine2.insert('users', [{ id: 'new-1', name: 'New', note: 'paged' }]);
await (engine2 as any).lsm.flush();
await engine2.close();
// 阶段 3：混合库重开，新旧数据都在
const engine3 = new AriaEngine({ storageBackend: 'opfs', checkpointInterval: 100000 });
await engine3.open('page-db-4', 1);
expect(await engine3.count('users')).toBe(2);
const both = await engine3.find('users', { table: 'users' });
const byId = Object.fromEntries(both.map((r) => [r.id, r]));
expect(byId['legacy-1'].note).toBe('whole-value');
expect(byId['new-1'].note).toBe('paged');
await engine3.close();
});
it('页面化 + 加密 + 压缩组合：往返完整', async () => {
const files = installOPFSMock(new Map());
const engine = new AriaEngine({
storageBackend: 'opfs',
compression: true,
encryption: { password: 'page-pass' },
memtableSizeThreshold: 64 * 1024 * 1024,
checkpointInterval: 100000,
});
await engine.open('page-db-5', 1);
await engine.createTable(SCHEMA());
const rows = [] as Record<string, unknown>[];
for (let i = 0; i < 20; i++) {
rows.push({ id: `e-${i}`, name: `E${i}`, note: '加密的页面化内容'.repeat(30) });
}
await engine.insert('users', rows);
await (engine as any).lsm.flush();
await engine.close();
const engine2 = new AriaEngine({
storageBackend: 'opfs',
compression: true,
encryption: { password: 'page-pass' },
checkpointInterval: 100000,
});
await engine2.open('page-db-5', 1);
expect(await engine2.count('users')).toBe(20);
const one = await engine2.find('users', { table: 'users', where: { id: 'e-7' } });
expect(one[0].note).toBe('加密的页面化内容'.repeat(30));
await engine2.close();
});
});
tests/engine/aria-page.test.ts
-337
View File
@@ -1,337 +0,0 @@
/**
* AriaEngine Page 格式单元测试
* 覆盖: PageHeader / Slot / Tuple 编解码 + PageFormat 整合
*/
import {
PAGE_SIZE, PageType, PAGE_HEADER_SIZE, SLOT_ENTRY_SIZE,
} from '../../src/engine/aria/types';
import {
encodePageHeader, decodePageHeader, initPageHeader, getPageType,
getSlotCount, getFreeStart, setFreeStart, setFreeEnd,
} from '../../src/engine/aria/page/header';
import {
getSlotEntry, setSlotEntry, getSlotDirectorySize,
getFreeSpace, hasEnoughSpace, allocateSlot, readSlotData, freeSlot,
} from '../../src/engine/aria/page/slot';
import {
encodeTuple, decodeTuple, getColumnEncodingMap,
} from '../../src/engine/aria/page/tuple';
import { ColumnEncoding } from '../../src/engine/aria/types';
import {
createPage, pageFromBuffer, pageInsertRow, pageReadRow,
pageDeleteRow, pageUpdateRow, computeChecksum, verifyChecksum, updateChecksum,
} from '../../src/engine/aria/page/format';
// ===================================================================
// PageHeader
// ===================================================================
describe('AriaEngine Page — Header', () => {
let buf: ArrayBuffer;
beforeEach(() => {
buf = new ArrayBuffer(PAGE_SIZE);
});
it('initPageHeader 初始化头部字段', () => {
initPageHeader(buf, 42, PageType.DATA);
const h = decodePageHeader(buf);
expect(h.pageId).toBe(42);
expect(h.type).toBe(PageType.DATA);
expect(h.slotCount).toBe(0);
expect(h.freeStart).toBe(PAGE_HEADER_SIZE);
expect(h.freeEnd).toBe(PAGE_SIZE);
});
it('initPageHeader — INDEX 类型页面', () => {
initPageHeader(buf, 99, PageType.INDEX);
expect(getPageType(buf)).toBe(PageType.INDEX);
});
it('encodePageHeader + decodePageHeader 往返一致', () => {
const header = { pageId: 7, type: PageType.META, freeStart: 32, freeEnd: 4000, slotCount: 5, checksum: 0xdeadbeef };
encodePageHeader(header, buf);
const decoded = decodePageHeader(buf);
expect(decoded.pageId).toBe(7);
expect(decoded.type).toBe(PageType.META);
expect(decoded.freeStart).toBe(32);
expect(decoded.freeEnd).toBe(4000);
expect(decoded.slotCount).toBe(5);
});
it('不同 pageId 正确编解码', () => {
for (const id of [0, 1, 255, 65535, 0xffffffff]) {
initPageHeader(buf, id, PageType.DATA);
expect(decodePageHeader(buf).pageId).toBe(id >>> 0);
}
});
it('setFreeStart / setFreeEnd 修改字段', () => {
initPageHeader(buf, 1, PageType.DATA);
setFreeStart(buf, 100);
setFreeEnd(buf, 3000);
expect(getFreeStart(buf)).toBe(100);
expect(decodePageHeader(buf).freeEnd).toBe(3000);
});
});
// ===================================================================
// Slot Directory
// ===================================================================
describe('AriaEngine Page — Slot', () => {
let buf: ArrayBuffer;
beforeEach(() => {
buf = new ArrayBuffer(PAGE_SIZE);
initPageHeader(buf, 1, PageType.DATA);
});
it('getSlotEntry — 空页面 slotCount 为 0', () => {
expect(getSlotCount(buf)).toBe(0);
});
it('setSlotEntry + getSlotEntry 往返', () => {
// 手动写一个 slot（不通过 allocateSlot）
new DataView(buf).setUint16(9, 1, false); // slotCount = 1
setSlotEntry(buf, 0, { offset: 1000, length: 50 });
const entry = getSlotEntry(buf, 0);
expect(entry.offset).toBe(1000);
expect(entry.length).toBe(50);
});
it('getSlotDirectorySize 计算正确', () => {
expect(getSlotDirectorySize(0)).toBe(0);
expect(getSlotDirectorySize(1)).toBe(SLOT_ENTRY_SIZE);
expect(getSlotDirectorySize(10)).toBe(10 * SLOT_ENTRY_SIZE);
});
it('getFreeSpace — 空页面有最大空闲空间', () => {
const free = getFreeSpace(buf);
expect(free).toBe(PAGE_SIZE - PAGE_HEADER_SIZE);
});
it('hasEnoughSpace — 小数据返回 true', () => {
expect(hasEnoughSpace(buf, 100)).toBe(true);
});
it('hasEnoughSpace — 超大数据返回 false', () => {
expect(hasEnoughSpace(buf, PAGE_SIZE * 2)).toBe(false);
});
it('allocateSlot 分配并写入数据', () => {
const data = new Uint8Array([1, 2, 3, 4, 5]);
const idx = allocateSlot(buf, data);
expect(idx).toBe(0);
expect(getSlotCount(buf)).toBe(1);
const readBack = readSlotData(buf, 0);
expect(readBack).not.toBeNull();
expect(Array.from(readBack!)).toEqual([1, 2, 3, 4, 5]);
});
it('allocateSlot 多次分配', () => {
for (let i = 0; i < 10; i++) {
const data = new Uint8Array([i, i + 1]);
const idx = allocateSlot(buf, data);
expect(idx).toBe(i);
}
expect(getSlotCount(buf)).toBe(10);
// 验证每个 slot 数据正确
for (let i = 0; i < 10; i++) {
const data = readSlotData(buf, i);
expect(Array.from(data!)).toEqual([i, i + 1]);
}
});
it('freeSlot 标记为删除', () => {
const data = new Uint8Array([10, 20, 30]);
allocateSlot(buf, data);
freeSlot(buf, 0);
const entry = getSlotEntry(buf, 0);
expect(entry.offset).toBe(0);
expect(entry.length).toBe(0);
});
it('readSlotData — 已删除 slot 返回 null', () => {
allocateSlot(buf, new Uint8Array([1]));
freeSlot(buf, 0);
expect(readSlotData(buf, 0)).toBeNull();
});
it('allocateSlot — 空间不足返回 -1', () => {
// 填满页面
const big = new Uint8Array(PAGE_SIZE - PAGE_HEADER_SIZE - SLOT_ENTRY_SIZE);
const idx1 = allocateSlot(buf, big);
expect(idx1).toBe(0);
const idx2 = allocateSlot(buf, new Uint8Array([1]));
expect(idx2).toBe(-1);
});
});
// ===================================================================
// Tuple Codec
// ===================================================================
describe('AriaEngine Page — Tuple', () => {
const colOrder = ['id', 'name', 'age', 'active', 'data'];
const colTypes: Record<string, string> = {
id: 'string', name: 'string', age: 'number', active: 'boolean', data: 'json',
};
it('encodeTuple + decodeTuple 完整往返', () => {
const row = { id: '1', name: 'Alice', age: 30, active: true, data: { x: 1 } };
const encoded = encodeTuple(row, colOrder, colTypes);
expect(encoded.byteLength).toBeGreaterThan(0);
const decoded = decodeTuple(encoded, colOrder, colTypes);
expect(decoded).not.toBeNull();
expect(decoded!.id).toBe('1');
expect(decoded!.name).toBe('Alice');
expect(decoded!.age).toBe(30);
expect(decoded!.active).toBe(true);
expect(decoded!.data).toEqual({ x: 1 });
});
it('encodeTuple — null 值正确处理', () => {
const row = { id: '2', name: null, age: 25, active: null, data: null };
const encoded = encodeTuple(row, colOrder, colTypes);
const decoded = decodeTuple(encoded, colOrder, colTypes);
expect(decoded!.name).toBeNull();
expect(decoded!.active).toBeNull();
expect(decoded!.data).toBeNull();
});
it('encodeTuple — undefined 值按 null 处理', () => {
const row = { id: '3', age: 30 } as any;
const encoded = encodeTuple(row, colOrder, colTypes);
const decoded = decodeTuple(encoded, colOrder, colTypes);
expect(decoded!.id).toBe('3');
expect(decoded!.name).toBeNull();
});
it('encodeTuple — date 类型', () => {
const order = ['ts'];
const types = { ts: 'date' };
const row = { ts: '2024-01-15T00:00:00.000Z' };
const encoded = encodeTuple(row, order, types);
const decoded = decodeTuple(encoded, order, types);
expect(decoded!.ts).toBe('2024-01-15T00:00:00.000Z');
});
it('encodeTuple — boolean false', () => {
const order = ['flag'];
const types = { flag: 'boolean' };
const encoded = encodeTuple({ flag: false }, order, types);
const decoded = decodeTuple(encoded, order, types);
expect(decoded!.flag).toBe(false);
});
it('encodeTuple — 负数', () => {
const order = ['val'];
const types = { val: 'number' };
const encoded = encodeTuple({ val: -42.5 }, order, types);
const decoded = decodeTuple(encoded, order, types);
expect(decoded!.val).toBe(-42.5);
});
it('encodeTuple — 空字符串', () => {
const order = ['s'];
const types = { s: 'string' };
const encoded = encodeTuple({ s: '' }, order, types);
const decoded = decodeTuple(encoded, order, types);
expect(decoded!.s).toBe('');
});
it('encodeTuple — 长字符串', () => {
const order = ['s'];
const types = { s: 'string' };
const long = 'x'.repeat(10000);
const encoded = encodeTuple({ s: long }, order, types);
const decoded = decodeTuple(encoded, order, types);
expect(decoded!.s).toBe(long);
});
it('getColumnEncodingMap 返回正确映射', () => {
const map = getColumnEncodingMap(colOrder, colTypes);
expect(map.get('id')).toBe(ColumnEncoding.STRING);
expect(map.get('age')).toBe(ColumnEncoding.NUMBER);
expect(map.get('active')).toBe(ColumnEncoding.BOOLEAN);
expect(map.get('data')).toBe(ColumnEncoding.JSON);
});
it('decodeTuple — 损坏数据返回 null', () => {
const broken = new Uint8Array([0xff, 0xff, 0xff]);
expect(decodeTuple(broken, colOrder, colTypes)).toBeNull();
});
});
// ===================================================================
// Page Format 整合
// ===================================================================
describe('AriaEngine Page — Format', () => {
const colOrder = ['id', 'name'];
const colTypes: Record<string, string> = { id: 'string', name: 'string' };
it('createPage 创建合法页面', () => {
const page = createPage(100, PageType.DATA);
expect(page.pageId).toBe(100);
expect(page.type).toBe(PageType.DATA);
expect(page.pins).toBe(0);
expect(page.dirty).toBe(true);
});
it('pageInsertRow + pageReadRow 往返', () => {
const page = createPage(1, PageType.DATA);
const row = { id: 'u1', name: 'Test' };
const idx = pageInsertRow(page, row, colOrder, colTypes);
expect(idx).toBe(0);
const read = pageReadRow(page, 0, colOrder, colTypes);
expect(read).not.toBeNull();
expect(read!.id).toBe('u1');
expect(read!.name).toBe('Test');
});
it('pageInsertRow — 多次插入', () => {
const page = createPage(1, PageType.DATA);
for (let i = 0; i < 50; i++) {
const idx = pageInsertRow(page, { id: `${i}`, name: `User${i}` }, colOrder, colTypes);
expect(idx).toBe(i);
}
for (let i = 0; i < 50; i++) {
const row = pageReadRow(page, i, colOrder, colTypes);
expect(row!.id).toBe(`${i}`);
}
});
it('pageDeleteRow 标记删除', () => {
const page = createPage(1, PageType.DATA);
pageInsertRow(page, { id: '1', name: 'A' }, colOrder, colTypes);
pageInsertRow(page, { id: '2', name: 'B' }, colOrder, colTypes);
pageDeleteRow(page, 0);
expect(page.dirty).toBe(true);
// 已删除的行读取失败
const row = pageReadRow(page, 0, colOrder, colTypes);
expect(row).toBeNull();
// 未删除的行仍然可读
const row2 = pageReadRow(page, 1, colOrder, colTypes);
expect(row2!.id).toBe('2');
});
it('pageFromBuffer 从 ArrayBuffer 恢复', () => {
const page = createPage(5, PageType.INDEX);
const restored = pageFromBuffer(5, page.data);
expect(restored.pageId).toBe(5);
expect(restored.type).toBe(PageType.INDEX);
expect(restored.dirty).toBe(false);
});
it('computeChecksum + verifyChecksum', () => {
const page = createPage(1, PageType.DATA);
updateChecksum(page);
expect(verifyChecksum(page)).toBe(true);
// 修改页面 → 校验和失效
new Uint8Array(page.data)[100] = 0xff;
expect(verifyChecksum(page)).toBe(false);
});
});
tests/engine/aria-repair-hardening.test.ts
+249
View File
@@ -0,0 +1,249 @@
/**
* AriaEngine — repair 自愈增强 + 随机操作压力测试
*
* 覆盖：
* 1. repair 清理孤儿页面（meta 未引用的 pg_ 文件）
* 2. repair 清理 OPFS 残留临时文件
* 3. WAL 空洞 + repair → 截断清空（不再重放错位数据）
* 4. 随机操作压力：insert/update/delete + 模拟崩溃（不 checkpoint 断开）→ 重开全量验证
* 5. 随机操作 + 页面化 + 模拟崩溃 → 重开验证
* 6. repair 幂等
*/
import { AriaEngine } from '../../src/engine/aria/index';
import { createSchema } from '../../src/table/schema';
import { installOPFSMock } from '../helpers/opfs-mock';
import 'fake-indexeddb/auto';
let idbCounter = 0;
function uniqueDB(): string {
return `rph-${Date.now()}-${++idbCounter}-${Math.random().toString(36).slice(2, 8)}`;
}
const SCHEMA = () => createSchema('items', {
id: { type: 'string', primaryKey: true },
val: { type: 'number' },
tag: { type: 'string', index: true },
});
// ===================================================================
// repair 增强
// ===================================================================
describe('AriaEngine — repair 自愈增强', () => {
it('清理孤儿页面（meta 未引用的 pg_ 文件）', async () => {
const dbName = uniqueDB();
const engine = new AriaEngine({
storageBackend: 'indexeddb',
pageStorage: true, // 强制页面化（IDB 上也启用，便于断言页面文件）
memtableSizeThreshold: 64 * 1024 * 1024,
checkpointInterval: 100000,
});
await engine.open(dbName, 1);
await engine.createTable(SCHEMA());
await engine.insert('items', [{ id: 'a', val: 1, tag: 'x' }]);
await (engine as any).lsm.flush();
const backend = (engine as any).backend;
const keys = await backend.listKeys();
const pgKeys = keys.filter((k: string) => k.startsWith('pg_'));
expect(pgKeys.length).toBeGreaterThan(0);
// 伪造孤儿页面（模拟崩溃中断的删除流程）
await backend.write('pg_999999', new Uint8Array(4096).buffer);
await backend.write('pg_999998', new Uint8Array(4096).buffer);
expect((await backend.listKeys()).filter((k: string) => k.startsWith('pg_99999'))).toHaveLength(2);
await (engine as any).repair();
// 孤儿页面被清理，正常页面保留
const after = await backend.listKeys();
expect(after).not.toContain('pg_999999');
expect(after).not.toContain('pg_999998');
expect(await engine.count('items')).toBe(1);
await engine.close();
});
it('清理 OPFS 残留临时文件（.crswap/.tmp）', async () => {
// 用 OPFS mock 后端验证 cleanupStaleFiles 被调用
const { files, dir } = installOPFSMock(new Map());
const engine = new AriaEngine({ storageBackend: 'opfs', checkpointInterval: 100000 });
await engine.open('repair-opfs-1', 1);
await engine.createTable(SCHEMA());
await engine.insert('items', [{ id: 'a', val: 1, tag: 'x' }]);
// 制造残留
await dir.getFileHandle('junk.crswap', { create: true });
await dir.getFileHandle('junk2.tmp', { create: true });
expect(Array.from(files.keys()).some((k) => k.endsWith('.crswap'))).toBe(true);
await (engine as any).repair();
const after = Array.from(files.keys());
expect(after.some((k) => k.endsWith('.crswap'))).toBe(false);
expect(after.some((k) => k.endsWith('.tmp'))).toBe(false);
expect(await engine.count('items')).toBe(1);
await engine.close();
});
it('WAL 空洞 + repair → 截断清空（不再重放错位数据）', async () => {
const dbName = uniqueDB();
const engine = new AriaEngine({ storageBackend: 'indexeddb', checkpointInterval: 100000 });
await engine.open(dbName, 1);
await engine.createTable(SCHEMA());
// 数据落盘 + checkpoint 清 WAL
await engine.insert('items', [{ id: 'a', val: 1, tag: 'x' }]);
await (engine as any).lsm.flush();
await (engine as any).wal.checkpoint();
// 新数据只进 WAL
await engine.insert('items', [{ id: 'b', val: 2, tag: 'y' }]);
const backend = (engine as any).backend;
const keys = await backend.listKeys();
const walSeg = keys.find((k: string) => k.startsWith('__wal_'));
// 制造空洞：删掉当前分片（数据仍在内存）
await backend.delete(walSeg);
await (engine as any).repair();
// repair 后 WAL 已清空，无空洞残留
const after = await backend.listKeys();
expect(after.some((k: string) => k.startsWith('__wal_'))).toBe(false);
// 在线自愈：memtable 中的 'b'（已确认写入）随 flush 落盘 → 数据完整不丢
const rows = await engine.find('items', { table: 'items' });
expect(rows).toHaveLength(2);
expect(rows.some((r) => r.id === 'a')).toBe(true);
expect(rows.some((r) => r.id === 'b')).toBe(true);
await engine.close();
});
it('repair 幂等（连续调用无副作用）', async () => {
const dbName = uniqueDB();
const engine = new AriaEngine({ storageBackend: 'indexeddb', checkpointInterval: 100000 });
await engine.open(dbName, 1);
await engine.createTable(SCHEMA());
await engine.insert('items', [{ id: 'a', val: 1, tag: 'x' }]);
await (engine as any).repair();
await (engine as any).repair();
await (engine as any).repair();
expect(await engine.count('items')).toBe(1);
await engine.close();
});
});
// ===================================================================
// 随机操作压力 + 模拟崩溃
// ===================================================================
describe('AriaEngine — 随机操作压力 + 模拟崩溃', () => {
it('500 随机操作（insert/update/delete）→ 模拟崩溃 → 重开验证全部已确认写入', async () => {
const dbName = uniqueDB();
const engine = new AriaEngine({
storageBackend: 'indexeddb',
memtableSizeThreshold: 8 * 1024,
checkpointInterval: 50,
walSyncMode: 'full',
});
await engine.open(dbName, 1);
await engine.createTable(SCHEMA());
let seed = 12345;
const rand = () => { seed = (seed * 1103515245 + 12345) & 0x7fffffff; return seed / 0x7fffffff; };
// 已确认写入集合（模拟应用层已收到成功返回）
const confirmed = new Map<string, { val: number; tag: string }>();
for (let i = 0; i < 500; i++) {
const r = rand();
const id = `k-${Math.floor(rand() * 100)}`;
if (r < 0.6) {
// insert/upsert（随机 id 可能碰撞 → 已存在则跳过，confirmed 保持首个值）
const row = { id, val: Math.floor(rand() * 1000), tag: `t${Math.floor(rand() * 5)}` };
try {
await engine.insert('items', [row]);
confirmed.set(id, row);
} catch (e) {
if ((e as { code?: string }).code !== 'DUPLICATE_KEY') throw e;
}
} else if (r < 0.8) {
// update
const upd = { val: Math.floor(rand() * 1000) };
await engine.update('items', { table: 'items', where: { id } }, upd);
const cur = confirmed.get(id);
if (cur) confirmed.set(id, { ...cur, ...upd });
} else {
// delete
await engine.delete('items', { table: 'items', where: { id } });
confirmed.delete(id);
}
}
// 模拟崩溃：不 close（不 checkpoint），直接断开 backend
await (engine as any).backend.close();
(engine as any).opened = false;
// 重开：WAL 重放 + SSTable 加载
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', checkpointInterval: 50, walSyncMode: 'full' });
await engine2.open(dbName, 1);
const rows = await engine2.find('items', { table: 'items' });
const byId = new Map(rows.map((r) => [r.id, r]));
// 全部已确认写入可见
expect(byId.size).toBe(confirmed.size);
for (const [id, expected] of confirmed) {
expect(byId.has(id)).toBe(true);
expect(byId.get(id)!.val).toBe(expected.val);
}
await engine2.close();
});
it('随机操作 + 页面化 + 模拟崩溃 → 重开验证', async () => {
const files = installOPFSMock(new Map());
const dbName = `rand-opfs-${Date.now()}-${Math.random().toString(36).slice(2, 6)}`;
const engine = new AriaEngine({
storageBackend: 'opfs',
memtableSizeThreshold: 8 * 1024,
checkpointInterval: 50,
walSyncMode: 'full',
});
await engine.open(dbName, 1);
await engine.createTable(SCHEMA());
let seed = 999;
const rand = () => { seed = (seed * 1103515245 + 12345) & 0x7fffffff; return seed / 0x7fffffff; };
const confirmed = new Map<string, { val: number; tag: string }>();
for (let i = 0; i < 300; i++) {
const r = rand();
const id = `p-${Math.floor(rand() * 80)}`;
if (r < 0.6) {
const row = { id, val: Math.floor(rand() * 500), tag: `tag${Math.floor(rand() * 4)}` };
try {
await engine.insert('items', [row]);
confirmed.set(id, row);
} catch (e) {
if ((e as { code?: string }).code !== 'DUPLICATE_KEY') throw e;
}
} else if (r < 0.8) {
const upd = { val: Math.floor(rand() * 500) };
await engine.update('items', { table: 'items', where: { id } }, upd);
const cur = confirmed.get(id);
if (cur) confirmed.set(id, { ...cur, ...upd });
} else {
await engine.delete('items', { table: 'items', where: { id } });
confirmed.delete(id);
}
}
// 模拟崩溃
await (engine as any).backend.close();
(engine as any).opened = false;
const engine2 = new AriaEngine({ storageBackend: 'opfs', checkpointInterval: 50, walSyncMode: 'full' });
await engine2.open(dbName, 1);
const rows = await engine2.find('items', { table: 'items' });
const byId = new Map(rows.map((r) => [r.id, r]));
expect(byId.size).toBe(confirmed.size);
for (const [id, expected] of confirmed) {
expect(byId.get(id)!.val).toBe(expected.val);
}
// 索引查询也验证
const tagged = await engine2.find('items', { table: 'items', where: { tag: 'tag0' } });
const expectedTagged = Array.from(confirmed.values()).filter((v) => v.tag === 'tag0').length;
expect(tagged.length).toBe(expectedTagged);
await engine2.close();
});
});
tests/engine/aria-sstable.test.ts
+264 -129
View File
@@ -1,129 +1,264 @@
/**
* AriaEngine SSTable Builder + Reader 单元测试
*/
import { SSTableBuilder } from '../../src/engine/aria/index/sstable_builder';
import { SSTableReader } from '../../src/engine/aria/index/sstable';
import type { SSTableMeta } from '../../src/engine/aria/types';
// ===================================================================
// SSTable Builder + Reader
// ===================================================================
describe('AriaEngine — SSTable Builder + Reader', () => {
const makeMeta = (data: Uint8Array): SSTableMeta => ({
id: 1, level: 0, minKey: '', maxKey: '\uffff',
blockCount: 1, totalSize: data.byteLength, bloomData: null,
});
it('构建单条目 SSTable 并精确读取', () => {
const builder = new SSTableBuilder(4096);
builder.add('key1', { name: 'Alice', age: 30 });
const { sstableData } = builder.build();
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
const result = reader.get('key1');
expect(result).not.toBeNull();
expect(result!.name).toBe('Alice');
expect(result!.age).toBe(30);
});
it('构建多条 SSTable 并全部读取', () => {
const builder = new SSTableBuilder(4096);
const items: [string, Record<string, unknown>][] = [];
for (let i = 0; i < 100; i++) {
const key = `user-${String(i).padStart(3, '0')}`;
const value = { idx: i, name: `User${i}` };
items.push([key, value]);
builder.add(key, value);
}
const { sstableData } = builder.build();
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
for (const [key, value] of items) {
const result = reader.get(key);
expect(result).not.toBeNull();
expect(result!.idx).toBe(value.idx);
}
});
it('get — 不存在的 key 返回 null', () => {
const builder = new SSTableBuilder(4096);
builder.add('a', { v: 1 });
const { sstableData } = builder.build();
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
expect(reader.get('nonexistent')).toBeNull();
});
it('rangeScan — 范围查询', () => {
const builder = new SSTableBuilder(4096);
for (let i = 0; i < 20; i++) {
builder.add(`k-${String(i).padStart(2, '0')}`, { v: i });
}
const { sstableData } = builder.build();
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
const results: [string, Record<string, unknown>][] = [];
reader.rangeScan('k-05', 'k-10', (k, v) => results.push([k, v]));
expect(results).toHaveLength(6);
expect(results[0][0]).toBe('k-05');
expect(results[results.length - 1][0]).toBe('k-10');
});
it('scanAll — 遍历所有条目', () => {
const builder = new SSTableBuilder(4096);
const count = 50;
for (let i = 0; i < count; i++) {
builder.add(`item-${i}`, { idx: i });
}
const { sstableData } = builder.build();
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
const items: [string, Record<string, unknown>][] = [];
reader.scanAll((k, v) => items.push([k, v]));
expect(items).toHaveLength(count);
});
it('getIndexBlockCount 返回索引块数', () => {
const builder = new SSTableBuilder(256); // 小 block size 触发多个 block
for (let i = 0; i < 100; i++) {
builder.add(`k-${i}`, { data: 'x'.repeat(50) });
}
const { sstableData } = builder.build();
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
expect(reader.getIndexBlockCount()).toBeGreaterThanOrEqual(1);
});
it('边界 — 空 SSTable 不抛异常', () => {
const builder = new SSTableBuilder(4096);
const { sstableData } = builder.build();
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
expect(reader.get('any')).toBeNull();
const results: [string, Record<string, unknown>][] = [];
reader.scanAll((k, v) => results.push([k, v]));
expect(results).toHaveLength(0);
});
it('带特殊字符的 key', () => {
const builder = new SSTableBuilder(4096);
// 必须按键排序添加（按 ASCII 排序：空格 < 短横 < 点号）
builder.add('key with space', { v: 3 });
builder.add('key-with-dash', { v: 1 });
builder.add('key.with.dot', { v: 2 });
const { sstableData } = builder.build();
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
expect(reader.get('key-with-dash')!.v).toBe(1);
expect(reader.get('key.with.dot')!.v).toBe(2);
expect(reader.get('key with space')!.v).toBe(3);
});
it('getEntryCount 返回正确条目数', () => {
const builder = new SSTableBuilder(4096);
builder.add('a', { v: 1 });
builder.add('b', { v: 2 });
builder.add('c', { v: 3 });
expect(builder.getEntryCount()).toBe(3);
});
});
/**
* AriaEngine SSTable Builder + Reader 单元测试
*/
import { SSTableBuilder } from '../../src/engine/aria/index/sstable_builder';
import { SSTableReader } from '../../src/engine/aria/index/sstable';
import type { SSTableMeta } from '../../src/engine/aria/types';
// ===================================================================
// SSTable Builder + Reader
// ===================================================================
describe('AriaEngine — SSTable Builder + Reader', () => {
const makeMeta = (data: Uint8Array): SSTableMeta => ({
id: 1, level: 0, minKey: '', maxKey: '\uffff',
blockCount: 1, totalSize: data.byteLength, bloomData: null,
});
it('构建单条目 SSTable 并精确读取', () => {
const builder = new SSTableBuilder(4096);
builder.add('key1', { name: 'Alice', age: 30 });
const { sstableData } = builder.build();
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
const result = reader.get('key1');
expect(result).not.toBeNull();
expect(result!.name).toBe('Alice');
expect(result!.age).toBe(30);
});
it('构建多条 SSTable 并全部读取', () => {
const builder = new SSTableBuilder(4096);
const items: [string, Record<string, unknown>][] = [];
for (let i = 0; i < 100; i++) {
const key = `user-${String(i).padStart(3, '0')}`;
const value = { idx: i, name: `User${i}` };
items.push([key, value]);
builder.add(key, value);
}
const { sstableData } = builder.build();
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
for (const [key, value] of items) {
const result = reader.get(key);
expect(result).not.toBeNull();
expect(result!.idx).toBe(value.idx);
}
});
it('get — 不存在的 key 返回 null', () => {
const builder = new SSTableBuilder(4096);
builder.add('a', { v: 1 });
const { sstableData } = builder.build();
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
expect(reader.get('nonexistent')).toBeNull();
});
it('rangeScan — 范围查询', () => {
const builder = new SSTableBuilder(4096);
for (let i = 0; i < 20; i++) {
builder.add(`k-${String(i).padStart(2, '0')}`, { v: i });
}
const { sstableData } = builder.build();
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
const results: [string, Record<string, unknown>][] = [];
reader.rangeScan('k-05', 'k-10', (k, v) => results.push([k, v]));
expect(results).toHaveLength(6);
expect(results[0][0]).toBe('k-05');
expect(results[results.length - 1][0]).toBe('k-10');
});
it('scanAll — 遍历所有条目', () => {
const builder = new SSTableBuilder(4096);
const count = 50;
for (let i = 0; i < count; i++) {
builder.add(`item-${i}`, { idx: i });
}
const { sstableData } = builder.build();
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
const items: [string, Record<string, unknown>][] = [];
reader.scanAll((k, v) => items.push([k, v]));
expect(items).toHaveLength(count);
});
it('边界 — 空 SSTable 不抛异常', () => {
const builder = new SSTableBuilder(4096);
const { sstableData } = builder.build();
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
expect(reader.get('any')).toBeNull();
const results: [string, Record<string, unknown>][] = [];
reader.scanAll((k, v) => results.push([k, v]));
expect(results).toHaveLength(0);
});
it('带特殊字符的 key', () => {
const builder = new SSTableBuilder(4096);
// 必须按键排序添加（按 ASCII 排序：空格 < 短横 < 点号）
builder.add('key with space', { v: 3 });
builder.add('key-with-dash', { v: 1 });
builder.add('key.with.dot', { v: 2 });
const { sstableData } = builder.build();
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
expect(reader.get('key-with-dash')!.v).toBe(1);
expect(reader.get('key.with.dot')!.v).toBe(2);
expect(reader.get('key with space')!.v).toBe(3);
});
it('getEntryCount 返回正确条目数', () => {
const builder = new SSTableBuilder(4096);
builder.add('a', { v: 1 });
builder.add('b', { v: 2 });
builder.add('c', { v: 3 });
expect(builder.getEntryCount()).toBe(3);
});
});
// ===================================================================
// v0.4.5: 整文件 CRC-32 校验
// ===================================================================
describe('AriaEngine — SSTable CRC-32 校验', () => {
const makeMeta = (data: Uint8Array): SSTableMeta => ({
id: 1, level: 0, minKey: '', maxKey: '\uffff',
blockCount: 1, totalSize: data.byteLength, bloomData: null,
});
it('新构建的 SSTable 写入真实非零 checksum', () => {
const builder = new SSTableBuilder(4096);
builder.add('key1', { name: 'Alice' });
const { sstableData } = builder.build();
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
// footer 最后 4 字节为真实 checksum（非 0）
const stored = new DataView(sstableData.buffer, sstableData.byteOffset + sstableData.byteLength - 4, 4).getUint32(0, false);
expect(stored).not.toBe(0);
expect(reader.verifyChecksum()).toBe(true);
});
it('正常文件校验通过后可正常读取', () => {
const builder = new SSTableBuilder(4096);
for (let i = 0; i < 50; i++) builder.add(`k-${i}`, { v: i });
const { sstableData } = builder.build();
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
expect(reader.verifyChecksum()).toBe(true);
expect(reader.get('k-30')).toEqual({ v: 30 });
});
it('数据块字节篡改 → 校验失败', () => {
const builder = new SSTableBuilder(4096);
for (let i = 0; i < 50; i++) builder.add(`k-${i}`, { v: i, pad: 'x'.repeat(64) });
const { sstableData } = builder.build();
// 篡改数据区中部一个字节（非 footer）
const corrupted = new Uint8Array(sstableData);
corrupted[Math.floor(corrupted.length / 2)] ^= 0xff;
const reader = new SSTableReader(corrupted, makeMeta(corrupted));
// 结构仍可解析（长度一致），但 CRC 必须失败
const stored = new DataView(corrupted.buffer, corrupted.byteOffset + corrupted.byteLength - 4, 4).getUint32(0, false);
expect(stored).not.toBe(0);
expect(reader.verifyChecksum()).toBe(false);
});
it('索引块字节篡改 → 校验失败', () => {
const builder = new SSTableBuilder(4096);
for (let i = 0; i < 20; i++) builder.add(`k-${i}`, { v: i });
const { sstableData } = builder.build();
// 从 footer 反推索引块位置并篡改
const footerOffset = sstableData.byteLength - 32;
const view = new DataView(sstableData.buffer, sstableData.byteOffset, sstableData.byteLength);
const indexOffset = view.getUint32(footerOffset, false);
const corrupted = new Uint8Array(sstableData);
corrupted[indexOffset + 8] ^= 0x01;
const reader = new SSTableReader(corrupted, makeMeta(corrupted));
expect(reader.verifyChecksum()).toBe(false);
});
it('旧版文件（checksum=0）跳过校验，保持兼容', () => {
const builder = new SSTableBuilder(4096);
builder.add('legacy', { v: 1 });
const { sstableData } = builder.build();
// 手工把 checksum 字段清零，模拟旧版写入产物
const legacy = new Uint8Array(sstableData);
new DataView(legacy.buffer).setUint32(legacy.byteLength - 4, 0, false);
const reader = new SSTableReader(legacy, makeMeta(legacy));
const stored = new DataView(legacy.buffer, legacy.byteOffset + legacy.byteLength - 4, 4).getUint32(0, false);
expect(stored).toBe(0);
expect(reader.verifyChecksum()).toBe(true); // 跳过校验
expect(reader.get('legacy')).toEqual({ v: 1 }); // 数据仍可读
});
it('v1 旧格式文件（magic SSTB，checksum=0）校验通过且可读', () => {
// 手工构造 v1 格式最小文件：单条记录 + 索引 + footer（magic SSTB, checksum 0）
const encoder = new TextEncoder();
const keyBytes = encoder.encode('old-key');
const valBytes = encoder.encode(JSON.stringify({ v: 42 }));
// data block: entryCount(u32)=1 + keyLen(u16)+key + valLen(u16)+val
const block = new ArrayBuffer(4 + 2 + keyBytes.length + 2 + valBytes.length);
const bv = new DataView(block);
let off = 0;
bv.setUint32(off, 1, false); off += 4;
bv.setUint16(off, keyBytes.length, false); off += 2;
new Uint8Array(block).set(keyBytes, off); off += keyBytes.length;
bv.setUint16(off, valBytes.length, false); off += 2;
new Uint8Array(block).set(valBytes, off); off += valBytes.length;
// index block: entryCount(u32)=1 + keyLen(u16)+key + blockOffset(u32) + blockSize(u32)
const blockSize = block.byteLength;
const idxKey = encoder.encode('old-key');
const idxBlock = new ArrayBuffer(4 + 2 + idxKey.length + 4 + 4);
const iv = new DataView(idxBlock);
let ioff = 0;
iv.setUint32(ioff, 1, false); ioff += 4;
iv.setUint16(ioff, idxKey.length, false); ioff += 2;
new Uint8Array(idxBlock).set(idxKey, ioff); ioff += idxKey.length;
iv.setUint32(ioff, 0, false); ioff += 4; // blockOffset = 0
iv.setUint32(ioff, blockSize, false); ioff += 4;
const footerSize = 32;
const total = block.byteLength + idxBlock.byteLength + footerSize;
const file = new ArrayBuffer(total);
const fv = new DataView(file);
new Uint8Array(file).set(new Uint8Array(block), 0);
new Uint8Array(file).set(new Uint8Array(idxBlock), block.byteLength);
const fo = block.byteLength + idxBlock.byteLength;
fv.setUint32(fo, block.byteLength, false); // index_offset
fv.setUint32(fo + 4, idxBlock.byteLength, false);
fv.setUint32(fo + 8, 0, false); // bloom_offset
fv.setUint32(fo + 12, 0, false); // bloom_size
fv.setUint32(fo + 16, 0, false); // bloom_hash_count
fv.setUint32(fo + 20, 1, false); // entry_count
fv.setUint32(fo + 24, 0x53535442, false); // magic "SSTB" (v1)
fv.setUint32(fo + 28, 0, false); // checksum = 0（旧版无校验）
const data = new Uint8Array(file);
const reader = new SSTableReader(data, makeMeta(data));
const stored = new DataView(data.buffer, data.byteOffset + data.byteLength - 4, 4).getUint32(0, false);
expect(stored).toBe(0);
expect(reader.verifyChecksum()).toBe(true);
expect(reader.get('old-key')).toEqual({ v: 42 });
});
it('footer checksum 字段自身篡改 → 校验失败', () => {
const builder = new SSTableBuilder(4096);
builder.add('a', { v: 1 });
const { sstableData } = builder.build();
const tampered = new Uint8Array(sstableData);
const view = new DataView(tampered.buffer);
view.setUint32(tampered.byteLength - 4, view.getUint32(tampered.byteLength - 4, false) ^ 0x000000ff, false);
const reader = new SSTableReader(tampered, makeMeta(tampered));
expect(reader.verifyChecksum()).toBe(false);
});
});
tests/engine/aria-wal-crc.test.ts
+218
View File
@@ -0,0 +1,218 @@
/**
* AriaEngine WAL — CRC-32 校验升级测试
*
* 覆盖：
* 1. 新算法记录往返（append → recover）
* 2. 篡改记录字节 → CRC 不匹配跳过
* 3. 旧版弱校验记录兼容（双算法探测）
* 4. appendBatch 组提交记录往返
* 5. 损坏记录跳过不影响后续完整记录
*/
import { WAL } from '../../src/engine/aria/wal/log';
import { WALRecordType } from '../../src/engine/aria/types';
// ===================================================================
// WAL 内存 Store（模拟 IStorageBackend 单记录语义）
// ===================================================================
class MemoryWALStore {
chunks: Uint8Array[] = [];
truncated = false;
async append(data: Uint8Array): Promise<void> {
this.chunks.push(data);
}
async readAll(): Promise<Uint8Array> {
const total = this.chunks.reduce((s, c) => s + c.byteLength, 0);
const combined = new Uint8Array(total);
let off = 0;
for (const c of this.chunks) { combined.set(c, off); off += c.byteLength; }
return combined;
}
async truncate(): Promise<void> {
this.chunks = [];
this.truncated = true;
}
async exists(): Promise<boolean> {
return this.chunks.length > 0;
}
}
function makeRecord(over: Partial<import('../../src/engine/aria/types').WALRecord> = {}) {
return {
type: WALRecordType.INSERT,
txnId: 0,
tableName: 'users',
key: 'u-1',
data: { id: 'u-1', name: 'Alice' },
...over,
};
}
describe('AriaEngine — WAL CRC-32 校验', () => {
it('新算法记录 append → recover 往返完整', async () => {
const store = new MemoryWALStore();
const wal = new WAL(store as never, true, 'full');
await wal.append(makeRecord({ key: 'u-1' }));
await wal.append(makeRecord({ key: 'u-2', data: { id: 'u-2', name: 'Bob' } }));
const records: import('../../src/engine/aria/types').WALRecord[] = [];
await wal.recover((r) => records.push(r));
expect(records).toHaveLength(2);
expect(records[0].key).toBe('u-1');
expect(records[1].data).toEqual({ id: 'u-2', name: 'Bob' });
});
it('appendBatch 组提交记录往返完整', async () => {
const store = new MemoryWALStore();
const wal = new WAL(store as never, true, 'full');
await wal.appendBatch([
makeRecord({ key: 'a' }),
makeRecord({ key: 'b' }),
makeRecord({ key: 'c' }),
]);
const records: import('../../src/engine/aria/types').WALRecord[] = [];
await wal.recover((r) => records.push(r));
expect(records.map((r) => r.key)).toEqual(['a', 'b', 'c']);
expect(records.map((r) => r.lsn)).toEqual([1, 2, 3]);
});
it('篡改记录数据 → CRC 不匹配跳过', async () => {
const store = new MemoryWALStore();
const wal = new WAL(store as never, true, 'full');
await wal.append(makeRecord({ key: 'good-1' }));
await wal.append(makeRecord({ key: 'evil' }));
await wal.append(makeRecord({ key: 'good-2' }));
// 篡改第二条记录的 data 字节（LSN=2 的记录）
const raw = await store.readAll();
const view = new DataView(raw.buffer, raw.byteOffset, raw.byteLength);
// 定位第二条记录起始：按记录格式顺序扫描直到 LSN=2
// [lsn:4][type:1][txnId:4][tblLen:2][table][keyLen:2][key][jsonLen:4][json][crc:4]
let offset = 0;
while (offset + 4 <= raw.byteLength && view.getUint32(offset, false) !== 2) {
const tblLen = view.getUint16(offset + 9, false);
const keyLenOff = offset + 11 + tblLen;
const keyLen = view.getUint16(keyLenOff, false);
const jsonLenOff = keyLenOff + 2 + keyLen;
const jsonLen = view.getUint32(jsonLenOff, false);
offset += 4 + 1 + 4 + 2 + tblLen + 2 + keyLen + 4 + jsonLen + 4;
}
expect(view.getUint32(offset, false)).toBe(2);
// 篡改 data 区（json 起始处第 4 字节）
const tblLen2 = view.getUint16(offset + 9, false);
const jsonStart = offset + 4 + 1 + 4 + 2 + tblLen2 + 2 + view.getUint16(offset + 11 + tblLen2, false) + 4;
raw[jsonStart + 4] ^= 0x01;
// 写回 store（readAll 返回的是合并副本）
store.chunks = [raw];
// 恢复：坏记录被跳过，前后好记录保留
const records: import('../../src/engine/aria/types').WALRecord[] = [];
await wal.recover((r) => records.push(r));
expect(records.map((r) => r.key)).toEqual(['good-1', 'good-2']);
});
it('旧版弱校验记录兼容（双算法探测）', async () => {
const store = new MemoryWALStore();
const wal = new WAL(store as never, true, 'full');
// 手工构造一条"旧算法"记录：与 encodeRecord 布局一致但 CRC 用旧算法计算
const record = {
lsn: 1,
type: WALRecordType.INSERT as number,
txnId: 0,
tableName: 'legacy',
key: 'old',
data: { id: 'old', v: 1 },
};
const encoder = new TextEncoder();
const tblB = encoder.encode(record.tableName);
const keyB = encoder.encode(record.key);
const jsonB = encoder.encode(JSON.stringify(record.data));
const size = 4 + 1 + 4 + 2 + tblB.length + 2 + keyB.length + 4 + jsonB.length + 4;
const buf = new ArrayBuffer(size);
const v = new DataView(buf);
let o = 0;
v.setUint32(o, record.lsn, false); o += 4;
v.setUint8(o, record.type); o += 1;
v.setUint32(o, record.txnId, false); o += 4;
v.setUint16(o, tblB.length, false); o += 2;
new Uint8Array(buf).set(tblB, o); o += tblB.length;
v.setUint16(o, keyB.length, false); o += 2;
new Uint8Array(buf).set(keyB, o); o += keyB.length;
v.setUint32(o, jsonB.length, false); o += 4;
new Uint8Array(buf).set(jsonB, o); o += jsonB.length;
// 旧算法 CRC
let crc = 0;
const u8 = new Uint8Array(buf, 0, o);
for (let i = 0; i < u8.length; i++) crc = ((crc << 5) - crc + u8[i]) | 0;
v.setUint32(o, crc >>> 0, false);
store.chunks.push(new Uint8Array(buf));
const records: import('../../src/engine/aria/types').WALRecord[] = [];
await wal.recover((r) => records.push(r));
expect(records).toHaveLength(1);
expect(records[0].tableName).toBe('legacy');
expect(records[0].data).toEqual({ id: 'old', v: 1 });
});
it('旧算法记录 + 新算法记录混合 → 全部正确恢复', async () => {
const store = new MemoryWALStore();
const wal = new WAL(store as never, true, 'full');
// 先放一条旧算法记录
const record = {
lsn: 1,
type: WALRecordType.DELETE as number,
txnId: 0,
tableName: 't',
key: 'old-key',
data: undefined as Record<string, unknown> | undefined,
};
const encoder = new TextEncoder();
const tblB = encoder.encode(record.tableName);
const keyB = encoder.encode(record.key);
const size = 4 + 1 + 4 + 2 + tblB.length + 2 + keyB.length + 4 + 4;
const buf = new ArrayBuffer(size);
const v = new DataView(buf);
let o = 0;
v.setUint32(o, record.lsn, false); o += 4;
v.setUint8(o, record.type); o += 1;
v.setUint32(o, record.txnId, false); o += 4;
v.setUint16(o, tblB.length, false); o += 2;
new Uint8Array(buf).set(tblB, o); o += tblB.length;
v.setUint16(o, keyB.length, false); o += 2;
new Uint8Array(buf).set(keyB, o); o += keyB.length;
v.setUint32(o, 0, false); o += 4; // jsonLen=0
let crc = 0;
const u8 = new Uint8Array(buf, 0, o);
for (let i = 0; i < u8.length; i++) crc = ((crc << 5) - crc + u8[i]) | 0;
v.setUint32(o, crc >>> 0, false);
store.chunks.push(new Uint8Array(buf));
// 再追加一条新算法记录（LSN=2）
await wal.append(makeRecord({ key: 'new-key', lsn: 2 } as never));
const records: import('../../src/engine/aria/types').WALRecord[] = [];
await wal.recover((r) => records.push(r));
expect(records.map((r) => r.key)).toEqual(['old-key', 'new-key']);
expect(records[0].type).toBe(WALRecordType.DELETE);
});
it('checkpoint 截断后 exists 返回 false', async () => {
const store = new MemoryWALStore();
const wal = new WAL(store as never, true, 'full');
await wal.append(makeRecord({ key: 'x' }));
expect(await store.exists()).toBe(true);
await wal.checkpoint();
expect(store.truncated).toBe(true);
expect(await store.exists()).toBe(false);
});
});
tests/engine/aria-wal-mvcc.test.ts
+225 -289
View File
@@ -1,289 +1,225 @@
/**
* AriaEngine WAL + MVCC 单元测试
*/
import { WAL, type WALStore } from '../../src/engine/aria/wal/log';
import { WALRecordType, type WALRecord } from '../../src/engine/aria/types';
import { CheckpointManager, type Flushable } from '../../src/engine/aria/wal/checkpoint';
import { MVCCManager } from '../../src/engine/aria/transaction/mvcc';
// ===================================================================
// WAL 存储 Mock
// ===================================================================
class MockWALStore implements WALStore {
chunks: Uint8Array[] = [];
async append(data: Uint8Array) { this.chunks.push(data); }
async readAll(): Promise<Uint8Array> {
const total = this.chunks.reduce((s, c) => s + c.byteLength, 0);
const combined = new Uint8Array(total);
let off = 0;
for (const c of this.chunks) { combined.set(c, off); off += c.byteLength; }
return combined;
}
async truncate() { this.chunks = []; }
async exists() { return this.chunks.length > 0; }
}
// ===================================================================
// WAL 测试
// ===================================================================
describe('AriaEngine — WAL', () => {
it('append 记录后可恢复', async () => {
const store = new MockWALStore();
const wal = new WAL(store, true, 'full');
wal.append({ type: WALRecordType.INSERT, txnId: 1, tableName: 'users', key: '1', data: { name: 'Alice' } });
wal.append({ type: WALRecordType.INSERT, txnId: 1, tableName: 'users', key: '2', data: { name: 'Bob' } });
const records: WALRecord[] = [];
await wal.recover((r) => records.push(r));
expect(records).toHaveLength(2);
expect(records[0].tableName).toBe('users');
expect(records[0].key).toBe('1');
});
it('batch 模式缓冲后 flush', async () => {
const store = new MockWALStore();
const wal = new WAL(store, true, 'batch');
wal.append({ type: WALRecordType.UPDATE, txnId: 2, tableName: 'items', key: 'a', data: { v: 1 } });
wal.append({ type: WALRecordType.DELETE, txnId: 2, tableName: 'items', key: 'b' });
// 未 flush 前无法恢复
let records: WALRecord[] = [];
await wal.recover((r) => records.push(r));
expect(records).toHaveLength(0);
await wal.flush();
records = [];
await wal.recover((r) => records.push(r));
expect(records).toHaveLength(2);
});
it('none 模式不记录', async () => {
const store = new MockWALStore();
const wal = new WAL(store, false, 'none');
wal.append({ type: WALRecordType.INSERT, txnId: 3, tableName: 'x', key: 'y', data: {} });
expect(store.chunks).toHaveLength(0);
});
it('checkpoint 清空 WAL', async () => {
const store = new MockWALStore();
const wal = new WAL(store, true, 'full');
wal.append({ type: WALRecordType.CREATE_TABLE, txnId: 0, tableName: 't', key: '' });
expect(await store.exists()).toBe(true);
await wal.checkpoint();
expect(await store.exists()).toBe(false);
});
it('getLSN 跟踪序列号', () => {
const store = new MockWALStore();
const wal = new WAL(store, true, 'full');
expect(wal.getLSN()).toBe(0);
wal.append({ type: WALRecordType.BEGIN, txnId: 10, tableName: '', key: '' });
expect(wal.getLSN()).toBe(1);
wal.append({ type: WALRecordType.COMMIT, txnId: 10, tableName: '', key: '' });
expect(wal.getLSN()).toBe(2);
});
it('isEnabled 反映配置', () => {
expect(new WAL(new MockWALStore(), true).isEnabled()).toBe(true);
expect(new WAL(new MockWALStore(), false).isEnabled()).toBe(false);
});
it('多种记录类型编解码', async () => {
const store = new MockWALStore();
const wal = new WAL(store, true, 'full');
wal.append({ type: WALRecordType.BEGIN, txnId: 100, tableName: '', key: '' });
wal.append({ type: WALRecordType.INSERT, txnId: 100, tableName: 'users', key: '1', data: { x: 'hello' } });
wal.append({ type: WALRecordType.UPDATE, txnId: 100, tableName: 'users', key: '1', data: { x: 'world' } });
wal.append({ type: WALRecordType.COMMIT, txnId: 100, tableName: '', key: '' });
const records: WALRecord[] = [];
await wal.recover((r) => records.push(r));
expect(records).toHaveLength(4);
expect(records[0].type).toBe(WALRecordType.BEGIN);
expect(records[1].type).toBe(WALRecordType.INSERT);
expect(records[2].type).toBe(WALRecordType.UPDATE);
expect(records[3].type).toBe(WALRecordType.COMMIT);
});
it('getBufferedCount 返回缓冲数', () => {
const wal = new WAL(new MockWALStore(), true, 'batch');
expect(wal.getBufferedCount()).toBe(0);
wal.append({ type: WALRecordType.INSERT, txnId: 0, tableName: 't', key: 'k' });
expect(wal.getBufferedCount()).toBe(1);
});
});
// ===================================================================
// Checkpoint 测试（使用安全 Mock，避免 null 引用导致 CI 卡死）
// ===================================================================
describe('AriaEngine — CheckpointManager', () => {
class MockFlushable implements Flushable { flushed = false; async flushAll() { this.flushed = true; } }
class MockLSM { flushed = false; async flush() { this.flushed = true; } }
class MockWAL { checkpointed = false; async checkpoint() { this.checkpointed = true; } async flush() {} }
it('tick 未达间隔不触发 checkpoint', async () => {
const lsm = new MockLSM();
const wal = new MockWAL();
const flushable = new MockFlushable();
const cm = new CheckpointManager(lsm as any, wal as any, flushable, 100);
await cm.tick();
await cm.tick();
expect(cm.getOpCount()).toBe(2);
expect(flushable.flushed).toBe(false);
expect(lsm.flushed).toBe(false);
});
it('setInterval 修改间隔后 tick 触发 checkpoint', async () => {
const lsm = new MockLSM();
const wal = new MockWAL();
const cm = new CheckpointManager(lsm as any, wal as any, null, 1000);
cm.setInterval(2);
await cm.tick();
await cm.tick();
expect(cm.getOpCount()).toBe(0); // reset after checkpoint
expect(lsm.flushed).toBe(true);
expect(wal.checkpointed).toBe(true);
});
it('forceCheckpoint 强制触发', async () => {
const lsm = new MockLSM();
const wal = new MockWAL();
const cm = new CheckpointManager(lsm as any, wal as any, null, 100);
await cm.forceCheckpoint();
expect(cm.getOpCount()).toBe(0);
expect(lsm.flushed).toBe(true);
expect(wal.checkpointed).toBe(true);
});
});
// ===================================================================
// MVCC 测试
// ===================================================================
describe('AriaEngine — MVCC', () => {
let mvcc: MVCCManager;
beforeEach(() => { mvcc = new MVCCManager(); });
it('beginTransaction 分配唯一 ID', () => {
const id1 = mvcc.beginTransaction();
const id2 = mvcc.beginTransaction();
expect(id1).not.toBe(id2);
expect(mvcc.isActive(id1)).toBe(true);
expect(mvcc.isActive(id2)).toBe(true);
});
it('commit 后 isActive 返回 false', () => {
const txnId = mvcc.beginTransaction();
mvcc.commitTransaction(txnId);
expect(mvcc.isActive(txnId)).toBe(false);
});
it('rollback 后 isActive 返回 false', () => {
const txnId = mvcc.beginTransaction();
mvcc.rollbackTransaction(txnId);
expect(mvcc.isActive(txnId)).toBe(false);
});
it('writeVersion + readVersion 往返', () => {
const txnId = mvcc.beginTransaction();
mvcc.writeVersion('users', '1', { name: 'Alice', age: 30 }, txnId);
const val = mvcc.readVersion('users', '1', txnId);
expect(val).not.toBeNull();
expect(val!.name).toBe('Alice');
});
it('未提交版本对其他事务不可见', () => {
const txn1 = mvcc.beginTransaction();
mvcc.writeVersion('users', '1', { name: 'Alice' }, txn1);
const txn2 = mvcc.beginTransaction();
const val = mvcc.readVersion('users', '1', txn2);
expect(val).toBeNull(); // txn1's write not yet committed
});
it('commit 后新事务可见', () => {
const txn1 = mvcc.beginTransaction();
mvcc.writeVersion('users', '1', { name: 'Alice' }, txn1);
mvcc.commitTransaction(txn1);
const txn2 = mvcc.beginTransaction();
const val = mvcc.readVersion('users', '1', txn2);
expect(val).not.toBeNull();
expect(val!.name).toBe('Alice');
});
it('rollback 移除写入的版本', () => {
const txnId = mvcc.beginTransaction();
mvcc.writeVersion('users', '1', { name: 'Temp' }, txnId);
mvcc.rollbackTransaction(txnId);
const txn2 = mvcc.beginTransaction();
expect(mvcc.readVersion('users', '1', txn2)).toBeNull();
});
it('deleteVersion 创建墓碑', () => {
const txnId = mvcc.beginTransaction();
mvcc.writeVersion('users', '1', { name: 'Alice' }, txnId);
mvcc.commitTransaction(txnId);
// delete
const txn2 = mvcc.beginTransaction();
mvcc.deleteVersion('users', '1', txn2);
mvcc.commitTransaction(txn2);
// 删除后读取
const txn3 = mvcc.beginTransaction();
const val = mvcc.readVersion('users', '1', txn3);
expect(val).not.toBeNull();
expect((val! as any).__mvcc_tombstone).toBe(true);
});
it('getLatestCommittedVersions 返回最新已提交', () => {
const txnId = mvcc.beginTransaction();
mvcc.writeVersion('users', '1', { name: 'Alice' }, txnId);
mvcc.writeVersion('users', '2', { name: 'Bob' }, txnId);
mvcc.commitTransaction(txnId);
const result = mvcc.getLatestCommittedVersions('users');
expect(result['1'].name).toBe('Alice');
expect(result['2'].name).toBe('Bob');
});
it('clearTable 清理指定表', () => {
const txnId = mvcc.beginTransaction();
mvcc.writeVersion('users', '1', { name: 'A' }, txnId);
mvcc.writeVersion('users', '2', { name: 'B' }, txnId);
mvcc.writeVersion('orders', 'o1', { amt: 100 }, txnId);
mvcc.commitTransaction(txnId);
mvcc.clearTable('users');
const txn2 = mvcc.beginTransaction();
expect(mvcc.readVersion('users', '1', txn2)).toBeNull();
expect(mvcc.readVersion('users', '2', txn2)).toBeNull();
expect(mvcc.readVersion('orders', 'o1', txn2)).not.toBeNull();
});
it('getActiveTxnCount 返回活跃事务数', () => {
expect(mvcc.getActiveTxnCount()).toBe(0);
mvcc.beginTransaction();
mvcc.beginTransaction();
expect(mvcc.getActiveTxnCount()).toBe(2);
});
it('gc 清理过旧版本', () => {
// 创建很多版本后 gc
for (let i = 0; i < 200; i++) {
const txnId = mvcc.beginTransaction();
mvcc.writeVersion('users', '1', { ver: i }, txnId);
mvcc.commitTransaction(txnId);
}
mvcc.gc(100);
// gc 后应可继续操作
const txnId = mvcc.beginTransaction();
const val = mvcc.readVersion('users', '1', txnId);
expect(val).not.toBeNull();
});
});
/**
* AriaEngine WAL + MVCC 单元测试
*/
import { WAL, type WALStore } from '../../src/engine/aria/wal/log';
import { WALRecordType, type WALRecord } from '../../src/engine/aria/types';
import { CheckpointManager, type Flushable } from '../../src/engine/aria/wal/checkpoint';
import { MVCCManager } from '../../src/engine/aria/transaction/mvcc';
// ===================================================================
// WAL 存储 Mock
// ===================================================================
class MockWALStore implements WALStore {
chunks: Uint8Array[] = [];
async append(data: Uint8Array) { this.chunks.push(data); }
async readAll(): Promise<Uint8Array> {
const total = this.chunks.reduce((s, c) => s + c.byteLength, 0);
const combined = new Uint8Array(total);
let off = 0;
for (const c of this.chunks) { combined.set(c, off); off += c.byteLength; }
return combined;
}
async truncate() { this.chunks = []; }
async exists() { return this.chunks.length > 0; }
}
// ===================================================================
// WAL 测试
// ===================================================================
describe('AriaEngine — WAL', () => {
it('append 记录后可恢复', async () => {
const store = new MockWALStore();
const wal = new WAL(store, true, 'full');
wal.append({ type: WALRecordType.INSERT, txnId: 1, tableName: 'users', key: '1', data: { name: 'Alice' } });
wal.append({ type: WALRecordType.INSERT, txnId: 1, tableName: 'users', key: '2', data: { name: 'Bob' } });
const records: WALRecord[] = [];
await wal.recover((r) => records.push(r));
expect(records).toHaveLength(2);
expect(records[0].tableName).toBe('users');
expect(records[0].key).toBe('1');
});
it('batch 模式缓冲后 flush', async () => {
const store = new MockWALStore();
const wal = new WAL(store, true, 'batch');
wal.append({ type: WALRecordType.UPDATE, txnId: 2, tableName: 'items', key: 'a', data: { v: 1 } });
wal.append({ type: WALRecordType.DELETE, txnId: 2, tableName: 'items', key: 'b' });
// 未 flush 前无法恢复
let records: WALRecord[] = [];
await wal.recover((r) => records.push(r));
expect(records).toHaveLength(0);
await wal.flush();
records = [];
await wal.recover((r) => records.push(r));
expect(records).toHaveLength(2);
});
it('none 模式不记录', async () => {
const store = new MockWALStore();
const wal = new WAL(store, false, 'none');
wal.append({ type: WALRecordType.INSERT, txnId: 3, tableName: 'x', key: 'y', data: {} });
expect(store.chunks).toHaveLength(0);
});
it('checkpoint 清空 WAL', async () => {
const store = new MockWALStore();
const wal = new WAL(store, true, 'full');
wal.append({ type: WALRecordType.CREATE_TABLE, txnId: 0, tableName: 't', key: '' });
expect(await store.exists()).toBe(true);
await wal.checkpoint();
expect(await store.exists()).toBe(false);
});
it('多种记录类型编解码', async () => {
const store = new MockWALStore();
const wal = new WAL(store, true, 'full');
wal.append({ type: WALRecordType.BEGIN, txnId: 100, tableName: '', key: '' });
wal.append({ type: WALRecordType.INSERT, txnId: 100, tableName: 'users', key: '1', data: { x: 'hello' } });
wal.append({ type: WALRecordType.UPDATE, txnId: 100, tableName: 'users', key: '1', data: { x: 'world' } });
wal.append({ type: WALRecordType.COMMIT, txnId: 100, tableName: '', key: '' });
const records: WALRecord[] = [];
await wal.recover((r) => records.push(r));
expect(records).toHaveLength(4);
expect(records[0].type).toBe(WALRecordType.BEGIN);
expect(records[1].type).toBe(WALRecordType.INSERT);
expect(records[2].type).toBe(WALRecordType.UPDATE);
expect(records[3].type).toBe(WALRecordType.COMMIT);
});
it('getBufferedCount 返回缓冲数', () => {
const wal = new WAL(new MockWALStore(), true, 'batch');
expect(wal.getBufferedCount()).toBe(0);
wal.append({ type: WALRecordType.INSERT, txnId: 0, tableName: 't', key: 'k' });
expect(wal.getBufferedCount()).toBe(1);
});
});
// ===================================================================
// Checkpoint 测试（使用安全 Mock，避免 null 引用导致 CI 卡死）
// ===================================================================
describe('AriaEngine — CheckpointManager', () => {
class MockFlushable implements Flushable { flushed = false; async flushAll() { this.flushed = true; } }
class MockLSM { flushed = false; async flush() { this.flushed = true; } }
class MockWAL { checkpointed = false; async checkpoint() { this.checkpointed = true; } async flush() {} }
it('tick 未达间隔不触发 checkpoint', async () => {
const lsm = new MockLSM();
const wal = new MockWAL();
const flushable = new MockFlushable();
const cm = new CheckpointManager(lsm as any, wal as any, flushable, 100);
await cm.tick();
await cm.tick();
expect(flushable.flushed).toBe(false);
expect(lsm.flushed).toBe(false);
});
it('tick 达到间隔触发 checkpoint', async () => {
const lsm = new MockLSM();
const wal = new MockWAL();
const cm = new CheckpointManager(lsm as any, wal as any, null, 2);
await cm.tick();
expect(lsm.flushed).toBe(false);
await cm.tick();
expect(lsm.flushed).toBe(true);
expect(wal.checkpointed).toBe(true);
});
it('checkpoint 直接触发', async () => {
const lsm = new MockLSM();
const wal = new MockWAL();
const cm = new CheckpointManager(lsm as any, wal as any, null, 100);
await cm.checkpoint();
expect(lsm.flushed).toBe(true);
expect(wal.checkpointed).toBe(true);
});
});
// ===================================================================
// MVCC 测试
// ===================================================================
describe('AriaEngine — MVCC', () => {
let mvcc: MVCCManager;
beforeEach(() => { mvcc = new MVCCManager(); });
it('beginTransaction 分配唯一 ID', () => {
const id1 = mvcc.beginTransaction();
const id2 = mvcc.beginTransaction();
expect(id1).not.toBe(id2);
});
it('commit 标记版本为已提交并清理事务登记', () => {
const txnId = mvcc.beginTransaction();
mvcc.writeVersion('users', '1', { name: 'Alice' }, txnId);
mvcc.commitTransaction(txnId);
const versions = (mvcc as unknown as { versionStore: Map<string, { committed: boolean }[]> }).versionStore.get('users.1');
expect(versions).toHaveLength(1);
expect(versions![0].committed).toBe(true);
});
it('rollback 移除写入的版本', () => {
const txnId = mvcc.beginTransaction();
mvcc.writeVersion('users', '1', { name: 'Temp' }, txnId);
mvcc.rollbackTransaction(txnId);
const store = (mvcc as unknown as { versionStore: Map<string, unknown> }).versionStore;
expect(store.has('users.1')).toBe(false);
});
it('writeVersion 记录版本链（undo log 语义）', () => {
const txnId = mvcc.beginTransaction();
mvcc.writeVersion('users', '1', { name: 'Alice', age: 30 }, txnId);
mvcc.writeVersion('users', '1', { name: 'Alice', age: 31 }, txnId);
const store = (mvcc as unknown as { versionStore: Map<string, { prevVersion: unknown }[]> }).versionStore;
const versions = store.get('users.1')!;
expect(versions).toHaveLength(2);
expect(versions[1].prevVersion).not.toBeNull(); // 版本链
});
it('deleteVersion 写入墓碑版本', () => {
const txnId = mvcc.beginTransaction();
mvcc.deleteVersion('users', '1', txnId);
mvcc.commitTransaction(txnId);
const store = (mvcc as unknown as { versionStore: Map<string, { data: Record<string, unknown>; committed: boolean }[]> }).versionStore;
const versions = store.get('users.1')!;
expect(versions[0].committed).toBe(true);
expect((versions[0].data as unknown as { __mvcc_tombstone: boolean }).__mvcc_tombstone).toBe(true);
});
it('活跃事务可并发登记', () => {
const t1 = mvcc.beginTransaction();
const t2 = mvcc.beginTransaction();
mvcc.writeVersion('users', '1', { v: 1 }, t1);
mvcc.writeVersion('users', '2', { v: 2 }, t2);
// 两个事务的写入各自独立记录
const store = (mvcc as unknown as { versionStore: Map<string, unknown[]> }).versionStore;
expect(store.has('users.1')).toBe(true);
expect(store.has('users.2')).toBe(true);
});
it('gc 清理过旧版本', () => {
// 创建很多版本后 gc
for (let i = 0; i < 200; i++) {
const txnId = mvcc.beginTransaction();
mvcc.writeVersion('users', '1', { ver: i }, txnId);
mvcc.commitTransaction(txnId);
}
mvcc.gc(100);
// gc 后版本保留最新 100 个
const store = (mvcc as unknown as { versionStore: Map<string, unknown[]> }).versionStore;
expect(store.get('users.1')!.length).toBe(100);
// gc 后引擎仍可继续写入
const txnId = mvcc.beginTransaction();
mvcc.writeVersion('users', '1', { ver: 999 }, txnId);
mvcc.commitTransaction(txnId);
expect(store.get('users.1')!.length).toBe(101);
});
});
tests/engine/aria-wal-segment.test.ts
+275
View File
@@ -0,0 +1,275 @@
/**
* AriaEngine — SegmentedWALStore 分片 WAL 存储测试
*
* 覆盖：
* 1. append → readAll 往返（分片切换、字节顺序）
* 2. 空洞检测：序号不连续 → 空洞后的分片丢弃
* 3. 旧格式 __wal_N + __wal_count 兼容读取
* 4. truncate / exists
* 5. 后端无 append 接口时回退 read+write（正确性一致）
* 6. AriaEngine 集成：高频写入跨重启数据完整（分片 + WAL 重放）
*/
import { SegmentedWALStore } from '../../src/engine/aria/wal/segmented_store';
import { MemoryBackend } from '../../src/engine/aria/store/backend';
import { AriaEngine } from '../../src/engine/aria/index';
import { createSchema } from '../../src/table/schema';
import 'fake-indexeddb/auto';
let idbCounter = 0;
function uniqueDB(): string {
return `seg-${Date.now()}-${++idbCounter}-${Math.random().toString(36).slice(2, 8)}`;
}
const enc = (s: string) => new Uint8Array(new TextEncoder().encode(s));
describe('AriaEngine — SegmentedWALStore 单元', () => {
it('append → readAll 往返，多批次字节顺序一致', async () => {
const backend = new MemoryBackend();
await backend.open('seg-unit-1');
const store = new SegmentedWALStore(backend, 1024);
expect(await store.exists()).toBe(false);
await store.append(enc('AAA'));
await store.append(enc('BBB'));
await store.append(enc('CCC'));
expect(await store.exists()).toBe(true);
const all = await store.readAll();
expect(new TextDecoder().decode(all)).toBe('AAABBBCCC');
// 后端是分片文件（单个 key）
const keys = await backend.listKeys();
expect(keys).toEqual(['__wal_000000.bin']);
await backend.close();
});
it('超过分片阈值自动切换分片', async () => {
const backend = new MemoryBackend();
await backend.open('seg-unit-2');
const store = new SegmentedWALStore(backend, 16); // 每分片 16 字节
await store.append(enc('AAAAAAAAAA')); // 10B → 分片 0
await store.append(enc('BBBBBBBBBB')); // 10B → 分片 1（0 已 10B+10B > 16）
await store.append(enc('CC')); // 2B → 分片 1（10+2 <= 16）
const all = await store.readAll();
expect(new TextDecoder().decode(all)).toBe('AAAAAAAAAABBBBBBBBBBCC');
const keys = await backend.listKeys().then((ks) => ks.sort());
expect(keys).toEqual(['__wal_000000.bin', '__wal_000001.bin']);
await backend.close();
});
it('空洞检测：分片序号不连续 → 空洞后的分片整体丢弃', async () => {
const backend = new MemoryBackend();
await backend.open('seg-unit-3');
const store = new SegmentedWALStore(backend, 16);
await store.append(enc('SEG0-SEG0-S')); // 10B → 分片 0
await store.append(enc('SEG1-SEG1-S')); // 10B → 分片 1（0 已 10B+10B > 16）
await store.append(enc('S2')); // 2B → 分片 1（10+2 <= 16）
// 模拟 truncate 部分完成：删除分片 1（空洞）
await backend.delete('__wal_000001.bin');
const all = await store.readAll();
// 空洞在分片 1 → 1 及之后丢弃，只保留分片 0
expect(new TextDecoder().decode(all)).toBe('SEG0-SEG0-S');
await backend.close();
});
it('空洞在开头（分片 0 缺失）→ 全部丢弃（保守截断）', async () => {
const backend = new MemoryBackend();
await backend.open('seg-unit-4');
const store = new SegmentedWALStore(backend, 16);
await store.append(enc('X0'));
await store.append(enc('X1'));
await backend.delete('__wal_000000.bin');
const all = await store.readAll();
expect(all.byteLength).toBe(0);
await backend.close();
});
it('旧格式 __wal_N + __wal_count 兼容读取（迁移前数据）', async () => {
const backend = new MemoryBackend();
await backend.open('seg-unit-5');
// 构造旧格式数据
await backend.write('__wal_0', enc('LEGACY0').buffer);
await backend.write('__wal_1', enc('LEGACY1').buffer);
await backend.write('__wal_count', enc('2').buffer);
const store = new SegmentedWALStore(backend, 1024);
expect(await store.exists()).toBe(true);
const all = await store.readAll();
expect(new TextDecoder().decode(all)).toBe('LEGACY0LEGACY1');
// 迁移中追加新格式：新记录写入分片文件（不与旧键冲突）
await store.append(enc('NEW'));
const keys = await backend.listKeys().then((ks) => ks.sort());
expect(keys).toContain('__wal_000000.bin');
expect(keys).toContain('__wal_0');
// truncate 清空新旧全部
await store.truncate();
expect(await store.exists()).toBe(false);
expect((await backend.listKeys()).length).toBe(0);
await backend.close();
});
it('truncate 清空分片并重置状态', async () => {
const backend = new MemoryBackend();
await backend.open('seg-unit-6');
const store = new SegmentedWALStore(backend, 16);
await store.append(enc('AAAAAA'));
await store.append(enc('BBBBBB'));
await store.truncate();
expect(await store.exists()).toBe(false);
expect((await backend.listKeys()).length).toBe(0);
// 截断后继续追加从分片 0 重新开始
await store.append(enc('CCC'));
const all = await store.readAll();
expect(new TextDecoder().decode(all)).toBe('CCC');
await backend.close();
});
it('后端无 append 接口 → 回退 read+write 语义一致', async () => {
// MemoryBackend 没有 append 接口 → 走回退路径
const backend = new MemoryBackend();
await backend.open('seg-unit-7');
const store = new SegmentedWALStore(backend, 1024);
await store.append(enc('PART1'));
await store.append(enc('PART2'));
const all = await store.readAll();
expect(new TextDecoder().decode(all)).toBe('PART1PART2');
await backend.close();
});
});
// ===================================================================
// AriaEngine 集成 — 分片 WAL 跨重启
// ===================================================================
describe('AriaEngine — 分片 WAL 集成', () => {
it('高频写入（多条 WAL 记录）→ close → reopen 数据完整', async () => {
const dbName = uniqueDB();
const engine = new AriaEngine({
storageBackend: 'indexeddb',
checkpointInterval: 100000,
walSyncMode: 'full',
});
await engine.open(dbName, 1);
await engine.createTable(createSchema('logs', {
id: { type: 'string', primaryKey: true },
msg: { type: 'string' },
}));
const total = 300;
for (let i = 0; i < total; i++) {
await engine.insert('logs', [{ id: `log-${i}`, msg: `message ${i}` }]);
}
// 不 flush 不 checkpoint，全部留在 WAL → 模拟崩溃后重放
await (engine as any).backend.close();
(engine as any).opened = false;
const engine2 = new AriaEngine({
storageBackend: 'indexeddb',
checkpointInterval: 100000,
walSyncMode: 'full',
});
await engine2.open(dbName, 1);
const rows = await engine2.find('logs', { table: 'logs' });
expect(rows).toHaveLength(total);
expect(rows.some((r) => r.id === 'log-299')).toBe(true);
await engine2.close();
});
it('分片结构落盘验证（backend 中是分片文件而非旧单记录键）', async () => {
const dbName = uniqueDB();
const engine = new AriaEngine({
storageBackend: 'indexeddb',
checkpointInterval: 100000,
walSyncMode: 'full',
});
await engine.open(dbName, 1);
await engine.createTable(createSchema('t', {
id: { type: 'string', primaryKey: true },
}));
await engine.insert('t', [{ id: '1' }, { id: '2' }, { id: '3' }]);
const backend = (engine as any).backend;
const keys = await backend.listKeys();
const walKeys = keys.filter((k: string) => k.startsWith('__wal_'));
// 新格式：__wal_000000.bin；无旧格式单记录键与 count 键
expect(walKeys).toContain('__wal_000000.bin');
expect(walKeys.some((k: string) => /^__wal_\d+$/.test(k) && !k.endsWith('.bin'))).toBe(false);
expect(walKeys).not.toContain('__wal_count');
await engine.close();
});
it('WAL 分片部分残留（空洞）→ 重开恢复已落盘数据，不丢已 checkpoint 数据', async () => {
const dbName = uniqueDB();
const engine = new AriaEngine({
storageBackend: 'indexeddb',
checkpointInterval: 100000,
walSyncMode: 'full',
});
await engine.open(dbName, 1);
await engine.createTable(createSchema('t', {
id: { type: 'string', primaryKey: true },
}));
// 第一批：flush 落盘 + checkpoint 清 WAL
await engine.insert('t', [{ id: 'a' }, { id: 'b' }]);
await (engine as any).lsm.flush();
await (engine as any).wal.checkpoint();
// 第二批：只进 WAL（不 flush）
await engine.insert('t', [{ id: 'c' }, { id: 'd' }]);
// 模拟 WAL 分片文件被外部删掉（空洞场景）
const backend = (engine as any).backend;
await backend.delete('__wal_000000.bin');
await engine.close();
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', checkpointInterval: 100000 });
await engine2.open(dbName, 1);
// 已落盘的 a/b 必须保留；c/d 在 WAL 中且文件被删 → 恢复不到（可接受的保守丢弃）
const rows = await engine2.find('t', { table: 't' });
expect(rows.some((r) => r.id === 'a')).toBe(true);
expect(rows.some((r) => r.id === 'b')).toBe(true);
await engine2.close();
});
it('旧格式 WAL 升级迁移：旧键重放后 checkpoint 清空', async () => {
const dbName = uniqueDB();
// 手工构造旧格式 WAL 库：直接写旧键（模拟 v0.4.4 库崩溃现场）
const engine = new AriaEngine({
storageBackend: 'indexeddb',
checkpointInterval: 100000,
walSyncMode: 'full',
});
await engine.open(dbName, 1);
await engine.createTable(createSchema('t', {
id: { type: 'string', primaryKey: true },
}));
// 写入一批数据但只保留在 WAL（新格式）→ close 前手工转成旧格式
await engine.insert('t', [{ id: 'x' }]);
const backend = (engine as any).backend;
const raw = await backend.read('__wal_000000.bin');
expect(raw).not.toBeNull();
// 清掉新格式，写成旧格式（单记录键）
await backend.delete('__wal_000000.bin');
await backend.write('__wal_0', raw);
await backend.write('__wal_count', new TextEncoder().encode('1').buffer);
// 同时清掉内存里的 WAL 状态，模拟"重启"
await (engine as any).backend.close();
(engine as any).opened = false;
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', checkpointInterval: 100000 });
await engine2.open(dbName, 1);
const rows = await engine2.find('t', { table: 't' });
expect(rows).toHaveLength(1);
expect(rows[0].id).toBe('x');
// 恢复后 checkpoint 清空旧键
const keys = await (engine2 as any).backend.listKeys();
expect(keys.some((k: string) => k.startsWith('__wal_'))).toBe(false);
await engine2.close();
});
});
tests/engine/crc32.test.ts
+89
View File
@@ -0,0 +1,89 @@
/**
* AriaEngine CRC32 单元测试
*/
import { crc32, crc32Continue } from '../../src/engine/aria/crc32';
const enc = new TextEncoder();
function bytes(s: string): Uint8Array {
return enc.encode(s);
}
describe('CRC32', () => {
test('标准测试向量：空串 = 0', () => {
expect(crc32(new Uint8Array(0))).toBe(0x00000000);
});
test('标准测试向量：123456789 = 0xCBF43926（IEEE 802.3 参考值）', () => {
expect(crc32(bytes('123456789'))).toBe(0xcbf43926);
});
test('标准测试向量：The quick brown fox... = 0x414FA339', () => {
expect(crc32(bytes('The quick brown fox jumps over the lazy dog'))).toBe(0x414fa339);
});
test('确定性：相同输入多次计算一致', () => {
const data = bytes('metona-sqlark aria engine');
const a = crc32(data);
const b = crc32(data);
expect(a).toBe(b);
});
test('区分性：不同输入不同校验和', () => {
expect(crc32(bytes('hello'))).not.toBe(crc32(bytes('hellp')));
expect(crc32(bytes('abc'))).not.toBe(crc32(bytes('abd')));
});
test('单字节翻转可被检测', () => {
const original = bytes('metona-sqlark aria engine');
const flipped = new Uint8Array(original);
flipped[10] ^= 0xff;
expect(crc32(original)).not.toBe(crc32(flipped));
});
test('分段计算与一次性计算一致（crc32Continue）', () => {
const data = bytes('The quick brown fox jumps over the lazy dog');
const whole = crc32(data);
// 按 1 字节分段
let seg = 0;
for (let i = 0; i < data.length; i++) {
seg = crc32Continue(seg, data.subarray(i, i + 1));
}
expect(seg).toBe(whole);
// 按 5 字节分段
let seg2 = 0;
for (let i = 0; i < data.length; i += 5) {
seg2 = crc32Continue(seg2, data.subarray(i, i + 5));
}
expect(seg2).toBe(whole);
});
test('seed 参数与 crc32Continue 等价', () => {
const data = bytes('metona-sqlark aria engine');
const head = data.subarray(0, 7);
const tail = data.subarray(7);
const whole = crc32(data);
const split = crc32(tail, crc32(head));
expect(split).toBe(whole);
});
test('中文 UTF-8 字节正确参与计算', () => {
const data = bytes('玥玥的测试数据：大段中文内容');
expect(crc32(data)).toBe(crc32(new Uint8Array(data)));
});
test('大缓冲区（1MB 随机字节）稳定计算', () => {
const data = new Uint8Array(1024 * 1024);
let x = 12345;
for (let i = 0; i < data.length; i++) {
x = (x * 1103515245 + 12345) & 0x7fffffff;
data[i] = x & 0xff;
}
const a = crc32(data);
const b = crc32(data);
expect(a).toBe(b);
expect(a).not.toBe(0);
});
});
tests/helpers/opfs-mock.ts
+88
View File
@@ -0,0 +1,88 @@
/**
* 测试共享 — 真实语义 OPFS mock
*
* 模拟 OPFS 的完整语义：
* - createWritable({ keepExistingData }) + position 追加写
* - 文件内容以 ArrayBuffer 存储（跨 backend 实例共享于同一 Map）
* - entries() 迭代、removeEntry 删除
*
* 用法：
* const files = installOPFSMock(new Map());
* const backend = new OPFSBackend();
* await backend.open('any');
*/
export interface MockOpfsFile {
content: ArrayBuffer;
}
export function installOPFSMock(files: Map<string, MockOpfsFile>): {
files: Map<string, MockOpfsFile>;
dir: {
getFileHandle: (name: string, opts?: { create?: boolean }) => Promise<{
getFile: () => Promise<{ size: number; arrayBuffer: () => Promise<ArrayBuffer> }>;
createWritable: (wOpts?: { keepExistingData?: boolean }) => Promise<{
write: (arg: ArrayBuffer | { type: string; position: number; data: ArrayBuffer }) => Promise<void>;
close: () => Promise<void>;
}>;
}>;
entries: () => AsyncGenerator<[string]>;
removeEntry: (name: string) => Promise<void>;
};
} {
const getFileHandle = async (name: string, opts?: { create?: boolean }) => {
if (!files.has(name)) {
if (!opts?.create) throw new Error(`NotFoundError: ${name}`);
files.set(name, { content: new ArrayBuffer(0) });
}
const entry = files.get(name)!;
return {
getFile: async () => ({ size: entry.content.byteLength, arrayBuffer: async () => entry.content }),
createWritable: async (wOpts?: { keepExistingData?: boolean }) => {
const w: {
write: (arg: ArrayBuffer | { type: string; position: number; data: ArrayBuffer }) => Promise<void>;
close: () => Promise<void>;
} = {
write: async (arg) => {
const keepExisting = wOpts?.keepExistingData ?? false;
const chunk = (arg as { type?: string }).type === 'write'
? { data: (arg as { data: ArrayBuffer }).data, position: (arg as { position: number }).position }
: { data: arg as ArrayBuffer, position: keepExisting ? entry.content.byteLength : 0 };
const merged = new Uint8Array(chunk.position + chunk.data.byteLength);
if (keepExisting || chunk.position > 0) {
merged.set(new Uint8Array(entry.content), 0);
}
merged.set(new Uint8Array(chunk.data), chunk.position);
entry.content = merged.buffer;
},
close: async () => { /* no-op */ },
};
return w;
},
};
};
const dir = {
getFileHandle,
entries: async function* () {
for (const [name] of files) yield [name];
},
removeEntry: async (name: string) => {
files.delete(name);
},
};
Object.defineProperty(globalThis, 'navigator', {
value: {
storage: {
getDirectory: async () => ({
getDirectoryHandle: async (_name: string, _opts?: unknown) => dir,
}),
},
},
configurable: true,
writable: true,
});
return { files, dir };
}
tests/hooks-crud.test.ts
+140
View File
@@ -0,0 +1,140 @@
/**
* v0.5.1 — CRUD 生命周期钩子真实接线测试
*
* 此前 beforeInsert/afterInsert/beforeUpdate/afterUpdate/beforeDelete/afterDelete
* 六个钩子虽在 HookName 类型与文档中宣称（"14 种钩子"），但从未被任何路径触发。
* 本次验证 Table API 与 SQL query 两条路径均真实触发全部 14 个钩子。
*/
import { MetonaSqlark } from '../src/core';
import 'fake-indexeddb/auto';
async function createDb(name: string): Promise<MetonaSqlark> {
const db = new MetonaSqlark({ name: `hooks-${name}-${Date.now()}-${Math.random().toString(36).slice(2, 6)}`, mode: 'memory' });
await db.init();
await db.defineTable('users', {
id: { type: 'string', primaryKey: true },
name: { type: 'string' },
age: { type: 'number' },
});
return db;
}
describe('v0.5.1 — CRUD hooks 真实接线', () => {
it('Table.insert / insertMany 触发 beforeInsert + afterInsert', async () => {
const db = await createDb('table-insert');
const events: string[] = [];
db.on('beforeInsert', (rows) => { events.push(`before:${(rows as unknown[]).length}`); });
db.on('afterInsert', (rows, pks) => { events.push(`after:${(rows as unknown[]).length}:${(pks as string[]).length}`); });
await db.table('users').insert({ id: '1', name: 'A' });
await db.table('users').insertMany([{ id: '2', name: 'B' }, { id: '3', name: 'C' }]);
expect(events).toEqual([
'before:1', 'after:1:1',
'before:2', 'after:2:2',
]);
await db.close();
});
it('Table.update().execute() 触发 beforeUpdate + afterUpdate（含影响行数）', async () => {
const db = await createDb('table-update');
await db.table('users').insertMany([{ id: '1', name: 'A', age: 1 }, { id: '2', name: 'B', age: 2 }]);
const events: string[] = [];
db.on('beforeUpdate', (query, updates) => { events.push(`before:${(updates as Record<string, unknown>).name}`); });
db.on('afterUpdate', (_query, _updates, count) => { events.push(`after:${count}`); });
await db.table('users').update({ name: 'X' }).where({ age: { $gt: 1 } }).execute();
expect(events).toEqual(['before:X', 'after:1']);
await db.close();
});
it('Table.delete().execute() 触发 beforeDelete + afterDelete（含影响行数）', async () => {
const db = await createDb('table-delete');
await db.table('users').insertMany([{ id: '1', name: 'A' }, { id: '2', name: 'B' }]);
const events: string[] = [];
db.on('beforeDelete', (query) => { events.push(`before:${(query as { where: Record<string, unknown> }).where.id}`); });
db.on('afterDelete', (_query, count) => { events.push(`after:${count}`); });
await db.table('users').delete().where({ id: '1' }).execute();
expect(events).toEqual(['before:1', 'after:1']);
await db.close();
});
it('SQL query 路径触发全部 CRUD hooks（INSERT/UPDATE/DELETE）', async () => {
const db = await createDb('sql-crud');
await db.query("INSERT INTO users VALUES ('1', 'Alice', 30)");
const events: string[] = [];
db.on('beforeInsert', () => events.push('beforeInsert'));
db.on('afterInsert', () => events.push('afterInsert'));
db.on('beforeUpdate', () => events.push('beforeUpdate'));
db.on('afterUpdate', () => events.push('afterUpdate'));
db.on('beforeDelete', () => events.push('beforeDelete'));
db.on('afterDelete', () => events.push('afterDelete'));
await db.query("INSERT INTO users VALUES ('2', 'Bob', 25)");
await db.query("UPDATE users SET age = 26 WHERE id = '2'");
await db.query("DELETE FROM users WHERE id = '1'");
expect(events).toEqual([
'beforeInsert', 'afterInsert',
'beforeUpdate', 'afterUpdate',
'beforeDelete', 'afterDelete',
]);
await db.close();
});
it('SQL INSERT 多行语句按列名映射触发 afterInsert', async () => {
const db = await createDb('sql-insert-cols');
const seen: unknown[] = [];
db.on('afterInsert', (_rows, pks) => seen.push(pks));
await db.query("INSERT INTO users (id, name) VALUES ('9', 'Zed')");
expect(seen).toEqual([['9']]);
await db.close();
});
it('全部 14 个钩子均可注册并真实触发', async () => {
const db = await createDb('all-hooks');
const fired = new Set<string>();
const hooks = [
'beforeCreateTable', 'afterCreateTable',
'beforeDropTable', 'afterDropTable',
'beforeInsert', 'afterInsert',
'beforeUpdate', 'afterUpdate',
'beforeDelete', 'afterDelete',
'beforeQuery', 'afterQuery',
'beforeTransaction', 'afterTransaction',
] as const;
for (const h of hooks) db.on(h, () => fired.add(h));
await db.defineTable('t2', { id: { type: 'string', primaryKey: true } }); // createTable hooks
await db.table('t2').insert({ id: '1' }); // insert hooks
await db.table('t2').update({}).where({ id: '1' }).execute(); // update hooks
await db.table('t2').delete().where({ id: '1' }).execute(); // delete hooks
await db.query('SELECT * FROM t2'); // query hooks
await db.transaction(async () => { /* noop */ }); // transaction hooks
await db.dropTable('t2'); // dropTable hooks
for (const h of hooks) {
expect(fired.has(h)).toBe(true);
if (!fired.has(h)) throw new Error(`hook ${h} 应被触发`);
}
await db.close();
});
it('事务路径不触发 CRUD hooks（Table 无 hooks 回调）', async () => {
const db = await createDb('tx-hooks');
let fired = false;
db.on('beforeInsert', () => { fired = true; });
await db.transaction(async (trx) => {
await trx.table('users').insert({ id: '1', name: 'Tx' });
});
expect(fired).toBe(false);
await db.close();
});
});
tests/maintenance-sql.test.ts
+172
View File
@@ -0,0 +1,172 @@
/**
* v0.5.1 — 维护语句 SQL 入口测试（EXPLAIN / ANALYZE / REINDEX / VACUUM / SAVEPOINT）
*
* 此前这些功能仅有引擎方法/executor 分支而无 SQL 入口（parser 无法解析），
* 属"宣称存在但用户无法触发"的半成品。本次补齐 SQL 语法并验证真实执行。
*/
import { MetonaSqlark } from '../src/core';
import { parse } from '../src/sql/parser';
import { createSchema } from '../src/table/schema';
import 'fake-indexeddb/auto';
let dbCounter = 0;
function uniqueDB(): string {
return `maint-${Date.now()}-${++dbCounter}-${Math.random().toString(36).slice(2, 8)}`;
}
describe('v0.5.1 — 维护语句解析', () => {
it('EXPLAIN SELECT 解析为 ExplainStatement', () => {
const stmt = parse('EXPLAIN SELECT * FROM users WHERE id = \'1\'');
expect(stmt.type).toBe('EXPLAIN');
expect((stmt as { query: { type: string } }).query.type).toBe('SELECT');
});
it('EXPLAIN 嵌套 EXPLAIN 拒绝', () => {
expect(() => parse('EXPLAIN EXPLAIN SELECT 1')).toThrow('Nested EXPLAIN');
});
it('ANALYZE / ANALYZE TABLE 解析', () => {
expect(parse('ANALYZE users')).toMatchObject({ type: 'ANALYZE', table: 'users' });
expect(parse('ANALYZE TABLE users')).toMatchObject({ type: 'ANALYZE', table: 'users' });
});
it('REINDEX / VACUUM 解析', () => {
expect(parse('REINDEX users')).toMatchObject({ type: 'REINDEX', table: 'users' });
expect(parse('REINDEX TABLE users')).toMatchObject({ type: 'REINDEX', table: 'users' });
expect(parse('VACUUM')).toMatchObject({ type: 'VACUUM' });
});
it('SAVEPOINT / ROLLBACK TO / RELEASE 解析', () => {
expect(parse('SAVEPOINT sp1')).toMatchObject({ type: 'SAVEPOINT', name: 'sp1', action: 'SAVE' });
expect(parse('ROLLBACK TO sp1')).toMatchObject({ type: 'SAVEPOINT', name: 'sp1', action: 'ROLLBACK' });
expect(parse('ROLLBACK TO SAVEPOINT sp1')).toMatchObject({ type: 'SAVEPOINT', name: 'sp1', action: 'ROLLBACK' });
expect(parse('RELEASE SAVEPOINT sp1')).toMatchObject({ type: 'SAVEPOINT', name: 'sp1', action: 'RELEASE' });
expect(parse('RELEASE sp1')).toMatchObject({ type: 'SAVEPOINT', name: 'sp1', action: 'RELEASE' });
// ROLLBACK [TRANSACTION] 保持原语义
expect(parse('ROLLBACK')).toMatchObject({ type: 'ROLLBACK' });
expect(parse('ROLLBACK TRANSACTION')).toMatchObject({ type: 'ROLLBACK' });
});
});
describe('v0.5.1 — 维护语句执行（Aria 引擎）', () => {
async function createAriaDb(): Promise<MetonaSqlark> {
const db = new MetonaSqlark({
name: uniqueDB(),
mode: 'aria',
diskEngine: 'indexeddb',
aria: { checkpointInterval: 100000, walSyncMode: 'full' },
});
await db.init();
await db.defineTable('users', {
id: { type: 'string', primaryKey: true },
name: { type: 'string', index: true },
age: { type: 'number' },
});
await db.table('users').insertMany([
{ id: '1', name: 'Alice', age: 30 },
{ id: '2', name: 'Bob', age: 25 },
{ id: '3', name: 'Carol', age: 35 },
]);
return db;
}
it('EXPLAIN SELECT 返回查询计划', async () => {
const db = await createAriaDb();
const plan = await db.query("EXPLAIN SELECT * FROM users WHERE id = '1'") as Record<string, unknown>;
expect(plan.type).toBe('SELECT');
expect(plan.table).toBe('users');
expect(plan.estimatedRows).toBe(1);
await db.close();
});
it('EXPLAIN UPDATE / DELETE 返回计划', async () => {
const db = await createAriaDb();
const planU = await db.query("EXPLAIN UPDATE users SET age = 31 WHERE id = '1'") as Record<string, unknown>;
expect(planU.type).toBe('UPDATE');
const planD = await db.query('EXPLAIN DELETE FROM users WHERE id = \'1\'') as Record<string, unknown>;
expect(planD.type).toBe('DELETE');
await db.close();
});
it('ANALYZE TABLE 收集统计信息', async () => {
const db = await createAriaDb();
const stats = await db.query('ANALYZE TABLE users') as Record<string, unknown>;
expect(stats.rowCount).toBe(3);
expect(stats.table).toBe('users');
await db.close();
});
it('REINDEX TABLE 重建索引后索引查询正常', async () => {
const db = await createAriaDb();
const count = await db.query('REINDEX TABLE users') as number;
expect(count).toBeGreaterThanOrEqual(0);
const rows = await db.query("SELECT * FROM users WHERE name = 'Bob'") as Record<string, unknown>[];
expect(rows).toHaveLength(1);
await db.close();
});
it('VACUUM 压缩成功', async () => {
const db = await createAriaDb();
const result = await db.query('VACUUM') as { compactedLevels: number };
expect(typeof result.compactedLevels).toBe('number');
await db.close();
});
it('SAVEPOINT / ROLLBACK TO / RELEASE 真实执行', async () => {
const db = await createAriaDb();
await db.query('BEGIN');
await db.query("UPDATE users SET age = 99 WHERE id = '1'");
await db.query('SAVEPOINT sp_a');
await db.query("UPDATE users SET age = 88 WHERE id = '1'");
// 回滚到 savepoint：恢复 age=99
await db.query('ROLLBACK TO SAVEPOINT sp_a');
await db.query('RELEASE SAVEPOINT sp_a');
await db.query('COMMIT');
const rows = await db.query("SELECT age FROM users WHERE id = '1'") as Record<string, unknown>[];
expect(rows[0].age).toBe(99);
await db.close();
});
it('savepoint 已存在 → 抛 SAVEPOINT_EXISTS', async () => {
const db = await createAriaDb();
await db.query('BEGIN');
await db.query('SAVEPOINT sp1');
await expect(db.query('SAVEPOINT sp1')).rejects.toMatchObject({ code: 'SAVEPOINT_EXISTS' });
await db.query('ROLLBACK');
await db.close();
});
it('回滚到不存在的 savepoint → 抛 SAVEPOINT_NOT_FOUND', async () => {
const db = await createAriaDb();
await db.query('BEGIN');
await expect(db.query('ROLLBACK TO nope')).rejects.toMatchObject({ code: 'SAVEPOINT_NOT_FOUND' });
await db.query('ROLLBACK');
await db.close();
});
});
describe('v0.5.1 — 维护语句执行（Memory 引擎不支持）', () => {
it('ANALYZE / REINDEX / VACUUM / SAVEPOINT → NOT_SUPPORTED', async () => {
const db = new MetonaSqlark({ name: uniqueDB(), mode: 'memory' });
await db.init();
await db.defineTable('t', { id: { type: 'string', primaryKey: true } });
await expect(db.query('ANALYZE TABLE t')).rejects.toMatchObject({ code: 'NOT_SUPPORTED' });
await expect(db.query('REINDEX TABLE t')).rejects.toMatchObject({ code: 'NOT_SUPPORTED' });
await expect(db.query('VACUUM')).rejects.toMatchObject({ code: 'NOT_SUPPORTED' });
await expect(db.query('SAVEPOINT sp')).rejects.toMatchObject({ code: 'NOT_SUPPORTED' });
await db.close();
});
it('EXPLAIN 在所有引擎可用（返回基本信息）', async () => {
const db = new MetonaSqlark({ name: uniqueDB(), mode: 'memory' });
await db.init();
await db.defineTable('t', { id: { type: 'string', primaryKey: true } });
await db.table('t').insert({ id: '1' });
const plan = await db.query('EXPLAIN SELECT * FROM t') as Record<string, unknown>;
expect(plan.type).toBe('SELECT');
expect(plan.table).toBe('t');
await db.close();
});
});
tests/sql-ext2.test.ts
+5 -7
View File
@@ -287,14 +287,13 @@ describe('[v0.3.1] WAL 批量组提交', () => {
await db.defineTable('users', { id: { type: 'string', primaryKey: true }, name: { type: 'string' } });
const engine = db.getEngine() as any;
// v0.4.2: WAL 记录与 count 改用 writeMany 单事务原子写入（记录仍合并为 1 次落盘）
const origWriteMany = engine.backend.writeMany.bind(engine.backend);
// v0.4.5: WAL 改用分片存储（SegmentedWALStore，无 append 接口的后端回退单 key write）；
// 组提交语义保持：批量记录仍合并为 1 次底层落盘
const origWrite = engine.backend.write.bind(engine.backend);
let walWrites = 0;
engine.backend.writeMany = async (entries: Record<string, ArrayBuffer>) => {
const keys = Object.keys(entries);
if (keys.some((k) => k.startsWith('__wal_') && !k.startsWith('__wal_count'))) walWrites++;
return origWriteMany(entries);
engine.backend.write = async (key: string, data: ArrayBuffer) => {
if (key.startsWith('__wal_')) walWrites++;
return origWrite(key, data);
};
await db.table('users').insertMany([
@@ -305,7 +304,6 @@ describe('[v0.3.1] WAL 批量组提交', () => {
]);
expect(walWrites).toBe(1); // 4 行合并为 1 次 WAL 落盘
void origWrite;
const rows = await db.query('SELECT * FROM users') as Record<string, unknown>[];
expect(rows).toHaveLength(4);
await db.close();
tests/utils.test.ts
-99
View File
@@ -1,99 +0,0 @@
/**
* utils.ts 单元测试模板
*/
import {
generateId,
escapeHTML,
debounce,
throttle,
deepMerge,
isBrowser,
} from '../src/utils';
describe('generateId', () => {
test('返回字符串', () => {
expect(typeof generateId()).toBe('string');
});
test('多次调用产生不同值', () => {
const ids = new Set<string>();
for (let i = 0; i < 100; i++) ids.add(generateId());
expect(ids.size).toBe(100);
});
});
describe('escapeHTML', () => {
test('转义 < > &', () => {
const out = escapeHTML('<a>&b</a>');
expect(out).toContain('&lt;');
expect(out).toContain('&gt;');
expect(out).toContain('&amp;');
});
test('null/undefined 返回空字符串', () => {
expect(escapeHTML(null)).toBe('');
expect(escapeHTML(undefined)).toBe('');
});
});
describe('debounce', () => {
test('延迟执行', (done) => {
let called = 0;
const fn = debounce(() => { called++; }, 20);
fn();
expect(called).toBe(0);
setTimeout(() => {
expect(called).toBe(1);
done();
}, 50);
});
test('多次调用只执行最后一次', (done) => {
let result = 0;
const fn = debounce((v: number) => { result = v; }, 20);
fn(1);
fn(2);
fn(3);
setTimeout(() => {
expect(result).toBe(3);
done();
}, 50);
});
});
describe('throttle', () => {
test('首次立即执行', () => {
let called = 0;
const fn = throttle(() => { called++; }, 50);
fn();
expect(called).toBe(1);
});
test('限流期内不重复执行', () => {
let called = 0;
const fn = throttle(() => { called++; }, 50);
fn();
fn();
fn();
expect(called).toBe(1);
});
});
describe('deepMerge', () => {
test('浅层合并', () => {
const out = deepMerge({ a: 1, b: 2 }, { b: 3, c: 4 });
expect(out).toEqual({ a: 1, b: 3, c: 4 });
});
test('深层对象合并', () => {
const out = deepMerge({ obj: { x: 1 } }, { obj: { y: 2 } });
expect(out).toEqual({ obj: { x: 1, y: 2 } });
});
});
describe('isBrowser', () => {
test('jsdom 环境下返回 true', () => {
expect(isBrowser()).toBe(true);
});
});
tests/v025-fixes.test.ts
+2 -3
View File
@@ -23,8 +23,8 @@ import { createSchema } from '../src/table/schema';
// ---------------------------------------------------------------------------
describe('[v0.2.5] P0-1: 版本号统一', () => {
test('VERSION 常量为当前版本（0.4.4）', () => {
expect(VERSION).toBe('0.4.4');
test('VERSION 常量为当前版本（0.5.1）', () => {
expect(VERSION).toBe('0.5.1');
});
});
@@ -339,7 +339,6 @@ describe('[v0.2.5] P2-12: WAL 大小阈值', () => {
const fakeWal = { flush: async () => {}, checkpoint: async () => {}, getBufferedCount: () => 0 };
const cm = new CheckpointManager(fakeLsm, fakeWal, null, 1000, 1024);
expect(cm).toBeDefined();
expect(cm.getOpCount()).toBe(0);
});
});
tests/v033-fixes.test.ts
+1 -1
View File
@@ -400,7 +400,7 @@ describe('[v0.3.3] P1-9: Savepoint + MVCC 一致性', () => {
describe('[v0.3.3] 端到端', () => {
test('全部修复点可共存于 MetonaSqlark API', async () => {
expect(VERSION).toBe('0.4.4');
expect(VERSION).toBe('0.5.1');
const db = new MetonaSqlark({ name: `e2e-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`, mode: 'memory' });
await db.init();
await db.defineTable('users', {
tests/v043-hardening.test.ts
+1 -1
View File
@@ -111,7 +111,7 @@ describe('P0 — 事务进行中 checkpoint 不得截断 WAL', () => {
await engine.insert('t', [{ id: 'tx1' }]);
await engine.rollbackTransaction();
// 事务结束后 checkpoint 可正常截断
await (engine as any).checkpointManager.forceCheckpoint();
await (engine as any).checkpointManager.checkpoint();
await (engine as any).backend.close();
(engine as any).opened = false;