release: v0.5.1 — 存储后端生产级硬化(CRC-32/全库加密/WAL分片/页面化存储/多标签页锁/e2e)+ 深度审查修复(假实现接线/死代码清理)
This commit is contained in:
@@ -40,3 +40,25 @@ jobs:
|
|||||||
|
|
||||||
- name: Build
|
- name: Build
|
||||||
run: npm run build
|
run: npm run build
|
||||||
|
|
||||||
|
e2e:
|
||||||
|
runs-on: debian-latest
|
||||||
|
# 与 MetonaEditor 一致:Playwright 官方镜像(自带 Chromium 与全部系统依赖),
|
||||||
|
# 避免 runner 宿主(Debian 11)不被 Playwright 1.62 支持的问题
|
||||||
|
container: mcr.microsoft.com/playwright:v1.62.1-noble
|
||||||
|
timeout-minutes: 15
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- name: Use Node.js
|
||||||
|
uses: actions/setup-node@v4
|
||||||
|
with:
|
||||||
|
node-version: 22.x
|
||||||
|
cache: 'npm'
|
||||||
|
- name: Install dependencies
|
||||||
|
run: npm ci
|
||||||
|
- name: Build
|
||||||
|
run: npm run build
|
||||||
|
- name: Verify Playwright browsers
|
||||||
|
run: npx playwright install chromium
|
||||||
|
- name: Run E2E smoke tests
|
||||||
|
run: npm run test:e2e
|
||||||
|
|||||||
@@ -7,3 +7,7 @@ coverage/
|
|||||||
|
|
||||||
# CatPaw / .zcode 工作区文件(本地 AI 会话计划,勿提交)
|
# CatPaw / .zcode 工作区文件(本地 AI 会话计划,勿提交)
|
||||||
.zcode/
|
.zcode/
|
||||||
|
|
||||||
|
# Playwright e2e 产物
|
||||||
|
test-results/
|
||||||
|
playwright-report/
|
||||||
|
|||||||
+109
@@ -2,6 +2,115 @@
|
|||||||
|
|
||||||
All notable changes to MetonaSqlark will be documented in this file.
|
All notable changes to MetonaSqlark will be documented in this file.
|
||||||
|
|
||||||
|
## [0.5.1] - 2026-08-10
|
||||||
|
|
||||||
|
### 深度审查:消除假实现/半成品/死代码
|
||||||
|
|
||||||
|
> 对全部源码做系统性审查,修复"宣称存在但从未接线"的假实现、补齐无入口的半成品、
|
||||||
|
> 删除无引用的死代码。
|
||||||
|
|
||||||
|
### Added(假实现接线)
|
||||||
|
|
||||||
|
- **CRUD 生命周期钩子真实接线** — 此前 `beforeInsert`/`afterInsert`/`beforeUpdate`/`afterUpdate`/`beforeDelete`/`afterDelete`
|
||||||
|
六个钩子仅存在于类型与文档宣称("14 种钩子"),从未被任何路径触发。现已在两条路径真实触发:
|
||||||
|
- Table API:`table().insert/insertMany/update().execute()/delete().execute()`
|
||||||
|
- SQL:`db.query("INSERT/UPDATE/DELETE ...")`
|
||||||
|
参数与文档一致(beforeInsert: rows[]、afterUpdate: query, updates, count 等);事务路径不触发(Table 无 hooks 回调,与现状一致)
|
||||||
|
- **EXPLAIN SQL 入口** — 此前 executor 有 EXPLAIN 分支但 parser 无法解析(无 token),只能手工构造 AST 测试。
|
||||||
|
补齐 `EXPLAIN <statement>` 语法(tokens/lexer/parser),任意引擎可用
|
||||||
|
- **ANALYZE / REINDEX / VACUUM SQL 入口** — 此前仅有 AriaEngine 方法(`(engine as any)` 才能调用),README 宣称能力但用户无法触发。
|
||||||
|
补齐 `ANALYZE [TABLE] name` / `REINDEX [TABLE] name` / `VACUUM` 语法;不支持的引擎抛 `NOT_SUPPORTED`
|
||||||
|
- **SAVEPOINT SQL 入口** — 补齐 `SAVEPOINT name` / `ROLLBACK TO [SAVEPOINT] name` / `RELEASE [SAVEPOINT] name`;
|
||||||
|
`ROLLBACK [TRANSACTION]` 保持原语义
|
||||||
|
- **db.backup() 公共方法** — 引擎级 backup(MVCC 一致性视图)此前无公共入口;Aria 走引擎级,其余引擎回退 exportAll
|
||||||
|
|
||||||
|
### Removed(死代码)
|
||||||
|
|
||||||
|
- **`src/utils.ts` 整文件** — generateId/escapeHTML/deepMerge/debounce/throttle/isBrowser 在 src 内零引用
|
||||||
|
(v0.0.1 遗留,仅测试锁定);连同 tests/utils.test.ts 删除
|
||||||
|
- **`estimateQueryCost`(未接线的"查询优化器")** — 实现与测试存在但从未接入查询路径(find 用 tryIndexLookup 硬编码),删除消除"优化器存在但没用"假象
|
||||||
|
- **MVCC 读侧全套** — readVersion/getLatestCommittedVersions/clearTable/getActiveTxnCount/isActive/getActiveWriteKeys:
|
||||||
|
引擎事务读取走 txnSnapshot(MVCC 仅作 undo log),读侧实现从未执行。删除并改写相关测试为验证版本链清理/GC 行为
|
||||||
|
- **统计/辅助死方法** — EvictionManager.getAllPages/getSize/popLRU/getCapacity、BufferPool.getCachedPageCount/getDirtyPageCount/getCapacity/newPage、
|
||||||
|
WAL.isEnabled/getLSN、CheckpointManager.forceCheckpoint/setInterval/getOpCount、LSM.isInitialized/getCacheSize/getCacheLimit、
|
||||||
|
SSTableReader.getMeta/getIndexBlockCount/getStoredChecksum、BloomFilter.getBitSize/getInsertedCount、MemTable.contains、
|
||||||
|
FileManager.allocateTableRootPage/createEmptyPage、OPFSEngine.loadTableIntoMemory、QueryExecutor.setMaxRowsPerQuery、
|
||||||
|
ConnectionManager.getRefCount
|
||||||
|
- **类型/常量死代码** — STORAGE_MODES/DISK_ENGINES、ColumnEncoding/DEFAULT_BLOCK_SIZE/DataBlock、CallbackEntrySource、StatementType、AggregateExpression、SubqueryExpression
|
||||||
|
- **Crypto 全局兼容层** — initCrypto/encryptPage/decryptPage/closeCrypto/isCryptoEnabled(v0.5.0 后引擎走 EncryptedBackend 实例,全局单例无消费者);删除过时的"全局加密落盘"测试(aria-encryption.test.ts 已覆盖等价路径)
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- 测试 1022 → **1010**(63 套件;删除 12 个死代码测试 + 新增 22 个 hooks/维护语句测试);覆盖率 86.68% → 待发布后统计
|
||||||
|
- README 补维护语句表(EXPLAIN/ANALYZE/REINDEX/VACUUM/SAVEPOINT SQL 入口)与 `db.backup()` API
|
||||||
|
- 14 个生命周期钩子现在全部真实触发(site/docs.html 与 CONTRIBUTING 宣称变为事实)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## [0.5.0] - 2026-08-10
|
||||||
|
|
||||||
|
### 里程碑:存储后端生产级硬化(自研 OPFS 后端全面落地)
|
||||||
|
|
||||||
|
> 本次版本对 AriaEngine 存储层做全链路生产级改造:修复"宣称有校验和/加密/页面式存储但实际未实现"的失实实现,
|
||||||
|
> WAL 重构为分片文件(消除多文件原子性与文件爆炸问题)、SSTable 页面化物理存储(BufferPool/FileManager 真正接入)、
|
||||||
|
> 多标签页独占锁、repair 自愈增强,并引入真实 Chromium e2e 验证。
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **全库 AES-256-GCM 加密(真实接线)** — `AriaEngineConfig.encryption.password`:
|
||||||
|
- `EncryptedBackend` 装饰器在 backend 层透明加解密(WAL/SSTable/Schema/元数据全覆盖,每个 value 独立随机 IV + GCM 认证)
|
||||||
|
- PBKDF2-SHA256(100000 迭代)派生密钥,salt + verifier 持久化于 `__aria_keymeta`(密码错误 → `ARIA_DECRYPT_ERROR`)
|
||||||
|
- 加密开关一致性检测:明文旧库 + 加密配置 → `ARIA_ENCRYPT_CONFIG_ERROR`;加密库 + 无密码 → `ARIA_ENCRYPT_REQUIRED`
|
||||||
|
- clearAll 保留密钥元数据(库身份保留,密码不变)
|
||||||
|
- **WAL 分片存储重构** — `SegmentedWALStore`(`__wal_%06d.bin`):
|
||||||
|
- 取代"每条记录一个 key"旧结构:分片阈值 4MB 自动切换,文件数量可控
|
||||||
|
- WAL 序号(LSN)内嵌记录字节流,移除 `__wal_count` 键 → append 单文件原子写(createWritable COW)
|
||||||
|
- 空洞检测:分片序号不连续 → 空洞之后保守丢弃(仅可能来自 truncate 部分完成,数据已落盘,无害)
|
||||||
|
- 旧格式 `__wal_N` + `__wal_count` 兼容:读取迁移重放,checkpoint 统一清空
|
||||||
|
- **OPFS 后端 v2** —
|
||||||
|
- `append()` 真追加:`createWritable({ keepExistingData })` + seek,O(chunk) 无需读旧内容
|
||||||
|
- 写队列健壮性:单次任务失败不中断队列链(错误返回给调用方),close 等待队列排空
|
||||||
|
- open 自动清理崩溃残留临时文件(`.crswap`/`.tmp`)
|
||||||
|
- 原子性文档化:单文件 createWritable 为 copy-on-write 原子替换
|
||||||
|
- **SSTable 页面化物理存储(BufferPool/FileManager 真正接入)** —
|
||||||
|
- `PageSSTableStore`:SSTable 切分为 4KB 页面,经 FileManager 分配 pageId、BufferPool LRU 缓存
|
||||||
|
- `save` 语义 = 已持久化(逐页 flushPage 落盘后才返回,WAL checkpoint 截断安全)
|
||||||
|
- 页面 ID 列表经 `SSTableMeta.pageIds` 持久化;旧 meta(无 pageIds)自动走整 value 读取(兼容)
|
||||||
|
- OPFS 后端默认启用(`pageStorage` 配置可覆盖);FileManager 批量分配 `allocatePageIds`(一次 meta 持久化)
|
||||||
|
- 读取缺失页面返回 null(此前返回空页会静默掩盖页面丢失,自愈路径依赖 null 触发清理)
|
||||||
|
- **多标签页独占锁** — Web Locks API(Chrome 69+ / Firefox 96+ / Safari 15.4+):
|
||||||
|
- `ifAvailable` 模式:第二个标签页打开同一库 → 抛 `ARIA_LOCKED`(不排队挂起)
|
||||||
|
- 打开失败自动释放锁(不泄漏);close 等待锁真正归还
|
||||||
|
- 不支持 Web Locks 的环境降级为无锁 + 告警(如实文档化)
|
||||||
|
- **repair() 自愈增强** — 孤儿页面清理(meta 未引用的 `pg_` 文件)、OPFS 残留临时文件清理、WAL 空洞截断落地
|
||||||
|
- **LZ4 格式 v2** — 压缩流前置 4 字节原始大小头:解压不再依赖外部估算(高压缩率数据下 `buf.length*2` 估算不足会截断),损坏头显式抛错
|
||||||
|
- **SSTable 真实 CRC-32 校验和** — footer checksum 字段从写死 0 升级为整文件标准 CRC32(checksum=0 识别旧版文件跳过校验,兼容旧库);打开校验/compaction 加载/运行期预加载三处均校验,损坏文件自动清理自愈
|
||||||
|
- **WAL 标准 CRC32** — 记录校验从弱滚动校验升级为标准 CRC32;双算法探测(新算法失败再试旧算法)兼容旧 WAL 记录
|
||||||
|
- **配置透传** — `DatabaseConfig.aria`:`MetonaSqlark.create()` 可配置 `walSyncMode`/`checkpointInterval`/`encryption`/`pageStorage`/`compression` 等 AriaEngine 专属参数
|
||||||
|
- **Playwright e2e(真实 Chromium + OPFS)** — 7 用例:持久化(刷新保留)、关闭重开、崩溃恢复(WAL 重放)、多标签页锁、加密密码往返、页面化端到端 + 索引查询、repair 自愈;`npm run test:e2e`
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **加密此前从未接线(严重失实)** — crypto 模块全局单例从未被引擎调用、salt 不持久化重启无法解密;README 宣称的 AES-GCM 实际不可用。改为 EncryptedBackend 装饰器 + keymeta 持久化 + 密码验证
|
||||||
|
- **SSTable checksum 写死 0(严重失实)** — 数据腐坏(bit flip)完全无法检测。升级为真实 CRC-32,损坏文件打开自愈
|
||||||
|
- **OPFS writeMany 无原子性(P0)** — WAL 记录 + count 多键写入非原子,记录空洞导致后续记录字节错位解析失败。分片重构后 WAL append 单文件原子写 + 空洞检测截断根治
|
||||||
|
- **OPFS 高频写入文件爆炸(P1)** — 每条 WAL 记录一个文件,8 万条记录 = 8 万次 createWritable。分片存储 + 真追加根治
|
||||||
|
- **LZ4 高压缩率解压截断(P1)** — `buf.length * 2` 估算原始大小,压缩率 >50% 时截断(SSTable 魔数在文件末尾被丢 → 整库数据丢失)。格式 v2 头部自描述根治
|
||||||
|
- **OPFS 写队列失败永久中断(P1)** — 队列任务抛错后链 rejected,后续所有写挂起。单次失败不中断链
|
||||||
|
- **OPFS close 不等挂起写(P1)** — close 立即置空目录句柄,挂起写静默丢失。close 等待队列排空
|
||||||
|
- **Web Locks 方法解构 this 绑定丢失(e2e 发现)** — `navigator.locks.request` 解构后调用抛 Illegal invocation。改用 `.bind(lockManager)` 保持 this
|
||||||
|
- **页面化 + 崩溃窗口数据丢失(压力测试发现)** — 修复前页面化路径在模拟崩溃测试中丢失已确认写入;根治后 500 随机操作 + 崩溃重放零丢失
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- `AriaEngineConfig` 新增 `encryption` / `pageStorage`;`DatabaseConfig` 新增 `aria` 透传
|
||||||
|
- 页面层精简:删除从未接入主路径的 slot/tuple 模块(假"页面式存储"实现),仅保留生产代码实际使用的 `initPageHeader`/`createPage`;架构宣称如实更新(4KB 页面承载 SSTable 字节切片,非行级 Slotted Page)
|
||||||
|
- LZ4 压缩格式不向后兼容(v0.2.6 已声明 compression 数据需重新导入,本次格式 v2 同理)
|
||||||
|
- 测试 958 → **1022**(62 套件);行覆盖率 84.2% → **86.68%**;新增 6 套件:crc32 / aria-checksum / aria-wal-crc / aria-wal-segment / aria-opfs-backend / aria-page-store / aria-encryption / aria-locks / aria-repair-hardening / aria-page-core
|
||||||
|
- 新增 Playwright e2e 基建(`npm run test:e2e`,7 用例真实 Chromium)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## [0.4.4] - 2026-08-09
|
## [0.4.4] - 2026-08-09
|
||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
|
|||||||
+3
-1
@@ -73,7 +73,9 @@ src/
|
|||||||
├── plugin/ # Plugin system (14 lifecycle hooks)
|
├── plugin/ # Plugin system (14 lifecycle hooks)
|
||||||
└── integrations/ # React & Vue hooks
|
└── integrations/ # React & Vue hooks
|
||||||
|
|
||||||
tests/ # Test suite (837+ test cases, 44 test suites)
|
tests/ # Test suite (1022 test cases, 62 suites + 7 e2e)
|
||||||
|
tests/helpers/ # 共享测试工具(OPFS mock 等)
|
||||||
|
tests/e2e/ # Playwright e2e(真实 Chromium + OPFS)
|
||||||
site/ # Documentation site (index / docs / demo)
|
site/ # Documentation site (index / docs / demo)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
@@ -1,21 +1,22 @@
|
|||||||
# MetonaSqlark
|
# MetonaSqlark
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<img src="https://img.shields.io/badge/version-0.4.4-blue?style=flat-square" alt="version">
|
<img src="https://img.shields.io/badge/version-0.5.0-blue?style=flat-square" alt="version">
|
||||||
<img src="https://img.shields.io/badge/license-MIT-green?style=flat-square" alt="license">
|
<img src="https://img.shields.io/badge/license-MIT-green?style=flat-square" alt="license">
|
||||||
<img src="https://img.shields.io/badge/coverage-84.2%25-brightgreen?style=flat-square" alt="coverage">
|
<img src="https://img.shields.io/badge/coverage-86.7%25-brightgreen?style=flat-square" alt="coverage">
|
||||||
<img src="https://img.shields.io/badge/tests-958%20passed-success?style=flat-square" alt="tests">
|
<img src="https://img.shields.io/badge/tests-1022%20passed-success?style=flat-square" alt="tests">
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
> 基于 TypeScript 的**前端关系型数据库**,支持完整 SQL 查询、Query Builder 链式 API、与 **AriaEngine 自研页面式存储引擎**。
|
> 基于 TypeScript 的**前端关系型数据库**,支持完整 SQL 查询、Query Builder 链式 API、与 **AriaEngine 自研存储引擎**。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## ✨ 特性
|
## ✨ 特性
|
||||||
|
|
||||||
- 🚀 **AriaEngine 自研存储引擎** — LSM-Tree 页面式存储,4KB Slotted Page、WAL 崩溃恢复(full 模式真正同步)、LZ4 压缩
|
- 🚀 **AriaEngine 自研存储引擎** — LSM-Tree + **SSTable 4KB 页面化物理存储**(BufferPool LRU 缓存 + FileManager 页面管理,v0.5.0 真实接入)、WAL 分片文件(真追加 + 空洞检测)、LZ4 压缩
|
||||||
- 💾 **OPFS 自研存储后端** — 纯浏览器文件系统,零 IndexedDB 依赖,二进制页面文件,schema 持久化(空表/索引跨重启完整保留)
|
- 💾 **OPFS 自研存储后端** — 纯浏览器文件系统,零 IndexedDB 依赖,二进制页面文件,schema 持久化,单文件 COW 原子写,崩溃残留自动清理
|
||||||
- 🔒 **生产级数据安全** — WAL 原子写入 + CRC 完整性校验、`RESTRICT` 外键约束、崩溃恢复自愈(`repair()` 无需删库重建)、SQL 注入防护
|
- 🔒 **生产级数据安全** — WAL 原子写入 + 标准 CRC32 完整性校验(SSTable 整文件校验 + WAL 记录校验)、**全库 AES-256-GCM 透明加密**(PBKDF2 密钥派生 + salt 持久化 + 密码验证,v0.5.0 真实接线)、`RESTRICT` 外键约束、崩溃恢复自愈
|
||||||
|
- 🔐 **多标签页独占锁** — Web Locks API,第二个标签页打开同一库抛 `ARIA_LOCKED`(v0.5.0)
|
||||||
- 🛡 **输入校验全覆盖** — `maxLength`/`min`/`max` 约束、类型检查、必填验证
|
- 🛡 **输入校验全覆盖** — `maxLength`/`min`/`max` 约束、类型检查、必填验证
|
||||||
- 💾 **多引擎架构** — Memory / IndexedDB / OPFS / Hybrid(write-through) / Aria 五种模式
|
- 💾 **多引擎架构** — Memory / IndexedDB / OPFS / Hybrid(write-through) / Aria 五种模式
|
||||||
- 📝 **完整 SQL 支持** — SELECT/JOIN/子查询/GROUP BY/HAVING/ORDER BY/LIMIT/BETWEEN/IF NOT EXISTS/ALTER TABLE/TRUNCATE TABLE/UNION/INSERT...SELECT/事务语句/CREATE INDEX/EXISTS(v0.3.0)+ CASE WHEN/哈希连接/组提交(v0.3.1)+ 多标签页同步(v0.3.2)
|
- 📝 **完整 SQL 支持** — SELECT/JOIN/子查询/GROUP BY/HAVING/ORDER BY/LIMIT/BETWEEN/IF NOT EXISTS/ALTER TABLE/TRUNCATE TABLE/UNION/INSERT...SELECT/事务语句/CREATE INDEX/EXISTS(v0.3.0)+ CASE WHEN/哈希连接/组提交(v0.3.1)+ 多标签页同步(v0.3.2)
|
||||||
@@ -24,13 +25,13 @@
|
|||||||
- 🔗 **Query Builder API** — 链式 `.select().where().orderBy().limit().execute()`
|
- 🔗 **Query Builder API** — 链式 `.select().where().orderBy().limit().execute()`
|
||||||
- 🔄 **事务回滚** — Memory/IndexedDB/Hybrid/Aria 四引擎事务原子性,自动回滚,MVCC 版本链接入读写路径
|
- 🔄 **事务回滚** — Memory/IndexedDB/Hybrid/Aria 四引擎事务原子性,自动回滚,MVCC 版本链接入读写路径
|
||||||
- 🔗 **外键级联** — ON DELETE + ON UPDATE(CASCADE / SET NULL / RESTRICT)全引擎支持,支持更新主键(v0.4.2)
|
- 🔗 **外键级联** — ON DELETE + ON UPDATE(CASCADE / SET NULL / RESTRICT)全引擎支持,支持更新主键(v0.4.2)
|
||||||
- 🛡 **崩溃恢复自愈** — 残缺 SSTable 打开自动跳过、`db.repair()` 自愈、`db.clearAll()` 重置、迁移版本持久化(v0.4.2)
|
- 🛡 **崩溃恢复自愈** — 残缺 SSTable 打开自动跳过、`db.repair()` 自愈(含孤儿页面清理与残留清理)、`db.clearAll()` 重置、迁移版本持久化(v0.4.2/v0.5.0)
|
||||||
- 🧵 **关闭时序与后台任务加固** — 后台 flush/compaction 串行入队(close 排空后才关闭存储,失败显式报告 `ARIA_BACKGROUND_ERROR`)、预加载等待链稳定(compaction 竞态修复)、事务提交先落 WAL 再合并快照(v0.4.3)
|
- 🧵 **关闭时序与后台任务加固** — 后台 flush/compaction 串行入队(close 排空后才关闭存储,失败显式报告 `ARIA_BACKGROUND_ERROR`)
|
||||||
- 📏 **SSTable 编码修复** — 块大小按 UTF-8 字节精确计算(大段中文内容不再因缓冲区低估崩溃)、长度字段 u32(>64KB value 不截断)、v1/v2 双格式兼容(旧库数据不丢)(v0.4.4)
|
- 📏 **SSTable 编码修复** — 块大小按 UTF-8 字节精确计算、长度字段 u32、v1/v2 双格式兼容(v0.4.4)
|
||||||
- 🌲 **RB-Tree 完整实现** — 标准红黑树插入+删除修复,O(log n) 保证
|
- 🌲 **RB-Tree 完整实现** — 标准红黑树插入+删除修复,O(log n) 保证
|
||||||
- ⚡ **性能优化** — SSTableReader 二分查找统一、IndexedDB 索引利用、crypto 实例化避免全局状态
|
- ⚡ **性能优化** — SSTableReader 二分查找统一、IndexedDB 索引利用、crypto 实例化避免全局状态
|
||||||
- 🌐 **浏览器兼容** — Chrome 80+ / Firefox 80+ / Safari 14+ / Edge 80+ / Node.js 16+
|
- 🌐 **浏览器兼容** — Chrome 80+ / Firefox 80+ / Safari 14+ / Edge 80+ / Node.js 16+
|
||||||
- 🧪 **958 测试 · 84.2% 覆盖率** — 52 套件,生产级质量保证
|
- 🧪 **1022 测试 · 86.7% 覆盖率** — 62 套件 + 7 个 Playwright 真实 Chromium e2e,生产级质量保证
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -217,6 +218,7 @@ await db2.disconnect(); // 引用计数 -1
|
|||||||
| `maxRowsPerQuery` | `number` | `0` | 查询结果行数上限(0=不限制)✅ v0.2.5 生效 |
|
| `maxRowsPerQuery` | `number` | `0` | 查询结果行数上限(0=不限制)✅ v0.2.5 生效 |
|
||||||
| `debug` | `boolean` | `false` | 调试模式,输出详细日志 🆕 |
|
| `debug` | `boolean` | `false` | 调试模式,输出详细日志 🆕 |
|
||||||
| `onError` | `(error) => void` | — | 全局错误回调 🆕 |
|
| `onError` | `(error) => void` | — | 全局错误回调 🆕 |
|
||||||
|
| `aria` | `AriaEngineConfig` | — | AriaEngine 专属配置透传:`walSyncMode`/`checkpointInterval`/`encryption`/`pageStorage`/`compression` 等 🆕 v0.5.0 |
|
||||||
|
|
||||||
### ColumnDef 列定义
|
### ColumnDef 列定义
|
||||||
|
|
||||||
@@ -254,9 +256,24 @@ await db2.disconnect(); // 引用计数 -1
|
|||||||
| `db.clearAll()` | 清空全部数据与表结构(保留库本身,实例可继续使用)🆕 v0.4.2 |
|
| `db.clearAll()` | 清空全部数据与表结构(保留库本身,实例可继续使用)🆕 v0.4.2 |
|
||||||
| `db.exportTable(name)` / `db.exportAll()` | 导出数据 JSON |
|
| `db.exportTable(name)` / `db.exportAll()` | 导出数据 JSON |
|
||||||
| `db.importTable(name, data)` | 导入数据 |
|
| `db.importTable(name, data)` | 导入数据 |
|
||||||
|
| `db.backup()` | 在线备份:全库一致性快照(Aria 引擎级备份 / 其余引擎回退 exportAll)🆕 v0.5.1 |
|
||||||
| `db.addMigration(v, fn)` / `db.migrateTo(v)` | 数据迁移(版本持久化到库内,重启不重跑)🆕 v0.4.2 |
|
| `db.addMigration(v, fn)` / `db.migrateTo(v)` | 数据迁移(版本持久化到库内,重启不重跑)🆕 v0.4.2 |
|
||||||
| `db.subscribe(table, fn)` | 订阅表变更 |
|
| `db.subscribe(table, fn)` | 订阅表变更 |
|
||||||
| `db.on(hook, fn)` | 注册钩子 (14 种) |
|
| `db.on(hook, fn)` | 注册钩子 (14 种,全部真实触发)🆕 v0.5.1 |
|
||||||
|
|
||||||
|
### 维护语句(v0.5.1 SQL 入口)
|
||||||
|
|
||||||
|
| 语句 | 说明 | 引擎支持 |
|
||||||
|
|------|------|----------|
|
||||||
|
| `EXPLAIN SELECT ...` | 输出查询计划(type/table/where/estimatedRows/actualTimeMs) | 全部 |
|
||||||
|
| `ANALYZE [TABLE] name` | 收集表统计信息(行数/平均行大小/索引深度/列基数) | Aria |
|
||||||
|
| `REINDEX [TABLE] name` | 重建表二级索引 | Aria |
|
||||||
|
| `VACUUM` | 压缩 LSM + 清理 MVCC 碎片 | Aria |
|
||||||
|
| `SAVEPOINT name` | 创建嵌套事务保存点 | Aria |
|
||||||
|
| `ROLLBACK TO [SAVEPOINT] name` | 回滚到保存点 | Aria |
|
||||||
|
| `RELEASE [SAVEPOINT] name` | 释放保存点 | Aria |
|
||||||
|
|
||||||
|
> 不支持的引擎执行维护语句抛 `NOT_SUPPORTED`。
|
||||||
|
|
||||||
### 连接池(v0.1.13)
|
### 连接池(v0.1.13)
|
||||||
|
|
||||||
@@ -289,10 +306,12 @@ const { data, loading, refresh } = useSqlarkQuery(db, 'SELECT * FROM users');
|
|||||||
| **事务回滚** | ✅ 快照 | ✅ 原子flush | ✅ 快照 | ✅ 双引擎 | ✅ MVCC |
|
| **事务回滚** | ✅ 快照 | ✅ 原子flush | ✅ 快照 | ✅ 双引擎 | ✅ MVCC |
|
||||||
| **二级索引** | ✅ Hash | ✅ Hash | ✅ Hash(重启恢复) | ✅ Hash | ✅ LSM(重启恢复) |
|
| **二级索引** | ✅ Hash | ✅ Hash | ✅ Hash(重启恢复) | ✅ Hash | ✅ LSM(重启恢复) |
|
||||||
| **查询性能** | ⚡ O(1) PK | 🟡 O(1) PK | 🟡 O(1) PK | ⚡ O(1) PK | ⚡ O(log n) |
|
| **查询性能** | ⚡ O(1) PK | 🟡 O(1) PK | 🟡 O(1) PK | ⚡ O(1) PK | ⚡ O(log n) |
|
||||||
| **数据上限** | 内存限制 | ~2GB(IDB限制) | ~磁盘可用(整表重写,≤1000行/表为宜) | ~2GB(IDB) | 内存限制 |
|
| **数据上限** | 内存限制 | ~2GB(IDB限制) | ~磁盘可用(页面化后大表可行) | ~2GB(IDB) | 内存限制 |
|
||||||
| **浏览器** | 全部 | 全部 | Chrome/Edge 102+ | 全部 | 全部 |
|
| **浏览器** | 全部 | 全部 | Chrome/Edge 102+ / Firefox 111+ / Safari 15.2+ | 全部 | 全部 |
|
||||||
| **适用场景** | 缓存/测试 | 标准持久化 | Chromium专有 | 速度+持久化 | 大规模/分析 |
|
| **多标签页** | — | ✅ | ❌(Web Locks 独占锁,v0.5.0) | ❌ | ✅ 锁保护 |
|
||||||
| **测试覆盖** | 30+ | 30+ | 15 | 15+ | 200+ |
|
| **全库加密** | — | — | — | — | ✅ AES-GCM(v0.5.0) |
|
||||||
|
| **适用场景** | 缓存/测试 | 标准持久化 | Chromium+ 持久化 | 速度+持久化 | 大规模/分析 |
|
||||||
|
| **测试覆盖** | 30+ | 30+ | 15+e2e | 15+ | 400+ |
|
||||||
|
|
||||||
### Memory 模式
|
### Memory 模式
|
||||||
- **环境**: 所有浏览器、Node.js
|
- **环境**: 所有浏览器、Node.js
|
||||||
@@ -307,10 +326,10 @@ const { data, loading, refresh } = useSqlarkQuery(db, 'SELECT * FROM users');
|
|||||||
- **适用**: 标准前端数据库持久化场景
|
- **适用**: 标准前端数据库持久化场景
|
||||||
|
|
||||||
### Disk (OPFS) 模式
|
### Disk (OPFS) 模式
|
||||||
- **环境**: **仅限** Chrome 102+ / Edge 102+(Origin Private File System)
|
- **环境**: Chrome 102+ / Edge 102+ / Firefox 111+ / Safari 15.2+(Origin Private File System)
|
||||||
- **限制**: Firefox/Safari 不支持 OPFS API;每次写入重写整表 JSON 文件(大表性能差,不建议 >1000 行)
|
- **限制**: 每次写入重写整表 JSON 文件(大表性能差,不建议 >1000 行);多标签页并发写无保护(Hybrid 模式)
|
||||||
- **能力**: 完整 CRUD、重启自动加载数据(空表/索引/schema 完整保留,v0.4.2)、事务回滚、并发写安全(内存快照一致)
|
- **能力**: 完整 CRUD、重启自动加载数据(空表/索引/schema 完整保留,v0.4.2)、事务回滚、并发写安全(内存快照一致)
|
||||||
- **适用**: Chromium 独占场景、小数据集持久化
|
- **适用**: 小数据集持久化(大表请用 Aria + OPFS 页面化存储)
|
||||||
|
|
||||||
### Hybrid 模式
|
### Hybrid 模式
|
||||||
- **环境**: 所有浏览器
|
- **环境**: 所有浏览器
|
||||||
@@ -320,8 +339,8 @@ const { data, loading, refresh } = useSqlarkQuery(db, 'SELECT * FROM users');
|
|||||||
|
|
||||||
### Aria 模式
|
### Aria 模式
|
||||||
- **环境**: 所有浏览器(后端可选 IndexedDB / OPFS / Memory)
|
- **环境**: 所有浏览器(后端可选 IndexedDB / OPFS / Memory)
|
||||||
- **限制**: Memory 后端重启丢失;IndexedDB 后端受配额限制;OPFS 后端仅 Chromium
|
- **限制**: Memory 后端重启丢失;IndexedDB 后端受配额限制;OPFS 后端需 OPFS 支持(Chrome 102+ / Firefox 111+ / Safari 15.2+);同一库仅允许一个标签页打开(Web Locks 独占锁,第二个标签页抛 `ARIA_LOCKED`)
|
||||||
- **能力**: LSM-Tree 存储引擎、二级索引(跨重启恢复)、MVCC 事务、WAL 原子写入崩溃恢复(残缺 SSTable 打开自动跳过)、ON UPDATE/DELETE 外键级联、Bloom Filter、AES-GCM 加密、Savepoint、EXPLAIN、ANALYZE、REINDEX、VACUUM、`repair()` 自愈
|
- **能力**: LSM-Tree 存储引擎、SSTable 4KB 页面化物理存储(OPFS 后端默认启用,BufferPool LRU 缓存)、二级索引(跨重启恢复)、MVCC 事务、WAL 分片文件原子写入崩溃恢复(真追加 + 空洞检测)、ON UPDATE/DELETE 外键级联、Bloom Filter、**全库 AES-GCM 透明加密**(v0.5.0)、LZ4 压缩、Savepoint、EXPLAIN、ANALYZE、REINDEX、VACUUM、`repair()` 自愈(含孤儿页面清理)
|
||||||
- **适用**: 大规模数据分析、需要自研引擎可控性的高级场景
|
- **适用**: 大规模数据分析、需要自研引擎可控性的高级场景
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -331,11 +350,15 @@ const { data, loading, refresh } = useSqlarkQuery(db, 'SELECT * FROM users');
|
|||||||
AriaEngine 是内置的页面式存储引擎,对标 SQLite 的设计理念:
|
AriaEngine 是内置的页面式存储引擎,对标 SQLite 的设计理念:
|
||||||
|
|
||||||
```typescript
|
```typescript
|
||||||
// 激活 AriaEngine
|
// 激活 AriaEngine(全库加密 + OPFS 页面化存储,v0.5.0)
|
||||||
const db = await MetonaSqlark.create({
|
const db = await MetonaSqlark.create({
|
||||||
name: 'my-app',
|
name: 'my-app',
|
||||||
mode: 'aria', // 🆕 自研引擎模式
|
mode: 'aria', // 🆕 自研引擎模式
|
||||||
diskEngine: 'indexeddb', // 'indexeddb' | 'opfs' | 'memory'
|
diskEngine: 'opfs', // 'indexeddb' | 'opfs' | 'memory'
|
||||||
|
aria: { // 🆕 AriaEngine 专属配置透传(v0.5.0)
|
||||||
|
walSyncMode: 'full', // WAL 同步模式
|
||||||
|
encryption: { password: 'my-password' }, // 全库 AES-GCM 加密
|
||||||
|
},
|
||||||
});
|
});
|
||||||
|
|
||||||
// 与现有 API 完全兼容
|
// 与现有 API 完全兼容
|
||||||
@@ -351,36 +374,39 @@ const rows = await db.query('SELECT * FROM users');
|
|||||||
|
|
||||||
```
|
```
|
||||||
┌──────────────────────────────────────────┐
|
┌──────────────────────────────────────────┐
|
||||||
│ AriaEngine v0.2.5 │
|
│ AriaEngine v0.5.0 │
|
||||||
│ (implements IStorageEngine) │
|
│ (implements IStorageEngine) │
|
||||||
├──────────────────────────────────────────┤
|
├──────────────────────────────────────────┤
|
||||||
│ LSM-Tree │ Buffer Pool │ WAL │
|
│ LSM-Tree │ Buffer Pool │ WAL │
|
||||||
│ MemTable │ LRU (256pp) │ Recovery │
|
│ MemTable │ LRU (256pp) │ Segmented │
|
||||||
│ +SSTable │ +FileManager │ +CRC │
|
│ +SSTable │ +FileManager │ +CRC-32 │
|
||||||
|
│ (4KB Page) │ │ +空洞检测 │
|
||||||
├──────────────────────────────────────────┤
|
├──────────────────────────────────────────┤
|
||||||
│ MVCC │ Bloom Filter │ LZ4 │
|
│ MVCC │ Bloom Filter │ LZ4 v2 │
|
||||||
│ Snapshot │ FNV-1a+Murmur│ Compress │
|
│ Snapshot │ FNV-1a+Murmur│ +大小头 │
|
||||||
│ +Savepoint│ +Serialize │ │
|
│ +Savepoint│ +Serialize │ │
|
||||||
├──────────────────────────────────────────┤
|
├──────────────────────────────────────────┤
|
||||||
│ AES-GCM │ 二级索引 │ ANALYZE │
|
│ EncryptedBackend │ 二级索引 │ ANALYZE │
|
||||||
│ Encrypt │ Per-Column │ EXPLAIN │
|
│ AES-GCM 全库 │ Per-Column│ EXPLAIN │
|
||||||
├──────────────────────────────────────────┤
|
├──────────────────────────────────────────┤
|
||||||
│ Storage Backend (IDB / OPFS / Memory) │
|
│ Web Locks 独占锁 │ OPFS Backend (v2) │
|
||||||
|
│ (多标签页) │ append/COW原子/清理 │
|
||||||
└──────────────────────────────────────────┘
|
└──────────────────────────────────────────┘
|
||||||
```
|
```
|
||||||
|
|
||||||
| 特性 | 说明 |
|
| 特性 | 说明 |
|
||||||
|------|------|
|
|------|------|
|
||||||
| **LSM-Tree** | MemTable (红黑树) → SSTable 多级索引,异步 Compaction(从存储兜底加载,不依赖缓存),写背压 |
|
| **LSM-Tree** | MemTable (红黑树) → SSTable 多级索引,异步 Compaction(从存储兜底加载,不依赖缓存),写背压 |
|
||||||
| **WAL** | Write-Ahead Log 二进制格式,CRC 校验,记录与计数单事务原子写入,full/batch/none 三种模式(full 模式真正同步 ✅ v0.2.5),16MB 阈值自动 checkpoint(活跃事务期间不截断 ✅ v0.4.2) |
|
| **页面化物理存储** | OPFS 后端下 SSTable 存为 4KB 页面(`pageStorage` 自动启用):FileManager 分配 pageId、BufferPool LRU 缓存(256 页 ≈ 1MB 可控内存)、save 即落盘(WAL checkpoint 截断安全)、`SSTableMeta.pageIds` 持久化(旧整 value 数据兼容)v0.5.0 |
|
||||||
| **崩溃恢复** | 打开时完整性校验(残缺 SSTable 自动跳过并清理)、WAL 按 key 扫描恢复(不丢记录)、恢复后自动重建二级索引 ✅ v0.4.2 |
|
| **WAL** | 分片文件 `__wal_%06d.bin`(4MB 阈值切换)+ 真追加(createWritable keepExistingData);标准 CRC32 记录校验;full/batch/none 三种模式(full 真正同步);空洞检测截断;16MB 阈值自动 checkpoint(活跃事务期间不截断)v0.5.0 |
|
||||||
|
| **崩溃恢复** | 打开时完整性校验(残缺 SSTable 自动跳过并清理 + 整文件 CRC-32 校验)、WAL 按 key 扫描恢复(不丢记录)、恢复后自动重建二级索引 |
|
||||||
|
| **全库加密** | `encryption.password` 配置 → EncryptedBackend 透明加解密(WAL/SSTable/Schema/元数据全密文);PBKDF2-SHA256 密钥派生 + salt/verifier 持久化;密码错误/数据篡改 → `ARIA_DECRYPT_ERROR`;明文库/加密库开关一致性检测 v0.5.0 |
|
||||||
| **MVCC** | 版本链 + 快照隔离,事务读写不互斥,提交/回滚按事务写入 key 精准清理,自动 GC |
|
| **MVCC** | 版本链 + 快照隔离,事务读写不互斥,提交/回滚按事务写入 key 精准清理,自动 GC |
|
||||||
| **Buffer Pool** | SSTable 缓存 LRU 上限(`bufferPoolPages` × `pageSize`,默认 256 页 ≈ 1MB 可控内存)✅ v0.2.6 生效,查询前异步预加载兜底,缓存驱逐不丢数据 |
|
|
||||||
| **Bloom Filter** | FNV-1a + Murmur 双哈希,SSTable footer 序列化,查询时 probe |
|
| **Bloom Filter** | FNV-1a + Murmur 双哈希,SSTable footer 序列化,查询时 probe |
|
||||||
| **二级索引** | 每列独立 LSM Tree,支持 $eq/$in/$gt/$lt 范围扫描,跨重启自动恢复,WAL 恢复后自动重建 ✅ v0.4.2 |
|
| **二级索引** | 每列独立 LSM Tree,支持 $eq/$in/$gt/$lt 范围扫描,跨重启自动恢复,WAL 恢复后自动重建 |
|
||||||
| **AES-GCM** | PBKDF2 密钥派生 + AES-256-GCM 页面级加密,CryptoManager 实例化 ✅ v0.2.5 |
|
| **多标签页锁** | Web Locks API 库级独占锁(`ifAvailable` 不排队):第二个标签页抛 `ARIA_LOCKED`;不支持的环境降级无锁并告警 v0.5.0 |
|
||||||
| **Compaction** | 异步 Leveled Compaction,Level 0 > 8 触发同步背压,compactLevel public 接口 ✅ v0.2.5 |
|
| **Compaction** | 异步 Leveled Compaction,Level 0 > 8 触发同步背压,compactLevel public 接口 |
|
||||||
| **OPFS Backend** | 纯浏览器文件系统,Promise 队列串行写,零外部依赖 |
|
| **OPFS Backend** | 零外部依赖纯浏览器文件系统:单文件 COW 原子写、`append` 真追加、写队列失败不中断、close 排空、崩溃残留自动清理 v0.5.0 |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -391,6 +417,7 @@ npm install # 安装依赖
|
|||||||
npm run dev # 开发模式(localhost:3001)
|
npm run dev # 开发模式(localhost:3001)
|
||||||
npm run build # 生产构建(生成 dist/)
|
npm run build # 生产构建(生成 dist/)
|
||||||
npm test # 运行测试
|
npm test # 运行测试
|
||||||
|
npm run test:e2e # Playwright e2e(真实 Chromium + OPFS,需先 build)
|
||||||
npm run lint # 代码检查
|
npm run lint # 代码检查
|
||||||
npm run typecheck # 类型检查
|
npm run typecheck # 类型检查
|
||||||
```
|
```
|
||||||
@@ -401,23 +428,29 @@ npm run typecheck # 类型检查
|
|||||||
|
|
||||||
| 指标 | 数值 |
|
| 指标 | 数值 |
|
||||||
|------|------|
|
|------|------|
|
||||||
| 测试用例 | 958 |
|
| 测试用例 | 1022 |
|
||||||
| 测试套件 | 51 |
|
| 测试套件 | 62(+7 Playwright e2e) |
|
||||||
| 行覆盖率 | 84.2% |
|
| 行覆盖率 | 86.7% |
|
||||||
| SQL 关键字 | 36 |
|
| SQL 关键字 | 36 |
|
||||||
| 存储引擎 | 5(Memory / IndexedDB / OPFS / Hybrid / **Aria**) |
|
| 存储引擎 | 5(Memory / IndexedDB / OPFS / Hybrid / **Aria**) |
|
||||||
|
|
||||||
### 🌐 浏览器兼容性
|
### 🌐 浏览器兼容性
|
||||||
|
|
||||||
| 浏览器 | 最低版本 | Memory | IndexedDB | OPFS | Aria |
|
| 浏览器 | 最低版本 | Memory | IndexedDB | OPFS | Web Locks | Aria |
|
||||||
|--------|----------|--------|-----------|------|------|
|
|--------|----------|--------|-----------|------|-----------|------|
|
||||||
| Chrome | 80+ | ✅ | ✅ | ✅ (102+) | ✅ |
|
| Chrome | 80+ | ✅ | ✅ | ✅ (102+) | ✅ (69+) | ✅ |
|
||||||
| Firefox | 80+ | ✅ | ✅ | ❌ | ✅ |
|
| Firefox | 80+ | ✅ | ✅ | ✅ (111+) | ✅ (96+) | ✅ |
|
||||||
| Safari | 14+ | ✅ | ✅ | ❌ | ✅ |
|
| Safari | 14+ | ✅ | ✅ | ✅ (15.2+) | ✅ (15.4+) | ✅ |
|
||||||
| Edge | 80+ | ✅ | ✅ | ✅ (102+) | ✅ |
|
| Edge | 80+ | ✅ | ✅ | ✅ (102+) | ✅ (79+) | ✅ |
|
||||||
| Node.js | 16+ | ✅ | ✅ (fake-idb) | ❌ | ✅ |
|
| Node.js | 16+ | ✅ | ✅ (fake-idb) | ❌ (测试用 mock) | ❌ | ✅ |
|
||||||
|
|
||||||
> **注意**: OPFS 模式仅限 Chromium 内核浏览器 (Chrome/Edge 102+),Firefox/Safari 请使用 `diskEngine: 'indexeddb'`。
|
> **OPFS 支持说明(v0.4.5 更新)**: Chromium(Chrome/Edge 102+)、Firefox 111+、Safari 15.2+
|
||||||
|
> 均已支持基础 OPFS API(`createWritable` 原子写)。OPFS 无跨文件事务,
|
||||||
|
> AriaEngine 以 WAL 分片单文件原子写 + 空洞检测截断保证崩溃一致性。
|
||||||
|
>
|
||||||
|
> **多标签页保护(v0.4.5)**: AriaEngine 打开库时通过 Web Locks API 获取库级独占锁,
|
||||||
|
> 第二个标签页打开同一库会抛 `ARIA_LOCKED`。Web Locks 不可用的环境降级为无锁并告警
|
||||||
|
> (仅理论上,现代浏览器均支持)。
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
Vendored
+1262
-522
File diff suppressed because it is too large
Load Diff
Vendored
+1
-1
File diff suppressed because one or more lines are too long
Vendored
+145
-62
@@ -1,3 +1,44 @@
|
|||||||
|
interface AriaEngineConfig {
|
||||||
|
/** 页面大小(默认 4096) */
|
||||||
|
pageSize?: number;
|
||||||
|
/** Buffer Pool 页面数量(默认 256) */
|
||||||
|
bufferPoolPages?: number;
|
||||||
|
/** MemTable 刷盘阈值(默认 4MB) */
|
||||||
|
memtableSizeThreshold?: number;
|
||||||
|
/** LSM 层级之间的容量倍数(默认 10) */
|
||||||
|
levelSizeMultiplier?: number;
|
||||||
|
/** Bloom Filter 每 key 位数(默认 10) */
|
||||||
|
bloomFilterBitsPerKey?: number;
|
||||||
|
/** 是否启用 WAL(默认 true) */
|
||||||
|
walEnabled?: boolean;
|
||||||
|
/** WAL 同步模式 */
|
||||||
|
walSyncMode?: 'full' | 'batch' | 'none';
|
||||||
|
/** Checkpoint 间隔(操作数,默认 1000) */
|
||||||
|
checkpointInterval?: number;
|
||||||
|
/** 是否启用页面压缩(默认 false) */
|
||||||
|
compression?: boolean;
|
||||||
|
/** 存储后端 */
|
||||||
|
storageBackend?: 'indexeddb' | 'opfs' | 'memory';
|
||||||
|
/** WAL 大小阈值(字节,超过则强制 checkpoint,默认 16MB) */
|
||||||
|
walSizeThreshold?: number;
|
||||||
|
/** 最大内存预算(MB,默认 64) */
|
||||||
|
maxMemoryMB?: number;
|
||||||
|
/**
|
||||||
|
* v0.4.5: 全库 AES-256-GCM 加密(backend 层透明加解密,WAL/SSTable/Schema/元数据全覆盖)。
|
||||||
|
* 密钥由 PBKDF2(salt 持久化于库内 __aria_keymeta)派生,重启用同一密码即可解密。
|
||||||
|
*/
|
||||||
|
encryption?: {
|
||||||
|
/** 加密密码 */
|
||||||
|
password: string;
|
||||||
|
};
|
||||||
|
/**
|
||||||
|
* v0.4.5: SSTable 页面化存储(4KB 页面 + BufferPool/FileManager 管理,LRU 缓存)。
|
||||||
|
* 默认:storageBackend === 'opfs' 时自动启用(大文件随机读/写放大优化);
|
||||||
|
* 显式 false 强制关闭(整 value 存储,兼容旧行为)。
|
||||||
|
*/
|
||||||
|
pageStorage?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* metona-sqlark Constants — 类型定义 / 默认配置 / 枚举
|
* metona-sqlark Constants — 类型定义 / 默认配置 / 枚举
|
||||||
* @module constants
|
* @module constants
|
||||||
@@ -64,6 +105,11 @@ interface DatabaseConfig {
|
|||||||
debug?: boolean;
|
debug?: boolean;
|
||||||
/** 多标签页同步(v0.3.2):BroadcastChannel 广播表变更,其他标签页自动刷新 */
|
/** 多标签页同步(v0.3.2):BroadcastChannel 广播表变更,其他标签页自动刷新 */
|
||||||
multiTabSync?: boolean;
|
multiTabSync?: boolean;
|
||||||
|
/**
|
||||||
|
* v0.4.5: AriaEngine 专属配置(mode='aria' 时透传):
|
||||||
|
* walSyncMode / checkpointInterval / encryption / pageStorage / compression 等
|
||||||
|
*/
|
||||||
|
aria?: AriaEngineConfig;
|
||||||
}
|
}
|
||||||
/** Where 条件操作符 */
|
/** Where 条件操作符 */
|
||||||
type WhereOperator = '$eq' | '$ne' | '$gt' | '$gte' | '$lt' | '$lte' | '$in' | '$nin' | '$like' | '$and' | '$or' | '$not';
|
type WhereOperator = '$eq' | '$ne' | '$gt' | '$gte' | '$lt' | '$lte' | '$in' | '$nin' | '$like' | '$and' | '$or' | '$not';
|
||||||
@@ -118,7 +164,7 @@ interface MetonaPlugin {
|
|||||||
/** 销毁 */
|
/** 销毁 */
|
||||||
destroy(): void;
|
destroy(): void;
|
||||||
}
|
}
|
||||||
declare const VERSION = "0.4.4";
|
declare const VERSION = "0.5.1";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* metona-sqlark Plugin — 插件系统
|
* metona-sqlark Plugin — 插件系统
|
||||||
@@ -350,7 +396,28 @@ interface CommitTransactionStatement {
|
|||||||
interface RollbackTransactionStatement {
|
interface RollbackTransactionStatement {
|
||||||
type: 'ROLLBACK';
|
type: 'ROLLBACK';
|
||||||
}
|
}
|
||||||
type Statement = SelectStatement | SelectUnionStatement | ExplainStatement | InsertStatement | UpdateStatement | DeleteStatement | CreateTableStatement | DropTableStatement | AlterTableStatement | TruncateTableStatement | CreateIndexStatement | DropIndexStatement | BeginTransactionStatement | CommitTransactionStatement | RollbackTransactionStatement;
|
/** SAVEPOINT name / ROLLBACK TO SAVEPOINT name / RELEASE SAVEPOINT name */
|
||||||
|
interface SavepointStatement {
|
||||||
|
type: 'SAVEPOINT';
|
||||||
|
name: string;
|
||||||
|
/** SAVE = 创建;ROLLBACK = 回滚到;RELEASE = 释放 */
|
||||||
|
action: 'SAVE' | 'ROLLBACK' | 'RELEASE';
|
||||||
|
}
|
||||||
|
/** ANALYZE TABLE name — 收集表统计信息 */
|
||||||
|
interface AnalyzeStatement {
|
||||||
|
type: 'ANALYZE';
|
||||||
|
table: string;
|
||||||
|
}
|
||||||
|
/** REINDEX TABLE name — 重建表二级索引 */
|
||||||
|
interface ReindexStatement {
|
||||||
|
type: 'REINDEX';
|
||||||
|
table: string;
|
||||||
|
}
|
||||||
|
/** VACUUM — 压缩 LSM + 清理碎片 */
|
||||||
|
interface VacuumStatement {
|
||||||
|
type: 'VACUUM';
|
||||||
|
}
|
||||||
|
type Statement = SelectStatement | SelectUnionStatement | ExplainStatement | InsertStatement | UpdateStatement | DeleteStatement | CreateTableStatement | DropTableStatement | AlterTableStatement | TruncateTableStatement | CreateIndexStatement | DropIndexStatement | BeginTransactionStatement | CommitTransactionStatement | RollbackTransactionStatement | SavepointStatement | AnalyzeStatement | ReindexStatement | VacuumStatement;
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* metona-sqlark Query Executor — AST 执行器
|
* metona-sqlark Query Executor — AST 执行器
|
||||||
@@ -363,8 +430,6 @@ declare class QueryExecutor {
|
|||||||
private engine;
|
private engine;
|
||||||
private maxRowsPerQuery;
|
private maxRowsPerQuery;
|
||||||
constructor(engine: IStorageEngine, maxRowsPerQuery?: number);
|
constructor(engine: IStorageEngine, maxRowsPerQuery?: number);
|
||||||
/** 设置查询结果行数上限 */
|
|
||||||
setMaxRowsPerQuery(max: number): void;
|
|
||||||
execute(stmt: Statement): Promise<unknown>;
|
execute(stmt: Statement): Promise<unknown>;
|
||||||
/** 递归执行 UNION / UNION ALL,返回合并结果 */
|
/** 递归执行 UNION / UNION ALL,返回合并结果 */
|
||||||
private executeSelectUnion;
|
private executeSelectUnion;
|
||||||
@@ -406,6 +471,14 @@ declare class QueryExecutor {
|
|||||||
private executeBegin;
|
private executeBegin;
|
||||||
private executeCommit;
|
private executeCommit;
|
||||||
private executeRollback;
|
private executeRollback;
|
||||||
|
/** SAVEPOINT name / ROLLBACK TO SAVEPOINT name / RELEASE SAVEPOINT name */
|
||||||
|
private executeSavepoint;
|
||||||
|
/** ANALYZE TABLE name — 收集表统计信息 */
|
||||||
|
private executeAnalyze;
|
||||||
|
/** REINDEX TABLE name — 重建表二级索引 */
|
||||||
|
private executeReindex;
|
||||||
|
/** VACUUM — 压缩 LSM + 清理碎片 */
|
||||||
|
private executeVacuum;
|
||||||
/** 列列表是否包含 CASE WHEN 表达式 */
|
/** 列列表是否包含 CASE WHEN 表达式 */
|
||||||
private hasCaseColumn;
|
private hasCaseColumn;
|
||||||
/**
|
/**
|
||||||
@@ -456,14 +529,6 @@ declare class QueryExecutor {
|
|||||||
private resolveOperatorSubqueries;
|
private resolveOperatorSubqueries;
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* metona-sqlark Query Builder — 链式查询构建器
|
|
||||||
* @module query/builder
|
|
||||||
*
|
|
||||||
* 链式调用 → 构建 AST → 执行引擎操作。
|
|
||||||
* 支持 JOIN(需要 Executor)。
|
|
||||||
*/
|
|
||||||
|
|
||||||
declare class SelectQueryBuilder {
|
declare class SelectQueryBuilder {
|
||||||
private engine;
|
private engine;
|
||||||
private tableName;
|
private tableName;
|
||||||
@@ -508,7 +573,9 @@ declare class UpdateQueryBuilder {
|
|||||||
private _updates;
|
private _updates;
|
||||||
private _where;
|
private _where;
|
||||||
private onWrite?;
|
private onWrite?;
|
||||||
constructor(engine: IStorageEngine, tableName: string, _updates: Record<string, unknown>, onWrite?: (table: string) => void);
|
/** v0.5.1: CRUD hooks 触发回调 */
|
||||||
|
private onHooks?;
|
||||||
|
constructor(engine: IStorageEngine, tableName: string, _updates: Record<string, unknown>, onWrite?: (table: string) => void, onHooks?: (hook: HookName, args: unknown[]) => Promise<void>);
|
||||||
where(condition: WhereCondition): this;
|
where(condition: WhereCondition): this;
|
||||||
execute(): Promise<number>;
|
execute(): Promise<number>;
|
||||||
toAST(): UpdateStatement;
|
toAST(): UpdateStatement;
|
||||||
@@ -518,17 +585,14 @@ declare class DeleteQueryBuilder {
|
|||||||
private tableName;
|
private tableName;
|
||||||
private _where;
|
private _where;
|
||||||
private onWrite?;
|
private onWrite?;
|
||||||
constructor(engine: IStorageEngine, tableName: string, onWrite?: (table: string) => void);
|
/** v0.5.1: CRUD hooks 触发回调 */
|
||||||
|
private onHooks?;
|
||||||
|
constructor(engine: IStorageEngine, tableName: string, onWrite?: (table: string) => void, onHooks?: (hook: HookName, args: unknown[]) => Promise<void>);
|
||||||
where(condition: WhereCondition): this;
|
where(condition: WhereCondition): this;
|
||||||
execute(): Promise<number>;
|
execute(): Promise<number>;
|
||||||
toAST(): DeleteStatement;
|
toAST(): DeleteStatement;
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* metona-sqlark Table — 表操作 API
|
|
||||||
* @module table/table
|
|
||||||
*/
|
|
||||||
|
|
||||||
declare class Table<T = Record<string, unknown>> {
|
declare class Table<T = Record<string, unknown>> {
|
||||||
readonly name: string;
|
readonly name: string;
|
||||||
private engine;
|
private engine;
|
||||||
@@ -536,7 +600,9 @@ declare class Table<T = Record<string, unknown>> {
|
|||||||
private executor;
|
private executor;
|
||||||
/** 写入回调(多标签页广播,v0.3.2) */
|
/** 写入回调(多标签页广播,v0.3.2) */
|
||||||
private onWrite?;
|
private onWrite?;
|
||||||
constructor(engine: IStorageEngine, tableName: string, executor?: QueryExecutor, onWrite?: (table: string) => void);
|
/** v0.5.1: CRUD 生命周期钩子触发回调(beforeInsert/afterInsert/... 真实接线) */
|
||||||
|
private onHooks?;
|
||||||
|
constructor(engine: IStorageEngine, tableName: string, executor?: QueryExecutor, onWrite?: (table: string) => void, onHooks?: (hook: HookName, args: unknown[]) => Promise<void>);
|
||||||
getSchema(): Promise<TableSchema>;
|
getSchema(): Promise<TableSchema>;
|
||||||
insert(row: T & Record<string, unknown>): Promise<string>;
|
insert(row: T & Record<string, unknown>): Promise<string>;
|
||||||
insertMany(rows: (T & Record<string, unknown>)[]): Promise<string[]>;
|
insertMany(rows: (T & Record<string, unknown>)[]): Promise<string[]>;
|
||||||
@@ -637,6 +703,11 @@ declare class MetonaSqlark {
|
|||||||
importTable(tableName: string, data: Record<string, unknown>[]): Promise<string[]>;
|
importTable(tableName: string, data: Record<string, unknown>[]): Promise<string[]>;
|
||||||
/** 导出整个数据库为 JSON */
|
/** 导出整个数据库为 JSON */
|
||||||
exportAll(): Promise<Record<string, Record<string, unknown>[]>>;
|
exportAll(): Promise<Record<string, Record<string, unknown>[]>>;
|
||||||
|
/**
|
||||||
|
* v0.5.1: 在线备份 — 导出全库一致性快照。
|
||||||
|
* Aria 引擎走引擎级 backup()(MVCC 一致性视图);其余引擎回退 exportAll()。
|
||||||
|
*/
|
||||||
|
backup(): Promise<Record<string, Record<string, unknown>[]>>;
|
||||||
private listeners;
|
private listeners;
|
||||||
/** 订阅表变更 */
|
/** 订阅表变更 */
|
||||||
subscribe(tableName: string, callback: (event: {
|
subscribe(tableName: string, callback: (event: {
|
||||||
@@ -654,6 +725,12 @@ declare class MetonaSqlark {
|
|||||||
broadcastChange(tableName: string): void;
|
broadcastChange(tableName: string): void;
|
||||||
/** 写语句对应的表名(多标签页广播用) */
|
/** 写语句对应的表名(多标签页广播用) */
|
||||||
private writeStatementTable;
|
private writeStatementTable;
|
||||||
|
/**
|
||||||
|
* v0.5.1: SQL 写语句触发 CRUD 生命周期钩子。
|
||||||
|
* INSERT/UPDATE/DELETE 分别触发 beforeInsert/afterInsert、beforeUpdate/afterUpdate、
|
||||||
|
* beforeDelete/afterDelete(参数与 Table API 路径一致)。
|
||||||
|
*/
|
||||||
|
private triggerStatementHooks;
|
||||||
private migrations;
|
private migrations;
|
||||||
/** 注册迁移 */
|
/** 注册迁移 */
|
||||||
addMigration(version: number, up: (db: MetonaSqlark) => Promise<void>): void;
|
addMigration(version: number, up: (db: MetonaSqlark) => Promise<void>): void;
|
||||||
@@ -901,35 +978,6 @@ declare class OPFSEngine implements IStorageEngine {
|
|||||||
* 空表不再消失、索引标记/主键/约束完整;无 schema 记录的旧库从数据推断(兼容)。
|
* 空表不再消失、索引标记/主键/约束完整;无 schema 记录的旧库从数据推断(兼容)。
|
||||||
*/
|
*/
|
||||||
private loadExistingTables;
|
private loadExistingTables;
|
||||||
/** 从 OPFS 加载表数据到内存缓存 */
|
|
||||||
loadTableIntoMemory(tableName: string, schema: TableSchema): Promise<void>;
|
|
||||||
}
|
|
||||||
|
|
||||||
interface AriaEngineConfig {
|
|
||||||
/** 页面大小(默认 4096) */
|
|
||||||
pageSize?: number;
|
|
||||||
/** Buffer Pool 页面数量(默认 256) */
|
|
||||||
bufferPoolPages?: number;
|
|
||||||
/** MemTable 刷盘阈值(默认 4MB) */
|
|
||||||
memtableSizeThreshold?: number;
|
|
||||||
/** LSM 层级之间的容量倍数(默认 10) */
|
|
||||||
levelSizeMultiplier?: number;
|
|
||||||
/** Bloom Filter 每 key 位数(默认 10) */
|
|
||||||
bloomFilterBitsPerKey?: number;
|
|
||||||
/** 是否启用 WAL(默认 true) */
|
|
||||||
walEnabled?: boolean;
|
|
||||||
/** WAL 同步模式 */
|
|
||||||
walSyncMode?: 'full' | 'batch' | 'none';
|
|
||||||
/** Checkpoint 间隔(操作数,默认 1000) */
|
|
||||||
checkpointInterval?: number;
|
|
||||||
/** 是否启用页面压缩(默认 false) */
|
|
||||||
compression?: boolean;
|
|
||||||
/** 存储后端 */
|
|
||||||
storageBackend?: 'indexeddb' | 'opfs' | 'memory';
|
|
||||||
/** WAL 大小阈值(字节,超过则强制 checkpoint,默认 16MB) */
|
|
||||||
walSizeThreshold?: number;
|
|
||||||
/** 最大内存预算(MB,默认 64) */
|
|
||||||
maxMemoryMB?: number;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
declare class AriaEngine implements IStorageEngine {
|
declare class AriaEngine implements IStorageEngine {
|
||||||
@@ -941,6 +989,9 @@ declare class AriaEngine implements IStorageEngine {
|
|||||||
private backend;
|
private backend;
|
||||||
private opened;
|
private opened;
|
||||||
private dbName;
|
private dbName;
|
||||||
|
private fileManager;
|
||||||
|
private bufferPool;
|
||||||
|
private dbLock;
|
||||||
private schemas;
|
private schemas;
|
||||||
private tablePKs;
|
private tablePKs;
|
||||||
private opCounter;
|
private opCounter;
|
||||||
@@ -949,7 +1000,6 @@ declare class AriaEngine implements IStorageEngine {
|
|||||||
private currentTxnId;
|
private currentTxnId;
|
||||||
private txnSnapshot;
|
private txnSnapshot;
|
||||||
private gcCounter;
|
private gcCounter;
|
||||||
private bufferPool;
|
|
||||||
constructor(config?: AriaEngineConfig);
|
constructor(config?: AriaEngineConfig);
|
||||||
open(dbName: string, _version: number): Promise<void>;
|
open(dbName: string, _version: number): Promise<void>;
|
||||||
/** open 内部实现(错误包装在 open 外层) */
|
/** open 内部实现(错误包装在 open 外层) */
|
||||||
@@ -957,9 +1007,15 @@ declare class AriaEngine implements IStorageEngine {
|
|||||||
close(): Promise<void>;
|
close(): Promise<void>;
|
||||||
/**
|
/**
|
||||||
* v0.4.2-fix: 崩溃恢复/自愈 — 校验并移除损坏 SSTable、截断 WAL、重建二级索引。
|
* v0.4.2-fix: 崩溃恢复/自愈 — 校验并移除损坏 SSTable、截断 WAL、重建二级索引。
|
||||||
|
* v0.4.5 增强:清理 OPFS 残留临时文件、清理孤儿页面(meta 未引用的 pg_ 文件)。
|
||||||
* 应用层检测到异常后调用,无需删库重建。
|
* 应用层检测到异常后调用,无需删库重建。
|
||||||
*/
|
*/
|
||||||
repair(): Promise<void>;
|
repair(): Promise<void>;
|
||||||
|
/**
|
||||||
|
* v0.4.5: 清理孤儿页面 — 扫描全部 pg_* 文件,未被任何 LSM 命名空间 meta 引用的删除。
|
||||||
|
* 孤儿页面来自:崩溃中断的 compaction/删除流程(旧 SSTable 页面残留)。
|
||||||
|
*/
|
||||||
|
private cleanupOrphanPages;
|
||||||
/**
|
/**
|
||||||
* v0.4.1: 重置数据库 — 清空全部数据与表结构(演示页刷新/重新初始化用)。
|
* v0.4.1: 重置数据库 — 清空全部数据与表结构(演示页刷新/重新初始化用)。
|
||||||
* 清空存储后端、LSM、WAL、MVCC 与二级索引,后续可继续使用本实例。
|
* 清空存储后端、LSM、WAL、MVCC 与二级索引,后续可继续使用本实例。
|
||||||
@@ -1044,6 +1100,8 @@ declare class AriaEngine implements IStorageEngine {
|
|||||||
* - id 序列独立(避免 v0.2.4 共享 id 空间导致的文件互相覆盖)
|
* - id 序列独立(避免 v0.2.4 共享 id 空间导致的文件互相覆盖)
|
||||||
*/
|
*/
|
||||||
private createSSTableStore;
|
private createSSTableStore;
|
||||||
|
/** v0.4.5: 是否启用页面化物理存储(默认 OPFS 后端启用,显式配置可覆盖) */
|
||||||
|
private isPageStorage;
|
||||||
private applyWALRecord;
|
private applyWALRecord;
|
||||||
/**
|
/**
|
||||||
* v0.3.3: DROP_TABLE 恢复 — 删除 schema 并清除主 LSM 中该表的所有残留数据。
|
* v0.3.3: DROP_TABLE 恢复 — 删除 schema 并清除主 LSM 中该表的所有残留数据。
|
||||||
@@ -1084,21 +1142,10 @@ declare class AriaEngine implements IStorageEngine {
|
|||||||
compactedLevels: number;
|
compactedLevels: number;
|
||||||
gcVersions: number;
|
gcVersions: number;
|
||||||
}>;
|
}>;
|
||||||
/**
|
|
||||||
* 查询优化器:估算各索引成本,选择最优方案
|
|
||||||
*/
|
|
||||||
estimateQueryCost(tableName: string, query: QueryPlan): {
|
|
||||||
strategy: string;
|
|
||||||
estimatedRows: number;
|
|
||||||
};
|
|
||||||
private ensureOpen;
|
private ensureOpen;
|
||||||
/** v0.4.2-fix: Aria 事务中 DDL 显式拒绝(结构变更无法通过行快照回滚) */
|
/** v0.4.2-fix: Aria 事务中 DDL 显式拒绝(结构变更无法通过行快照回滚) */
|
||||||
private ensureNoDDLInTransaction;
|
private ensureNoDDLInTransaction;
|
||||||
private ensureTable;
|
private ensureTable;
|
||||||
/** Get the number of WAL records stored */
|
|
||||||
private getWALCount;
|
|
||||||
/** Set the number of WAL records stored */
|
|
||||||
private setWALCount;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
declare class HybridEngine implements IStorageEngine {
|
declare class HybridEngine implements IStorageEngine {
|
||||||
@@ -1234,6 +1281,13 @@ declare enum TokenType {
|
|||||||
THEN = "THEN",
|
THEN = "THEN",
|
||||||
ELSE = "ELSE",
|
ELSE = "ELSE",
|
||||||
END = "END",
|
END = "END",
|
||||||
|
EXPLAIN = "EXPLAIN",
|
||||||
|
ANALYZE = "ANALYZE",
|
||||||
|
REINDEX = "REINDEX",
|
||||||
|
VACUUM = "VACUUM",
|
||||||
|
SAVEPOINT = "SAVEPOINT",
|
||||||
|
RELEASE = "RELEASE",
|
||||||
|
TO = "TO",
|
||||||
IDENTIFIER = "IDENTIFIER",
|
IDENTIFIER = "IDENTIFIER",
|
||||||
STRING = "STRING",
|
STRING = "STRING",
|
||||||
NUMBER = "NUMBER",
|
NUMBER = "NUMBER",
|
||||||
@@ -1286,6 +1340,13 @@ interface IStorageBackend {
|
|||||||
read(key: string): Promise<ArrayBuffer | null>;
|
read(key: string): Promise<ArrayBuffer | null>;
|
||||||
/** 写入数据块 */
|
/** 写入数据块 */
|
||||||
write(key: string, data: ArrayBuffer): Promise<void>;
|
write(key: string, data: ArrayBuffer): Promise<void>;
|
||||||
|
/**
|
||||||
|
* 追加写入(v0.4.5 WAL 分片用,可选):
|
||||||
|
* - OPFS 后端实现真追加(createWritable keepExistingData + seek,O(chunk))
|
||||||
|
* - 未实现的后端由调用方回退 read+write(EncryptedBackend 包装时整体重写保正确性)
|
||||||
|
* 语义:在 key 现有内容末尾追加 data;key 不存在时等同 write。
|
||||||
|
*/
|
||||||
|
append?(key: string, data: ArrayBuffer): Promise<void>;
|
||||||
/**
|
/**
|
||||||
* 批量原子写入(v0.4.2-fix):多个 key 在单个底层事务中提交,
|
* 批量原子写入(v0.4.2-fix):多个 key 在单个底层事务中提交,
|
||||||
* 中断时整体回滚,不留半写状态。WAL count 与记录同事务保证一致性。
|
* 中断时整体回滚,不留半写状态。WAL count 与记录同事务保证一致性。
|
||||||
@@ -1312,7 +1373,17 @@ interface IStorageBackend {
|
|||||||
* 零外部依赖,纯浏览器文件系统 API。
|
* 零外部依赖,纯浏览器文件系统 API。
|
||||||
* 每个 key 对应 OPFS 目录下的一个二进制文件。
|
* 每个 key 对应 OPFS 目录下的一个二进制文件。
|
||||||
*
|
*
|
||||||
* 浏览器要求:Chrome 102+ / Edge 102+
|
* 原子性与一致性保证(v0.4.5 固化):
|
||||||
|
* - 单文件 write/append:createWritable 为 copy-on-write —— close 前崩溃旧文件保持不变,
|
||||||
|
* close 后原子替换(单文件写入原子)
|
||||||
|
* - 多文件 writeMany/deleteMany:OPFS 无跨文件事务,串行逐个落盘;调用方(WAL 分片)
|
||||||
|
* 已改为单文件语义,多键操作仅用于一次性的 schema/meta 写入
|
||||||
|
* - 所有写操作串行队列化(同源同进程顺序一致);单次任务失败不中断队列链,
|
||||||
|
* 错误如实返回给该次调用的调用方
|
||||||
|
* - close() 等待写队列排空后再释放目录句柄(杜绝 close 后挂起写丢失/读旧数据)
|
||||||
|
* - open() 自动清理崩溃残留临时文件(Chromium createWritable 的 .crswap 等)
|
||||||
|
*
|
||||||
|
* 浏览器要求:Chrome 102+ / Edge 102+(Safari 15.2+ / Firefox 111+ 支持基础 OPFS)
|
||||||
*/
|
*/
|
||||||
|
|
||||||
declare class OPFSBackend implements IStorageBackend {
|
declare class OPFSBackend implements IStorageBackend {
|
||||||
@@ -1323,8 +1394,20 @@ declare class OPFSBackend implements IStorageBackend {
|
|||||||
open(name: string): Promise<void>;
|
open(name: string): Promise<void>;
|
||||||
close(): Promise<void>;
|
close(): Promise<void>;
|
||||||
isOpen(): boolean;
|
isOpen(): boolean;
|
||||||
|
/** 清理崩溃残留的临时文件(open 时自动调用,repair 也可调用) */
|
||||||
|
cleanupStaleFiles(): Promise<void>;
|
||||||
read(key: string): Promise<ArrayBuffer | null>;
|
read(key: string): Promise<ArrayBuffer | null>;
|
||||||
|
/**
|
||||||
|
* 单文件原子写:createWritable 为 copy-on-write,close 后原子替换;
|
||||||
|
* 写入期间崩溃 → 旧文件保持(原子性由浏览器 OPFS 实现保证)。
|
||||||
|
*/
|
||||||
write(key: string, data: ArrayBuffer): Promise<void>;
|
write(key: string, data: ArrayBuffer): Promise<void>;
|
||||||
|
/**
|
||||||
|
* v0.4.5: 真追加写 — createWritable(keepExistingData) + seek 到文件末尾。
|
||||||
|
* 单文件 COW 原子(close 前崩溃旧文件保持),无需读旧内容即实现 O(chunk) 追加
|
||||||
|
* (WAL 分片高频写入用)。
|
||||||
|
*/
|
||||||
|
append(key: string, data: ArrayBuffer): Promise<void>;
|
||||||
/** v0.4.2-fix: 批量写入 — 串行队列内逐个落盘(OPFS 无跨文件事务,顺序保证一致) */
|
/** v0.4.2-fix: 批量写入 — 串行队列内逐个落盘(OPFS 无跨文件事务,顺序保证一致) */
|
||||||
writeMany(entries: Record<string, ArrayBuffer>): Promise<void>;
|
writeMany(entries: Record<string, ArrayBuffer>): Promise<void>;
|
||||||
delete(key: string): Promise<void>;
|
delete(key: string): Promise<void>;
|
||||||
|
|||||||
Vendored
+1262
-522
File diff suppressed because it is too large
Load Diff
Vendored
+1
-1
File diff suppressed because one or more lines are too long
Vendored
+1262
-522
File diff suppressed because it is too large
Load Diff
Vendored
+1
-1
File diff suppressed because one or more lines are too long
Vendored
+1
-1
File diff suppressed because one or more lines are too long
@@ -8,6 +8,12 @@ module.exports = {
|
|||||||
'/node_modules/(?!(@rollup)/)',
|
'/node_modules/(?!(@rollup)/)',
|
||||||
],
|
],
|
||||||
moduleFileExtensions: ['ts', 'js', 'json'],
|
moduleFileExtensions: ['ts', 'js', 'json'],
|
||||||
|
// Playwright e2e(tests/e2e/)与共享测试工具(tests/helpers/)不属于 jest 单元测试
|
||||||
|
testPathIgnorePatterns: [
|
||||||
|
'/node_modules/',
|
||||||
|
'/tests/e2e/',
|
||||||
|
'/tests/helpers/',
|
||||||
|
],
|
||||||
collectCoverageFrom: [
|
collectCoverageFrom: [
|
||||||
'src/**/*.ts',
|
'src/**/*.ts',
|
||||||
'!src/index.ts',
|
'!src/index.ts',
|
||||||
|
|||||||
Generated
+67
-2
@@ -1,18 +1,19 @@
|
|||||||
{
|
{
|
||||||
"name": "@metona-team/metona-sqlark",
|
"name": "@metona-team/metona-sqlark",
|
||||||
"version": "0.4.1",
|
"version": "0.4.4",
|
||||||
"lockfileVersion": 3,
|
"lockfileVersion": 3,
|
||||||
"requires": true,
|
"requires": true,
|
||||||
"packages": {
|
"packages": {
|
||||||
"": {
|
"": {
|
||||||
"name": "@metona-team/metona-sqlark",
|
"name": "@metona-team/metona-sqlark",
|
||||||
"version": "0.4.1",
|
"version": "0.4.4",
|
||||||
"license": "MIT",
|
"license": "MIT",
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
"@babel/core": "^7.22.0",
|
"@babel/core": "^7.22.0",
|
||||||
"@babel/plugin-transform-modules-commonjs": "^7.22.0",
|
"@babel/plugin-transform-modules-commonjs": "^7.22.0",
|
||||||
"@babel/preset-env": "^7.22.0",
|
"@babel/preset-env": "^7.22.0",
|
||||||
"@babel/preset-typescript": "^7.22.0",
|
"@babel/preset-typescript": "^7.22.0",
|
||||||
|
"@playwright/test": "^1.62.1",
|
||||||
"@rollup/plugin-commonjs": "^25.0.0",
|
"@rollup/plugin-commonjs": "^25.0.0",
|
||||||
"@rollup/plugin-node-resolve": "^15.0.0",
|
"@rollup/plugin-node-resolve": "^15.0.0",
|
||||||
"@rollup/plugin-terser": "^0.4.0",
|
"@rollup/plugin-terser": "^0.4.0",
|
||||||
@@ -25,6 +26,7 @@
|
|||||||
"fake-indexeddb": "^6.2.5",
|
"fake-indexeddb": "^6.2.5",
|
||||||
"jest": "^29.5.0",
|
"jest": "^29.5.0",
|
||||||
"jest-environment-jsdom": "^29.5.0",
|
"jest-environment-jsdom": "^29.5.0",
|
||||||
|
"playwright": "^1.62.1",
|
||||||
"prettier": "^2.8.0",
|
"prettier": "^2.8.0",
|
||||||
"rollup": "^3.20.0",
|
"rollup": "^3.20.0",
|
||||||
"rollup-plugin-dts": "^5.3.0",
|
"rollup-plugin-dts": "^5.3.0",
|
||||||
@@ -2543,6 +2545,22 @@
|
|||||||
"node": ">= 8"
|
"node": ">= 8"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"node_modules/@playwright/test": {
|
||||||
|
"version": "1.62.1",
|
||||||
|
"resolved": "https://registry.npmmirror.com/@playwright/test/-/test-1.62.1.tgz",
|
||||||
|
"integrity": "sha512-DTcUc8qii+cpHvtOwggMtBRMjKZHXYWdw8syRYu2vtzuq4Wxphqq4NfCs5Zt44L6mA8rfDfj+PHnxFc/FeK6mQ==",
|
||||||
|
"dev": true,
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"dependencies": {
|
||||||
|
"playwright": "1.62.1"
|
||||||
|
},
|
||||||
|
"bin": {
|
||||||
|
"playwright": "cli.js"
|
||||||
|
},
|
||||||
|
"engines": {
|
||||||
|
"node": ">=20"
|
||||||
|
}
|
||||||
|
},
|
||||||
"node_modules/@rollup/plugin-commonjs": {
|
"node_modules/@rollup/plugin-commonjs": {
|
||||||
"version": "25.0.8",
|
"version": "25.0.8",
|
||||||
"resolved": "https://registry.npmmirror.com/@rollup/plugin-commonjs/-/plugin-commonjs-25.0.8.tgz",
|
"resolved": "https://registry.npmmirror.com/@rollup/plugin-commonjs/-/plugin-commonjs-25.0.8.tgz",
|
||||||
@@ -6750,6 +6768,53 @@
|
|||||||
"node": ">=8"
|
"node": ">=8"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"node_modules/playwright": {
|
||||||
|
"version": "1.62.1",
|
||||||
|
"resolved": "https://registry.npmmirror.com/playwright/-/playwright-1.62.1.tgz",
|
||||||
|
"integrity": "sha512-0M+L3LAD8/nm554LOla9Ayx0j0tmFZ0FBcoQ7F1VuVHpM/XpiC8RcDzBQB8W5+hA8L22THxELzeF+2WcUzvcLg==",
|
||||||
|
"dev": true,
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"dependencies": {
|
||||||
|
"playwright-core": "1.62.1"
|
||||||
|
},
|
||||||
|
"bin": {
|
||||||
|
"playwright": "cli.js"
|
||||||
|
},
|
||||||
|
"engines": {
|
||||||
|
"node": ">=20"
|
||||||
|
},
|
||||||
|
"optionalDependencies": {
|
||||||
|
"fsevents": "2.3.2"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/playwright-core": {
|
||||||
|
"version": "1.62.1",
|
||||||
|
"resolved": "https://registry.npmmirror.com/playwright-core/-/playwright-core-1.62.1.tgz",
|
||||||
|
"integrity": "sha512-wPYSwEBJY9GHraISXqyqtx0na0LpO3XEX7jNDhntbex7tzUS7kLnZsOlFruFJB4Hi/rhDMjXGqHewDZ68nYZVw==",
|
||||||
|
"dev": true,
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"bin": {
|
||||||
|
"playwright-core": "cli.js"
|
||||||
|
},
|
||||||
|
"engines": {
|
||||||
|
"node": ">=20"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/playwright/node_modules/fsevents": {
|
||||||
|
"version": "2.3.2",
|
||||||
|
"resolved": "https://registry.npmmirror.com/fsevents/-/fsevents-2.3.2.tgz",
|
||||||
|
"integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==",
|
||||||
|
"dev": true,
|
||||||
|
"hasInstallScript": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"optional": true,
|
||||||
|
"os": [
|
||||||
|
"darwin"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": "^8.16.0 || ^10.6.0 || >=11.0.0"
|
||||||
|
}
|
||||||
|
},
|
||||||
"node_modules/prelude-ls": {
|
"node_modules/prelude-ls": {
|
||||||
"version": "1.2.1",
|
"version": "1.2.1",
|
||||||
"resolved": "https://registry.npmmirror.com/prelude-ls/-/prelude-ls-1.2.1.tgz",
|
"resolved": "https://registry.npmmirror.com/prelude-ls/-/prelude-ls-1.2.1.tgz",
|
||||||
|
|||||||
+4
-1
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@metona-team/metona-sqlark",
|
"name": "@metona-team/metona-sqlark",
|
||||||
"version": "0.4.4",
|
"version": "0.5.1",
|
||||||
"description": "Frontend SQL database with in-memory and disk dual-mode storage",
|
"description": "Frontend SQL database with in-memory and disk dual-mode storage",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "dist/metona-sqlark.cjs",
|
"main": "dist/metona-sqlark.cjs",
|
||||||
@@ -35,6 +35,7 @@
|
|||||||
"test": "jest",
|
"test": "jest",
|
||||||
"test:coverage": "jest --coverage",
|
"test:coverage": "jest --coverage",
|
||||||
"test:watch": "jest --watch",
|
"test:watch": "jest --watch",
|
||||||
|
"test:e2e": "playwright test",
|
||||||
"lint": "eslint \"src/**/*.ts\"",
|
"lint": "eslint \"src/**/*.ts\"",
|
||||||
"lint:fix": "eslint \"src/**/*.ts\" --fix",
|
"lint:fix": "eslint \"src/**/*.ts\" --fix",
|
||||||
"format": "prettier --write \"src/**/*.ts\"",
|
"format": "prettier --write \"src/**/*.ts\"",
|
||||||
@@ -66,6 +67,7 @@
|
|||||||
"@babel/plugin-transform-modules-commonjs": "^7.22.0",
|
"@babel/plugin-transform-modules-commonjs": "^7.22.0",
|
||||||
"@babel/preset-env": "^7.22.0",
|
"@babel/preset-env": "^7.22.0",
|
||||||
"@babel/preset-typescript": "^7.22.0",
|
"@babel/preset-typescript": "^7.22.0",
|
||||||
|
"@playwright/test": "^1.62.1",
|
||||||
"@rollup/plugin-commonjs": "^25.0.0",
|
"@rollup/plugin-commonjs": "^25.0.0",
|
||||||
"@rollup/plugin-node-resolve": "^15.0.0",
|
"@rollup/plugin-node-resolve": "^15.0.0",
|
||||||
"@rollup/plugin-terser": "^0.4.0",
|
"@rollup/plugin-terser": "^0.4.0",
|
||||||
@@ -78,6 +80,7 @@
|
|||||||
"fake-indexeddb": "^6.2.5",
|
"fake-indexeddb": "^6.2.5",
|
||||||
"jest": "^29.5.0",
|
"jest": "^29.5.0",
|
||||||
"jest-environment-jsdom": "^29.5.0",
|
"jest-environment-jsdom": "^29.5.0",
|
||||||
|
"playwright": "^1.62.1",
|
||||||
"prettier": "^2.8.0",
|
"prettier": "^2.8.0",
|
||||||
"rollup": "^3.20.0",
|
"rollup": "^3.20.0",
|
||||||
"rollup-plugin-dts": "^5.3.0",
|
"rollup-plugin-dts": "^5.3.0",
|
||||||
|
|||||||
@@ -0,0 +1,24 @@
|
|||||||
|
/**
|
||||||
|
* Playwright E2E 配置 — 真实 Chromium 环境验证 OPFS 存储后端
|
||||||
|
*/
|
||||||
|
import { defineConfig } from '@playwright/test';
|
||||||
|
|
||||||
|
export default defineConfig({
|
||||||
|
testDir: './tests/e2e',
|
||||||
|
timeout: 60_000,
|
||||||
|
fullyParallel: false,
|
||||||
|
workers: 1, // OPFS 有锁/配额,串行执行避免互相干扰
|
||||||
|
retries: 1,
|
||||||
|
reporter: [['list']],
|
||||||
|
use: {
|
||||||
|
baseURL: 'http://127.0.0.1:3344',
|
||||||
|
trace: 'retain-on-failure',
|
||||||
|
},
|
||||||
|
webServer: {
|
||||||
|
// Node 静态服务(与 MetonaEditor 一致,不依赖 python3)
|
||||||
|
command: 'node tests/e2e/server.cjs 3344',
|
||||||
|
url: 'http://127.0.0.1:3344',
|
||||||
|
reuseExistingServer: !process.env.CI,
|
||||||
|
timeout: 30_000,
|
||||||
|
},
|
||||||
|
});
|
||||||
+61
-33
@@ -79,14 +79,14 @@
|
|||||||
<div class="container">
|
<div class="container">
|
||||||
<div class="hero">
|
<div class="hero">
|
||||||
<h1>⚡ 性能基准测试</h1>
|
<h1>⚡ 性能基准测试</h1>
|
||||||
<p>在浏览器中实测 <code>Memory</code> 与 <code>Aria</code>(LSM-Tree 自研引擎)两种模式的真实性能。
|
<p>在浏览器中实测 <code>Memory</code> 与 <code>Aria</code>(LSM-Tree 自研引擎)在纯内存 / IndexedDB / OPFS(页面化 4KB 存储)三种存储后端下的真实性能。
|
||||||
数据量分三档(1K / 10K / 50K 行),测量 <code>INSERT</code> / 主键查询 / 索引查询 / <code>UPDATE</code> / <code>DELETE</code> 的吞吐(ops/sec)。
|
数据量分档(1K / 10K 行),测量 <code>INSERT</code> / 主键查询 / 索引查询 / <code>UPDATE</code> / <code>DELETE</code> 的吞吐(ops/sec)。
|
||||||
<strong>测试结果取决于你的设备与浏览器</strong>,仅供参考。</p>
|
<strong>测试结果取决于你的设备与浏览器</strong>,仅供参考。磁盘后端结果反映真实持久化成本(WAL full 同步模式)。</p>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div class="controls">
|
<div class="controls">
|
||||||
<button class="btn btn-run" id="run-btn" onclick="runAll()">▶ 运行完整基准</button>
|
<button class="btn btn-run" id="run-btn" onclick="runAll()">▶ 运行完整基准</button>
|
||||||
<button class="btn btn-small" onclick="runAll(true)">⚡ 快速模式(1K/10K)</button>
|
<button class="btn btn-small" onclick="runAll(true)">⚡ 快速模式(仅 1K)</button>
|
||||||
<button class="btn btn-small" onclick="window.location.reload()">↻ 重置</button>
|
<button class="btn btn-small" onclick="window.location.reload()">↻ 重置</button>
|
||||||
<span class="status" id="status"></span>
|
<span class="status" id="status"></span>
|
||||||
</div>
|
</div>
|
||||||
@@ -99,7 +99,9 @@
|
|||||||
<thead><tr><th>引擎</th><th>数据量</th><th>INSERT</th><th>主键查询</th><th>索引查询</th><th>UPDATE</th><th>DELETE</th></tr></thead>
|
<thead><tr><th>引擎</th><th>数据量</th><th>INSERT</th><th>主键查询</th><th>索引查询</th><th>UPDATE</th><th>DELETE</th></tr></thead>
|
||||||
<tbody></tbody>
|
<tbody></tbody>
|
||||||
</table>
|
</table>
|
||||||
<p class="note">注:Memory 模式为纯内存 Map 存储;Aria 模式为 LSM-Tree + MemTable 存储(<code>storageBackend: 'memory'</code>,无磁盘 I/O)。
|
<p class="note">注:Memory = 纯内存 Map 引擎;Aria-memory = LSM-Tree(内存后端,无磁盘 I/O);
|
||||||
|
Aria-IndexedDB = LSM-Tree(IDB 后端,整 value SSTable);Aria-OPFS = LSM-Tree(OPFS 后端,<strong>4KB 页面化存储</strong> + BufferPool LRU 缓存 + WAL 分片真追加)。
|
||||||
|
磁盘后端使用 <code>walSyncMode: 'full'</code>(每条写入同步落盘,最安全的崩溃一致性)。
|
||||||
INSERT 为单行批量写入(含 WAL 记录);UPDATE/DELETE 按主键条件执行。</p>
|
INSERT 为单行批量写入(含 WAL 记录);UPDATE/DELETE 按主键条件执行。</p>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
@@ -107,8 +109,16 @@
|
|||||||
<script src="../dist/metona-sqlark.js"></script>
|
<script src="../dist/metona-sqlark.js"></script>
|
||||||
<script>
|
<script>
|
||||||
const M = window.MetonaSqlark;
|
const M = window.MetonaSqlark;
|
||||||
const SIZES = [1000, 10000, 50000];
|
const SIZES = [1000, 10000];
|
||||||
const SIZES_FAST = [1000, 10000];
|
const SIZES_FAST = [1000];
|
||||||
|
|
||||||
|
// [label, mode, diskEngine]
|
||||||
|
const ENGINES = [
|
||||||
|
['memory', 'memory', 'memory'],
|
||||||
|
['aria-memory', 'aria', 'memory'],
|
||||||
|
['aria-indexeddb', 'aria', 'indexeddb'],
|
||||||
|
['aria-opfs', 'aria', 'opfs'],
|
||||||
|
];
|
||||||
|
|
||||||
let cancelled = false;
|
let cancelled = false;
|
||||||
|
|
||||||
@@ -128,13 +138,25 @@ function makeRows(n) {
|
|||||||
return rows;
|
return rows;
|
||||||
}
|
}
|
||||||
|
|
||||||
async function createDb(mode) {
|
async function createDb(mode, diskEngine) {
|
||||||
const db = new M.MetonaSqlark({
|
if (mode === 'memory') {
|
||||||
name: 'bench-' + mode + '-' + Date.now(),
|
const db = new M.MetonaSqlark({ name: 'bench-mem-' + Date.now(), mode: 'memory' });
|
||||||
mode,
|
await db.init();
|
||||||
diskEngine: 'memory',
|
await db.defineTable('bench', {
|
||||||
|
id: { type: 'string', primaryKey: true },
|
||||||
|
name: { type: 'string' },
|
||||||
|
age: { type: 'number', index: true },
|
||||||
|
city: { type: 'string' },
|
||||||
|
});
|
||||||
|
return db;
|
||||||
|
}
|
||||||
|
// aria 模式:磁盘后端走真实持久化(WAL full 同步)
|
||||||
|
const db = await M.create({
|
||||||
|
name: 'bench-aria-' + diskEngine + '-' + Date.now(),
|
||||||
|
mode: 'aria',
|
||||||
|
diskEngine,
|
||||||
|
aria: { walSyncMode: 'full', checkpointInterval: 100000, memtableSizeThreshold: 64 * 1024 * 1024 },
|
||||||
});
|
});
|
||||||
await db.init();
|
|
||||||
await db.defineTable('bench', {
|
await db.defineTable('bench', {
|
||||||
id: { type: 'string', primaryKey: true },
|
id: { type: 'string', primaryKey: true },
|
||||||
name: { type: 'string' },
|
name: { type: 'string' },
|
||||||
@@ -150,39 +172,39 @@ async function timeIt(fn) {
|
|||||||
return performance.now() - start;
|
return performance.now() - start;
|
||||||
}
|
}
|
||||||
|
|
||||||
async function runBench(mode, size) {
|
async function runBench(label, mode, diskEngine, size) {
|
||||||
const db = await createDb(mode);
|
const db = await createDb(mode, diskEngine);
|
||||||
const rows = makeRows(size);
|
const rows = makeRows(size);
|
||||||
const result = { mode, size, insert: 0, pk: 0, idx: 0, update: 0, del: 0 };
|
const result = { mode: label, size, insert: 0, pk: 0, idx: 0, update: 0, del: 0 };
|
||||||
|
|
||||||
try {
|
try {
|
||||||
// INSERT(批量)
|
// INSERT(批量)
|
||||||
let ms = await timeIt(() => db.table('bench').insertMany(rows));
|
let ms = await timeIt(() => db.table('bench').insertMany(rows));
|
||||||
result.insert = Math.round(size / (ms / 1000));
|
result.insert = Math.round(size / (ms / 1000));
|
||||||
|
|
||||||
// 主键查询 × 500
|
// 主键查询 × 300
|
||||||
ms = await timeIt(async () => {
|
ms = await timeIt(async () => {
|
||||||
for (let i = 0; i < 500; i++) await db.query(`SELECT * FROM bench WHERE id = 'k${(i * 37) % size}'`);
|
for (let i = 0; i < 300; i++) await db.query(`SELECT * FROM bench WHERE id = 'k${(i * 37) % size}'`);
|
||||||
});
|
});
|
||||||
result.pk = Math.round(500 / (ms / 1000));
|
result.pk = Math.round(300 / (ms / 1000));
|
||||||
|
|
||||||
// 索引查询 × 500
|
// 索引查询 × 300
|
||||||
ms = await timeIt(async () => {
|
ms = await timeIt(async () => {
|
||||||
for (let i = 0; i < 500; i++) await db.query(`SELECT * FROM bench WHERE age = ${18 + ((i * 17) % 60)}`);
|
for (let i = 0; i < 300; i++) await db.query(`SELECT * FROM bench WHERE age = ${18 + ((i * 17) % 60)}`);
|
||||||
});
|
});
|
||||||
result.idx = Math.round(500 / (ms / 1000));
|
result.idx = Math.round(300 / (ms / 1000));
|
||||||
|
|
||||||
// UPDATE × 200(按主键)
|
// UPDATE × 150(按主键)
|
||||||
ms = await timeIt(async () => {
|
ms = await timeIt(async () => {
|
||||||
for (let i = 0; i < 200; i++) await db.query(`UPDATE bench SET name = 'X${i}' WHERE id = 'k${(i * 53) % size}'`);
|
for (let i = 0; i < 150; i++) await db.query(`UPDATE bench SET name = 'X${i}' WHERE id = 'k${(i * 53) % size}'`);
|
||||||
});
|
});
|
||||||
result.update = Math.round(200 / (ms / 1000));
|
result.update = Math.round(150 / (ms / 1000));
|
||||||
|
|
||||||
// DELETE × 200(按主键)
|
// DELETE × 150(按主键)
|
||||||
ms = await timeIt(async () => {
|
ms = await timeIt(async () => {
|
||||||
for (let i = 0; i < 200; i++) await db.query(`DELETE FROM bench WHERE id = 'k${(i * 71) % size}'`);
|
for (let i = 0; i < 150; i++) await db.query(`DELETE FROM bench WHERE id = 'k${(i * 71) % size}'`);
|
||||||
});
|
});
|
||||||
result.del = Math.round(200 / (ms / 1000));
|
result.del = Math.round(150 / (ms / 1000));
|
||||||
} finally {
|
} finally {
|
||||||
await db.close();
|
await db.close();
|
||||||
}
|
}
|
||||||
@@ -199,16 +221,16 @@ async function runAll(quick) {
|
|||||||
document.getElementById('summary').innerHTML = '';
|
document.getElementById('summary').innerHTML = '';
|
||||||
|
|
||||||
const results = [];
|
const results = [];
|
||||||
for (const mode of ['memory', 'aria']) {
|
for (const [label, mode, diskEngine] of ENGINES) {
|
||||||
for (const size of sizes) {
|
for (const size of sizes) {
|
||||||
setStatus(`正在测试 ${mode === 'memory' ? 'Memory' : 'Aria'} 引擎 · ${size.toLocaleString()} 行…`, true);
|
setStatus(`正在测试 ${label} · ${size.toLocaleString()} 行…`, true);
|
||||||
// 让 UI 刷新
|
// 让 UI 刷新
|
||||||
await new Promise((r) => setTimeout(r, 30));
|
await new Promise((r) => setTimeout(r, 30));
|
||||||
try {
|
try {
|
||||||
const r = await runBench(mode, size);
|
const r = await runBench(label, mode, diskEngine, size);
|
||||||
results.push(r);
|
results.push(r);
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
setStatus(`✗ ${mode}/${size} 失败: ${e.message}`, false);
|
setStatus(`✗ ${label}/${size} 失败: ${e.message}`, false);
|
||||||
console.error(e);
|
console.error(e);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -239,13 +261,19 @@ function renderResults(results) {
|
|||||||
`;
|
`;
|
||||||
|
|
||||||
// 表格(按引擎分组)
|
// 表格(按引擎分组)
|
||||||
|
const GROUP_LABEL = {
|
||||||
|
'memory': '🚀 Memory 引擎',
|
||||||
|
'aria-memory': '🌲 Aria 引擎(内存后端)',
|
||||||
|
'aria-indexeddb': '🌲 Aria 引擎(IndexedDB 后端)',
|
||||||
|
'aria-opfs': '🌲 Aria 引擎(OPFS · 4KB 页面化)',
|
||||||
|
};
|
||||||
let lastMode = '';
|
let lastMode = '';
|
||||||
for (const r of results) {
|
for (const r of results) {
|
||||||
if (r.mode !== lastMode) {
|
if (r.mode !== lastMode) {
|
||||||
lastMode = r.mode;
|
lastMode = r.mode;
|
||||||
const row = document.createElement('tr');
|
const row = document.createElement('tr');
|
||||||
row.className = 'group-header';
|
row.className = 'group-header';
|
||||||
row.innerHTML = `<td colspan="6">${r.mode === 'memory' ? '🚀 Memory 引擎' : '🌲 Aria 引擎(LSM-Tree)'}</td>`;
|
row.innerHTML = `<td colspan="6">${GROUP_LABEL[r.mode] || r.mode}</td>`;
|
||||||
tbody.appendChild(row);
|
tbody.appendChild(row);
|
||||||
}
|
}
|
||||||
const row = document.createElement('tr');
|
const row = document.createElement('tr');
|
||||||
|
|||||||
@@ -106,12 +106,6 @@ class ConnectionManager {
|
|||||||
return Array.from(this.connections.keys());
|
return Array.from(this.connections.keys());
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* 获取连接的引用计数
|
|
||||||
*/
|
|
||||||
getRefCount(dbName: string): number {
|
|
||||||
return this.refCount.get(dbName) ?? 0;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|||||||
+8
-8
@@ -13,12 +13,6 @@ export type StorageMode = 'memory' | 'disk' | 'hybrid' | 'aria';
|
|||||||
/** 磁盘引擎类型 */
|
/** 磁盘引擎类型 */
|
||||||
export type DiskEngine = 'indexeddb' | 'opfs';
|
export type DiskEngine = 'indexeddb' | 'opfs';
|
||||||
|
|
||||||
/** 所有存储模式 */
|
|
||||||
export const STORAGE_MODES: StorageMode[] = ['memory', 'disk', 'hybrid', 'aria'];
|
|
||||||
|
|
||||||
/** 所有磁盘引擎 */
|
|
||||||
export const DISK_ENGINES: DiskEngine[] = ['indexeddb', 'opfs'];
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// 字段类型
|
// 字段类型
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -95,10 +89,15 @@ export interface DatabaseConfig {
|
|||||||
debug?: boolean;
|
debug?: boolean;
|
||||||
/** 多标签页同步(v0.3.2):BroadcastChannel 广播表变更,其他标签页自动刷新 */
|
/** 多标签页同步(v0.3.2):BroadcastChannel 广播表变更,其他标签页自动刷新 */
|
||||||
multiTabSync?: boolean;
|
multiTabSync?: boolean;
|
||||||
|
/**
|
||||||
|
* v0.4.5: AriaEngine 专属配置(mode='aria' 时透传):
|
||||||
|
* walSyncMode / checkpointInterval / encryption / pageStorage / compression 等
|
||||||
|
*/
|
||||||
|
aria?: import('./engine/aria/types').AriaEngineConfig;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 数据库默认配置 */
|
/** 数据库默认配置 */
|
||||||
export const DB_DEFAULTS: Readonly<Required<Omit<DatabaseConfig, 'plugins' | 'onReady' | 'onError'>>> = Object.freeze({
|
export const DB_DEFAULTS: Readonly<Required<Omit<DatabaseConfig, 'plugins' | 'onReady' | 'onError' | 'aria'>>> & { aria: undefined } = Object.freeze({
|
||||||
name: 'metona-sqlark',
|
name: 'metona-sqlark',
|
||||||
mode: 'hybrid' as const,
|
mode: 'hybrid' as const,
|
||||||
diskEngine: 'indexeddb' as const,
|
diskEngine: 'indexeddb' as const,
|
||||||
@@ -106,6 +105,7 @@ export const DB_DEFAULTS: Readonly<Required<Omit<DatabaseConfig, 'plugins' | 'on
|
|||||||
maxRowsPerQuery: 0, // 0 = 不限制
|
maxRowsPerQuery: 0, // 0 = 不限制
|
||||||
debug: false,
|
debug: false,
|
||||||
multiTabSync: false,
|
multiTabSync: false,
|
||||||
|
aria: undefined,
|
||||||
});
|
});
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -214,4 +214,4 @@ export class DatabaseError extends Error {
|
|||||||
// 版本
|
// 版本
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
export const VERSION = '0.4.4';
|
export const VERSION = '0.5.1';
|
||||||
|
|||||||
+65
-2
@@ -152,7 +152,14 @@ export class MetonaSqlark {
|
|||||||
let t = this.tableCache.get(name);
|
let t = this.tableCache.get(name);
|
||||||
if (!t) {
|
if (!t) {
|
||||||
// v0.3.2: 表操作写入后广播变更(多标签页同步)
|
// v0.3.2: 表操作写入后广播变更(多标签页同步)
|
||||||
t = new Table(this.engine, name, this.executor, (tableName) => this.broadcastChange(tableName));
|
// v0.5.1: CRUD 生命周期钩子真实接线(beforeInsert/afterInsert/...)
|
||||||
|
t = new Table(
|
||||||
|
this.engine,
|
||||||
|
name,
|
||||||
|
this.executor,
|
||||||
|
(tableName) => this.broadcastChange(tableName),
|
||||||
|
(hook, args) => this.pluginManager.trigger(hook, ...args),
|
||||||
|
);
|
||||||
this.tableCache.set(name, t);
|
this.tableCache.set(name, t);
|
||||||
}
|
}
|
||||||
return t;
|
return t;
|
||||||
@@ -192,7 +199,10 @@ export class MetonaSqlark {
|
|||||||
// v0.3.0: 支持分号分隔的多语句,逐条顺序执行,返回最后一条的结果
|
// v0.3.0: 支持分号分隔的多语句,逐条顺序执行,返回最后一条的结果
|
||||||
const statements: Statement[] = parseAll(sql);
|
const statements: Statement[] = parseAll(sql);
|
||||||
for (const stmt of statements) {
|
for (const stmt of statements) {
|
||||||
|
// v0.5.1: SQL 写语句触发 CRUD 生命周期钩子(与 Table API 路径一致)
|
||||||
|
await this.triggerStatementHooks(stmt, 'before');
|
||||||
result = await this.executor.execute(stmt);
|
result = await this.executor.execute(stmt);
|
||||||
|
await this.triggerStatementHooks(stmt, 'after', result);
|
||||||
// v0.3.2: 写语句广播表变更(多标签页同步)
|
// v0.3.2: 写语句广播表变更(多标签页同步)
|
||||||
const table = this.writeStatementTable(stmt);
|
const table = this.writeStatementTable(stmt);
|
||||||
if (table) this.broadcastChange(table);
|
if (table) this.broadcastChange(table);
|
||||||
@@ -344,6 +354,18 @@ export class MetonaSqlark {
|
|||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.5.1: 在线备份 — 导出全库一致性快照。
|
||||||
|
* Aria 引擎走引擎级 backup()(MVCC 一致性视图);其余引擎回退 exportAll()。
|
||||||
|
*/
|
||||||
|
async backup(): Promise<Record<string, Record<string, unknown>[]>> {
|
||||||
|
this.ensureReady();
|
||||||
|
if (typeof this.engine.backup === 'function') {
|
||||||
|
return this.engine.backup();
|
||||||
|
}
|
||||||
|
return this.exportAll();
|
||||||
|
}
|
||||||
|
|
||||||
// ---- 发布订阅 ----
|
// ---- 发布订阅 ----
|
||||||
|
|
||||||
private listeners: Map<string, Set<(data: unknown) => void>> = new Map();
|
private listeners: Map<string, Set<(data: unknown) => void>> = new Map();
|
||||||
@@ -393,6 +415,43 @@ export class MetonaSqlark {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.5.1: SQL 写语句触发 CRUD 生命周期钩子。
|
||||||
|
* INSERT/UPDATE/DELETE 分别触发 beforeInsert/afterInsert、beforeUpdate/afterUpdate、
|
||||||
|
* beforeDelete/afterDelete(参数与 Table API 路径一致)。
|
||||||
|
*/
|
||||||
|
private async triggerStatementHooks(stmt: Statement, phase: 'before' | 'after', result?: unknown): Promise<void> {
|
||||||
|
switch (stmt.type) {
|
||||||
|
case 'INSERT': {
|
||||||
|
const rows: Record<string, unknown>[] = (stmt.values ?? []).map((vals: unknown[]) => {
|
||||||
|
const row: Record<string, unknown> = {};
|
||||||
|
const cols = stmt.columns ?? [];
|
||||||
|
for (let i = 0; i < vals.length; i++) {
|
||||||
|
row[cols[i] ?? String(i)] = vals[i];
|
||||||
|
}
|
||||||
|
return row;
|
||||||
|
});
|
||||||
|
if (phase === 'before') await this.pluginManager.trigger('beforeInsert', rows);
|
||||||
|
else await this.pluginManager.trigger('afterInsert', rows, result as string[]);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
case 'UPDATE': {
|
||||||
|
const query = { table: stmt.table, where: stmt.where };
|
||||||
|
if (phase === 'before') await this.pluginManager.trigger('beforeUpdate', query, stmt.sets);
|
||||||
|
else await this.pluginManager.trigger('afterUpdate', query, stmt.sets, result as number);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
case 'DELETE': {
|
||||||
|
const query = { table: stmt.from, where: stmt.where };
|
||||||
|
if (phase === 'before') await this.pluginManager.trigger('beforeDelete', query);
|
||||||
|
else await this.pluginManager.trigger('afterDelete', query, result as number);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
default:
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// ---- 迁移 ----
|
// ---- 迁移 ----
|
||||||
|
|
||||||
private migrations: Map<number, (db: MetonaSqlark) => Promise<void>> = new Map();
|
private migrations: Map<number, (db: MetonaSqlark) => Promise<void>> = new Map();
|
||||||
@@ -497,7 +556,11 @@ export class MetonaSqlark {
|
|||||||
case 'disk':
|
case 'disk':
|
||||||
return diskEngine === 'opfs' ? new OPFSEngine() : new IndexedDBEngine();
|
return diskEngine === 'opfs' ? new OPFSEngine() : new IndexedDBEngine();
|
||||||
case 'aria':
|
case 'aria':
|
||||||
return new AriaEngine({ storageBackend: diskEngine === 'opfs' ? 'opfs' : 'indexeddb' });
|
// v0.4.5: 透传 AriaEngine 专属配置(walSyncMode/checkpointInterval/encryption/pageStorage 等)
|
||||||
|
return new AriaEngine({
|
||||||
|
storageBackend: diskEngine === 'opfs' ? 'opfs' : 'indexeddb',
|
||||||
|
...(this.config.aria ?? {}),
|
||||||
|
});
|
||||||
case 'hybrid':
|
case 'hybrid':
|
||||||
return new HybridEngine(diskEngine);
|
return new HybridEngine(diskEngine);
|
||||||
default:
|
default:
|
||||||
|
|||||||
@@ -75,20 +75,6 @@ export class LRUList {
|
|||||||
page.next = null;
|
page.next = null;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 获取 LRU 尾部(最久未使用的页面) */
|
|
||||||
getLRU(): PageHandle | null {
|
|
||||||
return this.tail;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** 弹出 LRU 尾部 */
|
|
||||||
popLRU(): PageHandle | null {
|
|
||||||
const lru = this.tail;
|
|
||||||
if (lru) {
|
|
||||||
this.remove(lru);
|
|
||||||
}
|
|
||||||
return lru;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** 清空链表 */
|
/** 清空链表 */
|
||||||
clear(): void {
|
clear(): void {
|
||||||
this.head = null;
|
this.head = null;
|
||||||
@@ -96,16 +82,11 @@ export class LRUList {
|
|||||||
this._size = 0;
|
this._size = 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 获取所有页面(用于迭代) */
|
/** 获取 LRU 尾部(最久未使用的页面) */
|
||||||
getAllPages(): PageHandle[] {
|
getLRU(): PageHandle | null {
|
||||||
const pages: PageHandle[] = [];
|
return this.tail;
|
||||||
let current = this.head;
|
|
||||||
while (current) {
|
|
||||||
pages.push(current);
|
|
||||||
current = current.next;
|
|
||||||
}
|
|
||||||
return pages;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -189,16 +170,6 @@ export class EvictionManager {
|
|||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 获取当前大小 */
|
|
||||||
getSize(): number {
|
|
||||||
return this.lru.size;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** 获取容量 */
|
|
||||||
getCapacity(): number {
|
|
||||||
return this.capacity;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** 清空 */
|
/** 清空 */
|
||||||
clear(): void {
|
clear(): void {
|
||||||
this.lru.clear();
|
this.lru.clear();
|
||||||
|
|||||||
@@ -21,6 +21,8 @@ export interface PageIO {
|
|||||||
writePage(pageId: number, data: ArrayBuffer): Promise<void>;
|
writePage(pageId: number, data: ArrayBuffer): Promise<void>;
|
||||||
/** 分配新页面 ID */
|
/** 分配新页面 ID */
|
||||||
allocatePageId(): Promise<number>;
|
allocatePageId(): Promise<number>;
|
||||||
|
/** v0.4.5: 批量分配页面 ID(一次 meta 持久化) */
|
||||||
|
allocatePageIds?(count: number): Promise<number[]>;
|
||||||
/** 释放页面 ID */
|
/** 释放页面 ID */
|
||||||
freePageId(pageId: number): Promise<void>;
|
freePageId(pageId: number): Promise<void>;
|
||||||
}
|
}
|
||||||
@@ -87,18 +89,30 @@ export class BufferPool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* 创建新页面。
|
* v0.4.5: 批量创建新页面(一次页面 ID 分配,页面化 SSTable 保存用)。
|
||||||
|
* 返回的页面均 pin 且 dirty=false(调用方写入后需 markDirty + flushPage)。
|
||||||
*/
|
*/
|
||||||
async newPage(type: PageType = PageType.DATA): Promise<PageHandle> {
|
async newPages(count: number, type: PageType = PageType.DATA): Promise<PageHandle[]> {
|
||||||
const pageId = await this.pageIO.allocatePageId();
|
if (count <= 0) return [];
|
||||||
|
let pageIds: number[];
|
||||||
|
if (typeof this.pageIO.allocatePageIds === 'function') {
|
||||||
|
pageIds = await this.pageIO.allocatePageIds!(count);
|
||||||
|
} else {
|
||||||
|
pageIds = [];
|
||||||
|
for (let i = 0; i < count; i++) pageIds.push(await this.pageIO.allocatePageId());
|
||||||
|
}
|
||||||
|
|
||||||
await this.eviction.evictIfNeeded(1);
|
await this.eviction.evictIfNeeded(count);
|
||||||
|
|
||||||
const page = createPage(pageId, type);
|
const handles: PageHandle[] = [];
|
||||||
page.pins = 1;
|
for (const pageId of pageIds) {
|
||||||
this.pages.set(pageId, page);
|
const page = createPage(pageId, type);
|
||||||
this.eviction.add(page);
|
page.pins = 1;
|
||||||
return page;
|
this.pages.set(pageId, page);
|
||||||
|
this.eviction.add(page);
|
||||||
|
handles.push(page);
|
||||||
|
}
|
||||||
|
return handles;
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
@@ -160,26 +174,4 @@ export class BufferPool {
|
|||||||
this.eviction.clear();
|
this.eviction.clear();
|
||||||
}
|
}
|
||||||
|
|
||||||
// -----------------------------------------------------------------------
|
|
||||||
// 统计
|
|
||||||
// -----------------------------------------------------------------------
|
|
||||||
|
|
||||||
/** 获取当前缓存页面数 */
|
|
||||||
getCachedPageCount(): number {
|
|
||||||
return this.pages.size;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** 获取缓存容量 */
|
|
||||||
getCapacity(): number {
|
|
||||||
return this.eviction.getCapacity();
|
|
||||||
}
|
|
||||||
|
|
||||||
/** 获取脏页面数 */
|
|
||||||
getDirtyPageCount(): number {
|
|
||||||
let count = 0;
|
|
||||||
for (const [, page] of this.pages) {
|
|
||||||
if (page.dirty) count++;
|
|
||||||
}
|
|
||||||
return count;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -2,10 +2,9 @@
|
|||||||
* AriaEngine LZ4 Compression — 简化 LZ4 压缩/解压
|
* AriaEngine LZ4 Compression — 简化 LZ4 压缩/解压
|
||||||
* @module engine/aria/compression/lz4
|
* @module engine/aria/compression/lz4
|
||||||
*
|
*
|
||||||
* v0.2.6: 修复往返一致性
|
* v0.4.5 格式 v2:压缩流前增加 4 字节原始大小头(LE u32),
|
||||||
* - 匹配长度截断到 19 字节(matchField 上限 15 + MIN_MATCH),长匹配分段输出
|
* 解压不再依赖外部估算(高压缩率数据下 buf.length*2 估算不足会截断)。
|
||||||
* - 组合 token 的 matchField ∈ [1,15];matchField=0 且 lo=0 表示末尾纯字面量(无 offset)
|
* 旧版压缩数据(无头)视为损坏(compression 选项自 v0.2.6 起已声明不向后兼容)。
|
||||||
* - 消除"matchField=0 的组合 token 与纯字面量 token 歧义"
|
|
||||||
*
|
*
|
||||||
* Token 格式(1 字节):
|
* Token 格式(1 字节):
|
||||||
* hi 4bit = litLen (0-15)
|
* hi 4bit = litLen (0-15)
|
||||||
@@ -17,10 +16,16 @@
|
|||||||
|
|
||||||
const MIN_MATCH = 4;
|
const MIN_MATCH = 4;
|
||||||
const MAX_MATCH = MIN_MATCH + 15; // 19,匹配长度上限
|
const MAX_MATCH = MIN_MATCH + 15; // 19,匹配长度上限
|
||||||
|
/** 原始大小头字节数 */
|
||||||
|
const HEADER_SIZE = 4;
|
||||||
|
|
||||||
export function compressLZ4(input: Uint8Array): Uint8Array {
|
export function compressLZ4(input: Uint8Array): Uint8Array {
|
||||||
// 空输入直接返回(无 token 可输出)
|
// 空输入:仅头部(原始大小 0)
|
||||||
if (input.byteLength === 0) return input;
|
if (input.byteLength === 0) {
|
||||||
|
const empty = new Uint8Array(HEADER_SIZE);
|
||||||
|
new DataView(empty.buffer).setUint32(0, 0, true);
|
||||||
|
return empty;
|
||||||
|
}
|
||||||
|
|
||||||
// 最坏情况:纯字面量分块输出 len/15 个 token + 末尾 token
|
// 最坏情况:纯字面量分块输出 len/15 个 token + 末尾 token
|
||||||
// 上限:len + ceil(len/15) + 8(组合 token 的 offset 开销已包含在内)
|
// 上限:len + ceil(len/15) + 8(组合 token 的 offset 开销已包含在内)
|
||||||
@@ -73,24 +78,38 @@ export function compressLZ4(input: Uint8Array): Uint8Array {
|
|||||||
litStart += chunk;
|
litStart += chunk;
|
||||||
}
|
}
|
||||||
|
|
||||||
// 始终输出压缩流(即使比原数据略大)。
|
// v0.4.5: 前置原始大小头,解压端自描述
|
||||||
// 注意:不能返回原样 input —— 解压端无法区分"压缩流"与"原始数据",
|
const stream = out.slice(0, di);
|
||||||
// 原样返回会导致解压器将原始字节误解析为 token(v0.2.6 修复)
|
const combined = new Uint8Array(HEADER_SIZE + stream.byteLength);
|
||||||
return out.slice(0, di);
|
new DataView(combined.buffer).setUint32(0, input.byteLength, true);
|
||||||
|
combined.set(stream, HEADER_SIZE);
|
||||||
|
return combined;
|
||||||
}
|
}
|
||||||
|
|
||||||
export function decompressLZ4(input: Uint8Array, originalSize: number): Uint8Array {
|
export function decompressLZ4(input: Uint8Array, _originalSize?: number): Uint8Array {
|
||||||
|
if (input.byteLength < HEADER_SIZE) {
|
||||||
|
throw new Error('LZ4 stream too short: missing header');
|
||||||
|
}
|
||||||
|
const view = new DataView(input.buffer, input.byteOffset, input.byteLength);
|
||||||
|
const originalSize = view.getUint32(0, true);
|
||||||
|
if (originalSize === 0 && input.byteLength === HEADER_SIZE) {
|
||||||
|
return new Uint8Array(0); // 空输入
|
||||||
|
}
|
||||||
|
if (originalSize <= 0 || originalSize > 0x3fffffff) {
|
||||||
|
throw new Error('Invalid LZ4 header: bad original size');
|
||||||
|
}
|
||||||
|
const stream = input.subarray(HEADER_SIZE);
|
||||||
const out = new Uint8Array(originalSize);
|
const out = new Uint8Array(originalSize);
|
||||||
let si = 0, di = 0;
|
let si = 0, di = 0;
|
||||||
|
|
||||||
while (si < input.byteLength && di < originalSize) {
|
while (si < stream.byteLength && di < originalSize) {
|
||||||
const token = input[si++];
|
const token = stream[si++];
|
||||||
const litLen = (token >> 4) & 0x0F;
|
const litLen = (token >> 4) & 0x0F;
|
||||||
const matchField = token & 0x0F;
|
const matchField = token & 0x0F;
|
||||||
|
|
||||||
// 复制字面量
|
// 复制字面量
|
||||||
for (let i = 0; i < litLen && si < input.byteLength && di < originalSize; i++) {
|
for (let i = 0; i < litLen && si < stream.byteLength && di < originalSize; i++) {
|
||||||
out[di++] = input[si++];
|
out[di++] = stream[si++];
|
||||||
}
|
}
|
||||||
|
|
||||||
// matchField=0:纯字面量 token(无 offset 无匹配)。
|
// matchField=0:纯字面量 token(无 offset 无匹配)。
|
||||||
@@ -98,8 +117,8 @@ export function decompressLZ4(input: Uint8Array, originalSize: number): Uint8Arr
|
|||||||
if (matchField === 0) continue;
|
if (matchField === 0) continue;
|
||||||
|
|
||||||
// 组合 token:读取 offset + 复制匹配(可能自重叠)
|
// 组合 token:读取 offset + 复制匹配(可能自重叠)
|
||||||
if (si + 1 >= input.byteLength) break;
|
if (si + 1 >= stream.byteLength) break;
|
||||||
const offset = input[si++] | (input[si++] << 8);
|
const offset = stream[si++] | (stream[si++] << 8);
|
||||||
const matchLen = matchField + MIN_MATCH;
|
const matchLen = matchField + MIN_MATCH;
|
||||||
for (let i = 0; i < matchLen && di < originalSize; i++) {
|
for (let i = 0; i < matchLen && di < originalSize; i++) {
|
||||||
out[di] = out[di - offset];
|
out[di] = out[di - offset];
|
||||||
|
|||||||
@@ -0,0 +1,58 @@
|
|||||||
|
/**
|
||||||
|
* AriaEngine CRC32 — 标准 CRC-32(IEEE 802.3,多项式 0xEDB88320)
|
||||||
|
* @module engine/aria/crc32
|
||||||
|
*
|
||||||
|
* 查表法实现。
|
||||||
|
* 分段计算约定:
|
||||||
|
* crc32(head + tail) === crc32Finalize(crc32Update(crc32Update(0xFFFFFFFF, head), tail))
|
||||||
|
* === crc32(tail, crc32(head))
|
||||||
|
* 用于 SSTable 文件校验和与 WAL 记录完整性校验。
|
||||||
|
*/
|
||||||
|
|
||||||
|
/** CRC-32 查找表(0xEDB88320 反射多项式) */
|
||||||
|
const CRC32_TABLE = (() => {
|
||||||
|
const table = new Uint32Array(256);
|
||||||
|
for (let i = 0; i < 256; i++) {
|
||||||
|
let c = i;
|
||||||
|
for (let k = 0; k < 8; k++) {
|
||||||
|
c = c & 1 ? 0xedb88320 ^ (c >>> 1) : c >>> 1;
|
||||||
|
}
|
||||||
|
table[i] = c >>> 0;
|
||||||
|
}
|
||||||
|
return table;
|
||||||
|
})();
|
||||||
|
|
||||||
|
/** CRC-32 初始累加器状态 */
|
||||||
|
export const CRC32_INIT = 0xffffffff;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 更新内部 CRC 累加状态(供分段计算使用,接收中间状态返回中间状态)。
|
||||||
|
*/
|
||||||
|
export function crc32Update(state: number, data: Uint8Array): number {
|
||||||
|
let crc = state >>> 0;
|
||||||
|
for (let i = 0; i < data.byteLength; i++) {
|
||||||
|
crc = (CRC32_TABLE[(crc ^ data[i]) & 0xff] ^ (crc >>> 8)) >>> 0;
|
||||||
|
}
|
||||||
|
return crc >>> 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 将中间状态转换为最终校验和(终止计算) */
|
||||||
|
export function crc32Finalize(state: number): number {
|
||||||
|
return (state ^ CRC32_INIT) >>> 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 计算标准 CRC-32 校验和。
|
||||||
|
* @param data 输入字节
|
||||||
|
* @param seed 前序片段计算的最终校验和(首段传 0 或不传)
|
||||||
|
* @returns 32 位无符号校验和
|
||||||
|
*/
|
||||||
|
export function crc32(data: Uint8Array, seed = 0): number {
|
||||||
|
const state = crc32Update(seed ^ CRC32_INIT, data);
|
||||||
|
return crc32Finalize(state);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 分段计算便捷函数:crc32Continue(prevFinal, chunk) === crc32(chunk, prevFinal) */
|
||||||
|
export function crc32Continue(prev: number, data: Uint8Array): number {
|
||||||
|
return crc32(data, prev);
|
||||||
|
}
|
||||||
@@ -33,18 +33,20 @@ export class CryptoManager {
|
|||||||
return actualSalt as Uint8Array;
|
return actualSalt as Uint8Array;
|
||||||
}
|
}
|
||||||
|
|
||||||
async encryptPage(data: ArrayBuffer): Promise<{ iv: Uint8Array; data: ArrayBuffer }> {
|
async encryptPage(data: ArrayBuffer | Uint8Array): Promise<{ iv: Uint8Array; data: ArrayBuffer }> {
|
||||||
if (!this.cryptoKey) throw new Error('Crypto not initialized');
|
if (!this.cryptoKey) throw new Error('Crypto not initialized');
|
||||||
const iv = crypto.getRandomValues(new Uint8Array(IV_LENGTH)) as any;
|
const iv = crypto.getRandomValues(new Uint8Array(IV_LENGTH)) as any;
|
||||||
// 传 TypedArray 视图而非裸 ArrayBuffer:SubtleCrypto 通过 ArrayBuffer.isView 检查,
|
// 传 TypedArray 视图而非裸 ArrayBuffer:SubtleCrypto 通过 ArrayBuffer.isView 检查,
|
||||||
// 对跨 realm / 跨 vm 环境的 ArrayBuffer 兼容(Node 18/20 的 webcrypto 对裸 ArrayBuffer 检查严格)
|
// 对跨 realm / 跨 vm 环境的 ArrayBuffer 兼容(Node 18/20 的 webcrypto 对裸 ArrayBuffer 检查严格)
|
||||||
const ciphertext = await crypto.subtle.encrypt({ name: ALGO, iv } as any, this.cryptoKey, new Uint8Array(data));
|
const plain = (data instanceof Uint8Array ? data : new Uint8Array(data)) as BufferSource;
|
||||||
|
const ciphertext = await crypto.subtle.encrypt({ name: ALGO, iv } as any, this.cryptoKey, plain);
|
||||||
return { iv: iv as Uint8Array, data: ciphertext };
|
return { iv: iv as Uint8Array, data: ciphertext };
|
||||||
}
|
}
|
||||||
|
|
||||||
async decryptPage(iv: Uint8Array, data: ArrayBuffer): Promise<ArrayBuffer> {
|
async decryptPage(iv: Uint8Array, data: ArrayBuffer | Uint8Array): Promise<ArrayBuffer> {
|
||||||
if (!this.cryptoKey) throw new Error('Crypto not initialized');
|
if (!this.cryptoKey) throw new Error('Crypto not initialized');
|
||||||
return crypto.subtle.decrypt({ name: ALGO, iv } as any, this.cryptoKey, new Uint8Array(data));
|
const ciphertext = (data instanceof Uint8Array ? data : new Uint8Array(data)) as BufferSource;
|
||||||
|
return crypto.subtle.decrypt({ name: ALGO, iv } as any, this.cryptoKey, ciphertext);
|
||||||
}
|
}
|
||||||
|
|
||||||
close(): void {
|
close(): void {
|
||||||
@@ -53,31 +55,3 @@ export class CryptoManager {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// 全局兼容层(旧代码仍可使用全局函数)
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
const globalCrypto = new CryptoManager();
|
|
||||||
|
|
||||||
/** @deprecated 使用 CryptoManager 实例代替 */
|
|
||||||
export async function initCrypto(password: string, salt?: Uint8Array): Promise<Uint8Array> {
|
|
||||||
return globalCrypto.init(password, salt);
|
|
||||||
}
|
|
||||||
|
|
||||||
/** @deprecated 使用 CryptoManager 实例代替 */
|
|
||||||
export function isCryptoEnabled(): boolean { return globalCrypto.enabled; }
|
|
||||||
|
|
||||||
/** @deprecated 使用 CryptoManager 实例代替 */
|
|
||||||
export async function encryptPage(data: ArrayBuffer): Promise<{ iv: Uint8Array; data: ArrayBuffer }> {
|
|
||||||
return globalCrypto.encryptPage(data);
|
|
||||||
}
|
|
||||||
|
|
||||||
/** @deprecated 使用 CryptoManager 实例代替 */
|
|
||||||
export async function decryptPage(iv: Uint8Array, data: ArrayBuffer): Promise<ArrayBuffer> {
|
|
||||||
return globalCrypto.decryptPage(iv, data);
|
|
||||||
}
|
|
||||||
|
|
||||||
/** @deprecated 使用 CryptoManager 实例代替 */
|
|
||||||
export function closeCrypto(): void {
|
|
||||||
globalCrypto.close();
|
|
||||||
}
|
|
||||||
|
|||||||
+160
-139
@@ -17,15 +17,18 @@ import { DEFAULT_ARIA_CONFIG } from './types';
|
|||||||
import { LSM } from './index/lsm';
|
import { LSM } from './index/lsm';
|
||||||
import type { SSTableStore } from './index/lsm';
|
import type { SSTableStore } from './index/lsm';
|
||||||
import { WAL } from './wal/log';
|
import { WAL } from './wal/log';
|
||||||
|
import { SegmentedWALStore } from './wal/segmented_store';
|
||||||
|
import { DatabaseLock } from './locks';
|
||||||
import { WALRecordType, type WALRecord } from './types';
|
import { WALRecordType, type WALRecord } from './types';
|
||||||
import { CheckpointManager } from './wal/checkpoint';
|
import { CheckpointManager } from './wal/checkpoint';
|
||||||
import { IndexedDBBackend, MemoryBackend, type IStorageBackend } from './store/backend';
|
import { IndexedDBBackend, MemoryBackend, type IStorageBackend } from './store/backend';
|
||||||
import { OPFSBackend } from './store/opfs_backend';
|
import { OPFSBackend } from './store/opfs_backend';
|
||||||
|
import { EncryptedBackend } from './store/encrypted_backend';
|
||||||
|
import { PageSSTableStore } from './store/page_sstable_store';
|
||||||
import { FileManager } from './store/file_manager';
|
import { FileManager } from './store/file_manager';
|
||||||
import { MVCCManager } from './transaction/mvcc';
|
import { MVCCManager } from './transaction/mvcc';
|
||||||
import { BufferPool } from './buffer/pool';
|
import { BufferPool } from './buffer/pool';
|
||||||
import { compressLZ4, decompressLZ4 } from './compression/lz4';
|
import { compressLZ4, decompressLZ4 } from './compression/lz4';
|
||||||
import { isCryptoEnabled, encryptPage, decryptPage } from './crypto';
|
|
||||||
import type { WALRecord as _WALRecord } from './types';
|
import type { WALRecord as _WALRecord } from './types';
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -35,7 +38,8 @@ import type { WALRecord as _WALRecord } from './types';
|
|||||||
export class AriaEngine implements IStorageEngine {
|
export class AriaEngine implements IStorageEngine {
|
||||||
readonly name = 'aria';
|
readonly name = 'aria';
|
||||||
|
|
||||||
private config!: Required<AriaEngineConfig>;
|
private config!: Required<Omit<AriaEngineConfig, 'encryption' | 'pageStorage'>> &
|
||||||
|
Pick<AriaEngineConfig, 'encryption' | 'pageStorage'>;
|
||||||
private lsm!: LSM; // 主键索引 LSM
|
private lsm!: LSM; // 主键索引 LSM
|
||||||
private wal!: WAL;
|
private wal!: WAL;
|
||||||
private checkpointManager!: CheckpointManager;
|
private checkpointManager!: CheckpointManager;
|
||||||
@@ -43,6 +47,13 @@ export class AriaEngine implements IStorageEngine {
|
|||||||
private opened = false;
|
private opened = false;
|
||||||
private dbName = '';
|
private dbName = '';
|
||||||
|
|
||||||
|
// v0.4.5: 页面化物理存储(FileManager + BufferPool 提升为引擎字段,close/repair 时落盘/清理)
|
||||||
|
private fileManager!: FileManager;
|
||||||
|
private bufferPool!: BufferPool;
|
||||||
|
|
||||||
|
// v0.4.5: 多标签页独占锁(Web Locks API,OPFS 等无事务后端防并发写)
|
||||||
|
private dbLock: DatabaseLock | null = null;
|
||||||
|
|
||||||
// 表结构
|
// 表结构
|
||||||
private schemas: Map<string, TableSchema> = new Map();
|
private schemas: Map<string, TableSchema> = new Map();
|
||||||
private tablePKs: Map<string, string> = new Map();
|
private tablePKs: Map<string, string> = new Map();
|
||||||
@@ -57,9 +68,6 @@ export class AriaEngine implements IStorageEngine {
|
|||||||
private txnSnapshot: Map<string, Record<string, unknown>> | null = null;
|
private txnSnapshot: Map<string, Record<string, unknown>> | null = null;
|
||||||
private gcCounter = 0;
|
private gcCounter = 0;
|
||||||
|
|
||||||
// Buffer Pool(页面缓存,减少磁盘 I/O)
|
|
||||||
private bufferPool!: BufferPool;
|
|
||||||
|
|
||||||
constructor(config: AriaEngineConfig = {}) {
|
constructor(config: AriaEngineConfig = {}) {
|
||||||
this.config = { ...DEFAULT_ARIA_CONFIG, ...config };
|
this.config = { ...DEFAULT_ARIA_CONFIG, ...config };
|
||||||
}
|
}
|
||||||
@@ -75,6 +83,11 @@ export class AriaEngine implements IStorageEngine {
|
|||||||
try {
|
try {
|
||||||
await this.openInternal(dbName);
|
await this.openInternal(dbName);
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
|
// 打开失败:释放已获取的锁(避免锁泄漏阻塞其他标签页)
|
||||||
|
if (this.dbLock) {
|
||||||
|
try { await this.dbLock.release(); } catch { /* ignore */ }
|
||||||
|
this.dbLock = null;
|
||||||
|
}
|
||||||
if (error instanceof DatabaseError) throw error;
|
if (error instanceof DatabaseError) throw error;
|
||||||
throw new DatabaseError(
|
throw new DatabaseError(
|
||||||
`Failed to open AriaEngine database "${dbName}"`,
|
`Failed to open AriaEngine database "${dbName}"`,
|
||||||
@@ -88,20 +101,49 @@ export class AriaEngine implements IStorageEngine {
|
|||||||
private async openInternal(dbName: string): Promise<void> {
|
private async openInternal(dbName: string): Promise<void> {
|
||||||
this.dbName = dbName;
|
this.dbName = dbName;
|
||||||
|
|
||||||
// 1. 存储后端
|
// v0.4.5: 多标签页独占锁(Web Locks)— 不支持的环境降级为无锁(文档注明)
|
||||||
|
const lock = new DatabaseLock();
|
||||||
|
this.dbLock = lock;
|
||||||
|
const lockAcquired = await lock.acquire(dbName);
|
||||||
|
if (!lockAcquired) {
|
||||||
|
// eslint-disable-next-line no-console
|
||||||
|
console.warn(
|
||||||
|
`[AriaEngine] Web Locks API unavailable: no multi-tab protection for "${dbName}" ` +
|
||||||
|
'(open the same database in multiple tabs may corrupt data)',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// 1. 存储后端(可选全库加密包装)
|
||||||
|
let baseBackend: IStorageBackend;
|
||||||
if (this.config.storageBackend === 'opfs') {
|
if (this.config.storageBackend === 'opfs') {
|
||||||
this.backend = new OPFSBackend();
|
baseBackend = new OPFSBackend();
|
||||||
} else if (this.config.storageBackend === 'indexeddb') {
|
} else if (this.config.storageBackend === 'indexeddb') {
|
||||||
this.backend = new IndexedDBBackend();
|
baseBackend = new IndexedDBBackend();
|
||||||
} else {
|
} else {
|
||||||
this.backend = new MemoryBackend();
|
baseBackend = new MemoryBackend();
|
||||||
|
}
|
||||||
|
await baseBackend.open(dbName);
|
||||||
|
|
||||||
|
// v0.4.5: encryption 配置 → 透明加密封装(密码错误/数据损坏在 open 或首次读取时暴露)
|
||||||
|
if (this.config.encryption?.password) {
|
||||||
|
this.backend = new EncryptedBackend(baseBackend, this.config.encryption.password);
|
||||||
|
} else {
|
||||||
|
this.backend = baseBackend;
|
||||||
|
// 反向检测:库中存在密钥元数据但未提供密码 → 拒绝打开(避免密文被当明文解析成空库)
|
||||||
|
if (await baseBackend.exists('__aria_keymeta')) {
|
||||||
|
await baseBackend.close();
|
||||||
|
throw new DatabaseError(
|
||||||
|
'Database is encrypted: provide encryption.password to open it',
|
||||||
|
'ARIA_ENCRYPT_REQUIRED',
|
||||||
|
);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
await this.backend.open(dbName);
|
await this.backend.open(dbName);
|
||||||
|
|
||||||
// 2a. FileManager (PageIO 实现) + Buffer Pool
|
// 2a. FileManager (PageIO 实现) + Buffer Pool
|
||||||
const fileManager = new FileManager(this.backend);
|
this.fileManager = new FileManager(this.backend);
|
||||||
await fileManager.init(dbName);
|
await this.fileManager.init(dbName);
|
||||||
this.bufferPool = new BufferPool(fileManager, this.config.bufferPoolPages);
|
this.bufferPool = new BufferPool(this.fileManager, this.config.bufferPoolPages);
|
||||||
|
|
||||||
// 2. 构建 SSTableStore
|
// 2. 构建 SSTableStore
|
||||||
const sstableStore = this.createSSTableStore('main');
|
const sstableStore = this.createSSTableStore('main');
|
||||||
@@ -118,55 +160,10 @@ export class AriaEngine implements IStorageEngine {
|
|||||||
});
|
});
|
||||||
|
|
||||||
// 4. 初始化 WAL
|
// 4. 初始化 WAL
|
||||||
|
// v0.4.5: 分片式 WAL 存储(__wal_%06d.bin),序号内嵌记录字节流无需 count 键,
|
||||||
|
// append 单文件原子写;空洞检测截断;兼容旧格式 __wal_N + __wal_count
|
||||||
this.wal = new WAL(
|
this.wal = new WAL(
|
||||||
{
|
new SegmentedWALStore(this.backend),
|
||||||
append: async (data) => {
|
|
||||||
// v0.4.2-fix: 记录写入与 count 计数在同一底层事务中原子提交,
|
|
||||||
// 中断时整体回滚,杜绝"记录在、计数丢"导致恢复漏读的丢数据问题
|
|
||||||
const idx = await this.getWALCount();
|
|
||||||
const slice = new Uint8Array(data.buffer, data.byteOffset, data.byteLength);
|
|
||||||
const copy = slice.buffer.slice(slice.byteOffset, slice.byteOffset + slice.byteLength) as ArrayBuffer;
|
|
||||||
await this.backend.writeMany({
|
|
||||||
[`__wal_${idx}`]: copy,
|
|
||||||
__wal_count: new TextEncoder().encode(String(idx + 1)).buffer,
|
|
||||||
});
|
|
||||||
},
|
|
||||||
readAll: async () => {
|
|
||||||
// v0.4.2-fix: 不依赖 count 计数,直接扫描全部 __wal_* 键,
|
|
||||||
// 避免 count 与实际记录不一致时漏读(与 checkpoint/并发写入竞态无关)
|
|
||||||
const keys = (await this.backend.listKeys())
|
|
||||||
.filter((k) => k.startsWith('__wal_') && k !== '__wal_count')
|
|
||||||
.sort((a, b) => {
|
|
||||||
const na = parseInt(a.slice('__wal_'.length), 10);
|
|
||||||
const nb = parseInt(b.slice('__wal_'.length), 10);
|
|
||||||
return (isNaN(na) ? 0 : na) - (isNaN(nb) ? 0 : nb);
|
|
||||||
});
|
|
||||||
if (keys.length === 0) return new Uint8Array(0);
|
|
||||||
const chunks: Uint8Array[] = [];
|
|
||||||
for (const key of keys) {
|
|
||||||
const d = await this.backend.read(key);
|
|
||||||
if (d) chunks.push(new Uint8Array(d));
|
|
||||||
}
|
|
||||||
const total = chunks.reduce((s, c) => s + c.byteLength, 0);
|
|
||||||
const combined = new Uint8Array(total);
|
|
||||||
let off = 0;
|
|
||||||
for (const c of chunks) { combined.set(c, off); off += c.byteLength; }
|
|
||||||
return combined;
|
|
||||||
},
|
|
||||||
truncate: async () => {
|
|
||||||
// v0.4.2-fix: 扫描删除全部 WAL 记录键 + count 键(单事务原子清理)
|
|
||||||
const keys = (await this.backend.listKeys())
|
|
||||||
.filter((k) => k.startsWith('__wal_'));
|
|
||||||
await this.backend.deleteMany(keys);
|
|
||||||
await this.setWALCount(0);
|
|
||||||
},
|
|
||||||
exists: async () => {
|
|
||||||
// v0.4.2-fix: 与 readAll 一致按 key 扫描判断(count 可能因崩溃截断而滞后)
|
|
||||||
const keys = (await this.backend.listKeys())
|
|
||||||
.filter((k) => k.startsWith('__wal_') && k !== '__wal_count');
|
|
||||||
return keys.length > 0;
|
|
||||||
},
|
|
||||||
},
|
|
||||||
this.config.walEnabled,
|
this.config.walEnabled,
|
||||||
this.config.walSyncMode,
|
this.config.walSyncMode,
|
||||||
);
|
);
|
||||||
@@ -270,12 +267,19 @@ export class AriaEngine implements IStorageEngine {
|
|||||||
for (const idxLsm of this.secondaryIndexes.values()) {
|
for (const idxLsm of this.secondaryIndexes.values()) {
|
||||||
await idxLsm.flush();
|
await idxLsm.flush();
|
||||||
}
|
}
|
||||||
|
// v0.4.5: 页面化存储 — 落盘全部脏页(save 已逐页落盘,此处兜底)
|
||||||
|
await this.bufferPool.flushAll();
|
||||||
await this.wal.flush();
|
await this.wal.flush();
|
||||||
// v0.4.2-fix: close 前 checkpoint(截断 WAL)—
|
// v0.4.2-fix: close 前 checkpoint(截断 WAL)—
|
||||||
// 此前只 flush 不截断,下次打开会重放全部历史 WAL 记录(含已落盘 SSTable 的数据),
|
// 此前只 flush 不截断,下次打开会重放全部历史 WAL 记录(含已落盘 SSTable 的数据),
|
||||||
// 重复解析/重复 put 拖慢启动,并与恢复后 flush+checkpoint 竞争放大丢数据
|
// 重复解析/重复 put 拖慢启动,并与恢复后 flush+checkpoint 竞争放大丢数据
|
||||||
await this.wal.checkpoint();
|
await this.wal.checkpoint();
|
||||||
await this.backend.close();
|
await this.backend.close();
|
||||||
|
// v0.4.5: 释放多标签页独占锁(等待锁真正归还)
|
||||||
|
if (this.dbLock) {
|
||||||
|
await this.dbLock.release();
|
||||||
|
this.dbLock = null;
|
||||||
|
}
|
||||||
// v0.4.2-fix: 清空运行期状态(此前 close 后 mvcc/txn 残留,
|
// v0.4.2-fix: 清空运行期状态(此前 close 后 mvcc/txn 残留,
|
||||||
// 重开时 beginTransaction 报 TX_ACTIVE 或读到陈旧快照)
|
// 重开时 beginTransaction 报 TX_ACTIVE 或读到陈旧快照)
|
||||||
this.schemas.clear();
|
this.schemas.clear();
|
||||||
@@ -291,25 +295,77 @@ export class AriaEngine implements IStorageEngine {
|
|||||||
|
|
||||||
/**
|
/**
|
||||||
* v0.4.2-fix: 崩溃恢复/自愈 — 校验并移除损坏 SSTable、截断 WAL、重建二级索引。
|
* v0.4.2-fix: 崩溃恢复/自愈 — 校验并移除损坏 SSTable、截断 WAL、重建二级索引。
|
||||||
|
* v0.4.5 增强:清理 OPFS 残留临时文件、清理孤儿页面(meta 未引用的 pg_ 文件)。
|
||||||
* 应用层检测到异常后调用,无需删库重建。
|
* 应用层检测到异常后调用,无需删库重建。
|
||||||
*/
|
*/
|
||||||
async repair(): Promise<void> {
|
async repair(): Promise<void> {
|
||||||
this.ensureOpen();
|
this.ensureOpen();
|
||||||
// 1. 校验全部 SSTable,移除残缺项(打开时已做一次,此处兜底运行期损坏)
|
// 1. 校验全部 SSTable,移除残缺项(打开时已做一次,此处兜底运行期损坏)
|
||||||
const removed = await this.lsm.validateAll();
|
const removed = await this.lsm.validateAll();
|
||||||
// 2. 将 WAL 残留数据落盘并截断,避免无限重放
|
// v0.4.5: 清空页面缓存(损坏数据可能驻留 BufferPool,重新加载)
|
||||||
|
await this.bufferPool.clear();
|
||||||
|
// 2. 将 WAL 残留数据落盘并截断,避免无限重放(含空洞截断落地)
|
||||||
await this.lsm.flush();
|
await this.lsm.flush();
|
||||||
await this.wal.checkpoint();
|
await this.wal.checkpoint();
|
||||||
// 3. 重建所有表的二级索引(修复索引与主数据不一致)
|
// 3. 重建所有表的二级索引(修复索引与主数据不一致)
|
||||||
for (const tableName of this.schemas.keys()) {
|
for (const tableName of this.schemas.keys()) {
|
||||||
await this.reindexTable(tableName);
|
await this.reindexTable(tableName);
|
||||||
}
|
}
|
||||||
|
// v0.4.5: 4. 清理 OPFS 残留临时文件(.crswap/.tmp)
|
||||||
|
const backendAny = this.backend as unknown as { cleanupStaleFiles?: () => Promise<void> };
|
||||||
|
if (typeof backendAny.cleanupStaleFiles === 'function') {
|
||||||
|
try { await backendAny.cleanupStaleFiles(); } catch { /* 清理失败不阻塞 */ }
|
||||||
|
}
|
||||||
|
// v0.4.5: 5. 清理孤儿页面(所有 LSM 命名空间 meta 均未引用的 pg_ 文件)
|
||||||
|
await this.cleanupOrphanPages();
|
||||||
if (removed > 0) {
|
if (removed > 0) {
|
||||||
// eslint-disable-next-line no-console
|
// eslint-disable-next-line no-console
|
||||||
console.warn(`[AriaEngine] repair: removed ${removed} corrupted SSTable(s)`);
|
console.warn(`[AriaEngine] repair: removed ${removed} corrupted SSTable(s)`);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.4.5: 清理孤儿页面 — 扫描全部 pg_* 文件,未被任何 LSM 命名空间 meta 引用的删除。
|
||||||
|
* 孤儿页面来自:崩溃中断的 compaction/删除流程(旧 SSTable 页面残留)。
|
||||||
|
*/
|
||||||
|
private async cleanupOrphanPages(): Promise<void> {
|
||||||
|
const keys = await this.backend.listKeys();
|
||||||
|
const pgKeys = keys.filter((k) => /^pg_\d+$/.test(k));
|
||||||
|
if (pgKeys.length === 0) return;
|
||||||
|
|
||||||
|
const used = new Set<number>();
|
||||||
|
const collectMeta = async (ns: string): Promise<void> => {
|
||||||
|
const META_KEY = ns === 'main' ? '__aria_lsm_meta' : `__aria_lsm_meta_${ns}`;
|
||||||
|
const raw = await this.backend.read(META_KEY);
|
||||||
|
if (!raw) return;
|
||||||
|
try {
|
||||||
|
const metas = JSON.parse(new TextDecoder().decode(raw)) as SSTableMeta[];
|
||||||
|
for (const m of metas) {
|
||||||
|
if (m.pageIds) {
|
||||||
|
for (const pid of m.pageIds) used.add(pid);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch { /* 损坏的 meta 忽略(validateAll 已处理) */ }
|
||||||
|
};
|
||||||
|
|
||||||
|
await collectMeta('main');
|
||||||
|
// 收集全部二级索引命名空间
|
||||||
|
for (const [tableName, schema] of this.schemas) {
|
||||||
|
for (const [colName, colDef] of Object.entries(schema.columns)) {
|
||||||
|
if (colDef.index || colDef.unique) {
|
||||||
|
await collectMeta(`idx_${tableName}_${colName}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const orphanIds = pgKeys
|
||||||
|
.map((k) => Number(k.slice('pg_'.length)))
|
||||||
|
.filter((pid) => !used.has(pid));
|
||||||
|
if (orphanIds.length > 0) {
|
||||||
|
await this.backend.deleteMany(orphanIds.map((pid) => `pg_${pid}`));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* v0.4.1: 重置数据库 — 清空全部数据与表结构(演示页刷新/重新初始化用)。
|
* v0.4.1: 重置数据库 — 清空全部数据与表结构(演示页刷新/重新初始化用)。
|
||||||
* 清空存储后端、LSM、WAL、MVCC 与二级索引,后续可继续使用本实例。
|
* 清空存储后端、LSM、WAL、MVCC 与二级索引,后续可继续使用本实例。
|
||||||
@@ -318,6 +374,9 @@ export class AriaEngine implements IStorageEngine {
|
|||||||
this.ensureOpen();
|
this.ensureOpen();
|
||||||
// 清空存储后端(页面文件 / WAL 记录 / schema 记录 / 元数据)
|
// 清空存储后端(页面文件 / WAL 记录 / schema 记录 / 元数据)
|
||||||
await this.backend.clear();
|
await this.backend.clear();
|
||||||
|
// v0.4.5: 清空页面缓存与页面 ID 分配状态
|
||||||
|
await this.bufferPool.clear();
|
||||||
|
await this.fileManager.clearAll();
|
||||||
this.schemas.clear();
|
this.schemas.clear();
|
||||||
this.tablePKs.clear();
|
this.tablePKs.clear();
|
||||||
this.secondaryIndexes.clear();
|
this.secondaryIndexes.clear();
|
||||||
@@ -1319,6 +1378,10 @@ export class AriaEngine implements IStorageEngine {
|
|||||||
let seq = 0;
|
let seq = 0;
|
||||||
let seqLoaded = false;
|
let seqLoaded = false;
|
||||||
|
|
||||||
|
// v0.4.5: 页面化物理存储(OPFS 后端默认启用)— SSTable 存为 4KB 页面,BufferPool 缓存
|
||||||
|
const usePages = this.isPageStorage();
|
||||||
|
const pageStore = usePages ? new PageSSTableStore(this.fileManager, this.bufferPool) : null;
|
||||||
|
|
||||||
const encodeText = (text: string): ArrayBuffer => {
|
const encodeText = (text: string): ArrayBuffer => {
|
||||||
return new TextEncoder().encode(text).buffer;
|
return new TextEncoder().encode(text).buffer;
|
||||||
};
|
};
|
||||||
@@ -1335,45 +1398,46 @@ export class AriaEngine implements IStorageEngine {
|
|||||||
|
|
||||||
return {
|
return {
|
||||||
save: async (id, data) => {
|
save: async (id, data) => {
|
||||||
|
if (pageStore) {
|
||||||
|
// 页面化:切页写入 BufferPool 并逐页落盘(save 语义 = 已持久化)
|
||||||
|
await pageStore.save(id, data);
|
||||||
|
return;
|
||||||
|
}
|
||||||
let buf: ArrayBuffer = data.buffer.slice(data.byteOffset, data.byteOffset + data.byteLength) as ArrayBuffer;
|
let buf: ArrayBuffer = data.buffer.slice(data.byteOffset, data.byteOffset + data.byteLength) as ArrayBuffer;
|
||||||
// 压缩(若启用)
|
// 压缩(若启用)— 加密由 EncryptedBackend 在 backend 层透明处理(v0.4.5)
|
||||||
if (this.config.compression) {
|
if (this.config.compression) {
|
||||||
const compressed = compressLZ4(new Uint8Array(buf));
|
const compressed = compressLZ4(new Uint8Array(buf));
|
||||||
buf = compressed.buffer.slice(compressed.byteOffset, compressed.byteOffset + compressed.byteLength) as ArrayBuffer;
|
buf = compressed.buffer.slice(compressed.byteOffset, compressed.byteOffset + compressed.byteLength) as ArrayBuffer;
|
||||||
}
|
}
|
||||||
// 加密(若启用)
|
|
||||||
if (isCryptoEnabled()) {
|
|
||||||
const enc = await encryptPage(buf);
|
|
||||||
const header = new Uint8Array(12 + 4); // IV(12) + originalLen(4)
|
|
||||||
header.set(enc.iv, 0);
|
|
||||||
new DataView(header.buffer).setUint32(12, data.byteLength, false);
|
|
||||||
const combined = new Uint8Array(header.length + enc.data.byteLength);
|
|
||||||
combined.set(header, 0);
|
|
||||||
combined.set(new Uint8Array(enc.data), header.length);
|
|
||||||
buf = combined.buffer;
|
|
||||||
}
|
|
||||||
await this.backend.write(`${filePrefix}${id}`, buf);
|
await this.backend.write(`${filePrefix}${id}`, buf);
|
||||||
},
|
},
|
||||||
load: async (id) => {
|
load: async (id) => {
|
||||||
|
// 页面化读取:meta 有 pageIds → 页面拼接;无(旧数据)→ 整 value
|
||||||
|
if (pageStore) {
|
||||||
|
const metas = await readMetaList();
|
||||||
|
const meta = metas.find((m) => m.id === id);
|
||||||
|
if (meta && meta.pageIds && meta.pageIds.length > 0) {
|
||||||
|
return pageStore.load(id, meta.pageIds, meta.totalSize);
|
||||||
|
}
|
||||||
|
}
|
||||||
const raw = await this.backend.read(`${filePrefix}${id}`);
|
const raw = await this.backend.read(`${filePrefix}${id}`);
|
||||||
if (!raw) return null;
|
if (!raw) return null;
|
||||||
let buf = new Uint8Array(raw);
|
let buf = new Uint8Array(raw);
|
||||||
// 解密(若数据带加密头)
|
// 解压(若启用)— 解密由 EncryptedBackend 在 backend 层透明处理(v0.4.5)
|
||||||
if (isCryptoEnabled() && buf.length > 16) {
|
|
||||||
const iv = buf.slice(0, 12);
|
|
||||||
const origLen = new DataView(buf.buffer, buf.byteOffset + 12, 4).getUint32(0, false);
|
|
||||||
const ciphertext = buf.slice(16).buffer;
|
|
||||||
const decrypted = await decryptPage(iv, ciphertext);
|
|
||||||
buf = new Uint8Array(decrypted, 0, origLen);
|
|
||||||
}
|
|
||||||
// 解压(若启用)
|
|
||||||
if (this.config.compression) {
|
if (this.config.compression) {
|
||||||
const decompressed = decompressLZ4(buf, buf.length * 2); // 估计原始大小
|
// v0.4.5: 压缩流自带原始大小头,无需外部估算
|
||||||
buf = decompressed as any;
|
buf = decompressLZ4(buf) as Uint8Array<ArrayBuffer>;
|
||||||
}
|
}
|
||||||
return buf;
|
return buf;
|
||||||
},
|
},
|
||||||
delete: async (id) => {
|
delete: async (id) => {
|
||||||
|
if (pageStore) {
|
||||||
|
const metas = await readMetaList();
|
||||||
|
const meta = metas.find((m) => m.id === id);
|
||||||
|
if (meta && meta.pageIds && meta.pageIds.length > 0) {
|
||||||
|
await pageStore.delete(id, meta.pageIds);
|
||||||
|
}
|
||||||
|
}
|
||||||
await this.backend.delete(`${filePrefix}${id}`);
|
await this.backend.delete(`${filePrefix}${id}`);
|
||||||
},
|
},
|
||||||
allocateId: async () => {
|
allocateId: async () => {
|
||||||
@@ -1388,10 +1452,13 @@ export class AriaEngine implements IStorageEngine {
|
|||||||
listMeta: readMetaList,
|
listMeta: readMetaList,
|
||||||
saveMeta: async (meta) => {
|
saveMeta: async (meta) => {
|
||||||
const list = await readMetaList();
|
const list = await readMetaList();
|
||||||
|
// v0.4.5: 页面化时把页面 ID 列表注入 meta(save 后、saveMeta 前由 LSM 顺序调用)
|
||||||
|
const pageIds = pageStore?.getPageIds(meta.id);
|
||||||
|
const metaWithPages = pageIds && pageIds.length > 0 ? { ...meta, pageIds } : meta;
|
||||||
// 更新或添加
|
// 更新或添加
|
||||||
const idx = list.findIndex((m) => m.id === meta.id);
|
const idx = list.findIndex((m) => m.id === meta.id);
|
||||||
if (idx >= 0) list[idx] = meta;
|
if (idx >= 0) list[idx] = metaWithPages;
|
||||||
else list.push(meta);
|
else list.push(metaWithPages);
|
||||||
await this.backend.write(META_KEY, encodeText(JSON.stringify(list)));
|
await this.backend.write(META_KEY, encodeText(JSON.stringify(list)));
|
||||||
},
|
},
|
||||||
deleteMeta: async (id) => {
|
deleteMeta: async (id) => {
|
||||||
@@ -1402,6 +1469,13 @@ export class AriaEngine implements IStorageEngine {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** v0.4.5: 是否启用页面化物理存储(默认 OPFS 后端启用,显式配置可覆盖) */
|
||||||
|
private isPageStorage(): boolean {
|
||||||
|
if (this.config.pageStorage === true) return true;
|
||||||
|
if (this.config.pageStorage === false) return false;
|
||||||
|
return this.config.storageBackend === 'opfs';
|
||||||
|
}
|
||||||
|
|
||||||
// =======================================================================
|
// =======================================================================
|
||||||
// WAL 恢复
|
// WAL 恢复
|
||||||
// =======================================================================
|
// =======================================================================
|
||||||
@@ -1753,40 +1827,6 @@ export class AriaEngine implements IStorageEngine {
|
|||||||
return { compactedLevels: 6, gcVersions: beforeGC };
|
return { compactedLevels: 6, gcVersions: beforeGC };
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* 查询优化器:估算各索引成本,选择最优方案
|
|
||||||
*/
|
|
||||||
estimateQueryCost(tableName: string, query: QueryPlan): { strategy: string; estimatedRows: number } {
|
|
||||||
const schema = this.schemas.get(tableName);
|
|
||||||
if (!schema || !query.where) return { strategy: 'full_scan', estimatedRows: 0 };
|
|
||||||
|
|
||||||
const pkCol = this.tablePKs.get(tableName)!;
|
|
||||||
for (const [col, condition] of Object.entries(query.where)) {
|
|
||||||
if (col === '$and' || col === '$or' || col === '$not') continue;
|
|
||||||
|
|
||||||
// PK 等值 → 最快,估计 1 行
|
|
||||||
if (col === pkCol && (typeof condition !== 'object' || (condition as any).$eq)) {
|
|
||||||
return { strategy: 'pk_lookup', estimatedRows: 1 };
|
|
||||||
}
|
|
||||||
|
|
||||||
// 索引列等值 → 快
|
|
||||||
const colDef = schema.columns[col];
|
|
||||||
if (colDef?.index || colDef?.unique) {
|
|
||||||
if (typeof condition !== 'object' || (condition as any).$eq) {
|
|
||||||
return { strategy: `index_eq:${col}`, estimatedRows: 1 };
|
|
||||||
}
|
|
||||||
if ((condition as any).$in && Array.isArray((condition as any).$in)) {
|
|
||||||
return { strategy: `index_in:${col}`, estimatedRows: (condition as any).$in.length };
|
|
||||||
}
|
|
||||||
if ((condition as any).$gt || (condition as any).$lt || (condition as any).$gte || (condition as any).$lte) {
|
|
||||||
return { strategy: `index_range:${col}`, estimatedRows: 100 };
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return { strategy: 'full_scan', estimatedRows: 1000 };
|
|
||||||
}
|
|
||||||
|
|
||||||
private ensureOpen(): void {
|
private ensureOpen(): void {
|
||||||
if (!this.opened) throw new DatabaseError('AriaEngine not opened', 'DB_NOT_OPEN');
|
if (!this.opened) throw new DatabaseError('AriaEngine not opened', 'DB_NOT_OPEN');
|
||||||
}
|
}
|
||||||
@@ -1806,23 +1846,4 @@ export class AriaEngine implements IStorageEngine {
|
|||||||
throw new DatabaseError(`Table "${tableName}" does not exist`, 'TABLE_NOT_FOUND');
|
throw new DatabaseError(`Table "${tableName}" does not exist`, 'TABLE_NOT_FOUND');
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Get the number of WAL records stored */
|
|
||||||
private async getWALCount(): Promise<number> {
|
|
||||||
const raw = await this.backend.read('__wal_count');
|
|
||||||
if (!raw) return 0;
|
|
||||||
try {
|
|
||||||
const dec = new TextDecoder();
|
|
||||||
return parseInt(dec.decode(raw), 10) || 0;
|
|
||||||
} catch {
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Set the number of WAL records stored */
|
|
||||||
private async setWALCount(count: number): Promise<void> {
|
|
||||||
const enc = new TextEncoder();
|
|
||||||
const buf = enc.encode(String(count)).buffer;
|
|
||||||
await this.backend.write('__wal_count', buf);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -67,16 +67,6 @@ export class BloomFilter {
|
|||||||
return this.bits;
|
return this.bits;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** bit 数组大小 */
|
|
||||||
getBitSize(): number {
|
|
||||||
return this.bits.byteLength * 8;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** 已插入 key 数量 */
|
|
||||||
getInsertedCount(): number {
|
|
||||||
return this._inserted;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** hash 函数数量 */
|
/** hash 函数数量 */
|
||||||
getHashCount(): number {
|
getHashCount(): number {
|
||||||
return this.numHashes;
|
return this.numHashes;
|
||||||
|
|||||||
@@ -322,7 +322,8 @@ export class LSM {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
for (const id of toLoad) {
|
for (const id of toLoad) {
|
||||||
await this.preloadSSTable(id);
|
const meta = this.findMetaById(id);
|
||||||
|
await this.preloadSSTable(id, meta);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -345,10 +346,21 @@ export class LSM {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
for (const id of toLoad) {
|
for (const id of toLoad) {
|
||||||
await this.preloadSSTable(id);
|
const meta = this.findMetaById(id);
|
||||||
|
await this.preloadSSTable(id, meta);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** 按 id 查找 SSTable 元数据(prefetch 预加载用) */
|
||||||
|
private findMetaById(id: number): SSTableMeta | undefined {
|
||||||
|
for (let level = 0; level < MAX_LSM_LEVELS; level++) {
|
||||||
|
for (const meta of this.levels[level]) {
|
||||||
|
if (meta.id === id) return meta;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
|
||||||
get(key: string): Record<string, unknown> | null {
|
get(key: string): Record<string, unknown> | null {
|
||||||
// 1. 活跃 MemTable
|
// 1. 活跃 MemTable
|
||||||
let result = this.memtable.get(key);
|
let result = this.memtable.get(key);
|
||||||
@@ -491,6 +503,10 @@ export class LSM {
|
|||||||
let reader: SSTableReader;
|
let reader: SSTableReader;
|
||||||
try {
|
try {
|
||||||
reader = new SSTableReader(data, meta);
|
reader = new SSTableReader(data, meta);
|
||||||
|
if (!reader.verifyChecksum()) {
|
||||||
|
await this.dropInvalidSSTable(meta);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
} catch {
|
} catch {
|
||||||
await this.dropInvalidSSTable(meta);
|
await this.dropInvalidSSTable(meta);
|
||||||
continue;
|
continue;
|
||||||
@@ -603,10 +619,6 @@ export class LSM {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
isInitialized(): boolean {
|
|
||||||
return this.initialized;
|
|
||||||
}
|
|
||||||
|
|
||||||
// =======================================================================
|
// =======================================================================
|
||||||
// 内部
|
// 内部
|
||||||
// =======================================================================
|
// =======================================================================
|
||||||
@@ -636,6 +648,7 @@ export class LSM {
|
|||||||
* v0.4.2-fix: 校验单个 SSTable 的完整性。
|
* v0.4.2-fix: 校验单个 SSTable 的完整性。
|
||||||
* - 文件不存在 → 清理 meta,返回 false
|
* - 文件不存在 → 清理 meta,返回 false
|
||||||
* - 文件过小/魔数错误/索引越界(残缺写入产物)→ 清理 meta,返回 false
|
* - 文件过小/魔数错误/索引越界(残缺写入产物)→ 清理 meta,返回 false
|
||||||
|
* - v0.4.5: 整文件 CRC-32 校验失败(数据腐坏)→ 清理 meta,返回 false
|
||||||
* 校验通过的数据不缓存(保持内存预算),读路径按需预加载。
|
* 校验通过的数据不缓存(保持内存预算),读路径按需预加载。
|
||||||
*/
|
*/
|
||||||
private async validateSSTable(meta: SSTableMeta): Promise<boolean> {
|
private async validateSSTable(meta: SSTableMeta): Promise<boolean> {
|
||||||
@@ -650,7 +663,11 @@ export class LSM {
|
|||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
try {
|
try {
|
||||||
new SSTableReader(data, meta);
|
const reader = new SSTableReader(data, meta);
|
||||||
|
if (!reader.verifyChecksum()) {
|
||||||
|
this.dropInvalidSSTable(meta);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
} catch {
|
} catch {
|
||||||
this.dropInvalidSSTable(meta);
|
this.dropInvalidSSTable(meta);
|
||||||
return false;
|
return false;
|
||||||
@@ -703,12 +720,24 @@ export class LSM {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/** 预加载 SSTable 到缓存(受 cacheLimitBytes 上限约束) */
|
/** 预加载 SSTable 到缓存(受 cacheLimitBytes 上限约束) */
|
||||||
async preloadSSTable(id: number): Promise<void> {
|
async preloadSSTable(id: number, meta?: SSTableMeta): Promise<void> {
|
||||||
if (this.sstableCache.has(id)) return;
|
if (this.sstableCache.has(id)) return;
|
||||||
const data = await this.sstableStore.load(id);
|
const data = await this.sstableStore.load(id);
|
||||||
if (data) {
|
if (!data) return;
|
||||||
this.cacheSSTable(id, data);
|
// v0.4.5: 运行期加载同样校验整文件 CRC-32,损坏文件不缓存并清理(自愈)
|
||||||
|
if (meta) {
|
||||||
|
try {
|
||||||
|
const reader = new SSTableReader(data, meta);
|
||||||
|
if (!reader.verifyChecksum()) {
|
||||||
|
await this.dropInvalidSSTable(meta);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
await this.dropInvalidSSTable(meta);
|
||||||
|
return;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
this.cacheSSTable(id, data);
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 写入缓存。注意:不在加载时立即驱逐,避免破坏正在进行的同步扫描 */
|
/** 写入缓存。注意:不在加载时立即驱逐,避免破坏正在进行的同步扫描 */
|
||||||
@@ -736,13 +765,4 @@ export class LSM {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 获取当前 SSTable 缓存大小(字节) */
|
|
||||||
getCacheSize(): number {
|
|
||||||
return this.cacheSize;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** 获取 SSTable 缓存容量上限(字节) */
|
|
||||||
getCacheLimit(): number {
|
|
||||||
return this.cacheLimitBytes;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -441,11 +441,6 @@ export class MemTable {
|
|||||||
this._estimatedSize = 0;
|
this._estimatedSize = 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 检查 key 是否存在 */
|
|
||||||
contains(key: string): boolean {
|
|
||||||
return this.tree.find(key) !== null;
|
|
||||||
}
|
|
||||||
|
|
||||||
// -----------------------------------------------------------------------
|
// -----------------------------------------------------------------------
|
||||||
// 内部
|
// 内部
|
||||||
// -----------------------------------------------------------------------
|
// -----------------------------------------------------------------------
|
||||||
|
|||||||
@@ -35,29 +35,6 @@ export class ArrayEntrySource implements EntrySource {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 回调数据源的迭代器 */
|
|
||||||
export class CallbackEntrySource implements EntrySource {
|
|
||||||
private items: [string, Record<string, unknown>][] = [];
|
|
||||||
private index = 0;
|
|
||||||
private consumed = false;
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @param producer 产生所有条目的回调
|
|
||||||
*/
|
|
||||||
constructor(producer: (cb: (key: string, value: Record<string, unknown>) => void) => void) {
|
|
||||||
producer((key, value) => this.items.push([key, value]));
|
|
||||||
}
|
|
||||||
|
|
||||||
next(): [string, Record<string, unknown>] | null {
|
|
||||||
if (this.index >= this.items.length) return null;
|
|
||||||
return this.items[this.index++];
|
|
||||||
}
|
|
||||||
|
|
||||||
reset(): void {
|
|
||||||
this.index = 0;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// Heap 节点(用于多路归并)
|
// Heap 节点(用于多路归并)
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|||||||
@@ -8,6 +8,7 @@
|
|||||||
|
|
||||||
import type { IndexEntry, SSTableMeta } from '../types';
|
import type { IndexEntry, SSTableMeta } from '../types';
|
||||||
import { BloomFilter } from './bloom';
|
import { BloomFilter } from './bloom';
|
||||||
|
import { crc32 } from '../crc32';
|
||||||
import { SSTABLE_MAGIC_V1, SSTABLE_MAGIC_V2 } from './sstable_builder';
|
import { SSTABLE_MAGIC_V1, SSTABLE_MAGIC_V2 } from './sstable_builder';
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -23,6 +24,8 @@ export class SSTableReader {
|
|||||||
private bloomFilter: BloomFilter | null = null;
|
private bloomFilter: BloomFilter | null = null;
|
||||||
/** 格式版本:1 = u16 长度字段(旧),2 = u32 长度字段(v0.4.4) */
|
/** 格式版本:1 = u16 长度字段(旧),2 = u32 长度字段(v0.4.4) */
|
||||||
private format: 1 | 2 = 2;
|
private format: 1 | 2 = 2;
|
||||||
|
/** footer 中存储的 checksum(0 = 旧版无校验文件) */
|
||||||
|
private storedChecksum = 0;
|
||||||
|
|
||||||
constructor(data: Uint8Array, meta: SSTableMeta) {
|
constructor(data: Uint8Array, meta: SSTableMeta) {
|
||||||
this.data = data;
|
this.data = data;
|
||||||
@@ -31,6 +34,17 @@ export class SSTableReader {
|
|||||||
this.parseFooter();
|
this.parseFooter();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 校验整文件 CRC-32(覆盖除 checksum 字段外的全部字节)。
|
||||||
|
* checksum === 0 表示旧版文件(v1 / v0.4.4 及更早的 v2),跳过校验返回 true(兼容)。
|
||||||
|
*/
|
||||||
|
verifyChecksum(): boolean {
|
||||||
|
if (this.storedChecksum === 0) return true;
|
||||||
|
if (this.data.byteLength < 4) return false;
|
||||||
|
const computed = crc32(this.data.subarray(0, this.data.byteLength - 4));
|
||||||
|
return computed === this.storedChecksum;
|
||||||
|
}
|
||||||
|
|
||||||
/** 长度字段宽度:v2 = 4 字节 u32,v1 = 2 字节 u16 */
|
/** 长度字段宽度:v2 = 4 字节 u32,v1 = 2 字节 u16 */
|
||||||
private lenFieldSize(): number {
|
private lenFieldSize(): number {
|
||||||
return this.format === 2 ? 4 : 2;
|
return this.format === 2 ? 4 : 2;
|
||||||
@@ -170,16 +184,6 @@ export class SSTableReader {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 获取元数据 */
|
|
||||||
getMeta(): SSTableMeta {
|
|
||||||
return this.meta;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** 获取索引条目数 */
|
|
||||||
getIndexBlockCount(): number {
|
|
||||||
return this.indexEntries.length;
|
|
||||||
}
|
|
||||||
|
|
||||||
// -----------------------------------------------------------------------
|
// -----------------------------------------------------------------------
|
||||||
// 内部
|
// 内部
|
||||||
// -----------------------------------------------------------------------
|
// -----------------------------------------------------------------------
|
||||||
@@ -207,6 +211,7 @@ export class SSTableReader {
|
|||||||
const bloomSize = this.view.getUint32(footerOffset + 12, false);
|
const bloomSize = this.view.getUint32(footerOffset + 12, false);
|
||||||
const bloomHashCount = this.view.getUint32(footerOffset + 16, false);
|
const bloomHashCount = this.view.getUint32(footerOffset + 16, false);
|
||||||
this.entryCount = this.view.getUint32(footerOffset + 20, false);
|
this.entryCount = this.view.getUint32(footerOffset + 20, false);
|
||||||
|
this.storedChecksum = this.view.getUint32(footerOffset + 28, false);
|
||||||
|
|
||||||
// v0.4.1-fix: 完整性校验 — 索引块必须完全落在文件内,否则视为残缺文件跳过
|
// v0.4.1-fix: 完整性校验 — 索引块必须完全落在文件内,否则视为残缺文件跳过
|
||||||
if (indexOffset + 4 > this.data.byteLength || indexOffset + indexSize > this.data.byteLength) {
|
if (indexOffset + 4 > this.data.byteLength || indexOffset + indexSize > this.data.byteLength) {
|
||||||
|
|||||||
@@ -34,6 +34,7 @@
|
|||||||
*/
|
*/
|
||||||
|
|
||||||
import { BloomFilter } from './bloom';
|
import { BloomFilter } from './bloom';
|
||||||
|
import { crc32 } from '../crc32';
|
||||||
import type { IndexEntry } from '../types';
|
import type { IndexEntry } from '../types';
|
||||||
|
|
||||||
/** v1 格式魔数("SSTB",u16 长度字段,兼容旧文件读取) */
|
/** v1 格式魔数("SSTB",u16 长度字段,兼容旧文件读取) */
|
||||||
@@ -137,7 +138,15 @@ export class SSTableBuilder {
|
|||||||
view.setUint32(footerOffset + 16, bloomFilter.getHashCount(), false);
|
view.setUint32(footerOffset + 16, bloomFilter.getHashCount(), false);
|
||||||
view.setUint32(footerOffset + 20, this.entries.length, false);
|
view.setUint32(footerOffset + 20, this.entries.length, false);
|
||||||
view.setUint32(footerOffset + 24, SSTABLE_MAGIC_V2, false);
|
view.setUint32(footerOffset + 24, SSTABLE_MAGIC_V2, false);
|
||||||
view.setUint32(footerOffset + 28, 0, false); // checksum (simplified: 0)
|
// checksum 字段先写 0,全部字节就绪后计算整文件 CRC32 再回填
|
||||||
|
view.setUint32(footerOffset + 28, 0, false);
|
||||||
|
|
||||||
|
// v0.4.5: 真实 CRC-32 校验和 — 覆盖除自身(最后 4 字节)外的全部内容。
|
||||||
|
// checksum 永远非 0(计算结果为 0 时用 1 代替),读取端以 0 识别旧版无校验文件
|
||||||
|
const all = new Uint8Array(buf);
|
||||||
|
let checksum = crc32(all.subarray(0, all.byteLength - 4));
|
||||||
|
if (checksum === 0) checksum = 1;
|
||||||
|
view.setUint32(footerOffset + 28, checksum, false);
|
||||||
|
|
||||||
return {
|
return {
|
||||||
sstableData: new Uint8Array(buf),
|
sstableData: new Uint8Array(buf),
|
||||||
|
|||||||
@@ -0,0 +1,92 @@
|
|||||||
|
/**
|
||||||
|
* AriaEngine Database Lock — 多标签页独占锁(Web Locks API)
|
||||||
|
* @module engine/aria/locks
|
||||||
|
*
|
||||||
|
* v0.4.5: OPFS 等无事务后端缺乏多标签页并发协调,多个标签页同时打开同一库
|
||||||
|
* 会导致写竞态与数据损坏。用 Web Locks API(Chrome 69+ / Firefox 96+ / Safari 15.4+)
|
||||||
|
* 获取库级排他锁:
|
||||||
|
* - ifAvailable 模式:锁被其他标签页持有 → 立即抛 ARIA_LOCKED(不排队挂起)
|
||||||
|
* - 持锁期间回调挂起,close() 时释放
|
||||||
|
* - 浏览器不支持 navigator.locks → 返回 false(如实降级:无并发保护,文档注明)
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { DatabaseError } from '../../constants';
|
||||||
|
|
||||||
|
/** Web Locks 锁名(库级排他) */
|
||||||
|
export function lockName(dbName: string): string {
|
||||||
|
return `metona-sqlark:${dbName}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
export class DatabaseLock {
|
||||||
|
private acquired = false;
|
||||||
|
private supported = false;
|
||||||
|
private releaseResolve: (() => void) | null = null;
|
||||||
|
private releasePromise: Promise<void> | null = null;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 尝试获取独占锁。
|
||||||
|
* @returns true = 已持锁;false = 环境不支持 Web Locks(无并发保护,调用方可警告)
|
||||||
|
* @throws ARIA_LOCKED 锁被其他标签页持有
|
||||||
|
*/
|
||||||
|
async acquire(dbName: string): Promise<boolean> {
|
||||||
|
const nav = (globalThis as { navigator?: unknown }).navigator as {
|
||||||
|
locks?: { request: (name: string, opts: unknown, cb: (lock: { name: string } | null) => Promise<void> | void) => Promise<void> };
|
||||||
|
} | undefined;
|
||||||
|
const lockManager = nav?.locks;
|
||||||
|
if (!lockManager || typeof lockManager.request !== 'function') {
|
||||||
|
this.supported = false;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
this.supported = true;
|
||||||
|
|
||||||
|
await new Promise<void>((resolve, reject) => {
|
||||||
|
// 注意:必须直接调用(不能解构 request —— LockManager 方法依赖 this 绑定)
|
||||||
|
const request = lockManager.request.bind(lockManager);
|
||||||
|
request(
|
||||||
|
lockName(dbName),
|
||||||
|
{ ifAvailable: true, mode: 'exclusive' },
|
||||||
|
async (lock) => {
|
||||||
|
if (!lock) {
|
||||||
|
reject(new DatabaseError(
|
||||||
|
`Database "${dbName}" is already open in another tab (locked)`,
|
||||||
|
'ARIA_LOCKED',
|
||||||
|
));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
this.acquired = true;
|
||||||
|
const releasePromise = new Promise<void>((res) => { this.releaseResolve = res; });
|
||||||
|
this.releasePromise = releasePromise;
|
||||||
|
// 锁已获取:acquire 返回(open 流程继续)
|
||||||
|
resolve();
|
||||||
|
// 回调挂起:保持锁直到 release() 触发
|
||||||
|
await releasePromise;
|
||||||
|
},
|
||||||
|
);
|
||||||
|
});
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 释放锁(等待回调真正结束,保证锁已归还) */
|
||||||
|
async release(): Promise<void> {
|
||||||
|
if (!this.acquired) return;
|
||||||
|
if (this.releaseResolve) {
|
||||||
|
const res = this.releaseResolve;
|
||||||
|
const p = this.releasePromise!;
|
||||||
|
this.releaseResolve = null;
|
||||||
|
this.releasePromise = null;
|
||||||
|
res();
|
||||||
|
try { await p; } catch { /* 释放过程异常不阻塞 */ }
|
||||||
|
}
|
||||||
|
this.acquired = false;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 是否已持锁 */
|
||||||
|
isAcquired(): boolean {
|
||||||
|
return this.acquired;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 环境是否支持 Web Locks */
|
||||||
|
isSupported(): boolean {
|
||||||
|
return this.supported;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,21 +1,15 @@
|
|||||||
/**
|
/**
|
||||||
* AriaEngine Page Format — 页面格式整合层
|
* AriaEngine Page Format — 页面创建
|
||||||
* @module engine/aria/page/format
|
* @module engine/aria/page/format
|
||||||
*
|
*
|
||||||
* 组合 Header / Slot / Tuple 操作,提供统一的页面管理接口。
|
* v0.4.5: 仅保留生产代码实际使用的页面创建逻辑。
|
||||||
|
* (Slot Directory / Tuple 编解码曾为行级页面存储设计,但从未接入 LSM 主路径,
|
||||||
|
* 属"宣称页面式但未实现"的半成品,已删除 —— SSTable 以 4KB 页面承载,
|
||||||
|
* 页面内容为原始字节切片,由 PageSSTableStore 管理)
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { PAGE_SIZE, PageType, type PageHandle } from '../types';
|
import { PAGE_SIZE, PageType, type PageHandle } from '../types';
|
||||||
import {
|
import { initPageHeader } from './header';
|
||||||
initPageHeader,
|
|
||||||
getSlotCount,
|
|
||||||
} from './header';
|
|
||||||
import { allocateSlot, freeSlot, readSlotData } from './slot';
|
|
||||||
import { encodeTuple, decodeTuple } from './tuple';
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// 页面创建
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
/** 创建一个新的空页面 */
|
/** 创建一个新的空页面 */
|
||||||
export function createPage(pageId: number, type: PageType): PageHandle {
|
export function createPage(pageId: number, type: PageType): PageHandle {
|
||||||
@@ -32,133 +26,3 @@ export function createPage(pageId: number, type: PageType): PageHandle {
|
|||||||
lastAccess: Date.now(),
|
lastAccess: Date.now(),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 从 ArrayBuffer 恢复页面句柄 */
|
|
||||||
export function pageFromBuffer(
|
|
||||||
pageId: number,
|
|
||||||
buffer: ArrayBuffer,
|
|
||||||
): PageHandle {
|
|
||||||
return {
|
|
||||||
pageId,
|
|
||||||
type: new DataView(buffer).getUint8(4) as PageType,
|
|
||||||
data: buffer,
|
|
||||||
dirty: false,
|
|
||||||
pins: 0,
|
|
||||||
prev: null,
|
|
||||||
next: null,
|
|
||||||
lastAccess: Date.now(),
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// 行操作(页面级)
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
/**
|
|
||||||
* 插入一行到页面,返回 slot 索引,空间不足返回 -1。
|
|
||||||
*/
|
|
||||||
export function pageInsertRow(
|
|
||||||
page: PageHandle,
|
|
||||||
row: Record<string, unknown>,
|
|
||||||
columnOrder: string[],
|
|
||||||
columnTypes: Record<string, string>,
|
|
||||||
): number {
|
|
||||||
const encoded = encodeTuple(row, columnOrder, columnTypes);
|
|
||||||
const slotIdx = allocateSlot(page.data, encoded);
|
|
||||||
if (slotIdx >= 0) {
|
|
||||||
page.dirty = true;
|
|
||||||
page.lastAccess = Date.now();
|
|
||||||
}
|
|
||||||
return slotIdx;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* 读取指定 slot 的行数据并解码。
|
|
||||||
*/
|
|
||||||
export function pageReadRow(
|
|
||||||
page: PageHandle,
|
|
||||||
slotIndex: number,
|
|
||||||
columnOrder: string[],
|
|
||||||
columnTypes: Record<string, string>,
|
|
||||||
): Record<string, unknown> | null {
|
|
||||||
const slotData = readSlotData(page.data, slotIndex);
|
|
||||||
if (!slotData) return null;
|
|
||||||
return decodeTuple(slotData, columnOrder, columnTypes);
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* 读取页面中的所有行。
|
|
||||||
*/
|
|
||||||
export function pageReadAllRows(
|
|
||||||
page: PageHandle,
|
|
||||||
columnOrder: string[],
|
|
||||||
columnTypes: Record<string, string>,
|
|
||||||
): Record<string, unknown>[] {
|
|
||||||
const rows: Record<string, unknown>[] = [];
|
|
||||||
const slotCount = getSlotCount(page.data);
|
|
||||||
for (let i = 0; i < slotCount; i++) {
|
|
||||||
const row = pageReadRow(page, i, columnOrder, columnTypes);
|
|
||||||
if (row) rows.push(row);
|
|
||||||
}
|
|
||||||
return rows;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* 标记 slot 为已删除。
|
|
||||||
*/
|
|
||||||
export function pageDeleteRow(page: PageHandle, slotIndex: number): void {
|
|
||||||
freeSlot(page.data, slotIndex);
|
|
||||||
page.dirty = true;
|
|
||||||
page.lastAccess = Date.now();
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* 重写指定 slot 的行数据。
|
|
||||||
*/
|
|
||||||
export function pageUpdateRow(
|
|
||||||
page: PageHandle,
|
|
||||||
slotIndex: number,
|
|
||||||
row: Record<string, unknown>,
|
|
||||||
columnOrder: string[],
|
|
||||||
columnTypes: Record<string, string>,
|
|
||||||
): void {
|
|
||||||
// 先标记旧 slot 为删除
|
|
||||||
pageDeleteRow(page, slotIndex);
|
|
||||||
// 分配新 slot,可能会在不同位置
|
|
||||||
pageInsertRow(page, row, columnOrder, columnTypes);
|
|
||||||
// 注意:调用者需要自行维护 slot index → pk 的映射
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// 校验和
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
/** 简单 CRC32(使用预先计算的查找表简化) */
|
|
||||||
export function computeChecksum(data: ArrayBuffer): number {
|
|
||||||
const view = new Uint8Array(data);
|
|
||||||
let hash = 0;
|
|
||||||
for (let i = 0; i < view.byteLength; i++) {
|
|
||||||
hash = ((hash << 5) - hash + view[i]) | 0;
|
|
||||||
}
|
|
||||||
return hash >>> 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** 更新页面的校验和字段 */
|
|
||||||
export function updateChecksum(page: PageHandle): void {
|
|
||||||
// 先清零校验和字段
|
|
||||||
const view = new DataView(page.data);
|
|
||||||
view.setUint32(11, 0, false);
|
|
||||||
// 计算校验和
|
|
||||||
const cksum = computeChecksum(page.data);
|
|
||||||
view.setUint32(11, cksum, false);
|
|
||||||
}
|
|
||||||
|
|
||||||
/** 验证页面校验和 */
|
|
||||||
export function verifyChecksum(page: PageHandle): boolean {
|
|
||||||
const stored = new DataView(page.data).getUint32(11, false);
|
|
||||||
// 临时清零
|
|
||||||
new DataView(page.data).setUint32(11, 0, false);
|
|
||||||
const computed = computeChecksum(page.data);
|
|
||||||
new DataView(page.data).setUint32(11, stored, false);
|
|
||||||
return stored === computed;
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -1,17 +1,9 @@
|
|||||||
/**
|
/**
|
||||||
* AriaEngine Page Header — 页面头部编解码
|
* AriaEngine Page Header — 页面头初始化
|
||||||
* @module engine/aria/page/header
|
* @module engine/aria/page/header
|
||||||
*/
|
*
|
||||||
|
* v0.4.5: 仅保留生产代码实际使用的 initPageHeader。
|
||||||
import { PAGE_HEADER_SIZE, PageType, type PageHeader } from '../types';
|
* 页面头部布局(大端序):
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// 编码
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
/**
|
|
||||||
* 将 PageHeader 编码写入 ArrayBuffer 的前 16 字节。
|
|
||||||
* 布局(大端序):
|
|
||||||
* [0-3] page_id u32
|
* [0-3] page_id u32
|
||||||
* [4] type u8
|
* [4] type u8
|
||||||
* [5-6] free_start u16
|
* [5-6] free_start u16
|
||||||
@@ -19,32 +11,12 @@ import { PAGE_HEADER_SIZE, PageType, type PageHeader } from '../types';
|
|||||||
* [9-10] slot_count u16
|
* [9-10] slot_count u16
|
||||||
* [11-14] checksum u32
|
* [11-14] checksum u32
|
||||||
* [15] reserved u8
|
* [15] reserved u8
|
||||||
|
*
|
||||||
|
* 注:free_start/free_end/slot_count/checksum 字段为历史行级页面格式遗留,
|
||||||
|
* 页面化 SSTable 使用页面原始字节区(跳过头部),字段保留以维持 16 字节头部对齐。
|
||||||
*/
|
*/
|
||||||
export function encodePageHeader(header: PageHeader, buf: ArrayBuffer): void {
|
|
||||||
const view = new DataView(buf);
|
|
||||||
view.setUint32(0, header.pageId, false);
|
|
||||||
view.setUint8(4, header.type);
|
|
||||||
view.setUint16(5, header.freeStart, false);
|
|
||||||
view.setUint16(7, header.freeEnd, false);
|
|
||||||
view.setUint16(9, header.slotCount, false);
|
|
||||||
view.setUint32(11, header.checksum, false);
|
|
||||||
view.setUint8(15, 0); // reserved
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
import { PAGE_HEADER_SIZE } from '../types';
|
||||||
* 从 ArrayBuffer 解码 PageHeader。
|
|
||||||
*/
|
|
||||||
export function decodePageHeader(buf: ArrayBuffer): PageHeader {
|
|
||||||
const view = new DataView(buf);
|
|
||||||
return {
|
|
||||||
pageId: view.getUint32(0, false),
|
|
||||||
type: view.getUint8(4) as PageType,
|
|
||||||
freeStart: view.getUint16(5, false),
|
|
||||||
freeEnd: view.getUint16(7, false),
|
|
||||||
slotCount: view.getUint16(9, false),
|
|
||||||
checksum: view.getUint32(11, false),
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* 初始化新页面的 Header。
|
* 初始化新页面的 Header。
|
||||||
@@ -52,7 +24,7 @@ export function decodePageHeader(buf: ArrayBuffer): PageHeader {
|
|||||||
export function initPageHeader(
|
export function initPageHeader(
|
||||||
buf: ArrayBuffer,
|
buf: ArrayBuffer,
|
||||||
pageId: number,
|
pageId: number,
|
||||||
type: PageType,
|
type: number,
|
||||||
): void {
|
): void {
|
||||||
const view = new DataView(buf);
|
const view = new DataView(buf);
|
||||||
view.setUint32(0, pageId, false);
|
view.setUint32(0, pageId, false);
|
||||||
@@ -63,35 +35,3 @@ export function initPageHeader(
|
|||||||
view.setUint32(11, 0, false); // checksum = 0
|
view.setUint32(11, 0, false); // checksum = 0
|
||||||
view.setUint8(15, 0);
|
view.setUint8(15, 0);
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// 从 Buffer 中提取 Header 字段的辅助函数
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
export function getPageType(buf: ArrayBuffer): PageType {
|
|
||||||
return new DataView(buf).getUint8(4) as PageType;
|
|
||||||
}
|
|
||||||
|
|
||||||
export function getPageId(buf: ArrayBuffer): number {
|
|
||||||
return new DataView(buf).getUint32(0, false);
|
|
||||||
}
|
|
||||||
|
|
||||||
export function getSlotCount(buf: ArrayBuffer): number {
|
|
||||||
return new DataView(buf).getUint16(9, false);
|
|
||||||
}
|
|
||||||
|
|
||||||
export function getFreeStart(buf: ArrayBuffer): number {
|
|
||||||
return new DataView(buf).getUint16(5, false);
|
|
||||||
}
|
|
||||||
|
|
||||||
export function setFreeStart(buf: ArrayBuffer, val: number): void {
|
|
||||||
new DataView(buf).setUint16(5, val, false);
|
|
||||||
}
|
|
||||||
|
|
||||||
export function setFreeEnd(buf: ArrayBuffer, val: number): void {
|
|
||||||
new DataView(buf).setUint16(7, val, false);
|
|
||||||
}
|
|
||||||
|
|
||||||
export function setSlotCount(buf: ArrayBuffer, val: number): void {
|
|
||||||
new DataView(buf).setUint16(9, val, false);
|
|
||||||
}
|
|
||||||
|
|||||||
@@ -1,184 +0,0 @@
|
|||||||
/**
|
|
||||||
* AriaEngine Slot Directory — 页面内行槽位管理
|
|
||||||
* @module engine/aria/page/slot
|
|
||||||
*
|
|
||||||
* Slot 目录从页面 Header 之后向下增长,每条记录 4 字节。
|
|
||||||
*/
|
|
||||||
|
|
||||||
import {
|
|
||||||
PAGE_HEADER_SIZE,
|
|
||||||
SLOT_ENTRY_SIZE,
|
|
||||||
PAGE_SIZE,
|
|
||||||
type SlotEntry,
|
|
||||||
} from '../types';
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// Slot 读写
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
/**
|
|
||||||
* 读取 slot 号对应的 SlotEntry。
|
|
||||||
*/
|
|
||||||
export function getSlotEntry(buf: ArrayBuffer, slotIndex: number): SlotEntry {
|
|
||||||
const offset = PAGE_HEADER_SIZE + slotIndex * SLOT_ENTRY_SIZE;
|
|
||||||
const view = new DataView(buf);
|
|
||||||
return {
|
|
||||||
offset: view.getUint16(offset, false),
|
|
||||||
length: view.getUint16(offset + 2, false),
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* 写入 slot 号对应的 SlotEntry。
|
|
||||||
*/
|
|
||||||
export function setSlotEntry(
|
|
||||||
buf: ArrayBuffer,
|
|
||||||
slotIndex: number,
|
|
||||||
entry: SlotEntry,
|
|
||||||
): void {
|
|
||||||
const offset = PAGE_HEADER_SIZE + slotIndex * SLOT_ENTRY_SIZE;
|
|
||||||
const view = new DataView(buf);
|
|
||||||
view.setUint16(offset, entry.offset, false);
|
|
||||||
view.setUint16(offset + 2, entry.length, false);
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* 获取页面中所有 slot 条目。
|
|
||||||
*/
|
|
||||||
export function getAllSlots(
|
|
||||||
buf: ArrayBuffer,
|
|
||||||
slotCount: number,
|
|
||||||
): SlotEntry[] {
|
|
||||||
const entries: SlotEntry[] = [];
|
|
||||||
for (let i = 0; i < slotCount; i++) {
|
|
||||||
entries.push(getSlotEntry(buf, i));
|
|
||||||
}
|
|
||||||
return entries;
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// Slot 空间计算
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
/** 获取 slot 目录占用的总字节数 */
|
|
||||||
export function getSlotDirectorySize(slotCount: number): number {
|
|
||||||
return slotCount * SLOT_ENTRY_SIZE;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** 获取可用空闲空间(字节) */
|
|
||||||
export function getFreeSpace(buf: ArrayBuffer): number {
|
|
||||||
const view = new DataView(buf);
|
|
||||||
const freeStart = view.getUint16(5, false); // slot 区之后
|
|
||||||
const freeEnd = view.getUint16(7, false); // 数据区之前
|
|
||||||
return freeEnd - freeStart;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** 检查是否有足够空间存放长度为 len 的行 */
|
|
||||||
export function hasEnoughSpace(buf: ArrayBuffer, len: number): boolean {
|
|
||||||
const slotCount = new DataView(buf).getUint16(9, false);
|
|
||||||
const neededSlotSize = (slotCount + 1) * SLOT_ENTRY_SIZE;
|
|
||||||
const freeStart = PAGE_HEADER_SIZE + neededSlotSize;
|
|
||||||
const freeEnd = new DataView(buf).getUint16(7, false);
|
|
||||||
return freeEnd - freeStart >= len;
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// 插入 / 删除 slot
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
/**
|
|
||||||
* 在页面中分配一个 slot 并写入行数据。
|
|
||||||
* 返回分配的 slot 索引,失败返回 -1。
|
|
||||||
*/
|
|
||||||
export function allocateSlot(
|
|
||||||
buf: ArrayBuffer,
|
|
||||||
rowData: Uint8Array,
|
|
||||||
): number {
|
|
||||||
const view = new DataView(buf);
|
|
||||||
const slotCount = view.getUint16(9, false);
|
|
||||||
const neededSlotSpace = (slotCount + 1) * SLOT_ENTRY_SIZE;
|
|
||||||
const freeStart = PAGE_HEADER_SIZE + neededSlotSpace;
|
|
||||||
const freeEnd = view.getUint16(7, false);
|
|
||||||
|
|
||||||
if (freeEnd - freeStart < rowData.byteLength) {
|
|
||||||
return -1; // 空间不足
|
|
||||||
}
|
|
||||||
|
|
||||||
// 将数据放入页面底部
|
|
||||||
const dataOffset = freeEnd - rowData.byteLength;
|
|
||||||
const dest = new Uint8Array(buf, dataOffset, rowData.byteLength);
|
|
||||||
dest.set(rowData);
|
|
||||||
|
|
||||||
// 写入 slot 条目
|
|
||||||
setSlotEntry(buf, slotCount, { offset: dataOffset, length: rowData.byteLength });
|
|
||||||
|
|
||||||
// 更新 header
|
|
||||||
view.setUint16(5, freeStart, false); // freeStart
|
|
||||||
view.setUint16(7, dataOffset, false); // freeEnd
|
|
||||||
view.setUint16(9, slotCount + 1, false); // slotCount
|
|
||||||
|
|
||||||
return slotCount;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* 从页面中删除指定 slot 的数据(标记为无效,offset 置 0)。
|
|
||||||
* 注:简化实现,不做 slot 压缩。
|
|
||||||
*/
|
|
||||||
export function freeSlot(buf: ArrayBuffer, slotIndex: number): void {
|
|
||||||
setSlotEntry(buf, slotIndex, { offset: 0, length: 0 });
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* 压缩页面槽位:移除已删除 slot,整理碎片空间。
|
|
||||||
* 将有效数据紧凑排列,释放空洞。
|
|
||||||
*/
|
|
||||||
export function compactSlots(buf: ArrayBuffer): number {
|
|
||||||
const view = new DataView(buf);
|
|
||||||
const slotCount = view.getUint16(9, false);
|
|
||||||
if (slotCount === 0) return 0;
|
|
||||||
|
|
||||||
// 收集有效 slot(offset>0 的)
|
|
||||||
const validSlots: { index: number; offset: number; length: number; data: Uint8Array }[] = [];
|
|
||||||
for (let i = 0; i < slotCount; i++) {
|
|
||||||
const entry = getSlotEntry(buf, i);
|
|
||||||
if (entry.offset > 0 && entry.length > 0) {
|
|
||||||
const data = new Uint8Array(buf, entry.offset, entry.length);
|
|
||||||
validSlots.push({ index: i, offset: entry.offset, length: entry.length, data: new Uint8Array(data) });
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if (validSlots.length === slotCount) return 0; // 无碎片
|
|
||||||
|
|
||||||
// 从页面底部重新紧凑排列
|
|
||||||
let dataEnd = PAGE_SIZE;
|
|
||||||
const newSlots: { offset: number; length: number }[] = [];
|
|
||||||
|
|
||||||
for (let i = validSlots.length - 1; i >= 0; i--) {
|
|
||||||
const s = validSlots[i];
|
|
||||||
dataEnd -= s.length;
|
|
||||||
new Uint8Array(buf).set(s.data, dataEnd);
|
|
||||||
newSlots.unshift({ offset: dataEnd, length: s.length });
|
|
||||||
}
|
|
||||||
|
|
||||||
// 重写 slot directory
|
|
||||||
view.setUint16(9, validSlots.length, false); // slotCount
|
|
||||||
view.setUint16(7, dataEnd, false); // freeEnd
|
|
||||||
for (let i = 0; i < validSlots.length; i++) {
|
|
||||||
setSlotEntry(buf, i, newSlots[i]);
|
|
||||||
}
|
|
||||||
// 清除剩余 slot 条目
|
|
||||||
for (let i = validSlots.length; i < slotCount; i++) {
|
|
||||||
setSlotEntry(buf, i, { offset: 0, length: 0 });
|
|
||||||
}
|
|
||||||
|
|
||||||
return slotCount - validSlots.length; // 回收的 slot 数量
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* 读取指定 slot 的行数据。
|
|
||||||
*/
|
|
||||||
export function readSlotData(buf: ArrayBuffer, slotIndex: number): Uint8Array | null {
|
|
||||||
const entry = getSlotEntry(buf, slotIndex);
|
|
||||||
if (entry.offset === 0 || entry.length === 0) return null;
|
|
||||||
return new Uint8Array(buf, entry.offset, entry.length);
|
|
||||||
}
|
|
||||||
@@ -1,251 +0,0 @@
|
|||||||
/**
|
|
||||||
* AriaEngine Tuple Codec — 行数据的二进制编解码
|
|
||||||
* @module engine/aria/page/tuple
|
|
||||||
*
|
|
||||||
* 将 Record<string, unknown> 编码为紧凑的二进制格式。
|
|
||||||
*
|
|
||||||
* 格式:
|
|
||||||
* [null bitmap: ceil(colCount/8) bytes]
|
|
||||||
* [column 1 data]
|
|
||||||
* [column 2 data]
|
|
||||||
* ...
|
|
||||||
*
|
|
||||||
* 每列:
|
|
||||||
* type tag (u8) + data
|
|
||||||
* - STRING: [len: u16][UTF-8 bytes]
|
|
||||||
* - NUMBER: [f64: 8 bytes]
|
|
||||||
* - BOOLEAN: [u8: 1 byte]
|
|
||||||
* - DATE: [f64: 8 bytes] (epoch ms)
|
|
||||||
* - JSON: [len: u16][UTF-8 bytes]
|
|
||||||
* - NULL: (no data, just the tag)
|
|
||||||
*/
|
|
||||||
|
|
||||||
import { ColumnEncoding } from '../types';
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// 编码
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
/**
|
|
||||||
* 将行数据编码为二进制字节数组。
|
|
||||||
* @param row 行数据
|
|
||||||
* @param columnOrder 列名顺序列表(决定编码顺序)
|
|
||||||
* @param columnTypes 列名 → FieldType 映射
|
|
||||||
*/
|
|
||||||
export function encodeTuple(
|
|
||||||
row: Record<string, unknown>,
|
|
||||||
columnOrder: string[],
|
|
||||||
columnTypes: Record<string, string>,
|
|
||||||
): Uint8Array {
|
|
||||||
// 先计算总大小
|
|
||||||
let size = 0;
|
|
||||||
const nullBitmapBytes = Math.ceil(columnOrder.length / 8);
|
|
||||||
size += nullBitmapBytes;
|
|
||||||
|
|
||||||
// 预计算每列编码后的字节
|
|
||||||
const colData: (Uint8Array | null)[] = [];
|
|
||||||
for (let i = 0; i < columnOrder.length; i++) {
|
|
||||||
const col = columnOrder[i];
|
|
||||||
const val = row[col];
|
|
||||||
const encoded = encodeColumn(val, columnTypes[col] ?? 'string');
|
|
||||||
colData.push(encoded);
|
|
||||||
if (encoded) {
|
|
||||||
size += 1 + encoded.byteLength; // tag + data
|
|
||||||
} else {
|
|
||||||
size += 1; // just the NULL tag
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
const buf = new Uint8Array(size);
|
|
||||||
const view = new DataView(buf.buffer, buf.byteOffset, buf.byteLength);
|
|
||||||
let offset = 0;
|
|
||||||
|
|
||||||
// Null bitmap
|
|
||||||
const nullBitmap = new Uint8Array(nullBitmapBytes);
|
|
||||||
for (let i = 0; i < columnOrder.length; i++) {
|
|
||||||
if (colData[i] === null) {
|
|
||||||
nullBitmap[Math.floor(i / 8)] |= (1 << (i % 8));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
buf.set(nullBitmap, offset);
|
|
||||||
offset += nullBitmapBytes;
|
|
||||||
|
|
||||||
// Column data
|
|
||||||
for (let i = 0; i < columnOrder.length; i++) {
|
|
||||||
const encoded = colData[i];
|
|
||||||
if (encoded === null) {
|
|
||||||
view.setUint8(offset, ColumnEncoding.NULL);
|
|
||||||
offset += 1;
|
|
||||||
} else {
|
|
||||||
const tag = getEncodingTag(columnTypes[columnOrder[i]] ?? 'string');
|
|
||||||
view.setUint8(offset, tag);
|
|
||||||
offset += 1;
|
|
||||||
buf.set(encoded, offset);
|
|
||||||
offset += encoded.byteLength;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return buf;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* 编码单个列的值。
|
|
||||||
*/
|
|
||||||
function encodeColumn(value: unknown, fieldType: string): Uint8Array | null {
|
|
||||||
if (value === null || value === undefined) return null;
|
|
||||||
|
|
||||||
switch (fieldType) {
|
|
||||||
case 'string': {
|
|
||||||
const str = String(value);
|
|
||||||
const encoder = new TextEncoder();
|
|
||||||
const bytes = encoder.encode(str);
|
|
||||||
const buf = new Uint8Array(2 + bytes.byteLength);
|
|
||||||
new DataView(buf.buffer).setUint16(0, bytes.byteLength, false);
|
|
||||||
buf.set(bytes, 2);
|
|
||||||
return buf;
|
|
||||||
}
|
|
||||||
case 'number': {
|
|
||||||
const buf = new ArrayBuffer(8);
|
|
||||||
new DataView(buf).setFloat64(0, Number(value), false);
|
|
||||||
return new Uint8Array(buf);
|
|
||||||
}
|
|
||||||
case 'boolean': {
|
|
||||||
return new Uint8Array([value ? 1 : 0]);
|
|
||||||
}
|
|
||||||
case 'date': {
|
|
||||||
const ts = value instanceof Date ? value.getTime() : new Date(String(value)).getTime();
|
|
||||||
const buf = new ArrayBuffer(8);
|
|
||||||
new DataView(buf).setFloat64(0, ts, false);
|
|
||||||
return new Uint8Array(buf);
|
|
||||||
}
|
|
||||||
case 'json': {
|
|
||||||
const str = JSON.stringify(value);
|
|
||||||
const encoder = new TextEncoder();
|
|
||||||
const bytes = encoder.encode(str);
|
|
||||||
const buf = new Uint8Array(2 + bytes.byteLength);
|
|
||||||
new DataView(buf.buffer).setUint16(0, bytes.byteLength, false);
|
|
||||||
buf.set(bytes, 2);
|
|
||||||
return buf;
|
|
||||||
}
|
|
||||||
default:
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// 解码
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
/**
|
|
||||||
* 从二进制字节数组解码行数据。
|
|
||||||
* @returns 行数据,如果格式错误返回 null
|
|
||||||
*/
|
|
||||||
export function decodeTuple(
|
|
||||||
bytes: Uint8Array,
|
|
||||||
columnOrder: string[],
|
|
||||||
columnTypes: Record<string, string>,
|
|
||||||
): Record<string, unknown> | null {
|
|
||||||
try {
|
|
||||||
const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
|
|
||||||
let offset = 0;
|
|
||||||
|
|
||||||
const nullBitmapBytes = Math.ceil(columnOrder.length / 8);
|
|
||||||
if (offset + nullBitmapBytes > bytes.byteLength) return null;
|
|
||||||
|
|
||||||
offset += nullBitmapBytes;
|
|
||||||
|
|
||||||
const row: Record<string, unknown> = {};
|
|
||||||
|
|
||||||
for (let i = 0; i < columnOrder.length; i++) {
|
|
||||||
if (offset >= bytes.byteLength) break;
|
|
||||||
|
|
||||||
const tag = view.getUint8(offset);
|
|
||||||
offset += 1;
|
|
||||||
|
|
||||||
if (tag === ColumnEncoding.NULL) {
|
|
||||||
row[columnOrder[i]] = null;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
const col = columnOrder[i];
|
|
||||||
const fType = columnTypes[col] ?? 'string';
|
|
||||||
|
|
||||||
const result = decodeColumnValue(bytes, offset, tag, fType);
|
|
||||||
if (result === null) return null;
|
|
||||||
row[col] = result.value;
|
|
||||||
offset = result.nextOffset;
|
|
||||||
}
|
|
||||||
|
|
||||||
return row;
|
|
||||||
} catch {
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function decodeColumnValue(
|
|
||||||
bytes: Uint8Array,
|
|
||||||
offset: number,
|
|
||||||
tag: number,
|
|
||||||
_fieldType: string,
|
|
||||||
): { value: unknown; nextOffset: number } | null {
|
|
||||||
const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
|
|
||||||
|
|
||||||
switch (tag) {
|
|
||||||
case ColumnEncoding.STRING:
|
|
||||||
case ColumnEncoding.JSON: {
|
|
||||||
if (offset + 2 > bytes.byteLength) return null;
|
|
||||||
const len = view.getUint16(offset, false);
|
|
||||||
offset += 2;
|
|
||||||
if (offset + len > bytes.byteLength) return null;
|
|
||||||
const decoder = new TextDecoder();
|
|
||||||
const str = decoder.decode(bytes.slice(offset, offset + len));
|
|
||||||
return {
|
|
||||||
value: tag === ColumnEncoding.JSON ? JSON.parse(str) : str,
|
|
||||||
nextOffset: offset + len,
|
|
||||||
};
|
|
||||||
}
|
|
||||||
case ColumnEncoding.NUMBER: {
|
|
||||||
if (offset + 8 > bytes.byteLength) return null;
|
|
||||||
const val = view.getFloat64(offset, false);
|
|
||||||
return { value: val, nextOffset: offset + 8 };
|
|
||||||
}
|
|
||||||
case ColumnEncoding.BOOLEAN: {
|
|
||||||
if (offset >= bytes.byteLength) return null;
|
|
||||||
return { value: view.getUint8(offset) !== 0, nextOffset: offset + 1 };
|
|
||||||
}
|
|
||||||
case ColumnEncoding.DATE: {
|
|
||||||
if (offset + 8 > bytes.byteLength) return null;
|
|
||||||
const ts = view.getFloat64(offset, false);
|
|
||||||
return { value: new Date(ts).toISOString(), nextOffset: offset + 8 };
|
|
||||||
}
|
|
||||||
default:
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// 辅助
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
function getEncodingTag(fieldType: string): ColumnEncoding {
|
|
||||||
switch (fieldType) {
|
|
||||||
case 'string': return ColumnEncoding.STRING;
|
|
||||||
case 'number': return ColumnEncoding.NUMBER;
|
|
||||||
case 'boolean': return ColumnEncoding.BOOLEAN;
|
|
||||||
case 'date': return ColumnEncoding.DATE;
|
|
||||||
case 'json': return ColumnEncoding.JSON;
|
|
||||||
default: return ColumnEncoding.NULL;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/** 获取列类型到编码标签的映射 */
|
|
||||||
export function getColumnEncodingMap(
|
|
||||||
columnOrder: string[],
|
|
||||||
columnTypes: Record<string, string>,
|
|
||||||
): Map<string, ColumnEncoding> {
|
|
||||||
const map = new Map<string, ColumnEncoding>();
|
|
||||||
for (const col of columnOrder) {
|
|
||||||
map.set(col, getEncodingTag(columnTypes[col] ?? 'string'));
|
|
||||||
}
|
|
||||||
return map;
|
|
||||||
}
|
|
||||||
@@ -23,6 +23,13 @@ export interface IStorageBackend {
|
|||||||
read(key: string): Promise<ArrayBuffer | null>;
|
read(key: string): Promise<ArrayBuffer | null>;
|
||||||
/** 写入数据块 */
|
/** 写入数据块 */
|
||||||
write(key: string, data: ArrayBuffer): Promise<void>;
|
write(key: string, data: ArrayBuffer): Promise<void>;
|
||||||
|
/**
|
||||||
|
* 追加写入(v0.4.5 WAL 分片用,可选):
|
||||||
|
* - OPFS 后端实现真追加(createWritable keepExistingData + seek,O(chunk))
|
||||||
|
* - 未实现的后端由调用方回退 read+write(EncryptedBackend 包装时整体重写保正确性)
|
||||||
|
* 语义:在 key 现有内容末尾追加 data;key 不存在时等同 write。
|
||||||
|
*/
|
||||||
|
append?(key: string, data: ArrayBuffer): Promise<void>;
|
||||||
/**
|
/**
|
||||||
* 批量原子写入(v0.4.2-fix):多个 key 在单个底层事务中提交,
|
* 批量原子写入(v0.4.2-fix):多个 key 在单个底层事务中提交,
|
||||||
* 中断时整体回滚,不留半写状态。WAL count 与记录同事务保证一致性。
|
* 中断时整体回滚,不留半写状态。WAL count 与记录同事务保证一致性。
|
||||||
|
|||||||
@@ -0,0 +1,233 @@
|
|||||||
|
/**
|
||||||
|
* AriaEngine Encrypted Backend — 全库透明 AES-256-GCM 加密后端
|
||||||
|
* @module engine/aria/store/encrypted_backend
|
||||||
|
*
|
||||||
|
* 装饰器模式包装底层 IStorageBackend:
|
||||||
|
* - 写入时加密(每个 value 独立随机 IV:格式 [12B IV][AES-GCM ciphertext])
|
||||||
|
* - 读取时解密(GCM 认证标签同时保证完整性)
|
||||||
|
* - WAL / SSTable / Schema / 元数据 全部密文存储(除密钥元数据本身)
|
||||||
|
*
|
||||||
|
* 密钥管理:
|
||||||
|
* - PBKDF2-SHA256(100000 迭代)从密码派生 AES-256-GCM 密钥
|
||||||
|
* - `__aria_keymeta` 明文保存 { salt, verifier }:
|
||||||
|
* - salt:PBKDF2 盐(重启后用同一密码重新派生密钥)
|
||||||
|
* - verifier:对固定明文加密的密文(打开时解密验证密码正确性)
|
||||||
|
* - 密码错误 → GCM 认证失败 → 抛 ARIA_DECRYPT_ERROR
|
||||||
|
*
|
||||||
|
* 依赖浏览器/Node 的 WebCrypto(jest.setup.js 已提供 polyfill)。
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { DatabaseError } from '../../../constants';
|
||||||
|
import type { IStorageBackend } from './backend';
|
||||||
|
import { CryptoManager } from '../crypto';
|
||||||
|
|
||||||
|
const IV_LENGTH = 12;
|
||||||
|
const KEYMETA_KEY = '__aria_keymeta';
|
||||||
|
/** verifier 固定明文(无数据泄露风险) */
|
||||||
|
const VERIFIER_PLAIN = 'metona-sqlark-encryption-verifier';
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Base64 工具(浏览器 btoa/atob 与 Node Buffer 双环境)
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
function bytesToBase64(bytes: Uint8Array): string {
|
||||||
|
if (typeof Buffer !== 'undefined') {
|
||||||
|
return Buffer.from(bytes).toString('base64');
|
||||||
|
}
|
||||||
|
let bin = '';
|
||||||
|
for (let i = 0; i < bytes.byteLength; i++) bin += String.fromCharCode(bytes[i]);
|
||||||
|
return btoa(bin);
|
||||||
|
}
|
||||||
|
|
||||||
|
function base64ToBytes(b64: string): Uint8Array {
|
||||||
|
if (typeof Buffer !== 'undefined') {
|
||||||
|
return new Uint8Array(Buffer.from(b64, 'base64'));
|
||||||
|
}
|
||||||
|
const bin = atob(b64);
|
||||||
|
const bytes = new Uint8Array(bin.length);
|
||||||
|
for (let i = 0; i < bin.length; i++) bytes[i] = bin.charCodeAt(i);
|
||||||
|
return bytes;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// EncryptedBackend
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
export class EncryptedBackend implements IStorageBackend {
|
||||||
|
private crypto: CryptoManager = new CryptoManager();
|
||||||
|
private opened = false;
|
||||||
|
|
||||||
|
constructor(
|
||||||
|
private inner: IStorageBackend,
|
||||||
|
private password: string,
|
||||||
|
) {
|
||||||
|
if (!password || password.length === 0) {
|
||||||
|
throw new DatabaseError('Encryption password must not be empty', 'ARIA_ENCRYPT_CONFIG_ERROR');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 底层后端(测试/调试用) */
|
||||||
|
getInner(): IStorageBackend {
|
||||||
|
return this.inner;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 加密是否已初始化(打开并验证/创建密钥后为 true) */
|
||||||
|
isCryptoReady(): boolean {
|
||||||
|
return this.crypto.enabled;
|
||||||
|
}
|
||||||
|
|
||||||
|
async open(name: string): Promise<void> {
|
||||||
|
if (!this.inner.isOpen()) {
|
||||||
|
await this.inner.open(name);
|
||||||
|
}
|
||||||
|
|
||||||
|
const raw = await this.inner.read(KEYMETA_KEY);
|
||||||
|
if (raw) {
|
||||||
|
// 已有密钥元数据:用持久化 salt 重新派生并验证密码
|
||||||
|
let meta: { salt?: string; verifier?: string };
|
||||||
|
try {
|
||||||
|
meta = JSON.parse(new TextDecoder().decode(raw)) as { salt?: string; verifier?: string };
|
||||||
|
} catch {
|
||||||
|
throw new DatabaseError('Corrupted encryption key metadata', 'ARIA_DECRYPT_ERROR');
|
||||||
|
}
|
||||||
|
if (!meta.salt || !meta.verifier) {
|
||||||
|
throw new DatabaseError('Corrupted encryption key metadata', 'ARIA_DECRYPT_ERROR');
|
||||||
|
}
|
||||||
|
const salt = base64ToBytes(meta.salt);
|
||||||
|
await this.crypto.init(this.password, salt);
|
||||||
|
const verifierBytes = base64ToBytes(meta.verifier);
|
||||||
|
if (verifierBytes.byteLength <= IV_LENGTH) {
|
||||||
|
throw new DatabaseError('Corrupted encryption key metadata', 'ARIA_DECRYPT_ERROR');
|
||||||
|
}
|
||||||
|
const iv = verifierBytes.subarray(0, IV_LENGTH);
|
||||||
|
const ciphertext = verifierBytes.subarray(IV_LENGTH);
|
||||||
|
try {
|
||||||
|
await this.crypto.decryptPage(iv, ciphertext);
|
||||||
|
} catch {
|
||||||
|
// GCM 认证失败:密码错误(或密钥元数据被篡改)
|
||||||
|
throw new DatabaseError(
|
||||||
|
'Decryption failed: wrong password or corrupted key metadata',
|
||||||
|
'ARIA_DECRYPT_ERROR',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// 无 keymeta:若库中已存在其他数据 → 明文旧库或 keymeta 丢失,拒绝以加密模式打开
|
||||||
|
const keys = await this.inner.listKeys();
|
||||||
|
if (keys.some((k) => k !== KEYMETA_KEY)) {
|
||||||
|
throw new DatabaseError(
|
||||||
|
'Cannot open with encryption: existing database has no key metadata ' +
|
||||||
|
'(database was created without encryption, or key metadata was lost)',
|
||||||
|
'ARIA_ENCRYPT_CONFIG_ERROR',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
// 新库:生成随机 salt + 派生密钥 + 写入 verifier
|
||||||
|
const salt = await this.crypto.init(this.password);
|
||||||
|
const enc = await this.crypto.encryptPage(new TextEncoder().encode(VERIFIER_PLAIN).buffer);
|
||||||
|
const verifier = new Uint8Array(IV_LENGTH + enc.data.byteLength);
|
||||||
|
verifier.set(enc.iv, 0);
|
||||||
|
verifier.set(new Uint8Array(enc.data), IV_LENGTH);
|
||||||
|
const keymeta = JSON.stringify({
|
||||||
|
salt: bytesToBase64(salt),
|
||||||
|
verifier: bytesToBase64(verifier),
|
||||||
|
});
|
||||||
|
await this.inner.write(KEYMETA_KEY, new TextEncoder().encode(keymeta).buffer);
|
||||||
|
}
|
||||||
|
|
||||||
|
this.opened = true;
|
||||||
|
}
|
||||||
|
|
||||||
|
async close(): Promise<void> {
|
||||||
|
await this.inner.close();
|
||||||
|
this.crypto.close();
|
||||||
|
this.opened = false;
|
||||||
|
}
|
||||||
|
|
||||||
|
isOpen(): boolean {
|
||||||
|
return this.opened;
|
||||||
|
}
|
||||||
|
|
||||||
|
async read(key: string): Promise<ArrayBuffer | null> {
|
||||||
|
this.ensureReady();
|
||||||
|
const raw = await this.inner.read(key);
|
||||||
|
if (raw === null) return null;
|
||||||
|
return this.decrypt(raw);
|
||||||
|
}
|
||||||
|
|
||||||
|
async write(key: string, data: ArrayBuffer): Promise<void> {
|
||||||
|
this.ensureReady();
|
||||||
|
await this.inner.write(key, await this.encrypt(data));
|
||||||
|
}
|
||||||
|
|
||||||
|
async writeMany(entries: Record<string, ArrayBuffer>): Promise<void> {
|
||||||
|
this.ensureReady();
|
||||||
|
const encrypted: Record<string, ArrayBuffer> = {};
|
||||||
|
for (const [key, data] of Object.entries(entries)) {
|
||||||
|
encrypted[key] = await this.encrypt(data);
|
||||||
|
}
|
||||||
|
await this.inner.writeMany(encrypted);
|
||||||
|
}
|
||||||
|
|
||||||
|
async delete(key: string): Promise<void> {
|
||||||
|
this.ensureReady();
|
||||||
|
await this.inner.delete(key);
|
||||||
|
}
|
||||||
|
|
||||||
|
async deleteMany(keys: string[]): Promise<void> {
|
||||||
|
this.ensureReady();
|
||||||
|
await this.inner.deleteMany(keys);
|
||||||
|
}
|
||||||
|
|
||||||
|
async listKeys(): Promise<string[]> {
|
||||||
|
this.ensureReady();
|
||||||
|
return this.inner.listKeys();
|
||||||
|
}
|
||||||
|
|
||||||
|
async exists(key: string): Promise<boolean> {
|
||||||
|
this.ensureReady();
|
||||||
|
return this.inner.exists(key);
|
||||||
|
}
|
||||||
|
|
||||||
|
async clear(): Promise<void> {
|
||||||
|
// 清空全部数据但保留密钥元数据 —— clearAll 语义:库本身保留,密码不失效
|
||||||
|
const keys = await this.inner.listKeys();
|
||||||
|
const toDelete = keys.filter((k) => k !== KEYMETA_KEY);
|
||||||
|
if (toDelete.length > 0) {
|
||||||
|
await this.inner.deleteMany(toDelete);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// -----------------------------------------------------------------------
|
||||||
|
// 内部
|
||||||
|
// -----------------------------------------------------------------------
|
||||||
|
|
||||||
|
private ensureReady(): void {
|
||||||
|
if (!this.opened) throw new DatabaseError('EncryptedBackend not opened', 'ARIA_DB_NOT_OPEN');
|
||||||
|
if (!this.crypto.enabled) throw new DatabaseError('EncryptedBackend key not initialized', 'ARIA_DECRYPT_ERROR');
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 加密单块数据:[IV(12)][ciphertext] */
|
||||||
|
private async encrypt(data: ArrayBuffer): Promise<ArrayBuffer> {
|
||||||
|
const enc = await this.crypto.encryptPage(data);
|
||||||
|
const out = new ArrayBuffer(IV_LENGTH + enc.data.byteLength);
|
||||||
|
const outBytes = new Uint8Array(out);
|
||||||
|
outBytes.set(enc.iv, 0);
|
||||||
|
outBytes.set(new Uint8Array(enc.data), IV_LENGTH);
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 解密单块数据(GCM 认证失败抛错) */
|
||||||
|
private async decrypt(data: ArrayBuffer): Promise<ArrayBuffer> {
|
||||||
|
const bytes = new Uint8Array(data);
|
||||||
|
if (bytes.byteLength <= IV_LENGTH) {
|
||||||
|
throw new DatabaseError('Corrupted encrypted data block (too short)', 'ARIA_DECRYPT_ERROR');
|
||||||
|
}
|
||||||
|
const iv = bytes.subarray(0, IV_LENGTH);
|
||||||
|
const ciphertext = bytes.subarray(IV_LENGTH);
|
||||||
|
try {
|
||||||
|
return await this.crypto.decryptPage(iv, ciphertext);
|
||||||
|
} catch (error) {
|
||||||
|
if (error instanceof DatabaseError) throw error;
|
||||||
|
throw new DatabaseError('Decryption failed (data corrupted or wrong key)', 'ARIA_DECRYPT_ERROR', error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -44,8 +44,9 @@ export class FileManager implements PageIO {
|
|||||||
const key = `pg_${pageId}`;
|
const key = `pg_${pageId}`;
|
||||||
const data = await this.backend.read(key);
|
const data = await this.backend.read(key);
|
||||||
if (!data) {
|
if (!data) {
|
||||||
// 第一次访问:创建新页面
|
// v0.4.5: 页面总是先分配(allocatePageId 持久化)后写入 —— 读取缺失页面视为损坏
|
||||||
return this.createEmptyPage(pageId, PageType.DATA);
|
// (此前返回空页面会静默掩盖页面丢失,页面化 SSTable 依赖 null 触发自愈清理)
|
||||||
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
// 确保大小正确
|
// 确保大小正确
|
||||||
@@ -69,25 +70,23 @@ export class FileManager implements PageIO {
|
|||||||
return id;
|
return id;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** v0.4.5: 批量分配页面 ID(一次 meta 持久化,避免页面化 SSTable 保存时逐页写 meta) */
|
||||||
|
async allocatePageIds(count: number): Promise<number[]> {
|
||||||
|
if (count <= 0) return [];
|
||||||
|
const ids: number[] = [];
|
||||||
|
const start = this.nextPageId;
|
||||||
|
this.nextPageId += count;
|
||||||
|
for (let i = 0; i < count; i++) ids.push(start + i);
|
||||||
|
await this.saveMeta();
|
||||||
|
return ids;
|
||||||
|
}
|
||||||
|
|
||||||
async freePageId(_pageId: number): Promise<void> {
|
async freePageId(_pageId: number): Promise<void> {
|
||||||
// 简化实现:不回收 pageId
|
// 简化实现:不回收 pageId
|
||||||
const key = `pg_${_pageId}`;
|
const key = `pg_${_pageId}`;
|
||||||
await this.backend.delete(key);
|
await this.backend.delete(key);
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- 表页面分配 ----
|
|
||||||
|
|
||||||
/**
|
|
||||||
* 分配一个新的表元数据页面。
|
|
||||||
*/
|
|
||||||
async allocateTableRootPage(): Promise<number> {
|
|
||||||
const pageId = await this.allocatePageId();
|
|
||||||
const data = new ArrayBuffer(PAGE_SIZE);
|
|
||||||
initPageHeader(data, pageId, PageType.META);
|
|
||||||
await this.writePage(pageId, data);
|
|
||||||
return pageId;
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---- 辅助 ----
|
// ---- 辅助 ----
|
||||||
|
|
||||||
private async saveMeta(): Promise<void> {
|
private async saveMeta(): Promise<void> {
|
||||||
@@ -96,12 +95,6 @@ export class FileManager implements PageIO {
|
|||||||
await this.backend.write('__aria_meta', buf);
|
await this.backend.write('__aria_meta', buf);
|
||||||
}
|
}
|
||||||
|
|
||||||
private createEmptyPage(pageId: number, type: PageType): ArrayBuffer {
|
|
||||||
const buf = new ArrayBuffer(PAGE_SIZE);
|
|
||||||
initPageHeader(buf, pageId, type);
|
|
||||||
return buf;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** 清空所有数据 */
|
/** 清空所有数据 */
|
||||||
async clearAll(): Promise<void> {
|
async clearAll(): Promise<void> {
|
||||||
await this.backend.clear();
|
await this.backend.clear();
|
||||||
|
|||||||
@@ -5,10 +5,23 @@
|
|||||||
* 零外部依赖,纯浏览器文件系统 API。
|
* 零外部依赖,纯浏览器文件系统 API。
|
||||||
* 每个 key 对应 OPFS 目录下的一个二进制文件。
|
* 每个 key 对应 OPFS 目录下的一个二进制文件。
|
||||||
*
|
*
|
||||||
* 浏览器要求:Chrome 102+ / Edge 102+
|
* 原子性与一致性保证(v0.4.5 固化):
|
||||||
|
* - 单文件 write/append:createWritable 为 copy-on-write —— close 前崩溃旧文件保持不变,
|
||||||
|
* close 后原子替换(单文件写入原子)
|
||||||
|
* - 多文件 writeMany/deleteMany:OPFS 无跨文件事务,串行逐个落盘;调用方(WAL 分片)
|
||||||
|
* 已改为单文件语义,多键操作仅用于一次性的 schema/meta 写入
|
||||||
|
* - 所有写操作串行队列化(同源同进程顺序一致);单次任务失败不中断队列链,
|
||||||
|
* 错误如实返回给该次调用的调用方
|
||||||
|
* - close() 等待写队列排空后再释放目录句柄(杜绝 close 后挂起写丢失/读旧数据)
|
||||||
|
* - open() 自动清理崩溃残留临时文件(Chromium createWritable 的 .crswap 等)
|
||||||
|
*
|
||||||
|
* 浏览器要求:Chrome 102+ / Edge 102+(Safari 15.2+ / Firefox 111+ 支持基础 OPFS)
|
||||||
*/
|
*/
|
||||||
import type { IStorageBackend } from './backend';
|
import type { IStorageBackend } from './backend';
|
||||||
|
|
||||||
|
/** 崩溃残留临时文件后缀(createWritable 底层实现可能留下) */
|
||||||
|
const STALE_SUFFIXES = ['.crswap', '.tmp'];
|
||||||
|
|
||||||
export class OPFSBackend implements IStorageBackend {
|
export class OPFSBackend implements IStorageBackend {
|
||||||
private root: FileSystemDirectoryHandle | null = null;
|
private root: FileSystemDirectoryHandle | null = null;
|
||||||
private dbDir: FileSystemDirectoryHandle | null = null;
|
private dbDir: FileSystemDirectoryHandle | null = null;
|
||||||
@@ -19,9 +32,15 @@ export class OPFSBackend implements IStorageBackend {
|
|||||||
this.dbName = name;
|
this.dbName = name;
|
||||||
this.root = await navigator.storage.getDirectory();
|
this.root = await navigator.storage.getDirectory();
|
||||||
this.dbDir = await this.root.getDirectoryHandle(name, { create: true });
|
this.dbDir = await this.root.getDirectoryHandle(name, { create: true });
|
||||||
|
// v0.4.5: 清理崩溃残留临时文件(不阻塞打开)
|
||||||
|
await this.cleanupStaleFiles();
|
||||||
}
|
}
|
||||||
|
|
||||||
async close(): Promise<void> {
|
async close(): Promise<void> {
|
||||||
|
// v0.4.5-fix: 等待所有挂起写完成(否则 close 后挂起写被静默丢弃/读旧数据)
|
||||||
|
try {
|
||||||
|
await this.writeQueue;
|
||||||
|
} catch { /* 写失败已返回给调用方 */ }
|
||||||
this.dbDir = null;
|
this.dbDir = null;
|
||||||
this.root = null;
|
this.root = null;
|
||||||
}
|
}
|
||||||
@@ -30,6 +49,23 @@ export class OPFSBackend implements IStorageBackend {
|
|||||||
return this.dbDir !== null;
|
return this.dbDir !== null;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** 清理崩溃残留的临时文件(open 时自动调用,repair 也可调用) */
|
||||||
|
async cleanupStaleFiles(): Promise<void> {
|
||||||
|
if (!this.dbDir) return;
|
||||||
|
try {
|
||||||
|
const dir = this.dbDir as any;
|
||||||
|
const stale: string[] = [];
|
||||||
|
for await (const [name] of dir.entries()) {
|
||||||
|
if (STALE_SUFFIXES.some((s) => name.endsWith(s))) {
|
||||||
|
stale.push(name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (const name of stale) {
|
||||||
|
try { await this.dbDir!.removeEntry(name); } catch { /* ignore */ }
|
||||||
|
}
|
||||||
|
} catch { /* 清理失败不阻塞 */ }
|
||||||
|
}
|
||||||
|
|
||||||
async read(key: string): Promise<ArrayBuffer | null> {
|
async read(key: string): Promise<ArrayBuffer | null> {
|
||||||
if (!this.dbDir) return null;
|
if (!this.dbDir) return null;
|
||||||
try {
|
try {
|
||||||
@@ -41,21 +77,45 @@ export class OPFSBackend implements IStorageBackend {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 单文件原子写:createWritable 为 copy-on-write,close 后原子替换;
|
||||||
|
* 写入期间崩溃 → 旧文件保持(原子性由浏览器 OPFS 实现保证)。
|
||||||
|
*/
|
||||||
async write(key: string, data: ArrayBuffer): Promise<void> {
|
async write(key: string, data: ArrayBuffer): Promise<void> {
|
||||||
if (!this.dbDir) return;
|
if (!this.dbDir) return;
|
||||||
this.writeQueue = this.writeQueue.then(async () => {
|
const run = this.writeQueue.then(async () => {
|
||||||
const fh = await this.dbDir!.getFileHandle(key, { create: true });
|
const fh = await this.dbDir!.getFileHandle(key, { create: true });
|
||||||
const writable = await fh.createWritable();
|
const writable = await fh.createWritable();
|
||||||
await writable.write(data);
|
await writable.write(data);
|
||||||
await writable.close();
|
await writable.close();
|
||||||
});
|
});
|
||||||
return this.writeQueue;
|
// v0.4.5-fix: 单次任务失败不中断队列链(错误仍返回给本次调用方)
|
||||||
|
this.writeQueue = run.then(() => undefined, () => undefined);
|
||||||
|
return run;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* v0.4.5: 真追加写 — createWritable(keepExistingData) + seek 到文件末尾。
|
||||||
|
* 单文件 COW 原子(close 前崩溃旧文件保持),无需读旧内容即实现 O(chunk) 追加
|
||||||
|
* (WAL 分片高频写入用)。
|
||||||
|
*/
|
||||||
|
async append(key: string, data: ArrayBuffer): Promise<void> {
|
||||||
|
if (!this.dbDir) return;
|
||||||
|
const run = this.writeQueue.then(async () => {
|
||||||
|
const fh = await this.dbDir!.getFileHandle(key, { create: true });
|
||||||
|
const existing = await fh.getFile();
|
||||||
|
const writable = await fh.createWritable({ keepExistingData: true });
|
||||||
|
await writable.write({ type: 'write', position: existing.size, data });
|
||||||
|
await writable.close();
|
||||||
|
});
|
||||||
|
this.writeQueue = run.then(() => undefined, () => undefined);
|
||||||
|
return run;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** v0.4.2-fix: 批量写入 — 串行队列内逐个落盘(OPFS 无跨文件事务,顺序保证一致) */
|
/** v0.4.2-fix: 批量写入 — 串行队列内逐个落盘(OPFS 无跨文件事务,顺序保证一致) */
|
||||||
async writeMany(entries: Record<string, ArrayBuffer>): Promise<void> {
|
async writeMany(entries: Record<string, ArrayBuffer>): Promise<void> {
|
||||||
if (!this.dbDir) return;
|
if (!this.dbDir) return;
|
||||||
this.writeQueue = this.writeQueue.then(async () => {
|
const run = this.writeQueue.then(async () => {
|
||||||
for (const [key, data] of Object.entries(entries)) {
|
for (const [key, data] of Object.entries(entries)) {
|
||||||
const fh = await this.dbDir!.getFileHandle(key, { create: true });
|
const fh = await this.dbDir!.getFileHandle(key, { create: true });
|
||||||
const writable = await fh.createWritable();
|
const writable = await fh.createWritable();
|
||||||
@@ -63,26 +123,29 @@ export class OPFSBackend implements IStorageBackend {
|
|||||||
await writable.close();
|
await writable.close();
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
return this.writeQueue;
|
this.writeQueue = run.then(() => undefined, () => undefined);
|
||||||
|
return run;
|
||||||
}
|
}
|
||||||
|
|
||||||
async delete(key: string): Promise<void> {
|
async delete(key: string): Promise<void> {
|
||||||
if (!this.dbDir) return;
|
if (!this.dbDir) return;
|
||||||
this.writeQueue = this.writeQueue.then(async () => {
|
const run = this.writeQueue.then(async () => {
|
||||||
try { await this.dbDir!.removeEntry(key); } catch { /* ignore */ }
|
try { await this.dbDir!.removeEntry(key); } catch { /* ignore */ }
|
||||||
});
|
});
|
||||||
return this.writeQueue;
|
this.writeQueue = run.then(() => undefined, () => undefined);
|
||||||
|
return run;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** v0.4.2-fix: 批量删除 — 串行队列内逐个删除 */
|
/** v0.4.2-fix: 批量删除 — 串行队列内逐个删除 */
|
||||||
async deleteMany(keys: string[]): Promise<void> {
|
async deleteMany(keys: string[]): Promise<void> {
|
||||||
if (!this.dbDir) return;
|
if (!this.dbDir) return;
|
||||||
this.writeQueue = this.writeQueue.then(async () => {
|
const run = this.writeQueue.then(async () => {
|
||||||
for (const key of keys) {
|
for (const key of keys) {
|
||||||
try { await this.dbDir!.removeEntry(key); } catch { /* ignore */ }
|
try { await this.dbDir!.removeEntry(key); } catch { /* ignore */ }
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
return this.writeQueue;
|
this.writeQueue = run.then(() => undefined, () => undefined);
|
||||||
|
return run;
|
||||||
}
|
}
|
||||||
|
|
||||||
async listKeys(): Promise<string[]> {
|
async listKeys(): Promise<string[]> {
|
||||||
|
|||||||
@@ -0,0 +1,95 @@
|
|||||||
|
/**
|
||||||
|
* AriaEngine Page SSTable Store — SSTable 页面化物理存储
|
||||||
|
* @module engine/aria/store/page_sstable_store
|
||||||
|
*
|
||||||
|
* v0.4.5: 让 BufferPool/FileManager 真正接入 LSM 读写路径。
|
||||||
|
* SSTable 不再整体存为一个 backend value,而是切分为 4KB 页面:
|
||||||
|
* - 页面由 FileManager 分配 pageId,经 BufferPool 缓存(LRU 驱逐,脏页写回)
|
||||||
|
* - save 语义 = 数据已落盘:页面写入后逐个 flushPage(await 底层写)才返回,
|
||||||
|
* 保证 WAL checkpoint(截断)前 SSTable 数据真实持久化
|
||||||
|
* - 页面 ID 列表经 SSTableMeta.pageIds 持久化;旧 meta(无 pageIds)走整 value 读取
|
||||||
|
*
|
||||||
|
* 与 LSM 的 sstableCache(整文件 LRU)双层缓存并存:
|
||||||
|
* - BufferPool 缓存物理页面(跨 SSTable 共享、受 bufferPoolPages 上限约束)
|
||||||
|
* - LSM 缓存解析后的整文件字节(查询热点复用)
|
||||||
|
*/
|
||||||
|
|
||||||
|
import type { FileManager } from './file_manager';
|
||||||
|
import type { BufferPool } from '../buffer/pool';
|
||||||
|
import type { PageHandle } from '../types';
|
||||||
|
import { PAGE_SIZE, PageType } from '../types';
|
||||||
|
|
||||||
|
export class PageSSTableStore {
|
||||||
|
/** SSTable id → 页面 ID 列表(save 时记录,saveMeta 时注入 meta) */
|
||||||
|
private pageIds = new Map<number, number[]>();
|
||||||
|
|
||||||
|
constructor(
|
||||||
|
private fileManager: FileManager,
|
||||||
|
private bufferPool: BufferPool,
|
||||||
|
) {}
|
||||||
|
|
||||||
|
/** 保存数据:切页 → 写入 BufferPool → 逐页落盘 → 记录 pageIds */
|
||||||
|
async save(id: number, data: Uint8Array): Promise<void> {
|
||||||
|
const pageCount = Math.max(1, Math.ceil(data.byteLength / PAGE_SIZE));
|
||||||
|
const handles: PageHandle[] = await this.bufferPool.newPages(pageCount, PageType.DATA);
|
||||||
|
|
||||||
|
const ids: number[] = [];
|
||||||
|
for (let i = 0; i < pageCount; i++) {
|
||||||
|
const page = handles[i];
|
||||||
|
ids.push(page.pageId);
|
||||||
|
const dest = new Uint8Array(page.data);
|
||||||
|
dest.fill(0); // 清空(最后一页可能不满)
|
||||||
|
const slice = data.subarray(i * PAGE_SIZE, Math.min((i + 1) * PAGE_SIZE, data.byteLength));
|
||||||
|
dest.set(slice, 0);
|
||||||
|
page.dirty = true;
|
||||||
|
// save 语义 = 已持久化:立即落盘(WAL checkpoint 截断依赖此保证)
|
||||||
|
await this.bufferPool.flushPage(page.pageId);
|
||||||
|
this.bufferPool.unpin(page);
|
||||||
|
}
|
||||||
|
this.pageIds.set(id, ids);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 获取指定 SSTable 的页面 ID 列表(saveMeta 注入用) */
|
||||||
|
getPageIds(id: number): number[] | undefined {
|
||||||
|
return this.pageIds.get(id);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 按页面 ID 列表读取并拼接为完整字节流。
|
||||||
|
* @param totalSize SSTable 真实大小(meta 持久化)——最后一页可能有 0 填充,按真实大小截断
|
||||||
|
* @returns 缺失页面/读取失败返回 null(调用方视为损坏并清理)
|
||||||
|
*/
|
||||||
|
async load(id: number, pageIds: number[], totalSize: number): Promise<Uint8Array | null> {
|
||||||
|
if (pageIds.length === 0) return null;
|
||||||
|
const chunks: Uint8Array[] = [];
|
||||||
|
for (const pageId of pageIds) {
|
||||||
|
const page = await this.bufferPool.getPage(pageId);
|
||||||
|
if (!page) return null;
|
||||||
|
// 立即复制(后续驱逐安全)
|
||||||
|
chunks.push(new Uint8Array(page.data));
|
||||||
|
this.bufferPool.unpin(page);
|
||||||
|
}
|
||||||
|
const total = chunks.reduce((s, c) => s + c.byteLength, 0);
|
||||||
|
const out = new Uint8Array(Math.min(total, totalSize));
|
||||||
|
let off = 0;
|
||||||
|
for (const c of chunks) {
|
||||||
|
const take = Math.min(c.byteLength, out.byteLength - off);
|
||||||
|
if (take <= 0) break;
|
||||||
|
out.set(c.subarray(0, take), off);
|
||||||
|
off += take;
|
||||||
|
}
|
||||||
|
this.pageIds.delete(id);
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 释放页面(删除物理页面文件 + 移出 BufferPool) */
|
||||||
|
async delete(id: number, pageIds: number[]): Promise<void> {
|
||||||
|
for (const pageId of pageIds) {
|
||||||
|
this.bufferPool.removePage(pageId);
|
||||||
|
try {
|
||||||
|
await this.fileManager.freePageId(pageId);
|
||||||
|
} catch { /* 清理失败不阻塞 */ }
|
||||||
|
}
|
||||||
|
this.pageIds.delete(id);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -102,12 +102,6 @@ export class MVCCManager {
|
|||||||
this.txnWriteKeys.delete(txnId);
|
this.txnWriteKeys.delete(txnId);
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 检查事务是否活跃 */
|
|
||||||
isActive(txnId: number): boolean {
|
|
||||||
const txn = this.activeTxns.get(txnId);
|
|
||||||
return txn !== undefined && txn.state === TransactionState.ACTIVE;
|
|
||||||
}
|
|
||||||
|
|
||||||
// =======================================================================
|
// =======================================================================
|
||||||
// 版本读写
|
// 版本读写
|
||||||
// =======================================================================
|
// =======================================================================
|
||||||
@@ -137,42 +131,6 @@ export class MVCCManager {
|
|||||||
this.txnWriteKeys.get(txnId)?.add(tableKey);
|
this.txnWriteKeys.get(txnId)?.add(tableKey);
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* 读取一行(对指定事务可见的最新版本)。
|
|
||||||
*/
|
|
||||||
readVersion(
|
|
||||||
tableName: string,
|
|
||||||
key: string,
|
|
||||||
txnId: number,
|
|
||||||
): Record<string, unknown> | null {
|
|
||||||
const txn = this.activeTxns.get(txnId);
|
|
||||||
if (!txn) return null;
|
|
||||||
|
|
||||||
const tableKey = `${tableName}.${key}`;
|
|
||||||
const versions = this.versionStore.get(tableKey);
|
|
||||||
if (!versions || versions.length === 0) return null;
|
|
||||||
|
|
||||||
// 从最新版本向前遍历
|
|
||||||
for (let i = versions.length - 1; i >= 0; i--) {
|
|
||||||
const version = versions[i];
|
|
||||||
|
|
||||||
// 1. 如果是当前事务写入的(未提交),可见
|
|
||||||
if (version.txnId === txnId) {
|
|
||||||
return version.data;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 2. 如果是已提交的版本,且在快照 LSN 之前提交,可见
|
|
||||||
if (version.committed) {
|
|
||||||
// 简化:所有已提交版本都可见
|
|
||||||
return version.data;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 3. 其他事务的未提交版本,不可见,继续找更早的版本
|
|
||||||
}
|
|
||||||
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* 删除一行(创建墓碑版本)。
|
* 删除一行(创建墓碑版本)。
|
||||||
*/
|
*/
|
||||||
@@ -180,30 +138,6 @@ export class MVCCManager {
|
|||||||
this.writeVersion(tableName, key, { __mvcc_tombstone: true } as unknown as Record<string, unknown>, txnId);
|
this.writeVersion(tableName, key, { __mvcc_tombstone: true } as unknown as Record<string, unknown>, txnId);
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* 获取所有行的最新已提交版本(用于非事务读取)。
|
|
||||||
*/
|
|
||||||
getLatestCommittedVersions(
|
|
||||||
tableName: string,
|
|
||||||
): Record<string, Record<string, unknown>> {
|
|
||||||
const result: Record<string, Record<string, unknown>> = {};
|
|
||||||
|
|
||||||
for (const [tableKey, versions] of this.versionStore) {
|
|
||||||
if (!tableKey.startsWith(`${tableName}.`)) continue;
|
|
||||||
const key = tableKey.slice(tableName.length + 1);
|
|
||||||
|
|
||||||
for (let i = versions.length - 1; i >= 0; i--) {
|
|
||||||
const version = versions[i];
|
|
||||||
if (version.committed && !(version.data as unknown as Record<string, unknown>).__mvcc_tombstone) {
|
|
||||||
result[key] = version.data;
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return result;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* v0.3.3: 丢弃指定事务的所有版本记录,但保留事务登记(Savepoint 回滚用)。
|
* v0.3.3: 丢弃指定事务的所有版本记录,但保留事务登记(Savepoint 回滚用)。
|
||||||
* 快照数据由调用方(引擎 txnSnapshot)负责恢复。
|
* 快照数据由调用方(引擎 txnSnapshot)负责恢复。
|
||||||
@@ -238,43 +172,6 @@ export class MVCCManager {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* 获取所有未提交事务中的 key 列表。
|
|
||||||
*/
|
|
||||||
getActiveWriteKeys(tableName: string, txnId: number): Set<string> {
|
|
||||||
const keys = new Set<string>();
|
|
||||||
const prefix = `${tableName}.`;
|
|
||||||
|
|
||||||
for (const [tableKey, versions] of this.versionStore) {
|
|
||||||
if (!tableKey.startsWith(prefix)) continue;
|
|
||||||
const latestVersion = versions[versions.length - 1];
|
|
||||||
if (latestVersion.txnId === txnId && !latestVersion.committed) {
|
|
||||||
keys.add(tableKey.slice(prefix.length));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return keys;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* 清理指定表的所有版本。
|
|
||||||
*/
|
|
||||||
clearTable(tableName: string): void {
|
|
||||||
const prefix = `${tableName}.`;
|
|
||||||
for (const [tableKey] of this.versionStore) {
|
|
||||||
if (tableKey.startsWith(prefix)) {
|
|
||||||
this.versionStore.delete(tableKey);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* 获取活跃事务数。
|
|
||||||
*/
|
|
||||||
getActiveTxnCount(): number {
|
|
||||||
return this.activeTxns.size;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* 获取全局 LSN。
|
* 获取全局 LSN。
|
||||||
*/
|
*/
|
||||||
|
|||||||
+28
-81
@@ -15,21 +15,12 @@ export const PAGE_SIZE = 4096;
|
|||||||
/** 页面头大小:16 字节 */
|
/** 页面头大小:16 字节 */
|
||||||
export const PAGE_HEADER_SIZE = 16;
|
export const PAGE_HEADER_SIZE = 16;
|
||||||
|
|
||||||
/** 每个 Slot 目录项大小:4 字节 (offset: u16 + len: u16) */
|
|
||||||
export const SLOT_ENTRY_SIZE = 4;
|
|
||||||
|
|
||||||
/** 页面数据区起始偏移(头部之后) */
|
|
||||||
export const PAGE_DATA_START = PAGE_HEADER_SIZE;
|
|
||||||
|
|
||||||
/** 无效页面 ID */
|
|
||||||
export const INVALID_PAGE_ID = 0xFFFFFFFF;
|
|
||||||
|
|
||||||
// =============================================================================
|
// =============================================================================
|
||||||
// 页面类型
|
// 页面类型
|
||||||
// =============================================================================
|
// =============================================================================
|
||||||
|
|
||||||
export enum PageType {
|
export enum PageType {
|
||||||
/** 数据页面:存储行数据 */
|
/** 数据页面:存储 SSTable 字节切片 */
|
||||||
DATA = 1,
|
DATA = 1,
|
||||||
/** 索引页面:存储索引节点 */
|
/** 索引页面:存储索引节点 */
|
||||||
INDEX = 2,
|
INDEX = 2,
|
||||||
@@ -39,36 +30,6 @@ export enum PageType {
|
|||||||
META = 4,
|
META = 4,
|
||||||
}
|
}
|
||||||
|
|
||||||
// =============================================================================
|
|
||||||
// 页面头部(16 字节)
|
|
||||||
// =============================================================================
|
|
||||||
|
|
||||||
export interface PageHeader {
|
|
||||||
/** 页面 ID(全局唯一) */
|
|
||||||
pageId: number;
|
|
||||||
/** 页面类型 */
|
|
||||||
type: PageType;
|
|
||||||
/** 空闲空间起始偏移(slot 区结束位置) */
|
|
||||||
freeStart: number;
|
|
||||||
/** 数据区结束偏移(从页面底部向上增长) */
|
|
||||||
freeEnd: number;
|
|
||||||
/** 当前 slot 数量 */
|
|
||||||
slotCount: number;
|
|
||||||
/** CRC32 校验和 */
|
|
||||||
checksum: number;
|
|
||||||
}
|
|
||||||
|
|
||||||
// =============================================================================
|
|
||||||
// Slot 目录项(4 字节)
|
|
||||||
// =============================================================================
|
|
||||||
|
|
||||||
export interface SlotEntry {
|
|
||||||
/** 行数据在页面内的偏移 */
|
|
||||||
offset: number;
|
|
||||||
/** 行数据长度(字节) */
|
|
||||||
length: number;
|
|
||||||
}
|
|
||||||
|
|
||||||
// =============================================================================
|
// =============================================================================
|
||||||
// 页面句柄(Buffer Pool 中的页面)
|
// 页面句柄(Buffer Pool 中的页面)
|
||||||
// =============================================================================
|
// =============================================================================
|
||||||
@@ -93,27 +54,9 @@ export interface PageHandle {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// =============================================================================
|
// =============================================================================
|
||||||
// 行编解码
|
// 列类型(内部二进制编码用)
|
||||||
// =============================================================================
|
// =============================================================================
|
||||||
|
|
||||||
/** 行/元组的二进制表示 */
|
|
||||||
export interface SerializedTuple {
|
|
||||||
/** 序列化后的字节数组 */
|
|
||||||
bytes: Uint8Array;
|
|
||||||
/** 该行中 null 列的位图 */
|
|
||||||
nullBitmap: Uint8Array;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** 列类型(内部二进制编码用) */
|
|
||||||
export enum ColumnEncoding {
|
|
||||||
STRING = 1,
|
|
||||||
NUMBER = 2,
|
|
||||||
BOOLEAN = 3,
|
|
||||||
DATE = 4,
|
|
||||||
JSON = 5,
|
|
||||||
NULL = 6,
|
|
||||||
}
|
|
||||||
|
|
||||||
// =============================================================================
|
// =============================================================================
|
||||||
// LSM-Tree
|
// LSM-Tree
|
||||||
// =============================================================================
|
// =============================================================================
|
||||||
@@ -121,9 +64,6 @@ export enum ColumnEncoding {
|
|||||||
/** MemTable 最大大小(默认 4MB) */
|
/** MemTable 最大大小(默认 4MB) */
|
||||||
export const DEFAULT_MEMTABLE_SIZE = 4 * 1024 * 1024;
|
export const DEFAULT_MEMTABLE_SIZE = 4 * 1024 * 1024;
|
||||||
|
|
||||||
/** SSTable 中每个 Data Block 的默认大小 */
|
|
||||||
export const DEFAULT_BLOCK_SIZE = 4096;
|
|
||||||
|
|
||||||
/** Bloom Filter 每 key 的默认位数 */
|
/** Bloom Filter 每 key 的默认位数 */
|
||||||
export const DEFAULT_BLOOM_BITS_PER_KEY = 10;
|
export const DEFAULT_BLOOM_BITS_PER_KEY = 10;
|
||||||
|
|
||||||
@@ -149,18 +89,12 @@ export interface SSTableMeta {
|
|||||||
totalSize: number;
|
totalSize: number;
|
||||||
/** Bloom Filter 序列化数据 */
|
/** Bloom Filter 序列化数据 */
|
||||||
bloomData: Uint8Array | null;
|
bloomData: Uint8Array | null;
|
||||||
}
|
/**
|
||||||
|
* v0.4.5: 页面化存储(OPFS 后端)— SSTable 的物理页面 ID 列表。
|
||||||
/** SSTable 内部的 Data Block */
|
* 存在 → 数据以 4KB 页面存储(BufferPool/FileManager 管理);
|
||||||
export interface DataBlock {
|
* 缺失 → 整 value 存储(旧格式,兼容读取)。
|
||||||
/** 该块内的 key-value 条目数 */
|
*/
|
||||||
entryCount: number;
|
pageIds?: number[];
|
||||||
/** 该块数据区 */
|
|
||||||
data: Uint8Array;
|
|
||||||
/** 该块起始 key */
|
|
||||||
startKey: string;
|
|
||||||
/** 该块结束 key */
|
|
||||||
endKey: string;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 索引块条目:key → data block offset */
|
/** 索引块条目:key → data block offset */
|
||||||
@@ -211,12 +145,6 @@ export interface WALRecord {
|
|||||||
// MVCC
|
// MVCC
|
||||||
// =============================================================================
|
// =============================================================================
|
||||||
|
|
||||||
/** 事务隔离级别 */
|
|
||||||
export enum IsolationLevel {
|
|
||||||
READ_COMMITTED = 1,
|
|
||||||
SNAPSHOT = 2,
|
|
||||||
}
|
|
||||||
|
|
||||||
/** 事务状态 */
|
/** 事务状态 */
|
||||||
export enum TransactionState {
|
export enum TransactionState {
|
||||||
ACTIVE = 1,
|
ACTIVE = 1,
|
||||||
@@ -284,9 +212,26 @@ export interface AriaEngineConfig {
|
|||||||
walSizeThreshold?: number;
|
walSizeThreshold?: number;
|
||||||
/** 最大内存预算(MB,默认 64) */
|
/** 最大内存预算(MB,默认 64) */
|
||||||
maxMemoryMB?: number;
|
maxMemoryMB?: number;
|
||||||
|
/**
|
||||||
|
* v0.4.5: 全库 AES-256-GCM 加密(backend 层透明加解密,WAL/SSTable/Schema/元数据全覆盖)。
|
||||||
|
* 密钥由 PBKDF2(salt 持久化于库内 __aria_keymeta)派生,重启用同一密码即可解密。
|
||||||
|
*/
|
||||||
|
encryption?: {
|
||||||
|
/** 加密密码 */
|
||||||
|
password: string;
|
||||||
|
};
|
||||||
|
/**
|
||||||
|
* v0.4.5: SSTable 页面化存储(4KB 页面 + BufferPool/FileManager 管理,LRU 缓存)。
|
||||||
|
* 默认:storageBackend === 'opfs' 时自动启用(大文件随机读/写放大优化);
|
||||||
|
* 显式 false 强制关闭(整 value 存储,兼容旧行为)。
|
||||||
|
*/
|
||||||
|
pageStorage?: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
export const DEFAULT_ARIA_CONFIG: Required<AriaEngineConfig> = {
|
export const DEFAULT_ARIA_CONFIG: Required<Omit<AriaEngineConfig, 'encryption' | 'pageStorage'>> & {
|
||||||
|
encryption: undefined;
|
||||||
|
pageStorage: undefined;
|
||||||
|
} = {
|
||||||
pageSize: PAGE_SIZE,
|
pageSize: PAGE_SIZE,
|
||||||
bufferPoolPages: DEFAULT_BUFFER_POOL_PAGES,
|
bufferPoolPages: DEFAULT_BUFFER_POOL_PAGES,
|
||||||
memtableSizeThreshold: DEFAULT_MEMTABLE_SIZE,
|
memtableSizeThreshold: DEFAULT_MEMTABLE_SIZE,
|
||||||
@@ -299,4 +244,6 @@ export const DEFAULT_ARIA_CONFIG: Required<AriaEngineConfig> = {
|
|||||||
storageBackend: 'indexeddb',
|
storageBackend: 'indexeddb',
|
||||||
walSizeThreshold: 16 * 1024 * 1024, // 16MB
|
walSizeThreshold: 16 * 1024 * 1024, // 16MB
|
||||||
maxMemoryMB: 64,
|
maxMemoryMB: 64,
|
||||||
|
encryption: undefined,
|
||||||
|
pageStorage: undefined,
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -68,15 +68,4 @@ export class CheckpointManager {
|
|||||||
this.opCount = 0;
|
this.opCount = 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
async forceCheckpoint(): Promise<void> {
|
|
||||||
await this.checkpoint();
|
|
||||||
}
|
|
||||||
|
|
||||||
setInterval(ops: number): void {
|
|
||||||
this.interval = ops;
|
|
||||||
}
|
|
||||||
|
|
||||||
getOpCount(): number {
|
|
||||||
return this.opCount;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
+17
-20
@@ -21,6 +21,7 @@
|
|||||||
*/
|
*/
|
||||||
|
|
||||||
import { WALRecordType, type WALRecord } from '../types';
|
import { WALRecordType, type WALRecord } from '../types';
|
||||||
|
import { crc32 } from '../crc32';
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// WAL 存储接口
|
// WAL 存储接口
|
||||||
@@ -178,14 +179,6 @@ export class WAL {
|
|||||||
// 统计
|
// 统计
|
||||||
// =======================================================================
|
// =======================================================================
|
||||||
|
|
||||||
isEnabled(): boolean {
|
|
||||||
return this.enabled;
|
|
||||||
}
|
|
||||||
|
|
||||||
getLSN(): number {
|
|
||||||
return this.lsn;
|
|
||||||
}
|
|
||||||
|
|
||||||
getBufferedCount(): number {
|
getBufferedCount(): number {
|
||||||
return this.buffer.length;
|
return this.buffer.length;
|
||||||
}
|
}
|
||||||
@@ -199,6 +192,15 @@ export class WAL {
|
|||||||
// 编解码
|
// 编解码
|
||||||
// -----------------------------------------------------------------------
|
// -----------------------------------------------------------------------
|
||||||
|
|
||||||
|
/** 旧版弱滚动校验(v0.4.4 及更早写入的 WAL 记录使用,双算法探测兼容) */
|
||||||
|
private legacyChecksum(data: Uint8Array): number {
|
||||||
|
let crc = 0;
|
||||||
|
for (let i = 0; i < data.length; i++) {
|
||||||
|
crc = ((crc << 5) - crc + data[i]) | 0;
|
||||||
|
}
|
||||||
|
return crc >>> 0;
|
||||||
|
}
|
||||||
|
|
||||||
private encodeRecord(record: WALRecord): Uint8Array {
|
private encodeRecord(record: WALRecord): Uint8Array {
|
||||||
const encoder = new TextEncoder();
|
const encoder = new TextEncoder();
|
||||||
const tableBytes = encoder.encode(record.tableName);
|
const tableBytes = encoder.encode(record.tableName);
|
||||||
@@ -241,13 +243,10 @@ export class WAL {
|
|||||||
new Uint8Array(buf).set(jsonBytes, offset);
|
new Uint8Array(buf).set(jsonBytes, offset);
|
||||||
offset += jsonBytes.length;
|
offset += jsonBytes.length;
|
||||||
|
|
||||||
// 简单 CRC
|
// v0.4.5: 标准 CRC-32 校验(此前为弱滚动校验,误检率更高)
|
||||||
let crc = 0;
|
|
||||||
const u8 = new Uint8Array(buf, 0, offset);
|
const u8 = new Uint8Array(buf, 0, offset);
|
||||||
for (let i = 0; i < u8.length; i++) {
|
const crc = crc32(u8);
|
||||||
crc = ((crc << 5) - crc + u8[i]) | 0;
|
view.setUint32(offset, crc, false);
|
||||||
}
|
|
||||||
view.setUint32(offset, crc >>> 0, false);
|
|
||||||
|
|
||||||
return new Uint8Array(buf);
|
return new Uint8Array(buf);
|
||||||
}
|
}
|
||||||
@@ -291,15 +290,13 @@ export class WAL {
|
|||||||
}
|
}
|
||||||
offset += jsonLen;
|
offset += jsonLen;
|
||||||
|
|
||||||
// 验证 CRC(跨记录数据计算,不含 CRC 自身)
|
// 验证 CRC:先标准 CRC-32,失败再尝试旧版弱滚动校验(兼容旧库 WAL 记录)
|
||||||
const storedCrc = view.getUint32(offset, false);
|
const storedCrc = view.getUint32(offset, false);
|
||||||
offset += 4;
|
offset += 4;
|
||||||
const recordBytes = data.slice(recordStart, offset - 4);
|
const recordBytes = data.slice(recordStart, offset - 4);
|
||||||
let computedCrc = 0;
|
const computedNew = crc32(recordBytes);
|
||||||
for (let i = 0; i < recordBytes.length; i++) {
|
const computedLegacy = this.legacyChecksum(recordBytes);
|
||||||
computedCrc = ((computedCrc << 5) - computedCrc + recordBytes[i]) | 0;
|
if ((computedNew >>> 0) !== storedCrc && (computedLegacy >>> 0) !== storedCrc) {
|
||||||
}
|
|
||||||
if ((computedCrc >>> 0) !== storedCrc) {
|
|
||||||
// CRC 不匹配,跳过此损坏记录
|
// CRC 不匹配,跳过此损坏记录
|
||||||
// eslint-disable-next-line no-console
|
// eslint-disable-next-line no-console
|
||||||
console.warn(`[AriaEngine WAL] CRC mismatch at record LSN=${lsn}, skipping`);
|
console.warn(`[AriaEngine WAL] CRC mismatch at record LSN=${lsn}, skipping`);
|
||||||
|
|||||||
@@ -0,0 +1,161 @@
|
|||||||
|
/**
|
||||||
|
* AriaEngine Segmented WAL Store — 分片式 WAL 持久化存储
|
||||||
|
* @module engine/aria/wal/segmented_store
|
||||||
|
*
|
||||||
|
* v0.4.5: 取代"每条记录一个 key"的旧结构:
|
||||||
|
* - 分片文件 `__wal_%06d.bin`,达到阈值(默认 4MB)切新分片 → 文件数量可控
|
||||||
|
* - append 追加写当前分片(后端支持真追加则 O(chunk),否则回退 read+write)
|
||||||
|
* - WAL 序号(LSN)内嵌于记录字节流,无需独立 count 键 → append 单文件原子写
|
||||||
|
* - readAll 检测分片序号空洞:序号不连续 → 丢弃空洞之后的分片(保守截断,
|
||||||
|
* 空洞仅可能来自 truncate 部分完成——此时数据已落盘,丢弃无害)
|
||||||
|
* - 兼容旧格式 `__wal_N`(每条记录一个键)+ `__wal_count`:读取时迁移重放,
|
||||||
|
* checkpoint 时一并清空
|
||||||
|
*/
|
||||||
|
|
||||||
|
import type { IStorageBackend } from '../store/backend';
|
||||||
|
|
||||||
|
/** 分片文件名:__wal_%06d.bin */
|
||||||
|
export const WAL_SEGMENT_PREFIX = '__wal_';
|
||||||
|
const SEGMENT_REGEX = /^__wal_(\d{6})\.bin$/;
|
||||||
|
const LEGACY_RECORD_REGEX = /^__wal_(\d+)$/;
|
||||||
|
const LEGACY_COUNT_KEY = '__wal_count';
|
||||||
|
|
||||||
|
/** 默认分片阈值 */
|
||||||
|
export const DEFAULT_WAL_SEGMENT_SIZE = 4 * 1024 * 1024;
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// WAL 存储接口(与 WAL 类解耦)
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
export interface WALStore {
|
||||||
|
append(data: Uint8Array): Promise<void>;
|
||||||
|
readAll(): Promise<Uint8Array>;
|
||||||
|
truncate(): Promise<void>;
|
||||||
|
exists(): Promise<boolean>;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// SegmentedWALStore
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
export class SegmentedWALStore implements WALStore {
|
||||||
|
private segmentSize: number;
|
||||||
|
/** 当前分片序号(append 定位) */
|
||||||
|
private currentSegment = 0;
|
||||||
|
/** 当前分片字节数(内存跟踪,append 切分片判断) */
|
||||||
|
private currentSize = 0;
|
||||||
|
|
||||||
|
constructor(
|
||||||
|
private backend: IStorageBackend,
|
||||||
|
segmentSize: number = DEFAULT_WAL_SEGMENT_SIZE,
|
||||||
|
) {
|
||||||
|
this.segmentSize = segmentSize;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 分片 key 生成 */
|
||||||
|
private segmentKey(seq: number): string {
|
||||||
|
return `${WAL_SEGMENT_PREFIX}${String(seq).padStart(6, '0')}.bin`;
|
||||||
|
}
|
||||||
|
|
||||||
|
async append(data: Uint8Array): Promise<void> {
|
||||||
|
if (data.byteLength === 0) return;
|
||||||
|
if (this.currentSize + data.byteLength > this.segmentSize) {
|
||||||
|
// 当前分片放不下 → 切新分片
|
||||||
|
this.currentSegment++;
|
||||||
|
this.currentSize = 0;
|
||||||
|
}
|
||||||
|
const key = this.segmentKey(this.currentSegment);
|
||||||
|
const copy = data.buffer.slice(data.byteOffset, data.byteOffset + data.byteLength) as ArrayBuffer;
|
||||||
|
|
||||||
|
if (typeof this.backend.append === 'function') {
|
||||||
|
await this.backend.append!(key, copy);
|
||||||
|
} else {
|
||||||
|
// 回退:读旧 + 拼接 + 写(单文件原子写)
|
||||||
|
const existing = await this.backend.read(key);
|
||||||
|
if (existing) {
|
||||||
|
const combined = new ArrayBuffer(existing.byteLength + copy.byteLength);
|
||||||
|
new Uint8Array(combined).set(new Uint8Array(existing), 0);
|
||||||
|
new Uint8Array(combined).set(new Uint8Array(copy), existing.byteLength);
|
||||||
|
await this.backend.write(key, combined);
|
||||||
|
} else {
|
||||||
|
await this.backend.write(key, copy);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
this.currentSize += data.byteLength;
|
||||||
|
}
|
||||||
|
|
||||||
|
async readAll(): Promise<Uint8Array> {
|
||||||
|
const keys = await this.backend.listKeys();
|
||||||
|
|
||||||
|
// ---- 新格式分片 ----
|
||||||
|
const segments = keys
|
||||||
|
.filter((k) => SEGMENT_REGEX.test(k))
|
||||||
|
.map((k) => ({ seq: Number(k.match(SEGMENT_REGEX)![1]), key: k }))
|
||||||
|
.sort((a, b) => a.seq - b.seq);
|
||||||
|
|
||||||
|
// 空洞检测:分片序号必须从 0 严格连续,空洞后的分片整体丢弃
|
||||||
|
let keepCount = 0;
|
||||||
|
for (let i = 0; i < segments.length; i++) {
|
||||||
|
if (segments[i].seq !== i) break;
|
||||||
|
keepCount = i + 1;
|
||||||
|
}
|
||||||
|
const validSegments = segments.slice(0, keepCount);
|
||||||
|
|
||||||
|
// ---- 旧格式兼容:__wal_N 单记录键(迁移前数据) ----
|
||||||
|
const legacyKeys = keys
|
||||||
|
.filter((k) => LEGACY_RECORD_REGEX.test(k))
|
||||||
|
.map((k) => ({ seq: Number(k.match(LEGACY_RECORD_REGEX)![1]), key: k }))
|
||||||
|
.sort((a, b) => a.seq - b.seq);
|
||||||
|
|
||||||
|
const parts: Uint8Array[] = [];
|
||||||
|
// 旧格式在前(它们是最早的记录)
|
||||||
|
for (const { key } of legacyKeys) {
|
||||||
|
const raw = await this.backend.read(key);
|
||||||
|
if (raw) parts.push(new Uint8Array(raw));
|
||||||
|
}
|
||||||
|
// 新格式分片在后
|
||||||
|
for (const { key } of validSegments) {
|
||||||
|
const raw = await this.backend.read(key);
|
||||||
|
if (raw) parts.push(new Uint8Array(raw));
|
||||||
|
}
|
||||||
|
|
||||||
|
// 同步当前分片状态(追加定位)
|
||||||
|
if (validSegments.length > 0) {
|
||||||
|
this.currentSegment = validSegments[validSegments.length - 1].seq;
|
||||||
|
const lastRaw = await this.backend.read(validSegments[validSegments.length - 1].key);
|
||||||
|
this.currentSize = lastRaw ? lastRaw.byteLength : 0;
|
||||||
|
// 旧格式键存在时(迁移中),下一条记录另起分片,避免与旧键序号冲突
|
||||||
|
if (legacyKeys.length > 0) {
|
||||||
|
this.currentSegment++;
|
||||||
|
this.currentSize = 0;
|
||||||
|
}
|
||||||
|
} else if (legacyKeys.length > 0) {
|
||||||
|
// 仅有旧格式:迁移中,新写入从分片 0 开始(checkpoint 会清空旧键)
|
||||||
|
this.currentSegment = 0;
|
||||||
|
this.currentSize = 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const total = parts.reduce((s, c) => s + c.byteLength, 0);
|
||||||
|
const combined = new Uint8Array(total);
|
||||||
|
let off = 0;
|
||||||
|
for (const c of parts) { combined.set(c, off); off += c.byteLength; }
|
||||||
|
return combined;
|
||||||
|
}
|
||||||
|
|
||||||
|
async truncate(): Promise<void> {
|
||||||
|
const keys = await this.backend.listKeys();
|
||||||
|
const walKeys = keys.filter((k) =>
|
||||||
|
k.startsWith(WAL_SEGMENT_PREFIX) || k === LEGACY_COUNT_KEY);
|
||||||
|
if (walKeys.length > 0) {
|
||||||
|
await this.backend.deleteMany(walKeys);
|
||||||
|
}
|
||||||
|
this.currentSegment = 0;
|
||||||
|
this.currentSize = 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
async exists(): Promise<boolean> {
|
||||||
|
const keys = await this.backend.listKeys();
|
||||||
|
return keys.some((k) =>
|
||||||
|
k.startsWith(WAL_SEGMENT_PREFIX) || k === LEGACY_COUNT_KEY);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -360,16 +360,4 @@ export class OPFSEngine implements IStorageEngine {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 从 OPFS 加载表数据到内存缓存 */
|
|
||||||
async loadTableIntoMemory(tableName: string, schema: TableSchema): Promise<void> {
|
|
||||||
await this.memoryCache.createTable(schema);
|
|
||||||
const rows = await this.readTableData(tableName);
|
|
||||||
if (rows.length > 0) {
|
|
||||||
// 直接用 Map 设置绕过 insert 校验
|
|
||||||
for (const row of rows) {
|
|
||||||
await this.memoryCache.insert(tableName, [row]);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
+34
-39
@@ -12,23 +12,6 @@ import type { WhereCondition, OrderBy } from '../constants';
|
|||||||
// AST 语句类型枚举
|
// AST 语句类型枚举
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
export type StatementType =
|
|
||||||
| 'SELECT'
|
|
||||||
| 'SELECT_UNION'
|
|
||||||
| 'EXPLAIN'
|
|
||||||
| 'INSERT'
|
|
||||||
| 'UPDATE'
|
|
||||||
| 'DELETE'
|
|
||||||
| 'CREATE_TABLE'
|
|
||||||
| 'DROP_TABLE'
|
|
||||||
| 'ALTER_TABLE'
|
|
||||||
| 'TRUNCATE_TABLE'
|
|
||||||
| 'CREATE_INDEX'
|
|
||||||
| 'DROP_INDEX'
|
|
||||||
| 'BEGIN'
|
|
||||||
| 'COMMIT'
|
|
||||||
| 'ROLLBACK';
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// 列引用
|
// 列引用
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -51,31 +34,10 @@ export interface JoinClause {
|
|||||||
on: WhereCondition;
|
on: WhereCondition;
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// 聚合函数
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
|
|
||||||
/** 聚合函数类型 */
|
|
||||||
export type AggregateFunc = 'COUNT' | 'SUM' | 'AVG' | 'MIN' | 'MAX';
|
|
||||||
|
|
||||||
/** 聚合表达式 */
|
|
||||||
export interface AggregateExpression {
|
|
||||||
type: 'AGGREGATE';
|
|
||||||
func: AggregateFunc;
|
|
||||||
column: string; // '*' for COUNT(*)
|
|
||||||
alias?: string;
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// 子查询
|
// 子查询
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
/** 子查询表达式 */
|
|
||||||
export interface SubqueryExpression {
|
|
||||||
type: 'SUBQUERY';
|
|
||||||
statement: SelectStatement | SelectUnionStatement;
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// DDL: CREATE TABLE
|
// DDL: CREATE TABLE
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -255,6 +217,35 @@ export interface RollbackTransactionStatement {
|
|||||||
type: 'ROLLBACK';
|
type: 'ROLLBACK';
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// TCL: 维护语句(v0.5.1)
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/** SAVEPOINT name / ROLLBACK TO SAVEPOINT name / RELEASE SAVEPOINT name */
|
||||||
|
export interface SavepointStatement {
|
||||||
|
type: 'SAVEPOINT';
|
||||||
|
name: string;
|
||||||
|
/** SAVE = 创建;ROLLBACK = 回滚到;RELEASE = 释放 */
|
||||||
|
action: 'SAVE' | 'ROLLBACK' | 'RELEASE';
|
||||||
|
}
|
||||||
|
|
||||||
|
/** ANALYZE TABLE name — 收集表统计信息 */
|
||||||
|
export interface AnalyzeStatement {
|
||||||
|
type: 'ANALYZE';
|
||||||
|
table: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** REINDEX TABLE name — 重建表二级索引 */
|
||||||
|
export interface ReindexStatement {
|
||||||
|
type: 'REINDEX';
|
||||||
|
table: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** VACUUM — 压缩 LSM + 清理碎片 */
|
||||||
|
export interface VacuumStatement {
|
||||||
|
type: 'VACUUM';
|
||||||
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// AST 联合类型
|
// AST 联合类型
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -274,4 +265,8 @@ export type Statement =
|
|||||||
| DropIndexStatement
|
| DropIndexStatement
|
||||||
| BeginTransactionStatement
|
| BeginTransactionStatement
|
||||||
| CommitTransactionStatement
|
| CommitTransactionStatement
|
||||||
| RollbackTransactionStatement;
|
| RollbackTransactionStatement
|
||||||
|
| SavepointStatement
|
||||||
|
| AnalyzeStatement
|
||||||
|
| ReindexStatement
|
||||||
|
| VacuumStatement;
|
||||||
|
|||||||
+16
-2
@@ -135,14 +135,18 @@ export class SelectQueryBuilder {
|
|||||||
export class UpdateQueryBuilder {
|
export class UpdateQueryBuilder {
|
||||||
private _where: WhereCondition = {};
|
private _where: WhereCondition = {};
|
||||||
private onWrite?: (table: string) => void;
|
private onWrite?: (table: string) => void;
|
||||||
|
/** v0.5.1: CRUD hooks 触发回调 */
|
||||||
|
private onHooks?: (hook: import('../constants').HookName, args: unknown[]) => Promise<void>;
|
||||||
|
|
||||||
constructor(
|
constructor(
|
||||||
private engine: IStorageEngine,
|
private engine: IStorageEngine,
|
||||||
private tableName: string,
|
private tableName: string,
|
||||||
private _updates: Record<string, unknown>,
|
private _updates: Record<string, unknown>,
|
||||||
onWrite?: (table: string) => void,
|
onWrite?: (table: string) => void,
|
||||||
|
onHooks?: (hook: import('../constants').HookName, args: unknown[]) => Promise<void>,
|
||||||
) {
|
) {
|
||||||
this.onWrite = onWrite;
|
this.onWrite = onWrite;
|
||||||
|
this.onHooks = onHooks;
|
||||||
}
|
}
|
||||||
|
|
||||||
where(condition: WhereCondition): this {
|
where(condition: WhereCondition): this {
|
||||||
@@ -151,8 +155,11 @@ export class UpdateQueryBuilder {
|
|||||||
}
|
}
|
||||||
|
|
||||||
async execute(): Promise<number> {
|
async execute(): Promise<number> {
|
||||||
const count = await this.engine.update(this.tableName, { table: this.tableName, where: this._where }, this._updates);
|
const query = { table: this.tableName, where: this._where };
|
||||||
|
await this.onHooks?.('beforeUpdate', [query, this._updates]);
|
||||||
|
const count = await this.engine.update(this.tableName, query, this._updates);
|
||||||
this.onWrite?.(this.tableName);
|
this.onWrite?.(this.tableName);
|
||||||
|
await this.onHooks?.('afterUpdate', [query, this._updates, count]);
|
||||||
return count;
|
return count;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -168,13 +175,17 @@ export class UpdateQueryBuilder {
|
|||||||
export class DeleteQueryBuilder {
|
export class DeleteQueryBuilder {
|
||||||
private _where: WhereCondition = {};
|
private _where: WhereCondition = {};
|
||||||
private onWrite?: (table: string) => void;
|
private onWrite?: (table: string) => void;
|
||||||
|
/** v0.5.1: CRUD hooks 触发回调 */
|
||||||
|
private onHooks?: (hook: import('../constants').HookName, args: unknown[]) => Promise<void>;
|
||||||
|
|
||||||
constructor(
|
constructor(
|
||||||
private engine: IStorageEngine,
|
private engine: IStorageEngine,
|
||||||
private tableName: string,
|
private tableName: string,
|
||||||
onWrite?: (table: string) => void,
|
onWrite?: (table: string) => void,
|
||||||
|
onHooks?: (hook: import('../constants').HookName, args: unknown[]) => Promise<void>,
|
||||||
) {
|
) {
|
||||||
this.onWrite = onWrite;
|
this.onWrite = onWrite;
|
||||||
|
this.onHooks = onHooks;
|
||||||
}
|
}
|
||||||
|
|
||||||
where(condition: WhereCondition): this {
|
where(condition: WhereCondition): this {
|
||||||
@@ -183,8 +194,11 @@ export class DeleteQueryBuilder {
|
|||||||
}
|
}
|
||||||
|
|
||||||
async execute(): Promise<number> {
|
async execute(): Promise<number> {
|
||||||
const count = await this.engine.delete(this.tableName, { table: this.tableName, where: this._where });
|
const query = { table: this.tableName, where: this._where };
|
||||||
|
await this.onHooks?.('beforeDelete', [query]);
|
||||||
|
const count = await this.engine.delete(this.tableName, query);
|
||||||
this.onWrite?.(this.tableName);
|
this.onWrite?.(this.tableName);
|
||||||
|
await this.onHooks?.('afterDelete', [query, count]);
|
||||||
return count;
|
return count;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+84
-13
@@ -99,11 +99,6 @@ export class QueryExecutor {
|
|||||||
this.maxRowsPerQuery = maxRowsPerQuery;
|
this.maxRowsPerQuery = maxRowsPerQuery;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** 设置查询结果行数上限 */
|
|
||||||
setMaxRowsPerQuery(max: number): void {
|
|
||||||
this.maxRowsPerQuery = max;
|
|
||||||
}
|
|
||||||
|
|
||||||
async execute(stmt: Statement): Promise<unknown> {
|
async execute(stmt: Statement): Promise<unknown> {
|
||||||
switch (stmt.type) {
|
switch (stmt.type) {
|
||||||
case 'SELECT': return this.executeSelect(stmt);
|
case 'SELECT': return this.executeSelect(stmt);
|
||||||
@@ -121,6 +116,11 @@ export class QueryExecutor {
|
|||||||
case 'BEGIN': return this.executeBegin();
|
case 'BEGIN': return this.executeBegin();
|
||||||
case 'COMMIT': return this.executeCommit();
|
case 'COMMIT': return this.executeCommit();
|
||||||
case 'ROLLBACK': return this.executeRollback();
|
case 'ROLLBACK': return this.executeRollback();
|
||||||
|
// v0.5.1: 维护语句
|
||||||
|
case 'SAVEPOINT': return this.executeSavepoint(stmt);
|
||||||
|
case 'ANALYZE': return this.executeAnalyze(stmt);
|
||||||
|
case 'REINDEX': return this.executeReindex(stmt);
|
||||||
|
case 'VACUUM': return this.executeVacuum();
|
||||||
default: throw new DatabaseError('Unknown statement type', 'UNKNOWN_STATEMENT');
|
default: throw new DatabaseError('Unknown statement type', 'UNKNOWN_STATEMENT');
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -181,7 +181,6 @@ export class QueryExecutor {
|
|||||||
|
|
||||||
/** EXPLAIN: 输出查询计划 */
|
/** EXPLAIN: 输出查询计划 */
|
||||||
private async executeExplain(stmt: { query: Statement }): Promise<Record<string, unknown>> {
|
private async executeExplain(stmt: { query: Statement }): Promise<Record<string, unknown>> {
|
||||||
const plan = compileStatement(stmt.query.type === 'SELECT' ? stmt.query : stmt.query as any);
|
|
||||||
const startTime = Date.now();
|
const startTime = Date.now();
|
||||||
let result: unknown = null;
|
let result: unknown = null;
|
||||||
try {
|
try {
|
||||||
@@ -190,15 +189,21 @@ export class QueryExecutor {
|
|||||||
const elapsed = Date.now() - startTime;
|
const elapsed = Date.now() - startTime;
|
||||||
const rows = Array.isArray(result) ? result.length : 0;
|
const rows = Array.isArray(result) ? result.length : 0;
|
||||||
|
|
||||||
|
// v0.5.1: 仅 SELECT/DELETE/UPDATE 有引擎查询计划;其他语句输出基本信息
|
||||||
|
let plan: import('../constants').QueryPlan | null = null;
|
||||||
|
try {
|
||||||
|
plan = compileStatement(stmt.query);
|
||||||
|
} catch { /* 非查询语句无 QueryPlan */ }
|
||||||
|
|
||||||
return {
|
return {
|
||||||
type: stmt.query.type,
|
type: stmt.query.type,
|
||||||
table: plan.table,
|
table: plan?.table,
|
||||||
columns: plan.columns,
|
columns: plan?.columns,
|
||||||
where: plan.where || {},
|
where: plan?.where || {},
|
||||||
orderBy: plan.orderBy || [],
|
orderBy: plan?.orderBy || [],
|
||||||
limit: plan.limit,
|
limit: plan?.limit,
|
||||||
offset: plan.offset,
|
offset: plan?.offset,
|
||||||
usingIndex: plan.table ? 'auto' : 'none',
|
usingIndex: plan?.table ? 'auto' : 'none',
|
||||||
estimatedRows: rows,
|
estimatedRows: rows,
|
||||||
actualTimeMs: elapsed,
|
actualTimeMs: elapsed,
|
||||||
};
|
};
|
||||||
@@ -776,6 +781,72 @@ export class QueryExecutor {
|
|||||||
return this.engine.rollbackTransaction();
|
return this.engine.rollbackTransaction();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ===================================================================
|
||||||
|
// 维护语句(v0.5.1)
|
||||||
|
// ===================================================================
|
||||||
|
|
||||||
|
/** SAVEPOINT name / ROLLBACK TO SAVEPOINT name / RELEASE SAVEPOINT name */
|
||||||
|
private async executeSavepoint(stmt: import('./ast').SavepointStatement): Promise<void> {
|
||||||
|
const engine = this.engine as unknown as import('./ast').SavepointStatement & {
|
||||||
|
savepoint?: (name: string) => Promise<void>;
|
||||||
|
rollbackToSavepoint?: (name: string) => Promise<void>;
|
||||||
|
releaseSavepoint?: (name: string) => Promise<void>;
|
||||||
|
};
|
||||||
|
if (stmt.action === 'SAVE') {
|
||||||
|
if (typeof engine.savepoint !== 'function') {
|
||||||
|
throw new DatabaseError(`Engine "${this.engine.name}" does not support SAVEPOINT`, 'NOT_SUPPORTED');
|
||||||
|
}
|
||||||
|
return engine.savepoint(stmt.name);
|
||||||
|
}
|
||||||
|
if (stmt.action === 'ROLLBACK') {
|
||||||
|
if (typeof engine.rollbackToSavepoint !== 'function') {
|
||||||
|
throw new DatabaseError(`Engine "${this.engine.name}" does not support ROLLBACK TO SAVEPOINT`, 'NOT_SUPPORTED');
|
||||||
|
}
|
||||||
|
return engine.rollbackToSavepoint(stmt.name);
|
||||||
|
}
|
||||||
|
if (typeof engine.releaseSavepoint !== 'function') {
|
||||||
|
throw new DatabaseError(`Engine "${this.engine.name}" does not support RELEASE SAVEPOINT`, 'NOT_SUPPORTED');
|
||||||
|
}
|
||||||
|
return engine.releaseSavepoint(stmt.name);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** ANALYZE TABLE name — 收集表统计信息 */
|
||||||
|
private async executeAnalyze(stmt: import('./ast').AnalyzeStatement): Promise<Record<string, unknown>> {
|
||||||
|
const engine = this.engine as unknown as import('./ast').AnalyzeStatement & {
|
||||||
|
analyzeTable?: (table: string) => Promise<Record<string, unknown>>;
|
||||||
|
};
|
||||||
|
if (typeof engine.analyzeTable !== 'function') {
|
||||||
|
throw new DatabaseError(`Engine "${this.engine.name}" does not support ANALYZE`, 'NOT_SUPPORTED');
|
||||||
|
}
|
||||||
|
const exists = await this.engine.hasTable(stmt.table);
|
||||||
|
if (!exists) throw new DatabaseError(`Table "${stmt.table}" does not exist`, 'TABLE_NOT_FOUND');
|
||||||
|
return engine.analyzeTable(stmt.table);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** REINDEX TABLE name — 重建表二级索引 */
|
||||||
|
private async executeReindex(stmt: import('./ast').ReindexStatement): Promise<number> {
|
||||||
|
const engine = this.engine as unknown as import('./ast').ReindexStatement & {
|
||||||
|
reindexTable?: (table: string) => Promise<number>;
|
||||||
|
};
|
||||||
|
if (typeof engine.reindexTable !== 'function') {
|
||||||
|
throw new DatabaseError(`Engine "${this.engine.name}" does not support REINDEX`, 'NOT_SUPPORTED');
|
||||||
|
}
|
||||||
|
const exists = await this.engine.hasTable(stmt.table);
|
||||||
|
if (!exists) throw new DatabaseError(`Table "${stmt.table}" does not exist`, 'TABLE_NOT_FOUND');
|
||||||
|
return engine.reindexTable(stmt.table);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** VACUUM — 压缩 LSM + 清理碎片 */
|
||||||
|
private async executeVacuum(): Promise<unknown> {
|
||||||
|
const engine = this.engine as unknown as import('./ast').VacuumStatement & {
|
||||||
|
vacuum?: () => Promise<unknown>;
|
||||||
|
};
|
||||||
|
if (typeof engine.vacuum !== 'function') {
|
||||||
|
throw new DatabaseError(`Engine "${this.engine.name}" does not support VACUUM`, 'NOT_SUPPORTED');
|
||||||
|
}
|
||||||
|
return engine.vacuum();
|
||||||
|
}
|
||||||
|
|
||||||
/** 列列表是否包含 CASE WHEN 表达式 */
|
/** 列列表是否包含 CASE WHEN 表达式 */
|
||||||
private hasCaseColumn(columns: string[]): boolean {
|
private hasCaseColumn(columns: string[]): boolean {
|
||||||
return columns.some((col) => /^\s*CASE\b/i.test(col));
|
return columns.some((col) => /^\s*CASE\b/i.test(col));
|
||||||
|
|||||||
+77
-1
@@ -72,6 +72,19 @@ export class Parser {
|
|||||||
return this.parseCommit();
|
return this.parseCommit();
|
||||||
case TokenType.ROLLBACK:
|
case TokenType.ROLLBACK:
|
||||||
return this.parseRollback();
|
return this.parseRollback();
|
||||||
|
// v0.5.1: 维护语句入口
|
||||||
|
case TokenType.EXPLAIN:
|
||||||
|
return this.parseExplain();
|
||||||
|
case TokenType.ANALYZE:
|
||||||
|
return this.parseAnalyze();
|
||||||
|
case TokenType.REINDEX:
|
||||||
|
return this.parseReindex();
|
||||||
|
case TokenType.VACUUM:
|
||||||
|
return this.parseVacuum();
|
||||||
|
case TokenType.SAVEPOINT:
|
||||||
|
return this.parseSavepoint();
|
||||||
|
case TokenType.RELEASE:
|
||||||
|
return this.parseSavepoint();
|
||||||
default:
|
default:
|
||||||
throw this.error(`Unexpected token "${this.curToken.value}"`);
|
throw this.error(`Unexpected token "${this.curToken.value}"`);
|
||||||
}
|
}
|
||||||
@@ -95,6 +108,57 @@ export class Parser {
|
|||||||
return statements;
|
return statements;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ---- 维护语句(v0.5.1) ----
|
||||||
|
|
||||||
|
/** EXPLAIN <statement> — 输出查询计划 */
|
||||||
|
private parseExplain(): import('../query/ast').ExplainStatement {
|
||||||
|
this.expect(TokenType.EXPLAIN);
|
||||||
|
if (this.curTokenIs(TokenType.EXPLAIN)) {
|
||||||
|
throw this.error('Nested EXPLAIN is not allowed');
|
||||||
|
}
|
||||||
|
const query = this.parseStatement();
|
||||||
|
return { type: 'EXPLAIN', query };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** ANALYZE [TABLE] name — 收集表统计信息 */
|
||||||
|
private parseAnalyze(): import('../query/ast').AnalyzeStatement {
|
||||||
|
this.expect(TokenType.ANALYZE);
|
||||||
|
if (this._isKeywordAsIdent() && this.curToken.value.toUpperCase() === 'TABLE') {
|
||||||
|
this.nextToken();
|
||||||
|
}
|
||||||
|
return { type: 'ANALYZE', table: this.expectIdentifier('table name') };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** REINDEX [TABLE] name — 重建表二级索引 */
|
||||||
|
private parseReindex(): import('../query/ast').ReindexStatement {
|
||||||
|
this.expect(TokenType.REINDEX);
|
||||||
|
if (this._isKeywordAsIdent() && this.curToken.value.toUpperCase() === 'TABLE') {
|
||||||
|
this.nextToken();
|
||||||
|
}
|
||||||
|
return { type: 'REINDEX', table: this.expectIdentifier('table name') };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** VACUUM — 压缩 LSM + 清理碎片 */
|
||||||
|
private parseVacuum(): import('../query/ast').VacuumStatement {
|
||||||
|
this.expect(TokenType.VACUUM);
|
||||||
|
return { type: 'VACUUM' };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** SAVEPOINT name | RELEASE [SAVEPOINT] name */
|
||||||
|
private parseSavepoint(): import('../query/ast').SavepointStatement {
|
||||||
|
let action: 'SAVE' | 'ROLLBACK' | 'RELEASE';
|
||||||
|
if (this.curTokenIs(TokenType.RELEASE)) {
|
||||||
|
action = 'RELEASE';
|
||||||
|
this.nextToken();
|
||||||
|
} else {
|
||||||
|
action = 'SAVE';
|
||||||
|
this.expect(TokenType.SAVEPOINT);
|
||||||
|
}
|
||||||
|
// 可选 SAVEPOINT 关键字(RELEASE SAVEPOINT name)
|
||||||
|
if (this.curTokenIs(TokenType.SAVEPOINT)) this.nextToken();
|
||||||
|
return { type: 'SAVEPOINT', name: this.expectIdentifier('savepoint name'), action };
|
||||||
|
}
|
||||||
|
|
||||||
// ---- 事务语句 ----
|
// ---- 事务语句 ----
|
||||||
|
|
||||||
private parseBegin(): BeginTransactionStatement {
|
private parseBegin(): BeginTransactionStatement {
|
||||||
@@ -114,10 +178,22 @@ export class Parser {
|
|||||||
return { type: 'COMMIT' };
|
return { type: 'COMMIT' };
|
||||||
}
|
}
|
||||||
|
|
||||||
private parseRollback(): RollbackTransactionStatement {
|
/** ROLLBACK [TRANSACTION] | ROLLBACK TO [SAVEPOINT] name(v0.5.1) */
|
||||||
|
private parseRollback(): Statement {
|
||||||
this.expect(TokenType.ROLLBACK);
|
this.expect(TokenType.ROLLBACK);
|
||||||
if (this._isKeywordAsIdent() && this.curToken.value.toUpperCase() === 'TRANSACTION') {
|
if (this._isKeywordAsIdent() && this.curToken.value.toUpperCase() === 'TRANSACTION') {
|
||||||
this.nextToken();
|
this.nextToken();
|
||||||
|
return { type: 'ROLLBACK' };
|
||||||
|
}
|
||||||
|
// ROLLBACK TO [SAVEPOINT] name
|
||||||
|
if (this.curTokenIs(TokenType.TO) ||
|
||||||
|
(this._isKeywordAsIdent() && this.curToken.value.toUpperCase() === 'TO')) {
|
||||||
|
this.nextToken();
|
||||||
|
if (this.curTokenIs(TokenType.SAVEPOINT) ||
|
||||||
|
(this._isKeywordAsIdent() && this.curToken.value.toUpperCase() === 'SAVEPOINT')) {
|
||||||
|
this.nextToken();
|
||||||
|
}
|
||||||
|
return { type: 'SAVEPOINT', name: this.expectIdentifier('savepoint name'), action: 'ROLLBACK' };
|
||||||
}
|
}
|
||||||
return { type: 'ROLLBACK' };
|
return { type: 'ROLLBACK' };
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -83,6 +83,15 @@ export enum TokenType {
|
|||||||
ELSE = 'ELSE',
|
ELSE = 'ELSE',
|
||||||
END = 'END',
|
END = 'END',
|
||||||
|
|
||||||
|
// v0.5.1: 维护语句(EXPLAIN / ANALYZE / REINDEX / VACUUM / SAVEPOINT)
|
||||||
|
EXPLAIN = 'EXPLAIN',
|
||||||
|
ANALYZE = 'ANALYZE',
|
||||||
|
REINDEX = 'REINDEX',
|
||||||
|
VACUUM = 'VACUUM',
|
||||||
|
SAVEPOINT = 'SAVEPOINT',
|
||||||
|
RELEASE = 'RELEASE',
|
||||||
|
TO = 'TO',
|
||||||
|
|
||||||
// 标识符 & 字面量
|
// 标识符 & 字面量
|
||||||
IDENTIFIER = 'IDENTIFIER',
|
IDENTIFIER = 'IDENTIFIER',
|
||||||
STRING = 'STRING',
|
STRING = 'STRING',
|
||||||
@@ -195,4 +204,13 @@ export const KEYWORDS: Record<string, TokenType> = {
|
|||||||
'THEN': TokenType.THEN,
|
'THEN': TokenType.THEN,
|
||||||
'ELSE': TokenType.ELSE,
|
'ELSE': TokenType.ELSE,
|
||||||
'END': TokenType.END,
|
'END': TokenType.END,
|
||||||
|
|
||||||
|
// v0.5.1
|
||||||
|
'EXPLAIN': TokenType.EXPLAIN,
|
||||||
|
'ANALYZE': TokenType.ANALYZE,
|
||||||
|
'REINDEX': TokenType.REINDEX,
|
||||||
|
'VACUUM': TokenType.VACUUM,
|
||||||
|
'SAVEPOINT': TokenType.SAVEPOINT,
|
||||||
|
'RELEASE': TokenType.RELEASE,
|
||||||
|
'TO': TokenType.TO,
|
||||||
};
|
};
|
||||||
|
|||||||
+17
-3
@@ -20,12 +20,21 @@ export class Table<T = Record<string, unknown>> {
|
|||||||
private executor: QueryExecutor | undefined;
|
private executor: QueryExecutor | undefined;
|
||||||
/** 写入回调(多标签页广播,v0.3.2) */
|
/** 写入回调(多标签页广播,v0.3.2) */
|
||||||
private onWrite?: (table: string) => void;
|
private onWrite?: (table: string) => void;
|
||||||
|
/** v0.5.1: CRUD 生命周期钩子触发回调(beforeInsert/afterInsert/... 真实接线) */
|
||||||
|
private onHooks?: (hook: import('../constants').HookName, args: unknown[]) => Promise<void>;
|
||||||
|
|
||||||
constructor(engine: IStorageEngine, tableName: string, executor?: QueryExecutor, onWrite?: (table: string) => void) {
|
constructor(
|
||||||
|
engine: IStorageEngine,
|
||||||
|
tableName: string,
|
||||||
|
executor?: QueryExecutor,
|
||||||
|
onWrite?: (table: string) => void,
|
||||||
|
onHooks?: (hook: import('../constants').HookName, args: unknown[]) => Promise<void>,
|
||||||
|
) {
|
||||||
this.engine = engine;
|
this.engine = engine;
|
||||||
this.name = tableName;
|
this.name = tableName;
|
||||||
this.executor = executor;
|
this.executor = executor;
|
||||||
this.onWrite = onWrite;
|
this.onWrite = onWrite;
|
||||||
|
this.onHooks = onHooks;
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- Schema ----
|
// ---- Schema ----
|
||||||
@@ -42,14 +51,19 @@ export class Table<T = Record<string, unknown>> {
|
|||||||
// ---- 插入 ----
|
// ---- 插入 ----
|
||||||
|
|
||||||
async insert(row: T & Record<string, unknown>): Promise<string> {
|
async insert(row: T & Record<string, unknown>): Promise<string> {
|
||||||
|
// beforeInsert 参数约定:rows[](单行时也是数组)
|
||||||
|
await this.onHooks?.('beforeInsert', [[row]]);
|
||||||
const pks = await this.engine.insert(this.name, [row as Record<string, unknown>]);
|
const pks = await this.engine.insert(this.name, [row as Record<string, unknown>]);
|
||||||
this.onWrite?.(this.name);
|
this.onWrite?.(this.name);
|
||||||
|
await this.onHooks?.('afterInsert', [[row], pks]);
|
||||||
return pks[0];
|
return pks[0];
|
||||||
}
|
}
|
||||||
|
|
||||||
async insertMany(rows: (T & Record<string, unknown>)[]): Promise<string[]> {
|
async insertMany(rows: (T & Record<string, unknown>)[]): Promise<string[]> {
|
||||||
|
await this.onHooks?.('beforeInsert', [rows]);
|
||||||
const pks = await this.engine.insert(this.name, rows as Record<string, unknown>[]);
|
const pks = await this.engine.insert(this.name, rows as Record<string, unknown>[]);
|
||||||
this.onWrite?.(this.name);
|
this.onWrite?.(this.name);
|
||||||
|
await this.onHooks?.('afterInsert', [rows, pks]);
|
||||||
return pks;
|
return pks;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -87,13 +101,13 @@ export class Table<T = Record<string, unknown>> {
|
|||||||
// ---- 更新 ----
|
// ---- 更新 ----
|
||||||
|
|
||||||
update(updates: Partial<T> & Record<string, unknown>): UpdateQueryBuilder {
|
update(updates: Partial<T> & Record<string, unknown>): UpdateQueryBuilder {
|
||||||
return new UpdateQueryBuilder(this.engine, this.name, updates, this.onWrite);
|
return new UpdateQueryBuilder(this.engine, this.name, updates, this.onWrite, this.onHooks);
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- 删除 ----
|
// ---- 删除 ----
|
||||||
|
|
||||||
delete(): DeleteQueryBuilder {
|
delete(): DeleteQueryBuilder {
|
||||||
return new DeleteQueryBuilder(this.engine, this.name, this.onWrite);
|
return new DeleteQueryBuilder(this.engine, this.name, this.onWrite, this.onHooks);
|
||||||
}
|
}
|
||||||
|
|
||||||
// ---- 聚合 ----
|
// ---- 聚合 ----
|
||||||
|
|||||||
@@ -1,80 +0,0 @@
|
|||||||
/**
|
|
||||||
* metona-sqlark Utils — 工具函数
|
|
||||||
* @module utils
|
|
||||||
*/
|
|
||||||
|
|
||||||
/** 生成唯一 ID */
|
|
||||||
export const generateId = (): string => {
|
|
||||||
return 'id-' + Date.now().toString(36) + '-' + Math.random().toString(36).slice(2, 8);
|
|
||||||
};
|
|
||||||
|
|
||||||
/** HTML 转义 */
|
|
||||||
export const escapeHTML = (s: unknown): string => {
|
|
||||||
const str = String(s == null ? '' : s);
|
|
||||||
if (typeof document === 'undefined') {
|
|
||||||
return str
|
|
||||||
.replace(/&/g, '&')
|
|
||||||
.replace(/</g, '<')
|
|
||||||
.replace(/>/g, '>')
|
|
||||||
.replace(/"/g, '"')
|
|
||||||
.replace(/'/g, ''');
|
|
||||||
}
|
|
||||||
const div = document.createElement('div');
|
|
||||||
div.textContent = str;
|
|
||||||
return div.innerHTML;
|
|
||||||
};
|
|
||||||
|
|
||||||
/** 防抖 */
|
|
||||||
export function debounce<T extends (...args: any[]) => void>(
|
|
||||||
func: T,
|
|
||||||
wait: number,
|
|
||||||
immediate = false,
|
|
||||||
): (...args: Parameters<T>) => void {
|
|
||||||
let timeout: ReturnType<typeof setTimeout> | null;
|
|
||||||
return function (this: any, ...args: Parameters<T>) {
|
|
||||||
const context = this;
|
|
||||||
const later = () => {
|
|
||||||
timeout = null;
|
|
||||||
if (!immediate) func.apply(context, args);
|
|
||||||
};
|
|
||||||
const callNow = immediate && !timeout;
|
|
||||||
if (timeout) clearTimeout(timeout);
|
|
||||||
timeout = setTimeout(later, wait);
|
|
||||||
if (callNow) func.apply(context, args);
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
/** 节流 */
|
|
||||||
export function throttle<T extends (...args: any[]) => void>(
|
|
||||||
func: T,
|
|
||||||
limit: number,
|
|
||||||
): (...args: Parameters<T>) => void {
|
|
||||||
let inThrottle = false;
|
|
||||||
return function (this: any, ...args: Parameters<T>) {
|
|
||||||
if (!inThrottle) {
|
|
||||||
func.apply(this, args);
|
|
||||||
inThrottle = true;
|
|
||||||
setTimeout(() => { inThrottle = false; }, limit);
|
|
||||||
}
|
|
||||||
};
|
|
||||||
}
|
|
||||||
|
|
||||||
/** 深度合并 */
|
|
||||||
export const deepMerge = <T extends Record<string, unknown>>(target: T, source: Partial<T>): T => {
|
|
||||||
const output: Record<string, unknown> = { ...target };
|
|
||||||
for (const key of Object.keys(source as Record<string, unknown>)) {
|
|
||||||
const sv = (source as Record<string, unknown>)[key];
|
|
||||||
const tv = (target as Record<string, unknown>)[key];
|
|
||||||
if (sv instanceof Object && key in target && tv instanceof Object) {
|
|
||||||
output[key] = deepMerge(tv as Record<string, unknown>, sv as Record<string, unknown>);
|
|
||||||
} else {
|
|
||||||
output[key] = source[key];
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return output as T;
|
|
||||||
};
|
|
||||||
|
|
||||||
/** 浏览器环境检测 */
|
|
||||||
export const isBrowser = (): boolean => {
|
|
||||||
return typeof window !== 'undefined' && typeof document !== 'undefined';
|
|
||||||
};
|
|
||||||
@@ -0,0 +1,97 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html lang="zh-CN">
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8">
|
||||||
|
<title>MetonaSqlark E2E Harness</title>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<h1>MetonaSqlark E2E Harness</h1>
|
||||||
|
<div id="status">loading</div>
|
||||||
|
<script src="/dist/metona-sqlark.js"></script>
|
||||||
|
<script>
|
||||||
|
// E2E 驱动器:暴露全局 API,测试通过 page.evaluate 调用
|
||||||
|
// 使用高层 API(MetonaSqlark.create),保证 e2e 覆盖真实入口路径
|
||||||
|
window.__ms = {
|
||||||
|
version: () => (window.MetonaSqlark ? window.MetonaSqlark.VERSION : null),
|
||||||
|
|
||||||
|
/**
|
||||||
|
* 创建并打开 Aria 库。
|
||||||
|
* opts: {
|
||||||
|
* name, diskEngine ('opfs'|'indexeddb'|'memory'),
|
||||||
|
* walSyncMode, checkpointInterval,
|
||||||
|
* encryptionPassword, memtableSizeThreshold
|
||||||
|
* }
|
||||||
|
*/
|
||||||
|
open: async (opts) => {
|
||||||
|
const db = await window.MetonaSqlark.create({
|
||||||
|
name: opts.name,
|
||||||
|
mode: 'aria',
|
||||||
|
diskEngine: opts.diskEngine || 'opfs',
|
||||||
|
aria: {
|
||||||
|
walSyncMode: opts.walSyncMode || 'full',
|
||||||
|
checkpointInterval: opts.checkpointInterval || 100000,
|
||||||
|
memtableSizeThreshold: opts.memtableSizeThreshold || 64 * 1024 * 1024,
|
||||||
|
...(opts.encryptionPassword ? { encryption: { password: opts.encryptionPassword } } : {}),
|
||||||
|
},
|
||||||
|
});
|
||||||
|
window.__ms.__db = db;
|
||||||
|
return { ok: true, engineName: db.getEngine().name };
|
||||||
|
},
|
||||||
|
|
||||||
|
createTable: async (opts) => {
|
||||||
|
const db = window.__ms.__db;
|
||||||
|
const columns = {};
|
||||||
|
for (const col of opts.columns) {
|
||||||
|
columns[col.name] = { type: col.type, primaryKey: !!col.primaryKey, index: !!col.index };
|
||||||
|
}
|
||||||
|
await db.defineTable(opts.table, columns);
|
||||||
|
return { ok: true };
|
||||||
|
},
|
||||||
|
|
||||||
|
insert: async (opts) => {
|
||||||
|
const db = window.__ms.__db;
|
||||||
|
const pks = await db.table(opts.table).insertMany(opts.rows);
|
||||||
|
return { pks };
|
||||||
|
},
|
||||||
|
|
||||||
|
count: async (opts) => {
|
||||||
|
const db = window.__ms.__db;
|
||||||
|
return { count: await db.table(opts.table).count() };
|
||||||
|
},
|
||||||
|
|
||||||
|
find: async (opts) => {
|
||||||
|
const db = window.__ms.__db;
|
||||||
|
const rows = await db.table(opts.table).select().where(opts.where || {}).execute();
|
||||||
|
return { rows };
|
||||||
|
},
|
||||||
|
|
||||||
|
query: async (opts) => {
|
||||||
|
const db = window.__ms.__db;
|
||||||
|
return { rows: await db.query(opts.sql) };
|
||||||
|
},
|
||||||
|
|
||||||
|
repair: async () => {
|
||||||
|
const db = window.__ms.__db;
|
||||||
|
await db.repair();
|
||||||
|
return { ok: true };
|
||||||
|
},
|
||||||
|
|
||||||
|
clearAll: async () => {
|
||||||
|
const db = window.__ms.__db;
|
||||||
|
await db.clearAll();
|
||||||
|
return { ok: true };
|
||||||
|
},
|
||||||
|
|
||||||
|
close: async () => {
|
||||||
|
const db = window.__ms.__db;
|
||||||
|
await db.close();
|
||||||
|
window.__ms.__db = null;
|
||||||
|
return { ok: true };
|
||||||
|
},
|
||||||
|
|
||||||
|
getEngine: () => (window.__ms.__db ? window.__ms.__db.getEngine() : null),
|
||||||
|
};
|
||||||
|
document.getElementById('status').textContent = 'ready';
|
||||||
|
</script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -0,0 +1,261 @@
|
|||||||
|
/**
|
||||||
|
* MetonaSqlark E2E — 真实 Chromium + OPFS 环境验证
|
||||||
|
*
|
||||||
|
* 覆盖:
|
||||||
|
* 1. OPFS 持久化:写入 → 刷新页面 → 数据保留(跨页面生命周期)
|
||||||
|
* 2. 优雅关闭后重开:数据完整
|
||||||
|
* 3. 崩溃恢复:写入后强制终止(不 close)→ 重开 → WAL 重放数据完整
|
||||||
|
* 4. 多标签页锁:页 A 持锁 → 页 B 打开同一库 → ARIA_LOCKED
|
||||||
|
* 5. 加密库密码往返(真实 WebCrypto)
|
||||||
|
* 6. 页面化 + OPFS 端到端
|
||||||
|
* 7. repair() 自愈(真实文件系统)
|
||||||
|
*/
|
||||||
|
import { test, expect, type Page } from '@playwright/test';
|
||||||
|
|
||||||
|
const HARNESS = '/tests/e2e/harness.html';
|
||||||
|
|
||||||
|
async function openPage(page: Page): Promise<void> {
|
||||||
|
await page.goto(HARNESS);
|
||||||
|
await expect(page.locator('#status')).toHaveText('ready');
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 在页面中执行 harness 方法(统一包装错误信息) */
|
||||||
|
async function run<T>(page: Page, method: string, args: unknown): Promise<T> {
|
||||||
|
const result = await page.evaluate(
|
||||||
|
([m, a]) => (window as unknown as { __ms: Record<string, (o: unknown) => Promise<unknown>> }).__ms[m](a),
|
||||||
|
[method, args] as const,
|
||||||
|
);
|
||||||
|
return result as T;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 强制"崩溃"当前页面:终止渲染进程(不触发 beforeunload/close) */
|
||||||
|
async function crashPage(page: Page): Promise<void> {
|
||||||
|
await page.evaluate(() => {
|
||||||
|
// 模拟崩溃:直接终止。使用 CDP Page.crash 等价效果 —— 通过无限循环让渲染进程被杀
|
||||||
|
// 但更可靠的方式是关闭页面连接(Playwright 无 Page.crash 公共 API),
|
||||||
|
// 这里用 context.close 前先把页面里的清理路径断开:
|
||||||
|
const win = window as unknown as { __ms: { __db: unknown } | null };
|
||||||
|
// 断开所有引用(模拟进程死亡:不调用 close)
|
||||||
|
win.__ms = null;
|
||||||
|
});
|
||||||
|
await page.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
test.describe('OPFS 真实环境', () => {
|
||||||
|
test('写入 → 刷新页面 → 数据保留(持久化)', async ({ page }) => {
|
||||||
|
await openPage(page);
|
||||||
|
await run(page, 'open', { name: 'e2e-persist', diskEngine: 'opfs' });
|
||||||
|
await run(page, 'createTable', {
|
||||||
|
table: 'users',
|
||||||
|
columns: [
|
||||||
|
{ name: 'id', type: 'string', primaryKey: true },
|
||||||
|
{ name: 'name', type: 'string' },
|
||||||
|
{ name: 'age', type: 'number' },
|
||||||
|
],
|
||||||
|
});
|
||||||
|
await run(page, 'insert', {
|
||||||
|
table: 'users',
|
||||||
|
rows: [
|
||||||
|
{ id: '1', name: 'Alice', age: 30 },
|
||||||
|
{ id: '2', name: 'Bob', age: 25 },
|
||||||
|
],
|
||||||
|
});
|
||||||
|
expect(await run<{ count: number }>(page, 'count', { table: 'users' })).toEqual({ count: 2 });
|
||||||
|
|
||||||
|
// 刷新页面(完全重新加载,OPFS 数据必须保留)
|
||||||
|
await page.reload();
|
||||||
|
await expect(page.locator('#status')).toHaveText('ready');
|
||||||
|
await run(page, 'open', { name: 'e2e-persist', diskEngine: 'opfs' });
|
||||||
|
const res = await run<{ count: number }>(page, 'count', { table: 'users' });
|
||||||
|
expect(res.count).toBe(2);
|
||||||
|
const found = await run<{ rows: Record<string, unknown>[] }>(page, 'find', {
|
||||||
|
table: 'users', where: { id: '1' },
|
||||||
|
});
|
||||||
|
expect(found.rows[0].name).toBe('Alice');
|
||||||
|
await run(page, 'close');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('优雅关闭后重开:数据完整', async ({ page }) => {
|
||||||
|
await openPage(page);
|
||||||
|
await run(page, 'open', { name: 'e2e-close-reopen', diskEngine: 'opfs' });
|
||||||
|
await run(page, 'createTable', {
|
||||||
|
table: 't', columns: [{ name: 'id', type: 'string', primaryKey: true }],
|
||||||
|
});
|
||||||
|
for (let i = 0; i < 20; i++) {
|
||||||
|
await run(page, 'insert', { table: 't', rows: [{ id: `r-${i}` }] });
|
||||||
|
}
|
||||||
|
await run(page, 'close');
|
||||||
|
|
||||||
|
await run(page, 'open', { name: 'e2e-close-reopen', diskEngine: 'opfs' });
|
||||||
|
expect(await run<{ count: number }>(page, 'count', { table: 't' })).toEqual({ count: 20 });
|
||||||
|
await run(page, 'close');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('崩溃恢复:写入后强制终止 → 重开 WAL 重放数据完整', async ({ page }) => {
|
||||||
|
await openPage(page);
|
||||||
|
// 不 checkpoint(interval 极大),全部写入留在 WAL
|
||||||
|
await run(page, 'open', {
|
||||||
|
name: 'e2e-crash', diskEngine: 'opfs', checkpointInterval: 999999999,
|
||||||
|
});
|
||||||
|
await run(page, 'createTable', {
|
||||||
|
table: 'logs', columns: [{ name: 'id', type: 'string', primaryKey: true }],
|
||||||
|
});
|
||||||
|
const total = 50;
|
||||||
|
for (let i = 0; i < total; i++) {
|
||||||
|
await run(page, 'insert', { table: 'logs', rows: [{ id: `log-${i}` }] });
|
||||||
|
}
|
||||||
|
// 模拟崩溃:直接终止页面(不调用 close → 不 checkpoint)
|
||||||
|
await crashPage(page);
|
||||||
|
|
||||||
|
// 新页面重开:WAL 重放
|
||||||
|
const page2 = await page.context().newPage();
|
||||||
|
await openPage(page2);
|
||||||
|
await run(page2, 'open', {
|
||||||
|
name: 'e2e-crash', diskEngine: 'opfs', checkpointInterval: 999999999,
|
||||||
|
});
|
||||||
|
const res = await run<{ count: number }>(page2, 'count', { table: 'logs' });
|
||||||
|
expect(res.count).toBe(total);
|
||||||
|
await run(page2, 'close');
|
||||||
|
await page2.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
test('多标签页锁:第二个标签页打开同一库 → ARIA_LOCKED', async ({ page }) => {
|
||||||
|
await openPage(page);
|
||||||
|
await run(page, 'open', { name: 'e2e-lock', diskEngine: 'opfs' });
|
||||||
|
await run(page, 'createTable', {
|
||||||
|
table: 't', columns: [{ name: 'id', type: 'string', primaryKey: true }],
|
||||||
|
});
|
||||||
|
|
||||||
|
// 第二个标签页尝试打开同一库
|
||||||
|
const page2 = await page.context().newPage();
|
||||||
|
await openPage(page2);
|
||||||
|
const err = await page2.evaluate(async () => {
|
||||||
|
try {
|
||||||
|
await (window as unknown as { __ms: { open: (o: unknown) => Promise<unknown> } }).__ms.open(
|
||||||
|
{ name: 'e2e-lock', diskEngine: 'opfs' },
|
||||||
|
);
|
||||||
|
return { ok: true };
|
||||||
|
} catch (e) {
|
||||||
|
return { ok: false, code: (e as { code?: string }).code, msg: (e as Error).message };
|
||||||
|
}
|
||||||
|
});
|
||||||
|
expect(err.ok).toBe(false);
|
||||||
|
expect((err as { code?: string }).code).toBe('ARIA_LOCKED');
|
||||||
|
|
||||||
|
// 第一个标签页关闭后 → 第二标签页可打开
|
||||||
|
await run(page, 'close');
|
||||||
|
const retry = await page2.evaluate(async () => {
|
||||||
|
const ms = (window as unknown as { __ms: { open: (o: unknown) => Promise<unknown>; close: () => Promise<unknown> } }).__ms;
|
||||||
|
await ms.open({ name: 'e2e-lock', diskEngine: 'opfs' });
|
||||||
|
return { ok: true };
|
||||||
|
});
|
||||||
|
expect(retry.ok).toBe(true);
|
||||||
|
await run(page2, 'close');
|
||||||
|
await page2.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
test('加密库:密码往返(真实 WebCrypto AES-GCM)', async ({ page }) => {
|
||||||
|
await openPage(page);
|
||||||
|
await run(page, 'open', { name: 'e2e-enc', diskEngine: 'opfs', encryptionPassword: 'correct-horse' });
|
||||||
|
await run(page, 'createTable', {
|
||||||
|
table: 'secrets',
|
||||||
|
columns: [
|
||||||
|
{ name: 'id', type: 'string', primaryKey: true },
|
||||||
|
{ name: 'payload', type: 'string' },
|
||||||
|
],
|
||||||
|
});
|
||||||
|
await run(page, 'insert', {
|
||||||
|
table: 'secrets',
|
||||||
|
rows: [{ id: 's1', payload: 'top-secret-content' }],
|
||||||
|
});
|
||||||
|
await run(page, 'close');
|
||||||
|
|
||||||
|
// 错误密码 → 打开失败
|
||||||
|
const page2 = await page.context().newPage();
|
||||||
|
await openPage(page2);
|
||||||
|
const err = await page2.evaluate(async () => {
|
||||||
|
try {
|
||||||
|
await (window as unknown as { __ms: { open: (o: unknown) => Promise<unknown> } }).__ms.open(
|
||||||
|
{ name: 'e2e-enc', diskEngine: 'opfs', encryptionPassword: 'wrong-password' },
|
||||||
|
);
|
||||||
|
return { ok: true };
|
||||||
|
} catch (e) {
|
||||||
|
return { ok: false, code: (e as { code?: string }).code };
|
||||||
|
}
|
||||||
|
});
|
||||||
|
expect((err as { code?: string }).code).toBe('ARIA_DECRYPT_ERROR');
|
||||||
|
|
||||||
|
// 正确密码 → 数据完整
|
||||||
|
await run(page2, 'open', { name: 'e2e-enc', diskEngine: 'opfs', encryptionPassword: 'correct-horse' });
|
||||||
|
const found = await run<{ rows: Record<string, unknown>[] }>(page2, 'find', {
|
||||||
|
table: 'secrets', where: { id: 's1' },
|
||||||
|
});
|
||||||
|
expect(found.rows[0].payload).toBe('top-secret-content');
|
||||||
|
await run(page2, 'close');
|
||||||
|
await page2.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
test('OPFS 页面化端到端:大数据量 + 二级索引查询', async ({ page }) => {
|
||||||
|
await openPage(page);
|
||||||
|
await run(page, 'open', { name: 'e2e-paged', diskEngine: 'opfs' });
|
||||||
|
await run(page, 'createTable', {
|
||||||
|
table: 'docs',
|
||||||
|
columns: [
|
||||||
|
{ name: 'id', type: 'string', primaryKey: true },
|
||||||
|
{ name: 'tag', type: 'string', index: true },
|
||||||
|
{ name: 'body', type: 'string' },
|
||||||
|
],
|
||||||
|
});
|
||||||
|
const total = 100;
|
||||||
|
for (let batch = 0; batch < 5; batch++) {
|
||||||
|
const rows = [];
|
||||||
|
for (let i = 0; i < 20; i++) {
|
||||||
|
const idx = batch * 20 + i;
|
||||||
|
rows.push({ id: `d-${idx}`, tag: idx % 3 === 0 ? 'red' : 'blue', body: '页面化端到端内容'.repeat(30) });
|
||||||
|
}
|
||||||
|
await run(page, 'insert', { table: 'docs', rows });
|
||||||
|
}
|
||||||
|
expect(await run<{ count: number }>(page, 'count', { table: 'docs' })).toEqual({ count: total });
|
||||||
|
|
||||||
|
// 二级索引查询(真实浏览器中走 LSM 索引)
|
||||||
|
const reds = await run<{ rows: Record<string, unknown>[] }>(page, 'find', {
|
||||||
|
table: 'docs', where: { tag: 'red' },
|
||||||
|
});
|
||||||
|
expect(reds.rows.length).toBeGreaterThan(0);
|
||||||
|
|
||||||
|
// SQL 查询
|
||||||
|
const sql = await run<{ rows: Record<string, unknown>[] }>(page, 'query', {
|
||||||
|
sql: 'SELECT COUNT(*) AS n FROM docs WHERE tag = \'red\'',
|
||||||
|
});
|
||||||
|
expect(Number(sql.rows[0].n)).toBe(reds.rows.length);
|
||||||
|
|
||||||
|
await run(page, 'close');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('repair() 自愈(真实文件系统)', async ({ page }) => {
|
||||||
|
await openPage(page);
|
||||||
|
await run(page, 'open', { name: 'e2e-repair', diskEngine: 'opfs' });
|
||||||
|
await run(page, 'createTable', {
|
||||||
|
table: 't',
|
||||||
|
columns: [
|
||||||
|
{ name: 'id', type: 'string', primaryKey: true },
|
||||||
|
{ name: 'v', type: 'string' },
|
||||||
|
],
|
||||||
|
});
|
||||||
|
await run(page, 'insert', { table: 't', rows: [{ id: 'a', v: 'keep-me' }] });
|
||||||
|
// 通过 SQL 触发 checkpoint + 手动 repair
|
||||||
|
await run(page, 'repair');
|
||||||
|
expect(await run<{ count: number }>(page, 'count', { table: 't' })).toEqual({ count: 1 });
|
||||||
|
// clearAll 后重建表(表结构随 clearAll 清空)
|
||||||
|
await run(page, 'clearAll');
|
||||||
|
await run(page, 'createTable', {
|
||||||
|
table: 't',
|
||||||
|
columns: [
|
||||||
|
{ name: 'id', type: 'string', primaryKey: true },
|
||||||
|
{ name: 'v', type: 'string' },
|
||||||
|
],
|
||||||
|
});
|
||||||
|
expect(await run<{ count: number }>(page, 'count', { table: 't' })).toEqual({ count: 0 });
|
||||||
|
await run(page, 'close');
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
/**
|
||||||
|
* MetonaSqlark E2E 静态文件服务器(Playwright webServer 用)
|
||||||
|
* 与 MetonaEditor 的 e2e 服务模式保持一致:Node http 静态服务,不依赖 python3。
|
||||||
|
* 用法: node tests/e2e/server.cjs [port]
|
||||||
|
*/
|
||||||
|
const http = require('http');
|
||||||
|
const fs = require('fs');
|
||||||
|
const path = require('path');
|
||||||
|
|
||||||
|
const ROOT = path.resolve(__dirname, '..', '..');
|
||||||
|
const PORT = Number(process.argv[2] || process.env.E2E_PORT || 3344);
|
||||||
|
|
||||||
|
const MIME = {
|
||||||
|
'.html': 'text/html; charset=utf-8',
|
||||||
|
'.js': 'text/javascript',
|
||||||
|
'.mjs': 'text/javascript',
|
||||||
|
'.cjs': 'text/javascript',
|
||||||
|
'.css': 'text/css',
|
||||||
|
'.json': 'application/json',
|
||||||
|
'.svg': 'image/svg+xml',
|
||||||
|
'.png': 'image/png',
|
||||||
|
};
|
||||||
|
|
||||||
|
const server = http.createServer((req, res) => {
|
||||||
|
let p = decodeURIComponent((req.url || '/').split('?')[0]);
|
||||||
|
// 根路径指向 e2e harness(playwright webServer 探测要求 2xx/3xx)
|
||||||
|
if (p === '/') p = '/tests/e2e/harness.html';
|
||||||
|
const file = path.resolve(ROOT, '.' + p);
|
||||||
|
if (!file.startsWith(ROOT) || !fs.existsSync(file) || !fs.statSync(file).isFile()) {
|
||||||
|
res.writeHead(404); res.end('404 Not Found'); return;
|
||||||
|
}
|
||||||
|
res.writeHead(200, { 'Content-Type': MIME[path.extname(file)] || 'text/plain' });
|
||||||
|
fs.createReadStream(file).pipe(res);
|
||||||
|
});
|
||||||
|
|
||||||
|
server.listen(PORT, '127.0.0.1', () => {
|
||||||
|
console.log(`E2E server · http://127.0.0.1:${PORT}/`);
|
||||||
|
});
|
||||||
@@ -113,10 +113,10 @@ describe('AriaEngine — Bloom + WAL + BufferPool', () => {
|
|||||||
freePageId: async (_id: number) => {},
|
freePageId: async (_id: number) => {},
|
||||||
};
|
};
|
||||||
const pool = new BufferPool(pageIO, 4);
|
const pool = new BufferPool(pageIO, 4);
|
||||||
const page = await pool.newPage();
|
const pages = await pool.newPages(1);
|
||||||
expect(page.pageId).toBeGreaterThan(0);
|
expect(pages[0].pageId).toBeGreaterThan(0);
|
||||||
expect(page.pins).toBe(1);
|
expect(pages[0].pins).toBe(1);
|
||||||
pool.unpin(page);
|
pool.unpin(pages[0]);
|
||||||
await backend.close();
|
await backend.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -130,11 +130,12 @@ describe('AriaEngine — Bloom + WAL + BufferPool', () => {
|
|||||||
freePageId: async (_id: number) => {},
|
freePageId: async (_id: number) => {},
|
||||||
};
|
};
|
||||||
const pool = new BufferPool(pageIO, 4);
|
const pool = new BufferPool(pageIO, 4);
|
||||||
const page = await pool.newPage();
|
const pages = await pool.newPages(1);
|
||||||
pool.markDirty(page);
|
pool.markDirty(pages[0]);
|
||||||
pool.unpin(page);
|
pool.unpin(pages[0]);
|
||||||
await pool.flushAll();
|
await pool.flushAll();
|
||||||
expect(pool.getDirtyPageCount()).toBe(0);
|
const buf = await backend.read('pg_1');
|
||||||
|
expect(buf).not.toBeNull();
|
||||||
await backend.close();
|
await backend.close();
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -44,17 +44,6 @@ describe('AriaEngine — LRUList', () => {
|
|||||||
expect(list.getLRU()!.pageId).toBe(1);
|
expect(list.getLRU()!.pageId).toBe(1);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('popLRU 移除并返回最久未使用', () => {
|
|
||||||
const list = new LRUList();
|
|
||||||
const a = makePage(1);
|
|
||||||
const b = makePage(2);
|
|
||||||
list.moveToHead(a);
|
|
||||||
list.moveToHead(b);
|
|
||||||
const popped = list.popLRU();
|
|
||||||
expect(popped!.pageId).toBe(1);
|
|
||||||
expect(list.size).toBe(1);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('remove — 从中间移除', () => {
|
it('remove — 从中间移除', () => {
|
||||||
const list = new LRUList();
|
const list = new LRUList();
|
||||||
const a = makePage(1);
|
const a = makePage(1);
|
||||||
@@ -75,11 +64,12 @@ describe('AriaEngine — LRUList', () => {
|
|||||||
expect(list.size).toBe(0);
|
expect(list.size).toBe(0);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('getAllPages 返回所有页面', () => {
|
it('remove 从尾部移除', () => {
|
||||||
const list = new LRUList();
|
const list = new LRUList();
|
||||||
list.moveToHead(makePage(1));
|
const a = makePage(1);
|
||||||
list.moveToHead(makePage(2));
|
list.moveToHead(a);
|
||||||
expect(list.getAllPages()).toHaveLength(2);
|
list.remove(a);
|
||||||
|
expect(list.size).toBe(0);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('空 LRU getLRU 返回 null', () => {
|
it('空 LRU getLRU 返回 null', () => {
|
||||||
@@ -87,11 +77,6 @@ describe('AriaEngine — LRUList', () => {
|
|||||||
expect(list.getLRU()).toBeNull();
|
expect(list.getLRU()).toBeNull();
|
||||||
});
|
});
|
||||||
|
|
||||||
it('空 LRU popLRU 返回 null', () => {
|
|
||||||
const list = new LRUList();
|
|
||||||
expect(list.popLRU()).toBeNull();
|
|
||||||
});
|
|
||||||
|
|
||||||
it('moveToHead 同元素不移重复', () => {
|
it('moveToHead 同元素不移重复', () => {
|
||||||
const list = new LRUList();
|
const list = new LRUList();
|
||||||
const p = makePage(1);
|
const p = makePage(1);
|
||||||
@@ -118,7 +103,7 @@ describe('AriaEngine — EvictionManager', () => {
|
|||||||
const p = makePage(1);
|
const p = makePage(1);
|
||||||
em.add(p);
|
em.add(p);
|
||||||
em.access(p);
|
em.access(p);
|
||||||
expect(em.getSize()).toBe(1);
|
expect((em as unknown as { lru: { size: number } }).lru.size).toBe(1);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('add 不超过容量不触发驱逐', async () => {
|
it('add 不超过容量不触发驱逐', async () => {
|
||||||
@@ -129,7 +114,7 @@ describe('AriaEngine — EvictionManager', () => {
|
|||||||
em.add(makePage(3));
|
em.add(makePage(3));
|
||||||
await em.evictIfNeeded(1);
|
await em.evictIfNeeded(1);
|
||||||
expect(evictedPages).toHaveLength(0);
|
expect(evictedPages).toHaveLength(0);
|
||||||
expect(em.getSize()).toBe(3);
|
expect((em as unknown as { lru: { size: number } }).lru.size).toBe(3);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('evictIfNeeded 超容量触发驱逐', async () => {
|
it('evictIfNeeded 超容量触发驱逐', async () => {
|
||||||
@@ -140,7 +125,7 @@ describe('AriaEngine — EvictionManager', () => {
|
|||||||
em.add(makePage(3)); // 超容量
|
em.add(makePage(3)); // 超容量
|
||||||
await em.evictIfNeeded(0);
|
await em.evictIfNeeded(0);
|
||||||
// 驱逐后 size 应 <= 2
|
// 驱逐后 size 应 <= 2
|
||||||
expect(em.getSize()).toBeLessThanOrEqual(2);
|
expect((em as unknown as { lru: { size: number } }).lru.size).toBeLessThanOrEqual(2);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('remove 减少 size', () => {
|
it('remove 减少 size', () => {
|
||||||
@@ -149,12 +134,7 @@ describe('AriaEngine — EvictionManager', () => {
|
|||||||
em.add(p);
|
em.add(p);
|
||||||
em.add(makePage(2));
|
em.add(makePage(2));
|
||||||
em.remove(p);
|
em.remove(p);
|
||||||
expect(em.getSize()).toBe(1);
|
expect((em as unknown as { lru: { size: number } }).lru.size).toBe(1);
|
||||||
});
|
|
||||||
|
|
||||||
it('getCapacity 返回配置容量', () => {
|
|
||||||
const em = new EvictionManager(128, async () => {});
|
|
||||||
expect(em.getCapacity()).toBe(128);
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it('clear 清空', () => {
|
it('clear 清空', () => {
|
||||||
@@ -162,7 +142,7 @@ describe('AriaEngine — EvictionManager', () => {
|
|||||||
em.add(makePage(1));
|
em.add(makePage(1));
|
||||||
em.add(makePage(2));
|
em.add(makePage(2));
|
||||||
em.clear();
|
em.clear();
|
||||||
expect(em.getSize()).toBe(0);
|
expect((em as unknown as { lru: { size: number } }).lru.size).toBe(0);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('脏页驱逐前调用 onEvict 回调', async () => {
|
it('脏页驱逐前调用 onEvict 回调', async () => {
|
||||||
@@ -194,7 +174,7 @@ describe('AriaEngine — BufferPool', () => {
|
|||||||
it('newPage 创建页面并 pin', async () => {
|
it('newPage 创建页面并 pin', async () => {
|
||||||
const io = new MockPageIO();
|
const io = new MockPageIO();
|
||||||
const pool = new BufferPool(io, 4);
|
const pool = new BufferPool(io, 4);
|
||||||
const page = await pool.newPage();
|
const page = (await pool.newPages(1))[0];
|
||||||
expect(page.pageId).toBe(1);
|
expect(page.pageId).toBe(1);
|
||||||
expect(page.pins).toBe(1);
|
expect(page.pins).toBe(1);
|
||||||
expect(page.type).toBe(PageType.DATA);
|
expect(page.type).toBe(PageType.DATA);
|
||||||
@@ -204,7 +184,7 @@ describe('AriaEngine — BufferPool', () => {
|
|||||||
it('getPage — 池中已存在则 pin++', async () => {
|
it('getPage — 池中已存在则 pin++', async () => {
|
||||||
const io = new MockPageIO();
|
const io = new MockPageIO();
|
||||||
const pool = new BufferPool(io, 4);
|
const pool = new BufferPool(io, 4);
|
||||||
const p1 = await pool.newPage();
|
const p1 = (await pool.newPages(1))[0];
|
||||||
pool.unpin(p1);
|
pool.unpin(p1);
|
||||||
|
|
||||||
const p2 = await pool.getPage(p1.pageId);
|
const p2 = await pool.getPage(p1.pageId);
|
||||||
@@ -216,7 +196,7 @@ describe('AriaEngine — BufferPool', () => {
|
|||||||
it('markDirty + flushPage 写回', async () => {
|
it('markDirty + flushPage 写回', async () => {
|
||||||
const io = new MockPageIO();
|
const io = new MockPageIO();
|
||||||
const pool = new BufferPool(io, 4);
|
const pool = new BufferPool(io, 4);
|
||||||
const page = await pool.newPage();
|
const page = (await pool.newPages(1))[0];
|
||||||
new Uint8Array(page.data)[100] = 42;
|
new Uint8Array(page.data)[100] = 42;
|
||||||
pool.markDirty(page);
|
pool.markDirty(page);
|
||||||
pool.unpin(page);
|
pool.unpin(page);
|
||||||
@@ -228,8 +208,8 @@ describe('AriaEngine — BufferPool', () => {
|
|||||||
it('flushAll 刷新所有脏页', async () => {
|
it('flushAll 刷新所有脏页', async () => {
|
||||||
const io = new MockPageIO();
|
const io = new MockPageIO();
|
||||||
const pool = new BufferPool(io, 8);
|
const pool = new BufferPool(io, 8);
|
||||||
const p1 = await pool.newPage();
|
const p1 = (await pool.newPages(1))[0];
|
||||||
const p2 = await pool.newPage();
|
const p2 = (await pool.newPages(1))[0];
|
||||||
pool.markDirty(p1);
|
pool.markDirty(p1);
|
||||||
pool.markDirty(p2);
|
pool.markDirty(p2);
|
||||||
pool.unpin(p1);
|
pool.unpin(p1);
|
||||||
@@ -239,35 +219,22 @@ describe('AriaEngine — BufferPool', () => {
|
|||||||
expect(io.writes).toBe(2);
|
expect(io.writes).toBe(2);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('getCachedPageCount 返回缓存数', async () => {
|
it('newPages 批量创建页面并 pin', async () => {
|
||||||
const io = new MockPageIO();
|
const io = new MockPageIO();
|
||||||
const pool = new BufferPool(io, 4);
|
const pool = new BufferPool(io, 8);
|
||||||
await pool.newPage();
|
const pages = await pool.newPages(3);
|
||||||
await pool.newPage();
|
expect(pages.map((p) => p.pageId)).toEqual([1, 2, 3]);
|
||||||
expect(pool.getCachedPageCount()).toBe(2);
|
expect(pages.every((p) => p.pins === 1)).toBe(true);
|
||||||
});
|
expect(pages.every((p) => p.type === PageType.DATA)).toBe(true);
|
||||||
|
for (const p of pages) pool.unpin(p);
|
||||||
it('getDirtyPageCount 返回脏页数', async () => {
|
|
||||||
const io = new MockPageIO();
|
|
||||||
const pool = new BufferPool(io, 4);
|
|
||||||
const p = await pool.newPage();
|
|
||||||
pool.markDirty(p);
|
|
||||||
pool.unpin(p);
|
|
||||||
expect(pool.getDirtyPageCount()).toBe(1);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('getCapacity 返回容量', async () => {
|
|
||||||
const io = new MockPageIO();
|
|
||||||
const pool = new BufferPool(io, 64);
|
|
||||||
expect(pool.getCapacity()).toBe(64);
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it('removePage 移除缓存', async () => {
|
it('removePage 移除缓存', async () => {
|
||||||
const io = new MockPageIO();
|
const io = new MockPageIO();
|
||||||
const pool = new BufferPool(io, 4);
|
const pool = new BufferPool(io, 4);
|
||||||
const p = await pool.newPage();
|
const p = (await pool.newPages(1))[0];
|
||||||
pool.unpin(p);
|
pool.unpin(p);
|
||||||
pool.removePage(p.pageId);
|
pool.removePage(p.pageId);
|
||||||
expect(pool.getCachedPageCount()).toBe(0);
|
expect((pool as unknown as { pages: Map<number, unknown> }).pages.size).toBe(0);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -53,7 +53,7 @@ describe('AriaEngine SSTable 缓存内存上限', () => {
|
|||||||
for (let round = 0; round < 5; round++) {
|
for (let round = 0; round < 5; round++) {
|
||||||
const rows = await engine.find('users', { table: 'users', where: { age: 25 } });
|
const rows = await engine.find('users', { table: 'users', where: { age: 25 } });
|
||||||
expect(rows.length).toBe(10);
|
expect(rows.length).toBe(10);
|
||||||
expect(lsm.getCacheSize()).toBeLessThanOrEqual(lsm.getCacheLimit());
|
expect(lsm.cacheSize).toBeLessThanOrEqual(lsm.cacheLimitBytes);
|
||||||
}
|
}
|
||||||
|
|
||||||
await engine.close();
|
await engine.close();
|
||||||
@@ -74,7 +74,7 @@ describe('AriaEngine SSTable 缓存内存上限', () => {
|
|||||||
expect(all.length).toBe(300);
|
expect(all.length).toBe(300);
|
||||||
|
|
||||||
const lsm = (engine as any).lsm;
|
const lsm = (engine as any).lsm;
|
||||||
expect(lsm.getCacheSize()).toBeLessThanOrEqual(lsm.getCacheLimit());
|
expect(lsm.cacheSize).toBeLessThanOrEqual(lsm.cacheLimitBytes);
|
||||||
await engine.close();
|
await engine.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -165,7 +165,7 @@ describe('AriaEngine SSTable 缓存内存上限', () => {
|
|||||||
for (let batch = 0; batch < 10; batch++) {
|
for (let batch = 0; batch < 10; batch++) {
|
||||||
await engine.insert('users', makeRows(30).map((r, i) => ({ ...r, id: `b${batch}_u${i}` })));
|
await engine.insert('users', makeRows(30).map((r, i) => ({ ...r, id: `b${batch}_u${i}` })));
|
||||||
const lsm = (engine as any).lsm;
|
const lsm = (engine as any).lsm;
|
||||||
expect(lsm.getCacheSize()).toBeLessThanOrEqual(lsm.getCacheLimit());
|
expect(lsm.cacheSize).toBeLessThanOrEqual(lsm.cacheLimitBytes);
|
||||||
}
|
}
|
||||||
|
|
||||||
const all = await engine.find('users', { table: 'users' });
|
const all = await engine.find('users', { table: 'users' });
|
||||||
|
|||||||
@@ -0,0 +1,181 @@
|
|||||||
|
/**
|
||||||
|
* AriaEngine — SSTable CRC-32 损坏检测集成测试
|
||||||
|
*
|
||||||
|
* 覆盖:
|
||||||
|
* 1. 打开时整文件 CRC 校验失败 → 损坏 SSTable 被清理(自愈),打开不阻塞
|
||||||
|
* 2. 运行期预加载 CRC 校验失败 → 损坏文件不缓存并被清理
|
||||||
|
* 3. 旧版无校验文件(checksum=0)在 LSM 层仍正常加载
|
||||||
|
*/
|
||||||
|
import { AriaEngine } from '../../src/engine/aria/index';
|
||||||
|
import { createSchema } from '../../src/table/schema';
|
||||||
|
import 'fake-indexeddb/auto';
|
||||||
|
|
||||||
|
let idbCounter = 0;
|
||||||
|
function uniqueDB(): string {
|
||||||
|
return `crc-${Date.now()}-${++idbCounter}-${Math.random().toString(36).slice(2, 8)}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
const SCHEMA = () => createSchema('users', {
|
||||||
|
id: { type: 'string', primaryKey: true },
|
||||||
|
name: { type: 'string' },
|
||||||
|
age: { type: 'number' },
|
||||||
|
});
|
||||||
|
|
||||||
|
/** 篡改 MemoryBackend 中指定 key 的一个字节 */
|
||||||
|
async function corruptKey(engine: AriaEngine, key: string, byteOffset: number): Promise<void> {
|
||||||
|
const backend = (engine as any).backend;
|
||||||
|
const raw = await backend.read(key);
|
||||||
|
expect(raw).not.toBeNull();
|
||||||
|
const buf = new Uint8Array(raw as ArrayBuffer);
|
||||||
|
buf[Math.min(byteOffset, buf.length - 1)] ^= 0xff;
|
||||||
|
await backend.write(key, buf.buffer as ArrayBuffer);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** 列出 backend 中所有 sst_ 前缀 key */
|
||||||
|
async function listSSTKeys(engine: AriaEngine): Promise<string[]> {
|
||||||
|
const backend = (engine as any).backend;
|
||||||
|
const keys = await backend.listKeys();
|
||||||
|
return (keys as string[]).filter((k) => k.startsWith('sst_'));
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('AriaEngine — SSTable CRC 损坏检测(集成)', () => {
|
||||||
|
it('打开时 CRC 损坏的 SSTable 被清理,其余数据可读', async () => {
|
||||||
|
const engine = new AriaEngine({ storageBackend: 'indexeddb', memtableSizeThreshold: 64 * 1024 * 1024 });
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
|
||||||
|
// 写入 200 行(触发多次 flush → 多个 SSTable)
|
||||||
|
for (let batch = 0; batch < 3; batch++) {
|
||||||
|
const rows = [] as Record<string, unknown>[];
|
||||||
|
for (let i = 0; i < 50; i++) {
|
||||||
|
rows.push({ id: `u-${batch * 50 + i}`, name: `User${batch * 50 + i}`, age: batch * 50 + i });
|
||||||
|
}
|
||||||
|
await engine.insert('users', rows);
|
||||||
|
await (engine as any).lsm.flush();
|
||||||
|
}
|
||||||
|
await engine.close();
|
||||||
|
|
||||||
|
// 重新打开,篡改第一个 SSTable 文件的数据区
|
||||||
|
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', memtableSizeThreshold: 64 * 1024 * 1024 });
|
||||||
|
await engine2.open(dbName, 1);
|
||||||
|
const sstKeys = await listSSTKeys(engine2);
|
||||||
|
expect(sstKeys.length).toBeGreaterThanOrEqual(2);
|
||||||
|
await corruptKey(engine2, sstKeys[0], 64);
|
||||||
|
|
||||||
|
// 再次打开:损坏文件应被跳过并清理,打开不抛错
|
||||||
|
await engine2.close();
|
||||||
|
const engine3 = new AriaEngine({ storageBackend: 'indexeddb', memtableSizeThreshold: 64 * 1024 * 1024 });
|
||||||
|
await engine3.open(dbName, 1);
|
||||||
|
|
||||||
|
// 剩余未损坏文件的数据应可查询
|
||||||
|
const remaining = await engine3.find('users', { table: 'users' });
|
||||||
|
expect(remaining.length).toBeGreaterThan(0);
|
||||||
|
expect(remaining.length).toBeLessThan(200);
|
||||||
|
|
||||||
|
// 损坏文件已被清理(meta 移除)
|
||||||
|
const afterKeys = await listSSTKeys(engine3);
|
||||||
|
expect(afterKeys).not.toContain(sstKeys[0]);
|
||||||
|
const lsm = (engine3 as any).lsm;
|
||||||
|
const metas = await lsm.sstableStore.listMeta();
|
||||||
|
expect(metas.some((m: { id: number }) => m.id === Number(sstKeys[0].replace('sst_', '')))).toBe(false);
|
||||||
|
|
||||||
|
await engine3.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('打开时损坏全部 SSTable → 库仍可打开,数据为空但不崩溃', async () => {
|
||||||
|
const engine = new AriaEngine({ storageBackend: 'indexeddb', memtableSizeThreshold: 64 * 1024 * 1024 });
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
for (let batch = 0; batch < 2; batch++) {
|
||||||
|
const rows = [] as Record<string, unknown>[];
|
||||||
|
for (let i = 0; i < 50; i++) {
|
||||||
|
rows.push({ id: `u-${batch * 50 + i}`, name: `User${batch * 50 + i}`, age: batch * 50 + i });
|
||||||
|
}
|
||||||
|
await engine.insert('users', rows);
|
||||||
|
await (engine as any).lsm.flush();
|
||||||
|
}
|
||||||
|
await engine.close();
|
||||||
|
|
||||||
|
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', memtableSizeThreshold: 64 * 1024 * 1024 });
|
||||||
|
await engine2.open(dbName, 1);
|
||||||
|
for (const key of await listSSTKeys(engine2)) {
|
||||||
|
await corruptKey(engine2, key, 16);
|
||||||
|
}
|
||||||
|
await engine2.close();
|
||||||
|
|
||||||
|
const engine3 = new AriaEngine({ storageBackend: 'indexeddb', memtableSizeThreshold: 64 * 1024 * 1024 });
|
||||||
|
// 不应抛 ARIA_OPEN_ERROR
|
||||||
|
await engine3.open(dbName, 1);
|
||||||
|
const rows3 = await engine3.find('users', { table: 'users' });
|
||||||
|
expect(rows3.length).toBe(0);
|
||||||
|
await engine3.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('运行期 CRC 损坏 → 预加载不缓存损坏文件并清理(自愈)', async () => {
|
||||||
|
const engine = new AriaEngine({ storageBackend: 'indexeddb', memtableSizeThreshold: 64 * 1024 * 1024 });
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
for (let batch = 0; batch < 2; batch++) {
|
||||||
|
const rows = [] as Record<string, unknown>[];
|
||||||
|
for (let i = 0; i < 50; i++) {
|
||||||
|
rows.push({ id: `u-${batch * 50 + i}`, name: `User${batch * 50 + i}`, age: batch * 50 + i });
|
||||||
|
}
|
||||||
|
await engine.insert('users', rows);
|
||||||
|
await (engine as any).lsm.flush();
|
||||||
|
}
|
||||||
|
|
||||||
|
// 运行期:直接篡改后端中的某个 SSTable 文件(不触发打开校验路径)
|
||||||
|
const sstKeys = await listSSTKeys(engine);
|
||||||
|
expect(sstKeys.length).toBeGreaterThanOrEqual(2);
|
||||||
|
const victim = sstKeys[sstKeys.length - 1]; // 篡改最新(memtable 已 flush 后的文件)
|
||||||
|
await corruptKey(engine, victim, 100);
|
||||||
|
|
||||||
|
// 缓存里已有该文件(flush 时缓存)→ 先清缓存模拟运行期重载
|
||||||
|
const lsm = (engine as any).lsm;
|
||||||
|
lsm.sstableCache.delete(Number(victim.replace('sst_', '')));
|
||||||
|
|
||||||
|
// 查询触发 prefetchRange → preloadSSTable 发现 CRC 失败 → 清理 + 不缓存
|
||||||
|
const remaining = await engine.find('users', { table: 'users' });
|
||||||
|
expect(remaining.length).toBeLessThan(100);
|
||||||
|
expect(lsm.sstableCache.has(Number(victim.replace('sst_', '')))).toBe(false);
|
||||||
|
const metas = await lsm.sstableStore.listMeta();
|
||||||
|
expect(metas.some((m: { id: number }) => m.id === Number(victim.replace('sst_', '')))).toBe(false);
|
||||||
|
|
||||||
|
await engine.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('旧版无校验文件(checksum=0)在 LSM 中正常加载', async () => {
|
||||||
|
const engine = new AriaEngine({ storageBackend: 'indexeddb', memtableSizeThreshold: 64 * 1024 * 1024 });
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
await engine.insert('users', [
|
||||||
|
{ id: 'a', name: 'Alice', age: 30 },
|
||||||
|
{ id: 'b', name: 'Bob', age: 25 },
|
||||||
|
]);
|
||||||
|
await (engine as any).lsm.flush();
|
||||||
|
await engine.close();
|
||||||
|
|
||||||
|
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', memtableSizeThreshold: 64 * 1024 * 1024 });
|
||||||
|
await engine2.open(dbName, 1);
|
||||||
|
const sstKeys = await listSSTKeys(engine2);
|
||||||
|
expect(sstKeys.length).toBe(1);
|
||||||
|
const backend = (engine2 as any).backend;
|
||||||
|
const raw = await backend.read(sstKeys[0]);
|
||||||
|
const buf = new Uint8Array(raw as ArrayBuffer);
|
||||||
|
new DataView(buf.buffer).setUint32(buf.byteLength - 4, 0, false); // checksum 清零 → 旧版
|
||||||
|
await backend.write(sstKeys[0], buf.buffer as ArrayBuffer);
|
||||||
|
await engine2.close();
|
||||||
|
|
||||||
|
// 重开:checksum=0 跳过校验,数据完整可读
|
||||||
|
const engine3 = new AriaEngine({ storageBackend: 'indexeddb', memtableSizeThreshold: 64 * 1024 * 1024 });
|
||||||
|
await engine3.open(dbName, 1);
|
||||||
|
const rows3 = await engine3.find('users', { table: 'users' });
|
||||||
|
expect(rows3.length).toBe(2);
|
||||||
|
expect(rows3.map((r) => r.id).sort()).toEqual(['a', 'b']);
|
||||||
|
await engine3.close();
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -9,12 +9,14 @@ import { MergeIterator, ArrayEntrySource } from '../../src/engine/aria/index/mer
|
|||||||
// LZ4 压缩 — 安全烟雾测试(不卡死)
|
// LZ4 压缩 — 安全烟雾测试(不卡死)
|
||||||
// ===================================================================
|
// ===================================================================
|
||||||
describe('AriaEngine — LZ4 Compression', () => {
|
describe('AriaEngine — LZ4 Compression', () => {
|
||||||
it('短于 4 字节时压缩为纯字面量 token 且往返一致', () => {
|
it('短于 4 字节时压缩为纯字面量 token 且往返一致(v0.4.5 带 4 字节原始大小头)', () => {
|
||||||
const input = new Uint8Array([1, 2]);
|
const input = new Uint8Array([1, 2]);
|
||||||
const compressed = compressLZ4(input);
|
const compressed = compressLZ4(input);
|
||||||
// token(lo=0) + 2 字节字面量
|
// HEADER(4) + token(lo=0) + 2 字节字面量
|
||||||
expect(compressed.byteLength).toBe(3);
|
expect(compressed.byteLength).toBe(7);
|
||||||
expect(compressed[0]).toBe(0x20); // litLen=2, matchField=0
|
// 头部 = 原始大小 2(LE)
|
||||||
|
expect(new DataView(compressed.buffer).getUint32(0, true)).toBe(2);
|
||||||
|
expect(compressed[4]).toBe(0x20); // litLen=2, matchField=0
|
||||||
const restored = decompressLZ4(compressed, 2);
|
const restored = decompressLZ4(compressed, 2);
|
||||||
expect(Array.from(restored)).toEqual([1, 2]);
|
expect(Array.from(restored)).toEqual([1, 2]);
|
||||||
});
|
});
|
||||||
@@ -54,7 +56,8 @@ describe('AriaEngine — LZ4 Compression', () => {
|
|||||||
const input = new Uint8Array(size);
|
const input = new Uint8Array(size);
|
||||||
for (let i = 0; i < size; i++) input[i] = i % 256;
|
for (let i = 0; i < size; i++) input[i] = i % 256;
|
||||||
const compressed = compressLZ4(input);
|
const compressed = compressLZ4(input);
|
||||||
expect(compressed.byteLength).toBeLessThanOrEqual(input.byteLength + 16);
|
// 头部 +4 字节开销
|
||||||
|
expect(compressed.byteLength).toBeLessThanOrEqual(input.byteLength + 4 + 16);
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -115,6 +118,35 @@ describe('AriaEngine — LZ4 Compression', () => {
|
|||||||
const restored = decompressLZ4(compressed, input.byteLength);
|
const restored = decompressLZ4(compressed, input.byteLength);
|
||||||
expect(Array.from(restored)).toEqual(Array.from(input));
|
expect(Array.from(restored)).toEqual(Array.from(input));
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it('往返一致:空输入', () => {
|
||||||
|
const compressed = compressLZ4(new Uint8Array(0));
|
||||||
|
expect(compressed.byteLength).toBe(4); // 仅头部
|
||||||
|
expect(new DataView(compressed.buffer).getUint32(0, true)).toBe(0);
|
||||||
|
const restored = decompressLZ4(compressed);
|
||||||
|
expect(restored.byteLength).toBe(0);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('往返一致:头部自描述,无需外部原始大小(高压缩率回归)', () => {
|
||||||
|
// 中文重复内容 → 极高压缩率(此前 buf.length*2 估算不足导致解压截断)
|
||||||
|
const text = '这是需要加密的敏感内容'.repeat(500);
|
||||||
|
const input = new TextEncoder().encode(text);
|
||||||
|
const compressed = compressLZ4(input);
|
||||||
|
// 压缩率验证:远小于原始
|
||||||
|
expect(compressed.byteLength).toBeLessThan(input.byteLength / 2);
|
||||||
|
// 不传 originalSize:从头部恢复
|
||||||
|
const restored = decompressLZ4(compressed);
|
||||||
|
expect(Array.from(restored)).toEqual(Array.from(input));
|
||||||
|
});
|
||||||
|
|
||||||
|
it('损坏头(原始大小非法)→ 抛错而非静默截断', () => {
|
||||||
|
const bad = new Uint8Array([0xff, 0xff, 0xff, 0xff, 0x20, 0x01, 0x02]);
|
||||||
|
expect(() => decompressLZ4(bad)).toThrow('Invalid LZ4 header');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('过短输入(无头)→ 抛错', () => {
|
||||||
|
expect(() => decompressLZ4(new Uint8Array([1, 2, 3]))).toThrow('missing header');
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
// ===================================================================
|
// ===================================================================
|
||||||
|
|||||||
@@ -5,13 +5,7 @@
|
|||||||
* v0.2.6 补强:此前仅验证实例化,现在验证真实的加解密往返一致性。
|
* v0.2.6 补强:此前仅验证实例化,现在验证真实的加解密往返一致性。
|
||||||
*/
|
*/
|
||||||
import 'fake-indexeddb/auto';
|
import 'fake-indexeddb/auto';
|
||||||
import {
|
import { CryptoManager } from '../../src/engine/aria/crypto';
|
||||||
CryptoManager,
|
|
||||||
initCrypto,
|
|
||||||
encryptPage,
|
|
||||||
decryptPage,
|
|
||||||
closeCrypto,
|
|
||||||
} from '../../src/engine/aria/crypto';
|
|
||||||
|
|
||||||
function toBytes(data: ArrayBuffer): number[] {
|
function toBytes(data: ArrayBuffer): number[] {
|
||||||
return Array.from(new Uint8Array(data));
|
return Array.from(new Uint8Array(data));
|
||||||
@@ -89,15 +83,6 @@ describe('AriaEngine — CryptoManager', () => {
|
|||||||
cm2.close();
|
cm2.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
test('全局兼容层往返一致', async () => {
|
|
||||||
await initCrypto('global-password');
|
|
||||||
const original = new TextEncoder().encode('global compat layer').buffer;
|
|
||||||
const { iv, data } = await encryptPage(original);
|
|
||||||
const decrypted = await decryptPage(iv, data);
|
|
||||||
expect(toBytes(decrypted)).toEqual(toBytes(original));
|
|
||||||
closeCrypto();
|
|
||||||
});
|
|
||||||
|
|
||||||
test('大块数据(接近页面大小)往返一致', async () => {
|
test('大块数据(接近页面大小)往返一致', async () => {
|
||||||
const cm = new CryptoManager();
|
const cm = new CryptoManager();
|
||||||
await cm.init('page-size-test');
|
await cm.init('page-size-test');
|
||||||
@@ -111,47 +96,3 @@ describe('AriaEngine — CryptoManager', () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
// ===================================================================
|
|
||||||
// 真实存储路径:SSTable 加密 → 持久化 → 重载解密(v0.3.2 CI 修复)
|
|
||||||
// ===================================================================
|
|
||||||
|
|
||||||
describe('AriaEngine — SSTable 加密存储真实路径', () => {
|
|
||||||
test('加密落盘后重载可完整解密', async () => {
|
|
||||||
// eslint-disable-next-line @typescript-eslint/no-var-requires
|
|
||||||
const { AriaEngine } = require('../../src/engine/aria/index');
|
|
||||||
// eslint-disable-next-line @typescript-eslint/no-var-requires
|
|
||||||
const { createSchema } = require('../../src/table/schema');
|
|
||||||
const dbName = `enc-store-${Date.now()}`;
|
|
||||||
|
|
||||||
await initCrypto('engine-store-password');
|
|
||||||
|
|
||||||
const engine = new AriaEngine({
|
|
||||||
storageBackend: 'indexeddb',
|
|
||||||
walSyncMode: 'none',
|
|
||||||
} as any);
|
|
||||||
await engine.open(dbName, 1);
|
|
||||||
await engine.createTable(createSchema('t', {
|
|
||||||
id: { type: 'string', primaryKey: true },
|
|
||||||
v: { type: 'number' },
|
|
||||||
}));
|
|
||||||
await engine.insert('t', [{ id: '1', v: 42 }, { id: '2', v: 99 }]);
|
|
||||||
// 强制刷盘:SSTable 经加密保存(encryptPage 真实路径)
|
|
||||||
await (engine as any).lsm.flush();
|
|
||||||
await engine.close();
|
|
||||||
|
|
||||||
// 重载:SSTable 解密恢复
|
|
||||||
const engine2 = new AriaEngine({
|
|
||||||
storageBackend: 'indexeddb',
|
|
||||||
walSyncMode: 'none',
|
|
||||||
} as any);
|
|
||||||
await engine2.open(dbName, 1);
|
|
||||||
const rows = await engine2.find('t', { table: 't' });
|
|
||||||
expect(rows).toHaveLength(2);
|
|
||||||
const byId = Object.fromEntries(rows.map((r: any) => [r.id, r.v]));
|
|
||||||
expect(byId['1']).toBe(42);
|
|
||||||
expect(byId['2']).toBe(99);
|
|
||||||
await engine2.close();
|
|
||||||
|
|
||||||
closeCrypto();
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|||||||
@@ -0,0 +1,349 @@
|
|||||||
|
/**
|
||||||
|
* AriaEngine — 全库 AES-256-GCM 加密测试
|
||||||
|
*
|
||||||
|
* 覆盖:
|
||||||
|
* 1. EncryptedBackend 单元:读写往返 / writeMany / delete / clear / 密文落盘验证
|
||||||
|
* 2. AriaEngine 集成:密码往返(open → 写入 → close → 同密码重开数据完整)
|
||||||
|
* 3. 错误密码 → ARIA_DECRYPT_ERROR
|
||||||
|
* 4. 明文旧库 + 加密配置 → ARIA_ENCRYPT_CONFIG_ERROR
|
||||||
|
* 5. 加密库 + 无密码 → ARIA_ENCRYPT_REQUIRED
|
||||||
|
* 6. clearAll 后可换新密码重建
|
||||||
|
* 7. 元数据(迁移版本)加密往返
|
||||||
|
*/
|
||||||
|
import { AriaEngine } from '../../src/engine/aria/index';
|
||||||
|
import { EncryptedBackend } from '../../src/engine/aria/store/encrypted_backend';
|
||||||
|
import { MemoryBackend } from '../../src/engine/aria/store/backend';
|
||||||
|
import { createSchema } from '../../src/table/schema';
|
||||||
|
import 'fake-indexeddb/auto';
|
||||||
|
|
||||||
|
let idbCounter = 0;
|
||||||
|
function uniqueDB(): string {
|
||||||
|
return `enc-${Date.now()}-${++idbCounter}-${Math.random().toString(36).slice(2, 8)}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
const SCHEMA = () => createSchema('users', {
|
||||||
|
id: { type: 'string', primaryKey: true },
|
||||||
|
name: { type: 'string' },
|
||||||
|
secret: { type: 'string' },
|
||||||
|
});
|
||||||
|
|
||||||
|
// ===================================================================
|
||||||
|
// EncryptedBackend 单元
|
||||||
|
// ===================================================================
|
||||||
|
describe('AriaEngine — EncryptedBackend 单元', () => {
|
||||||
|
it('open 创建 keymeta,读写往返完整', async () => {
|
||||||
|
const inner = new MemoryBackend();
|
||||||
|
const backend = new EncryptedBackend(inner, 's3cret-password');
|
||||||
|
await backend.open('enc-unit-1');
|
||||||
|
|
||||||
|
// keymeta 已创建
|
||||||
|
const keymeta = await inner.read('__aria_keymeta');
|
||||||
|
expect(keymeta).not.toBeNull();
|
||||||
|
|
||||||
|
// 读写往返
|
||||||
|
const payload = new TextEncoder().encode('hello encrypted world').buffer;
|
||||||
|
await backend.write('k1', payload);
|
||||||
|
const back = await backend.read('k1');
|
||||||
|
expect(back).not.toBeNull();
|
||||||
|
expect(new TextDecoder().decode(back)).toBe('hello encrypted world');
|
||||||
|
|
||||||
|
// 底层是密文(非明文)
|
||||||
|
const raw = await inner.read('k1');
|
||||||
|
const rawStr = new TextDecoder().decode(raw);
|
||||||
|
expect(rawStr).not.toContain('hello encrypted world');
|
||||||
|
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('writeMany / deleteMany 往返', async () => {
|
||||||
|
const inner = new MemoryBackend();
|
||||||
|
const backend = new EncryptedBackend(inner, 'pw');
|
||||||
|
await backend.open('enc-unit-2');
|
||||||
|
|
||||||
|
const enc = (s: string) => new TextEncoder().encode(s).buffer;
|
||||||
|
await backend.writeMany({ a: enc('AAA'), b: enc('BBB'), c: enc('CCC') });
|
||||||
|
expect(new TextDecoder().decode(await backend.read('a'))).toBe('AAA');
|
||||||
|
expect(new TextDecoder().decode(await backend.read('c'))).toBe('CCC');
|
||||||
|
|
||||||
|
await backend.deleteMany(['a', 'c']);
|
||||||
|
expect(await backend.exists('a')).toBe(false);
|
||||||
|
expect(await backend.exists('b')).toBe(true);
|
||||||
|
|
||||||
|
await backend.delete('b');
|
||||||
|
expect(await backend.exists('b')).toBe(false);
|
||||||
|
expect(await backend.listKeys()).toEqual(['__aria_keymeta']);
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('每个 value 独立随机 IV(相同明文两次加密密文不同)', async () => {
|
||||||
|
const inner = new MemoryBackend();
|
||||||
|
const backend = new EncryptedBackend(inner, 'pw');
|
||||||
|
await backend.open('enc-unit-3');
|
||||||
|
|
||||||
|
const payload = new TextEncoder().encode('same plaintext').buffer;
|
||||||
|
await backend.write('x', payload);
|
||||||
|
await backend.write('y', payload);
|
||||||
|
const rawX = await inner.read('x');
|
||||||
|
const rawY = await inner.read('y');
|
||||||
|
// Jest toEqual 对 ArrayBuffer 不比较内容 → 字节级比较
|
||||||
|
expect(new Uint8Array(rawX as ArrayBuffer)).not.toEqual(new Uint8Array(rawY as ArrayBuffer));
|
||||||
|
// 但 IV 前 12 字节后的长度一致(密文长度 = 明文 + GCM tag 16B)
|
||||||
|
expect((rawX as ArrayBuffer).byteLength).toBe((rawY as ArrayBuffer).byteLength);
|
||||||
|
|
||||||
|
// 但解密一致
|
||||||
|
expect(new TextDecoder().decode(await backend.read('x'))).toBe('same plaintext');
|
||||||
|
expect(new TextDecoder().decode(await backend.read('y'))).toBe('same plaintext');
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('clear 清空全部数据但保留密钥元数据(库身份保留)', async () => {
|
||||||
|
const inner = new MemoryBackend();
|
||||||
|
const backend = new EncryptedBackend(inner, 'pw');
|
||||||
|
await backend.open('enc-unit-4');
|
||||||
|
await backend.write('k', new TextEncoder().encode('v').buffer);
|
||||||
|
expect((await backend.listKeys()).length).toBe(2);
|
||||||
|
await backend.clear();
|
||||||
|
// 数据清空,keymeta 保留
|
||||||
|
expect(await backend.listKeys()).toEqual(['__aria_keymeta']);
|
||||||
|
expect(await backend.exists('k')).toBe(false);
|
||||||
|
await backend.close();
|
||||||
|
|
||||||
|
// 重新打开:keymeta 仍在,同密码可用
|
||||||
|
const backend2 = new EncryptedBackend(inner, 'pw');
|
||||||
|
await backend2.open('enc-unit-4');
|
||||||
|
await backend2.write('k2', new TextEncoder().encode('v2').buffer);
|
||||||
|
expect(new TextDecoder().decode(await backend2.read('k2'))).toBe('v2');
|
||||||
|
|
||||||
|
// 换密码被拒(keymeta 与旧密码绑定)——注意:MemoryBackend close 清空 store,
|
||||||
|
// 因此在 close 前验证(backend2 仍持有 inner)
|
||||||
|
const backend3 = new EncryptedBackend(inner, 'other-pw');
|
||||||
|
await expect(backend3.open('enc-unit-4')).rejects.toMatchObject({ code: 'ARIA_DECRYPT_ERROR' });
|
||||||
|
await backend2.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('明文旧库(有数据无 keymeta)+ 加密 → ARIA_ENCRYPT_CONFIG_ERROR', async () => {
|
||||||
|
const inner = new MemoryBackend();
|
||||||
|
await inner.open('enc-unit-5');
|
||||||
|
await inner.write('__aria_schemas', new TextEncoder().encode('{}').buffer);
|
||||||
|
|
||||||
|
const backend = new EncryptedBackend(inner, 'pw');
|
||||||
|
await expect(backend.open('enc-unit-5')).rejects.toMatchObject({ code: 'ARIA_ENCRYPT_CONFIG_ERROR' });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('空密码配置直接抛错', () => {
|
||||||
|
const inner = new MemoryBackend();
|
||||||
|
try {
|
||||||
|
new EncryptedBackend(inner, '');
|
||||||
|
fail('should throw');
|
||||||
|
} catch (error) {
|
||||||
|
expect((error as { code?: string }).code).toBe('ARIA_ENCRYPT_CONFIG_ERROR');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it('getInner / isCryptoReady 访问器', async () => {
|
||||||
|
const inner = new MemoryBackend();
|
||||||
|
const backend = new EncryptedBackend(inner, 'pw');
|
||||||
|
await backend.open('enc-unit-6');
|
||||||
|
expect(backend.getInner()).toBe(inner);
|
||||||
|
expect(backend.isCryptoReady()).toBe(true);
|
||||||
|
await backend.close();
|
||||||
|
expect(backend.isCryptoReady()).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('密文块被篡改 → 读取抛 ARIA_DECRYPT_ERROR(GCM 认证失败)', async () => {
|
||||||
|
const inner = new MemoryBackend();
|
||||||
|
const backend = new EncryptedBackend(inner, 'pw');
|
||||||
|
await backend.open('enc-unit-7');
|
||||||
|
await backend.write('k', new TextEncoder().encode('integrity-check').buffer);
|
||||||
|
|
||||||
|
// 篡改密文一个字节(绕过 IV 区)
|
||||||
|
const raw = (await inner.read('k')) as ArrayBuffer;
|
||||||
|
const buf = new Uint8Array(raw);
|
||||||
|
buf[16] ^= 0xff;
|
||||||
|
await inner.write('k', buf.buffer as ArrayBuffer);
|
||||||
|
|
||||||
|
await expect(backend.read('k')).rejects.toMatchObject({ code: 'ARIA_DECRYPT_ERROR' });
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('过短密文块 → 抛 ARIA_DECRYPT_ERROR', async () => {
|
||||||
|
const inner = new MemoryBackend();
|
||||||
|
const backend = new EncryptedBackend(inner, 'pw');
|
||||||
|
await backend.open('enc-unit-8');
|
||||||
|
await inner.write('short', new Uint8Array(4).buffer); // 不足 IV(12) + tag
|
||||||
|
await expect(backend.read('short')).rejects.toMatchObject({ code: 'ARIA_DECRYPT_ERROR' });
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ===================================================================
|
||||||
|
// AriaEngine 集成 — 加密往返
|
||||||
|
// ===================================================================
|
||||||
|
describe('AriaEngine — 全库加密(集成)', () => {
|
||||||
|
it('加密库:写入 → close → 同密码重开数据完整', async () => {
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
const engine = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'master-pass' } });
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
await engine.insert('users', [
|
||||||
|
{ id: '1', name: 'Alice', secret: 'top-secret-a' },
|
||||||
|
{ id: '2', name: 'Bob', secret: 'top-secret-b' },
|
||||||
|
]);
|
||||||
|
await engine.setMeta('__metona_version', '3');
|
||||||
|
await engine.close();
|
||||||
|
|
||||||
|
// 重开(同密码)
|
||||||
|
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'master-pass' } });
|
||||||
|
await engine2.open(dbName, 1);
|
||||||
|
const rows = await engine2.find('users', { table: 'users' });
|
||||||
|
expect(rows).toHaveLength(2);
|
||||||
|
const byId = Object.fromEntries(rows.map((r) => [r.id, r]));
|
||||||
|
expect(byId['1'].secret).toBe('top-secret-a');
|
||||||
|
expect(byId['2'].secret).toBe('top-secret-b');
|
||||||
|
expect(await engine2.getMeta('__metona_version')).toBe('3');
|
||||||
|
await engine2.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('加密库:错误密码重开 → ARIA_DECRYPT_ERROR', async () => {
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
const engine = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'right-pass' } });
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
await engine.insert('users', [{ id: '1', name: 'A', secret: 'x' }]);
|
||||||
|
await engine.close();
|
||||||
|
|
||||||
|
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'wrong-pass' } });
|
||||||
|
await expect(engine2.open(dbName, 1)).rejects.toMatchObject({ code: 'ARIA_DECRYPT_ERROR' });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('加密库:无密码打开 → ARIA_ENCRYPT_REQUIRED', async () => {
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
const engine = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'pw' } });
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
await engine.close();
|
||||||
|
|
||||||
|
const engine2 = new AriaEngine({ storageBackend: 'indexeddb' });
|
||||||
|
await expect(engine2.open(dbName, 1)).rejects.toMatchObject({ code: 'ARIA_ENCRYPT_REQUIRED' });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('明文库:加密配置打开 → ARIA_ENCRYPT_CONFIG_ERROR', async () => {
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
const engine = new AriaEngine({ storageBackend: 'indexeddb' });
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
await engine.insert('users', [{ id: '1', name: 'A', secret: 'plain' }]);
|
||||||
|
await engine.close();
|
||||||
|
|
||||||
|
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'pw' } });
|
||||||
|
await expect(engine2.open(dbName, 1)).rejects.toMatchObject({ code: 'ARIA_ENCRYPT_CONFIG_ERROR' });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('加密库:底层存储全部为密文(SSTable/Schema/WAL 均不可见明文)', async () => {
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
const engine = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'pw' } });
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
await engine.insert('users', [{ id: '1', name: 'Alice', secret: 'needle-in-cipher' }]);
|
||||||
|
await (engine as any).lsm.flush();
|
||||||
|
// 强制落盘后 WAL 仍有记录(checkpoint 截断前)→ 全量扫描检查
|
||||||
|
const backend = (engine as any).backend as EncryptedBackend;
|
||||||
|
const inner = backend.getInner();
|
||||||
|
const keys = await inner.listKeys();
|
||||||
|
expect(keys.length).toBeGreaterThan(0);
|
||||||
|
|
||||||
|
// 任何一个底层 key 的内容都不包含明文数据(需解密才可见)
|
||||||
|
for (const key of keys) {
|
||||||
|
if (key === '__aria_keymeta') continue;
|
||||||
|
const raw = await inner.read(key);
|
||||||
|
const str = new TextDecoder().decode(raw as ArrayBuffer);
|
||||||
|
expect(str).not.toContain('needle-in-cipher');
|
||||||
|
expect(str).not.toContain('Alice');
|
||||||
|
expect(str).not.toContain('users');
|
||||||
|
}
|
||||||
|
await engine.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('加密 + 压缩组合:往返完整(压缩层先压,加密层后加密)', async () => {
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
const engine = new AriaEngine({
|
||||||
|
storageBackend: 'indexeddb',
|
||||||
|
compression: true,
|
||||||
|
encryption: { password: 'pw-compress' },
|
||||||
|
memtableSizeThreshold: 64 * 1024 * 1024,
|
||||||
|
});
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
const rows = [] as Record<string, unknown>[];
|
||||||
|
for (let i = 0; i < 200; i++) {
|
||||||
|
rows.push({ id: `u-${i}`, name: `User${i}`, secret: '这是需要加密的敏感内容'.repeat(5) });
|
||||||
|
}
|
||||||
|
await engine.insert('users', rows);
|
||||||
|
await (engine as any).lsm.flush();
|
||||||
|
await engine.close();
|
||||||
|
|
||||||
|
const engine2 = new AriaEngine({
|
||||||
|
storageBackend: 'indexeddb',
|
||||||
|
compression: true,
|
||||||
|
encryption: { password: 'pw-compress' },
|
||||||
|
});
|
||||||
|
await engine2.open(dbName, 1);
|
||||||
|
expect(await engine2.count('users')).toBe(200);
|
||||||
|
const one = await engine2.find('users', { table: 'users', where: { id: 'u-42' } });
|
||||||
|
expect(one[0].secret).toBe('这是需要加密的敏感内容'.repeat(5));
|
||||||
|
await engine2.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('加密库:clearAll 保留密钥(库身份),同密码重开重建,新密码被拒', async () => {
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
const engine = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'old-pw' } });
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
await engine.insert('users', [{ id: '1', name: 'A', secret: 'x' }]);
|
||||||
|
await engine.clearAll();
|
||||||
|
await engine.close();
|
||||||
|
|
||||||
|
// clearAll 保留 keymeta → 换新密码会被拒绝
|
||||||
|
const engineWrong = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'new-pw' } });
|
||||||
|
await expect(engineWrong.open(dbName, 1)).rejects.toMatchObject({ code: 'ARIA_DECRYPT_ERROR' });
|
||||||
|
|
||||||
|
// 同密码重开可重建
|
||||||
|
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'old-pw' } });
|
||||||
|
await engine2.open(dbName, 1);
|
||||||
|
await engine2.createTable(SCHEMA());
|
||||||
|
await engine2.insert('users', [{ id: '1', name: 'A', secret: 'new' }]);
|
||||||
|
expect(await engine2.count('users')).toBe(1);
|
||||||
|
await engine2.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('加密库:WAL 崩溃恢复路径可用(含加密 WAL)', async () => {
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
const engine = new AriaEngine({
|
||||||
|
storageBackend: 'indexeddb',
|
||||||
|
encryption: { password: 'pw' },
|
||||||
|
walSyncMode: 'full',
|
||||||
|
checkpointInterval: 100000,
|
||||||
|
});
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
// 写入后不 flush 不 checkpoint,直接"崩溃"(close 会 checkpoint,这里模拟崩溃:直接断 backend)
|
||||||
|
for (let i = 0; i < 10; i++) {
|
||||||
|
await engine.insert('users', [{ id: `w-${i}`, name: `W${i}`, secret: `s${i}` }]);
|
||||||
|
}
|
||||||
|
// 模拟崩溃:不 close,直接断开(真实崩溃时 WAL 留在存储中)
|
||||||
|
await (engine as any).backend.close();
|
||||||
|
(engine as any).opened = false;
|
||||||
|
|
||||||
|
// 重开:WAL 重放(加密 WAL 解密后解析)
|
||||||
|
const engine2 = new AriaEngine({
|
||||||
|
storageBackend: 'indexeddb',
|
||||||
|
encryption: { password: 'pw' },
|
||||||
|
walSyncMode: 'full',
|
||||||
|
checkpointInterval: 100000,
|
||||||
|
});
|
||||||
|
await engine2.open(dbName, 1);
|
||||||
|
const rows = await engine2.find('users', { table: 'users' });
|
||||||
|
expect(rows).toHaveLength(10);
|
||||||
|
await engine2.close();
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -46,19 +46,6 @@ describe('AriaEngine — BloomFilter', () => {
|
|||||||
expect(fp).toBeLessThan(25);
|
expect(fp).toBeLessThan(25);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('getBitSize 返回正确位数', () => {
|
|
||||||
const bf = new BloomFilter(100, 10);
|
|
||||||
expect(bf.getBitSize()).toBeGreaterThanOrEqual(64);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('getInsertedCount 追踪插入数', () => {
|
|
||||||
const bf = new BloomFilter(100);
|
|
||||||
bf.insert('a');
|
|
||||||
bf.insert('b');
|
|
||||||
bf.insert('c');
|
|
||||||
expect(bf.getInsertedCount()).toBe(3);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('getHashCount 返回哈希函数数量', () => {
|
it('getHashCount 返回哈希函数数量', () => {
|
||||||
const bf = new BloomFilter(1000, 10);
|
const bf = new BloomFilter(1000, 10);
|
||||||
expect(bf.getHashCount()).toBeGreaterThan(0);
|
expect(bf.getHashCount()).toBeGreaterThan(0);
|
||||||
@@ -161,11 +148,6 @@ describe('AriaEngine — MemTable', () => {
|
|||||||
expect(mt.getEntryCount()).toBe(1);
|
expect(mt.getEntryCount()).toBe(1);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('contains 检查存在性', () => {
|
|
||||||
mt.put('x', { v: 1 });
|
|
||||||
expect(mt.contains('x')).toBe(true);
|
|
||||||
expect(mt.contains('y')).toBe(false);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('shouldFlush — 未达阈值返回 false', () => {
|
it('shouldFlush — 未达阈值返回 false', () => {
|
||||||
expect(mt.shouldFlush()).toBe(false);
|
expect(mt.shouldFlush()).toBe(false);
|
||||||
|
|||||||
@@ -0,0 +1,170 @@
|
|||||||
|
/**
|
||||||
|
* AriaEngine — 多标签页独占锁(Web Locks)测试
|
||||||
|
*
|
||||||
|
* 覆盖:
|
||||||
|
* 1. 第一个引擎持锁 → 第二个引擎打开抛 ARIA_LOCKED
|
||||||
|
* 2. 关闭后锁释放 → 可再次打开
|
||||||
|
* 3. 环境不支持 Web Locks → 降级打开(返回 false 语义 + 警告)
|
||||||
|
* 4. open 失败(如加密密码错误)→ 锁释放(其他标签页可打开)
|
||||||
|
* 5. 锁名按库隔离(不同库不互斥)
|
||||||
|
*/
|
||||||
|
import { AriaEngine } from '../../src/engine/aria/index';
|
||||||
|
import { DatabaseLock, lockName } from '../../src/engine/aria/locks';
|
||||||
|
import { createSchema } from '../../src/table/schema';
|
||||||
|
import 'fake-indexeddb/auto';
|
||||||
|
|
||||||
|
let idbCounter = 0;
|
||||||
|
function uniqueDB(): string {
|
||||||
|
return `lock-${Date.now()}-${++idbCounter}-${Math.random().toString(36).slice(2, 8)}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ===================================================================
|
||||||
|
// Web Locks mock:真实语义(ifAvailable / 回调结束锁自动释放)
|
||||||
|
// ===================================================================
|
||||||
|
let lockHolders: Map<string, true>;
|
||||||
|
|
||||||
|
function installLocksMock() {
|
||||||
|
lockHolders = new Map();
|
||||||
|
Object.defineProperty(globalThis, 'navigator', {
|
||||||
|
value: {
|
||||||
|
locks: {
|
||||||
|
request: async (_name: string, opts: { ifAvailable?: boolean }, cb: (lock: { name: string } | null) => Promise<void> | void) => {
|
||||||
|
const name = _name;
|
||||||
|
if (lockHolders.has(name)) {
|
||||||
|
if (opts?.ifAvailable) {
|
||||||
|
await cb(null);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
throw new Error('unreachable: non-ifAvailable request not used');
|
||||||
|
}
|
||||||
|
lockHolders.set(name, true);
|
||||||
|
try {
|
||||||
|
// 真实 Web Locks 语义:回调结束 → 锁自动释放
|
||||||
|
await cb({ name });
|
||||||
|
} finally {
|
||||||
|
lockHolders.delete(name);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
configurable: true,
|
||||||
|
writable: true,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function removeLocksMock() {
|
||||||
|
Object.defineProperty(globalThis, 'navigator', {
|
||||||
|
value: {},
|
||||||
|
configurable: true,
|
||||||
|
writable: true,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// ===================================================================
|
||||||
|
// DatabaseLock 单元
|
||||||
|
// ===================================================================
|
||||||
|
describe('AriaEngine — DatabaseLock 单元', () => {
|
||||||
|
beforeEach(installLocksMock);
|
||||||
|
|
||||||
|
it('acquire 持锁 → 二次 acquire 抛 ARIA_LOCKED → release 后恢复', async () => {
|
||||||
|
const lock1 = new DatabaseLock();
|
||||||
|
expect(await lock1.acquire('db-a')).toBe(true);
|
||||||
|
expect(lock1.isAcquired()).toBe(true);
|
||||||
|
expect(lock1.isSupported()).toBe(true);
|
||||||
|
expect(lockHolders.has(lockName('db-a'))).toBe(true);
|
||||||
|
|
||||||
|
const lock2 = new DatabaseLock();
|
||||||
|
await expect(lock2.acquire('db-a')).rejects.toMatchObject({ code: 'ARIA_LOCKED' });
|
||||||
|
expect(lock2.isAcquired()).toBe(false);
|
||||||
|
|
||||||
|
await lock1.release();
|
||||||
|
expect(lock1.isAcquired()).toBe(false);
|
||||||
|
expect(lockHolders.has(lockName('db-a'))).toBe(false);
|
||||||
|
|
||||||
|
const lock3 = new DatabaseLock();
|
||||||
|
expect(await lock3.acquire('db-a')).toBe(true);
|
||||||
|
await lock3.release();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('不同库名互不阻塞', async () => {
|
||||||
|
const l1 = new DatabaseLock();
|
||||||
|
const l2 = new DatabaseLock();
|
||||||
|
expect(await l1.acquire('db-x')).toBe(true);
|
||||||
|
expect(await l2.acquire('db-y')).toBe(true);
|
||||||
|
await l1.release();
|
||||||
|
await l2.release();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('无 Web Locks 环境 → acquire 返回 false(降级)', async () => {
|
||||||
|
removeLocksMock();
|
||||||
|
const lock = new DatabaseLock();
|
||||||
|
expect(await lock.acquire('db-z')).toBe(false);
|
||||||
|
expect(lock.isSupported()).toBe(false);
|
||||||
|
expect(lock.isAcquired()).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('重复 release 幂等', async () => {
|
||||||
|
const lock = new DatabaseLock();
|
||||||
|
await lock.acquire('db-r');
|
||||||
|
await lock.release();
|
||||||
|
await lock.release();
|
||||||
|
expect(lock.isAcquired()).toBe(false);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ===================================================================
|
||||||
|
// AriaEngine 集成 — 多标签页锁
|
||||||
|
// ===================================================================
|
||||||
|
describe('AriaEngine — 多标签页锁(集成)', () => {
|
||||||
|
beforeEach(installLocksMock);
|
||||||
|
|
||||||
|
it('第二个标签页打开同一库 → ARIA_LOCKED', async () => {
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
const engine1 = new AriaEngine({ storageBackend: 'indexeddb' });
|
||||||
|
await engine1.open(dbName, 1);
|
||||||
|
await engine1.createTable(createSchema('t', { id: { type: 'string', primaryKey: true } }));
|
||||||
|
|
||||||
|
const engine2 = new AriaEngine({ storageBackend: 'indexeddb' });
|
||||||
|
await expect(engine2.open(dbName, 1)).rejects.toMatchObject({ code: 'ARIA_LOCKED' });
|
||||||
|
// engine2 未持锁、未 opened
|
||||||
|
expect((engine2 as any).opened).toBe(false);
|
||||||
|
|
||||||
|
await engine1.close();
|
||||||
|
|
||||||
|
// 释放后可打开
|
||||||
|
const engine3 = new AriaEngine({ storageBackend: 'indexeddb' });
|
||||||
|
await engine3.open(dbName, 1);
|
||||||
|
expect(await engine3.count('t')).toBe(0);
|
||||||
|
await engine3.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('open 失败(错误密码)→ 锁释放,其他标签页可打开', async () => {
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
const engine1 = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'pw' } });
|
||||||
|
await engine1.open(dbName, 1);
|
||||||
|
await engine1.createTable(createSchema('t', { id: { type: 'string', primaryKey: true } }));
|
||||||
|
await engine1.close();
|
||||||
|
|
||||||
|
// 错误密码 → open 失败
|
||||||
|
const engineBad = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'wrong' } });
|
||||||
|
await expect(engineBad.open(dbName, 1)).rejects.toMatchObject({ code: 'ARIA_DECRYPT_ERROR' });
|
||||||
|
// 失败后锁已释放(未泄漏)
|
||||||
|
expect(lockHolders.has(lockName(dbName))).toBe(false);
|
||||||
|
|
||||||
|
// 正确密码可打开
|
||||||
|
const engineOk = new AriaEngine({ storageBackend: 'indexeddb', encryption: { password: 'pw' } });
|
||||||
|
await engineOk.open(dbName, 1);
|
||||||
|
await engineOk.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('无 Web Locks 环境 → 降级打开(不抛错)', async () => {
|
||||||
|
removeLocksMock();
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
const engine = new AriaEngine({ storageBackend: 'indexeddb' });
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(createSchema('t', { id: { type: 'string', primaryKey: true } }));
|
||||||
|
await engine.insert('t', [{ id: '1' }]);
|
||||||
|
expect(await engine.count('t')).toBe(1);
|
||||||
|
await engine.close();
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -81,23 +81,6 @@ describe('AriaEngine — ANALYZE/VACUUM/REINDEX + EXPLAIN', () => {
|
|||||||
expect(plan.actualTimeMs).toBeGreaterThanOrEqual(0);
|
expect(plan.actualTimeMs).toBeGreaterThanOrEqual(0);
|
||||||
});
|
});
|
||||||
|
|
||||||
// ---- 查询优化器 ----
|
|
||||||
it('estimateQueryCost PK 等值 → pk_lookup', () => {
|
|
||||||
const cost = (engine as any).estimateQueryCost('users', { table: 'users', where: { id: '1' } });
|
|
||||||
expect(cost.strategy).toBe('pk_lookup');
|
|
||||||
expect(cost.estimatedRows).toBe(1);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('estimateQueryCost 索引等值 → index_eq', () => {
|
|
||||||
const cost = (engine as any).estimateQueryCost('users', { table: 'users', where: { name: 'Alice' } });
|
|
||||||
expect(cost.strategy).toBe('index_eq:name');
|
|
||||||
});
|
|
||||||
|
|
||||||
it('estimateQueryCost 无索引 → full_scan', () => {
|
|
||||||
const cost = (engine as any).estimateQueryCost('users', { table: 'users', where: { age: { $gt: 20 } } });
|
|
||||||
expect(cost.strategy).toMatch(/index_range|full_scan/);
|
|
||||||
});
|
|
||||||
|
|
||||||
// ---- 在线备份 ----
|
// ---- 在线备份 ----
|
||||||
it('backup 导出全库一致性快照', async () => {
|
it('backup 导出全库一致性快照', async () => {
|
||||||
const backup = await (engine as any).backup();
|
const backup = await (engine as any).backup();
|
||||||
|
|||||||
@@ -27,31 +27,20 @@ describe('AriaEngine — MVCC 事务 + Savepoint', () => {
|
|||||||
expect(id1).not.toBe(id2);
|
expect(id1).not.toBe(id2);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('commit 后事务标记为非活跃', () => {
|
it('commit 后版本标记已提交', () => {
|
||||||
const txnId = (engine as any).mvcc.beginTransaction();
|
const txnId = (engine as any).mvcc.beginTransaction();
|
||||||
|
(engine as any).mvcc.writeVersion('users', '1', { name: 'Alice' }, txnId);
|
||||||
(engine as any).mvcc.commitTransaction(txnId);
|
(engine as any).mvcc.commitTransaction(txnId);
|
||||||
expect((engine as any).mvcc.isActive(txnId)).toBe(false);
|
const store = (engine as any).mvcc.versionStore;
|
||||||
|
expect(store.get('users.1')[0].committed).toBe(true);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('rollback 后事务标记为非活跃', () => {
|
it('rollback 移除本事务版本', () => {
|
||||||
const txnId = (engine as any).mvcc.beginTransaction();
|
const txnId = (engine as any).mvcc.beginTransaction();
|
||||||
|
(engine as any).mvcc.writeVersion('users', '1', { name: 'Alice' }, txnId);
|
||||||
(engine as any).mvcc.rollbackTransaction(txnId);
|
(engine as any).mvcc.rollbackTransaction(txnId);
|
||||||
expect((engine as any).mvcc.isActive(txnId)).toBe(false);
|
const store = (engine as any).mvcc.versionStore;
|
||||||
});
|
expect(store.has('users.1')).toBe(false);
|
||||||
|
|
||||||
it('事务中写入版本可读', () => {
|
|
||||||
const txnId = (engine as any).mvcc.beginTransaction();
|
|
||||||
(engine as any).mvcc.writeVersion('users', '1', { name: 'Alice', balance: 100 }, txnId);
|
|
||||||
const val = (engine as any).mvcc.readVersion('users', '1', txnId);
|
|
||||||
expect(val).not.toBeNull();
|
|
||||||
expect(val!.name).toBe('Alice');
|
|
||||||
});
|
|
||||||
|
|
||||||
it('未提交版本对其他事务不可见', () => {
|
|
||||||
const txn1 = (engine as any).mvcc.beginTransaction();
|
|
||||||
(engine as any).mvcc.writeVersion('users', '1', { name: 'Alice' }, txn1);
|
|
||||||
const txn2 = (engine as any).mvcc.beginTransaction();
|
|
||||||
expect((engine as any).mvcc.readVersion('users', '1', txn2)).toBeNull();
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it('引擎层事务 commit 后数据可见', async () => {
|
it('引擎层事务 commit 后数据可见', async () => {
|
||||||
@@ -69,16 +58,14 @@ describe('AriaEngine — MVCC 事务 + Savepoint', () => {
|
|||||||
expect(await engine.count('users')).toBe(1);
|
expect(await engine.count('users')).toBe(1);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('GC 清理过旧版本', () => {
|
it('GC 清理过旧版本(保留最新 N 个)', () => {
|
||||||
for (let i = 0; i < 200; i++) {
|
for (let i = 0; i < 200; i++) {
|
||||||
const txnId = (engine as any).mvcc.beginTransaction();
|
const txnId = (engine as any).mvcc.beginTransaction();
|
||||||
(engine as any).mvcc.writeVersion('users', '1', { ver: i }, txnId);
|
(engine as any).mvcc.writeVersion('users', '1', { ver: i }, txnId);
|
||||||
(engine as any).mvcc.commitTransaction(txnId);
|
(engine as any).mvcc.commitTransaction(txnId);
|
||||||
}
|
}
|
||||||
(engine as any).mvcc.gc(50);
|
(engine as any).mvcc.gc(50);
|
||||||
const txnId = (engine as any).mvcc.beginTransaction();
|
expect((engine as any).mvcc.versionStore.get('users.1').length).toBe(50);
|
||||||
expect((engine as any).mvcc.readVersion('users', '1', txnId)).not.toBeNull();
|
|
||||||
(engine as any).mvcc.commitTransaction(txnId);
|
|
||||||
});
|
});
|
||||||
|
|
||||||
// ---- Savepoint ----
|
// ---- Savepoint ----
|
||||||
|
|||||||
@@ -0,0 +1,228 @@
|
|||||||
|
/**
|
||||||
|
* AriaEngine — OPFSBackend v2 测试
|
||||||
|
*
|
||||||
|
* 覆盖:
|
||||||
|
* 1. read/write/append/exists/delete/clear 基本语义
|
||||||
|
* 2. append 追加语义(keepExistingData + position)与字节顺序
|
||||||
|
* 3. 写队列串行化(并发写顺序一致)
|
||||||
|
* 4. 单次任务失败不中断队列链(后续写正常)
|
||||||
|
* 5. close 等待挂起写完成
|
||||||
|
* 6. open 清理崩溃残留临时文件(.crswap/.tmp)
|
||||||
|
* 7. writeMany/deleteMany 语义
|
||||||
|
*/
|
||||||
|
import { OPFSBackend } from '../../src/engine/aria/store/opfs_backend';
|
||||||
|
|
||||||
|
// ===================================================================
|
||||||
|
// 真实语义 OPFS mock:记录文件内容、支持 keepExistingData+position
|
||||||
|
// ===================================================================
|
||||||
|
interface MockFile {
|
||||||
|
content: ArrayBuffer;
|
||||||
|
writeCalls: { position?: number; data: ArrayBuffer; keepExistingData?: boolean }[];
|
||||||
|
}
|
||||||
|
|
||||||
|
function createOPFSMock() {
|
||||||
|
const files = new Map<string, MockFile>();
|
||||||
|
|
||||||
|
const getFileHandle = async (name: string, opts?: { create?: boolean }) => {
|
||||||
|
if (!files.has(name)) {
|
||||||
|
if (!opts?.create) throw new Error(`NotFoundError: ${name}`);
|
||||||
|
files.set(name, { content: new ArrayBuffer(0), writeCalls: [] });
|
||||||
|
}
|
||||||
|
const entry = files.get(name)!;
|
||||||
|
return {
|
||||||
|
getFile: async () => ({ size: entry.content.byteLength, arrayBuffer: async () => entry.content }),
|
||||||
|
createWritable: async (wOpts?: { keepExistingData?: boolean }) => {
|
||||||
|
const w: {
|
||||||
|
write: (arg: ArrayBuffer | { type: string; position: number; data: ArrayBuffer }) => Promise<void>;
|
||||||
|
close: () => Promise<void>;
|
||||||
|
} = {
|
||||||
|
write: async (arg) => {
|
||||||
|
const keepExisting = wOpts?.keepExistingData ?? false;
|
||||||
|
const isChunk = typeof arg !== 'object' || !('type' in (arg as object)) || (arg as { type?: string }).type === undefined
|
||||||
|
? { data: arg as ArrayBuffer, position: keepExisting ? entry.content.byteLength : 0 }
|
||||||
|
: { data: (arg as { data: ArrayBuffer }).data, position: (arg as { position: number }).position };
|
||||||
|
entry.writeCalls.push({ position: isChunk.position, data: isChunk.data, keepExistingData: keepExisting });
|
||||||
|
const merged = new Uint8Array(isChunk.position + isChunk.data.byteLength);
|
||||||
|
if (keepExisting || isChunk.position > 0) {
|
||||||
|
merged.set(new Uint8Array(entry.content.subarray ? entry.content : entry.content), 0);
|
||||||
|
}
|
||||||
|
merged.set(new Uint8Array(isChunk.data), isChunk.position);
|
||||||
|
entry.content = merged.buffer;
|
||||||
|
},
|
||||||
|
close: async () => { /* no-op */ },
|
||||||
|
};
|
||||||
|
return w;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
const dir = {
|
||||||
|
getFileHandle,
|
||||||
|
entries: async function* () {
|
||||||
|
for (const [name] of files) yield [name];
|
||||||
|
},
|
||||||
|
removeEntry: async (name: string) => {
|
||||||
|
files.delete(name);
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
Object.defineProperty(globalThis, 'navigator', {
|
||||||
|
value: {
|
||||||
|
storage: {
|
||||||
|
getDirectory: async () => ({
|
||||||
|
getDirectoryHandle: async (_name: string, _opts?: unknown) => dir,
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
configurable: true,
|
||||||
|
writable: true,
|
||||||
|
});
|
||||||
|
|
||||||
|
return { files, dir };
|
||||||
|
}
|
||||||
|
|
||||||
|
const enc = (s: string) => new Uint8Array(new TextEncoder().encode(s)).buffer;
|
||||||
|
|
||||||
|
describe('AriaEngine — OPFSBackend v2', () => {
|
||||||
|
beforeEach(() => {
|
||||||
|
createOPFSMock();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('read/write/exists/delete 基本语义', async () => {
|
||||||
|
const backend = new OPFSBackend();
|
||||||
|
await backend.open('opfs-test-1');
|
||||||
|
expect(backend.isOpen()).toBe(true);
|
||||||
|
|
||||||
|
await backend.write('k1', enc('hello'));
|
||||||
|
expect(await backend.exists('k1')).toBe(true);
|
||||||
|
expect(new TextDecoder().decode(await backend.read('k1'))).toBe('hello');
|
||||||
|
expect(await backend.read('missing')).toBeNull();
|
||||||
|
|
||||||
|
await backend.delete('k1');
|
||||||
|
expect(await backend.exists('k1')).toBe(false);
|
||||||
|
await backend.close();
|
||||||
|
expect(backend.isOpen()).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('append 追加语义:字节顺序与位置正确', async () => {
|
||||||
|
const backend = new OPFSBackend();
|
||||||
|
await backend.open('opfs-test-2');
|
||||||
|
await backend.append('wal', enc('AAA'));
|
||||||
|
await backend.append('wal', enc('BBB'));
|
||||||
|
await backend.append('wal', enc('CCC'));
|
||||||
|
const all = await backend.read('wal');
|
||||||
|
expect(new TextDecoder().decode(all)).toBe('AAABBBCCC');
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('write 覆盖语义(COW 原子替换)', async () => {
|
||||||
|
const backend = new OPFSBackend();
|
||||||
|
await backend.open('opfs-test-3');
|
||||||
|
await backend.write('f', enc('OLD-CONTENT'));
|
||||||
|
await backend.write('f', enc('NEW'));
|
||||||
|
expect(new TextDecoder().decode(await backend.read('f'))).toBe('NEW');
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('并发写串行化:写入顺序与调用顺序一致', async () => {
|
||||||
|
const backend = new OPFSBackend();
|
||||||
|
await backend.open('opfs-test-4');
|
||||||
|
// 同时发起多个写(不 await 中间结果)
|
||||||
|
const p1 = backend.append('log', enc('A'));
|
||||||
|
const p2 = backend.append('log', enc('B'));
|
||||||
|
const p3 = backend.append('log', enc('C'));
|
||||||
|
await Promise.all([p1, p2, p3]);
|
||||||
|
expect(new TextDecoder().decode(await backend.read('log'))).toBe('ABC');
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('单次写失败不中断队列链,后续写正常', async () => {
|
||||||
|
const backend = new OPFSBackend();
|
||||||
|
await backend.open('opfs-test-5');
|
||||||
|
|
||||||
|
// 注入失败:让 write 抛错一次
|
||||||
|
let failNext = true;
|
||||||
|
const origWrite = backend.write.bind(backend);
|
||||||
|
const mockDir = (backend as any).dbDir;
|
||||||
|
const origGetFileHandle = mockDir.getFileHandle.bind(mockDir);
|
||||||
|
mockDir.getFileHandle = async (name: string, opts?: { create?: boolean }) => {
|
||||||
|
if (failNext && name === 'boom') {
|
||||||
|
failNext = false;
|
||||||
|
throw new Error('Injected write failure');
|
||||||
|
}
|
||||||
|
return origGetFileHandle(name, opts);
|
||||||
|
};
|
||||||
|
(backend as any).dbDir = mockDir;
|
||||||
|
|
||||||
|
await expect(origWrite('boom', enc('x'))).rejects.toThrow('Injected write failure');
|
||||||
|
// 队列链恢复:后续写成功
|
||||||
|
await backend.write('ok', enc('fine'));
|
||||||
|
expect(new TextDecoder().decode(await backend.read('ok'))).toBe('fine');
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('close 等待挂起写完成(挂起写不丢失)', async () => {
|
||||||
|
const backend = new OPFSBackend();
|
||||||
|
await backend.open('opfs-test-6');
|
||||||
|
|
||||||
|
// 发起一个写但不 await,立即 close
|
||||||
|
const pending = backend.append('late', enc('LATE-DATA'));
|
||||||
|
await backend.close();
|
||||||
|
await pending; // close 已等队列 → 写已完成
|
||||||
|
expect(pending).resolves.toBeUndefined();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('open 清理崩溃残留临时文件(.crswap/.tmp)', async () => {
|
||||||
|
const backend = new OPFSBackend();
|
||||||
|
await backend.open('opfs-test-7');
|
||||||
|
await backend.write('data', enc('real'));
|
||||||
|
// 手工制造残留
|
||||||
|
const mockDir = (backend as any).dbDir;
|
||||||
|
await mockDir.getFileHandle('data.crswap', { create: true });
|
||||||
|
await mockDir.getFileHandle('data.tmp', { create: true });
|
||||||
|
await backend.close();
|
||||||
|
|
||||||
|
// 重开:open 自动清理
|
||||||
|
const backend2 = new OPFSBackend();
|
||||||
|
await backend2.open('opfs-test-7');
|
||||||
|
const keys = await backend2.listKeys();
|
||||||
|
expect(keys).toEqual(['data']);
|
||||||
|
expect(keys.some((k) => k.endsWith('.crswap') || k.endsWith('.tmp'))).toBe(false);
|
||||||
|
// 正常数据不受影响
|
||||||
|
expect(new TextDecoder().decode(await backend2.read('data'))).toBe('real');
|
||||||
|
await backend2.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('writeMany / deleteMany 语义', async () => {
|
||||||
|
const backend = new OPFSBackend();
|
||||||
|
await backend.open('opfs-test-8');
|
||||||
|
await backend.writeMany({ a: enc('AAA'), b: enc('BBB') });
|
||||||
|
expect(new TextDecoder().decode(await backend.read('a'))).toBe('AAA');
|
||||||
|
expect(new TextDecoder().decode(await backend.read('b'))).toBe('BBB');
|
||||||
|
await backend.deleteMany(['a']);
|
||||||
|
expect(await backend.exists('a')).toBe(false);
|
||||||
|
expect(await backend.exists('b')).toBe(true);
|
||||||
|
await backend.clear();
|
||||||
|
expect(await backend.listKeys()).toEqual([]);
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('close 后所有操作安全返回(dbDir 为 null 不抛错)', async () => {
|
||||||
|
const backend = new OPFSBackend();
|
||||||
|
await backend.open('opfs-test-9');
|
||||||
|
await backend.write('k', enc('v'));
|
||||||
|
await backend.close();
|
||||||
|
|
||||||
|
expect(backend.isOpen()).toBe(false);
|
||||||
|
expect(await backend.read('k')).toBeNull();
|
||||||
|
await backend.write('late', enc('x')); // 静默忽略
|
||||||
|
await backend.append('late2', enc('x'));
|
||||||
|
await backend.writeMany({ a: enc('1') });
|
||||||
|
await backend.delete('k');
|
||||||
|
await backend.deleteMany(['a']);
|
||||||
|
expect(await backend.listKeys()).toEqual([]);
|
||||||
|
expect(await backend.exists('k')).toBe(false);
|
||||||
|
await backend.clear();
|
||||||
|
await backend.cleanupStaleFiles();
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,62 @@
|
|||||||
|
/**
|
||||||
|
* AriaEngine Page — 页面核心逻辑测试
|
||||||
|
*
|
||||||
|
* v0.4.5: 页面层精简后仅保留生产代码实际使用的函数:
|
||||||
|
* - initPageHeader(FileManager 创建空页面)
|
||||||
|
* - createPage(BufferPool 分配新页面)
|
||||||
|
* - PAGE_HEADER_SIZE / PAGE_SIZE 常量
|
||||||
|
* 页面内容为 SSTable 原始字节切片(PageSSTableStore 管理),无行级 slot 结构。
|
||||||
|
*/
|
||||||
|
import { createPage } from '../../src/engine/aria/page/format';
|
||||||
|
import { initPageHeader } from '../../src/engine/aria/page/header';
|
||||||
|
import { PAGE_SIZE, PAGE_HEADER_SIZE, PageType } from '../../src/engine/aria/types';
|
||||||
|
|
||||||
|
describe('AriaEngine — Page 核心', () => {
|
||||||
|
it('initPageHeader 初始化 4KB 页面头部字段', () => {
|
||||||
|
const buf = new ArrayBuffer(PAGE_SIZE);
|
||||||
|
initPageHeader(buf, 42, PageType.DATA);
|
||||||
|
const view = new DataView(buf);
|
||||||
|
expect(view.getUint32(0, false)).toBe(42); // page_id
|
||||||
|
expect(view.getUint8(4)).toBe(PageType.DATA); // type
|
||||||
|
expect(view.getUint16(5, false)).toBe(PAGE_HEADER_SIZE); // free_start
|
||||||
|
expect(view.getUint16(7, false)).toBe(PAGE_SIZE); // free_end
|
||||||
|
expect(view.getUint16(9, false)).toBe(0); // slot_count
|
||||||
|
expect(view.getUint8(15)).toBe(0); // reserved
|
||||||
|
});
|
||||||
|
|
||||||
|
it('initPageHeader 支持不同页面类型', () => {
|
||||||
|
const buf = new ArrayBuffer(PAGE_SIZE);
|
||||||
|
initPageHeader(buf, 1, PageType.META);
|
||||||
|
expect(new DataView(buf).getUint8(4)).toBe(PageType.META);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('createPage 创建脏页面(pins=0,初始化头部)', () => {
|
||||||
|
const page = createPage(7, PageType.DATA);
|
||||||
|
expect(page.pageId).toBe(7);
|
||||||
|
expect(page.type).toBe(PageType.DATA);
|
||||||
|
expect(page.data.byteLength).toBe(PAGE_SIZE);
|
||||||
|
expect(page.dirty).toBe(true);
|
||||||
|
expect(page.pins).toBe(0);
|
||||||
|
// 头部已初始化
|
||||||
|
expect(new DataView(page.data).getUint32(0, false)).toBe(7);
|
||||||
|
expect(new DataView(page.data).getUint8(4)).toBe(PageType.DATA);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('页面数据区从头部之后开始(跳过 16 字节头)', () => {
|
||||||
|
const page = createPage(1, PageType.DATA);
|
||||||
|
// 头部之后写入数据,头部字段不受影响
|
||||||
|
const view = new Uint8Array(page.data);
|
||||||
|
view[PAGE_HEADER_SIZE] = 0xAB;
|
||||||
|
view[PAGE_HEADER_SIZE + 1] = 0xCD;
|
||||||
|
expect(view[PAGE_HEADER_SIZE]).toBe(0xAB);
|
||||||
|
expect(view[PAGE_HEADER_SIZE + 1]).toBe(0xCD);
|
||||||
|
// 头部字段保持
|
||||||
|
expect(new DataView(page.data).getUint32(0, false)).toBe(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('createPage 的 dirty 标记在写入后保持(PageSSTableStore 依赖)', () => {
|
||||||
|
const page = createPage(2, PageType.DATA);
|
||||||
|
new Uint8Array(page.data)[100] = 0x01;
|
||||||
|
expect(page.dirty).toBe(true);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,318 @@
|
|||||||
|
/**
|
||||||
|
* AriaEngine — 页面化 SSTable 存储测试
|
||||||
|
*
|
||||||
|
* 覆盖:
|
||||||
|
* 1. PageSSTableStore 单元:save/load 往返(页边界、非整数页)、delete 释放页面
|
||||||
|
* 2. AriaEngine + OPFS(页面模式):写入 → flush → close → reopen 数据完整
|
||||||
|
* 3. 大 SSTable 多页 + pageIds 注入 meta 验证
|
||||||
|
* 4. 页面损坏(CRC 校验失败)→ 打开自愈清理
|
||||||
|
* 5. 页面模式与非页面模式混合(pageIds 兼容:新页面写 + 旧整 value 读)
|
||||||
|
* 6. 页面化 + 加密 + 压缩组合
|
||||||
|
*/
|
||||||
|
import { AriaEngine } from '../../src/engine/aria/index';
|
||||||
|
import { PageSSTableStore } from '../../src/engine/aria/store/page_sstable_store';
|
||||||
|
import { FileManager } from '../../src/engine/aria/store/file_manager';
|
||||||
|
import { BufferPool } from '../../src/engine/aria/buffer/pool';
|
||||||
|
import { MemoryBackend } from '../../src/engine/aria/store/backend';
|
||||||
|
import { createSchema } from '../../src/table/schema';
|
||||||
|
import { PAGE_SIZE } from '../../src/engine/aria/types';
|
||||||
|
|
||||||
|
// ===================================================================
|
||||||
|
// OPFS mock(共享工具)
|
||||||
|
// ===================================================================
|
||||||
|
import { installOPFSMock } from '../helpers/opfs-mock';
|
||||||
|
|
||||||
|
const SCHEMA = () => createSchema('users', {
|
||||||
|
id: { type: 'string', primaryKey: true },
|
||||||
|
name: { type: 'string' },
|
||||||
|
note: { type: 'string' },
|
||||||
|
});
|
||||||
|
|
||||||
|
// ===================================================================
|
||||||
|
// PageSSTableStore 单元
|
||||||
|
// ===================================================================
|
||||||
|
describe('AriaEngine — PageSSTableStore 单元', () => {
|
||||||
|
it('save/load 往返:单页数据', async () => {
|
||||||
|
const backend = new MemoryBackend();
|
||||||
|
await backend.open('pg-unit-1');
|
||||||
|
const fm = new FileManager(backend);
|
||||||
|
await fm.init('pg-unit-1');
|
||||||
|
const bp = new BufferPool(fm, 64);
|
||||||
|
const store = new PageSSTableStore(fm, bp);
|
||||||
|
|
||||||
|
const data = new Uint8Array(new TextEncoder().encode('hello page store'));
|
||||||
|
await store.save(1, data);
|
||||||
|
const pageIds = store.getPageIds(1)!;
|
||||||
|
expect(pageIds.length).toBe(1);
|
||||||
|
|
||||||
|
const back = await store.load(1, pageIds, data.byteLength);
|
||||||
|
expect(back).not.toBeNull();
|
||||||
|
expect(new TextDecoder().decode(back)).toBe('hello page store');
|
||||||
|
|
||||||
|
// 页面已落盘(backend 有 pg_ 键)
|
||||||
|
expect(await backend.exists('pg_1')).toBe(true);
|
||||||
|
|
||||||
|
await store.delete(1, pageIds);
|
||||||
|
expect(await backend.exists('pg_1')).toBe(false);
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('save/load 往返:多页数据(跨 4KB 边界)', async () => {
|
||||||
|
const backend = new MemoryBackend();
|
||||||
|
await backend.open('pg-unit-2');
|
||||||
|
const fm = new FileManager(backend);
|
||||||
|
await fm.init('pg-unit-2');
|
||||||
|
const bp = new BufferPool(fm, 64);
|
||||||
|
const store = new PageSSTableStore(fm, bp);
|
||||||
|
|
||||||
|
// 3 页:4KB * 2 + 1KB
|
||||||
|
const size = PAGE_SIZE * 2 + 1024;
|
||||||
|
const data = new Uint8Array(size);
|
||||||
|
let x = 42;
|
||||||
|
for (let i = 0; i < size; i++) {
|
||||||
|
x = (x * 31 + 7) & 0xffff;
|
||||||
|
data[i] = x & 0xff;
|
||||||
|
}
|
||||||
|
await store.save(7, data);
|
||||||
|
const pageIds = store.getPageIds(7)!;
|
||||||
|
expect(pageIds.length).toBe(3);
|
||||||
|
|
||||||
|
const back = await store.load(7, pageIds, size);
|
||||||
|
expect(back).not.toBeNull();
|
||||||
|
expect(back!.byteLength).toBe(size);
|
||||||
|
expect(back).toEqual(data);
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('load 使用 totalSize 截断最后一页的 0 填充', async () => {
|
||||||
|
const backend = new MemoryBackend();
|
||||||
|
await backend.open('pg-unit-3');
|
||||||
|
const fm = new FileManager(backend);
|
||||||
|
await fm.init('pg-unit-3');
|
||||||
|
const bp = new BufferPool(fm, 64);
|
||||||
|
const store = new PageSSTableStore(fm, bp);
|
||||||
|
|
||||||
|
// 非整数页:4096 + 100 字节
|
||||||
|
const real = new Uint8Array(PAGE_SIZE + 100);
|
||||||
|
for (let i = 0; i < real.length; i++) real[i] = i & 0xff;
|
||||||
|
await store.save(9, real);
|
||||||
|
const pageIds = store.getPageIds(9)!;
|
||||||
|
expect(pageIds.length).toBe(2);
|
||||||
|
|
||||||
|
const back = await store.load(9, pageIds, real.length);
|
||||||
|
expect(back!.byteLength).toBe(real.length);
|
||||||
|
expect(back).toEqual(real);
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('页面缺失(磁盘文件丢失 + 缓存清空)→ load 返回 null', async () => {
|
||||||
|
const backend = new MemoryBackend();
|
||||||
|
await backend.open('pg-unit-4');
|
||||||
|
const fm = new FileManager(backend);
|
||||||
|
await fm.init('pg-unit-4');
|
||||||
|
const bp = new BufferPool(fm, 64);
|
||||||
|
const store = new PageSSTableStore(fm, bp);
|
||||||
|
|
||||||
|
await store.save(5, new Uint8Array(100));
|
||||||
|
const pageIds = store.getPageIds(5)!;
|
||||||
|
// 模拟崩溃重开:磁盘文件丢失 + 页面缓存清空
|
||||||
|
await backend.delete('pg_' + pageIds[0]);
|
||||||
|
bp.removePage(pageIds[0]);
|
||||||
|
|
||||||
|
const back = await store.load(5, pageIds, 100);
|
||||||
|
expect(back).toBeNull();
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ===================================================================
|
||||||
|
// AriaEngine + OPFS 页面模式集成
|
||||||
|
// ===================================================================
|
||||||
|
describe('AriaEngine — 页面化 SSTable 集成(OPFS)', () => {
|
||||||
|
it('写入 → flush → close → reopen 数据完整(页面模式默认启用)', async () => {
|
||||||
|
const files = installOPFSMock(new Map());
|
||||||
|
const engine = new AriaEngine({
|
||||||
|
storageBackend: 'opfs',
|
||||||
|
memtableSizeThreshold: 64 * 1024 * 1024,
|
||||||
|
checkpointInterval: 100000,
|
||||||
|
});
|
||||||
|
await engine.open('page-db-1', 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
|
||||||
|
// 多行 + 大 note → 超过 4KB → 多页
|
||||||
|
const rows = [] as Record<string, unknown>[];
|
||||||
|
for (let i = 0; i < 30; i++) {
|
||||||
|
rows.push({ id: `u-${i}`, name: `User${i}`, note: 'page-store-note-'.repeat(40) });
|
||||||
|
}
|
||||||
|
await engine.insert('users', rows);
|
||||||
|
await (engine as any).lsm.flush();
|
||||||
|
|
||||||
|
// 页面化落盘:backend 有 pg_ 键,meta 含 pageIds
|
||||||
|
const backend = (engine as any).backend;
|
||||||
|
const keys = await backend.listKeys();
|
||||||
|
const pgKeys = keys.filter((k: string) => k.startsWith('pg_'));
|
||||||
|
const sstKeys = keys.filter((k: string) => k.startsWith('sst_'));
|
||||||
|
expect(pgKeys.length).toBeGreaterThan(0);
|
||||||
|
expect(sstKeys.length).toBe(0); // 页面模式下不写整 value SSTable
|
||||||
|
|
||||||
|
const metas = await (engine as any).lsm.sstableStore.listMeta();
|
||||||
|
expect(metas.length).toBeGreaterThan(0);
|
||||||
|
expect(metas[0].pageIds).toBeDefined();
|
||||||
|
expect(metas[0].pageIds.length).toBeGreaterThan(0);
|
||||||
|
|
||||||
|
await engine.close();
|
||||||
|
|
||||||
|
// reopen:数据完整
|
||||||
|
const engine2 = new AriaEngine({ storageBackend: 'opfs', checkpointInterval: 100000 });
|
||||||
|
await engine2.open('page-db-1', 1);
|
||||||
|
const found = await engine2.find('users', { table: 'users' });
|
||||||
|
expect(found).toHaveLength(30);
|
||||||
|
expect(found[0].note).toBe('page-store-note-'.repeat(40));
|
||||||
|
expect(await engine2.count('users')).toBe(30);
|
||||||
|
await engine2.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('多级 compaction 后页面化数据仍完整', async () => {
|
||||||
|
const files = installOPFSMock(new Map());
|
||||||
|
const engine = new AriaEngine({
|
||||||
|
storageBackend: 'opfs',
|
||||||
|
memtableSizeThreshold: 16 * 1024,
|
||||||
|
checkpointInterval: 100000,
|
||||||
|
});
|
||||||
|
await engine.open('page-db-2', 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
|
||||||
|
// 大量小批量写入 → 多次 flush → 触发 compaction
|
||||||
|
for (let batch = 0; batch < 8; batch++) {
|
||||||
|
const rows = [] as Record<string, unknown>[];
|
||||||
|
for (let i = 0; i < 20; i++) {
|
||||||
|
rows.push({ id: `b${batch}-u${i}`, name: `U${i}`, note: 'x'.repeat(200) });
|
||||||
|
}
|
||||||
|
await engine.insert('users', rows);
|
||||||
|
await (engine as any).lsm.flush();
|
||||||
|
}
|
||||||
|
// 强制压缩
|
||||||
|
await (engine as any).lsm.compactLevel(0);
|
||||||
|
|
||||||
|
await engine.close();
|
||||||
|
const engine2 = new AriaEngine({ storageBackend: 'opfs', checkpointInterval: 100000 });
|
||||||
|
await engine2.open('page-db-2', 1);
|
||||||
|
expect(await engine2.count('users')).toBe(160);
|
||||||
|
const one = await engine2.find('users', { table: 'users', where: { id: 'b3-u10' } });
|
||||||
|
expect(one).toHaveLength(1);
|
||||||
|
await engine2.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('页面损坏(篡改 pg_ 文件)→ 打开自愈清理,其余数据可读', async () => {
|
||||||
|
const files = installOPFSMock(new Map());
|
||||||
|
const engine = new AriaEngine({
|
||||||
|
storageBackend: 'opfs',
|
||||||
|
memtableSizeThreshold: 64 * 1024 * 1024,
|
||||||
|
checkpointInterval: 100000,
|
||||||
|
});
|
||||||
|
await engine.open('page-db-3', 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
for (let batch = 0; batch < 2; batch++) {
|
||||||
|
const rows = [] as Record<string, unknown>[];
|
||||||
|
for (let i = 0; i < 20; i++) {
|
||||||
|
rows.push({ id: `b${batch}-u${i}`, name: `U${i}`, note: 'n'.repeat(100) });
|
||||||
|
}
|
||||||
|
await engine.insert('users', rows);
|
||||||
|
await (engine as any).lsm.flush();
|
||||||
|
}
|
||||||
|
await engine.close();
|
||||||
|
|
||||||
|
// 篡改第一个 SSTable 的第一页
|
||||||
|
const engine2 = new AriaEngine({ storageBackend: 'opfs', checkpointInterval: 100000 });
|
||||||
|
await engine2.open('page-db-3', 1);
|
||||||
|
const metas = await (engine2 as any).lsm.sstableStore.listMeta();
|
||||||
|
expect(metas.length).toBeGreaterThanOrEqual(2);
|
||||||
|
const victim = metas[0];
|
||||||
|
const backend = (engine2 as any).backend;
|
||||||
|
const pageRaw = await backend.read(`pg_${victim.pageIds[0]}`);
|
||||||
|
const corrupted = new Uint8Array(pageRaw);
|
||||||
|
corrupted[100] ^= 0xff;
|
||||||
|
await backend.write(`pg_${victim.pageIds[0]}`, corrupted.buffer as ArrayBuffer);
|
||||||
|
await engine2.close();
|
||||||
|
|
||||||
|
// 重开:损坏页面的 SSTable 被清理,其余数据可读
|
||||||
|
const engine3 = new AriaEngine({ storageBackend: 'opfs', checkpointInterval: 100000 });
|
||||||
|
await engine3.open('page-db-3', 1);
|
||||||
|
const rows3 = await engine3.find('users', { table: 'users' });
|
||||||
|
expect(rows3.length).toBeGreaterThan(0);
|
||||||
|
expect(rows3.length).toBeLessThan(40);
|
||||||
|
await engine3.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('页面模式与非页面模式混合兼容(pageIds 缺失 → 整 value 读取)', async () => {
|
||||||
|
const { files } = installOPFSMock(new Map());
|
||||||
|
// 阶段 1:非页面模式写入(pageStorage: false → 整 value SSTable)
|
||||||
|
const engine = new AriaEngine({
|
||||||
|
storageBackend: 'opfs',
|
||||||
|
pageStorage: false,
|
||||||
|
memtableSizeThreshold: 64 * 1024 * 1024,
|
||||||
|
checkpointInterval: 100000,
|
||||||
|
});
|
||||||
|
await engine.open('page-db-4', 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
await engine.insert('users', [{ id: 'legacy-1', name: 'Old', note: 'whole-value' }]);
|
||||||
|
await (engine as any).lsm.flush();
|
||||||
|
await engine.close();
|
||||||
|
|
||||||
|
const backendKeys1 = Array.from(files.keys());
|
||||||
|
expect(backendKeys1.some((k) => k.startsWith('sst_'))).toBe(true);
|
||||||
|
|
||||||
|
// 阶段 2:页面模式打开(默认 opfs → 启用),读旧数据 + 写新数据
|
||||||
|
const engine2 = new AriaEngine({ storageBackend: 'opfs', checkpointInterval: 100000 });
|
||||||
|
await engine2.open('page-db-4', 1);
|
||||||
|
expect(await engine2.count('users')).toBe(1);
|
||||||
|
const legacy = await engine2.find('users', { table: 'users', where: { id: 'legacy-1' } });
|
||||||
|
expect(legacy[0].note).toBe('whole-value');
|
||||||
|
// 新写入 → 页面化
|
||||||
|
await engine2.insert('users', [{ id: 'new-1', name: 'New', note: 'paged' }]);
|
||||||
|
await (engine2 as any).lsm.flush();
|
||||||
|
await engine2.close();
|
||||||
|
|
||||||
|
// 阶段 3:混合库重开,新旧数据都在
|
||||||
|
const engine3 = new AriaEngine({ storageBackend: 'opfs', checkpointInterval: 100000 });
|
||||||
|
await engine3.open('page-db-4', 1);
|
||||||
|
expect(await engine3.count('users')).toBe(2);
|
||||||
|
const both = await engine3.find('users', { table: 'users' });
|
||||||
|
const byId = Object.fromEntries(both.map((r) => [r.id, r]));
|
||||||
|
expect(byId['legacy-1'].note).toBe('whole-value');
|
||||||
|
expect(byId['new-1'].note).toBe('paged');
|
||||||
|
await engine3.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('页面化 + 加密 + 压缩组合:往返完整', async () => {
|
||||||
|
const files = installOPFSMock(new Map());
|
||||||
|
const engine = new AriaEngine({
|
||||||
|
storageBackend: 'opfs',
|
||||||
|
compression: true,
|
||||||
|
encryption: { password: 'page-pass' },
|
||||||
|
memtableSizeThreshold: 64 * 1024 * 1024,
|
||||||
|
checkpointInterval: 100000,
|
||||||
|
});
|
||||||
|
await engine.open('page-db-5', 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
const rows = [] as Record<string, unknown>[];
|
||||||
|
for (let i = 0; i < 20; i++) {
|
||||||
|
rows.push({ id: `e-${i}`, name: `E${i}`, note: '加密的页面化内容'.repeat(30) });
|
||||||
|
}
|
||||||
|
await engine.insert('users', rows);
|
||||||
|
await (engine as any).lsm.flush();
|
||||||
|
await engine.close();
|
||||||
|
|
||||||
|
const engine2 = new AriaEngine({
|
||||||
|
storageBackend: 'opfs',
|
||||||
|
compression: true,
|
||||||
|
encryption: { password: 'page-pass' },
|
||||||
|
checkpointInterval: 100000,
|
||||||
|
});
|
||||||
|
await engine2.open('page-db-5', 1);
|
||||||
|
expect(await engine2.count('users')).toBe(20);
|
||||||
|
const one = await engine2.find('users', { table: 'users', where: { id: 'e-7' } });
|
||||||
|
expect(one[0].note).toBe('加密的页面化内容'.repeat(30));
|
||||||
|
await engine2.close();
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -1,337 +0,0 @@
|
|||||||
/**
|
|
||||||
* AriaEngine Page 格式单元测试
|
|
||||||
* 覆盖: PageHeader / Slot / Tuple 编解码 + PageFormat 整合
|
|
||||||
*/
|
|
||||||
import {
|
|
||||||
PAGE_SIZE, PageType, PAGE_HEADER_SIZE, SLOT_ENTRY_SIZE,
|
|
||||||
} from '../../src/engine/aria/types';
|
|
||||||
import {
|
|
||||||
encodePageHeader, decodePageHeader, initPageHeader, getPageType,
|
|
||||||
getSlotCount, getFreeStart, setFreeStart, setFreeEnd,
|
|
||||||
} from '../../src/engine/aria/page/header';
|
|
||||||
import {
|
|
||||||
getSlotEntry, setSlotEntry, getSlotDirectorySize,
|
|
||||||
getFreeSpace, hasEnoughSpace, allocateSlot, readSlotData, freeSlot,
|
|
||||||
} from '../../src/engine/aria/page/slot';
|
|
||||||
import {
|
|
||||||
encodeTuple, decodeTuple, getColumnEncodingMap,
|
|
||||||
} from '../../src/engine/aria/page/tuple';
|
|
||||||
import { ColumnEncoding } from '../../src/engine/aria/types';
|
|
||||||
import {
|
|
||||||
createPage, pageFromBuffer, pageInsertRow, pageReadRow,
|
|
||||||
pageDeleteRow, pageUpdateRow, computeChecksum, verifyChecksum, updateChecksum,
|
|
||||||
} from '../../src/engine/aria/page/format';
|
|
||||||
|
|
||||||
// ===================================================================
|
|
||||||
// PageHeader
|
|
||||||
// ===================================================================
|
|
||||||
describe('AriaEngine Page — Header', () => {
|
|
||||||
let buf: ArrayBuffer;
|
|
||||||
|
|
||||||
beforeEach(() => {
|
|
||||||
buf = new ArrayBuffer(PAGE_SIZE);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('initPageHeader 初始化头部字段', () => {
|
|
||||||
initPageHeader(buf, 42, PageType.DATA);
|
|
||||||
const h = decodePageHeader(buf);
|
|
||||||
expect(h.pageId).toBe(42);
|
|
||||||
expect(h.type).toBe(PageType.DATA);
|
|
||||||
expect(h.slotCount).toBe(0);
|
|
||||||
expect(h.freeStart).toBe(PAGE_HEADER_SIZE);
|
|
||||||
expect(h.freeEnd).toBe(PAGE_SIZE);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('initPageHeader — INDEX 类型页面', () => {
|
|
||||||
initPageHeader(buf, 99, PageType.INDEX);
|
|
||||||
expect(getPageType(buf)).toBe(PageType.INDEX);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('encodePageHeader + decodePageHeader 往返一致', () => {
|
|
||||||
const header = { pageId: 7, type: PageType.META, freeStart: 32, freeEnd: 4000, slotCount: 5, checksum: 0xdeadbeef };
|
|
||||||
encodePageHeader(header, buf);
|
|
||||||
const decoded = decodePageHeader(buf);
|
|
||||||
expect(decoded.pageId).toBe(7);
|
|
||||||
expect(decoded.type).toBe(PageType.META);
|
|
||||||
expect(decoded.freeStart).toBe(32);
|
|
||||||
expect(decoded.freeEnd).toBe(4000);
|
|
||||||
expect(decoded.slotCount).toBe(5);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('不同 pageId 正确编解码', () => {
|
|
||||||
for (const id of [0, 1, 255, 65535, 0xffffffff]) {
|
|
||||||
initPageHeader(buf, id, PageType.DATA);
|
|
||||||
expect(decodePageHeader(buf).pageId).toBe(id >>> 0);
|
|
||||||
}
|
|
||||||
});
|
|
||||||
|
|
||||||
it('setFreeStart / setFreeEnd 修改字段', () => {
|
|
||||||
initPageHeader(buf, 1, PageType.DATA);
|
|
||||||
setFreeStart(buf, 100);
|
|
||||||
setFreeEnd(buf, 3000);
|
|
||||||
expect(getFreeStart(buf)).toBe(100);
|
|
||||||
expect(decodePageHeader(buf).freeEnd).toBe(3000);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
// ===================================================================
|
|
||||||
// Slot Directory
|
|
||||||
// ===================================================================
|
|
||||||
describe('AriaEngine Page — Slot', () => {
|
|
||||||
let buf: ArrayBuffer;
|
|
||||||
|
|
||||||
beforeEach(() => {
|
|
||||||
buf = new ArrayBuffer(PAGE_SIZE);
|
|
||||||
initPageHeader(buf, 1, PageType.DATA);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('getSlotEntry — 空页面 slotCount 为 0', () => {
|
|
||||||
expect(getSlotCount(buf)).toBe(0);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('setSlotEntry + getSlotEntry 往返', () => {
|
|
||||||
// 手动写一个 slot(不通过 allocateSlot)
|
|
||||||
new DataView(buf).setUint16(9, 1, false); // slotCount = 1
|
|
||||||
setSlotEntry(buf, 0, { offset: 1000, length: 50 });
|
|
||||||
const entry = getSlotEntry(buf, 0);
|
|
||||||
expect(entry.offset).toBe(1000);
|
|
||||||
expect(entry.length).toBe(50);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('getSlotDirectorySize 计算正确', () => {
|
|
||||||
expect(getSlotDirectorySize(0)).toBe(0);
|
|
||||||
expect(getSlotDirectorySize(1)).toBe(SLOT_ENTRY_SIZE);
|
|
||||||
expect(getSlotDirectorySize(10)).toBe(10 * SLOT_ENTRY_SIZE);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('getFreeSpace — 空页面有最大空闲空间', () => {
|
|
||||||
const free = getFreeSpace(buf);
|
|
||||||
expect(free).toBe(PAGE_SIZE - PAGE_HEADER_SIZE);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('hasEnoughSpace — 小数据返回 true', () => {
|
|
||||||
expect(hasEnoughSpace(buf, 100)).toBe(true);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('hasEnoughSpace — 超大数据返回 false', () => {
|
|
||||||
expect(hasEnoughSpace(buf, PAGE_SIZE * 2)).toBe(false);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('allocateSlot 分配并写入数据', () => {
|
|
||||||
const data = new Uint8Array([1, 2, 3, 4, 5]);
|
|
||||||
const idx = allocateSlot(buf, data);
|
|
||||||
expect(idx).toBe(0);
|
|
||||||
expect(getSlotCount(buf)).toBe(1);
|
|
||||||
|
|
||||||
const readBack = readSlotData(buf, 0);
|
|
||||||
expect(readBack).not.toBeNull();
|
|
||||||
expect(Array.from(readBack!)).toEqual([1, 2, 3, 4, 5]);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('allocateSlot 多次分配', () => {
|
|
||||||
for (let i = 0; i < 10; i++) {
|
|
||||||
const data = new Uint8Array([i, i + 1]);
|
|
||||||
const idx = allocateSlot(buf, data);
|
|
||||||
expect(idx).toBe(i);
|
|
||||||
}
|
|
||||||
expect(getSlotCount(buf)).toBe(10);
|
|
||||||
|
|
||||||
// 验证每个 slot 数据正确
|
|
||||||
for (let i = 0; i < 10; i++) {
|
|
||||||
const data = readSlotData(buf, i);
|
|
||||||
expect(Array.from(data!)).toEqual([i, i + 1]);
|
|
||||||
}
|
|
||||||
});
|
|
||||||
|
|
||||||
it('freeSlot 标记为删除', () => {
|
|
||||||
const data = new Uint8Array([10, 20, 30]);
|
|
||||||
allocateSlot(buf, data);
|
|
||||||
freeSlot(buf, 0);
|
|
||||||
const entry = getSlotEntry(buf, 0);
|
|
||||||
expect(entry.offset).toBe(0);
|
|
||||||
expect(entry.length).toBe(0);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('readSlotData — 已删除 slot 返回 null', () => {
|
|
||||||
allocateSlot(buf, new Uint8Array([1]));
|
|
||||||
freeSlot(buf, 0);
|
|
||||||
expect(readSlotData(buf, 0)).toBeNull();
|
|
||||||
});
|
|
||||||
|
|
||||||
it('allocateSlot — 空间不足返回 -1', () => {
|
|
||||||
// 填满页面
|
|
||||||
const big = new Uint8Array(PAGE_SIZE - PAGE_HEADER_SIZE - SLOT_ENTRY_SIZE);
|
|
||||||
const idx1 = allocateSlot(buf, big);
|
|
||||||
expect(idx1).toBe(0);
|
|
||||||
const idx2 = allocateSlot(buf, new Uint8Array([1]));
|
|
||||||
expect(idx2).toBe(-1);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
// ===================================================================
|
|
||||||
// Tuple Codec
|
|
||||||
// ===================================================================
|
|
||||||
describe('AriaEngine Page — Tuple', () => {
|
|
||||||
const colOrder = ['id', 'name', 'age', 'active', 'data'];
|
|
||||||
const colTypes: Record<string, string> = {
|
|
||||||
id: 'string', name: 'string', age: 'number', active: 'boolean', data: 'json',
|
|
||||||
};
|
|
||||||
|
|
||||||
it('encodeTuple + decodeTuple 完整往返', () => {
|
|
||||||
const row = { id: '1', name: 'Alice', age: 30, active: true, data: { x: 1 } };
|
|
||||||
const encoded = encodeTuple(row, colOrder, colTypes);
|
|
||||||
expect(encoded.byteLength).toBeGreaterThan(0);
|
|
||||||
|
|
||||||
const decoded = decodeTuple(encoded, colOrder, colTypes);
|
|
||||||
expect(decoded).not.toBeNull();
|
|
||||||
expect(decoded!.id).toBe('1');
|
|
||||||
expect(decoded!.name).toBe('Alice');
|
|
||||||
expect(decoded!.age).toBe(30);
|
|
||||||
expect(decoded!.active).toBe(true);
|
|
||||||
expect(decoded!.data).toEqual({ x: 1 });
|
|
||||||
});
|
|
||||||
|
|
||||||
it('encodeTuple — null 值正确处理', () => {
|
|
||||||
const row = { id: '2', name: null, age: 25, active: null, data: null };
|
|
||||||
const encoded = encodeTuple(row, colOrder, colTypes);
|
|
||||||
const decoded = decodeTuple(encoded, colOrder, colTypes);
|
|
||||||
expect(decoded!.name).toBeNull();
|
|
||||||
expect(decoded!.active).toBeNull();
|
|
||||||
expect(decoded!.data).toBeNull();
|
|
||||||
});
|
|
||||||
|
|
||||||
it('encodeTuple — undefined 值按 null 处理', () => {
|
|
||||||
const row = { id: '3', age: 30 } as any;
|
|
||||||
const encoded = encodeTuple(row, colOrder, colTypes);
|
|
||||||
const decoded = decodeTuple(encoded, colOrder, colTypes);
|
|
||||||
expect(decoded!.id).toBe('3');
|
|
||||||
expect(decoded!.name).toBeNull();
|
|
||||||
});
|
|
||||||
|
|
||||||
it('encodeTuple — date 类型', () => {
|
|
||||||
const order = ['ts'];
|
|
||||||
const types = { ts: 'date' };
|
|
||||||
const row = { ts: '2024-01-15T00:00:00.000Z' };
|
|
||||||
const encoded = encodeTuple(row, order, types);
|
|
||||||
const decoded = decodeTuple(encoded, order, types);
|
|
||||||
expect(decoded!.ts).toBe('2024-01-15T00:00:00.000Z');
|
|
||||||
});
|
|
||||||
|
|
||||||
it('encodeTuple — boolean false', () => {
|
|
||||||
const order = ['flag'];
|
|
||||||
const types = { flag: 'boolean' };
|
|
||||||
const encoded = encodeTuple({ flag: false }, order, types);
|
|
||||||
const decoded = decodeTuple(encoded, order, types);
|
|
||||||
expect(decoded!.flag).toBe(false);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('encodeTuple — 负数', () => {
|
|
||||||
const order = ['val'];
|
|
||||||
const types = { val: 'number' };
|
|
||||||
const encoded = encodeTuple({ val: -42.5 }, order, types);
|
|
||||||
const decoded = decodeTuple(encoded, order, types);
|
|
||||||
expect(decoded!.val).toBe(-42.5);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('encodeTuple — 空字符串', () => {
|
|
||||||
const order = ['s'];
|
|
||||||
const types = { s: 'string' };
|
|
||||||
const encoded = encodeTuple({ s: '' }, order, types);
|
|
||||||
const decoded = decodeTuple(encoded, order, types);
|
|
||||||
expect(decoded!.s).toBe('');
|
|
||||||
});
|
|
||||||
|
|
||||||
it('encodeTuple — 长字符串', () => {
|
|
||||||
const order = ['s'];
|
|
||||||
const types = { s: 'string' };
|
|
||||||
const long = 'x'.repeat(10000);
|
|
||||||
const encoded = encodeTuple({ s: long }, order, types);
|
|
||||||
const decoded = decodeTuple(encoded, order, types);
|
|
||||||
expect(decoded!.s).toBe(long);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('getColumnEncodingMap 返回正确映射', () => {
|
|
||||||
const map = getColumnEncodingMap(colOrder, colTypes);
|
|
||||||
expect(map.get('id')).toBe(ColumnEncoding.STRING);
|
|
||||||
expect(map.get('age')).toBe(ColumnEncoding.NUMBER);
|
|
||||||
expect(map.get('active')).toBe(ColumnEncoding.BOOLEAN);
|
|
||||||
expect(map.get('data')).toBe(ColumnEncoding.JSON);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('decodeTuple — 损坏数据返回 null', () => {
|
|
||||||
const broken = new Uint8Array([0xff, 0xff, 0xff]);
|
|
||||||
expect(decodeTuple(broken, colOrder, colTypes)).toBeNull();
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
// ===================================================================
|
|
||||||
// Page Format 整合
|
|
||||||
// ===================================================================
|
|
||||||
describe('AriaEngine Page — Format', () => {
|
|
||||||
const colOrder = ['id', 'name'];
|
|
||||||
const colTypes: Record<string, string> = { id: 'string', name: 'string' };
|
|
||||||
|
|
||||||
it('createPage 创建合法页面', () => {
|
|
||||||
const page = createPage(100, PageType.DATA);
|
|
||||||
expect(page.pageId).toBe(100);
|
|
||||||
expect(page.type).toBe(PageType.DATA);
|
|
||||||
expect(page.pins).toBe(0);
|
|
||||||
expect(page.dirty).toBe(true);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('pageInsertRow + pageReadRow 往返', () => {
|
|
||||||
const page = createPage(1, PageType.DATA);
|
|
||||||
const row = { id: 'u1', name: 'Test' };
|
|
||||||
const idx = pageInsertRow(page, row, colOrder, colTypes);
|
|
||||||
expect(idx).toBe(0);
|
|
||||||
|
|
||||||
const read = pageReadRow(page, 0, colOrder, colTypes);
|
|
||||||
expect(read).not.toBeNull();
|
|
||||||
expect(read!.id).toBe('u1');
|
|
||||||
expect(read!.name).toBe('Test');
|
|
||||||
});
|
|
||||||
|
|
||||||
it('pageInsertRow — 多次插入', () => {
|
|
||||||
const page = createPage(1, PageType.DATA);
|
|
||||||
for (let i = 0; i < 50; i++) {
|
|
||||||
const idx = pageInsertRow(page, { id: `${i}`, name: `User${i}` }, colOrder, colTypes);
|
|
||||||
expect(idx).toBe(i);
|
|
||||||
}
|
|
||||||
for (let i = 0; i < 50; i++) {
|
|
||||||
const row = pageReadRow(page, i, colOrder, colTypes);
|
|
||||||
expect(row!.id).toBe(`${i}`);
|
|
||||||
}
|
|
||||||
});
|
|
||||||
|
|
||||||
it('pageDeleteRow 标记删除', () => {
|
|
||||||
const page = createPage(1, PageType.DATA);
|
|
||||||
pageInsertRow(page, { id: '1', name: 'A' }, colOrder, colTypes);
|
|
||||||
pageInsertRow(page, { id: '2', name: 'B' }, colOrder, colTypes);
|
|
||||||
pageDeleteRow(page, 0);
|
|
||||||
expect(page.dirty).toBe(true);
|
|
||||||
// 已删除的行读取失败
|
|
||||||
const row = pageReadRow(page, 0, colOrder, colTypes);
|
|
||||||
expect(row).toBeNull();
|
|
||||||
// 未删除的行仍然可读
|
|
||||||
const row2 = pageReadRow(page, 1, colOrder, colTypes);
|
|
||||||
expect(row2!.id).toBe('2');
|
|
||||||
});
|
|
||||||
|
|
||||||
it('pageFromBuffer 从 ArrayBuffer 恢复', () => {
|
|
||||||
const page = createPage(5, PageType.INDEX);
|
|
||||||
const restored = pageFromBuffer(5, page.data);
|
|
||||||
expect(restored.pageId).toBe(5);
|
|
||||||
expect(restored.type).toBe(PageType.INDEX);
|
|
||||||
expect(restored.dirty).toBe(false);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('computeChecksum + verifyChecksum', () => {
|
|
||||||
const page = createPage(1, PageType.DATA);
|
|
||||||
updateChecksum(page);
|
|
||||||
expect(verifyChecksum(page)).toBe(true);
|
|
||||||
|
|
||||||
// 修改页面 → 校验和失效
|
|
||||||
new Uint8Array(page.data)[100] = 0xff;
|
|
||||||
expect(verifyChecksum(page)).toBe(false);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
@@ -0,0 +1,249 @@
|
|||||||
|
/**
|
||||||
|
* AriaEngine — repair 自愈增强 + 随机操作压力测试
|
||||||
|
*
|
||||||
|
* 覆盖:
|
||||||
|
* 1. repair 清理孤儿页面(meta 未引用的 pg_ 文件)
|
||||||
|
* 2. repair 清理 OPFS 残留临时文件
|
||||||
|
* 3. WAL 空洞 + repair → 截断清空(不再重放错位数据)
|
||||||
|
* 4. 随机操作压力:insert/update/delete + 模拟崩溃(不 checkpoint 断开)→ 重开全量验证
|
||||||
|
* 5. 随机操作 + 页面化 + 模拟崩溃 → 重开验证
|
||||||
|
* 6. repair 幂等
|
||||||
|
*/
|
||||||
|
import { AriaEngine } from '../../src/engine/aria/index';
|
||||||
|
import { createSchema } from '../../src/table/schema';
|
||||||
|
import { installOPFSMock } from '../helpers/opfs-mock';
|
||||||
|
import 'fake-indexeddb/auto';
|
||||||
|
|
||||||
|
let idbCounter = 0;
|
||||||
|
function uniqueDB(): string {
|
||||||
|
return `rph-${Date.now()}-${++idbCounter}-${Math.random().toString(36).slice(2, 8)}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
const SCHEMA = () => createSchema('items', {
|
||||||
|
id: { type: 'string', primaryKey: true },
|
||||||
|
val: { type: 'number' },
|
||||||
|
tag: { type: 'string', index: true },
|
||||||
|
});
|
||||||
|
|
||||||
|
// ===================================================================
|
||||||
|
// repair 增强
|
||||||
|
// ===================================================================
|
||||||
|
describe('AriaEngine — repair 自愈增强', () => {
|
||||||
|
it('清理孤儿页面(meta 未引用的 pg_ 文件)', async () => {
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
const engine = new AriaEngine({
|
||||||
|
storageBackend: 'indexeddb',
|
||||||
|
pageStorage: true, // 强制页面化(IDB 上也启用,便于断言页面文件)
|
||||||
|
memtableSizeThreshold: 64 * 1024 * 1024,
|
||||||
|
checkpointInterval: 100000,
|
||||||
|
});
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
await engine.insert('items', [{ id: 'a', val: 1, tag: 'x' }]);
|
||||||
|
await (engine as any).lsm.flush();
|
||||||
|
|
||||||
|
const backend = (engine as any).backend;
|
||||||
|
const keys = await backend.listKeys();
|
||||||
|
const pgKeys = keys.filter((k: string) => k.startsWith('pg_'));
|
||||||
|
expect(pgKeys.length).toBeGreaterThan(0);
|
||||||
|
|
||||||
|
// 伪造孤儿页面(模拟崩溃中断的删除流程)
|
||||||
|
await backend.write('pg_999999', new Uint8Array(4096).buffer);
|
||||||
|
await backend.write('pg_999998', new Uint8Array(4096).buffer);
|
||||||
|
expect((await backend.listKeys()).filter((k: string) => k.startsWith('pg_99999'))).toHaveLength(2);
|
||||||
|
|
||||||
|
await (engine as any).repair();
|
||||||
|
|
||||||
|
// 孤儿页面被清理,正常页面保留
|
||||||
|
const after = await backend.listKeys();
|
||||||
|
expect(after).not.toContain('pg_999999');
|
||||||
|
expect(after).not.toContain('pg_999998');
|
||||||
|
expect(await engine.count('items')).toBe(1);
|
||||||
|
await engine.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('清理 OPFS 残留临时文件(.crswap/.tmp)', async () => {
|
||||||
|
// 用 OPFS mock 后端验证 cleanupStaleFiles 被调用
|
||||||
|
const { files, dir } = installOPFSMock(new Map());
|
||||||
|
const engine = new AriaEngine({ storageBackend: 'opfs', checkpointInterval: 100000 });
|
||||||
|
await engine.open('repair-opfs-1', 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
await engine.insert('items', [{ id: 'a', val: 1, tag: 'x' }]);
|
||||||
|
|
||||||
|
// 制造残留
|
||||||
|
await dir.getFileHandle('junk.crswap', { create: true });
|
||||||
|
await dir.getFileHandle('junk2.tmp', { create: true });
|
||||||
|
expect(Array.from(files.keys()).some((k) => k.endsWith('.crswap'))).toBe(true);
|
||||||
|
|
||||||
|
await (engine as any).repair();
|
||||||
|
|
||||||
|
const after = Array.from(files.keys());
|
||||||
|
expect(after.some((k) => k.endsWith('.crswap'))).toBe(false);
|
||||||
|
expect(after.some((k) => k.endsWith('.tmp'))).toBe(false);
|
||||||
|
expect(await engine.count('items')).toBe(1);
|
||||||
|
await engine.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('WAL 空洞 + repair → 截断清空(不再重放错位数据)', async () => {
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
const engine = new AriaEngine({ storageBackend: 'indexeddb', checkpointInterval: 100000 });
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
// 数据落盘 + checkpoint 清 WAL
|
||||||
|
await engine.insert('items', [{ id: 'a', val: 1, tag: 'x' }]);
|
||||||
|
await (engine as any).lsm.flush();
|
||||||
|
await (engine as any).wal.checkpoint();
|
||||||
|
// 新数据只进 WAL
|
||||||
|
await engine.insert('items', [{ id: 'b', val: 2, tag: 'y' }]);
|
||||||
|
const backend = (engine as any).backend;
|
||||||
|
const keys = await backend.listKeys();
|
||||||
|
const walSeg = keys.find((k: string) => k.startsWith('__wal_'));
|
||||||
|
// 制造空洞:删掉当前分片(数据仍在内存)
|
||||||
|
await backend.delete(walSeg);
|
||||||
|
|
||||||
|
await (engine as any).repair();
|
||||||
|
|
||||||
|
// repair 后 WAL 已清空,无空洞残留
|
||||||
|
const after = await backend.listKeys();
|
||||||
|
expect(after.some((k: string) => k.startsWith('__wal_'))).toBe(false);
|
||||||
|
// 在线自愈:memtable 中的 'b'(已确认写入)随 flush 落盘 → 数据完整不丢
|
||||||
|
const rows = await engine.find('items', { table: 'items' });
|
||||||
|
expect(rows).toHaveLength(2);
|
||||||
|
expect(rows.some((r) => r.id === 'a')).toBe(true);
|
||||||
|
expect(rows.some((r) => r.id === 'b')).toBe(true);
|
||||||
|
await engine.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('repair 幂等(连续调用无副作用)', async () => {
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
const engine = new AriaEngine({ storageBackend: 'indexeddb', checkpointInterval: 100000 });
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
await engine.insert('items', [{ id: 'a', val: 1, tag: 'x' }]);
|
||||||
|
await (engine as any).repair();
|
||||||
|
await (engine as any).repair();
|
||||||
|
await (engine as any).repair();
|
||||||
|
expect(await engine.count('items')).toBe(1);
|
||||||
|
await engine.close();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ===================================================================
|
||||||
|
// 随机操作压力 + 模拟崩溃
|
||||||
|
// ===================================================================
|
||||||
|
describe('AriaEngine — 随机操作压力 + 模拟崩溃', () => {
|
||||||
|
it('500 随机操作(insert/update/delete)→ 模拟崩溃 → 重开验证全部已确认写入', async () => {
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
const engine = new AriaEngine({
|
||||||
|
storageBackend: 'indexeddb',
|
||||||
|
memtableSizeThreshold: 8 * 1024,
|
||||||
|
checkpointInterval: 50,
|
||||||
|
walSyncMode: 'full',
|
||||||
|
});
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
|
||||||
|
let seed = 12345;
|
||||||
|
const rand = () => { seed = (seed * 1103515245 + 12345) & 0x7fffffff; return seed / 0x7fffffff; };
|
||||||
|
|
||||||
|
// 已确认写入集合(模拟应用层已收到成功返回)
|
||||||
|
const confirmed = new Map<string, { val: number; tag: string }>();
|
||||||
|
for (let i = 0; i < 500; i++) {
|
||||||
|
const r = rand();
|
||||||
|
const id = `k-${Math.floor(rand() * 100)}`;
|
||||||
|
if (r < 0.6) {
|
||||||
|
// insert/upsert(随机 id 可能碰撞 → 已存在则跳过,confirmed 保持首个值)
|
||||||
|
const row = { id, val: Math.floor(rand() * 1000), tag: `t${Math.floor(rand() * 5)}` };
|
||||||
|
try {
|
||||||
|
await engine.insert('items', [row]);
|
||||||
|
confirmed.set(id, row);
|
||||||
|
} catch (e) {
|
||||||
|
if ((e as { code?: string }).code !== 'DUPLICATE_KEY') throw e;
|
||||||
|
}
|
||||||
|
} else if (r < 0.8) {
|
||||||
|
// update
|
||||||
|
const upd = { val: Math.floor(rand() * 1000) };
|
||||||
|
await engine.update('items', { table: 'items', where: { id } }, upd);
|
||||||
|
const cur = confirmed.get(id);
|
||||||
|
if (cur) confirmed.set(id, { ...cur, ...upd });
|
||||||
|
} else {
|
||||||
|
// delete
|
||||||
|
await engine.delete('items', { table: 'items', where: { id } });
|
||||||
|
confirmed.delete(id);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 模拟崩溃:不 close(不 checkpoint),直接断开 backend
|
||||||
|
await (engine as any).backend.close();
|
||||||
|
(engine as any).opened = false;
|
||||||
|
|
||||||
|
// 重开:WAL 重放 + SSTable 加载
|
||||||
|
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', checkpointInterval: 50, walSyncMode: 'full' });
|
||||||
|
await engine2.open(dbName, 1);
|
||||||
|
const rows = await engine2.find('items', { table: 'items' });
|
||||||
|
const byId = new Map(rows.map((r) => [r.id, r]));
|
||||||
|
// 全部已确认写入可见
|
||||||
|
expect(byId.size).toBe(confirmed.size);
|
||||||
|
for (const [id, expected] of confirmed) {
|
||||||
|
expect(byId.has(id)).toBe(true);
|
||||||
|
expect(byId.get(id)!.val).toBe(expected.val);
|
||||||
|
}
|
||||||
|
await engine2.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('随机操作 + 页面化 + 模拟崩溃 → 重开验证', async () => {
|
||||||
|
const files = installOPFSMock(new Map());
|
||||||
|
const dbName = `rand-opfs-${Date.now()}-${Math.random().toString(36).slice(2, 6)}`;
|
||||||
|
const engine = new AriaEngine({
|
||||||
|
storageBackend: 'opfs',
|
||||||
|
memtableSizeThreshold: 8 * 1024,
|
||||||
|
checkpointInterval: 50,
|
||||||
|
walSyncMode: 'full',
|
||||||
|
});
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(SCHEMA());
|
||||||
|
|
||||||
|
let seed = 999;
|
||||||
|
const rand = () => { seed = (seed * 1103515245 + 12345) & 0x7fffffff; return seed / 0x7fffffff; };
|
||||||
|
const confirmed = new Map<string, { val: number; tag: string }>();
|
||||||
|
for (let i = 0; i < 300; i++) {
|
||||||
|
const r = rand();
|
||||||
|
const id = `p-${Math.floor(rand() * 80)}`;
|
||||||
|
if (r < 0.6) {
|
||||||
|
const row = { id, val: Math.floor(rand() * 500), tag: `tag${Math.floor(rand() * 4)}` };
|
||||||
|
try {
|
||||||
|
await engine.insert('items', [row]);
|
||||||
|
confirmed.set(id, row);
|
||||||
|
} catch (e) {
|
||||||
|
if ((e as { code?: string }).code !== 'DUPLICATE_KEY') throw e;
|
||||||
|
}
|
||||||
|
} else if (r < 0.8) {
|
||||||
|
const upd = { val: Math.floor(rand() * 500) };
|
||||||
|
await engine.update('items', { table: 'items', where: { id } }, upd);
|
||||||
|
const cur = confirmed.get(id);
|
||||||
|
if (cur) confirmed.set(id, { ...cur, ...upd });
|
||||||
|
} else {
|
||||||
|
await engine.delete('items', { table: 'items', where: { id } });
|
||||||
|
confirmed.delete(id);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 模拟崩溃
|
||||||
|
await (engine as any).backend.close();
|
||||||
|
(engine as any).opened = false;
|
||||||
|
|
||||||
|
const engine2 = new AriaEngine({ storageBackend: 'opfs', checkpointInterval: 50, walSyncMode: 'full' });
|
||||||
|
await engine2.open(dbName, 1);
|
||||||
|
const rows = await engine2.find('items', { table: 'items' });
|
||||||
|
const byId = new Map(rows.map((r) => [r.id, r]));
|
||||||
|
expect(byId.size).toBe(confirmed.size);
|
||||||
|
for (const [id, expected] of confirmed) {
|
||||||
|
expect(byId.get(id)!.val).toBe(expected.val);
|
||||||
|
}
|
||||||
|
// 索引查询也验证
|
||||||
|
const tagged = await engine2.find('items', { table: 'items', where: { tag: 'tag0' } });
|
||||||
|
const expectedTagged = Array.from(confirmed.values()).filter((v) => v.tag === 'tag0').length;
|
||||||
|
expect(tagged.length).toBe(expectedTagged);
|
||||||
|
await engine2.close();
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -83,16 +83,6 @@ describe('AriaEngine — SSTable Builder + Reader', () => {
|
|||||||
expect(items).toHaveLength(count);
|
expect(items).toHaveLength(count);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('getIndexBlockCount 返回索引块数', () => {
|
|
||||||
const builder = new SSTableBuilder(256); // 小 block size 触发多个 block
|
|
||||||
for (let i = 0; i < 100; i++) {
|
|
||||||
builder.add(`k-${i}`, { data: 'x'.repeat(50) });
|
|
||||||
}
|
|
||||||
const { sstableData } = builder.build();
|
|
||||||
|
|
||||||
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
|
|
||||||
expect(reader.getIndexBlockCount()).toBeGreaterThanOrEqual(1);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('边界 — 空 SSTable 不抛异常', () => {
|
it('边界 — 空 SSTable 不抛异常', () => {
|
||||||
const builder = new SSTableBuilder(4096);
|
const builder = new SSTableBuilder(4096);
|
||||||
@@ -127,3 +117,148 @@ describe('AriaEngine — SSTable Builder + Reader', () => {
|
|||||||
expect(builder.getEntryCount()).toBe(3);
|
expect(builder.getEntryCount()).toBe(3);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// ===================================================================
|
||||||
|
// v0.4.5: 整文件 CRC-32 校验
|
||||||
|
// ===================================================================
|
||||||
|
describe('AriaEngine — SSTable CRC-32 校验', () => {
|
||||||
|
const makeMeta = (data: Uint8Array): SSTableMeta => ({
|
||||||
|
id: 1, level: 0, minKey: '', maxKey: '\uffff',
|
||||||
|
blockCount: 1, totalSize: data.byteLength, bloomData: null,
|
||||||
|
});
|
||||||
|
|
||||||
|
it('新构建的 SSTable 写入真实非零 checksum', () => {
|
||||||
|
const builder = new SSTableBuilder(4096);
|
||||||
|
builder.add('key1', { name: 'Alice' });
|
||||||
|
const { sstableData } = builder.build();
|
||||||
|
|
||||||
|
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
|
||||||
|
// footer 最后 4 字节为真实 checksum(非 0)
|
||||||
|
const stored = new DataView(sstableData.buffer, sstableData.byteOffset + sstableData.byteLength - 4, 4).getUint32(0, false);
|
||||||
|
expect(stored).not.toBe(0);
|
||||||
|
expect(reader.verifyChecksum()).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('正常文件校验通过后可正常读取', () => {
|
||||||
|
const builder = new SSTableBuilder(4096);
|
||||||
|
for (let i = 0; i < 50; i++) builder.add(`k-${i}`, { v: i });
|
||||||
|
const { sstableData } = builder.build();
|
||||||
|
|
||||||
|
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
|
||||||
|
expect(reader.verifyChecksum()).toBe(true);
|
||||||
|
expect(reader.get('k-30')).toEqual({ v: 30 });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('数据块字节篡改 → 校验失败', () => {
|
||||||
|
const builder = new SSTableBuilder(4096);
|
||||||
|
for (let i = 0; i < 50; i++) builder.add(`k-${i}`, { v: i, pad: 'x'.repeat(64) });
|
||||||
|
const { sstableData } = builder.build();
|
||||||
|
|
||||||
|
// 篡改数据区中部一个字节(非 footer)
|
||||||
|
const corrupted = new Uint8Array(sstableData);
|
||||||
|
corrupted[Math.floor(corrupted.length / 2)] ^= 0xff;
|
||||||
|
|
||||||
|
const reader = new SSTableReader(corrupted, makeMeta(corrupted));
|
||||||
|
// 结构仍可解析(长度一致),但 CRC 必须失败
|
||||||
|
const stored = new DataView(corrupted.buffer, corrupted.byteOffset + corrupted.byteLength - 4, 4).getUint32(0, false);
|
||||||
|
expect(stored).not.toBe(0);
|
||||||
|
expect(reader.verifyChecksum()).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('索引块字节篡改 → 校验失败', () => {
|
||||||
|
const builder = new SSTableBuilder(4096);
|
||||||
|
for (let i = 0; i < 20; i++) builder.add(`k-${i}`, { v: i });
|
||||||
|
const { sstableData } = builder.build();
|
||||||
|
|
||||||
|
// 从 footer 反推索引块位置并篡改
|
||||||
|
const footerOffset = sstableData.byteLength - 32;
|
||||||
|
const view = new DataView(sstableData.buffer, sstableData.byteOffset, sstableData.byteLength);
|
||||||
|
const indexOffset = view.getUint32(footerOffset, false);
|
||||||
|
const corrupted = new Uint8Array(sstableData);
|
||||||
|
corrupted[indexOffset + 8] ^= 0x01;
|
||||||
|
|
||||||
|
const reader = new SSTableReader(corrupted, makeMeta(corrupted));
|
||||||
|
expect(reader.verifyChecksum()).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('旧版文件(checksum=0)跳过校验,保持兼容', () => {
|
||||||
|
const builder = new SSTableBuilder(4096);
|
||||||
|
builder.add('legacy', { v: 1 });
|
||||||
|
const { sstableData } = builder.build();
|
||||||
|
|
||||||
|
// 手工把 checksum 字段清零,模拟旧版写入产物
|
||||||
|
const legacy = new Uint8Array(sstableData);
|
||||||
|
new DataView(legacy.buffer).setUint32(legacy.byteLength - 4, 0, false);
|
||||||
|
|
||||||
|
const reader = new SSTableReader(legacy, makeMeta(legacy));
|
||||||
|
const stored = new DataView(legacy.buffer, legacy.byteOffset + legacy.byteLength - 4, 4).getUint32(0, false);
|
||||||
|
expect(stored).toBe(0);
|
||||||
|
expect(reader.verifyChecksum()).toBe(true); // 跳过校验
|
||||||
|
expect(reader.get('legacy')).toEqual({ v: 1 }); // 数据仍可读
|
||||||
|
});
|
||||||
|
|
||||||
|
it('v1 旧格式文件(magic SSTB,checksum=0)校验通过且可读', () => {
|
||||||
|
// 手工构造 v1 格式最小文件:单条记录 + 索引 + footer(magic SSTB, checksum 0)
|
||||||
|
const encoder = new TextEncoder();
|
||||||
|
const keyBytes = encoder.encode('old-key');
|
||||||
|
const valBytes = encoder.encode(JSON.stringify({ v: 42 }));
|
||||||
|
|
||||||
|
// data block: entryCount(u32)=1 + keyLen(u16)+key + valLen(u16)+val
|
||||||
|
const block = new ArrayBuffer(4 + 2 + keyBytes.length + 2 + valBytes.length);
|
||||||
|
const bv = new DataView(block);
|
||||||
|
let off = 0;
|
||||||
|
bv.setUint32(off, 1, false); off += 4;
|
||||||
|
bv.setUint16(off, keyBytes.length, false); off += 2;
|
||||||
|
new Uint8Array(block).set(keyBytes, off); off += keyBytes.length;
|
||||||
|
bv.setUint16(off, valBytes.length, false); off += 2;
|
||||||
|
new Uint8Array(block).set(valBytes, off); off += valBytes.length;
|
||||||
|
|
||||||
|
// index block: entryCount(u32)=1 + keyLen(u16)+key + blockOffset(u32) + blockSize(u32)
|
||||||
|
const blockSize = block.byteLength;
|
||||||
|
const idxKey = encoder.encode('old-key');
|
||||||
|
const idxBlock = new ArrayBuffer(4 + 2 + idxKey.length + 4 + 4);
|
||||||
|
const iv = new DataView(idxBlock);
|
||||||
|
let ioff = 0;
|
||||||
|
iv.setUint32(ioff, 1, false); ioff += 4;
|
||||||
|
iv.setUint16(ioff, idxKey.length, false); ioff += 2;
|
||||||
|
new Uint8Array(idxBlock).set(idxKey, ioff); ioff += idxKey.length;
|
||||||
|
iv.setUint32(ioff, 0, false); ioff += 4; // blockOffset = 0
|
||||||
|
iv.setUint32(ioff, blockSize, false); ioff += 4;
|
||||||
|
|
||||||
|
const footerSize = 32;
|
||||||
|
const total = block.byteLength + idxBlock.byteLength + footerSize;
|
||||||
|
const file = new ArrayBuffer(total);
|
||||||
|
const fv = new DataView(file);
|
||||||
|
new Uint8Array(file).set(new Uint8Array(block), 0);
|
||||||
|
new Uint8Array(file).set(new Uint8Array(idxBlock), block.byteLength);
|
||||||
|
const fo = block.byteLength + idxBlock.byteLength;
|
||||||
|
fv.setUint32(fo, block.byteLength, false); // index_offset
|
||||||
|
fv.setUint32(fo + 4, idxBlock.byteLength, false);
|
||||||
|
fv.setUint32(fo + 8, 0, false); // bloom_offset
|
||||||
|
fv.setUint32(fo + 12, 0, false); // bloom_size
|
||||||
|
fv.setUint32(fo + 16, 0, false); // bloom_hash_count
|
||||||
|
fv.setUint32(fo + 20, 1, false); // entry_count
|
||||||
|
fv.setUint32(fo + 24, 0x53535442, false); // magic "SSTB" (v1)
|
||||||
|
fv.setUint32(fo + 28, 0, false); // checksum = 0(旧版无校验)
|
||||||
|
|
||||||
|
const data = new Uint8Array(file);
|
||||||
|
const reader = new SSTableReader(data, makeMeta(data));
|
||||||
|
const stored = new DataView(data.buffer, data.byteOffset + data.byteLength - 4, 4).getUint32(0, false);
|
||||||
|
expect(stored).toBe(0);
|
||||||
|
expect(reader.verifyChecksum()).toBe(true);
|
||||||
|
expect(reader.get('old-key')).toEqual({ v: 42 });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('footer checksum 字段自身篡改 → 校验失败', () => {
|
||||||
|
const builder = new SSTableBuilder(4096);
|
||||||
|
builder.add('a', { v: 1 });
|
||||||
|
const { sstableData } = builder.build();
|
||||||
|
|
||||||
|
const tampered = new Uint8Array(sstableData);
|
||||||
|
const view = new DataView(tampered.buffer);
|
||||||
|
view.setUint32(tampered.byteLength - 4, view.getUint32(tampered.byteLength - 4, false) ^ 0x000000ff, false);
|
||||||
|
|
||||||
|
const reader = new SSTableReader(tampered, makeMeta(tampered));
|
||||||
|
expect(reader.verifyChecksum()).toBe(false);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|||||||
@@ -0,0 +1,218 @@
|
|||||||
|
/**
|
||||||
|
* AriaEngine WAL — CRC-32 校验升级测试
|
||||||
|
*
|
||||||
|
* 覆盖:
|
||||||
|
* 1. 新算法记录往返(append → recover)
|
||||||
|
* 2. 篡改记录字节 → CRC 不匹配跳过
|
||||||
|
* 3. 旧版弱校验记录兼容(双算法探测)
|
||||||
|
* 4. appendBatch 组提交记录往返
|
||||||
|
* 5. 损坏记录跳过不影响后续完整记录
|
||||||
|
*/
|
||||||
|
import { WAL } from '../../src/engine/aria/wal/log';
|
||||||
|
import { WALRecordType } from '../../src/engine/aria/types';
|
||||||
|
|
||||||
|
// ===================================================================
|
||||||
|
// WAL 内存 Store(模拟 IStorageBackend 单记录语义)
|
||||||
|
// ===================================================================
|
||||||
|
class MemoryWALStore {
|
||||||
|
chunks: Uint8Array[] = [];
|
||||||
|
truncated = false;
|
||||||
|
|
||||||
|
async append(data: Uint8Array): Promise<void> {
|
||||||
|
this.chunks.push(data);
|
||||||
|
}
|
||||||
|
|
||||||
|
async readAll(): Promise<Uint8Array> {
|
||||||
|
const total = this.chunks.reduce((s, c) => s + c.byteLength, 0);
|
||||||
|
const combined = new Uint8Array(total);
|
||||||
|
let off = 0;
|
||||||
|
for (const c of this.chunks) { combined.set(c, off); off += c.byteLength; }
|
||||||
|
return combined;
|
||||||
|
}
|
||||||
|
|
||||||
|
async truncate(): Promise<void> {
|
||||||
|
this.chunks = [];
|
||||||
|
this.truncated = true;
|
||||||
|
}
|
||||||
|
|
||||||
|
async exists(): Promise<boolean> {
|
||||||
|
return this.chunks.length > 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function makeRecord(over: Partial<import('../../src/engine/aria/types').WALRecord> = {}) {
|
||||||
|
return {
|
||||||
|
type: WALRecordType.INSERT,
|
||||||
|
txnId: 0,
|
||||||
|
tableName: 'users',
|
||||||
|
key: 'u-1',
|
||||||
|
data: { id: 'u-1', name: 'Alice' },
|
||||||
|
...over,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('AriaEngine — WAL CRC-32 校验', () => {
|
||||||
|
it('新算法记录 append → recover 往返完整', async () => {
|
||||||
|
const store = new MemoryWALStore();
|
||||||
|
const wal = new WAL(store as never, true, 'full');
|
||||||
|
|
||||||
|
await wal.append(makeRecord({ key: 'u-1' }));
|
||||||
|
await wal.append(makeRecord({ key: 'u-2', data: { id: 'u-2', name: 'Bob' } }));
|
||||||
|
|
||||||
|
const records: import('../../src/engine/aria/types').WALRecord[] = [];
|
||||||
|
await wal.recover((r) => records.push(r));
|
||||||
|
expect(records).toHaveLength(2);
|
||||||
|
expect(records[0].key).toBe('u-1');
|
||||||
|
expect(records[1].data).toEqual({ id: 'u-2', name: 'Bob' });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('appendBatch 组提交记录往返完整', async () => {
|
||||||
|
const store = new MemoryWALStore();
|
||||||
|
const wal = new WAL(store as never, true, 'full');
|
||||||
|
|
||||||
|
await wal.appendBatch([
|
||||||
|
makeRecord({ key: 'a' }),
|
||||||
|
makeRecord({ key: 'b' }),
|
||||||
|
makeRecord({ key: 'c' }),
|
||||||
|
]);
|
||||||
|
|
||||||
|
const records: import('../../src/engine/aria/types').WALRecord[] = [];
|
||||||
|
await wal.recover((r) => records.push(r));
|
||||||
|
expect(records.map((r) => r.key)).toEqual(['a', 'b', 'c']);
|
||||||
|
expect(records.map((r) => r.lsn)).toEqual([1, 2, 3]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('篡改记录数据 → CRC 不匹配跳过', async () => {
|
||||||
|
const store = new MemoryWALStore();
|
||||||
|
const wal = new WAL(store as never, true, 'full');
|
||||||
|
|
||||||
|
await wal.append(makeRecord({ key: 'good-1' }));
|
||||||
|
await wal.append(makeRecord({ key: 'evil' }));
|
||||||
|
await wal.append(makeRecord({ key: 'good-2' }));
|
||||||
|
|
||||||
|
// 篡改第二条记录的 data 字节(LSN=2 的记录)
|
||||||
|
const raw = await store.readAll();
|
||||||
|
const view = new DataView(raw.buffer, raw.byteOffset, raw.byteLength);
|
||||||
|
// 定位第二条记录起始:按记录格式顺序扫描直到 LSN=2
|
||||||
|
// [lsn:4][type:1][txnId:4][tblLen:2][table][keyLen:2][key][jsonLen:4][json][crc:4]
|
||||||
|
let offset = 0;
|
||||||
|
while (offset + 4 <= raw.byteLength && view.getUint32(offset, false) !== 2) {
|
||||||
|
const tblLen = view.getUint16(offset + 9, false);
|
||||||
|
const keyLenOff = offset + 11 + tblLen;
|
||||||
|
const keyLen = view.getUint16(keyLenOff, false);
|
||||||
|
const jsonLenOff = keyLenOff + 2 + keyLen;
|
||||||
|
const jsonLen = view.getUint32(jsonLenOff, false);
|
||||||
|
offset += 4 + 1 + 4 + 2 + tblLen + 2 + keyLen + 4 + jsonLen + 4;
|
||||||
|
}
|
||||||
|
expect(view.getUint32(offset, false)).toBe(2);
|
||||||
|
// 篡改 data 区(json 起始处第 4 字节)
|
||||||
|
const tblLen2 = view.getUint16(offset + 9, false);
|
||||||
|
const jsonStart = offset + 4 + 1 + 4 + 2 + tblLen2 + 2 + view.getUint16(offset + 11 + tblLen2, false) + 4;
|
||||||
|
raw[jsonStart + 4] ^= 0x01;
|
||||||
|
// 写回 store(readAll 返回的是合并副本)
|
||||||
|
store.chunks = [raw];
|
||||||
|
|
||||||
|
// 恢复:坏记录被跳过,前后好记录保留
|
||||||
|
const records: import('../../src/engine/aria/types').WALRecord[] = [];
|
||||||
|
await wal.recover((r) => records.push(r));
|
||||||
|
expect(records.map((r) => r.key)).toEqual(['good-1', 'good-2']);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('旧版弱校验记录兼容(双算法探测)', async () => {
|
||||||
|
const store = new MemoryWALStore();
|
||||||
|
const wal = new WAL(store as never, true, 'full');
|
||||||
|
|
||||||
|
// 手工构造一条"旧算法"记录:与 encodeRecord 布局一致但 CRC 用旧算法计算
|
||||||
|
const record = {
|
||||||
|
lsn: 1,
|
||||||
|
type: WALRecordType.INSERT as number,
|
||||||
|
txnId: 0,
|
||||||
|
tableName: 'legacy',
|
||||||
|
key: 'old',
|
||||||
|
data: { id: 'old', v: 1 },
|
||||||
|
};
|
||||||
|
const encoder = new TextEncoder();
|
||||||
|
const tblB = encoder.encode(record.tableName);
|
||||||
|
const keyB = encoder.encode(record.key);
|
||||||
|
const jsonB = encoder.encode(JSON.stringify(record.data));
|
||||||
|
const size = 4 + 1 + 4 + 2 + tblB.length + 2 + keyB.length + 4 + jsonB.length + 4;
|
||||||
|
const buf = new ArrayBuffer(size);
|
||||||
|
const v = new DataView(buf);
|
||||||
|
let o = 0;
|
||||||
|
v.setUint32(o, record.lsn, false); o += 4;
|
||||||
|
v.setUint8(o, record.type); o += 1;
|
||||||
|
v.setUint32(o, record.txnId, false); o += 4;
|
||||||
|
v.setUint16(o, tblB.length, false); o += 2;
|
||||||
|
new Uint8Array(buf).set(tblB, o); o += tblB.length;
|
||||||
|
v.setUint16(o, keyB.length, false); o += 2;
|
||||||
|
new Uint8Array(buf).set(keyB, o); o += keyB.length;
|
||||||
|
v.setUint32(o, jsonB.length, false); o += 4;
|
||||||
|
new Uint8Array(buf).set(jsonB, o); o += jsonB.length;
|
||||||
|
// 旧算法 CRC
|
||||||
|
let crc = 0;
|
||||||
|
const u8 = new Uint8Array(buf, 0, o);
|
||||||
|
for (let i = 0; i < u8.length; i++) crc = ((crc << 5) - crc + u8[i]) | 0;
|
||||||
|
v.setUint32(o, crc >>> 0, false);
|
||||||
|
|
||||||
|
store.chunks.push(new Uint8Array(buf));
|
||||||
|
|
||||||
|
const records: import('../../src/engine/aria/types').WALRecord[] = [];
|
||||||
|
await wal.recover((r) => records.push(r));
|
||||||
|
expect(records).toHaveLength(1);
|
||||||
|
expect(records[0].tableName).toBe('legacy');
|
||||||
|
expect(records[0].data).toEqual({ id: 'old', v: 1 });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('旧算法记录 + 新算法记录混合 → 全部正确恢复', async () => {
|
||||||
|
const store = new MemoryWALStore();
|
||||||
|
const wal = new WAL(store as never, true, 'full');
|
||||||
|
|
||||||
|
// 先放一条旧算法记录
|
||||||
|
const record = {
|
||||||
|
lsn: 1,
|
||||||
|
type: WALRecordType.DELETE as number,
|
||||||
|
txnId: 0,
|
||||||
|
tableName: 't',
|
||||||
|
key: 'old-key',
|
||||||
|
data: undefined as Record<string, unknown> | undefined,
|
||||||
|
};
|
||||||
|
const encoder = new TextEncoder();
|
||||||
|
const tblB = encoder.encode(record.tableName);
|
||||||
|
const keyB = encoder.encode(record.key);
|
||||||
|
const size = 4 + 1 + 4 + 2 + tblB.length + 2 + keyB.length + 4 + 4;
|
||||||
|
const buf = new ArrayBuffer(size);
|
||||||
|
const v = new DataView(buf);
|
||||||
|
let o = 0;
|
||||||
|
v.setUint32(o, record.lsn, false); o += 4;
|
||||||
|
v.setUint8(o, record.type); o += 1;
|
||||||
|
v.setUint32(o, record.txnId, false); o += 4;
|
||||||
|
v.setUint16(o, tblB.length, false); o += 2;
|
||||||
|
new Uint8Array(buf).set(tblB, o); o += tblB.length;
|
||||||
|
v.setUint16(o, keyB.length, false); o += 2;
|
||||||
|
new Uint8Array(buf).set(keyB, o); o += keyB.length;
|
||||||
|
v.setUint32(o, 0, false); o += 4; // jsonLen=0
|
||||||
|
let crc = 0;
|
||||||
|
const u8 = new Uint8Array(buf, 0, o);
|
||||||
|
for (let i = 0; i < u8.length; i++) crc = ((crc << 5) - crc + u8[i]) | 0;
|
||||||
|
v.setUint32(o, crc >>> 0, false);
|
||||||
|
store.chunks.push(new Uint8Array(buf));
|
||||||
|
|
||||||
|
// 再追加一条新算法记录(LSN=2)
|
||||||
|
await wal.append(makeRecord({ key: 'new-key', lsn: 2 } as never));
|
||||||
|
|
||||||
|
const records: import('../../src/engine/aria/types').WALRecord[] = [];
|
||||||
|
await wal.recover((r) => records.push(r));
|
||||||
|
expect(records.map((r) => r.key)).toEqual(['old-key', 'new-key']);
|
||||||
|
expect(records[0].type).toBe(WALRecordType.DELETE);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('checkpoint 截断后 exists 返回 false', async () => {
|
||||||
|
const store = new MemoryWALStore();
|
||||||
|
const wal = new WAL(store as never, true, 'full');
|
||||||
|
await wal.append(makeRecord({ key: 'x' }));
|
||||||
|
expect(await store.exists()).toBe(true);
|
||||||
|
await wal.checkpoint();
|
||||||
|
expect(store.truncated).toBe(true);
|
||||||
|
expect(await store.exists()).toBe(false);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -77,21 +77,6 @@ describe('AriaEngine — WAL', () => {
|
|||||||
expect(await store.exists()).toBe(false);
|
expect(await store.exists()).toBe(false);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('getLSN 跟踪序列号', () => {
|
|
||||||
const store = new MockWALStore();
|
|
||||||
const wal = new WAL(store, true, 'full');
|
|
||||||
expect(wal.getLSN()).toBe(0);
|
|
||||||
wal.append({ type: WALRecordType.BEGIN, txnId: 10, tableName: '', key: '' });
|
|
||||||
expect(wal.getLSN()).toBe(1);
|
|
||||||
wal.append({ type: WALRecordType.COMMIT, txnId: 10, tableName: '', key: '' });
|
|
||||||
expect(wal.getLSN()).toBe(2);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('isEnabled 反映配置', () => {
|
|
||||||
expect(new WAL(new MockWALStore(), true).isEnabled()).toBe(true);
|
|
||||||
expect(new WAL(new MockWALStore(), false).isEnabled()).toBe(false);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('多种记录类型编解码', async () => {
|
it('多种记录类型编解码', async () => {
|
||||||
const store = new MockWALStore();
|
const store = new MockWALStore();
|
||||||
const wal = new WAL(store, true, 'full');
|
const wal = new WAL(store, true, 'full');
|
||||||
@@ -133,29 +118,26 @@ describe('AriaEngine — CheckpointManager', () => {
|
|||||||
const cm = new CheckpointManager(lsm as any, wal as any, flushable, 100);
|
const cm = new CheckpointManager(lsm as any, wal as any, flushable, 100);
|
||||||
await cm.tick();
|
await cm.tick();
|
||||||
await cm.tick();
|
await cm.tick();
|
||||||
expect(cm.getOpCount()).toBe(2);
|
|
||||||
expect(flushable.flushed).toBe(false);
|
expect(flushable.flushed).toBe(false);
|
||||||
expect(lsm.flushed).toBe(false);
|
expect(lsm.flushed).toBe(false);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('setInterval 修改间隔后 tick 触发 checkpoint', async () => {
|
it('tick 达到间隔触发 checkpoint', async () => {
|
||||||
const lsm = new MockLSM();
|
const lsm = new MockLSM();
|
||||||
const wal = new MockWAL();
|
const wal = new MockWAL();
|
||||||
const cm = new CheckpointManager(lsm as any, wal as any, null, 1000);
|
const cm = new CheckpointManager(lsm as any, wal as any, null, 2);
|
||||||
cm.setInterval(2);
|
|
||||||
await cm.tick();
|
await cm.tick();
|
||||||
|
expect(lsm.flushed).toBe(false);
|
||||||
await cm.tick();
|
await cm.tick();
|
||||||
expect(cm.getOpCount()).toBe(0); // reset after checkpoint
|
|
||||||
expect(lsm.flushed).toBe(true);
|
expect(lsm.flushed).toBe(true);
|
||||||
expect(wal.checkpointed).toBe(true);
|
expect(wal.checkpointed).toBe(true);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('forceCheckpoint 强制触发', async () => {
|
it('checkpoint 直接触发', async () => {
|
||||||
const lsm = new MockLSM();
|
const lsm = new MockLSM();
|
||||||
const wal = new MockWAL();
|
const wal = new MockWAL();
|
||||||
const cm = new CheckpointManager(lsm as any, wal as any, null, 100);
|
const cm = new CheckpointManager(lsm as any, wal as any, null, 100);
|
||||||
await cm.forceCheckpoint();
|
await cm.checkpoint();
|
||||||
expect(cm.getOpCount()).toBe(0);
|
|
||||||
expect(lsm.flushed).toBe(true);
|
expect(lsm.flushed).toBe(true);
|
||||||
expect(wal.checkpointed).toBe(true);
|
expect(wal.checkpointed).toBe(true);
|
||||||
});
|
});
|
||||||
@@ -173,104 +155,54 @@ describe('AriaEngine — MVCC', () => {
|
|||||||
const id1 = mvcc.beginTransaction();
|
const id1 = mvcc.beginTransaction();
|
||||||
const id2 = mvcc.beginTransaction();
|
const id2 = mvcc.beginTransaction();
|
||||||
expect(id1).not.toBe(id2);
|
expect(id1).not.toBe(id2);
|
||||||
expect(mvcc.isActive(id1)).toBe(true);
|
|
||||||
expect(mvcc.isActive(id2)).toBe(true);
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it('commit 后 isActive 返回 false', () => {
|
it('commit 标记版本为已提交并清理事务登记', () => {
|
||||||
const txnId = mvcc.beginTransaction();
|
const txnId = mvcc.beginTransaction();
|
||||||
|
mvcc.writeVersion('users', '1', { name: 'Alice' }, txnId);
|
||||||
mvcc.commitTransaction(txnId);
|
mvcc.commitTransaction(txnId);
|
||||||
expect(mvcc.isActive(txnId)).toBe(false);
|
const versions = (mvcc as unknown as { versionStore: Map<string, { committed: boolean }[]> }).versionStore.get('users.1');
|
||||||
});
|
expect(versions).toHaveLength(1);
|
||||||
|
expect(versions![0].committed).toBe(true);
|
||||||
it('rollback 后 isActive 返回 false', () => {
|
|
||||||
const txnId = mvcc.beginTransaction();
|
|
||||||
mvcc.rollbackTransaction(txnId);
|
|
||||||
expect(mvcc.isActive(txnId)).toBe(false);
|
|
||||||
});
|
|
||||||
|
|
||||||
it('writeVersion + readVersion 往返', () => {
|
|
||||||
const txnId = mvcc.beginTransaction();
|
|
||||||
mvcc.writeVersion('users', '1', { name: 'Alice', age: 30 }, txnId);
|
|
||||||
const val = mvcc.readVersion('users', '1', txnId);
|
|
||||||
expect(val).not.toBeNull();
|
|
||||||
expect(val!.name).toBe('Alice');
|
|
||||||
});
|
|
||||||
|
|
||||||
it('未提交版本对其他事务不可见', () => {
|
|
||||||
const txn1 = mvcc.beginTransaction();
|
|
||||||
mvcc.writeVersion('users', '1', { name: 'Alice' }, txn1);
|
|
||||||
|
|
||||||
const txn2 = mvcc.beginTransaction();
|
|
||||||
const val = mvcc.readVersion('users', '1', txn2);
|
|
||||||
expect(val).toBeNull(); // txn1's write not yet committed
|
|
||||||
});
|
|
||||||
|
|
||||||
it('commit 后新事务可见', () => {
|
|
||||||
const txn1 = mvcc.beginTransaction();
|
|
||||||
mvcc.writeVersion('users', '1', { name: 'Alice' }, txn1);
|
|
||||||
mvcc.commitTransaction(txn1);
|
|
||||||
|
|
||||||
const txn2 = mvcc.beginTransaction();
|
|
||||||
const val = mvcc.readVersion('users', '1', txn2);
|
|
||||||
expect(val).not.toBeNull();
|
|
||||||
expect(val!.name).toBe('Alice');
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it('rollback 移除写入的版本', () => {
|
it('rollback 移除写入的版本', () => {
|
||||||
const txnId = mvcc.beginTransaction();
|
const txnId = mvcc.beginTransaction();
|
||||||
mvcc.writeVersion('users', '1', { name: 'Temp' }, txnId);
|
mvcc.writeVersion('users', '1', { name: 'Temp' }, txnId);
|
||||||
mvcc.rollbackTransaction(txnId);
|
mvcc.rollbackTransaction(txnId);
|
||||||
|
const store = (mvcc as unknown as { versionStore: Map<string, unknown> }).versionStore;
|
||||||
const txn2 = mvcc.beginTransaction();
|
expect(store.has('users.1')).toBe(false);
|
||||||
expect(mvcc.readVersion('users', '1', txn2)).toBeNull();
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it('deleteVersion 创建墓碑', () => {
|
it('writeVersion 记录版本链(undo log 语义)', () => {
|
||||||
const txnId = mvcc.beginTransaction();
|
const txnId = mvcc.beginTransaction();
|
||||||
mvcc.writeVersion('users', '1', { name: 'Alice' }, txnId);
|
mvcc.writeVersion('users', '1', { name: 'Alice', age: 30 }, txnId);
|
||||||
mvcc.commitTransaction(txnId);
|
mvcc.writeVersion('users', '1', { name: 'Alice', age: 31 }, txnId);
|
||||||
// delete
|
const store = (mvcc as unknown as { versionStore: Map<string, { prevVersion: unknown }[]> }).versionStore;
|
||||||
const txn2 = mvcc.beginTransaction();
|
const versions = store.get('users.1')!;
|
||||||
mvcc.deleteVersion('users', '1', txn2);
|
expect(versions).toHaveLength(2);
|
||||||
mvcc.commitTransaction(txn2);
|
expect(versions[1].prevVersion).not.toBeNull(); // 版本链
|
||||||
// 删除后读取
|
|
||||||
const txn3 = mvcc.beginTransaction();
|
|
||||||
const val = mvcc.readVersion('users', '1', txn3);
|
|
||||||
expect(val).not.toBeNull();
|
|
||||||
expect((val! as any).__mvcc_tombstone).toBe(true);
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it('getLatestCommittedVersions 返回最新已提交', () => {
|
it('deleteVersion 写入墓碑版本', () => {
|
||||||
const txnId = mvcc.beginTransaction();
|
const txnId = mvcc.beginTransaction();
|
||||||
mvcc.writeVersion('users', '1', { name: 'Alice' }, txnId);
|
mvcc.deleteVersion('users', '1', txnId);
|
||||||
mvcc.writeVersion('users', '2', { name: 'Bob' }, txnId);
|
|
||||||
mvcc.commitTransaction(txnId);
|
mvcc.commitTransaction(txnId);
|
||||||
|
const store = (mvcc as unknown as { versionStore: Map<string, { data: Record<string, unknown>; committed: boolean }[]> }).versionStore;
|
||||||
const result = mvcc.getLatestCommittedVersions('users');
|
const versions = store.get('users.1')!;
|
||||||
expect(result['1'].name).toBe('Alice');
|
expect(versions[0].committed).toBe(true);
|
||||||
expect(result['2'].name).toBe('Bob');
|
expect((versions[0].data as unknown as { __mvcc_tombstone: boolean }).__mvcc_tombstone).toBe(true);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('clearTable 清理指定表', () => {
|
it('活跃事务可并发登记', () => {
|
||||||
const txnId = mvcc.beginTransaction();
|
const t1 = mvcc.beginTransaction();
|
||||||
mvcc.writeVersion('users', '1', { name: 'A' }, txnId);
|
const t2 = mvcc.beginTransaction();
|
||||||
mvcc.writeVersion('users', '2', { name: 'B' }, txnId);
|
mvcc.writeVersion('users', '1', { v: 1 }, t1);
|
||||||
mvcc.writeVersion('orders', 'o1', { amt: 100 }, txnId);
|
mvcc.writeVersion('users', '2', { v: 2 }, t2);
|
||||||
mvcc.commitTransaction(txnId);
|
// 两个事务的写入各自独立记录
|
||||||
|
const store = (mvcc as unknown as { versionStore: Map<string, unknown[]> }).versionStore;
|
||||||
mvcc.clearTable('users');
|
expect(store.has('users.1')).toBe(true);
|
||||||
const txn2 = mvcc.beginTransaction();
|
expect(store.has('users.2')).toBe(true);
|
||||||
expect(mvcc.readVersion('users', '1', txn2)).toBeNull();
|
|
||||||
expect(mvcc.readVersion('users', '2', txn2)).toBeNull();
|
|
||||||
expect(mvcc.readVersion('orders', 'o1', txn2)).not.toBeNull();
|
|
||||||
});
|
|
||||||
|
|
||||||
it('getActiveTxnCount 返回活跃事务数', () => {
|
|
||||||
expect(mvcc.getActiveTxnCount()).toBe(0);
|
|
||||||
mvcc.beginTransaction();
|
|
||||||
mvcc.beginTransaction();
|
|
||||||
expect(mvcc.getActiveTxnCount()).toBe(2);
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it('gc 清理过旧版本', () => {
|
it('gc 清理过旧版本', () => {
|
||||||
@@ -281,9 +213,13 @@ describe('AriaEngine — MVCC', () => {
|
|||||||
mvcc.commitTransaction(txnId);
|
mvcc.commitTransaction(txnId);
|
||||||
}
|
}
|
||||||
mvcc.gc(100);
|
mvcc.gc(100);
|
||||||
// gc 后应可继续操作
|
// gc 后版本保留最新 100 个
|
||||||
|
const store = (mvcc as unknown as { versionStore: Map<string, unknown[]> }).versionStore;
|
||||||
|
expect(store.get('users.1')!.length).toBe(100);
|
||||||
|
// gc 后引擎仍可继续写入
|
||||||
const txnId = mvcc.beginTransaction();
|
const txnId = mvcc.beginTransaction();
|
||||||
const val = mvcc.readVersion('users', '1', txnId);
|
mvcc.writeVersion('users', '1', { ver: 999 }, txnId);
|
||||||
expect(val).not.toBeNull();
|
mvcc.commitTransaction(txnId);
|
||||||
|
expect(store.get('users.1')!.length).toBe(101);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -0,0 +1,275 @@
|
|||||||
|
/**
|
||||||
|
* AriaEngine — SegmentedWALStore 分片 WAL 存储测试
|
||||||
|
*
|
||||||
|
* 覆盖:
|
||||||
|
* 1. append → readAll 往返(分片切换、字节顺序)
|
||||||
|
* 2. 空洞检测:序号不连续 → 空洞后的分片丢弃
|
||||||
|
* 3. 旧格式 __wal_N + __wal_count 兼容读取
|
||||||
|
* 4. truncate / exists
|
||||||
|
* 5. 后端无 append 接口时回退 read+write(正确性一致)
|
||||||
|
* 6. AriaEngine 集成:高频写入跨重启数据完整(分片 + WAL 重放)
|
||||||
|
*/
|
||||||
|
import { SegmentedWALStore } from '../../src/engine/aria/wal/segmented_store';
|
||||||
|
import { MemoryBackend } from '../../src/engine/aria/store/backend';
|
||||||
|
import { AriaEngine } from '../../src/engine/aria/index';
|
||||||
|
import { createSchema } from '../../src/table/schema';
|
||||||
|
import 'fake-indexeddb/auto';
|
||||||
|
|
||||||
|
let idbCounter = 0;
|
||||||
|
function uniqueDB(): string {
|
||||||
|
return `seg-${Date.now()}-${++idbCounter}-${Math.random().toString(36).slice(2, 8)}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
const enc = (s: string) => new Uint8Array(new TextEncoder().encode(s));
|
||||||
|
|
||||||
|
describe('AriaEngine — SegmentedWALStore 单元', () => {
|
||||||
|
it('append → readAll 往返,多批次字节顺序一致', async () => {
|
||||||
|
const backend = new MemoryBackend();
|
||||||
|
await backend.open('seg-unit-1');
|
||||||
|
const store = new SegmentedWALStore(backend, 1024);
|
||||||
|
|
||||||
|
expect(await store.exists()).toBe(false);
|
||||||
|
await store.append(enc('AAA'));
|
||||||
|
await store.append(enc('BBB'));
|
||||||
|
await store.append(enc('CCC'));
|
||||||
|
expect(await store.exists()).toBe(true);
|
||||||
|
|
||||||
|
const all = await store.readAll();
|
||||||
|
expect(new TextDecoder().decode(all)).toBe('AAABBBCCC');
|
||||||
|
|
||||||
|
// 后端是分片文件(单个 key)
|
||||||
|
const keys = await backend.listKeys();
|
||||||
|
expect(keys).toEqual(['__wal_000000.bin']);
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('超过分片阈值自动切换分片', async () => {
|
||||||
|
const backend = new MemoryBackend();
|
||||||
|
await backend.open('seg-unit-2');
|
||||||
|
const store = new SegmentedWALStore(backend, 16); // 每分片 16 字节
|
||||||
|
|
||||||
|
await store.append(enc('AAAAAAAAAA')); // 10B → 分片 0
|
||||||
|
await store.append(enc('BBBBBBBBBB')); // 10B → 分片 1(0 已 10B+10B > 16)
|
||||||
|
await store.append(enc('CC')); // 2B → 分片 1(10+2 <= 16)
|
||||||
|
|
||||||
|
const all = await store.readAll();
|
||||||
|
expect(new TextDecoder().decode(all)).toBe('AAAAAAAAAABBBBBBBBBBCC');
|
||||||
|
const keys = await backend.listKeys().then((ks) => ks.sort());
|
||||||
|
expect(keys).toEqual(['__wal_000000.bin', '__wal_000001.bin']);
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('空洞检测:分片序号不连续 → 空洞后的分片整体丢弃', async () => {
|
||||||
|
const backend = new MemoryBackend();
|
||||||
|
await backend.open('seg-unit-3');
|
||||||
|
const store = new SegmentedWALStore(backend, 16);
|
||||||
|
|
||||||
|
await store.append(enc('SEG0-SEG0-S')); // 10B → 分片 0
|
||||||
|
await store.append(enc('SEG1-SEG1-S')); // 10B → 分片 1(0 已 10B+10B > 16)
|
||||||
|
await store.append(enc('S2')); // 2B → 分片 1(10+2 <= 16)
|
||||||
|
|
||||||
|
// 模拟 truncate 部分完成:删除分片 1(空洞)
|
||||||
|
await backend.delete('__wal_000001.bin');
|
||||||
|
|
||||||
|
const all = await store.readAll();
|
||||||
|
// 空洞在分片 1 → 1 及之后丢弃,只保留分片 0
|
||||||
|
expect(new TextDecoder().decode(all)).toBe('SEG0-SEG0-S');
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('空洞在开头(分片 0 缺失)→ 全部丢弃(保守截断)', async () => {
|
||||||
|
const backend = new MemoryBackend();
|
||||||
|
await backend.open('seg-unit-4');
|
||||||
|
const store = new SegmentedWALStore(backend, 16);
|
||||||
|
await store.append(enc('X0'));
|
||||||
|
await store.append(enc('X1'));
|
||||||
|
await backend.delete('__wal_000000.bin');
|
||||||
|
|
||||||
|
const all = await store.readAll();
|
||||||
|
expect(all.byteLength).toBe(0);
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('旧格式 __wal_N + __wal_count 兼容读取(迁移前数据)', async () => {
|
||||||
|
const backend = new MemoryBackend();
|
||||||
|
await backend.open('seg-unit-5');
|
||||||
|
// 构造旧格式数据
|
||||||
|
await backend.write('__wal_0', enc('LEGACY0').buffer);
|
||||||
|
await backend.write('__wal_1', enc('LEGACY1').buffer);
|
||||||
|
await backend.write('__wal_count', enc('2').buffer);
|
||||||
|
|
||||||
|
const store = new SegmentedWALStore(backend, 1024);
|
||||||
|
expect(await store.exists()).toBe(true);
|
||||||
|
const all = await store.readAll();
|
||||||
|
expect(new TextDecoder().decode(all)).toBe('LEGACY0LEGACY1');
|
||||||
|
|
||||||
|
// 迁移中追加新格式:新记录写入分片文件(不与旧键冲突)
|
||||||
|
await store.append(enc('NEW'));
|
||||||
|
const keys = await backend.listKeys().then((ks) => ks.sort());
|
||||||
|
expect(keys).toContain('__wal_000000.bin');
|
||||||
|
expect(keys).toContain('__wal_0');
|
||||||
|
|
||||||
|
// truncate 清空新旧全部
|
||||||
|
await store.truncate();
|
||||||
|
expect(await store.exists()).toBe(false);
|
||||||
|
expect((await backend.listKeys()).length).toBe(0);
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('truncate 清空分片并重置状态', async () => {
|
||||||
|
const backend = new MemoryBackend();
|
||||||
|
await backend.open('seg-unit-6');
|
||||||
|
const store = new SegmentedWALStore(backend, 16);
|
||||||
|
await store.append(enc('AAAAAA'));
|
||||||
|
await store.append(enc('BBBBBB'));
|
||||||
|
await store.truncate();
|
||||||
|
expect(await store.exists()).toBe(false);
|
||||||
|
expect((await backend.listKeys()).length).toBe(0);
|
||||||
|
|
||||||
|
// 截断后继续追加从分片 0 重新开始
|
||||||
|
await store.append(enc('CCC'));
|
||||||
|
const all = await store.readAll();
|
||||||
|
expect(new TextDecoder().decode(all)).toBe('CCC');
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('后端无 append 接口 → 回退 read+write 语义一致', async () => {
|
||||||
|
// MemoryBackend 没有 append 接口 → 走回退路径
|
||||||
|
const backend = new MemoryBackend();
|
||||||
|
await backend.open('seg-unit-7');
|
||||||
|
const store = new SegmentedWALStore(backend, 1024);
|
||||||
|
await store.append(enc('PART1'));
|
||||||
|
await store.append(enc('PART2'));
|
||||||
|
const all = await store.readAll();
|
||||||
|
expect(new TextDecoder().decode(all)).toBe('PART1PART2');
|
||||||
|
await backend.close();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ===================================================================
|
||||||
|
// AriaEngine 集成 — 分片 WAL 跨重启
|
||||||
|
// ===================================================================
|
||||||
|
describe('AriaEngine — 分片 WAL 集成', () => {
|
||||||
|
it('高频写入(多条 WAL 记录)→ close → reopen 数据完整', async () => {
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
const engine = new AriaEngine({
|
||||||
|
storageBackend: 'indexeddb',
|
||||||
|
checkpointInterval: 100000,
|
||||||
|
walSyncMode: 'full',
|
||||||
|
});
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(createSchema('logs', {
|
||||||
|
id: { type: 'string', primaryKey: true },
|
||||||
|
msg: { type: 'string' },
|
||||||
|
}));
|
||||||
|
const total = 300;
|
||||||
|
for (let i = 0; i < total; i++) {
|
||||||
|
await engine.insert('logs', [{ id: `log-${i}`, msg: `message ${i}` }]);
|
||||||
|
}
|
||||||
|
// 不 flush 不 checkpoint,全部留在 WAL → 模拟崩溃后重放
|
||||||
|
await (engine as any).backend.close();
|
||||||
|
(engine as any).opened = false;
|
||||||
|
|
||||||
|
const engine2 = new AriaEngine({
|
||||||
|
storageBackend: 'indexeddb',
|
||||||
|
checkpointInterval: 100000,
|
||||||
|
walSyncMode: 'full',
|
||||||
|
});
|
||||||
|
await engine2.open(dbName, 1);
|
||||||
|
const rows = await engine2.find('logs', { table: 'logs' });
|
||||||
|
expect(rows).toHaveLength(total);
|
||||||
|
expect(rows.some((r) => r.id === 'log-299')).toBe(true);
|
||||||
|
await engine2.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('分片结构落盘验证(backend 中是分片文件而非旧单记录键)', async () => {
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
const engine = new AriaEngine({
|
||||||
|
storageBackend: 'indexeddb',
|
||||||
|
checkpointInterval: 100000,
|
||||||
|
walSyncMode: 'full',
|
||||||
|
});
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(createSchema('t', {
|
||||||
|
id: { type: 'string', primaryKey: true },
|
||||||
|
}));
|
||||||
|
await engine.insert('t', [{ id: '1' }, { id: '2' }, { id: '3' }]);
|
||||||
|
|
||||||
|
const backend = (engine as any).backend;
|
||||||
|
const keys = await backend.listKeys();
|
||||||
|
const walKeys = keys.filter((k: string) => k.startsWith('__wal_'));
|
||||||
|
// 新格式:__wal_000000.bin;无旧格式单记录键与 count 键
|
||||||
|
expect(walKeys).toContain('__wal_000000.bin');
|
||||||
|
expect(walKeys.some((k: string) => /^__wal_\d+$/.test(k) && !k.endsWith('.bin'))).toBe(false);
|
||||||
|
expect(walKeys).not.toContain('__wal_count');
|
||||||
|
await engine.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('WAL 分片部分残留(空洞)→ 重开恢复已落盘数据,不丢已 checkpoint 数据', async () => {
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
const engine = new AriaEngine({
|
||||||
|
storageBackend: 'indexeddb',
|
||||||
|
checkpointInterval: 100000,
|
||||||
|
walSyncMode: 'full',
|
||||||
|
});
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(createSchema('t', {
|
||||||
|
id: { type: 'string', primaryKey: true },
|
||||||
|
}));
|
||||||
|
// 第一批:flush 落盘 + checkpoint 清 WAL
|
||||||
|
await engine.insert('t', [{ id: 'a' }, { id: 'b' }]);
|
||||||
|
await (engine as any).lsm.flush();
|
||||||
|
await (engine as any).wal.checkpoint();
|
||||||
|
// 第二批:只进 WAL(不 flush)
|
||||||
|
await engine.insert('t', [{ id: 'c' }, { id: 'd' }]);
|
||||||
|
|
||||||
|
// 模拟 WAL 分片文件被外部删掉(空洞场景)
|
||||||
|
const backend = (engine as any).backend;
|
||||||
|
await backend.delete('__wal_000000.bin');
|
||||||
|
|
||||||
|
await engine.close();
|
||||||
|
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', checkpointInterval: 100000 });
|
||||||
|
await engine2.open(dbName, 1);
|
||||||
|
// 已落盘的 a/b 必须保留;c/d 在 WAL 中且文件被删 → 恢复不到(可接受的保守丢弃)
|
||||||
|
const rows = await engine2.find('t', { table: 't' });
|
||||||
|
expect(rows.some((r) => r.id === 'a')).toBe(true);
|
||||||
|
expect(rows.some((r) => r.id === 'b')).toBe(true);
|
||||||
|
await engine2.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('旧格式 WAL 升级迁移:旧键重放后 checkpoint 清空', async () => {
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
// 手工构造旧格式 WAL 库:直接写旧键(模拟 v0.4.4 库崩溃现场)
|
||||||
|
const engine = new AriaEngine({
|
||||||
|
storageBackend: 'indexeddb',
|
||||||
|
checkpointInterval: 100000,
|
||||||
|
walSyncMode: 'full',
|
||||||
|
});
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(createSchema('t', {
|
||||||
|
id: { type: 'string', primaryKey: true },
|
||||||
|
}));
|
||||||
|
// 写入一批数据但只保留在 WAL(新格式)→ close 前手工转成旧格式
|
||||||
|
await engine.insert('t', [{ id: 'x' }]);
|
||||||
|
const backend = (engine as any).backend;
|
||||||
|
const raw = await backend.read('__wal_000000.bin');
|
||||||
|
expect(raw).not.toBeNull();
|
||||||
|
// 清掉新格式,写成旧格式(单记录键)
|
||||||
|
await backend.delete('__wal_000000.bin');
|
||||||
|
await backend.write('__wal_0', raw);
|
||||||
|
await backend.write('__wal_count', new TextEncoder().encode('1').buffer);
|
||||||
|
// 同时清掉内存里的 WAL 状态,模拟"重启"
|
||||||
|
await (engine as any).backend.close();
|
||||||
|
(engine as any).opened = false;
|
||||||
|
|
||||||
|
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', checkpointInterval: 100000 });
|
||||||
|
await engine2.open(dbName, 1);
|
||||||
|
const rows = await engine2.find('t', { table: 't' });
|
||||||
|
expect(rows).toHaveLength(1);
|
||||||
|
expect(rows[0].id).toBe('x');
|
||||||
|
// 恢复后 checkpoint 清空旧键
|
||||||
|
const keys = await (engine2 as any).backend.listKeys();
|
||||||
|
expect(keys.some((k: string) => k.startsWith('__wal_'))).toBe(false);
|
||||||
|
await engine2.close();
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
/**
|
||||||
|
* AriaEngine CRC32 单元测试
|
||||||
|
*/
|
||||||
|
import { crc32, crc32Continue } from '../../src/engine/aria/crc32';
|
||||||
|
|
||||||
|
const enc = new TextEncoder();
|
||||||
|
|
||||||
|
function bytes(s: string): Uint8Array {
|
||||||
|
return enc.encode(s);
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('CRC32', () => {
|
||||||
|
test('标准测试向量:空串 = 0', () => {
|
||||||
|
expect(crc32(new Uint8Array(0))).toBe(0x00000000);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('标准测试向量:123456789 = 0xCBF43926(IEEE 802.3 参考值)', () => {
|
||||||
|
expect(crc32(bytes('123456789'))).toBe(0xcbf43926);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('标准测试向量:The quick brown fox... = 0x414FA339', () => {
|
||||||
|
expect(crc32(bytes('The quick brown fox jumps over the lazy dog'))).toBe(0x414fa339);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('确定性:相同输入多次计算一致', () => {
|
||||||
|
const data = bytes('metona-sqlark aria engine');
|
||||||
|
const a = crc32(data);
|
||||||
|
const b = crc32(data);
|
||||||
|
expect(a).toBe(b);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('区分性:不同输入不同校验和', () => {
|
||||||
|
expect(crc32(bytes('hello'))).not.toBe(crc32(bytes('hellp')));
|
||||||
|
expect(crc32(bytes('abc'))).not.toBe(crc32(bytes('abd')));
|
||||||
|
});
|
||||||
|
|
||||||
|
test('单字节翻转可被检测', () => {
|
||||||
|
const original = bytes('metona-sqlark aria engine');
|
||||||
|
const flipped = new Uint8Array(original);
|
||||||
|
flipped[10] ^= 0xff;
|
||||||
|
expect(crc32(original)).not.toBe(crc32(flipped));
|
||||||
|
});
|
||||||
|
|
||||||
|
test('分段计算与一次性计算一致(crc32Continue)', () => {
|
||||||
|
const data = bytes('The quick brown fox jumps over the lazy dog');
|
||||||
|
const whole = crc32(data);
|
||||||
|
|
||||||
|
// 按 1 字节分段
|
||||||
|
let seg = 0;
|
||||||
|
for (let i = 0; i < data.length; i++) {
|
||||||
|
seg = crc32Continue(seg, data.subarray(i, i + 1));
|
||||||
|
}
|
||||||
|
expect(seg).toBe(whole);
|
||||||
|
|
||||||
|
// 按 5 字节分段
|
||||||
|
let seg2 = 0;
|
||||||
|
for (let i = 0; i < data.length; i += 5) {
|
||||||
|
seg2 = crc32Continue(seg2, data.subarray(i, i + 5));
|
||||||
|
}
|
||||||
|
expect(seg2).toBe(whole);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('seed 参数与 crc32Continue 等价', () => {
|
||||||
|
const data = bytes('metona-sqlark aria engine');
|
||||||
|
const head = data.subarray(0, 7);
|
||||||
|
const tail = data.subarray(7);
|
||||||
|
const whole = crc32(data);
|
||||||
|
const split = crc32(tail, crc32(head));
|
||||||
|
expect(split).toBe(whole);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('中文 UTF-8 字节正确参与计算', () => {
|
||||||
|
const data = bytes('玥玥的测试数据:大段中文内容');
|
||||||
|
expect(crc32(data)).toBe(crc32(new Uint8Array(data)));
|
||||||
|
});
|
||||||
|
|
||||||
|
test('大缓冲区(1MB 随机字节)稳定计算', () => {
|
||||||
|
const data = new Uint8Array(1024 * 1024);
|
||||||
|
let x = 12345;
|
||||||
|
for (let i = 0; i < data.length; i++) {
|
||||||
|
x = (x * 1103515245 + 12345) & 0x7fffffff;
|
||||||
|
data[i] = x & 0xff;
|
||||||
|
}
|
||||||
|
const a = crc32(data);
|
||||||
|
const b = crc32(data);
|
||||||
|
expect(a).toBe(b);
|
||||||
|
expect(a).not.toBe(0);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
/**
|
||||||
|
* 测试共享 — 真实语义 OPFS mock
|
||||||
|
*
|
||||||
|
* 模拟 OPFS 的完整语义:
|
||||||
|
* - createWritable({ keepExistingData }) + position 追加写
|
||||||
|
* - 文件内容以 ArrayBuffer 存储(跨 backend 实例共享于同一 Map)
|
||||||
|
* - entries() 迭代、removeEntry 删除
|
||||||
|
*
|
||||||
|
* 用法:
|
||||||
|
* const files = installOPFSMock(new Map());
|
||||||
|
* const backend = new OPFSBackend();
|
||||||
|
* await backend.open('any');
|
||||||
|
*/
|
||||||
|
|
||||||
|
export interface MockOpfsFile {
|
||||||
|
content: ArrayBuffer;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function installOPFSMock(files: Map<string, MockOpfsFile>): {
|
||||||
|
files: Map<string, MockOpfsFile>;
|
||||||
|
dir: {
|
||||||
|
getFileHandle: (name: string, opts?: { create?: boolean }) => Promise<{
|
||||||
|
getFile: () => Promise<{ size: number; arrayBuffer: () => Promise<ArrayBuffer> }>;
|
||||||
|
createWritable: (wOpts?: { keepExistingData?: boolean }) => Promise<{
|
||||||
|
write: (arg: ArrayBuffer | { type: string; position: number; data: ArrayBuffer }) => Promise<void>;
|
||||||
|
close: () => Promise<void>;
|
||||||
|
}>;
|
||||||
|
}>;
|
||||||
|
entries: () => AsyncGenerator<[string]>;
|
||||||
|
removeEntry: (name: string) => Promise<void>;
|
||||||
|
};
|
||||||
|
} {
|
||||||
|
const getFileHandle = async (name: string, opts?: { create?: boolean }) => {
|
||||||
|
if (!files.has(name)) {
|
||||||
|
if (!opts?.create) throw new Error(`NotFoundError: ${name}`);
|
||||||
|
files.set(name, { content: new ArrayBuffer(0) });
|
||||||
|
}
|
||||||
|
const entry = files.get(name)!;
|
||||||
|
return {
|
||||||
|
getFile: async () => ({ size: entry.content.byteLength, arrayBuffer: async () => entry.content }),
|
||||||
|
createWritable: async (wOpts?: { keepExistingData?: boolean }) => {
|
||||||
|
const w: {
|
||||||
|
write: (arg: ArrayBuffer | { type: string; position: number; data: ArrayBuffer }) => Promise<void>;
|
||||||
|
close: () => Promise<void>;
|
||||||
|
} = {
|
||||||
|
write: async (arg) => {
|
||||||
|
const keepExisting = wOpts?.keepExistingData ?? false;
|
||||||
|
const chunk = (arg as { type?: string }).type === 'write'
|
||||||
|
? { data: (arg as { data: ArrayBuffer }).data, position: (arg as { position: number }).position }
|
||||||
|
: { data: arg as ArrayBuffer, position: keepExisting ? entry.content.byteLength : 0 };
|
||||||
|
const merged = new Uint8Array(chunk.position + chunk.data.byteLength);
|
||||||
|
if (keepExisting || chunk.position > 0) {
|
||||||
|
merged.set(new Uint8Array(entry.content), 0);
|
||||||
|
}
|
||||||
|
merged.set(new Uint8Array(chunk.data), chunk.position);
|
||||||
|
entry.content = merged.buffer;
|
||||||
|
},
|
||||||
|
close: async () => { /* no-op */ },
|
||||||
|
};
|
||||||
|
return w;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
const dir = {
|
||||||
|
getFileHandle,
|
||||||
|
entries: async function* () {
|
||||||
|
for (const [name] of files) yield [name];
|
||||||
|
},
|
||||||
|
removeEntry: async (name: string) => {
|
||||||
|
files.delete(name);
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
Object.defineProperty(globalThis, 'navigator', {
|
||||||
|
value: {
|
||||||
|
storage: {
|
||||||
|
getDirectory: async () => ({
|
||||||
|
getDirectoryHandle: async (_name: string, _opts?: unknown) => dir,
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
configurable: true,
|
||||||
|
writable: true,
|
||||||
|
});
|
||||||
|
|
||||||
|
return { files, dir };
|
||||||
|
}
|
||||||
@@ -0,0 +1,140 @@
|
|||||||
|
/**
|
||||||
|
* v0.5.1 — CRUD 生命周期钩子真实接线测试
|
||||||
|
*
|
||||||
|
* 此前 beforeInsert/afterInsert/beforeUpdate/afterUpdate/beforeDelete/afterDelete
|
||||||
|
* 六个钩子虽在 HookName 类型与文档中宣称("14 种钩子"),但从未被任何路径触发。
|
||||||
|
* 本次验证 Table API 与 SQL query 两条路径均真实触发全部 14 个钩子。
|
||||||
|
*/
|
||||||
|
import { MetonaSqlark } from '../src/core';
|
||||||
|
import 'fake-indexeddb/auto';
|
||||||
|
|
||||||
|
async function createDb(name: string): Promise<MetonaSqlark> {
|
||||||
|
const db = new MetonaSqlark({ name: `hooks-${name}-${Date.now()}-${Math.random().toString(36).slice(2, 6)}`, mode: 'memory' });
|
||||||
|
await db.init();
|
||||||
|
await db.defineTable('users', {
|
||||||
|
id: { type: 'string', primaryKey: true },
|
||||||
|
name: { type: 'string' },
|
||||||
|
age: { type: 'number' },
|
||||||
|
});
|
||||||
|
return db;
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('v0.5.1 — CRUD hooks 真实接线', () => {
|
||||||
|
it('Table.insert / insertMany 触发 beforeInsert + afterInsert', async () => {
|
||||||
|
const db = await createDb('table-insert');
|
||||||
|
const events: string[] = [];
|
||||||
|
db.on('beforeInsert', (rows) => { events.push(`before:${(rows as unknown[]).length}`); });
|
||||||
|
db.on('afterInsert', (rows, pks) => { events.push(`after:${(rows as unknown[]).length}:${(pks as string[]).length}`); });
|
||||||
|
|
||||||
|
await db.table('users').insert({ id: '1', name: 'A' });
|
||||||
|
await db.table('users').insertMany([{ id: '2', name: 'B' }, { id: '3', name: 'C' }]);
|
||||||
|
|
||||||
|
expect(events).toEqual([
|
||||||
|
'before:1', 'after:1:1',
|
||||||
|
'before:2', 'after:2:2',
|
||||||
|
]);
|
||||||
|
await db.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('Table.update().execute() 触发 beforeUpdate + afterUpdate(含影响行数)', async () => {
|
||||||
|
const db = await createDb('table-update');
|
||||||
|
await db.table('users').insertMany([{ id: '1', name: 'A', age: 1 }, { id: '2', name: 'B', age: 2 }]);
|
||||||
|
const events: string[] = [];
|
||||||
|
db.on('beforeUpdate', (query, updates) => { events.push(`before:${(updates as Record<string, unknown>).name}`); });
|
||||||
|
db.on('afterUpdate', (_query, _updates, count) => { events.push(`after:${count}`); });
|
||||||
|
|
||||||
|
await db.table('users').update({ name: 'X' }).where({ age: { $gt: 1 } }).execute();
|
||||||
|
|
||||||
|
expect(events).toEqual(['before:X', 'after:1']);
|
||||||
|
await db.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('Table.delete().execute() 触发 beforeDelete + afterDelete(含影响行数)', async () => {
|
||||||
|
const db = await createDb('table-delete');
|
||||||
|
await db.table('users').insertMany([{ id: '1', name: 'A' }, { id: '2', name: 'B' }]);
|
||||||
|
const events: string[] = [];
|
||||||
|
db.on('beforeDelete', (query) => { events.push(`before:${(query as { where: Record<string, unknown> }).where.id}`); });
|
||||||
|
db.on('afterDelete', (_query, count) => { events.push(`after:${count}`); });
|
||||||
|
|
||||||
|
await db.table('users').delete().where({ id: '1' }).execute();
|
||||||
|
|
||||||
|
expect(events).toEqual(['before:1', 'after:1']);
|
||||||
|
await db.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('SQL query 路径触发全部 CRUD hooks(INSERT/UPDATE/DELETE)', async () => {
|
||||||
|
const db = await createDb('sql-crud');
|
||||||
|
await db.query("INSERT INTO users VALUES ('1', 'Alice', 30)");
|
||||||
|
const events: string[] = [];
|
||||||
|
db.on('beforeInsert', () => events.push('beforeInsert'));
|
||||||
|
db.on('afterInsert', () => events.push('afterInsert'));
|
||||||
|
db.on('beforeUpdate', () => events.push('beforeUpdate'));
|
||||||
|
db.on('afterUpdate', () => events.push('afterUpdate'));
|
||||||
|
db.on('beforeDelete', () => events.push('beforeDelete'));
|
||||||
|
db.on('afterDelete', () => events.push('afterDelete'));
|
||||||
|
|
||||||
|
await db.query("INSERT INTO users VALUES ('2', 'Bob', 25)");
|
||||||
|
await db.query("UPDATE users SET age = 26 WHERE id = '2'");
|
||||||
|
await db.query("DELETE FROM users WHERE id = '1'");
|
||||||
|
|
||||||
|
expect(events).toEqual([
|
||||||
|
'beforeInsert', 'afterInsert',
|
||||||
|
'beforeUpdate', 'afterUpdate',
|
||||||
|
'beforeDelete', 'afterDelete',
|
||||||
|
]);
|
||||||
|
await db.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('SQL INSERT 多行语句按列名映射触发 afterInsert', async () => {
|
||||||
|
const db = await createDb('sql-insert-cols');
|
||||||
|
const seen: unknown[] = [];
|
||||||
|
db.on('afterInsert', (_rows, pks) => seen.push(pks));
|
||||||
|
|
||||||
|
await db.query("INSERT INTO users (id, name) VALUES ('9', 'Zed')");
|
||||||
|
|
||||||
|
expect(seen).toEqual([['9']]);
|
||||||
|
await db.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('全部 14 个钩子均可注册并真实触发', async () => {
|
||||||
|
const db = await createDb('all-hooks');
|
||||||
|
const fired = new Set<string>();
|
||||||
|
const hooks = [
|
||||||
|
'beforeCreateTable', 'afterCreateTable',
|
||||||
|
'beforeDropTable', 'afterDropTable',
|
||||||
|
'beforeInsert', 'afterInsert',
|
||||||
|
'beforeUpdate', 'afterUpdate',
|
||||||
|
'beforeDelete', 'afterDelete',
|
||||||
|
'beforeQuery', 'afterQuery',
|
||||||
|
'beforeTransaction', 'afterTransaction',
|
||||||
|
] as const;
|
||||||
|
for (const h of hooks) db.on(h, () => fired.add(h));
|
||||||
|
|
||||||
|
await db.defineTable('t2', { id: { type: 'string', primaryKey: true } }); // createTable hooks
|
||||||
|
await db.table('t2').insert({ id: '1' }); // insert hooks
|
||||||
|
await db.table('t2').update({}).where({ id: '1' }).execute(); // update hooks
|
||||||
|
await db.table('t2').delete().where({ id: '1' }).execute(); // delete hooks
|
||||||
|
await db.query('SELECT * FROM t2'); // query hooks
|
||||||
|
await db.transaction(async () => { /* noop */ }); // transaction hooks
|
||||||
|
await db.dropTable('t2'); // dropTable hooks
|
||||||
|
|
||||||
|
for (const h of hooks) {
|
||||||
|
expect(fired.has(h)).toBe(true);
|
||||||
|
if (!fired.has(h)) throw new Error(`hook ${h} 应被触发`);
|
||||||
|
}
|
||||||
|
await db.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('事务路径不触发 CRUD hooks(Table 无 hooks 回调)', async () => {
|
||||||
|
const db = await createDb('tx-hooks');
|
||||||
|
let fired = false;
|
||||||
|
db.on('beforeInsert', () => { fired = true; });
|
||||||
|
|
||||||
|
await db.transaction(async (trx) => {
|
||||||
|
await trx.table('users').insert({ id: '1', name: 'Tx' });
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(fired).toBe(false);
|
||||||
|
await db.close();
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,172 @@
|
|||||||
|
/**
|
||||||
|
* v0.5.1 — 维护语句 SQL 入口测试(EXPLAIN / ANALYZE / REINDEX / VACUUM / SAVEPOINT)
|
||||||
|
*
|
||||||
|
* 此前这些功能仅有引擎方法/executor 分支而无 SQL 入口(parser 无法解析),
|
||||||
|
* 属"宣称存在但用户无法触发"的半成品。本次补齐 SQL 语法并验证真实执行。
|
||||||
|
*/
|
||||||
|
import { MetonaSqlark } from '../src/core';
|
||||||
|
import { parse } from '../src/sql/parser';
|
||||||
|
import { createSchema } from '../src/table/schema';
|
||||||
|
import 'fake-indexeddb/auto';
|
||||||
|
|
||||||
|
let dbCounter = 0;
|
||||||
|
function uniqueDB(): string {
|
||||||
|
return `maint-${Date.now()}-${++dbCounter}-${Math.random().toString(36).slice(2, 8)}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('v0.5.1 — 维护语句解析', () => {
|
||||||
|
it('EXPLAIN SELECT 解析为 ExplainStatement', () => {
|
||||||
|
const stmt = parse('EXPLAIN SELECT * FROM users WHERE id = \'1\'');
|
||||||
|
expect(stmt.type).toBe('EXPLAIN');
|
||||||
|
expect((stmt as { query: { type: string } }).query.type).toBe('SELECT');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('EXPLAIN 嵌套 EXPLAIN 拒绝', () => {
|
||||||
|
expect(() => parse('EXPLAIN EXPLAIN SELECT 1')).toThrow('Nested EXPLAIN');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('ANALYZE / ANALYZE TABLE 解析', () => {
|
||||||
|
expect(parse('ANALYZE users')).toMatchObject({ type: 'ANALYZE', table: 'users' });
|
||||||
|
expect(parse('ANALYZE TABLE users')).toMatchObject({ type: 'ANALYZE', table: 'users' });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('REINDEX / VACUUM 解析', () => {
|
||||||
|
expect(parse('REINDEX users')).toMatchObject({ type: 'REINDEX', table: 'users' });
|
||||||
|
expect(parse('REINDEX TABLE users')).toMatchObject({ type: 'REINDEX', table: 'users' });
|
||||||
|
expect(parse('VACUUM')).toMatchObject({ type: 'VACUUM' });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('SAVEPOINT / ROLLBACK TO / RELEASE 解析', () => {
|
||||||
|
expect(parse('SAVEPOINT sp1')).toMatchObject({ type: 'SAVEPOINT', name: 'sp1', action: 'SAVE' });
|
||||||
|
expect(parse('ROLLBACK TO sp1')).toMatchObject({ type: 'SAVEPOINT', name: 'sp1', action: 'ROLLBACK' });
|
||||||
|
expect(parse('ROLLBACK TO SAVEPOINT sp1')).toMatchObject({ type: 'SAVEPOINT', name: 'sp1', action: 'ROLLBACK' });
|
||||||
|
expect(parse('RELEASE SAVEPOINT sp1')).toMatchObject({ type: 'SAVEPOINT', name: 'sp1', action: 'RELEASE' });
|
||||||
|
expect(parse('RELEASE sp1')).toMatchObject({ type: 'SAVEPOINT', name: 'sp1', action: 'RELEASE' });
|
||||||
|
// ROLLBACK [TRANSACTION] 保持原语义
|
||||||
|
expect(parse('ROLLBACK')).toMatchObject({ type: 'ROLLBACK' });
|
||||||
|
expect(parse('ROLLBACK TRANSACTION')).toMatchObject({ type: 'ROLLBACK' });
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('v0.5.1 — 维护语句执行(Aria 引擎)', () => {
|
||||||
|
async function createAriaDb(): Promise<MetonaSqlark> {
|
||||||
|
const db = new MetonaSqlark({
|
||||||
|
name: uniqueDB(),
|
||||||
|
mode: 'aria',
|
||||||
|
diskEngine: 'indexeddb',
|
||||||
|
aria: { checkpointInterval: 100000, walSyncMode: 'full' },
|
||||||
|
});
|
||||||
|
await db.init();
|
||||||
|
await db.defineTable('users', {
|
||||||
|
id: { type: 'string', primaryKey: true },
|
||||||
|
name: { type: 'string', index: true },
|
||||||
|
age: { type: 'number' },
|
||||||
|
});
|
||||||
|
await db.table('users').insertMany([
|
||||||
|
{ id: '1', name: 'Alice', age: 30 },
|
||||||
|
{ id: '2', name: 'Bob', age: 25 },
|
||||||
|
{ id: '3', name: 'Carol', age: 35 },
|
||||||
|
]);
|
||||||
|
return db;
|
||||||
|
}
|
||||||
|
|
||||||
|
it('EXPLAIN SELECT 返回查询计划', async () => {
|
||||||
|
const db = await createAriaDb();
|
||||||
|
const plan = await db.query("EXPLAIN SELECT * FROM users WHERE id = '1'") as Record<string, unknown>;
|
||||||
|
expect(plan.type).toBe('SELECT');
|
||||||
|
expect(plan.table).toBe('users');
|
||||||
|
expect(plan.estimatedRows).toBe(1);
|
||||||
|
await db.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('EXPLAIN UPDATE / DELETE 返回计划', async () => {
|
||||||
|
const db = await createAriaDb();
|
||||||
|
const planU = await db.query("EXPLAIN UPDATE users SET age = 31 WHERE id = '1'") as Record<string, unknown>;
|
||||||
|
expect(planU.type).toBe('UPDATE');
|
||||||
|
const planD = await db.query('EXPLAIN DELETE FROM users WHERE id = \'1\'') as Record<string, unknown>;
|
||||||
|
expect(planD.type).toBe('DELETE');
|
||||||
|
await db.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('ANALYZE TABLE 收集统计信息', async () => {
|
||||||
|
const db = await createAriaDb();
|
||||||
|
const stats = await db.query('ANALYZE TABLE users') as Record<string, unknown>;
|
||||||
|
expect(stats.rowCount).toBe(3);
|
||||||
|
expect(stats.table).toBe('users');
|
||||||
|
await db.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('REINDEX TABLE 重建索引后索引查询正常', async () => {
|
||||||
|
const db = await createAriaDb();
|
||||||
|
const count = await db.query('REINDEX TABLE users') as number;
|
||||||
|
expect(count).toBeGreaterThanOrEqual(0);
|
||||||
|
const rows = await db.query("SELECT * FROM users WHERE name = 'Bob'") as Record<string, unknown>[];
|
||||||
|
expect(rows).toHaveLength(1);
|
||||||
|
await db.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('VACUUM 压缩成功', async () => {
|
||||||
|
const db = await createAriaDb();
|
||||||
|
const result = await db.query('VACUUM') as { compactedLevels: number };
|
||||||
|
expect(typeof result.compactedLevels).toBe('number');
|
||||||
|
await db.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('SAVEPOINT / ROLLBACK TO / RELEASE 真实执行', async () => {
|
||||||
|
const db = await createAriaDb();
|
||||||
|
await db.query('BEGIN');
|
||||||
|
await db.query("UPDATE users SET age = 99 WHERE id = '1'");
|
||||||
|
await db.query('SAVEPOINT sp_a');
|
||||||
|
await db.query("UPDATE users SET age = 88 WHERE id = '1'");
|
||||||
|
// 回滚到 savepoint:恢复 age=99
|
||||||
|
await db.query('ROLLBACK TO SAVEPOINT sp_a');
|
||||||
|
await db.query('RELEASE SAVEPOINT sp_a');
|
||||||
|
await db.query('COMMIT');
|
||||||
|
|
||||||
|
const rows = await db.query("SELECT age FROM users WHERE id = '1'") as Record<string, unknown>[];
|
||||||
|
expect(rows[0].age).toBe(99);
|
||||||
|
await db.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('savepoint 已存在 → 抛 SAVEPOINT_EXISTS', async () => {
|
||||||
|
const db = await createAriaDb();
|
||||||
|
await db.query('BEGIN');
|
||||||
|
await db.query('SAVEPOINT sp1');
|
||||||
|
await expect(db.query('SAVEPOINT sp1')).rejects.toMatchObject({ code: 'SAVEPOINT_EXISTS' });
|
||||||
|
await db.query('ROLLBACK');
|
||||||
|
await db.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('回滚到不存在的 savepoint → 抛 SAVEPOINT_NOT_FOUND', async () => {
|
||||||
|
const db = await createAriaDb();
|
||||||
|
await db.query('BEGIN');
|
||||||
|
await expect(db.query('ROLLBACK TO nope')).rejects.toMatchObject({ code: 'SAVEPOINT_NOT_FOUND' });
|
||||||
|
await db.query('ROLLBACK');
|
||||||
|
await db.close();
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('v0.5.1 — 维护语句执行(Memory 引擎不支持)', () => {
|
||||||
|
it('ANALYZE / REINDEX / VACUUM / SAVEPOINT → NOT_SUPPORTED', async () => {
|
||||||
|
const db = new MetonaSqlark({ name: uniqueDB(), mode: 'memory' });
|
||||||
|
await db.init();
|
||||||
|
await db.defineTable('t', { id: { type: 'string', primaryKey: true } });
|
||||||
|
|
||||||
|
await expect(db.query('ANALYZE TABLE t')).rejects.toMatchObject({ code: 'NOT_SUPPORTED' });
|
||||||
|
await expect(db.query('REINDEX TABLE t')).rejects.toMatchObject({ code: 'NOT_SUPPORTED' });
|
||||||
|
await expect(db.query('VACUUM')).rejects.toMatchObject({ code: 'NOT_SUPPORTED' });
|
||||||
|
await expect(db.query('SAVEPOINT sp')).rejects.toMatchObject({ code: 'NOT_SUPPORTED' });
|
||||||
|
await db.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('EXPLAIN 在所有引擎可用(返回基本信息)', async () => {
|
||||||
|
const db = new MetonaSqlark({ name: uniqueDB(), mode: 'memory' });
|
||||||
|
await db.init();
|
||||||
|
await db.defineTable('t', { id: { type: 'string', primaryKey: true } });
|
||||||
|
await db.table('t').insert({ id: '1' });
|
||||||
|
const plan = await db.query('EXPLAIN SELECT * FROM t') as Record<string, unknown>;
|
||||||
|
expect(plan.type).toBe('SELECT');
|
||||||
|
expect(plan.table).toBe('t');
|
||||||
|
await db.close();
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -287,14 +287,13 @@ describe('[v0.3.1] WAL 批量组提交', () => {
|
|||||||
await db.defineTable('users', { id: { type: 'string', primaryKey: true }, name: { type: 'string' } });
|
await db.defineTable('users', { id: { type: 'string', primaryKey: true }, name: { type: 'string' } });
|
||||||
|
|
||||||
const engine = db.getEngine() as any;
|
const engine = db.getEngine() as any;
|
||||||
// v0.4.2: WAL 记录与 count 改用 writeMany 单事务原子写入(记录仍合并为 1 次落盘)
|
// v0.4.5: WAL 改用分片存储(SegmentedWALStore,无 append 接口的后端回退单 key write);
|
||||||
const origWriteMany = engine.backend.writeMany.bind(engine.backend);
|
// 组提交语义保持:批量记录仍合并为 1 次底层落盘
|
||||||
const origWrite = engine.backend.write.bind(engine.backend);
|
const origWrite = engine.backend.write.bind(engine.backend);
|
||||||
let walWrites = 0;
|
let walWrites = 0;
|
||||||
engine.backend.writeMany = async (entries: Record<string, ArrayBuffer>) => {
|
engine.backend.write = async (key: string, data: ArrayBuffer) => {
|
||||||
const keys = Object.keys(entries);
|
if (key.startsWith('__wal_')) walWrites++;
|
||||||
if (keys.some((k) => k.startsWith('__wal_') && !k.startsWith('__wal_count'))) walWrites++;
|
return origWrite(key, data);
|
||||||
return origWriteMany(entries);
|
|
||||||
};
|
};
|
||||||
|
|
||||||
await db.table('users').insertMany([
|
await db.table('users').insertMany([
|
||||||
@@ -305,7 +304,6 @@ describe('[v0.3.1] WAL 批量组提交', () => {
|
|||||||
]);
|
]);
|
||||||
|
|
||||||
expect(walWrites).toBe(1); // 4 行合并为 1 次 WAL 落盘
|
expect(walWrites).toBe(1); // 4 行合并为 1 次 WAL 落盘
|
||||||
void origWrite;
|
|
||||||
const rows = await db.query('SELECT * FROM users') as Record<string, unknown>[];
|
const rows = await db.query('SELECT * FROM users') as Record<string, unknown>[];
|
||||||
expect(rows).toHaveLength(4);
|
expect(rows).toHaveLength(4);
|
||||||
await db.close();
|
await db.close();
|
||||||
|
|||||||
@@ -1,99 +0,0 @@
|
|||||||
/**
|
|
||||||
* utils.ts 单元测试模板
|
|
||||||
*/
|
|
||||||
|
|
||||||
import {
|
|
||||||
generateId,
|
|
||||||
escapeHTML,
|
|
||||||
debounce,
|
|
||||||
throttle,
|
|
||||||
deepMerge,
|
|
||||||
isBrowser,
|
|
||||||
} from '../src/utils';
|
|
||||||
|
|
||||||
describe('generateId', () => {
|
|
||||||
test('返回字符串', () => {
|
|
||||||
expect(typeof generateId()).toBe('string');
|
|
||||||
});
|
|
||||||
|
|
||||||
test('多次调用产生不同值', () => {
|
|
||||||
const ids = new Set<string>();
|
|
||||||
for (let i = 0; i < 100; i++) ids.add(generateId());
|
|
||||||
expect(ids.size).toBe(100);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
describe('escapeHTML', () => {
|
|
||||||
test('转义 < > &', () => {
|
|
||||||
const out = escapeHTML('<a>&b</a>');
|
|
||||||
expect(out).toContain('<');
|
|
||||||
expect(out).toContain('>');
|
|
||||||
expect(out).toContain('&');
|
|
||||||
});
|
|
||||||
|
|
||||||
test('null/undefined 返回空字符串', () => {
|
|
||||||
expect(escapeHTML(null)).toBe('');
|
|
||||||
expect(escapeHTML(undefined)).toBe('');
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
describe('debounce', () => {
|
|
||||||
test('延迟执行', (done) => {
|
|
||||||
let called = 0;
|
|
||||||
const fn = debounce(() => { called++; }, 20);
|
|
||||||
fn();
|
|
||||||
expect(called).toBe(0);
|
|
||||||
setTimeout(() => {
|
|
||||||
expect(called).toBe(1);
|
|
||||||
done();
|
|
||||||
}, 50);
|
|
||||||
});
|
|
||||||
|
|
||||||
test('多次调用只执行最后一次', (done) => {
|
|
||||||
let result = 0;
|
|
||||||
const fn = debounce((v: number) => { result = v; }, 20);
|
|
||||||
fn(1);
|
|
||||||
fn(2);
|
|
||||||
fn(3);
|
|
||||||
setTimeout(() => {
|
|
||||||
expect(result).toBe(3);
|
|
||||||
done();
|
|
||||||
}, 50);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
describe('throttle', () => {
|
|
||||||
test('首次立即执行', () => {
|
|
||||||
let called = 0;
|
|
||||||
const fn = throttle(() => { called++; }, 50);
|
|
||||||
fn();
|
|
||||||
expect(called).toBe(1);
|
|
||||||
});
|
|
||||||
|
|
||||||
test('限流期内不重复执行', () => {
|
|
||||||
let called = 0;
|
|
||||||
const fn = throttle(() => { called++; }, 50);
|
|
||||||
fn();
|
|
||||||
fn();
|
|
||||||
fn();
|
|
||||||
expect(called).toBe(1);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
describe('deepMerge', () => {
|
|
||||||
test('浅层合并', () => {
|
|
||||||
const out = deepMerge({ a: 1, b: 2 }, { b: 3, c: 4 });
|
|
||||||
expect(out).toEqual({ a: 1, b: 3, c: 4 });
|
|
||||||
});
|
|
||||||
|
|
||||||
test('深层对象合并', () => {
|
|
||||||
const out = deepMerge({ obj: { x: 1 } }, { obj: { y: 2 } });
|
|
||||||
expect(out).toEqual({ obj: { x: 1, y: 2 } });
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
describe('isBrowser', () => {
|
|
||||||
test('jsdom 环境下返回 true', () => {
|
|
||||||
expect(isBrowser()).toBe(true);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
@@ -23,8 +23,8 @@ import { createSchema } from '../src/table/schema';
|
|||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
describe('[v0.2.5] P0-1: 版本号统一', () => {
|
describe('[v0.2.5] P0-1: 版本号统一', () => {
|
||||||
test('VERSION 常量为当前版本(0.4.4)', () => {
|
test('VERSION 常量为当前版本(0.5.1)', () => {
|
||||||
expect(VERSION).toBe('0.4.4');
|
expect(VERSION).toBe('0.5.1');
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -339,7 +339,6 @@ describe('[v0.2.5] P2-12: WAL 大小阈值', () => {
|
|||||||
const fakeWal = { flush: async () => {}, checkpoint: async () => {}, getBufferedCount: () => 0 };
|
const fakeWal = { flush: async () => {}, checkpoint: async () => {}, getBufferedCount: () => 0 };
|
||||||
const cm = new CheckpointManager(fakeLsm, fakeWal, null, 1000, 1024);
|
const cm = new CheckpointManager(fakeLsm, fakeWal, null, 1000, 1024);
|
||||||
expect(cm).toBeDefined();
|
expect(cm).toBeDefined();
|
||||||
expect(cm.getOpCount()).toBe(0);
|
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
@@ -400,7 +400,7 @@ describe('[v0.3.3] P1-9: Savepoint + MVCC 一致性', () => {
|
|||||||
|
|
||||||
describe('[v0.3.3] 端到端', () => {
|
describe('[v0.3.3] 端到端', () => {
|
||||||
test('全部修复点可共存于 MetonaSqlark API', async () => {
|
test('全部修复点可共存于 MetonaSqlark API', async () => {
|
||||||
expect(VERSION).toBe('0.4.4');
|
expect(VERSION).toBe('0.5.1');
|
||||||
const db = new MetonaSqlark({ name: `e2e-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`, mode: 'memory' });
|
const db = new MetonaSqlark({ name: `e2e-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`, mode: 'memory' });
|
||||||
await db.init();
|
await db.init();
|
||||||
await db.defineTable('users', {
|
await db.defineTable('users', {
|
||||||
|
|||||||
@@ -111,7 +111,7 @@ describe('P0 — 事务进行中 checkpoint 不得截断 WAL', () => {
|
|||||||
await engine.insert('t', [{ id: 'tx1' }]);
|
await engine.insert('t', [{ id: 'tx1' }]);
|
||||||
await engine.rollbackTransaction();
|
await engine.rollbackTransaction();
|
||||||
// 事务结束后 checkpoint 可正常截断
|
// 事务结束后 checkpoint 可正常截断
|
||||||
await (engine as any).checkpointManager.forceCheckpoint();
|
await (engine as any).checkpointManager.checkpoint();
|
||||||
await (engine as any).backend.close();
|
await (engine as any).backend.close();
|
||||||
(engine as any).opened = false;
|
(engine as any).opened = false;
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user