feat: v0.6.1 — AriaEngine 可选自研 KVStore 后端(storageBackend: 'kv')+ KVStore APPEND 日志类型 + 10 个 aria+kv 集成测试 + 文档全量同步
CI / test (20.x) (push) Successful in 10m55s
CI / test (22.x) (push) Successful in 10m49s
CI / e2e (push) Successful in 9m54s
CI / test (18.x) (push) Successful in 10m58s
CI / test (24.x) (push) Successful in 10m41s

This commit is contained in:
thzxx
2026-08-10 14:52:38 +08:00
parent a56a496148
commit 25bab0f9aa
21 changed files with 895 additions and 42 deletions
+6
View File
@@ -29,6 +29,12 @@ All notable changes to MetonaSqlark will be documented in this file.
### Added
- **AriaEngine 可选自研 KVStore 后端**`storageBackend: 'kv'` / `diskEngine: 'kv'`):
- aria 完全跑在自研存储栈上(KVStore 日志+快照),不再依赖浏览器 OPFS APINode/测试用 SharedMemory 介质)
- KVStore 新增 APPEND 日志操作类型:aria WAL 分片追加 O(chunk) 高效,恢复按序拼接,checkpoint 快照含最终值
- writeMany/deleteMany 经 KVStore 单日志记录真原子(此前 OPFS 逐文件写)
- aria 崩溃恢复 = KVStore 双层恢复 + WAL 重放;二级索引/页面化兼容(kv 后端 SSTable 整 value 存储)
- 10 个 aria+kv 集成测试(后端单元/CRUD 持久化/WAL 追加恢复/checkpoint 混合/索引/大数据崩溃/高层 API 事务级联)
- 27 个生产异常场景测试(`tests/production-abnormal.test.ts`):
- KVStore 边界:空 value/空 key、1MB 大 value、中文/emoji/冒号/超长 key、未 open、
重复 open/close 幂等、clear 后写、1 万小 key、写失败后继续可用
+11 -11
View File
@@ -3,8 +3,8 @@
<p align="center">
<img src="https://img.shields.io/badge/version-0.6.1-blue?style=flat-square" alt="version">
<img src="https://img.shields.io/badge/license-MIT-green?style=flat-square" alt="license">
<img src="https://img.shields.io/badge/coverage-89.3%25-brightgreen?style=flat-square" alt="coverage">
<img src="https://img.shields.io/badge/tests-1049%20passed-success?style=flat-square" alt="tests">
<img src="https://img.shields.io/badge/coverage-89.4%25-brightgreen?style=flat-square" alt="coverage">
<img src="https://img.shields.io/badge/tests-1064%20passed-success?style=flat-square" alt="tests">
</p>
> 基于 TypeScript 的**前端关系型数据库**,支持完整 SQL 查询、Query Builder 链式 API、与 **AriaEngine 自研存储引擎**。
@@ -20,7 +20,7 @@
- 🛠 **维护语句 SQL 入口**`EXPLAIN`/`ANALYZE`/`REINDEX`/`VACUUM`/`SAVEPOINT` 原生 SQL 支持(v0.5.1 补齐)
- 🛡 **输入校验全覆盖**`maxLength`/`min`/`max` 约束、类型检查、必填验证
- 💾 **多引擎架构** — Memory / **KVStore**(自研 KV 引擎)/ OPFS / Hybrid(write-through) / Aria 五种模式(v0.6.0: IndexedDB 完全移除)
- 💾 **KVStore 自研 KV 引擎** — 日志结构化事务存储:多 key 原子写(putMany/deleteMany 单记录原子追加)、快照 checkpoint、崩溃两阶段恢复、CRC-32 自愈;替代 IndexedDBv0.6.0
- 💾 **KVStore 自研 KV 引擎** — 日志结构化事务存储:多 key 原子写(putMany/deleteMany 单记录原子追加)、快照 checkpoint、崩溃两阶段恢复、CRC-32 自愈、APPEND 追加写入(aria WAL 分片用);替代 IndexedDBv0.6.0)且可作 AriaEngine 后端(v0.6.1
- 🔄 **旧库一键迁移**`migrateFromIndexedDB()` 把旧 IndexedDB 数据(schema/索引/行)导入新引擎(v0.6.0)
- 📝 **完整 SQL 支持** — SELECT/JOIN/子查询/GROUP BY/HAVING/ORDER BY/LIMIT/BETWEEN/IF NOT EXISTS/ALTER TABLE/TRUNCATE TABLE/UNION/INSERT...SELECT/事务语句/CREATE INDEX/EXISTSv0.3.0+ CASE WHEN/哈希连接/组提交(v0.3.1+ 多标签页同步(v0.3.2
- 🚰 **流式查询**`queryStream`/`stream()` 逐行回调,Aria LSM 惰性扫描不物化结果集(v0.4.0)
@@ -36,7 +36,7 @@
- 🛡 **级联环路保护** — A→B→A 循环引用 + CASCADE 不无限递归(visited 保护,全引擎)(v0.6.1
- ⚛️ **多表事务真原子** — 事务 commit / 级联更新合并单条日志记录(崩溃无部分提交)(v0.6.1)
- 🌐 **浏览器兼容** — Chrome 102+ / Firefox 111+ / Safari 15.2+ / Edge 102+ / Node.js 16+
- 🧪 **1049 测试 · 89.3% 覆盖率** — 65 套件 + 12 个 Playwright 真实 Chromium e2e(含崩溃注入),生产级质量保证
- 🧪 **1064 测试 · 89.4% 覆盖率** — 66 套件 + 12 个 Playwright 真实 Chromium e2e(含崩溃注入),生产级质量保证
---
@@ -111,7 +111,7 @@ import { MetonaSqlark } from '@metona-team/metona-sqlark';
const db = await MetonaSqlark.create({
name: 'my-app',
mode: 'hybrid', // 'memory' | 'disk' | 'hybrid' | 'aria'
diskEngine: 'opfs', // 'opfs' | 'memory'v0.6.0: IndexedDB 已移除)
diskEngine: 'opfs', // 'opfs' | 'kv'(自研 KVStore 后端)| 'memory'v0.6.0: IndexedDB 已移除)
});
// 定义表 — 支持外键级联
@@ -218,7 +218,7 @@ await db2.disconnect(); // 引用计数 -1
|------|------|------|------|
| `name` | `string` | `'metona-sqlark'` | 数据库名称 |
| `mode` | `'memory' \| 'disk' \| 'hybrid' \| 'aria'` | `'hybrid'` | 存储模式 🆕 aria |
| `diskEngine` | `'opfs' \| 'memory'` | `'opfs'` | 磁盘引擎(aria 模式下为存储后端;v0.6.0: IndexedDB 已移除 |
| `diskEngine` | `'opfs' \| 'memory' \| 'kv'` | `'opfs'` | 磁盘引擎(aria 模式下为存储后端;'kv' = 自研 KVStore 后端 v0.6.1 |
| `version` | `number` | `1` | 版本号 |
| `maxRowsPerQuery` | `number` | `0` | 查询结果行数上限(0=不限制)✅ v0.2.5 生效 |
| `debug` | `boolean` | `false` | 调试模式,输出详细日志 🆕 |
@@ -363,7 +363,7 @@ const { data, loading, refresh } = useSqlarkQuery(db, 'SELECT * FROM users');
- **适用**: 需要内存速度 + 磁盘持久化的混合场景
### Aria 模式
- **环境**: 所有浏览器(后端可选 OPFS / Memory
- **环境**: 所有浏览器(后端可选 OPFS / **KVStore**(自研,v0.6.1/ Memory
- **限制**: Memory 后端重启丢失;OPFS 后端需 OPFS 支持(Chrome 102+ / Firefox 111+ / Safari 15.2+);同一库仅允许一个标签页打开(Web Locks 独占锁,第二个标签页抛 `ARIA_LOCKED`
- **能力**: LSM-Tree 存储引擎、SSTable 4KB 页面化物理存储(OPFS 后端默认启用,BufferPool LRU 缓存)、二级索引(跨重启恢复)、MVCC 事务、WAL 分片文件原子写入崩溃恢复(真追加 + 空洞检测)、ON UPDATE/DELETE 外键级联、Bloom Filter、**全库 AES-GCM 透明加密**v0.5.0)、LZ4 压缩、Savepoint、EXPLAIN、ANALYZE、REINDEX、VACUUM、`repair()` 自愈(含孤儿页面清理)
- **适用**: 大规模数据分析、需要自研引擎可控性的高级场景
@@ -379,7 +379,7 @@ AriaEngine 是内置的页面式存储引擎,对标 SQLite 的设计理念:
const db = await MetonaSqlark.create({
name: 'my-app',
mode: 'aria', // 🆕 自研引擎模式
diskEngine: 'opfs', // 'opfs' | 'memory'
diskEngine: 'opfs', // 'opfs' | 'kv'(自研 KVStore 后端)| 'memory'
aria: { // 🆕 AriaEngine 专属配置透传(v0.5.0
walSyncMode: 'full', // WAL 同步模式
encryption: { password: 'my-password' }, // 全库 AES-GCM 加密
@@ -453,9 +453,9 @@ npm run typecheck # 类型检查
| 指标 | 数值 |
|------|------|
| 测试用例 | 1049 |
| 测试套件 | 65+12 Playwright e2e |
| 行覆盖率 | 89.3% |
| 测试用例 | 1064 |
| 测试套件 | 66+12 Playwright e2e |
| 行覆盖率 | 89.4% |
| SQL 关键字 | 72 |
| 存储引擎 | 5Memory / **KVStore** / OPFS / Hybrid / **Aria** |
+113 -4
View File
@@ -1124,14 +1124,17 @@ var KVLogOp;
(function (KVLogOp) {
KVLogOp[KVLogOp["PUT"] = 1] = "PUT";
KVLogOp[KVLogOp["DELETE"] = 2] = "DELETE";
/** v0.6.1: 追加写入(value 拼接语义,恢复时按序 concataria WAL 分片用) */
KVLogOp[KVLogOp["APPEND"] = 3] = "APPEND";
})(KVLogOp || (KVLogOp = {}));
/**
* 编码一条日志记录
* @param seq 日志序号
* @param puts key value 写入条目
* @param deletes 删除 key 列表
* @param appends key 追加块列表APPEND 语义恢复时拼接
*/
function encodeLogRecord(seq, puts, deletes = []) {
function encodeLogRecord(seq, puts, deletes = [], appends = {}) {
const encoder = new TextEncoder();
const entries = [];
for (const [key, value] of Object.entries(puts)) {
@@ -1140,6 +1143,9 @@ function encodeLogRecord(seq, puts, deletes = []) {
for (const key of deletes) {
entries.push({ op: KVLogOp.DELETE, key, value: new ArrayBuffer(0) });
}
for (const [key, value] of Object.entries(appends)) {
entries.push({ op: KVLogOp.APPEND, key, value });
}
// 预编码 key 字节,计算总长度
const entryBytes = [];
let total = 4 + 4 + 4; // recordLen + seq + entryCount
@@ -1545,6 +1551,18 @@ class KVStore {
await this.appendRecord({}, keys);
});
}
/**
* v0.6.1: 追加写入value 拼接语义 aria WAL 分片等追加型数据用
* 日志记录 APPEND 类型O(chunk) 高效恢复时按 seq 顺序拼接
* checkpoint 后快照含最终值崩溃时该次追加全有或全无单记录原子
*/
async appendValue(key, chunk) {
if (chunk.byteLength === 0)
return;
await this.enqueue(async () => {
await this.appendRecord({}, [], { [key]: chunk });
});
}
// =======================================================================
// 维护
// =======================================================================
@@ -1617,9 +1635,9 @@ class KVStore {
return run;
}
/** 追加一条日志记录并更新内存索引(队列内调用,无并发) */
async appendRecord(puts, deletes) {
async appendRecord(puts, deletes, appends = {}) {
this.seq++;
const record = encodeLogRecord(this.seq, puts, deletes);
const record = encodeLogRecord(this.seq, puts, deletes, appends);
try {
// 日志追加:介质 append(真追加)或回退读+拼+写
const data = record.buffer.slice(record.byteOffset, record.byteOffset + record.byteLength);
@@ -1652,6 +1670,18 @@ class KVStore {
for (const key of deletes) {
this.index.delete(key);
}
for (const [key, chunk] of Object.entries(appends)) {
const existing = this.index.get(key);
if (existing) {
const combined = new Uint8Array(existing.byteLength + chunk.byteLength);
combined.set(new Uint8Array(existing), 0);
combined.set(new Uint8Array(chunk), existing.byteLength);
this.index.set(key, combined.buffer);
}
else {
this.index.set(key, chunk);
}
}
this.logBytes += record.byteLength;
// 自动 checkpoint(日志超阈值)
if (this.checkpointThreshold > 0 && this.logBytes >= this.checkpointThreshold) {
@@ -1675,6 +1705,18 @@ class KVStore {
if (e.op === KVLogOp.PUT) {
this.index.set(e.key, e.value);
}
else if (e.op === KVLogOp.APPEND) {
const existing = this.index.get(e.key);
if (existing) {
const combined = new Uint8Array(existing.byteLength + e.value.byteLength);
combined.set(new Uint8Array(existing), 0);
combined.set(new Uint8Array(e.value), existing.byteLength);
this.index.set(e.key, combined.buffer);
}
else {
this.index.set(e.key, e.value);
}
}
else {
this.index.delete(e.key);
}
@@ -4822,6 +4864,68 @@ class MemoryBackend {
}
}
/**
* KVStoreBackend 基于自研 KVStore AriaEngine 存储后端
* @module engine/aria/store/kvstore_backend
*
* v0.6.1: AriaEngine 可选后端storageBackend: 'kv'完全跑在自研 KVStore
* - write/read/delete/listKeys/exists/clear KVStore
* - append KVStore.appendValue日志 APPEND 类型O(chunk) 高效aria WAL 分片用
* - writeMany/deleteMany KVStore.putMany/deleteMany单日志记录真原子
* 此前 OPFS 逐文件写靠空洞检测兜底KV 后端原生原子
* - 崩溃恢复KVStore 快照+日志恢复 aria 打开重放自己的 WAL双层恢复
*
* 介质浏览器 OPFSKVStore 默认 Node SharedMemoryaria 不再依赖浏览器 OPFS API
*/
class KVStoreBackend {
constructor(medium, checkpointThreshold) {
this.kv = new KVStore(medium, checkpointThreshold);
}
/** 底层 KVStore(测试/诊断用) */
getKV() {
return this.kv;
}
async open(name) {
await this.kv.open(name);
}
async close() {
await this.kv.close();
}
isOpen() {
return this.kv.isOpen();
}
async read(key) {
return this.kv.get(key);
}
async write(key, data) {
await this.kv.put(key, data);
}
/** 追加写入(KVStore APPEND 日志,O(chunk) */
async append(key, data) {
await this.kv.appendValue(key, data);
}
/** 多 key 原子写入(单日志记录) */
async writeMany(entries) {
await this.kv.putMany(entries);
}
async delete(key) {
await this.kv.delete(key);
}
/** 多 key 原子删除(单日志记录) */
async deleteMany(keys) {
await this.kv.deleteMany(keys);
}
async listKeys() {
return this.kv.listKeys();
}
async exists(key) {
return this.kv.exists(key);
}
async clear() {
await this.kv.clear();
}
}
/**
* AriaEngine Crypto 页面级 AES-GCM 加密
* @module engine/aria/crypto
@@ -5957,6 +6061,10 @@ class AriaEngine {
if (this.config.storageBackend === 'opfs') {
baseBackend = new OPFSBackend();
}
else if (this.config.storageBackend === 'kv') {
// v0.6.1: 自研 KVStore 后端(aria 完全跑在自研存储栈上,不依赖浏览器 OPFS)
baseBackend = new KVStoreBackend();
}
else {
baseBackend = new MemoryBackend();
}
@@ -11594,8 +11702,9 @@ class MetonaSqlark {
return new KVStoreEngine();
case 'aria':
// v0.4.5: 透传 AriaEngine 专属配置(walSyncMode/checkpointInterval/encryption/pageStorage 等)
// v0.6.1: diskEngine 'kv' → 自研 KVStore 后端
return new AriaEngine({
storageBackend: diskEngine === 'memory' ? 'memory' : 'opfs',
storageBackend: diskEngine === 'memory' ? 'memory' : diskEngine === 'kv' ? 'kv' : 'opfs',
...(this.config.aria ?? {}),
});
case 'hybrid':
+1 -1
View File
File diff suppressed because one or more lines are too long
+3 -3
View File
@@ -18,7 +18,7 @@ interface AriaEngineConfig {
/** 是否启用页面压缩(默认 false) */
compression?: boolean;
/** 存储后端 */
storageBackend?: 'opfs' | 'memory';
storageBackend?: 'opfs' | 'memory' | 'kv';
/** WAL 大小阈值(字节,超过则强制 checkpoint,默认 16MB */
walSizeThreshold?: number;
/** 最大内存预算(MB,默认 64) */
@@ -45,8 +45,8 @@ interface AriaEngineConfig {
*/
/** 存储模式 */
type StorageMode = 'memory' | 'disk' | 'hybrid' | 'aria';
/** 磁盘引擎类型(v0.6.0: IndexedDB 已移除'memory' 供 aria 内存后端) */
type DiskEngine = 'opfs' | 'memory';
/** 磁盘引擎类型(v0.6.0: IndexedDB 已移除'memory' 供 aria 内存后端;'kv' 供 aria 自研 KVStore 后端) */
type DiskEngine = 'opfs' | 'memory' | 'kv';
/** 字段数据类型 */
type FieldType = 'string' | 'number' | 'boolean' | 'date' | 'json';
/** 列定义 */
+113 -4
View File
@@ -1120,14 +1120,17 @@ var KVLogOp;
(function (KVLogOp) {
KVLogOp[KVLogOp["PUT"] = 1] = "PUT";
KVLogOp[KVLogOp["DELETE"] = 2] = "DELETE";
/** v0.6.1: 追加写入(value 拼接语义,恢复时按序 concataria WAL 分片用) */
KVLogOp[KVLogOp["APPEND"] = 3] = "APPEND";
})(KVLogOp || (KVLogOp = {}));
/**
* 编码一条日志记录
* @param seq 日志序号
* @param puts key value 写入条目
* @param deletes 删除 key 列表
* @param appends key 追加块列表APPEND 语义恢复时拼接
*/
function encodeLogRecord(seq, puts, deletes = []) {
function encodeLogRecord(seq, puts, deletes = [], appends = {}) {
const encoder = new TextEncoder();
const entries = [];
for (const [key, value] of Object.entries(puts)) {
@@ -1136,6 +1139,9 @@ function encodeLogRecord(seq, puts, deletes = []) {
for (const key of deletes) {
entries.push({ op: KVLogOp.DELETE, key, value: new ArrayBuffer(0) });
}
for (const [key, value] of Object.entries(appends)) {
entries.push({ op: KVLogOp.APPEND, key, value });
}
// 预编码 key 字节,计算总长度
const entryBytes = [];
let total = 4 + 4 + 4; // recordLen + seq + entryCount
@@ -1541,6 +1547,18 @@ class KVStore {
await this.appendRecord({}, keys);
});
}
/**
* v0.6.1: 追加写入value 拼接语义 aria WAL 分片等追加型数据用
* 日志记录 APPEND 类型O(chunk) 高效恢复时按 seq 顺序拼接
* checkpoint 后快照含最终值崩溃时该次追加全有或全无单记录原子
*/
async appendValue(key, chunk) {
if (chunk.byteLength === 0)
return;
await this.enqueue(async () => {
await this.appendRecord({}, [], { [key]: chunk });
});
}
// =======================================================================
// 维护
// =======================================================================
@@ -1613,9 +1631,9 @@ class KVStore {
return run;
}
/** 追加一条日志记录并更新内存索引(队列内调用,无并发) */
async appendRecord(puts, deletes) {
async appendRecord(puts, deletes, appends = {}) {
this.seq++;
const record = encodeLogRecord(this.seq, puts, deletes);
const record = encodeLogRecord(this.seq, puts, deletes, appends);
try {
// 日志追加:介质 append(真追加)或回退读+拼+写
const data = record.buffer.slice(record.byteOffset, record.byteOffset + record.byteLength);
@@ -1648,6 +1666,18 @@ class KVStore {
for (const key of deletes) {
this.index.delete(key);
}
for (const [key, chunk] of Object.entries(appends)) {
const existing = this.index.get(key);
if (existing) {
const combined = new Uint8Array(existing.byteLength + chunk.byteLength);
combined.set(new Uint8Array(existing), 0);
combined.set(new Uint8Array(chunk), existing.byteLength);
this.index.set(key, combined.buffer);
}
else {
this.index.set(key, chunk);
}
}
this.logBytes += record.byteLength;
// 自动 checkpoint(日志超阈值)
if (this.checkpointThreshold > 0 && this.logBytes >= this.checkpointThreshold) {
@@ -1671,6 +1701,18 @@ class KVStore {
if (e.op === KVLogOp.PUT) {
this.index.set(e.key, e.value);
}
else if (e.op === KVLogOp.APPEND) {
const existing = this.index.get(e.key);
if (existing) {
const combined = new Uint8Array(existing.byteLength + e.value.byteLength);
combined.set(new Uint8Array(existing), 0);
combined.set(new Uint8Array(e.value), existing.byteLength);
this.index.set(e.key, combined.buffer);
}
else {
this.index.set(e.key, e.value);
}
}
else {
this.index.delete(e.key);
}
@@ -4818,6 +4860,68 @@ class MemoryBackend {
}
}
/**
* KVStoreBackend 基于自研 KVStore AriaEngine 存储后端
* @module engine/aria/store/kvstore_backend
*
* v0.6.1: AriaEngine 可选后端storageBackend: 'kv'完全跑在自研 KVStore
* - write/read/delete/listKeys/exists/clear KVStore
* - append KVStore.appendValue日志 APPEND 类型O(chunk) 高效aria WAL 分片用
* - writeMany/deleteMany KVStore.putMany/deleteMany单日志记录真原子
* 此前 OPFS 逐文件写靠空洞检测兜底KV 后端原生原子
* - 崩溃恢复KVStore 快照+日志恢复 aria 打开重放自己的 WAL双层恢复
*
* 介质浏览器 OPFSKVStore 默认 Node SharedMemoryaria 不再依赖浏览器 OPFS API
*/
class KVStoreBackend {
constructor(medium, checkpointThreshold) {
this.kv = new KVStore(medium, checkpointThreshold);
}
/** 底层 KVStore(测试/诊断用) */
getKV() {
return this.kv;
}
async open(name) {
await this.kv.open(name);
}
async close() {
await this.kv.close();
}
isOpen() {
return this.kv.isOpen();
}
async read(key) {
return this.kv.get(key);
}
async write(key, data) {
await this.kv.put(key, data);
}
/** 追加写入(KVStore APPEND 日志,O(chunk) */
async append(key, data) {
await this.kv.appendValue(key, data);
}
/** 多 key 原子写入(单日志记录) */
async writeMany(entries) {
await this.kv.putMany(entries);
}
async delete(key) {
await this.kv.delete(key);
}
/** 多 key 原子删除(单日志记录) */
async deleteMany(keys) {
await this.kv.deleteMany(keys);
}
async listKeys() {
return this.kv.listKeys();
}
async exists(key) {
return this.kv.exists(key);
}
async clear() {
await this.kv.clear();
}
}
/**
* AriaEngine Crypto 页面级 AES-GCM 加密
* @module engine/aria/crypto
@@ -5953,6 +6057,10 @@ class AriaEngine {
if (this.config.storageBackend === 'opfs') {
baseBackend = new OPFSBackend();
}
else if (this.config.storageBackend === 'kv') {
// v0.6.1: 自研 KVStore 后端(aria 完全跑在自研存储栈上,不依赖浏览器 OPFS)
baseBackend = new KVStoreBackend();
}
else {
baseBackend = new MemoryBackend();
}
@@ -11590,8 +11698,9 @@ class MetonaSqlark {
return new KVStoreEngine();
case 'aria':
// v0.4.5: 透传 AriaEngine 专属配置(walSyncMode/checkpointInterval/encryption/pageStorage 等)
// v0.6.1: diskEngine 'kv' → 自研 KVStore 后端
return new AriaEngine({
storageBackend: diskEngine === 'memory' ? 'memory' : 'opfs',
storageBackend: diskEngine === 'memory' ? 'memory' : diskEngine === 'kv' ? 'kv' : 'opfs',
...(this.config.aria ?? {}),
});
case 'hybrid':
+1 -1
View File
File diff suppressed because one or more lines are too long
+113 -4
View File
@@ -1126,14 +1126,17 @@
(function (KVLogOp) {
KVLogOp[KVLogOp["PUT"] = 1] = "PUT";
KVLogOp[KVLogOp["DELETE"] = 2] = "DELETE";
/** v0.6.1: 追加写入(value 拼接语义,恢复时按序 concataria WAL 分片用) */
KVLogOp[KVLogOp["APPEND"] = 3] = "APPEND";
})(KVLogOp || (KVLogOp = {}));
/**
* 编码一条日志记录
* @param seq 日志序号
* @param puts key value 写入条目
* @param deletes 删除 key 列表
* @param appends key 追加块列表APPEND 语义恢复时拼接
*/
function encodeLogRecord(seq, puts, deletes = []) {
function encodeLogRecord(seq, puts, deletes = [], appends = {}) {
const encoder = new TextEncoder();
const entries = [];
for (const [key, value] of Object.entries(puts)) {
@@ -1142,6 +1145,9 @@
for (const key of deletes) {
entries.push({ op: KVLogOp.DELETE, key, value: new ArrayBuffer(0) });
}
for (const [key, value] of Object.entries(appends)) {
entries.push({ op: KVLogOp.APPEND, key, value });
}
// 预编码 key 字节,计算总长度
const entryBytes = [];
let total = 4 + 4 + 4; // recordLen + seq + entryCount
@@ -1547,6 +1553,18 @@
await this.appendRecord({}, keys);
});
}
/**
* v0.6.1: 追加写入value 拼接语义 aria WAL 分片等追加型数据用
* 日志记录 APPEND 类型O(chunk) 高效恢复时按 seq 顺序拼接
* checkpoint 后快照含最终值崩溃时该次追加全有或全无单记录原子
*/
async appendValue(key, chunk) {
if (chunk.byteLength === 0)
return;
await this.enqueue(async () => {
await this.appendRecord({}, [], { [key]: chunk });
});
}
// =======================================================================
// 维护
// =======================================================================
@@ -1619,9 +1637,9 @@
return run;
}
/** 追加一条日志记录并更新内存索引(队列内调用,无并发) */
async appendRecord(puts, deletes) {
async appendRecord(puts, deletes, appends = {}) {
this.seq++;
const record = encodeLogRecord(this.seq, puts, deletes);
const record = encodeLogRecord(this.seq, puts, deletes, appends);
try {
// 日志追加:介质 append(真追加)或回退读+拼+写
const data = record.buffer.slice(record.byteOffset, record.byteOffset + record.byteLength);
@@ -1654,6 +1672,18 @@
for (const key of deletes) {
this.index.delete(key);
}
for (const [key, chunk] of Object.entries(appends)) {
const existing = this.index.get(key);
if (existing) {
const combined = new Uint8Array(existing.byteLength + chunk.byteLength);
combined.set(new Uint8Array(existing), 0);
combined.set(new Uint8Array(chunk), existing.byteLength);
this.index.set(key, combined.buffer);
}
else {
this.index.set(key, chunk);
}
}
this.logBytes += record.byteLength;
// 自动 checkpoint(日志超阈值)
if (this.checkpointThreshold > 0 && this.logBytes >= this.checkpointThreshold) {
@@ -1677,6 +1707,18 @@
if (e.op === KVLogOp.PUT) {
this.index.set(e.key, e.value);
}
else if (e.op === KVLogOp.APPEND) {
const existing = this.index.get(e.key);
if (existing) {
const combined = new Uint8Array(existing.byteLength + e.value.byteLength);
combined.set(new Uint8Array(existing), 0);
combined.set(new Uint8Array(e.value), existing.byteLength);
this.index.set(e.key, combined.buffer);
}
else {
this.index.set(e.key, e.value);
}
}
else {
this.index.delete(e.key);
}
@@ -4824,6 +4866,68 @@
}
}
/**
* KVStoreBackend 基于自研 KVStore AriaEngine 存储后端
* @module engine/aria/store/kvstore_backend
*
* v0.6.1: AriaEngine 可选后端storageBackend: 'kv'完全跑在自研 KVStore
* - write/read/delete/listKeys/exists/clear KVStore
* - append KVStore.appendValue日志 APPEND 类型O(chunk) 高效aria WAL 分片用
* - writeMany/deleteMany KVStore.putMany/deleteMany单日志记录真原子
* 此前 OPFS 逐文件写靠空洞检测兜底KV 后端原生原子
* - 崩溃恢复KVStore 快照+日志恢复 aria 打开重放自己的 WAL双层恢复
*
* 介质浏览器 OPFSKVStore 默认 Node SharedMemoryaria 不再依赖浏览器 OPFS API
*/
class KVStoreBackend {
constructor(medium, checkpointThreshold) {
this.kv = new KVStore(medium, checkpointThreshold);
}
/** 底层 KVStore(测试/诊断用) */
getKV() {
return this.kv;
}
async open(name) {
await this.kv.open(name);
}
async close() {
await this.kv.close();
}
isOpen() {
return this.kv.isOpen();
}
async read(key) {
return this.kv.get(key);
}
async write(key, data) {
await this.kv.put(key, data);
}
/** 追加写入(KVStore APPEND 日志,O(chunk) */
async append(key, data) {
await this.kv.appendValue(key, data);
}
/** 多 key 原子写入(单日志记录) */
async writeMany(entries) {
await this.kv.putMany(entries);
}
async delete(key) {
await this.kv.delete(key);
}
/** 多 key 原子删除(单日志记录) */
async deleteMany(keys) {
await this.kv.deleteMany(keys);
}
async listKeys() {
return this.kv.listKeys();
}
async exists(key) {
return this.kv.exists(key);
}
async clear() {
await this.kv.clear();
}
}
/**
* AriaEngine Crypto 页面级 AES-GCM 加密
* @module engine/aria/crypto
@@ -5959,6 +6063,10 @@
if (this.config.storageBackend === 'opfs') {
baseBackend = new OPFSBackend();
}
else if (this.config.storageBackend === 'kv') {
// v0.6.1: 自研 KVStore 后端(aria 完全跑在自研存储栈上,不依赖浏览器 OPFS)
baseBackend = new KVStoreBackend();
}
else {
baseBackend = new MemoryBackend();
}
@@ -11596,8 +11704,9 @@
return new KVStoreEngine();
case 'aria':
// v0.4.5: 透传 AriaEngine 专属配置(walSyncMode/checkpointInterval/encryption/pageStorage 等)
// v0.6.1: diskEngine 'kv' → 自研 KVStore 后端
return new AriaEngine({
storageBackend: diskEngine === 'memory' ? 'memory' : 'opfs',
storageBackend: diskEngine === 'memory' ? 'memory' : diskEngine === 'kv' ? 'kv' : 'opfs',
...(this.config.aria ?? {}),
});
case 'hybrid':
+1 -1
View File
File diff suppressed because one or more lines are too long
+1 -1
View File
File diff suppressed because one or more lines are too long
+2 -2
View File
@@ -768,7 +768,7 @@ db.<span class="f">broadcastChange</span>(<span class="s">'users'</span>);</pre>
<strong>v0.5.0 存储后端生产级硬化</strong> — 真实 CRC-32 完整性校验(SSTable 整文件 + WAL 记录,旧文件兼容)· 全库 AES-256-GCM 透明加密(EncryptedBackend + PBKDF2 密钥派生 + 密码验证)· WAL 分片文件重构(真追加 + 空洞检测 + 旧格式迁移)· SSTable 4KB 页面化物理存储(BufferPool/FileManager 真实接入,meta 存 pageIds 兼容旧数据)· OPFS 后端 v2(append 真追加 / 写队列健壮性 / 残留清理)· Web Locks 多标签页独占锁(ARIA_LOCKED)· LZ4 v2 原始大小头 · Playwright 真实 Chromium e2e7 用例)· DatabaseConfig.aria 配置透传。<br>
<strong>v0.5.1 深度审查修复</strong> — 14 个生命周期钩子全部真实接线(此前 6 个 CRUD 钩子从未触发)· EXPLAIN / ANALYZE / REINDEX / VACUUM / SAVEPOINT SQL 入口补齐(此前仅有引擎方法无法触发)· db.backup() 公共方法 · 删除全部死代码(utils.ts 整文件 / MVCC 读侧 / estimateQueryCost 未接线优化器 / 40+ 统计辅助方法)。<br>
<strong>v0.6.0 完全移除 IndexedDB</strong> — 自研 KVStore 事务存储引擎(多 key 原子写 = 单日志记录原子追加 · 快照 checkpoint + 两阶段崩溃恢复 · CRC-32 自愈)· disk 模式切换 KVStoreEngine(替代 IndexedDBEngine + OPFSEngine)· 事务内 DDL / 外键级联 / 二级索引完整持久化 · migrateFromIndexedDB() 旧库一键迁移 · 10 万 key 压力验证 · e2e 崩溃注入 + KVStoreEngine 真实环境(12 用例)。<br>
<strong>v0.6.1 生产可用性深度审查</strong> — MemoryEngine 级联环(A→B→A)无限递归修复(visited 保护,与 Aria 对齐)· KVStore 快照损坏水位 bug 修复(metaSeq 误跳日志)· 多表事务 / 级联写入合并单条日志记录真原子(崩溃无部分提交)· 未 open 防护统一 · 27 个异常场景测试(空值/大 value/特殊字符 key/写失败/事务故障/级联环/批量删除等)· 1049 测试 65 套件 · 89.3% 行覆盖率。</p>
<strong>v0.6.1 生产可用性深度审查</strong> — MemoryEngine 级联环(A→B→A)无限递归修复(visited 保护,与 Aria 对齐)· KVStore 快照损坏水位 bug 修复(metaSeq 误跳日志)· 多表事务 / 级联写入合并单条日志记录真原子(崩溃无部分提交)· 未 open 防护统一 · <strong>AriaEngine 可选自研 KVStore 后端</strong>storageBackend: 'kv'KVStore APPEND 日志支持 WAL 追加、writeMany 真原子、不再依赖浏览器 OPFS)· 27 个异常场景测试 + 10 个 aria+kv 集成测试 · 1064 测试 66 套件 · 89.4% 行覆盖率。</p>
<h3>存储模式对比</h3>
<table>
@@ -829,7 +829,7 @@ db.<span class="f">broadcastChange</span>(<span class="s">'users'</span>);</pre>
<tr><td><code>checkpointInterval</code></td><td><code>number</code></td><td><code>1000</code></td><td>Checkpoint 触发间隔(操作数)</td></tr>
<tr><td><code>walSizeThreshold</code></td><td><code>number</code></td><td><code>16777216</code></td><td>WAL 大小阈值(字节),超阈值触发 checkpoint ✅ v0.2.5</td></tr>
<tr><td><code>compression</code></td><td><code>boolean</code></td><td><code>false</code></td><td>是否启用页面压缩</td></tr>
<tr><td><code>storageBackend</code></td><td><code>'opfs'|'memory'</code></td><td><code>'opfs'</code></td><td>存储后端类型(v0.6.0: IndexedDB 已移除</td></tr>
<tr><td><code>storageBackend</code></td><td><code>'opfs'|'memory'|'kv'</code></td><td><code>'opfs'</code></td><td>存储后端类型('kv' = 自研 KVStore 后端 🆕 v0.6.1</td></tr>
<tr><td><code>encryption</code></td><td><code>{ password: string }</code></td><td>-</td><td>全库 AES-256-GCM 透明加密(PBKDF2 派生 + salt 持久化 + 密码验证)🆕 v0.5.0</td></tr>
<tr><td><code>pageStorage</code></td><td><code>boolean</code></td><td>OPFS 自动启用</td><td>SSTable 4KB 页面化物理存储(BufferPool 缓存)🆕 v0.5.0</td></tr>
</table>
+4 -4
View File
@@ -153,7 +153,7 @@
<!-- Hero -->
<section class="hero">
<div class="container">
<div class="badge" style="margin-bottom:24px;"><span class="dot"></span> v0.6.1 生产可用性硬化 — 1049测试 65套件 · 自研 KVStore 事务引擎 · 多 key 原子写 · 级联环保护 · 崩溃恢复 · 旧库一键迁移</div>
<div class="badge" style="margin-bottom:24px;"><span class="dot"></span> v0.6.1 生产可用性硬化 — 1064测试 66套件 · 自研 KVStore 事务引擎 · 多 key 原子写 · 级联环保护 · 崩溃恢复 · 旧库一键迁移</div>
<h1>前端的 <span class="gradient-text">SQL 数据库</span></h1>
<p>TypeScript 原生构建,5 种存储引擎,支持完整 SQL 查询。<br>零运行时依赖,开箱即用。AriaEngine 自研引擎:LSM-Tree + WAL 同步 + MVCC。</p>
<div class="actions">
@@ -414,12 +414,12 @@ npm install @metona-team/metona-sqlark
<p>MetonaSqlark 的核心指标</p>
</div>
<div class="stats">
<div class="stat-card"><div class="num">1049</div><div class="label">测试用例</div></div>
<div class="stat-card"><div class="num">89.3%</div><div class="label">行覆盖率</div></div>
<div class="stat-card"><div class="num">1064</div><div class="label">测试用例</div></div>
<div class="stat-card"><div class="num">89.4%</div><div class="label">行覆盖率</div></div>
<div class="stat-card"><div class="num">~27KB</div><div class="label">gzip 体积</div></div>
<div class="stat-card"><div class="num">5</div><div class="label">存储引擎</div></div>
<div class="stat-card"><div class="num">72</div><div class="label">SQL 关键字</div></div>
<div class="stat-card"><div class="num">65</div><div class="label">测试套件</div></div>
<div class="stat-card"><div class="num">66</div><div class="label">测试套件</div></div>
</div>
</div>
</section>
+2 -2
View File
@@ -10,8 +10,8 @@
/** 存储模式 */
export type StorageMode = 'memory' | 'disk' | 'hybrid' | 'aria';
/** 磁盘引擎类型(v0.6.0: IndexedDB 已移除'memory' 供 aria 内存后端) */
export type DiskEngine = 'opfs' | 'memory';
/** 磁盘引擎类型(v0.6.0: IndexedDB 已移除'memory' 供 aria 内存后端;'kv' 供 aria 自研 KVStore 后端) */
export type DiskEngine = 'opfs' | 'memory' | 'kv';
// ---------------------------------------------------------------------------
// 字段类型
+2 -1
View File
@@ -557,8 +557,9 @@ export class MetonaSqlark {
return new KVStoreEngine();
case 'aria':
// v0.4.5: 透传 AriaEngine 专属配置(walSyncMode/checkpointInterval/encryption/pageStorage 等)
// v0.6.1: diskEngine 'kv' → 自研 KVStore 后端
return new AriaEngine({
storageBackend: diskEngine === 'memory' ? 'memory' : 'opfs',
storageBackend: diskEngine === 'memory' ? 'memory' : diskEngine === 'kv' ? 'kv' : 'opfs',
...(this.config.aria ?? {}),
});
case 'hybrid':
+4
View File
@@ -22,6 +22,7 @@ import { DatabaseLock } from './locks';
import { WALRecordType, type WALRecord } from './types';
import { CheckpointManager } from './wal/checkpoint';
import { MemoryBackend, type IStorageBackend } from './store/backend';
import { KVStoreBackend } from './store/kvstore_backend';
import { OPFSBackend } from './store/opfs_backend';
import { EncryptedBackend } from './store/encrypted_backend';
import { PageSSTableStore } from './store/page_sstable_store';
@@ -117,6 +118,9 @@ export class AriaEngine implements IStorageEngine {
let baseBackend: IStorageBackend;
if (this.config.storageBackend === 'opfs') {
baseBackend = new OPFSBackend();
} else if (this.config.storageBackend === 'kv') {
// v0.6.1: 自研 KVStore 后端(aria 完全跑在自研存储栈上,不依赖浏览器 OPFS)
baseBackend = new KVStoreBackend();
} else {
baseBackend = new MemoryBackend();
}
+80
View File
@@ -0,0 +1,80 @@
/**
* KVStoreBackend KVStore AriaEngine
* @module engine/aria/store/kvstore_backend
*
* v0.6.1: AriaEngine storageBackend: 'kv' KVStore
* - write/read/delete/listKeys/exists/clear KVStore
* - append KVStore.appendValue APPEND O(chunk) aria WAL
* - writeMany/deleteMany KVStore.putMany/deleteMany
* OPFS KV
* - KVStore + aria WAL
*
* OPFSKVStore Node SharedMemoryaria OPFS API
*/
import type { IStorageBackend } from './backend';
import { KVStore } from '../../kvstore/index';
export class KVStoreBackend implements IStorageBackend {
private kv: KVStore;
constructor(medium?: IStorageBackend, checkpointThreshold?: number) {
this.kv = new KVStore(medium, checkpointThreshold);
}
/** 底层 KVStore(测试/诊断用) */
getKV(): KVStore {
return this.kv;
}
async open(name: string): Promise<void> {
await this.kv.open(name);
}
async close(): Promise<void> {
await this.kv.close();
}
isOpen(): boolean {
return this.kv.isOpen();
}
async read(key: string): Promise<ArrayBuffer | null> {
return this.kv.get(key);
}
async write(key: string, data: ArrayBuffer): Promise<void> {
await this.kv.put(key, data);
}
/** 追加写入(KVStore APPEND 日志,O(chunk) */
async append(key: string, data: ArrayBuffer): Promise<void> {
await this.kv.appendValue(key, data);
}
/** 多 key 原子写入(单日志记录) */
async writeMany(entries: Record<string, ArrayBuffer>): Promise<void> {
await this.kv.putMany(entries);
}
async delete(key: string): Promise<void> {
await this.kv.delete(key);
}
/** 多 key 原子删除(单日志记录) */
async deleteMany(keys: string[]): Promise<void> {
await this.kv.deleteMany(keys);
}
async listKeys(): Promise<string[]> {
return this.kv.listKeys();
}
async exists(key: string): Promise<boolean> {
return this.kv.exists(key);
}
async clear(): Promise<void> {
await this.kv.clear();
}
}
+1 -1
View File
@@ -207,7 +207,7 @@ export interface AriaEngineConfig {
/** 是否启用页面压缩(默认 false) */
compression?: boolean;
/** 存储后端 */
storageBackend?: 'opfs' | 'memory';
storageBackend?: 'opfs' | 'memory' | 'kv';
/** WAL 大小阈值(字节,超过则强制 checkpoint,默认 16MB */
walSizeThreshold?: number;
/** 最大内存预算(MB,默认 64) */
+39 -2
View File
@@ -223,6 +223,18 @@ export class KVStore {
});
}
/**
* v0.6.1: 追加写入value aria WAL
* APPEND O(chunk) seq
* checkpoint
*/
async appendValue(key: string, chunk: ArrayBuffer): Promise<void> {
if (chunk.byteLength === 0) return;
await this.enqueue(async () => {
await this.appendRecord({}, [], { [key]: chunk });
});
}
// =======================================================================
// 维护
// =======================================================================
@@ -301,9 +313,13 @@ export class KVStore {
}
/** 追加一条日志记录并更新内存索引(队列内调用,无并发) */
private async appendRecord(puts: Record<string, ArrayBuffer>, deletes: string[]): Promise<void> {
private async appendRecord(
puts: Record<string, ArrayBuffer>,
deletes: string[],
appends: Record<string, ArrayBuffer> = {},
): Promise<void> {
this.seq++;
const record = encodeLogRecord(this.seq, puts, deletes);
const record = encodeLogRecord(this.seq, puts, deletes, appends);
try {
// 日志追加:介质 append(真追加)或回退读+拼+写
const data = record.buffer.slice(record.byteOffset, record.byteOffset + record.byteLength) as ArrayBuffer;
@@ -334,6 +350,17 @@ export class KVStore {
for (const key of deletes) {
this.index.delete(key);
}
for (const [key, chunk] of Object.entries(appends)) {
const existing = this.index.get(key);
if (existing) {
const combined = new Uint8Array(existing.byteLength + chunk.byteLength);
combined.set(new Uint8Array(existing), 0);
combined.set(new Uint8Array(chunk), existing.byteLength);
this.index.set(key, combined.buffer as ArrayBuffer);
} else {
this.index.set(key, chunk);
}
}
this.logBytes += record.byteLength;
// 自动 checkpoint(日志超阈值)
@@ -358,6 +385,16 @@ export class KVStore {
for (const e of entries) {
if (e.op === KVLogOp.PUT) {
this.index.set(e.key, e.value);
} else if (e.op === KVLogOp.APPEND) {
const existing = this.index.get(e.key);
if (existing) {
const combined = new Uint8Array(existing.byteLength + e.value.byteLength);
combined.set(new Uint8Array(existing), 0);
combined.set(new Uint8Array(e.value), existing.byteLength);
this.index.set(e.key, combined.buffer as ArrayBuffer);
} else {
this.index.set(e.key, e.value);
}
} else {
this.index.delete(e.key);
}
+7
View File
@@ -23,6 +23,8 @@ import { crc32 } from '../aria/crc32';
export const enum KVLogOp {
PUT = 1,
DELETE = 2,
/** v0.6.1: 追加写入(value 拼接语义,恢复时按序 concataria WAL 分片用) */
APPEND = 3,
}
/** 解析后的日志记录 */
@@ -40,11 +42,13 @@ export interface KVLogRecord {
* @param seq
* @param puts key value
* @param deletes key
* @param appends key APPEND
*/
export function encodeLogRecord(
seq: number,
puts: Record<string, ArrayBuffer>,
deletes: string[] = [],
appends: Record<string, ArrayBuffer> = {},
): Uint8Array {
const encoder = new TextEncoder();
const entries: { op: KVLogOp; key: string; value: ArrayBuffer }[] = [];
@@ -54,6 +58,9 @@ export function encodeLogRecord(
for (const key of deletes) {
entries.push({ op: KVLogOp.DELETE, key, value: new ArrayBuffer(0) });
}
for (const [key, value] of Object.entries(appends)) {
entries.push({ op: KVLogOp.APPEND, key, value });
}
// 预编码 key 字节,计算总长度
const entryBytes: { op: KVLogOp; key: Uint8Array; value: Uint8Array }[] = [];
+291
View File
@@ -0,0 +1,291 @@
/**
* KVStoreBackend + AriaEngine(kv )
*
*
* 1. KVStoreBackend ////clear/close
* 2. AriaEngine + kv CRUD
* 3. WAL backend.append KVStore APPEND
* 4. kv WAL
* 5.
* 6. SSTable KVStore key
* 7. writeMany aria
*/
import { KVStoreBackend } from '../../src/engine/aria/store/kvstore_backend';
import { SharedMemoryBackend } from '../../src/engine/kvstore/shared_memory_medium';
import { AriaEngine } from '../../src/engine/aria/index';
import { createSchema } from '../../src/table/schema';
let counter = 0;
function uniqueDB(): string {
return `kvbe-${Date.now()}-${++counter}-${Math.random().toString(36).slice(2, 6)}`;
}
const enc = (s: string) => new TextEncoder().encode(s).buffer as ArrayBuffer;
const dec = (b: ArrayBuffer | null) => (b ? new TextDecoder().decode(b) : null);
beforeEach(() => {
SharedMemoryBackend.clearRegistry();
});
describe('KVStoreBackend — 单元', () => {
it('读写/追加/批量/删除/clear 全接口', async () => {
const backend = new KVStoreBackend(new SharedMemoryBackend());
await backend.open(uniqueDB());
await backend.write('k1', enc('V1'));
expect(dec(await backend.read('k1'))).toBe('V1');
expect(await backend.exists('k1')).toBe(true);
// 追加
await backend.append('wal', enc('A'));
await backend.append('wal', enc('B'));
expect(dec(await backend.read('wal'))).toBe('AB');
// 批量原子
await backend.writeMany({ a: enc('1'), b: enc('2') });
expect(dec(await backend.read('b'))).toBe('2');
await backend.deleteMany(['a']);
expect(await backend.exists('a')).toBe(false);
// listKeys / delete / clear
const keys = await backend.listKeys();
expect(keys).toContain('k1');
expect(keys).toContain('wal');
await backend.delete('k1');
expect(await backend.exists('k1')).toBe(false);
await backend.clear();
expect(await backend.listKeys()).toEqual([]);
await backend.close();
expect(backend.isOpen()).toBe(false);
});
it('跨实例持久化(SharedMemory 语义)', async () => {
const dbName = uniqueDB();
const b1 = new KVStoreBackend(new SharedMemoryBackend());
await b1.open(dbName);
await b1.write('k', enc('PERSIST'));
await b1.append('wal', enc('LOG'));
await b1.close();
const b2 = new KVStoreBackend(new SharedMemoryBackend());
await b2.open(dbName);
expect(dec(await b2.read('k'))).toBe('PERSIST');
expect(dec(await b2.read('wal'))).toBe('LOG');
await b2.close();
});
});
describe('AriaEngine + kv 后端(storageBackend: kv', () => {
function createEngine(): AriaEngine {
const engine = new AriaEngine({
storageBackend: 'kv',
checkpointInterval: 100000,
walSyncMode: 'full',
memtableSizeThreshold: 64 * 1024 * 1024,
});
return engine;
}
it('CRUD 持久化:写 → close → 重开数据完整(KVStore 快照/日志)', async () => {
const dbName = uniqueDB();
const e1 = createEngine();
await e1.open(dbName, 1);
await e1.createTable(createSchema('users', {
id: { type: 'string', primaryKey: true },
name: { type: 'string', required: true },
age: { type: 'number', default: 0 },
}));
await e1.insert('users', [
{ id: '1', name: 'Alice', age: 30 },
{ id: '2', name: 'Bob', age: 25 },
]);
await e1.close();
const e2 = createEngine();
await e2.open(dbName, 1);
expect(await e2.count('users')).toBe(2);
const rows = await e2.find('users', { table: 'users', where: { name: 'Alice' } });
expect(rows[0].age).toBe(30);
await e2.close();
});
it('WAL 分片经 KVStore APPEND 追加:崩溃恢复重放', async () => {
const dbName = uniqueDB();
const e1 = createEngine();
await e1.open(dbName, 1);
await e1.createTable(createSchema('logs', {
id: { type: 'string', primaryKey: true },
msg: { type: 'string' },
}));
// 不 flush 不 checkpoint:数据全在 WALKVStore APPEND 记录)
for (let i = 0; i < 50; i++) {
await e1.insert('logs', [{ id: `l-${i}`, msg: `msg-${i}` }]);
}
// 模拟崩溃:直接断开(不 close)
await (e1 as any).backend.close();
(e1 as any).opened = false;
const e2 = createEngine();
await e2.open(dbName, 1);
expect(await e2.count('logs')).toBe(50);
await e2.close();
});
it('checkpoint 后崩溃:快照 + WAL 混合恢复', async () => {
const dbName = uniqueDB();
const e1 = createEngine();
await e1.open(dbName, 1);
await e1.createTable(createSchema('t', {
id: { type: 'string', primaryKey: true },
v: { type: 'number' },
}));
await e1.insert('t', [{ id: 'a', v: 1 }]);
// 完整 checkpointflush LSMa 落盘为 SSTable+ 截断 WAL
await (e1 as any).checkpointManager.checkpoint();
await e1.insert('t', [{ id: 'b', v: 2 }]); // b 只在 WAL
await (e1 as any).backend.close();
(e1 as any).opened = false;
const e2 = createEngine();
await e2.open(dbName, 1);
expect(await e2.count('t')).toBe(2);
await e2.close();
});
it('二级索引跨重启恢复', async () => {
const dbName = uniqueDB();
const e1 = createEngine();
await e1.open(dbName, 1);
await e1.createTable(createSchema('items', {
id: { type: 'string', primaryKey: true },
tag: { type: 'string', index: true },
}));
await e1.insert('items', [
{ id: '1', tag: 'a' }, { id: '2', tag: 'b' }, { id: '3', tag: 'a' },
]);
await e1.close();
const e2 = createEngine();
await e2.open(dbName, 1);
const rows = await e2.find('items', { table: 'items', where: { tag: 'a' } });
expect(rows).toHaveLength(2);
await e2.close();
});
it('页面化存储:SSTable 页面经 KVStore 读写', async () => {
const dbName = uniqueDB();
const e1 = createEngine();
await e1.open(dbName, 1);
await e1.createTable(createSchema('docs', {
id: { type: 'string', primaryKey: true },
body: { type: 'string' },
}));
const rows = [] as Record<string, unknown>[];
for (let i = 0; i < 30; i++) {
rows.push({ id: `d-${i}`, body: '页面化内容'.repeat(40) });
}
await e1.insert('docs', rows);
await (e1 as any).lsm.flush();
// kv 后端:SSTable 整 value 存储(KVStore 日志型天然适合大块,无需拆 4KB 页面)
const kv = (e1 as any).backend.getKV();
const keys = await kv.listKeys();
expect(keys.some((k: string) => k.startsWith('sst_'))).toBe(true);
await e1.close();
const e2 = createEngine();
await e2.open(dbName, 1);
expect(await e2.count('docs')).toBe(30);
await e2.close();
});
it('大数据量 + 崩溃模拟:KV 后端零丢失', async () => {
const dbName = uniqueDB();
const e1 = createEngine();
await e1.open(dbName, 1);
await e1.createTable(createSchema('big', {
id: { type: 'string', primaryKey: true },
v: { type: 'number' },
}));
for (let batch = 0; batch < 10; batch++) {
const rows = [] as Record<string, unknown>[];
for (let i = 0; i < 100; i++) rows.push({ id: `k${batch}-${i}`, v: i });
await e1.insert('big', rows);
if ((batch + 1) % 3 === 0) await (e1 as any).kvCheckpoint?.();
}
await (e1 as any).backend.close();
(e1 as any).opened = false;
const e2 = createEngine();
await e2.open(dbName, 1);
expect(await e2.count('big')).toBe(1000);
await e2.close();
}, 60000);
});
// ===================================================================
// 高层 APIMetonaSqlark + diskEngine 'kv'
// ===================================================================
describe('MetonaSqlark + diskEngine: kv(高层 API', () => {
it('创建 → 建表 → CRUD → SQL → 持久化重开', async () => {
const { MetonaSqlark } = require('../../src/core');
const dbName = uniqueDB();
const db = new MetonaSqlark({
name: dbName,
mode: 'aria',
diskEngine: 'kv',
aria: { walSyncMode: 'full', checkpointInterval: 100000, memtableSizeThreshold: 64 * 1024 * 1024 },
});
await db.init();
await db.defineTable('users', {
id: { type: 'string', primaryKey: true },
name: { type: 'string', required: true },
age: { type: 'number', default: 0 },
});
await db.query("INSERT INTO users VALUES ('1', 'Alice', 30)");
await db.query("INSERT INTO users VALUES ('2', 'Bob', 25)");
const rows = await db.query("SELECT * FROM users WHERE age > 26") as Record<string, unknown>[];
expect(rows).toHaveLength(1);
expect(rows[0].name).toBe('Alice');
await db.close();
// 重开:数据完整
const db2 = new MetonaSqlark({
name: dbName,
mode: 'aria',
diskEngine: 'kv',
aria: { walSyncMode: 'full', checkpointInterval: 100000, memtableSizeThreshold: 64 * 1024 * 1024 },
});
await db2.init();
expect(await db2.table('users').count()).toBe(2);
await db2.close();
});
it('事务回滚 + 外键级联在 kv 后端工作', async () => {
const { MetonaSqlark } = require('../../src/core');
const db = new MetonaSqlark({
name: uniqueDB(), mode: 'aria', diskEngine: 'kv',
aria: { walSyncMode: 'full', checkpointInterval: 100000 },
});
await db.init();
await db.defineTable('users', { id: { type: 'string', primaryKey: true } });
await db.defineTable('orders', {
id: { type: 'string', primaryKey: true },
user_id: { type: 'string', references: 'users.id', onDelete: 'CASCADE' },
});
// 事务回滚
await expect(db.transaction(async (trx) => {
await trx.table('users').insert({ id: '1' });
throw new Error('boom');
})).rejects.toThrow('boom');
expect(await db.table('users').count()).toBe(0);
// 级联删除
await db.table('users').insert({ id: '1' });
await db.table('orders').insert({ id: 'o1', user_id: '1' });
await db.table('users').delete().where({ id: '1' }).execute();
expect(await db.table('orders').count()).toBe(0);
await db.close();
});
});
+100
View File
@@ -366,3 +366,103 @@ describe('KVStore — 编解码单元', () => {
expect(decodeSnapshot(bytes)).toBeNull();
});
});
// ===================================================================
// APPEND 追加写入(v0.6.1
// ===================================================================
describe('KVStore — APPEND 追加写入', () => {
it('appendValue 拼接 + 恢复完整(跨实例)', async () => {
const dbName = uniqueDB();
const medium = new SharedMemoryBackend();
const kv = new KVStore(medium, 0);
await kv.open(dbName);
await kv.appendValue('log', enc('AAA'));
await kv.appendValue('log', enc('BBB'));
await kv.appendValue('log', enc('CCC'));
expect(dec(await kv.get('log'))).toBe('AAABBBCCC');
await kv.close();
const kv2 = new KVStore(medium, 0);
await kv2.open(dbName);
expect(dec(await kv2.get('log'))).toBe('AAABBBCCC');
await kv2.close();
});
it('append 与 put/delete 混合 + checkpoint 后恢复', async () => {
const dbName = uniqueDB();
const medium = new SharedMemoryBackend();
const kv = new KVStore(medium, 0);
await kv.open(dbName);
await kv.appendValue('log', enc('A'));
await kv.put('meta', enc('M'));
await kv.appendValue('log', enc('B'));
await kv.checkpoint();
await kv.appendValue('log', enc('C'));
await kv.put('meta2', enc('M2'));
await kv.close();
const kv2 = new KVStore(medium, 0);
await kv2.open(dbName);
expect(dec(await kv2.get('log'))).toBe('ABC');
expect(dec(await kv2.get('meta'))).toBe('M');
expect(dec(await kv2.get('meta2'))).toBe('M2');
await kv2.close();
});
it('append 覆盖语义:先 put 后 append 拼接', async () => {
const dbName = uniqueDB();
const medium = new SharedMemoryBackend();
const kv = new KVStore(medium, 0);
await kv.open(dbName);
await kv.put('f', enc('HEAD-'));
await kv.appendValue('f', enc('BODY'));
expect(dec(await kv.get('f'))).toBe('HEAD-BODY');
await kv.close();
const kv2 = new KVStore(medium, 0);
await kv2.open(dbName);
expect(dec(await kv2.get('f'))).toBe('HEAD-BODY');
await kv2.close();
});
it('append 后 delete → 删除生效(索引与恢复一致)', async () => {
const dbName = uniqueDB();
const medium = new SharedMemoryBackend();
const kv = new KVStore(medium, 0);
await kv.open(dbName);
await kv.appendValue('log', enc('X'));
await kv.delete('log');
expect(await kv.exists('log')).toBe(false);
await kv.close();
const kv2 = new KVStore(medium, 0);
await kv2.open(dbName);
expect(await kv2.exists('log')).toBe(false);
await kv2.close();
});
it('日志损坏截断后已确认的 append 数据保留', async () => {
const dbName = uniqueDB();
const medium = new SharedMemoryBackend();
const kv = new KVStore(medium, 0);
await kv.open(dbName);
await kv.appendValue('log', enc('KEEP'));
await kv.close();
// 追加损坏记录
const disk = new SharedMemoryBackend();
await disk.open(dbName);
const log = await disk.read('__kv_log');
const bad = new Uint8Array(encodeLogRecord(999, { x: enc('X') }));
bad[15] ^= 0xff;
const combined = new Uint8Array((log as ArrayBuffer).byteLength + bad.byteLength);
combined.set(new Uint8Array(log as ArrayBuffer), 0);
combined.set(bad, (log as ArrayBuffer).byteLength);
await disk.write('__kv_log', combined.buffer as ArrayBuffer);
const kv2 = new KVStore(medium, 0);
await kv2.open(dbName);
expect(dec(await kv2.get('log'))).toBe('KEEP');
await kv2.close();
});
});