release: v0.4.4 — SSTable 编码修复(UTF-8 字节估算 + u32 长度字段 + v1/v2 双格式兼容),大内容不再崩溃
CI / test (18.x) (push) Successful in 10m3s
CI / test (20.x) (push) Successful in 10m4s
CI / test (22.x) (push) Successful in 9m58s
CI / test (24.x) (push) Successful in 9m56s

This commit is contained in:
thzxx
2026-08-09 21:34:28 +08:00
parent d269bdfb75
commit cff98b0903
21 changed files with 1166 additions and 852 deletions
+18
View File
@@ -2,6 +2,24 @@
All notable changes to MetonaSqlark will be documented in this file. All notable changes to MetonaSqlark will be documented in this file.
## [0.4.4] - 2026-08-09
### Fixed
- **SSTable 编码缺陷导致大内容写入崩溃(P0)** — AriaEngine 保存大段中文内容(如 300KB 笔记)→ close 构建 SSTable 崩溃 → 半写文件 → 下次打开 ARIA_OPEN_ERROR
- **块大小估算用 UTF-16 码元而非 UTF-8 字节** — `computeBlockSize``json.length`(中文 1 字 = 1 码元)估算,实际写入 UTF-8 字节(中文 1 字 = 3 字节)→ 300KB 中文实际 900KB → 缓冲区低估 → `Uint8Array.set` 越界崩溃。改为 TextEncoder 预编码,全部按字节长度计算(切块、块大小、索引块、总大小)
- **valueLen/keyLen 字段 u16 截断** — 长度字段 u16 上限 64KB,大 value 被截断 → 线性格式整体错乱。格式 v2magic "SSTC")升级为 u32 长度字段
- **大 value 块切分** — 单条 value 超过块大小(4096)时独立成块(基于字节估算正确切分)
- **格式兼容** — reader 支持 v1"SSTB"u16)与 v2"SSTC"u32)双格式:旧库 v1 文件仍可读(小 value 场景无缺陷),新写入用 v2,旧库数据不丢失
- 连带受益:中文主键 key、大内容二级索引列(索引 key 含列值)同样支持 u32 长度字段
### Changed
- 测试 950 → **958**52 套件),新增 `tests/v045-hardening.test.ts`8 条:300KB 中文 builder/端到端跨重启、>64KB value、大 value 独立成块、中文 key、v1 兼容、v2 魔数)
- 版本号升至 v0.4.4
---
## [0.4.3] - 2026-08-09 ## [0.4.3] - 2026-08-09
### Fixed ### Fixed
+5 -4
View File
@@ -1,10 +1,10 @@
# MetonaSqlark # MetonaSqlark
<p align="center"> <p align="center">
<img src="https://img.shields.io/badge/version-0.4.3-blue?style=flat-square" alt="version"> <img src="https://img.shields.io/badge/version-0.4.4-blue?style=flat-square" alt="version">
<img src="https://img.shields.io/badge/license-MIT-green?style=flat-square" alt="license"> <img src="https://img.shields.io/badge/license-MIT-green?style=flat-square" alt="license">
<img src="https://img.shields.io/badge/coverage-84.2%25-brightgreen?style=flat-square" alt="coverage"> <img src="https://img.shields.io/badge/coverage-84.2%25-brightgreen?style=flat-square" alt="coverage">
<img src="https://img.shields.io/badge/tests-950%20passed-success?style=flat-square" alt="tests"> <img src="https://img.shields.io/badge/tests-958%20passed-success?style=flat-square" alt="tests">
</p> </p>
> 基于 TypeScript 的**前端关系型数据库**,支持完整 SQL 查询、Query Builder 链式 API、与 **AriaEngine 自研页面式存储引擎**。 > 基于 TypeScript 的**前端关系型数据库**,支持完整 SQL 查询、Query Builder 链式 API、与 **AriaEngine 自研页面式存储引擎**。
@@ -26,10 +26,11 @@
- 🔗 **外键级联** — ON DELETE + ON UPDATECASCADE / SET NULL / RESTRICT)全引擎支持,支持更新主键(v0.4.2) - 🔗 **外键级联** — ON DELETE + ON UPDATECASCADE / SET NULL / RESTRICT)全引擎支持,支持更新主键(v0.4.2)
- 🛡 **崩溃恢复自愈** — 残缺 SSTable 打开自动跳过、`db.repair()` 自愈、`db.clearAll()` 重置、迁移版本持久化(v0.4.2) - 🛡 **崩溃恢复自愈** — 残缺 SSTable 打开自动跳过、`db.repair()` 自愈、`db.clearAll()` 重置、迁移版本持久化(v0.4.2)
- 🧵 **关闭时序与后台任务加固** — 后台 flush/compaction 串行入队(close 排空后才关闭存储,失败显式报告 `ARIA_BACKGROUND_ERROR`)、预加载等待链稳定(compaction 竞态修复)、事务提交先落 WAL 再合并快照(v0.4.3) - 🧵 **关闭时序与后台任务加固** — 后台 flush/compaction 串行入队(close 排空后才关闭存储,失败显式报告 `ARIA_BACKGROUND_ERROR`)、预加载等待链稳定(compaction 竞态修复)、事务提交先落 WAL 再合并快照(v0.4.3)
- 📏 **SSTable 编码修复** — 块大小按 UTF-8 字节精确计算(大段中文内容不再因缓冲区低估崩溃)、长度字段 u32(>64KB value 不截断)、v1/v2 双格式兼容(旧库数据不丢)(v0.4.4)
- 🌲 **RB-Tree 完整实现** — 标准红黑树插入+删除修复,O(log n) 保证 - 🌲 **RB-Tree 完整实现** — 标准红黑树插入+删除修复,O(log n) 保证
-**性能优化** — SSTableReader 二分查找统一、IndexedDB 索引利用、crypto 实例化避免全局状态 -**性能优化** — SSTableReader 二分查找统一、IndexedDB 索引利用、crypto 实例化避免全局状态
- 🌐 **浏览器兼容** — Chrome 80+ / Firefox 80+ / Safari 14+ / Edge 80+ / Node.js 16+ - 🌐 **浏览器兼容** — Chrome 80+ / Firefox 80+ / Safari 14+ / Edge 80+ / Node.js 16+
- 🧪 **950 测试 · 84.2% 覆盖率** — 51 套件,生产级质量保证 - 🧪 **958 测试 · 84.2% 覆盖率** — 52 套件,生产级质量保证
--- ---
@@ -400,7 +401,7 @@ npm run typecheck # 类型检查
| 指标 | 数值 | | 指标 | 数值 |
|------|------| |------|------|
| 测试用例 | 950 | | 测试用例 | 958 |
| 测试套件 | 51 | | 测试套件 | 51 |
| 行覆盖率 | 84.2% | | 行覆盖率 | 84.2% |
| SQL 关键字 | 36 | | SQL 关键字 | 36 |
+113 -97
View File
@@ -34,7 +34,7 @@ class DatabaseError extends Error {
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// 版本 // 版本
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
const VERSION = '0.4.3'; const VERSION = '0.4.4';
/** /**
* metona-sqlark Shared WHERE Matcher 统一的条件匹配逻辑 * metona-sqlark Shared WHERE Matcher 统一的条件匹配逻辑
@@ -2713,7 +2713,15 @@ class BloomFilter {
* *
* 将排序后的 key-value 数据写入 SSTable 格式 * 将排序后的 key-value 数据写入 SSTable 格式
* *
* SSTable 文件布局: * v0.4.4 格式 v2magic "SSTC"修复
* - 块大小估算改用 UTF-8 字节长度TextEncoder 预编码
* 此前用字符串 .lengthUTF-16 码元估算而实际写入 UTF-8 字节
* 中文内容1 3 字节导致缓冲区低估 写入越界崩溃
* - keyLen/valueLen u16 升级为 u32此前 >64KB value 长度被截断
* 线性格式整体错乱
* - value 单条独立成块切分逻辑基于字节估算
*
* SSTable 文件布局 (v2):
* *
* Data Block 0 * Data Block 0
* Data Block 1 * Data Block 1
@@ -2727,58 +2735,56 @@ class BloomFilter {
* - bloom_size (u32) * - bloom_size (u32)
* - bloom_hash_count (u32) * - bloom_hash_count (u32)
* - entry_count (u32) * - entry_count (u32)
* - magic_number (u32, 0x53535442 ="SSTB") * - magic_number (u32, 0x53535443 ="SSTC")
* - checksum (u32) * - checksum (u32)
* *
* 数据块条目: entryCount(u32) + [keyLen(u32) + key + valueLen(u32) + value]
* 索引块条目: [keyLen(u32) + key + blockOffset(u32) + blockSize(u32)]
*/ */
const SSTABLE_MAGIC$1 = 0x53535442; // "SSTB" /** v1 格式魔数("SSTB",u16 长度字段,兼容旧文件读取) */
const SSTABLE_MAGIC_V1 = 0x53535442;
/** v2 格式魔数("SSTC"u32 长度字段 + 字节精确估算,v0.4.4) */
const SSTABLE_MAGIC_V2 = 0x53535443;
const SSTABLE_FOOTER_SIZE = 32; const SSTABLE_FOOTER_SIZE = 32;
// ---------------------------------------------------------------------------
// SSTableBuilder
// ---------------------------------------------------------------------------
class SSTableBuilder { class SSTableBuilder {
constructor(blockSizeLimit = 4096) { constructor(blockSizeLimit = 4096) {
this.entries = []; this.entries = [];
this.currentBlock = [];
this.currentBlockStartKey = '';
this.blockSizeLimit = blockSizeLimit; this.blockSizeLimit = blockSizeLimit;
} }
/** 添加一个 key-value 条目(必须按键排序添加) */ /** 添加一个 key-value 条目(必须按键排序添加) */
add(key, value) { add(key, value) {
if (this.currentBlock.length === 0) {
this.currentBlockStartKey = key;
}
this.currentBlock.push([key, value]);
this.entries.push([key, value]); this.entries.push([key, value]);
// 如果当前 Block 达到大小限制,切割
const estimated = this.estimateBlockSize();
if (estimated >= this.blockSizeLimit && this.currentBlock.length > 1) ;
} }
/** /**
* 构建 SSTable 文件的二进制数据 * 构建 SSTable 文件的二进制数据v2 格式
* 返回 { data: Uint8Array, indexEntries: IndexEntry[], bloomFilter: BloomFilter } * 返回 { data: Uint8Array, indexEntries: IndexEntry[] }
*/ */
build() { build() {
const blocks = this.splitIntoBlocks(); // v0.4.4-fix: 预编码全部条目 — 块大小估算必须基于 UTF-8 字节长度,
// 字符串 .length 是 UTF-16 码元(中文 1 字 3 字节 vs 1 码元)→ 缓冲区低估越界
const encoder = new TextEncoder();
const encoded = this.entries.map(([key, value]) => ({
key,
keyBytes: encoder.encode(key),
valueBytes: encoder.encode(JSON.stringify(value)),
}));
const blocks = this.splitIntoBlocks(encoded);
const bloomFilter = new BloomFilter(this.entries.length); const bloomFilter = new BloomFilter(this.entries.length);
// 预计算总大小 // 预计算总大小(字节)
let totalSize = 0; let totalSize = 0;
const blockOffsets = []; const blockOffsets = [];
for (const block of blocks) { for (const block of blocks) {
blockOffsets.push(totalSize); blockOffsets.push(totalSize);
const blockSize = this.computeBlockSize(block); totalSize += this.computeBlockSize(block);
totalSize += blockSize;
} }
// 索引块 // 索引块(块内最后一个 key 作为索引键)
const indexEntries = []; const indexEntries = [];
for (let i = 0; i < blocks.length; i++) { for (let i = 0; i < blocks.length; i++) {
const block = blocks[i]; const block = blocks[i];
const lastKey = block[block.length - 1][0];
const blockSize = this.computeBlockSize(block);
indexEntries.push({ indexEntries.push({
key: lastKey, key: block[block.length - 1].key,
blockOffset: blockOffsets[i], blockOffset: blockOffsets[i],
blockSize, blockSize: this.computeBlockSize(block),
}); });
} }
const indexBlockSize = this.estimateIndexBlockSize(indexEntries); const indexBlockSize = this.estimateIndexBlockSize(indexEntries);
@@ -2809,7 +2815,7 @@ class SSTableBuilder {
view.setUint32(footerOffset + 12, bloomSize, false); // bloom_size view.setUint32(footerOffset + 12, bloomSize, false); // bloom_size
view.setUint32(footerOffset + 16, bloomFilter.getHashCount(), false); view.setUint32(footerOffset + 16, bloomFilter.getHashCount(), false);
view.setUint32(footerOffset + 20, this.entries.length, false); view.setUint32(footerOffset + 20, this.entries.length, false);
view.setUint32(footerOffset + 24, SSTABLE_MAGIC$1, false); view.setUint32(footerOffset + 24, SSTABLE_MAGIC_V2, false);
view.setUint32(footerOffset + 28, 0, false); // checksum (simplified: 0) view.setUint32(footerOffset + 28, 0, false); // checksum (simplified: 0)
return { return {
sstableData: new Uint8Array(buf), sstableData: new Uint8Array(buf),
@@ -2823,12 +2829,14 @@ class SSTableBuilder {
// ----------------------------------------------------------------------- // -----------------------------------------------------------------------
// 内部 // 内部
// ----------------------------------------------------------------------- // -----------------------------------------------------------------------
splitIntoBlocks() { /** 按 UTF-8 字节大小切分数据块;大 value 单条独立成块 */
splitIntoBlocks(encoded) {
const blocks = []; const blocks = [];
let current = []; let current = [];
for (const entry of this.entries) { for (const entry of encoded) {
current.push(entry); current.push(entry);
if (this.estimateBlockSizeFromEntries(current) >= this.blockSizeLimit && current.length > 1) { // v0.4.4-fix: 基于字节估算;单条超大条目(length===1)独立成块不强行切分
if (this.computeBlockSize(current) >= this.blockSizeLimit && current.length > 1) {
blocks.push(current.slice(0, -1)); blocks.push(current.slice(0, -1));
current = [entry]; current = [entry];
} }
@@ -2837,22 +2845,11 @@ class SSTableBuilder {
blocks.push(current); blocks.push(current);
return blocks; return blocks;
} }
estimateBlockSize() { /** 块字节大小:entryCount(u32) + 每对 [keyLen(u32) + key + valueLen(u32) + value] */
return this.estimateBlockSizeFromEntries(this.currentBlock);
}
estimateBlockSizeFromEntries(entries) {
let size = 0;
for (const [key, value] of entries) {
size += 4 + key.length + JSON.stringify(value).length;
}
return size;
}
computeBlockSize(block) { computeBlockSize(block) {
// entryCount (u32) + 每对: keyLen(u16) + key + valueLen(u16) + value json
let size = 4; let size = 4;
for (const [key, value] of block) { for (const e of block) {
const json = JSON.stringify(value); size += 4 + e.keyBytes.length + 4 + e.valueBytes.length;
size += 2 + key.length + 2 + json.length;
} }
return size; return size;
} }
@@ -2860,32 +2857,30 @@ class SSTableBuilder {
// entry count // entry count
view.setUint32(offset, block.length, false); view.setUint32(offset, block.length, false);
offset += 4; offset += 4;
for (const [key, value] of block) { for (const e of block) {
const encoder = new TextEncoder(); // v0.4.4-fix: 长度字段 u32(此前 u16 截断 >64KB 的 value
const keyBytes = encoder.encode(key); if (e.keyBytes.length > 0xFFFFFFFF || e.valueBytes.length > 0xFFFFFFFF) {
const valueBytes = encoder.encode(JSON.stringify(value)); throw new Error('SSTable entry too large (exceeds u32 length field)');
// key length }
view.setUint16(offset, keyBytes.length, false); view.setUint32(offset, e.keyBytes.length, false);
offset += 2; offset += 4;
// key new Uint8Array(view.buffer).set(e.keyBytes, offset);
new Uint8Array(view.buffer).set(keyBytes, offset); offset += e.keyBytes.length;
offset += keyBytes.length; view.setUint32(offset, e.valueBytes.length, false);
// value length offset += 4;
view.setUint16(offset, valueBytes.length, false); new Uint8Array(view.buffer).set(e.valueBytes, offset);
offset += 2; offset += e.valueBytes.length;
// value
new Uint8Array(view.buffer).set(valueBytes, offset);
offset += valueBytes.length;
// 插入 bloom filter // 插入 bloom filter
bloomFilter.insert(key); bloomFilter.insert(e.key);
} }
return offset; return offset;
} }
estimateIndexBlockSize(entries) { estimateIndexBlockSize(entries) {
// entryCount(u32) + each: keyLen(u16)+key+blockOffset(u32)+blockSize(u32) // entryCount(u32) + each: keyLen(u32)+key+blockOffset(u32)+blockSize(u32)
let size = 4; let size = 4;
const encoder = new TextEncoder();
for (const entry of entries) { for (const entry of entries) {
size += 2 + entry.key.length + 8; size += 4 + encoder.encode(entry.key).byteLength + 8;
} }
return size; return size;
} }
@@ -2895,8 +2890,8 @@ class SSTableBuilder {
for (const entry of entries) { for (const entry of entries) {
const encoder = new TextEncoder(); const encoder = new TextEncoder();
const keyBytes = encoder.encode(entry.key); const keyBytes = encoder.encode(entry.key);
view.setUint16(offset, keyBytes.length, false); view.setUint32(offset, keyBytes.length, false);
offset += 2; offset += 4;
new Uint8Array(view.buffer).set(keyBytes, offset); new Uint8Array(view.buffer).set(keyBytes, offset);
offset += keyBytes.length; offset += keyBytes.length;
view.setUint32(offset, entry.blockOffset, false); view.setUint32(offset, entry.blockOffset, false);
@@ -2911,8 +2906,10 @@ class SSTableBuilder {
/** /**
* AriaEngine SSTable Reader SSTable 二进制数据中读取 * AriaEngine SSTable Reader SSTable 二进制数据中读取
* @module engine/aria/index/sstable * @module engine/aria/index/sstable
*
* v0.4.4: 支持 v1"SSTB"u16 长度字段 v2"SSTC"u32 长度字段双格式
* 旧库 v1 文件仍可读 value 场景无缺陷新写入使用 v2
*/ */
const SSTABLE_MAGIC = 0x53535442; // "SSTB"
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// SSTableReader // SSTableReader
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -2921,11 +2918,20 @@ class SSTableReader {
this.indexEntries = []; this.indexEntries = [];
this.entryCount = 0; this.entryCount = 0;
this.bloomFilter = null; this.bloomFilter = null;
/** 格式版本:1 = u16 长度字段(旧),2 = u32 长度字段(v0.4.4 */
this.format = 2;
this.data = data; this.data = data;
this.view = new DataView(data.buffer, data.byteOffset, data.byteLength); this.view = new DataView(data.buffer, data.byteOffset, data.byteLength);
this.meta = meta; this.meta = meta;
this.parseFooter(); this.parseFooter();
} }
/** 长度字段宽度:v2 = 4 字节 u32v1 = 2 字节 u16 */
lenFieldSize() {
return this.format === 2 ? 4 : 2;
}
readLen(offset) {
return this.format === 2 ? this.view.getUint32(offset, false) : this.view.getUint16(offset, false);
}
// ----------------------------------------------------------------------- // -----------------------------------------------------------------------
// 查询 // 查询
// ----------------------------------------------------------------------- // -----------------------------------------------------------------------
@@ -2939,24 +2945,25 @@ class SSTableReader {
return null; return null;
const entry = this.indexEntries[blockIdx]; const entry = this.indexEntries[blockIdx];
const blockData = this.getBlockData(entry); const blockData = this.getBlockData(entry);
// v0.4.2-fix: 残缺文件(meta 偏移超出实际长度)跳过该块,而非抛 RangeError // v0.4.1-fix: 残缺文件(meta 偏移超出实际长度)跳过该块,而非抛 RangeError
if (!blockData) if (!blockData)
return null; return null;
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength); const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
const lenSize = this.lenFieldSize();
const entryCount = blockView.getUint32(0, false); const entryCount = blockView.getUint32(0, false);
let offset = 4; let offset = 4;
// 顺序扫描 block 内的条目(生产中应二分查找) // 顺序扫描 block 内的条目(生产中应二分查找)
for (let i = 0; i < entryCount; i++) { for (let i = 0; i < entryCount; i++) {
if (offset + 2 > blockData.byteLength) if (offset + lenSize > blockData.byteLength)
break; break;
const keyLen = blockView.getUint16(offset, false); const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + keyLen + 2 > blockData.byteLength) if (offset + keyLen + lenSize > blockData.byteLength)
break; break;
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen)); const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
offset += keyLen; offset += keyLen;
const valLen = blockView.getUint16(offset, false); const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + valLen > blockData.byteLength) if (offset + valLen > blockData.byteLength)
break; break;
const valBytes = blockData.slice(offset, offset + valLen); const valBytes = blockData.slice(offset, offset + valLen);
@@ -2980,26 +2987,27 @@ class SSTableReader {
const endBlockIdx = Math.min(this.indexEntries.length - 1, this.locateBlockLE(endKey)); const endBlockIdx = Math.min(this.indexEntries.length - 1, this.locateBlockLE(endKey));
if (startBlockIdx < 0 || endBlockIdx < 0 || startBlockIdx > endBlockIdx) if (startBlockIdx < 0 || endBlockIdx < 0 || startBlockIdx > endBlockIdx)
return; return;
const lenSize = this.lenFieldSize();
for (let bi = startBlockIdx; bi <= endBlockIdx && bi >= 0; bi++) { for (let bi = startBlockIdx; bi <= endBlockIdx && bi >= 0; bi++) {
const entry = this.indexEntries[bi]; const entry = this.indexEntries[bi];
const blockData = this.getBlockData(entry); const blockData = this.getBlockData(entry);
// v0.4.2-fix: 残缺块跳过(rangeScan 继续后续块,不抛异常) // v0.4.1-fix: 残缺块跳过(rangeScan 继续后续块,不抛异常)
if (!blockData) if (!blockData)
continue; continue;
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength); const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
const blockEntryCount = blockView.getUint32(0, false); const blockEntryCount = blockView.getUint32(0, false);
let offset = 4; let offset = 4;
for (let i = 0; i < blockEntryCount; i++) { for (let i = 0; i < blockEntryCount; i++) {
if (offset + 2 > blockData.byteLength) if (offset + lenSize > blockData.byteLength)
break; break;
const keyLen = blockView.getUint16(offset, false); const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + keyLen + 2 > blockData.byteLength) if (offset + keyLen + lenSize > blockData.byteLength)
break; break;
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen)); const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
offset += keyLen; offset += keyLen;
const valLen = blockView.getUint16(offset, false); const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + valLen > blockData.byteLength) if (offset + valLen > blockData.byteLength)
break; break;
const valBytes = blockData.slice(offset, offset + valLen); const valBytes = blockData.slice(offset, offset + valLen);
@@ -3018,25 +3026,26 @@ class SSTableReader {
} }
/** 扫描所有条目 */ /** 扫描所有条目 */
scanAll(callback) { scanAll(callback) {
const lenSize = this.lenFieldSize();
for (const entry of this.indexEntries) { for (const entry of this.indexEntries) {
const blockData = this.getBlockData(entry); const blockData = this.getBlockData(entry);
// v0.4.2-fix: 残缺块跳过(scanAll 继续后续块,不抛异常) // v0.4.1-fix: 残缺块跳过(scanAll 继续后续块,不抛异常)
if (!blockData) if (!blockData)
continue; continue;
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength); const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
const blockEntryCount = blockView.getUint32(0, false); const blockEntryCount = blockView.getUint32(0, false);
let offset = 4; let offset = 4;
for (let i = 0; i < blockEntryCount; i++) { for (let i = 0; i < blockEntryCount; i++) {
if (offset + 2 > blockData.byteLength) if (offset + lenSize > blockData.byteLength)
break; break;
const keyLen = blockView.getUint16(offset, false); const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + keyLen + 2 > blockData.byteLength) if (offset + keyLen + lenSize > blockData.byteLength)
break; break;
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen)); const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
offset += keyLen; offset += keyLen;
const valLen = blockView.getUint16(offset, false); const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + valLen > blockData.byteLength) if (offset + valLen > blockData.byteLength)
break; break;
const valBytes = blockData.slice(offset, offset + valLen); const valBytes = blockData.slice(offset, offset + valLen);
@@ -3067,10 +3076,16 @@ class SSTableReader {
throw new Error('SSTable too small: missing footer'); throw new Error('SSTable too small: missing footer');
} }
const footerOffset = this.data.byteLength - 32; const footerOffset = this.data.byteLength - 32;
// 验证魔数 // 验证魔数v1 "SSTB" / v2 "SSTC"
const magic = this.view.getUint32(footerOffset + 24, false); const magic = this.view.getUint32(footerOffset + 24, false);
if (magic !== SSTABLE_MAGIC) { if (magic === SSTABLE_MAGIC_V1) {
throw new Error(`Invalid SSTable magic: expected ${SSTABLE_MAGIC}, got ${magic}`); this.format = 1;
}
else if (magic === SSTABLE_MAGIC_V2) {
this.format = 2;
}
else {
throw new Error(`Invalid SSTable magic: expected ${SSTABLE_MAGIC_V1} or ${SSTABLE_MAGIC_V2}, got ${magic}`);
} }
const indexOffset = this.view.getUint32(footerOffset, false); const indexOffset = this.view.getUint32(footerOffset, false);
const indexSize = this.view.getUint32(footerOffset + 4, false); const indexSize = this.view.getUint32(footerOffset + 4, false);
@@ -3078,7 +3093,7 @@ class SSTableReader {
const bloomSize = this.view.getUint32(footerOffset + 12, false); const bloomSize = this.view.getUint32(footerOffset + 12, false);
const bloomHashCount = this.view.getUint32(footerOffset + 16, false); const bloomHashCount = this.view.getUint32(footerOffset + 16, false);
this.entryCount = this.view.getUint32(footerOffset + 20, false); this.entryCount = this.view.getUint32(footerOffset + 20, false);
// v0.4.2-fix: 完整性校验 — 索引块必须完全落在文件内,否则视为残缺文件跳过 // v0.4.1-fix: 完整性校验 — 索引块必须完全落在文件内,否则视为残缺文件跳过
if (indexOffset + 4 > this.data.byteLength || indexOffset + indexSize > this.data.byteLength) { if (indexOffset + 4 > this.data.byteLength || indexOffset + indexSize > this.data.byteLength) {
return; // 残缺文件:无索引块可读,get/rangeScan 均返回空 return; // 残缺文件:无索引块可读,get/rangeScan 均返回空
} }
@@ -3098,13 +3113,14 @@ class SSTableReader {
parseIndexBlock(offset, _size) { parseIndexBlock(offset, _size) {
const entryCount = this.view.getUint32(offset, false); const entryCount = this.view.getUint32(offset, false);
offset += 4; offset += 4;
const lenSize = this.lenFieldSize();
for (let i = 0; i < entryCount; i++) { for (let i = 0; i < entryCount; i++) {
// v0.4.2-fix: 索引条目越界(keyLen/blockOffset/blockSize 超过文件长度)时中止解析, // v0.4.1-fix: 索引条目越界(keyLen/blockOffset/blockSize 超过文件长度)时中止解析,
// 已解析的有效条目仍可用于查询 // 已解析的有效条目仍可用于查询
if (offset + 2 > this.data.byteLength) if (offset + lenSize > this.data.byteLength)
break; break;
const keyLen = this.view.getUint16(offset, false); const keyLen = this.readLen(offset);
offset += 2; offset += lenSize;
if (offset + keyLen + 8 > this.data.byteLength) if (offset + keyLen + 8 > this.data.byteLength)
break; break;
const key = new TextDecoder().decode(this.data.slice(offset, offset + keyLen)); const key = new TextDecoder().decode(this.data.slice(offset, offset + keyLen));
@@ -3120,7 +3136,7 @@ class SSTableReader {
} }
} }
/** /**
* v0.4.2-fix: 获取索引条目对应的数据块 * v0.4.1-fix: 获取索引条目对应的数据块
* 块偏移/大小越界残缺 SSTable时返回 null由调用方跳过而非抛 RangeError * 块偏移/大小越界残缺 SSTable时返回 null由调用方跳过而非抛 RangeError
*/ */
getBlockData(entry) { getBlockData(entry) {
+1 -1
View File
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -118,7 +118,7 @@ interface MetonaPlugin {
/** 销毁 */ /** 销毁 */
destroy(): void; destroy(): void;
} }
declare const VERSION = "0.4.3"; declare const VERSION = "0.4.4";
/** /**
* metona-sqlark Plugin — 插件系统 * metona-sqlark Plugin — 插件系统
+113 -97
View File
@@ -30,7 +30,7 @@ class DatabaseError extends Error {
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// 版本 // 版本
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
const VERSION = '0.4.3'; const VERSION = '0.4.4';
/** /**
* metona-sqlark Shared WHERE Matcher 统一的条件匹配逻辑 * metona-sqlark Shared WHERE Matcher 统一的条件匹配逻辑
@@ -2709,7 +2709,15 @@ class BloomFilter {
* *
* 将排序后的 key-value 数据写入 SSTable 格式 * 将排序后的 key-value 数据写入 SSTable 格式
* *
* SSTable 文件布局: * v0.4.4 格式 v2magic "SSTC"修复
* - 块大小估算改用 UTF-8 字节长度TextEncoder 预编码
* 此前用字符串 .lengthUTF-16 码元估算而实际写入 UTF-8 字节
* 中文内容1 3 字节导致缓冲区低估 写入越界崩溃
* - keyLen/valueLen u16 升级为 u32此前 >64KB value 长度被截断
* 线性格式整体错乱
* - value 单条独立成块切分逻辑基于字节估算
*
* SSTable 文件布局 (v2):
* *
* Data Block 0 * Data Block 0
* Data Block 1 * Data Block 1
@@ -2723,58 +2731,56 @@ class BloomFilter {
* - bloom_size (u32) * - bloom_size (u32)
* - bloom_hash_count (u32) * - bloom_hash_count (u32)
* - entry_count (u32) * - entry_count (u32)
* - magic_number (u32, 0x53535442 ="SSTB") * - magic_number (u32, 0x53535443 ="SSTC")
* - checksum (u32) * - checksum (u32)
* *
* 数据块条目: entryCount(u32) + [keyLen(u32) + key + valueLen(u32) + value]
* 索引块条目: [keyLen(u32) + key + blockOffset(u32) + blockSize(u32)]
*/ */
const SSTABLE_MAGIC$1 = 0x53535442; // "SSTB" /** v1 格式魔数("SSTB",u16 长度字段,兼容旧文件读取) */
const SSTABLE_MAGIC_V1 = 0x53535442;
/** v2 格式魔数("SSTC"u32 长度字段 + 字节精确估算,v0.4.4) */
const SSTABLE_MAGIC_V2 = 0x53535443;
const SSTABLE_FOOTER_SIZE = 32; const SSTABLE_FOOTER_SIZE = 32;
// ---------------------------------------------------------------------------
// SSTableBuilder
// ---------------------------------------------------------------------------
class SSTableBuilder { class SSTableBuilder {
constructor(blockSizeLimit = 4096) { constructor(blockSizeLimit = 4096) {
this.entries = []; this.entries = [];
this.currentBlock = [];
this.currentBlockStartKey = '';
this.blockSizeLimit = blockSizeLimit; this.blockSizeLimit = blockSizeLimit;
} }
/** 添加一个 key-value 条目(必须按键排序添加) */ /** 添加一个 key-value 条目(必须按键排序添加) */
add(key, value) { add(key, value) {
if (this.currentBlock.length === 0) {
this.currentBlockStartKey = key;
}
this.currentBlock.push([key, value]);
this.entries.push([key, value]); this.entries.push([key, value]);
// 如果当前 Block 达到大小限制,切割
const estimated = this.estimateBlockSize();
if (estimated >= this.blockSizeLimit && this.currentBlock.length > 1) ;
} }
/** /**
* 构建 SSTable 文件的二进制数据 * 构建 SSTable 文件的二进制数据v2 格式
* 返回 { data: Uint8Array, indexEntries: IndexEntry[], bloomFilter: BloomFilter } * 返回 { data: Uint8Array, indexEntries: IndexEntry[] }
*/ */
build() { build() {
const blocks = this.splitIntoBlocks(); // v0.4.4-fix: 预编码全部条目 — 块大小估算必须基于 UTF-8 字节长度,
// 字符串 .length 是 UTF-16 码元(中文 1 字 3 字节 vs 1 码元)→ 缓冲区低估越界
const encoder = new TextEncoder();
const encoded = this.entries.map(([key, value]) => ({
key,
keyBytes: encoder.encode(key),
valueBytes: encoder.encode(JSON.stringify(value)),
}));
const blocks = this.splitIntoBlocks(encoded);
const bloomFilter = new BloomFilter(this.entries.length); const bloomFilter = new BloomFilter(this.entries.length);
// 预计算总大小 // 预计算总大小(字节)
let totalSize = 0; let totalSize = 0;
const blockOffsets = []; const blockOffsets = [];
for (const block of blocks) { for (const block of blocks) {
blockOffsets.push(totalSize); blockOffsets.push(totalSize);
const blockSize = this.computeBlockSize(block); totalSize += this.computeBlockSize(block);
totalSize += blockSize;
} }
// 索引块 // 索引块(块内最后一个 key 作为索引键)
const indexEntries = []; const indexEntries = [];
for (let i = 0; i < blocks.length; i++) { for (let i = 0; i < blocks.length; i++) {
const block = blocks[i]; const block = blocks[i];
const lastKey = block[block.length - 1][0];
const blockSize = this.computeBlockSize(block);
indexEntries.push({ indexEntries.push({
key: lastKey, key: block[block.length - 1].key,
blockOffset: blockOffsets[i], blockOffset: blockOffsets[i],
blockSize, blockSize: this.computeBlockSize(block),
}); });
} }
const indexBlockSize = this.estimateIndexBlockSize(indexEntries); const indexBlockSize = this.estimateIndexBlockSize(indexEntries);
@@ -2805,7 +2811,7 @@ class SSTableBuilder {
view.setUint32(footerOffset + 12, bloomSize, false); // bloom_size view.setUint32(footerOffset + 12, bloomSize, false); // bloom_size
view.setUint32(footerOffset + 16, bloomFilter.getHashCount(), false); view.setUint32(footerOffset + 16, bloomFilter.getHashCount(), false);
view.setUint32(footerOffset + 20, this.entries.length, false); view.setUint32(footerOffset + 20, this.entries.length, false);
view.setUint32(footerOffset + 24, SSTABLE_MAGIC$1, false); view.setUint32(footerOffset + 24, SSTABLE_MAGIC_V2, false);
view.setUint32(footerOffset + 28, 0, false); // checksum (simplified: 0) view.setUint32(footerOffset + 28, 0, false); // checksum (simplified: 0)
return { return {
sstableData: new Uint8Array(buf), sstableData: new Uint8Array(buf),
@@ -2819,12 +2825,14 @@ class SSTableBuilder {
// ----------------------------------------------------------------------- // -----------------------------------------------------------------------
// 内部 // 内部
// ----------------------------------------------------------------------- // -----------------------------------------------------------------------
splitIntoBlocks() { /** 按 UTF-8 字节大小切分数据块;大 value 单条独立成块 */
splitIntoBlocks(encoded) {
const blocks = []; const blocks = [];
let current = []; let current = [];
for (const entry of this.entries) { for (const entry of encoded) {
current.push(entry); current.push(entry);
if (this.estimateBlockSizeFromEntries(current) >= this.blockSizeLimit && current.length > 1) { // v0.4.4-fix: 基于字节估算;单条超大条目(length===1)独立成块不强行切分
if (this.computeBlockSize(current) >= this.blockSizeLimit && current.length > 1) {
blocks.push(current.slice(0, -1)); blocks.push(current.slice(0, -1));
current = [entry]; current = [entry];
} }
@@ -2833,22 +2841,11 @@ class SSTableBuilder {
blocks.push(current); blocks.push(current);
return blocks; return blocks;
} }
estimateBlockSize() { /** 块字节大小:entryCount(u32) + 每对 [keyLen(u32) + key + valueLen(u32) + value] */
return this.estimateBlockSizeFromEntries(this.currentBlock);
}
estimateBlockSizeFromEntries(entries) {
let size = 0;
for (const [key, value] of entries) {
size += 4 + key.length + JSON.stringify(value).length;
}
return size;
}
computeBlockSize(block) { computeBlockSize(block) {
// entryCount (u32) + 每对: keyLen(u16) + key + valueLen(u16) + value json
let size = 4; let size = 4;
for (const [key, value] of block) { for (const e of block) {
const json = JSON.stringify(value); size += 4 + e.keyBytes.length + 4 + e.valueBytes.length;
size += 2 + key.length + 2 + json.length;
} }
return size; return size;
} }
@@ -2856,32 +2853,30 @@ class SSTableBuilder {
// entry count // entry count
view.setUint32(offset, block.length, false); view.setUint32(offset, block.length, false);
offset += 4; offset += 4;
for (const [key, value] of block) { for (const e of block) {
const encoder = new TextEncoder(); // v0.4.4-fix: 长度字段 u32(此前 u16 截断 >64KB 的 value
const keyBytes = encoder.encode(key); if (e.keyBytes.length > 0xFFFFFFFF || e.valueBytes.length > 0xFFFFFFFF) {
const valueBytes = encoder.encode(JSON.stringify(value)); throw new Error('SSTable entry too large (exceeds u32 length field)');
// key length }
view.setUint16(offset, keyBytes.length, false); view.setUint32(offset, e.keyBytes.length, false);
offset += 2; offset += 4;
// key new Uint8Array(view.buffer).set(e.keyBytes, offset);
new Uint8Array(view.buffer).set(keyBytes, offset); offset += e.keyBytes.length;
offset += keyBytes.length; view.setUint32(offset, e.valueBytes.length, false);
// value length offset += 4;
view.setUint16(offset, valueBytes.length, false); new Uint8Array(view.buffer).set(e.valueBytes, offset);
offset += 2; offset += e.valueBytes.length;
// value
new Uint8Array(view.buffer).set(valueBytes, offset);
offset += valueBytes.length;
// 插入 bloom filter // 插入 bloom filter
bloomFilter.insert(key); bloomFilter.insert(e.key);
} }
return offset; return offset;
} }
estimateIndexBlockSize(entries) { estimateIndexBlockSize(entries) {
// entryCount(u32) + each: keyLen(u16)+key+blockOffset(u32)+blockSize(u32) // entryCount(u32) + each: keyLen(u32)+key+blockOffset(u32)+blockSize(u32)
let size = 4; let size = 4;
const encoder = new TextEncoder();
for (const entry of entries) { for (const entry of entries) {
size += 2 + entry.key.length + 8; size += 4 + encoder.encode(entry.key).byteLength + 8;
} }
return size; return size;
} }
@@ -2891,8 +2886,8 @@ class SSTableBuilder {
for (const entry of entries) { for (const entry of entries) {
const encoder = new TextEncoder(); const encoder = new TextEncoder();
const keyBytes = encoder.encode(entry.key); const keyBytes = encoder.encode(entry.key);
view.setUint16(offset, keyBytes.length, false); view.setUint32(offset, keyBytes.length, false);
offset += 2; offset += 4;
new Uint8Array(view.buffer).set(keyBytes, offset); new Uint8Array(view.buffer).set(keyBytes, offset);
offset += keyBytes.length; offset += keyBytes.length;
view.setUint32(offset, entry.blockOffset, false); view.setUint32(offset, entry.blockOffset, false);
@@ -2907,8 +2902,10 @@ class SSTableBuilder {
/** /**
* AriaEngine SSTable Reader SSTable 二进制数据中读取 * AriaEngine SSTable Reader SSTable 二进制数据中读取
* @module engine/aria/index/sstable * @module engine/aria/index/sstable
*
* v0.4.4: 支持 v1"SSTB"u16 长度字段 v2"SSTC"u32 长度字段双格式
* 旧库 v1 文件仍可读 value 场景无缺陷新写入使用 v2
*/ */
const SSTABLE_MAGIC = 0x53535442; // "SSTB"
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// SSTableReader // SSTableReader
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -2917,11 +2914,20 @@ class SSTableReader {
this.indexEntries = []; this.indexEntries = [];
this.entryCount = 0; this.entryCount = 0;
this.bloomFilter = null; this.bloomFilter = null;
/** 格式版本:1 = u16 长度字段(旧),2 = u32 长度字段(v0.4.4 */
this.format = 2;
this.data = data; this.data = data;
this.view = new DataView(data.buffer, data.byteOffset, data.byteLength); this.view = new DataView(data.buffer, data.byteOffset, data.byteLength);
this.meta = meta; this.meta = meta;
this.parseFooter(); this.parseFooter();
} }
/** 长度字段宽度:v2 = 4 字节 u32v1 = 2 字节 u16 */
lenFieldSize() {
return this.format === 2 ? 4 : 2;
}
readLen(offset) {
return this.format === 2 ? this.view.getUint32(offset, false) : this.view.getUint16(offset, false);
}
// ----------------------------------------------------------------------- // -----------------------------------------------------------------------
// 查询 // 查询
// ----------------------------------------------------------------------- // -----------------------------------------------------------------------
@@ -2935,24 +2941,25 @@ class SSTableReader {
return null; return null;
const entry = this.indexEntries[blockIdx]; const entry = this.indexEntries[blockIdx];
const blockData = this.getBlockData(entry); const blockData = this.getBlockData(entry);
// v0.4.2-fix: 残缺文件(meta 偏移超出实际长度)跳过该块,而非抛 RangeError // v0.4.1-fix: 残缺文件(meta 偏移超出实际长度)跳过该块,而非抛 RangeError
if (!blockData) if (!blockData)
return null; return null;
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength); const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
const lenSize = this.lenFieldSize();
const entryCount = blockView.getUint32(0, false); const entryCount = blockView.getUint32(0, false);
let offset = 4; let offset = 4;
// 顺序扫描 block 内的条目(生产中应二分查找) // 顺序扫描 block 内的条目(生产中应二分查找)
for (let i = 0; i < entryCount; i++) { for (let i = 0; i < entryCount; i++) {
if (offset + 2 > blockData.byteLength) if (offset + lenSize > blockData.byteLength)
break; break;
const keyLen = blockView.getUint16(offset, false); const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + keyLen + 2 > blockData.byteLength) if (offset + keyLen + lenSize > blockData.byteLength)
break; break;
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen)); const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
offset += keyLen; offset += keyLen;
const valLen = blockView.getUint16(offset, false); const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + valLen > blockData.byteLength) if (offset + valLen > blockData.byteLength)
break; break;
const valBytes = blockData.slice(offset, offset + valLen); const valBytes = blockData.slice(offset, offset + valLen);
@@ -2976,26 +2983,27 @@ class SSTableReader {
const endBlockIdx = Math.min(this.indexEntries.length - 1, this.locateBlockLE(endKey)); const endBlockIdx = Math.min(this.indexEntries.length - 1, this.locateBlockLE(endKey));
if (startBlockIdx < 0 || endBlockIdx < 0 || startBlockIdx > endBlockIdx) if (startBlockIdx < 0 || endBlockIdx < 0 || startBlockIdx > endBlockIdx)
return; return;
const lenSize = this.lenFieldSize();
for (let bi = startBlockIdx; bi <= endBlockIdx && bi >= 0; bi++) { for (let bi = startBlockIdx; bi <= endBlockIdx && bi >= 0; bi++) {
const entry = this.indexEntries[bi]; const entry = this.indexEntries[bi];
const blockData = this.getBlockData(entry); const blockData = this.getBlockData(entry);
// v0.4.2-fix: 残缺块跳过(rangeScan 继续后续块,不抛异常) // v0.4.1-fix: 残缺块跳过(rangeScan 继续后续块,不抛异常)
if (!blockData) if (!blockData)
continue; continue;
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength); const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
const blockEntryCount = blockView.getUint32(0, false); const blockEntryCount = blockView.getUint32(0, false);
let offset = 4; let offset = 4;
for (let i = 0; i < blockEntryCount; i++) { for (let i = 0; i < blockEntryCount; i++) {
if (offset + 2 > blockData.byteLength) if (offset + lenSize > blockData.byteLength)
break; break;
const keyLen = blockView.getUint16(offset, false); const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + keyLen + 2 > blockData.byteLength) if (offset + keyLen + lenSize > blockData.byteLength)
break; break;
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen)); const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
offset += keyLen; offset += keyLen;
const valLen = blockView.getUint16(offset, false); const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + valLen > blockData.byteLength) if (offset + valLen > blockData.byteLength)
break; break;
const valBytes = blockData.slice(offset, offset + valLen); const valBytes = blockData.slice(offset, offset + valLen);
@@ -3014,25 +3022,26 @@ class SSTableReader {
} }
/** 扫描所有条目 */ /** 扫描所有条目 */
scanAll(callback) { scanAll(callback) {
const lenSize = this.lenFieldSize();
for (const entry of this.indexEntries) { for (const entry of this.indexEntries) {
const blockData = this.getBlockData(entry); const blockData = this.getBlockData(entry);
// v0.4.2-fix: 残缺块跳过(scanAll 继续后续块,不抛异常) // v0.4.1-fix: 残缺块跳过(scanAll 继续后续块,不抛异常)
if (!blockData) if (!blockData)
continue; continue;
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength); const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
const blockEntryCount = blockView.getUint32(0, false); const blockEntryCount = blockView.getUint32(0, false);
let offset = 4; let offset = 4;
for (let i = 0; i < blockEntryCount; i++) { for (let i = 0; i < blockEntryCount; i++) {
if (offset + 2 > blockData.byteLength) if (offset + lenSize > blockData.byteLength)
break; break;
const keyLen = blockView.getUint16(offset, false); const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + keyLen + 2 > blockData.byteLength) if (offset + keyLen + lenSize > blockData.byteLength)
break; break;
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen)); const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
offset += keyLen; offset += keyLen;
const valLen = blockView.getUint16(offset, false); const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + valLen > blockData.byteLength) if (offset + valLen > blockData.byteLength)
break; break;
const valBytes = blockData.slice(offset, offset + valLen); const valBytes = blockData.slice(offset, offset + valLen);
@@ -3063,10 +3072,16 @@ class SSTableReader {
throw new Error('SSTable too small: missing footer'); throw new Error('SSTable too small: missing footer');
} }
const footerOffset = this.data.byteLength - 32; const footerOffset = this.data.byteLength - 32;
// 验证魔数 // 验证魔数v1 "SSTB" / v2 "SSTC"
const magic = this.view.getUint32(footerOffset + 24, false); const magic = this.view.getUint32(footerOffset + 24, false);
if (magic !== SSTABLE_MAGIC) { if (magic === SSTABLE_MAGIC_V1) {
throw new Error(`Invalid SSTable magic: expected ${SSTABLE_MAGIC}, got ${magic}`); this.format = 1;
}
else if (magic === SSTABLE_MAGIC_V2) {
this.format = 2;
}
else {
throw new Error(`Invalid SSTable magic: expected ${SSTABLE_MAGIC_V1} or ${SSTABLE_MAGIC_V2}, got ${magic}`);
} }
const indexOffset = this.view.getUint32(footerOffset, false); const indexOffset = this.view.getUint32(footerOffset, false);
const indexSize = this.view.getUint32(footerOffset + 4, false); const indexSize = this.view.getUint32(footerOffset + 4, false);
@@ -3074,7 +3089,7 @@ class SSTableReader {
const bloomSize = this.view.getUint32(footerOffset + 12, false); const bloomSize = this.view.getUint32(footerOffset + 12, false);
const bloomHashCount = this.view.getUint32(footerOffset + 16, false); const bloomHashCount = this.view.getUint32(footerOffset + 16, false);
this.entryCount = this.view.getUint32(footerOffset + 20, false); this.entryCount = this.view.getUint32(footerOffset + 20, false);
// v0.4.2-fix: 完整性校验 — 索引块必须完全落在文件内,否则视为残缺文件跳过 // v0.4.1-fix: 完整性校验 — 索引块必须完全落在文件内,否则视为残缺文件跳过
if (indexOffset + 4 > this.data.byteLength || indexOffset + indexSize > this.data.byteLength) { if (indexOffset + 4 > this.data.byteLength || indexOffset + indexSize > this.data.byteLength) {
return; // 残缺文件:无索引块可读,get/rangeScan 均返回空 return; // 残缺文件:无索引块可读,get/rangeScan 均返回空
} }
@@ -3094,13 +3109,14 @@ class SSTableReader {
parseIndexBlock(offset, _size) { parseIndexBlock(offset, _size) {
const entryCount = this.view.getUint32(offset, false); const entryCount = this.view.getUint32(offset, false);
offset += 4; offset += 4;
const lenSize = this.lenFieldSize();
for (let i = 0; i < entryCount; i++) { for (let i = 0; i < entryCount; i++) {
// v0.4.2-fix: 索引条目越界(keyLen/blockOffset/blockSize 超过文件长度)时中止解析, // v0.4.1-fix: 索引条目越界(keyLen/blockOffset/blockSize 超过文件长度)时中止解析,
// 已解析的有效条目仍可用于查询 // 已解析的有效条目仍可用于查询
if (offset + 2 > this.data.byteLength) if (offset + lenSize > this.data.byteLength)
break; break;
const keyLen = this.view.getUint16(offset, false); const keyLen = this.readLen(offset);
offset += 2; offset += lenSize;
if (offset + keyLen + 8 > this.data.byteLength) if (offset + keyLen + 8 > this.data.byteLength)
break; break;
const key = new TextDecoder().decode(this.data.slice(offset, offset + keyLen)); const key = new TextDecoder().decode(this.data.slice(offset, offset + keyLen));
@@ -3116,7 +3132,7 @@ class SSTableReader {
} }
} }
/** /**
* v0.4.2-fix: 获取索引条目对应的数据块 * v0.4.1-fix: 获取索引条目对应的数据块
* 块偏移/大小越界残缺 SSTable时返回 null由调用方跳过而非抛 RangeError * 块偏移/大小越界残缺 SSTable时返回 null由调用方跳过而非抛 RangeError
*/ */
getBlockData(entry) { getBlockData(entry) {
+1 -1
View File
File diff suppressed because one or more lines are too long
+113 -97
View File
@@ -36,7 +36,7 @@
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// 版本 // 版本
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
const VERSION = '0.4.3'; const VERSION = '0.4.4';
/** /**
* metona-sqlark Shared WHERE Matcher 统一的条件匹配逻辑 * metona-sqlark Shared WHERE Matcher 统一的条件匹配逻辑
@@ -2715,7 +2715,15 @@
* *
* 将排序后的 key-value 数据写入 SSTable 格式 * 将排序后的 key-value 数据写入 SSTable 格式
* *
* SSTable 文件布局: * v0.4.4 格式 v2magic "SSTC"修复
* - 块大小估算改用 UTF-8 字节长度TextEncoder 预编码
* 此前用字符串 .lengthUTF-16 码元估算而实际写入 UTF-8 字节
* 中文内容1 3 字节导致缓冲区低估 写入越界崩溃
* - keyLen/valueLen u16 升级为 u32此前 >64KB value 长度被截断
* 线性格式整体错乱
* - value 单条独立成块切分逻辑基于字节估算
*
* SSTable 文件布局 (v2):
* *
* Data Block 0 * Data Block 0
* Data Block 1 * Data Block 1
@@ -2729,58 +2737,56 @@
* - bloom_size (u32) * - bloom_size (u32)
* - bloom_hash_count (u32) * - bloom_hash_count (u32)
* - entry_count (u32) * - entry_count (u32)
* - magic_number (u32, 0x53535442 ="SSTB") * - magic_number (u32, 0x53535443 ="SSTC")
* - checksum (u32) * - checksum (u32)
* *
* 数据块条目: entryCount(u32) + [keyLen(u32) + key + valueLen(u32) + value]
* 索引块条目: [keyLen(u32) + key + blockOffset(u32) + blockSize(u32)]
*/ */
const SSTABLE_MAGIC$1 = 0x53535442; // "SSTB" /** v1 格式魔数("SSTB",u16 长度字段,兼容旧文件读取) */
const SSTABLE_MAGIC_V1 = 0x53535442;
/** v2 格式魔数("SSTC"u32 长度字段 + 字节精确估算,v0.4.4) */
const SSTABLE_MAGIC_V2 = 0x53535443;
const SSTABLE_FOOTER_SIZE = 32; const SSTABLE_FOOTER_SIZE = 32;
// ---------------------------------------------------------------------------
// SSTableBuilder
// ---------------------------------------------------------------------------
class SSTableBuilder { class SSTableBuilder {
constructor(blockSizeLimit = 4096) { constructor(blockSizeLimit = 4096) {
this.entries = []; this.entries = [];
this.currentBlock = [];
this.currentBlockStartKey = '';
this.blockSizeLimit = blockSizeLimit; this.blockSizeLimit = blockSizeLimit;
} }
/** 添加一个 key-value 条目(必须按键排序添加) */ /** 添加一个 key-value 条目(必须按键排序添加) */
add(key, value) { add(key, value) {
if (this.currentBlock.length === 0) {
this.currentBlockStartKey = key;
}
this.currentBlock.push([key, value]);
this.entries.push([key, value]); this.entries.push([key, value]);
// 如果当前 Block 达到大小限制,切割
const estimated = this.estimateBlockSize();
if (estimated >= this.blockSizeLimit && this.currentBlock.length > 1) ;
} }
/** /**
* 构建 SSTable 文件的二进制数据 * 构建 SSTable 文件的二进制数据v2 格式
* 返回 { data: Uint8Array, indexEntries: IndexEntry[], bloomFilter: BloomFilter } * 返回 { data: Uint8Array, indexEntries: IndexEntry[] }
*/ */
build() { build() {
const blocks = this.splitIntoBlocks(); // v0.4.4-fix: 预编码全部条目 — 块大小估算必须基于 UTF-8 字节长度,
// 字符串 .length 是 UTF-16 码元(中文 1 字 3 字节 vs 1 码元)→ 缓冲区低估越界
const encoder = new TextEncoder();
const encoded = this.entries.map(([key, value]) => ({
key,
keyBytes: encoder.encode(key),
valueBytes: encoder.encode(JSON.stringify(value)),
}));
const blocks = this.splitIntoBlocks(encoded);
const bloomFilter = new BloomFilter(this.entries.length); const bloomFilter = new BloomFilter(this.entries.length);
// 预计算总大小 // 预计算总大小(字节)
let totalSize = 0; let totalSize = 0;
const blockOffsets = []; const blockOffsets = [];
for (const block of blocks) { for (const block of blocks) {
blockOffsets.push(totalSize); blockOffsets.push(totalSize);
const blockSize = this.computeBlockSize(block); totalSize += this.computeBlockSize(block);
totalSize += blockSize;
} }
// 索引块 // 索引块(块内最后一个 key 作为索引键)
const indexEntries = []; const indexEntries = [];
for (let i = 0; i < blocks.length; i++) { for (let i = 0; i < blocks.length; i++) {
const block = blocks[i]; const block = blocks[i];
const lastKey = block[block.length - 1][0];
const blockSize = this.computeBlockSize(block);
indexEntries.push({ indexEntries.push({
key: lastKey, key: block[block.length - 1].key,
blockOffset: blockOffsets[i], blockOffset: blockOffsets[i],
blockSize, blockSize: this.computeBlockSize(block),
}); });
} }
const indexBlockSize = this.estimateIndexBlockSize(indexEntries); const indexBlockSize = this.estimateIndexBlockSize(indexEntries);
@@ -2811,7 +2817,7 @@
view.setUint32(footerOffset + 12, bloomSize, false); // bloom_size view.setUint32(footerOffset + 12, bloomSize, false); // bloom_size
view.setUint32(footerOffset + 16, bloomFilter.getHashCount(), false); view.setUint32(footerOffset + 16, bloomFilter.getHashCount(), false);
view.setUint32(footerOffset + 20, this.entries.length, false); view.setUint32(footerOffset + 20, this.entries.length, false);
view.setUint32(footerOffset + 24, SSTABLE_MAGIC$1, false); view.setUint32(footerOffset + 24, SSTABLE_MAGIC_V2, false);
view.setUint32(footerOffset + 28, 0, false); // checksum (simplified: 0) view.setUint32(footerOffset + 28, 0, false); // checksum (simplified: 0)
return { return {
sstableData: new Uint8Array(buf), sstableData: new Uint8Array(buf),
@@ -2825,12 +2831,14 @@
// ----------------------------------------------------------------------- // -----------------------------------------------------------------------
// 内部 // 内部
// ----------------------------------------------------------------------- // -----------------------------------------------------------------------
splitIntoBlocks() { /** 按 UTF-8 字节大小切分数据块;大 value 单条独立成块 */
splitIntoBlocks(encoded) {
const blocks = []; const blocks = [];
let current = []; let current = [];
for (const entry of this.entries) { for (const entry of encoded) {
current.push(entry); current.push(entry);
if (this.estimateBlockSizeFromEntries(current) >= this.blockSizeLimit && current.length > 1) { // v0.4.4-fix: 基于字节估算;单条超大条目(length===1)独立成块不强行切分
if (this.computeBlockSize(current) >= this.blockSizeLimit && current.length > 1) {
blocks.push(current.slice(0, -1)); blocks.push(current.slice(0, -1));
current = [entry]; current = [entry];
} }
@@ -2839,22 +2847,11 @@
blocks.push(current); blocks.push(current);
return blocks; return blocks;
} }
estimateBlockSize() { /** 块字节大小:entryCount(u32) + 每对 [keyLen(u32) + key + valueLen(u32) + value] */
return this.estimateBlockSizeFromEntries(this.currentBlock);
}
estimateBlockSizeFromEntries(entries) {
let size = 0;
for (const [key, value] of entries) {
size += 4 + key.length + JSON.stringify(value).length;
}
return size;
}
computeBlockSize(block) { computeBlockSize(block) {
// entryCount (u32) + 每对: keyLen(u16) + key + valueLen(u16) + value json
let size = 4; let size = 4;
for (const [key, value] of block) { for (const e of block) {
const json = JSON.stringify(value); size += 4 + e.keyBytes.length + 4 + e.valueBytes.length;
size += 2 + key.length + 2 + json.length;
} }
return size; return size;
} }
@@ -2862,32 +2859,30 @@
// entry count // entry count
view.setUint32(offset, block.length, false); view.setUint32(offset, block.length, false);
offset += 4; offset += 4;
for (const [key, value] of block) { for (const e of block) {
const encoder = new TextEncoder(); // v0.4.4-fix: 长度字段 u32(此前 u16 截断 >64KB 的 value
const keyBytes = encoder.encode(key); if (e.keyBytes.length > 0xFFFFFFFF || e.valueBytes.length > 0xFFFFFFFF) {
const valueBytes = encoder.encode(JSON.stringify(value)); throw new Error('SSTable entry too large (exceeds u32 length field)');
// key length }
view.setUint16(offset, keyBytes.length, false); view.setUint32(offset, e.keyBytes.length, false);
offset += 2; offset += 4;
// key new Uint8Array(view.buffer).set(e.keyBytes, offset);
new Uint8Array(view.buffer).set(keyBytes, offset); offset += e.keyBytes.length;
offset += keyBytes.length; view.setUint32(offset, e.valueBytes.length, false);
// value length offset += 4;
view.setUint16(offset, valueBytes.length, false); new Uint8Array(view.buffer).set(e.valueBytes, offset);
offset += 2; offset += e.valueBytes.length;
// value
new Uint8Array(view.buffer).set(valueBytes, offset);
offset += valueBytes.length;
// 插入 bloom filter // 插入 bloom filter
bloomFilter.insert(key); bloomFilter.insert(e.key);
} }
return offset; return offset;
} }
estimateIndexBlockSize(entries) { estimateIndexBlockSize(entries) {
// entryCount(u32) + each: keyLen(u16)+key+blockOffset(u32)+blockSize(u32) // entryCount(u32) + each: keyLen(u32)+key+blockOffset(u32)+blockSize(u32)
let size = 4; let size = 4;
const encoder = new TextEncoder();
for (const entry of entries) { for (const entry of entries) {
size += 2 + entry.key.length + 8; size += 4 + encoder.encode(entry.key).byteLength + 8;
} }
return size; return size;
} }
@@ -2897,8 +2892,8 @@
for (const entry of entries) { for (const entry of entries) {
const encoder = new TextEncoder(); const encoder = new TextEncoder();
const keyBytes = encoder.encode(entry.key); const keyBytes = encoder.encode(entry.key);
view.setUint16(offset, keyBytes.length, false); view.setUint32(offset, keyBytes.length, false);
offset += 2; offset += 4;
new Uint8Array(view.buffer).set(keyBytes, offset); new Uint8Array(view.buffer).set(keyBytes, offset);
offset += keyBytes.length; offset += keyBytes.length;
view.setUint32(offset, entry.blockOffset, false); view.setUint32(offset, entry.blockOffset, false);
@@ -2913,8 +2908,10 @@
/** /**
* AriaEngine SSTable Reader SSTable 二进制数据中读取 * AriaEngine SSTable Reader SSTable 二进制数据中读取
* @module engine/aria/index/sstable * @module engine/aria/index/sstable
*
* v0.4.4: 支持 v1"SSTB"u16 长度字段 v2"SSTC"u32 长度字段双格式
* 旧库 v1 文件仍可读 value 场景无缺陷新写入使用 v2
*/ */
const SSTABLE_MAGIC = 0x53535442; // "SSTB"
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// SSTableReader // SSTableReader
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -2923,11 +2920,20 @@
this.indexEntries = []; this.indexEntries = [];
this.entryCount = 0; this.entryCount = 0;
this.bloomFilter = null; this.bloomFilter = null;
/** 格式版本:1 = u16 长度字段(旧),2 = u32 长度字段(v0.4.4 */
this.format = 2;
this.data = data; this.data = data;
this.view = new DataView(data.buffer, data.byteOffset, data.byteLength); this.view = new DataView(data.buffer, data.byteOffset, data.byteLength);
this.meta = meta; this.meta = meta;
this.parseFooter(); this.parseFooter();
} }
/** 长度字段宽度:v2 = 4 字节 u32v1 = 2 字节 u16 */
lenFieldSize() {
return this.format === 2 ? 4 : 2;
}
readLen(offset) {
return this.format === 2 ? this.view.getUint32(offset, false) : this.view.getUint16(offset, false);
}
// ----------------------------------------------------------------------- // -----------------------------------------------------------------------
// 查询 // 查询
// ----------------------------------------------------------------------- // -----------------------------------------------------------------------
@@ -2941,24 +2947,25 @@
return null; return null;
const entry = this.indexEntries[blockIdx]; const entry = this.indexEntries[blockIdx];
const blockData = this.getBlockData(entry); const blockData = this.getBlockData(entry);
// v0.4.2-fix: 残缺文件(meta 偏移超出实际长度)跳过该块,而非抛 RangeError // v0.4.1-fix: 残缺文件(meta 偏移超出实际长度)跳过该块,而非抛 RangeError
if (!blockData) if (!blockData)
return null; return null;
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength); const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
const lenSize = this.lenFieldSize();
const entryCount = blockView.getUint32(0, false); const entryCount = blockView.getUint32(0, false);
let offset = 4; let offset = 4;
// 顺序扫描 block 内的条目(生产中应二分查找) // 顺序扫描 block 内的条目(生产中应二分查找)
for (let i = 0; i < entryCount; i++) { for (let i = 0; i < entryCount; i++) {
if (offset + 2 > blockData.byteLength) if (offset + lenSize > blockData.byteLength)
break; break;
const keyLen = blockView.getUint16(offset, false); const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + keyLen + 2 > blockData.byteLength) if (offset + keyLen + lenSize > blockData.byteLength)
break; break;
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen)); const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
offset += keyLen; offset += keyLen;
const valLen = blockView.getUint16(offset, false); const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + valLen > blockData.byteLength) if (offset + valLen > blockData.byteLength)
break; break;
const valBytes = blockData.slice(offset, offset + valLen); const valBytes = blockData.slice(offset, offset + valLen);
@@ -2982,26 +2989,27 @@
const endBlockIdx = Math.min(this.indexEntries.length - 1, this.locateBlockLE(endKey)); const endBlockIdx = Math.min(this.indexEntries.length - 1, this.locateBlockLE(endKey));
if (startBlockIdx < 0 || endBlockIdx < 0 || startBlockIdx > endBlockIdx) if (startBlockIdx < 0 || endBlockIdx < 0 || startBlockIdx > endBlockIdx)
return; return;
const lenSize = this.lenFieldSize();
for (let bi = startBlockIdx; bi <= endBlockIdx && bi >= 0; bi++) { for (let bi = startBlockIdx; bi <= endBlockIdx && bi >= 0; bi++) {
const entry = this.indexEntries[bi]; const entry = this.indexEntries[bi];
const blockData = this.getBlockData(entry); const blockData = this.getBlockData(entry);
// v0.4.2-fix: 残缺块跳过(rangeScan 继续后续块,不抛异常) // v0.4.1-fix: 残缺块跳过(rangeScan 继续后续块,不抛异常)
if (!blockData) if (!blockData)
continue; continue;
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength); const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
const blockEntryCount = blockView.getUint32(0, false); const blockEntryCount = blockView.getUint32(0, false);
let offset = 4; let offset = 4;
for (let i = 0; i < blockEntryCount; i++) { for (let i = 0; i < blockEntryCount; i++) {
if (offset + 2 > blockData.byteLength) if (offset + lenSize > blockData.byteLength)
break; break;
const keyLen = blockView.getUint16(offset, false); const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + keyLen + 2 > blockData.byteLength) if (offset + keyLen + lenSize > blockData.byteLength)
break; break;
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen)); const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
offset += keyLen; offset += keyLen;
const valLen = blockView.getUint16(offset, false); const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + valLen > blockData.byteLength) if (offset + valLen > blockData.byteLength)
break; break;
const valBytes = blockData.slice(offset, offset + valLen); const valBytes = blockData.slice(offset, offset + valLen);
@@ -3020,25 +3028,26 @@
} }
/** 扫描所有条目 */ /** 扫描所有条目 */
scanAll(callback) { scanAll(callback) {
const lenSize = this.lenFieldSize();
for (const entry of this.indexEntries) { for (const entry of this.indexEntries) {
const blockData = this.getBlockData(entry); const blockData = this.getBlockData(entry);
// v0.4.2-fix: 残缺块跳过(scanAll 继续后续块,不抛异常) // v0.4.1-fix: 残缺块跳过(scanAll 继续后续块,不抛异常)
if (!blockData) if (!blockData)
continue; continue;
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength); const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
const blockEntryCount = blockView.getUint32(0, false); const blockEntryCount = blockView.getUint32(0, false);
let offset = 4; let offset = 4;
for (let i = 0; i < blockEntryCount; i++) { for (let i = 0; i < blockEntryCount; i++) {
if (offset + 2 > blockData.byteLength) if (offset + lenSize > blockData.byteLength)
break; break;
const keyLen = blockView.getUint16(offset, false); const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + keyLen + 2 > blockData.byteLength) if (offset + keyLen + lenSize > blockData.byteLength)
break; break;
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen)); const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
offset += keyLen; offset += keyLen;
const valLen = blockView.getUint16(offset, false); const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + valLen > blockData.byteLength) if (offset + valLen > blockData.byteLength)
break; break;
const valBytes = blockData.slice(offset, offset + valLen); const valBytes = blockData.slice(offset, offset + valLen);
@@ -3069,10 +3078,16 @@
throw new Error('SSTable too small: missing footer'); throw new Error('SSTable too small: missing footer');
} }
const footerOffset = this.data.byteLength - 32; const footerOffset = this.data.byteLength - 32;
// 验证魔数 // 验证魔数v1 "SSTB" / v2 "SSTC"
const magic = this.view.getUint32(footerOffset + 24, false); const magic = this.view.getUint32(footerOffset + 24, false);
if (magic !== SSTABLE_MAGIC) { if (magic === SSTABLE_MAGIC_V1) {
throw new Error(`Invalid SSTable magic: expected ${SSTABLE_MAGIC}, got ${magic}`); this.format = 1;
}
else if (magic === SSTABLE_MAGIC_V2) {
this.format = 2;
}
else {
throw new Error(`Invalid SSTable magic: expected ${SSTABLE_MAGIC_V1} or ${SSTABLE_MAGIC_V2}, got ${magic}`);
} }
const indexOffset = this.view.getUint32(footerOffset, false); const indexOffset = this.view.getUint32(footerOffset, false);
const indexSize = this.view.getUint32(footerOffset + 4, false); const indexSize = this.view.getUint32(footerOffset + 4, false);
@@ -3080,7 +3095,7 @@
const bloomSize = this.view.getUint32(footerOffset + 12, false); const bloomSize = this.view.getUint32(footerOffset + 12, false);
const bloomHashCount = this.view.getUint32(footerOffset + 16, false); const bloomHashCount = this.view.getUint32(footerOffset + 16, false);
this.entryCount = this.view.getUint32(footerOffset + 20, false); this.entryCount = this.view.getUint32(footerOffset + 20, false);
// v0.4.2-fix: 完整性校验 — 索引块必须完全落在文件内,否则视为残缺文件跳过 // v0.4.1-fix: 完整性校验 — 索引块必须完全落在文件内,否则视为残缺文件跳过
if (indexOffset + 4 > this.data.byteLength || indexOffset + indexSize > this.data.byteLength) { if (indexOffset + 4 > this.data.byteLength || indexOffset + indexSize > this.data.byteLength) {
return; // 残缺文件:无索引块可读,get/rangeScan 均返回空 return; // 残缺文件:无索引块可读,get/rangeScan 均返回空
} }
@@ -3100,13 +3115,14 @@
parseIndexBlock(offset, _size) { parseIndexBlock(offset, _size) {
const entryCount = this.view.getUint32(offset, false); const entryCount = this.view.getUint32(offset, false);
offset += 4; offset += 4;
const lenSize = this.lenFieldSize();
for (let i = 0; i < entryCount; i++) { for (let i = 0; i < entryCount; i++) {
// v0.4.2-fix: 索引条目越界(keyLen/blockOffset/blockSize 超过文件长度)时中止解析, // v0.4.1-fix: 索引条目越界(keyLen/blockOffset/blockSize 超过文件长度)时中止解析,
// 已解析的有效条目仍可用于查询 // 已解析的有效条目仍可用于查询
if (offset + 2 > this.data.byteLength) if (offset + lenSize > this.data.byteLength)
break; break;
const keyLen = this.view.getUint16(offset, false); const keyLen = this.readLen(offset);
offset += 2; offset += lenSize;
if (offset + keyLen + 8 > this.data.byteLength) if (offset + keyLen + 8 > this.data.byteLength)
break; break;
const key = new TextDecoder().decode(this.data.slice(offset, offset + keyLen)); const key = new TextDecoder().decode(this.data.slice(offset, offset + keyLen));
@@ -3122,7 +3138,7 @@
} }
} }
/** /**
* v0.4.2-fix: 获取索引条目对应的数据块 * v0.4.1-fix: 获取索引条目对应的数据块
* 块偏移/大小越界残缺 SSTable时返回 null由调用方跳过而非抛 RangeError * 块偏移/大小越界残缺 SSTable时返回 null由调用方跳过而非抛 RangeError
*/ */
getBlockData(entry) { getBlockData(entry) {
+1 -1
View File
File diff suppressed because one or more lines are too long
+1 -1
View File
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -1,6 +1,6 @@
{ {
"name": "@metona-team/metona-sqlark", "name": "@metona-team/metona-sqlark",
"version": "0.4.3", "version": "0.4.4",
"description": "Frontend SQL database with in-memory and disk dual-mode storage", "description": "Frontend SQL database with in-memory and disk dual-mode storage",
"type": "module", "type": "module",
"main": "dist/metona-sqlark.cjs", "main": "dist/metona-sqlark.cjs",
+1 -1
View File
@@ -3,7 +3,7 @@
<head> <head>
<meta charset="UTF-8"> <meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>⚡ 性能基准 — MetonaSqlark v0.4.3</title> <title>⚡ 性能基准 — MetonaSqlark v0.4.4</title>
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><rect width='32' height='32' rx='8' fill='%236366f1'/><text x='16' y='22' text-anchor='middle' font-size='20' fill='white'>◈</text></svg>"> <link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><rect width='32' height='32' rx='8' fill='%236366f1'/><text x='16' y='22' text-anchor='middle' font-size='20' fill='white'>◈</text></svg>">
<style> <style>
:root { :root {
+4 -4
View File
@@ -3,7 +3,7 @@
<head> <head>
<meta charset="UTF-8"> <meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>🧪 在线演示 — MetonaSqlark v0.4.3</title> <title>🧪 在线演示 — MetonaSqlark v0.4.4</title>
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><rect width='32' height='32' rx='8' fill='%236366f1'/><text x='16' y='22' text-anchor='middle' font-size='20' fill='white'>◈</text></svg>"> <link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><rect width='32' height='32' rx='8' fill='%236366f1'/><text x='16' y='22' text-anchor='middle' font-size='20' fill='white'>◈</text></svg>">
<style> <style>
:root { :root {
@@ -84,7 +84,7 @@
<a href="demo.html" class="nav-active">演示</a> <a href="demo.html" class="nav-active">演示</a>
<a href="benchmark.html">基准</a> <a href="benchmark.html">基准</a>
</nav> </nav>
<div class="status"><span class="dot" id="engine-dot"></span> <span id="engine-status">Memory</span> 模式 — v0.4.3</div> <div class="status"><span class="dot" id="engine-dot"></span> <span id="engine-status">Memory</span> 模式 — v0.4.4</div>
<button class="btn btn-preset" onclick="switchEngine('memory')" id="btn-memory" style="margin:6px 4px 6px 0;padding:6px 12px;">⚡ Memory</button> <button class="btn btn-preset" onclick="switchEngine('memory')" id="btn-memory" style="margin:6px 4px 6px 0;padding:6px 12px;">⚡ Memory</button>
<button class="btn btn-preset" onclick="switchEngine('aria')" id="btn-aria" style="margin:6px 0;padding:6px 12px;color:#ec4899;border-color:#ec4899;">🌲 Aria</button> <button class="btn btn-preset" onclick="switchEngine('aria')" id="btn-aria" style="margin:6px 0;padding:6px 12px;color:#ec4899;border-color:#ec4899;">🌲 Aria</button>
</header> </header>
@@ -92,7 +92,7 @@
<div class="main"> <div class="main">
<div class="editor-panel"> <div class="editor-panel">
<div class="editor-area"> <div class="editor-area">
<textarea id="sql-input" placeholder="输入 SQL 语句...&#10;&#10;SELECT * FROM users;&#10;INSERT INTO users VALUES ('4', 'Diana', 'diana@test.com', 28);&#10;SELECT u.name, o.amount FROM users u INNER JOIN orders o ON u.id = o.user_id;">-- 🚀 MetonaSqlark v0.4.3 在线演示 <textarea id="sql-input" placeholder="输入 SQL 语句...&#10;&#10;SELECT * FROM users;&#10;INSERT INTO users VALUES ('4', 'Diana', 'diana@test.com', 28);&#10;SELECT u.name, o.amount FROM users u INNER JOIN orders o ON u.id = o.user_id;">-- 🚀 MetonaSqlark v0.4.4 在线演示
-- 已预置 users / orders / products 表数据 -- 已预置 users / orders / products 表数据
-- 新特性: ALTER TABLE · TRUNCATE TABLE · WAL同步 · MVCC · SQL注入防护 -- 新特性: ALTER TABLE · TRUNCATE TABLE · WAL同步 · MVCC · SQL注入防护
@@ -554,7 +554,7 @@ SELECT COUNT(*) as total FROM temp_logs;
-- 清理 -- 清理
DROP TABLE temp_logs;`, DROP TABLE temp_logs;`,
aria: `-- 🌲 AriaEngine 演示 (v0.4.3) aria: `-- 🌲 AriaEngine 演示 (v0.4.4)
-- 点击右上角「🌲 Aria」按钮切换数据库引擎到 AriaEngine -- 点击右上角「🌲 Aria」按钮切换数据库引擎到 AriaEngine
-- 当前数据库即运行在 Aria 引擎上(LSM-Tree · WAL 崩溃恢复 · MVCC · BloomFilter -- 当前数据库即运行在 Aria 引擎上(LSM-Tree · WAL 崩溃恢复 · MVCC · BloomFilter
-- 基础 CRUD 与 Memory 引擎完全兼容 -- 基础 CRUD 与 Memory 引擎完全兼容
+3 -2
View File
@@ -3,7 +3,7 @@
<head> <head>
<meta charset="UTF-8"> <meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>📖 API 文档 — MetonaSqlark v0.4.3</title> <title>📖 API 文档 — MetonaSqlark v0.4.4</title>
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><rect width='32' height='32' rx='8' fill='%236366f1'/><text x='16' y='22' text-anchor='middle' font-size='20' fill='white'>◈</text></svg>"> <link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><rect width='32' height='32' rx='8' fill='%236366f1'/><text x='16' y='22' text-anchor='middle' font-size='20' fill='white'>◈</text></svg>">
<style> <style>
:root { :root {
@@ -762,7 +762,8 @@ db.<span class="f">broadcastChange</span>(<span class="s">'users'</span>);</pre>
<strong>v0.2.4 生产级</strong> — 二级索引 · MVCC · BloomFilter · WAL CRC全同步 · AES-GCM加密 · Savepoint · EXPLAIN · ANALYZE · REINDEX · VACUUM · BufferPool · 零死代码。<br> <strong>v0.2.4 生产级</strong> — 二级索引 · MVCC · BloomFilter · WAL CRC全同步 · AES-GCM加密 · Savepoint · EXPLAIN · ANALYZE · REINDEX · VACUUM · BufferPool · 零死代码。<br>
<strong>v0.3.2 表达式与并发</strong> — WAL full模式真正同步 · MVCC接入读写路径 · SSTableReader二分查找统一 · crypto实例化 · IndexedDB索引利用 · compactLevel public接口 · WAL大小阈值自动checkpoint · SQL注入防护 · ALTER TABLE · TRUNCATE TABLE · 多标签页同步 · IDB schema持久化。<br> <strong>v0.3.2 表达式与并发</strong> — WAL full模式真正同步 · MVCC接入读写路径 · SSTableReader二分查找统一 · crypto实例化 · IndexedDB索引利用 · compactLevel public接口 · WAL大小阈值自动checkpoint · SQL注入防护 · ALTER TABLE · TRUNCATE TABLE · 多标签页同步 · IDB schema持久化。<br>
<strong>v0.4.2 生产就绪与崩溃自愈</strong> — 残缺 SSTable 打开自动跳过(不删库)· WAL 记录与计数原子写入 + 按 key 扫描恢复 · 事务进行中 checkpoint 不截断 WAL · 二级索引跨重启自动恢复 · ALTER TABLE / 事务内 DDL 全引擎持久化 · ON UPDATE 外键级联(含更新主键)· OPFS schema 持久化(空表/索引完整保留)· `repair()` / `clearAll()` 统一自愈接口 · 迁移版本持久化到库内。<br> <strong>v0.4.2 生产就绪与崩溃自愈</strong> — 残缺 SSTable 打开自动跳过(不删库)· WAL 记录与计数原子写入 + 按 key 扫描恢复 · 事务进行中 checkpoint 不截断 WAL · 二级索引跨重启自动恢复 · ALTER TABLE / 事务内 DDL 全引擎持久化 · ON UPDATE 外键级联(含更新主键)· OPFS schema 持久化(空表/索引完整保留)· `repair()` / `clearAll()` 统一自愈接口 · 迁移版本持久化到库内。<br>
<strong>v0.4.3 关闭时序与后台任务加固</strong> — 后台 flush/compaction 不再使用 setTimeout 延迟(close 排空全部任务后才关闭存储,杜绝"backend 关闭后写存储/重开污染")· 后台失败在 `flush()`/`close()` 显式报告(`ARIA_BACKGROUND_ERROR`,不静默吞错)· 预加载等待链稳定(修复 compaction 竞态跳块丢数据)· 事务提交先落 WAL 再合并快照(崩溃一致)· OPFS 写操作串行队列 + close 等待 · 950 测试 51 套件。</p> <strong>v0.4.3 关闭时序与后台任务加固</strong> — 后台 flush/compaction 不再使用 setTimeout 延迟(close 排空全部任务后才关闭存储,杜绝"backend 关闭后写存储/重开污染")· 后台失败在 `flush()`/`close()` 显式报告(`ARIA_BACKGROUND_ERROR`,不静默吞错)· 预加载等待链稳定(修复 compaction 竞态跳块丢数据)· 事务提交先落 WAL 再合并快照(崩溃一致)· OPFS 写操作串行队列 + close 等待<br>
<strong>v0.4.4 SSTable 编码修复</strong> — 大段中文内容(如 300KB 笔记)写入 AriaEngine 不再崩溃:块大小估算改 UTF-8 字节精确计算(修复中文 3 字节 vs 1 码元导致的缓冲区低估越界)· 长度字段 u16 → u32(修复 >64KB value 截断)· 大 value 独立成块 · 格式 v2"SSTC")与 v1"SSTB")双格式兼容(旧库数据不丢)· 中文主键 / 大内容索引列同步支持 · 958 测试 52 套件。</p>
<h3>存储模式对比</h3> <h3>存储模式对比</h3>
<table> <table>
+3 -3
View File
@@ -153,7 +153,7 @@
<!-- Hero --> <!-- Hero -->
<section class="hero"> <section class="hero">
<div class="container"> <div class="container">
<div class="badge" style="margin-bottom:24px;"><span class="dot"></span> v0.4.3 关闭时序与后台任务加固 — 950测试 51套件 · close排空后台任务 · 失败不吞错 · compaction竞态修复 · 提交先WAL · 零回归</div> <div class="badge" style="margin-bottom:24px;"><span class="dot"></span> v0.4.4 SSTable 编码修复 — 958测试 52套件 · 大内容(300KB中文)不再崩溃 · u32长度字段 · 双格式兼容 · 零回归</div>
<h1>前端的 <span class="gradient-text">SQL 数据库</span></h1> <h1>前端的 <span class="gradient-text">SQL 数据库</span></h1>
<p>TypeScript 原生构建,5 种存储引擎,支持完整 SQL 查询。<br>零运行时依赖,开箱即用。AriaEngine 自研引擎:LSM-Tree + WAL 同步 + MVCC。</p> <p>TypeScript 原生构建,5 种存储引擎,支持完整 SQL 查询。<br>零运行时依赖,开箱即用。AriaEngine 自研引擎:LSM-Tree + WAL 同步 + MVCC。</p>
<div class="actions"> <div class="actions">
@@ -243,7 +243,7 @@ npm install @metona-team/metona-sqlark
</div> </div>
<div class="feature-card"> <div class="feature-card">
<div class="icon">🌲</div> <div class="icon">🌲</div>
<h3>AriaEngine <span style="font-size:0.65rem;color:var(--accent);vertical-align:super;">v0.4.3</span></h3> <h3>AriaEngine <span style="font-size:0.65rem;color:var(--accent);vertical-align:super;">v0.4.4</span></h3>
<p>自研 LSM-Tree 页面式存储引擎。MemTable 红黑树 + 多级 SSTable、Bloom Filter 快速判存、WAL 原子写入 full模式真正同步、MVCC 快照隔离、崩溃恢复自动跳过残缺 SSTable、二级索引跨重启恢复、ON UPDATE/DELETE 外键级联。</p> <p>自研 LSM-Tree 页面式存储引擎。MemTable 红黑树 + 多级 SSTable、Bloom Filter 快速判存、WAL 原子写入 full模式真正同步、MVCC 快照隔离、崩溃恢复自动跳过残缺 SSTable、二级索引跨重启恢复、ON UPDATE/DELETE 外键级联。</p>
</div> </div>
<div class="feature-card"> <div class="feature-card">
@@ -404,7 +404,7 @@ npm install @metona-team/metona-sqlark
<p>MetonaSqlark 的核心指标</p> <p>MetonaSqlark 的核心指标</p>
</div> </div>
<div class="stats"> <div class="stats">
<div class="stat-card"><div class="num">950</div><div class="label">测试用例</div></div> <div class="stat-card"><div class="num">958</div><div class="label">测试用例</div></div>
<div class="stat-card"><div class="num">84.2%</div><div class="label">行覆盖率</div></div> <div class="stat-card"><div class="num">84.2%</div><div class="label">行覆盖率</div></div>
<div class="stat-card"><div class="num">~27KB</div><div class="label">gzip 体积</div></div> <div class="stat-card"><div class="num">~27KB</div><div class="label">gzip 体积</div></div>
<div class="stat-card"><div class="num">5</div><div class="label">存储引擎</div></div> <div class="stat-card"><div class="num">5</div><div class="label">存储引擎</div></div>
+1 -1
View File
@@ -214,4 +214,4 @@ export class DatabaseError extends Error {
// 版本 // 版本
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
export const VERSION = '0.4.3'; export const VERSION = '0.4.4';
+53 -32
View File
@@ -1,12 +1,14 @@
/** /**
* AriaEngine SSTable Reader — 从 SSTable 二进制数据中读取 * AriaEngine SSTable Reader — 从 SSTable 二进制数据中读取
* @module engine/aria/index/sstable * @module engine/aria/index/sstable
*
* v0.4.4: 支持 v1"SSTB"u16 长度字段)与 v2"SSTC"u32 长度字段)双格式,
* 旧库 v1 文件仍可读(小 value 场景无缺陷),新写入使用 v2。
*/ */
import type { IndexEntry, SSTableMeta } from '../types'; import type { IndexEntry, SSTableMeta } from '../types';
import { BloomFilter } from './bloom'; import { BloomFilter } from './bloom';
import { SSTABLE_MAGIC_V1, SSTABLE_MAGIC_V2 } from './sstable_builder';
const SSTABLE_MAGIC = 0x53535442; // "SSTB"
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// SSTableReader // SSTableReader
@@ -19,6 +21,8 @@ export class SSTableReader {
private entryCount = 0; private entryCount = 0;
private meta: SSTableMeta; private meta: SSTableMeta;
private bloomFilter: BloomFilter | null = null; private bloomFilter: BloomFilter | null = null;
/** 格式版本:1 = u16 长度字段(旧),2 = u32 长度字段(v0.4.4 */
private format: 1 | 2 = 2;
constructor(data: Uint8Array, meta: SSTableMeta) { constructor(data: Uint8Array, meta: SSTableMeta) {
this.data = data; this.data = data;
@@ -27,6 +31,15 @@ export class SSTableReader {
this.parseFooter(); this.parseFooter();
} }
/** 长度字段宽度:v2 = 4 字节 u32v1 = 2 字节 u16 */
private lenFieldSize(): number {
return this.format === 2 ? 4 : 2;
}
private readLen(offset: number): number {
return this.format === 2 ? this.view.getUint32(offset, false) : this.view.getUint16(offset, false);
}
// ----------------------------------------------------------------------- // -----------------------------------------------------------------------
// 查询 // 查询
// ----------------------------------------------------------------------- // -----------------------------------------------------------------------
@@ -41,23 +54,24 @@ export class SSTableReader {
const entry = this.indexEntries[blockIdx]; const entry = this.indexEntries[blockIdx];
const blockData = this.getBlockData(entry); const blockData = this.getBlockData(entry);
// v0.4.2-fix: 残缺文件(meta 偏移超出实际长度)跳过该块,而非抛 RangeError // v0.4.1-fix: 残缺文件(meta 偏移超出实际长度)跳过该块,而非抛 RangeError
if (!blockData) return null; if (!blockData) return null;
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength); const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
const lenSize = this.lenFieldSize();
const entryCount = blockView.getUint32(0, false); const entryCount = blockView.getUint32(0, false);
let offset = 4; let offset = 4;
// 顺序扫描 block 内的条目(生产中应二分查找) // 顺序扫描 block 内的条目(生产中应二分查找)
for (let i = 0; i < entryCount; i++) { for (let i = 0; i < entryCount; i++) {
if (offset + 2 > blockData.byteLength) break; if (offset + lenSize > blockData.byteLength) break;
const keyLen = blockView.getUint16(offset, false); const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + keyLen + 2 > blockData.byteLength) break; if (offset + keyLen + lenSize > blockData.byteLength) break;
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen)); const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
offset += keyLen; offset += keyLen;
const valLen = blockView.getUint16(offset, false); const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + valLen > blockData.byteLength) break; if (offset + valLen > blockData.byteLength) break;
const valBytes = blockData.slice(offset, offset + valLen); const valBytes = blockData.slice(offset, offset + valLen);
offset += valLen; offset += valLen;
@@ -85,10 +99,11 @@ export class SSTableReader {
const endBlockIdx = Math.min(this.indexEntries.length - 1, this.locateBlockLE(endKey)); const endBlockIdx = Math.min(this.indexEntries.length - 1, this.locateBlockLE(endKey));
if (startBlockIdx < 0 || endBlockIdx < 0 || startBlockIdx > endBlockIdx) return; if (startBlockIdx < 0 || endBlockIdx < 0 || startBlockIdx > endBlockIdx) return;
const lenSize = this.lenFieldSize();
for (let bi = startBlockIdx; bi <= endBlockIdx && bi >= 0; bi++) { for (let bi = startBlockIdx; bi <= endBlockIdx && bi >= 0; bi++) {
const entry = this.indexEntries[bi]; const entry = this.indexEntries[bi];
const blockData = this.getBlockData(entry); const blockData = this.getBlockData(entry);
// v0.4.2-fix: 残缺块跳过(rangeScan 继续后续块,不抛异常) // v0.4.1-fix: 残缺块跳过(rangeScan 继续后续块,不抛异常)
if (!blockData) continue; if (!blockData) continue;
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength); const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
@@ -96,14 +111,14 @@ export class SSTableReader {
let offset = 4; let offset = 4;
for (let i = 0; i < blockEntryCount; i++) { for (let i = 0; i < blockEntryCount; i++) {
if (offset + 2 > blockData.byteLength) break; if (offset + lenSize > blockData.byteLength) break;
const keyLen = blockView.getUint16(offset, false); const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + keyLen + 2 > blockData.byteLength) break; if (offset + keyLen + lenSize > blockData.byteLength) break;
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen)); const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
offset += keyLen; offset += keyLen;
const valLen = blockView.getUint16(offset, false); const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + valLen > blockData.byteLength) break; if (offset + valLen > blockData.byteLength) break;
const valBytes = blockData.slice(offset, offset + valLen); const valBytes = blockData.slice(offset, offset + valLen);
offset += valLen; offset += valLen;
@@ -122,9 +137,10 @@ export class SSTableReader {
/** 扫描所有条目 */ /** 扫描所有条目 */
scanAll(callback: (key: string, value: Record<string, unknown>) => void): void { scanAll(callback: (key: string, value: Record<string, unknown>) => void): void {
const lenSize = this.lenFieldSize();
for (const entry of this.indexEntries) { for (const entry of this.indexEntries) {
const blockData = this.getBlockData(entry); const blockData = this.getBlockData(entry);
// v0.4.2-fix: 残缺块跳过(scanAll 继续后续块,不抛异常) // v0.4.1-fix: 残缺块跳过(scanAll 继续后续块,不抛异常)
if (!blockData) continue; if (!blockData) continue;
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength); const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
@@ -132,14 +148,14 @@ export class SSTableReader {
let offset = 4; let offset = 4;
for (let i = 0; i < blockEntryCount; i++) { for (let i = 0; i < blockEntryCount; i++) {
if (offset + 2 > blockData.byteLength) break; if (offset + lenSize > blockData.byteLength) break;
const keyLen = blockView.getUint16(offset, false); const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + keyLen + 2 > blockData.byteLength) break; if (offset + keyLen + lenSize > blockData.byteLength) break;
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen)); const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
offset += keyLen; offset += keyLen;
const valLen = blockView.getUint16(offset, false); const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
offset += 2; offset += lenSize;
if (offset + valLen > blockData.byteLength) break; if (offset + valLen > blockData.byteLength) break;
const valBytes = blockData.slice(offset, offset + valLen); const valBytes = blockData.slice(offset, offset + valLen);
offset += valLen; offset += valLen;
@@ -175,10 +191,14 @@ export class SSTableReader {
const footerOffset = this.data.byteLength - 32; const footerOffset = this.data.byteLength - 32;
// 验证魔数 // 验证魔数v1 "SSTB" / v2 "SSTC"
const magic = this.view.getUint32(footerOffset + 24, false); const magic = this.view.getUint32(footerOffset + 24, false);
if (magic !== SSTABLE_MAGIC) { if (magic === SSTABLE_MAGIC_V1) {
throw new Error(`Invalid SSTable magic: expected ${SSTABLE_MAGIC}, got ${magic}`); this.format = 1;
} else if (magic === SSTABLE_MAGIC_V2) {
this.format = 2;
} else {
throw new Error(`Invalid SSTable magic: expected ${SSTABLE_MAGIC_V1} or ${SSTABLE_MAGIC_V2}, got ${magic}`);
} }
const indexOffset = this.view.getUint32(footerOffset, false); const indexOffset = this.view.getUint32(footerOffset, false);
@@ -188,7 +208,7 @@ export class SSTableReader {
const bloomHashCount = this.view.getUint32(footerOffset + 16, false); const bloomHashCount = this.view.getUint32(footerOffset + 16, false);
this.entryCount = this.view.getUint32(footerOffset + 20, false); this.entryCount = this.view.getUint32(footerOffset + 20, false);
// v0.4.2-fix: 完整性校验 — 索引块必须完全落在文件内,否则视为残缺文件跳过 // v0.4.1-fix: 完整性校验 — 索引块必须完全落在文件内,否则视为残缺文件跳过
if (indexOffset + 4 > this.data.byteLength || indexOffset + indexSize > this.data.byteLength) { if (indexOffset + 4 > this.data.byteLength || indexOffset + indexSize > this.data.byteLength) {
return; // 残缺文件:无索引块可读,get/rangeScan 均返回空 return; // 残缺文件:无索引块可读,get/rangeScan 均返回空
} }
@@ -210,13 +230,14 @@ export class SSTableReader {
private parseIndexBlock(offset: number, _size: number): void { private parseIndexBlock(offset: number, _size: number): void {
const entryCount = this.view.getUint32(offset, false); const entryCount = this.view.getUint32(offset, false);
offset += 4; offset += 4;
const lenSize = this.lenFieldSize();
for (let i = 0; i < entryCount; i++) { for (let i = 0; i < entryCount; i++) {
// v0.4.2-fix: 索引条目越界(keyLen/blockOffset/blockSize 超过文件长度)时中止解析, // v0.4.1-fix: 索引条目越界(keyLen/blockOffset/blockSize 超过文件长度)时中止解析,
// 已解析的有效条目仍可用于查询 // 已解析的有效条目仍可用于查询
if (offset + 2 > this.data.byteLength) break; if (offset + lenSize > this.data.byteLength) break;
const keyLen = this.view.getUint16(offset, false); const keyLen = this.readLen(offset);
offset += 2; offset += lenSize;
if (offset + keyLen + 8 > this.data.byteLength) break; if (offset + keyLen + 8 > this.data.byteLength) break;
const key = new TextDecoder().decode(this.data.slice(offset, offset + keyLen)); const key = new TextDecoder().decode(this.data.slice(offset, offset + keyLen));
offset += keyLen; offset += keyLen;
@@ -233,7 +254,7 @@ export class SSTableReader {
} }
/** /**
* v0.4.2-fix: 获取索引条目对应的数据块。 * v0.4.1-fix: 获取索引条目对应的数据块。
* 块偏移/大小越界(残缺 SSTable)时返回 null,由调用方跳过而非抛 RangeError。 * 块偏移/大小越界(残缺 SSTable)时返回 null,由调用方跳过而非抛 RangeError。
*/ */
private getBlockData(entry: IndexEntry): Uint8Array | null { private getBlockData(entry: IndexEntry): Uint8Array | null {
+74 -77
View File
@@ -4,7 +4,15 @@
* *
* 将排序后的 key-value 数据写入 SSTable 格式。 * 将排序后的 key-value 数据写入 SSTable 格式。
* *
* SSTable 文件布局: * v0.4.4 格式 v2magic "SSTC")修复:
* - 块大小估算改用 UTF-8 字节长度(TextEncoder 预编码),
* 此前用字符串 .length(UTF-16 码元)估算而实际写入 UTF-8 字节,
* 中文内容(1 字 3 字节)导致缓冲区低估 → 写入越界崩溃
* - keyLen/valueLen 从 u16 升级为 u32(此前 >64KB 的 value 长度被截断,
* 线性格式整体错乱)
* - 大 value 单条独立成块(切分逻辑基于字节估算)
*
* SSTable 文件布局 (v2):
* ┌──────────────────────────────────────────────┐ * ┌──────────────────────────────────────────────┐
* │ Data Block 0 │ * │ Data Block 0 │
* │ Data Block 1 │ * │ Data Block 1 │
@@ -18,25 +26,34 @@
* │ - bloom_size (u32) │ * │ - bloom_size (u32) │
* │ - bloom_hash_count (u32) │ * │ - bloom_hash_count (u32) │
* │ - entry_count (u32) │ * │ - entry_count (u32) │
* │ - magic_number (u32, 0x53535442 ="SSTB")│ * │ - magic_number (u32, 0x53535443 ="SSTC")│
* │ - checksum (u32) │ * │ - checksum (u32) │
* └──────────────────────────────────────────────┘ * └──────────────────────────────────────────────┘
* 数据块条目: entryCount(u32) + [keyLen(u32) + key + valueLen(u32) + value]
* 索引块条目: [keyLen(u32) + key + blockOffset(u32) + blockSize(u32)]
*/ */
import { BloomFilter } from './bloom'; import { BloomFilter } from './bloom';
import type { IndexEntry } from '../types'; import type { IndexEntry } from '../types';
const SSTABLE_MAGIC = 0x53535442; // "SSTB" /** v1 格式魔数("SSTB",u16 长度字段,兼容旧文件读取) */
export const SSTABLE_MAGIC_V1 = 0x53535442;
/** v2 格式魔数("SSTC"u32 长度字段 + 字节精确估算,v0.4.4) */
export const SSTABLE_MAGIC_V2 = 0x53535443;
const SSTABLE_FOOTER_SIZE = 32; const SSTABLE_FOOTER_SIZE = 32;
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// SSTableBuilder // SSTableBuilder
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
interface EncodedEntry {
key: string;
keyBytes: Uint8Array;
valueBytes: Uint8Array;
}
export class SSTableBuilder { export class SSTableBuilder {
private entries: [string, Record<string, unknown>][] = []; private entries: [string, Record<string, unknown>][] = [];
private currentBlock: [string, Record<string, unknown>][] = [];
private currentBlockStartKey = '';
private blockSizeLimit: number; private blockSizeLimit: number;
constructor(blockSizeLimit: number = 4096) { constructor(blockSizeLimit: number = 4096) {
@@ -45,48 +62,42 @@ export class SSTableBuilder {
/** 添加一个 key-value 条目(必须按键排序添加) */ /** 添加一个 key-value 条目(必须按键排序添加) */
add(key: string, value: Record<string, unknown>): void { add(key: string, value: Record<string, unknown>): void {
if (this.currentBlock.length === 0) {
this.currentBlockStartKey = key;
}
this.currentBlock.push([key, value]);
this.entries.push([key, value]); this.entries.push([key, value]);
// 如果当前 Block 达到大小限制,切割
const estimated = this.estimateBlockSize();
if (estimated >= this.blockSizeLimit && this.currentBlock.length > 1) {
// 当前 block 结束(不在这里切割,在 build 时统一处理)
}
} }
/** /**
* 构建 SSTable 文件的二进制数据。 * 构建 SSTable 文件的二进制数据v2 格式)
* 返回 { data: Uint8Array, indexEntries: IndexEntry[], bloomFilter: BloomFilter } * 返回 { data: Uint8Array, indexEntries: IndexEntry[] }
*/ */
build(): { sstableData: Uint8Array; indexEntries: IndexEntry[] } { build(): { sstableData: Uint8Array; indexEntries: IndexEntry[] } {
const blocks = this.splitIntoBlocks(); // v0.4.4-fix: 预编码全部条目 — 块大小估算必须基于 UTF-8 字节长度,
// 字符串 .length 是 UTF-16 码元(中文 1 字 3 字节 vs 1 码元)→ 缓冲区低估越界
const encoder = new TextEncoder();
const encoded: EncodedEntry[] = this.entries.map(([key, value]) => ({
key,
keyBytes: encoder.encode(key),
valueBytes: encoder.encode(JSON.stringify(value)),
}));
const blocks = this.splitIntoBlocks(encoded);
const bloomFilter = new BloomFilter(this.entries.length); const bloomFilter = new BloomFilter(this.entries.length);
// 预计算总大小 // 预计算总大小(字节)
let totalSize = 0; let totalSize = 0;
const blockOffsets: number[] = []; const blockOffsets: number[] = [];
for (const block of blocks) { for (const block of blocks) {
blockOffsets.push(totalSize); blockOffsets.push(totalSize);
const blockSize = this.computeBlockSize(block); totalSize += this.computeBlockSize(block);
totalSize += blockSize;
} }
// 索引块 // 索引块(块内最后一个 key 作为索引键)
const indexEntries: IndexEntry[] = []; const indexEntries: IndexEntry[] = [];
for (let i = 0; i < blocks.length; i++) { for (let i = 0; i < blocks.length; i++) {
const block = blocks[i]; const block = blocks[i];
const lastKey = block[block.length - 1][0];
const blockSize = this.computeBlockSize(block);
indexEntries.push({ indexEntries.push({
key: lastKey, key: block[block.length - 1].key,
blockOffset: blockOffsets[i], blockOffset: blockOffsets[i],
blockSize, blockSize: this.computeBlockSize(block),
}); });
} }
@@ -119,14 +130,14 @@ export class SSTableBuilder {
// ---- Footer ---- // ---- Footer ----
const footerOffset = offset; const footerOffset = offset;
view.setUint32(footerOffset, indexOffset, false); // index_offset view.setUint32(footerOffset, indexOffset, false); // index_offset
view.setUint32(footerOffset + 4, indexBlockSize, false); // index_size view.setUint32(footerOffset + 4, indexBlockSize, false); // index_size
view.setUint32(footerOffset + 8, bloomOffset, false); // bloom_offset view.setUint32(footerOffset + 8, bloomOffset, false); // bloom_offset
view.setUint32(footerOffset + 12, bloomSize, false); // bloom_size view.setUint32(footerOffset + 12, bloomSize, false); // bloom_size
view.setUint32(footerOffset + 16, bloomFilter.getHashCount(), false); view.setUint32(footerOffset + 16, bloomFilter.getHashCount(), false);
view.setUint32(footerOffset + 20, this.entries.length, false); view.setUint32(footerOffset + 20, this.entries.length, false);
view.setUint32(footerOffset + 24, SSTABLE_MAGIC, false); view.setUint32(footerOffset + 24, SSTABLE_MAGIC_V2, false);
view.setUint32(footerOffset + 28, 0, false); // checksum (simplified: 0) view.setUint32(footerOffset + 28, 0, false); // checksum (simplified: 0)
return { return {
sstableData: new Uint8Array(buf), sstableData: new Uint8Array(buf),
@@ -143,13 +154,15 @@ export class SSTableBuilder {
// 内部 // 内部
// ----------------------------------------------------------------------- // -----------------------------------------------------------------------
private splitIntoBlocks(): [string, Record<string, unknown>][][] { /** 按 UTF-8 字节大小切分数据块;大 value 单条独立成块 */
const blocks: [string, Record<string, unknown>][][] = []; private splitIntoBlocks(encoded: EncodedEntry[]): EncodedEntry[][] {
let current: [string, Record<string, unknown>][] = []; const blocks: EncodedEntry[][] = [];
let current: EncodedEntry[] = [];
for (const entry of this.entries) { for (const entry of encoded) {
current.push(entry); current.push(entry);
if (this.estimateBlockSizeFromEntries(current) >= this.blockSizeLimit && current.length > 1) { // v0.4.4-fix: 基于字节估算;单条超大条目(length===1)独立成块不强行切分
if (this.computeBlockSize(current) >= this.blockSizeLimit && current.length > 1) {
blocks.push(current.slice(0, -1)); blocks.push(current.slice(0, -1));
current = [entry]; current = [entry];
} }
@@ -159,24 +172,11 @@ export class SSTableBuilder {
return blocks; return blocks;
} }
private estimateBlockSize(): number { /** 块字节大小:entryCount(u32) + 每对 [keyLen(u32) + key + valueLen(u32) + value] */
return this.estimateBlockSizeFromEntries(this.currentBlock); private computeBlockSize(block: EncodedEntry[]): number {
}
private estimateBlockSizeFromEntries(entries: [string, unknown][]): number {
let size = 0;
for (const [key, value] of entries) {
size += 4 + key.length + JSON.stringify(value).length;
}
return size;
}
private computeBlockSize(block: [string, unknown][]): number {
// entryCount (u32) + 每对: keyLen(u16) + key + valueLen(u16) + value json
let size = 4; let size = 4;
for (const [key, value] of block) { for (const e of block) {
const json = JSON.stringify(value); size += 4 + e.keyBytes.length + 4 + e.valueBytes.length;
size += 2 + key.length + 2 + json.length;
} }
return size; return size;
} }
@@ -184,43 +184,40 @@ export class SSTableBuilder {
private writeDataBlock( private writeDataBlock(
view: DataView, view: DataView,
offset: number, offset: number,
block: [string, Record<string, unknown>][], block: EncodedEntry[],
bloomFilter: BloomFilter, bloomFilter: BloomFilter,
): number { ): number {
// entry count // entry count
view.setUint32(offset, block.length, false); view.setUint32(offset, block.length, false);
offset += 4; offset += 4;
for (const [key, value] of block) { for (const e of block) {
const encoder = new TextEncoder(); // v0.4.4-fix: 长度字段 u32(此前 u16 截断 >64KB 的 value
const keyBytes = encoder.encode(key); if (e.keyBytes.length > 0xFFFFFFFF || e.valueBytes.length > 0xFFFFFFFF) {
const valueBytes = encoder.encode(JSON.stringify(value)); throw new Error('SSTable entry too large (exceeds u32 length field)');
}
// key length view.setUint32(offset, e.keyBytes.length, false);
view.setUint16(offset, keyBytes.length, false); offset += 4;
offset += 2; new Uint8Array(view.buffer).set(e.keyBytes, offset);
// key offset += e.keyBytes.length;
new Uint8Array(view.buffer).set(keyBytes, offset); view.setUint32(offset, e.valueBytes.length, false);
offset += keyBytes.length; offset += 4;
// value length new Uint8Array(view.buffer).set(e.valueBytes, offset);
view.setUint16(offset, valueBytes.length, false); offset += e.valueBytes.length;
offset += 2;
// value
new Uint8Array(view.buffer).set(valueBytes, offset);
offset += valueBytes.length;
// 插入 bloom filter // 插入 bloom filter
bloomFilter.insert(key); bloomFilter.insert(e.key);
} }
return offset; return offset;
} }
private estimateIndexBlockSize(entries: IndexEntry[]): number { private estimateIndexBlockSize(entries: IndexEntry[]): number {
// entryCount(u32) + each: keyLen(u16)+key+blockOffset(u32)+blockSize(u32) // entryCount(u32) + each: keyLen(u32)+key+blockOffset(u32)+blockSize(u32)
let size = 4; let size = 4;
const encoder = new TextEncoder();
for (const entry of entries) { for (const entry of entries) {
size += 2 + entry.key.length + 8; size += 4 + encoder.encode(entry.key).byteLength + 8;
} }
return size; return size;
} }
@@ -232,8 +229,8 @@ export class SSTableBuilder {
for (const entry of entries) { for (const entry of entries) {
const encoder = new TextEncoder(); const encoder = new TextEncoder();
const keyBytes = encoder.encode(entry.key); const keyBytes = encoder.encode(entry.key);
view.setUint16(offset, keyBytes.length, false); view.setUint32(offset, keyBytes.length, false);
offset += 2; offset += 4;
new Uint8Array(view.buffer).set(keyBytes, offset); new Uint8Array(view.buffer).set(keyBytes, offset);
offset += keyBytes.length; offset += keyBytes.length;
view.setUint32(offset, entry.blockOffset, false); view.setUint32(offset, entry.blockOffset, false);
+2 -2
View File
@@ -23,8 +23,8 @@ import { createSchema } from '../src/table/schema';
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
describe('[v0.2.5] P0-1: 版本号统一', () => { describe('[v0.2.5] P0-1: 版本号统一', () => {
test('VERSION 常量为当前版本(0.4.3', () => { test('VERSION 常量为当前版本(0.4.4', () => {
expect(VERSION).toBe('0.4.3'); expect(VERSION).toBe('0.4.4');
}); });
}); });
+1 -1
View File
@@ -400,7 +400,7 @@ describe('[v0.3.3] P1-9: Savepoint + MVCC 一致性', () => {
describe('[v0.3.3] 端到端', () => { describe('[v0.3.3] 端到端', () => {
test('全部修复点可共存于 MetonaSqlark API', async () => { test('全部修复点可共存于 MetonaSqlark API', async () => {
expect(VERSION).toBe('0.4.3'); expect(VERSION).toBe('0.4.4');
const db = new MetonaSqlark({ name: `e2e-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`, mode: 'memory' }); const db = new MetonaSqlark({ name: `e2e-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`, mode: 'memory' });
await db.init(); await db.init();
await db.defineTable('users', { await db.defineTable('users', {
+228
View File
@@ -0,0 +1,228 @@
/**
* v0.4.4 SSTable
*
* - P0: 大段中文内容300KB aria flush close reopen
* - P0: >64KB valueu16
* - P0: value 4096
* - P1: v1
* - P1: 中文 key /
*/
import { AriaEngine } from '../src/engine/aria/index';
import { createSchema } from '../src/table/schema';
import { SSTableBuilder } from '../src/engine/aria/index/sstable_builder';
import { SSTableReader } from '../src/engine/aria/index/sstable';
import type { SSTableMeta } from '../src/engine/aria/types';
import 'fake-indexeddb/auto';
let idbCounter = 0;
function uniqueDB(): string {
return `r45-${Date.now()}-${++idbCounter}-${Math.random().toString(36).slice(2, 8)}`;
}
const makeMeta = (data: Uint8Array): SSTableMeta => ({
id: 1, level: 0, minKey: '', maxKey: '\uffff',
blockCount: 1, totalSize: data.byteLength, bloomData: null,
});
/** 300KB 中文内容(模拟真实笔记/文档数据) */
function makeBigChineseContent(size = 300 * 1024): string {
const chunk = '这是一段用于测试的中文内容。数据持久化必须正确处理 UTF-8 多字节编码!';
return chunk.repeat(Math.ceil(size / chunk.length)).slice(0, size);
}
// ===================================================================
// P0: 大内容编码(builder 字节估算)
// ===================================================================
describe('P0 — SSTable 编码:UTF-8 字节估算', () => {
it('构建含 300KB 中文内容的 SSTable 不越界,可完整读回', () => {
const builder = new SSTableBuilder(4096);
const content = makeBigChineseContent();
builder.add('note:1', { id: '1', title: '长文笔记', body: content });
builder.add('note:2', { id: '2', body: '小内容' });
const { sstableData } = builder.build();
// 修复前:缓冲区低估 → build 抛 RangeError
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
const note1 = reader.get('note:1');
expect(note1).not.toBeNull();
expect((note1 as any).body).toBe(content);
expect((note1 as any).body.length).toBe(content.length);
});
it('构建含 >64KB value 的 SSTableu32 长度字段,修复前 u16 截断)', () => {
const builder = new SSTableBuilder(4096);
const content = makeBigChineseContent(70 * 1024); // 70KB > 64KB
builder.add('doc:big', { id: 'big', data: content });
builder.add('doc:small', { id: 'small', data: 'x' });
const { sstableData } = builder.build();
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
const big = reader.get('doc:big');
expect((big as any).data).toBe(content);
expect((reader.get('doc:small') as any).data).toBe('x');
});
it('大 value>4096 块大小)单条独立成块,scanAll/get 均正确', () => {
const builder = new SSTableBuilder(64); // 极小块阈值
const big = makeBigChineseContent(8 * 1024);
for (let i = 0; i < 5; i++) {
builder.add(`k-${i}`, { v: i, body: i === 2 ? big : 'small' });
}
const { sstableData, indexEntries } = builder.build();
// 大条目独立成块(块数 > 1)
expect(indexEntries.length).toBeGreaterThan(1);
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
const collected: string[] = [];
reader.scanAll((k) => collected.push(k));
expect(collected).toEqual(['k-0', 'k-1', 'k-2', 'k-3', 'k-4']);
expect((reader.get('k-2') as any).body).toBe(big);
});
it('中文 key(多字节)与混合条目', () => {
const builder = new SSTableBuilder(4096);
builder.add('表:笔记一', { : '中文标题', : '内容' });
builder.add('表:笔记二', { : '第二篇' });
const { sstableData } = builder.build();
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
expect((reader.get('表:笔记一') as any)['标题']).toBe('中文标题');
expect((reader.get('表:笔记二') as any)['标题']).toBe('第二篇');
});
it('v2 魔数写入正确(SSTC', () => {
const builder = new SSTableBuilder(4096);
builder.add('a', { v: 1 });
const { sstableData } = builder.build();
const magic = new DataView(sstableData.buffer, sstableData.byteOffset, sstableData.byteLength)
.getUint32(sstableData.byteLength - 8, false);
expect(magic).toBe(0x53535443); // "SSTC"
});
});
// ===================================================================
// P1: v1 旧格式兼容
// ===================================================================
describe('P1 — v1 旧格式兼容', () => {
/** 手工构造 v1 格式(keyLen/valueLen u16 + magic SSTB */
function buildV1SSTable(entries: [string, Record<string, unknown>][]): Uint8Array {
const enc = new TextEncoder();
const encoded = entries.map(([k, v]) => ({ key: k, keyBytes: enc.encode(k), valueBytes: enc.encode(JSON.stringify(v)) }));
// 块(全部放一个块)
const entryCount = entries.length;
let blockSize = 4;
for (const e of encoded) blockSize += 2 + e.keyBytes.length + 2 + e.valueBytes.length;
const indexSize = 4 + 2 + encoded[encoded.length - 1].keyBytes.length + 8;
const bloomSize = 0;
const total = blockSize + indexSize + 32;
const buf = new ArrayBuffer(total);
const view = new DataView(buf);
let offset = 0;
view.setUint32(offset, entryCount, false); offset += 4;
for (const e of encoded) {
view.setUint16(offset, e.keyBytes.length, false); offset += 2;
new Uint8Array(buf).set(e.keyBytes, offset); offset += e.keyBytes.length;
view.setUint16(offset, e.valueBytes.length, false); offset += 2;
new Uint8Array(buf).set(e.valueBytes, offset); offset += e.valueBytes.length;
}
const indexOffset = offset;
view.setUint32(offset, 1, false); offset += 4;
const last = encoded[encoded.length - 1];
view.setUint16(offset, last.keyBytes.length, false); offset += 2;
new Uint8Array(buf).set(last.keyBytes, offset); offset += last.keyBytes.length;
view.setUint32(offset, 0, false); offset += 4;
view.setUint32(offset, blockSize, false); offset += 4;
const footer = offset;
view.setUint32(footer, indexOffset, false);
view.setUint32(footer + 4, indexSize, false);
view.setUint32(footer + 8, 0, false); // bloom_offset
view.setUint32(footer + 12, 0, false); // bloom_size
view.setUint32(footer + 16, 0, false); // hash_count
view.setUint32(footer + 20, entryCount, false);
view.setUint32(footer + 24, 0x53535442, false); // SSTB
view.setUint32(footer + 28, 0, false);
return new Uint8Array(buf);
}
it('v1 格式文件可读(旧库数据不丢)', () => {
const v1 = buildV1SSTable([
['u:1', { name: 'Alice', age: 30 }],
['u:2', { name: 'Bob', age: 25 }],
]);
const reader = new SSTableReader(v1, makeMeta(v1));
expect((reader.get('u:1') as any).name).toBe('Alice');
expect((reader.get('u:2') as any).name).toBe('Bob');
const keys: string[] = [];
reader.scanAll((k) => keys.push(k));
expect(keys).toEqual(['u:1', 'u:2']);
});
});
// ===================================================================
// P0: 端到端 — 大内容跨重启(爸爸 Electron 冒烟场景)
// ===================================================================
describe('P0 — 大内容端到端(aria 模式)', () => {
it('300KB 中文写入 → close → reopen 数据完整(修复前打开必崩)', async () => {
const dbName = uniqueDB();
const engine = new AriaEngine({ storageBackend: 'indexeddb', checkpointInterval: 100000 });
await engine.open(dbName, 1);
await engine.createTable(createSchema('notes', {
id: { type: 'string', primaryKey: true },
title: { type: 'string' },
body: { type: 'string' },
}));
const content = makeBigChineseContent();
await engine.insert('notes', [
{ id: 'n1', title: '长文', body: content },
{ id: 'n2', title: '短文', body: 'hello' },
]);
// 强制落盘(修复前:flush 编码越界崩溃)
await (engine as any).lsm.flush();
await engine.close();
// 修复前:打开解析半写文件 → ARIA_OPEN_ERROR
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', checkpointInterval: 100000 });
await expect(engine2.open(dbName, 1)).resolves.toBeUndefined();
const rows = await engine2.find('notes', { table: 'notes' });
expect(rows).toHaveLength(2);
const byId = Object.fromEntries(rows.map((r) => [r.id, r]));
expect(byId['n1'].body).toBe(content);
expect(byId['n1'].body.length).toBe(content.length);
expect(byId['n2'].body).toBe('hello');
await engine2.close();
});
it('多行 >64KB 内容批量写入重开完整(含索引列)', async () => {
const dbName = uniqueDB();
const engine = new AriaEngine({
storageBackend: 'indexeddb',
memtableSizeThreshold: 32 * 1024,
checkpointInterval: 100000,
});
await engine.open(dbName, 1);
await engine.createTable(createSchema('docs', {
id: { type: 'string', primaryKey: true },
tag: { type: 'string', index: true },
body: { type: 'string' },
}));
const contents: string[] = [];
for (let i = 0; i < 10; i++) {
const c = makeBigChineseContent(70 * 1024 + i * 100);
contents.push(c);
await engine.insert('docs', [{ id: `d${i}`, tag: i % 2 === 0 ? 'even' : 'odd', body: c }]);
}
await engine.close();
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', checkpointInterval: 100000 });
await engine2.open(dbName, 1);
expect(await engine2.count('docs')).toBe(10);
const evens = await engine2.find('docs', { table: 'docs', where: { tag: 'even' } });
expect(evens).toHaveLength(5);
for (let i = 0; i < 10; i++) {
const rows = await engine2.find('docs', { table: 'docs', where: { id: `d${i}` } });
expect(rows[0].body).toBe(contents[i]);
}
await engine2.close();
});
});