release: v0.4.4 — SSTable 编码修复(UTF-8 字节估算 + u32 长度字段 + v1/v2 双格式兼容),大内容不再崩溃
This commit is contained in:
@@ -2,6 +2,24 @@
|
|||||||
|
|
||||||
All notable changes to MetonaSqlark will be documented in this file.
|
All notable changes to MetonaSqlark will be documented in this file.
|
||||||
|
|
||||||
|
## [0.4.4] - 2026-08-09
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **SSTable 编码缺陷导致大内容写入崩溃(P0)** — AriaEngine 保存大段中文内容(如 300KB 笔记)→ close 构建 SSTable 崩溃 → 半写文件 → 下次打开 ARIA_OPEN_ERROR:
|
||||||
|
- **块大小估算用 UTF-16 码元而非 UTF-8 字节** — `computeBlockSize` 用 `json.length`(中文 1 字 = 1 码元)估算,实际写入 UTF-8 字节(中文 1 字 = 3 字节)→ 300KB 中文实际 900KB → 缓冲区低估 → `Uint8Array.set` 越界崩溃。改为 TextEncoder 预编码,全部按字节长度计算(切块、块大小、索引块、总大小)
|
||||||
|
- **valueLen/keyLen 字段 u16 截断** — 长度字段 u16 上限 64KB,大 value 被截断 → 线性格式整体错乱。格式 v2(magic "SSTC")升级为 u32 长度字段
|
||||||
|
- **大 value 块切分** — 单条 value 超过块大小(4096)时独立成块(基于字节估算正确切分)
|
||||||
|
- **格式兼容** — reader 支持 v1("SSTB",u16)与 v2("SSTC",u32)双格式:旧库 v1 文件仍可读(小 value 场景无缺陷),新写入用 v2,旧库数据不丢失
|
||||||
|
- 连带受益:中文主键 key、大内容二级索引列(索引 key 含列值)同样支持 u32 长度字段
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- 测试 950 → **958**(52 套件),新增 `tests/v045-hardening.test.ts`(8 条:300KB 中文 builder/端到端跨重启、>64KB value、大 value 独立成块、中文 key、v1 兼容、v2 魔数)
|
||||||
|
- 版本号升至 v0.4.4
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## [0.4.3] - 2026-08-09
|
## [0.4.3] - 2026-08-09
|
||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
|
|||||||
@@ -1,10 +1,10 @@
|
|||||||
# MetonaSqlark
|
# MetonaSqlark
|
||||||
|
|
||||||
<p align="center">
|
<p align="center">
|
||||||
<img src="https://img.shields.io/badge/version-0.4.3-blue?style=flat-square" alt="version">
|
<img src="https://img.shields.io/badge/version-0.4.4-blue?style=flat-square" alt="version">
|
||||||
<img src="https://img.shields.io/badge/license-MIT-green?style=flat-square" alt="license">
|
<img src="https://img.shields.io/badge/license-MIT-green?style=flat-square" alt="license">
|
||||||
<img src="https://img.shields.io/badge/coverage-84.2%25-brightgreen?style=flat-square" alt="coverage">
|
<img src="https://img.shields.io/badge/coverage-84.2%25-brightgreen?style=flat-square" alt="coverage">
|
||||||
<img src="https://img.shields.io/badge/tests-950%20passed-success?style=flat-square" alt="tests">
|
<img src="https://img.shields.io/badge/tests-958%20passed-success?style=flat-square" alt="tests">
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
> 基于 TypeScript 的**前端关系型数据库**,支持完整 SQL 查询、Query Builder 链式 API、与 **AriaEngine 自研页面式存储引擎**。
|
> 基于 TypeScript 的**前端关系型数据库**,支持完整 SQL 查询、Query Builder 链式 API、与 **AriaEngine 自研页面式存储引擎**。
|
||||||
@@ -26,10 +26,11 @@
|
|||||||
- 🔗 **外键级联** — ON DELETE + ON UPDATE(CASCADE / SET NULL / RESTRICT)全引擎支持,支持更新主键(v0.4.2)
|
- 🔗 **外键级联** — ON DELETE + ON UPDATE(CASCADE / SET NULL / RESTRICT)全引擎支持,支持更新主键(v0.4.2)
|
||||||
- 🛡 **崩溃恢复自愈** — 残缺 SSTable 打开自动跳过、`db.repair()` 自愈、`db.clearAll()` 重置、迁移版本持久化(v0.4.2)
|
- 🛡 **崩溃恢复自愈** — 残缺 SSTable 打开自动跳过、`db.repair()` 自愈、`db.clearAll()` 重置、迁移版本持久化(v0.4.2)
|
||||||
- 🧵 **关闭时序与后台任务加固** — 后台 flush/compaction 串行入队(close 排空后才关闭存储,失败显式报告 `ARIA_BACKGROUND_ERROR`)、预加载等待链稳定(compaction 竞态修复)、事务提交先落 WAL 再合并快照(v0.4.3)
|
- 🧵 **关闭时序与后台任务加固** — 后台 flush/compaction 串行入队(close 排空后才关闭存储,失败显式报告 `ARIA_BACKGROUND_ERROR`)、预加载等待链稳定(compaction 竞态修复)、事务提交先落 WAL 再合并快照(v0.4.3)
|
||||||
|
- 📏 **SSTable 编码修复** — 块大小按 UTF-8 字节精确计算(大段中文内容不再因缓冲区低估崩溃)、长度字段 u32(>64KB value 不截断)、v1/v2 双格式兼容(旧库数据不丢)(v0.4.4)
|
||||||
- 🌲 **RB-Tree 完整实现** — 标准红黑树插入+删除修复,O(log n) 保证
|
- 🌲 **RB-Tree 完整实现** — 标准红黑树插入+删除修复,O(log n) 保证
|
||||||
- ⚡ **性能优化** — SSTableReader 二分查找统一、IndexedDB 索引利用、crypto 实例化避免全局状态
|
- ⚡ **性能优化** — SSTableReader 二分查找统一、IndexedDB 索引利用、crypto 实例化避免全局状态
|
||||||
- 🌐 **浏览器兼容** — Chrome 80+ / Firefox 80+ / Safari 14+ / Edge 80+ / Node.js 16+
|
- 🌐 **浏览器兼容** — Chrome 80+ / Firefox 80+ / Safari 14+ / Edge 80+ / Node.js 16+
|
||||||
- 🧪 **950 测试 · 84.2% 覆盖率** — 51 套件,生产级质量保证
|
- 🧪 **958 测试 · 84.2% 覆盖率** — 52 套件,生产级质量保证
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -400,7 +401,7 @@ npm run typecheck # 类型检查
|
|||||||
|
|
||||||
| 指标 | 数值 |
|
| 指标 | 数值 |
|
||||||
|------|------|
|
|------|------|
|
||||||
| 测试用例 | 950 |
|
| 测试用例 | 958 |
|
||||||
| 测试套件 | 51 |
|
| 测试套件 | 51 |
|
||||||
| 行覆盖率 | 84.2% |
|
| 行覆盖率 | 84.2% |
|
||||||
| SQL 关键字 | 36 |
|
| SQL 关键字 | 36 |
|
||||||
|
|||||||
Vendored
+113
-97
@@ -34,7 +34,7 @@ class DatabaseError extends Error {
|
|||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// 版本
|
// 版本
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
const VERSION = '0.4.3';
|
const VERSION = '0.4.4';
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* metona-sqlark Shared WHERE Matcher — 统一的条件匹配逻辑
|
* metona-sqlark Shared WHERE Matcher — 统一的条件匹配逻辑
|
||||||
@@ -2713,7 +2713,15 @@ class BloomFilter {
|
|||||||
*
|
*
|
||||||
* 将排序后的 key-value 数据写入 SSTable 格式。
|
* 将排序后的 key-value 数据写入 SSTable 格式。
|
||||||
*
|
*
|
||||||
* SSTable 文件布局:
|
* v0.4.4 格式 v2(magic "SSTC")修复:
|
||||||
|
* - 块大小估算改用 UTF-8 字节长度(TextEncoder 预编码),
|
||||||
|
* 此前用字符串 .length(UTF-16 码元)估算而实际写入 UTF-8 字节,
|
||||||
|
* 中文内容(1 字 3 字节)导致缓冲区低估 → 写入越界崩溃
|
||||||
|
* - keyLen/valueLen 从 u16 升级为 u32(此前 >64KB 的 value 长度被截断,
|
||||||
|
* 线性格式整体错乱)
|
||||||
|
* - 大 value 单条独立成块(切分逻辑基于字节估算)
|
||||||
|
*
|
||||||
|
* SSTable 文件布局 (v2):
|
||||||
* ┌──────────────────────────────────────────────┐
|
* ┌──────────────────────────────────────────────┐
|
||||||
* │ Data Block 0 │
|
* │ Data Block 0 │
|
||||||
* │ Data Block 1 │
|
* │ Data Block 1 │
|
||||||
@@ -2727,58 +2735,56 @@ class BloomFilter {
|
|||||||
* │ - bloom_size (u32) │
|
* │ - bloom_size (u32) │
|
||||||
* │ - bloom_hash_count (u32) │
|
* │ - bloom_hash_count (u32) │
|
||||||
* │ - entry_count (u32) │
|
* │ - entry_count (u32) │
|
||||||
* │ - magic_number (u32, 0x53535442 ="SSTB")│
|
* │ - magic_number (u32, 0x53535443 ="SSTC")│
|
||||||
* │ - checksum (u32) │
|
* │ - checksum (u32) │
|
||||||
* └──────────────────────────────────────────────┘
|
* └──────────────────────────────────────────────┘
|
||||||
|
* 数据块条目: entryCount(u32) + [keyLen(u32) + key + valueLen(u32) + value]
|
||||||
|
* 索引块条目: [keyLen(u32) + key + blockOffset(u32) + blockSize(u32)]
|
||||||
*/
|
*/
|
||||||
const SSTABLE_MAGIC$1 = 0x53535442; // "SSTB"
|
/** v1 格式魔数("SSTB",u16 长度字段,兼容旧文件读取) */
|
||||||
|
const SSTABLE_MAGIC_V1 = 0x53535442;
|
||||||
|
/** v2 格式魔数("SSTC",u32 长度字段 + 字节精确估算,v0.4.4) */
|
||||||
|
const SSTABLE_MAGIC_V2 = 0x53535443;
|
||||||
const SSTABLE_FOOTER_SIZE = 32;
|
const SSTABLE_FOOTER_SIZE = 32;
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// SSTableBuilder
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
class SSTableBuilder {
|
class SSTableBuilder {
|
||||||
constructor(blockSizeLimit = 4096) {
|
constructor(blockSizeLimit = 4096) {
|
||||||
this.entries = [];
|
this.entries = [];
|
||||||
this.currentBlock = [];
|
|
||||||
this.currentBlockStartKey = '';
|
|
||||||
this.blockSizeLimit = blockSizeLimit;
|
this.blockSizeLimit = blockSizeLimit;
|
||||||
}
|
}
|
||||||
/** 添加一个 key-value 条目(必须按键排序添加) */
|
/** 添加一个 key-value 条目(必须按键排序添加) */
|
||||||
add(key, value) {
|
add(key, value) {
|
||||||
if (this.currentBlock.length === 0) {
|
|
||||||
this.currentBlockStartKey = key;
|
|
||||||
}
|
|
||||||
this.currentBlock.push([key, value]);
|
|
||||||
this.entries.push([key, value]);
|
this.entries.push([key, value]);
|
||||||
// 如果当前 Block 达到大小限制,切割
|
|
||||||
const estimated = this.estimateBlockSize();
|
|
||||||
if (estimated >= this.blockSizeLimit && this.currentBlock.length > 1) ;
|
|
||||||
}
|
}
|
||||||
/**
|
/**
|
||||||
* 构建 SSTable 文件的二进制数据。
|
* 构建 SSTable 文件的二进制数据(v2 格式)。
|
||||||
* 返回 { data: Uint8Array, indexEntries: IndexEntry[], bloomFilter: BloomFilter }
|
* 返回 { data: Uint8Array, indexEntries: IndexEntry[] }
|
||||||
*/
|
*/
|
||||||
build() {
|
build() {
|
||||||
const blocks = this.splitIntoBlocks();
|
// v0.4.4-fix: 预编码全部条目 — 块大小估算必须基于 UTF-8 字节长度,
|
||||||
|
// 字符串 .length 是 UTF-16 码元(中文 1 字 3 字节 vs 1 码元)→ 缓冲区低估越界
|
||||||
|
const encoder = new TextEncoder();
|
||||||
|
const encoded = this.entries.map(([key, value]) => ({
|
||||||
|
key,
|
||||||
|
keyBytes: encoder.encode(key),
|
||||||
|
valueBytes: encoder.encode(JSON.stringify(value)),
|
||||||
|
}));
|
||||||
|
const blocks = this.splitIntoBlocks(encoded);
|
||||||
const bloomFilter = new BloomFilter(this.entries.length);
|
const bloomFilter = new BloomFilter(this.entries.length);
|
||||||
// 预计算总大小
|
// 预计算总大小(字节)
|
||||||
let totalSize = 0;
|
let totalSize = 0;
|
||||||
const blockOffsets = [];
|
const blockOffsets = [];
|
||||||
for (const block of blocks) {
|
for (const block of blocks) {
|
||||||
blockOffsets.push(totalSize);
|
blockOffsets.push(totalSize);
|
||||||
const blockSize = this.computeBlockSize(block);
|
totalSize += this.computeBlockSize(block);
|
||||||
totalSize += blockSize;
|
|
||||||
}
|
}
|
||||||
// 索引块
|
// 索引块(块内最后一个 key 作为索引键)
|
||||||
const indexEntries = [];
|
const indexEntries = [];
|
||||||
for (let i = 0; i < blocks.length; i++) {
|
for (let i = 0; i < blocks.length; i++) {
|
||||||
const block = blocks[i];
|
const block = blocks[i];
|
||||||
const lastKey = block[block.length - 1][0];
|
|
||||||
const blockSize = this.computeBlockSize(block);
|
|
||||||
indexEntries.push({
|
indexEntries.push({
|
||||||
key: lastKey,
|
key: block[block.length - 1].key,
|
||||||
blockOffset: blockOffsets[i],
|
blockOffset: blockOffsets[i],
|
||||||
blockSize,
|
blockSize: this.computeBlockSize(block),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
const indexBlockSize = this.estimateIndexBlockSize(indexEntries);
|
const indexBlockSize = this.estimateIndexBlockSize(indexEntries);
|
||||||
@@ -2809,7 +2815,7 @@ class SSTableBuilder {
|
|||||||
view.setUint32(footerOffset + 12, bloomSize, false); // bloom_size
|
view.setUint32(footerOffset + 12, bloomSize, false); // bloom_size
|
||||||
view.setUint32(footerOffset + 16, bloomFilter.getHashCount(), false);
|
view.setUint32(footerOffset + 16, bloomFilter.getHashCount(), false);
|
||||||
view.setUint32(footerOffset + 20, this.entries.length, false);
|
view.setUint32(footerOffset + 20, this.entries.length, false);
|
||||||
view.setUint32(footerOffset + 24, SSTABLE_MAGIC$1, false);
|
view.setUint32(footerOffset + 24, SSTABLE_MAGIC_V2, false);
|
||||||
view.setUint32(footerOffset + 28, 0, false); // checksum (simplified: 0)
|
view.setUint32(footerOffset + 28, 0, false); // checksum (simplified: 0)
|
||||||
return {
|
return {
|
||||||
sstableData: new Uint8Array(buf),
|
sstableData: new Uint8Array(buf),
|
||||||
@@ -2823,12 +2829,14 @@ class SSTableBuilder {
|
|||||||
// -----------------------------------------------------------------------
|
// -----------------------------------------------------------------------
|
||||||
// 内部
|
// 内部
|
||||||
// -----------------------------------------------------------------------
|
// -----------------------------------------------------------------------
|
||||||
splitIntoBlocks() {
|
/** 按 UTF-8 字节大小切分数据块;大 value 单条独立成块 */
|
||||||
|
splitIntoBlocks(encoded) {
|
||||||
const blocks = [];
|
const blocks = [];
|
||||||
let current = [];
|
let current = [];
|
||||||
for (const entry of this.entries) {
|
for (const entry of encoded) {
|
||||||
current.push(entry);
|
current.push(entry);
|
||||||
if (this.estimateBlockSizeFromEntries(current) >= this.blockSizeLimit && current.length > 1) {
|
// v0.4.4-fix: 基于字节估算;单条超大条目(length===1)独立成块不强行切分
|
||||||
|
if (this.computeBlockSize(current) >= this.blockSizeLimit && current.length > 1) {
|
||||||
blocks.push(current.slice(0, -1));
|
blocks.push(current.slice(0, -1));
|
||||||
current = [entry];
|
current = [entry];
|
||||||
}
|
}
|
||||||
@@ -2837,22 +2845,11 @@ class SSTableBuilder {
|
|||||||
blocks.push(current);
|
blocks.push(current);
|
||||||
return blocks;
|
return blocks;
|
||||||
}
|
}
|
||||||
estimateBlockSize() {
|
/** 块字节大小:entryCount(u32) + 每对 [keyLen(u32) + key + valueLen(u32) + value] */
|
||||||
return this.estimateBlockSizeFromEntries(this.currentBlock);
|
|
||||||
}
|
|
||||||
estimateBlockSizeFromEntries(entries) {
|
|
||||||
let size = 0;
|
|
||||||
for (const [key, value] of entries) {
|
|
||||||
size += 4 + key.length + JSON.stringify(value).length;
|
|
||||||
}
|
|
||||||
return size;
|
|
||||||
}
|
|
||||||
computeBlockSize(block) {
|
computeBlockSize(block) {
|
||||||
// entryCount (u32) + 每对: keyLen(u16) + key + valueLen(u16) + value json
|
|
||||||
let size = 4;
|
let size = 4;
|
||||||
for (const [key, value] of block) {
|
for (const e of block) {
|
||||||
const json = JSON.stringify(value);
|
size += 4 + e.keyBytes.length + 4 + e.valueBytes.length;
|
||||||
size += 2 + key.length + 2 + json.length;
|
|
||||||
}
|
}
|
||||||
return size;
|
return size;
|
||||||
}
|
}
|
||||||
@@ -2860,32 +2857,30 @@ class SSTableBuilder {
|
|||||||
// entry count
|
// entry count
|
||||||
view.setUint32(offset, block.length, false);
|
view.setUint32(offset, block.length, false);
|
||||||
offset += 4;
|
offset += 4;
|
||||||
for (const [key, value] of block) {
|
for (const e of block) {
|
||||||
const encoder = new TextEncoder();
|
// v0.4.4-fix: 长度字段 u32(此前 u16 截断 >64KB 的 value)
|
||||||
const keyBytes = encoder.encode(key);
|
if (e.keyBytes.length > 0xFFFFFFFF || e.valueBytes.length > 0xFFFFFFFF) {
|
||||||
const valueBytes = encoder.encode(JSON.stringify(value));
|
throw new Error('SSTable entry too large (exceeds u32 length field)');
|
||||||
// key length
|
}
|
||||||
view.setUint16(offset, keyBytes.length, false);
|
view.setUint32(offset, e.keyBytes.length, false);
|
||||||
offset += 2;
|
offset += 4;
|
||||||
// key
|
new Uint8Array(view.buffer).set(e.keyBytes, offset);
|
||||||
new Uint8Array(view.buffer).set(keyBytes, offset);
|
offset += e.keyBytes.length;
|
||||||
offset += keyBytes.length;
|
view.setUint32(offset, e.valueBytes.length, false);
|
||||||
// value length
|
offset += 4;
|
||||||
view.setUint16(offset, valueBytes.length, false);
|
new Uint8Array(view.buffer).set(e.valueBytes, offset);
|
||||||
offset += 2;
|
offset += e.valueBytes.length;
|
||||||
// value
|
|
||||||
new Uint8Array(view.buffer).set(valueBytes, offset);
|
|
||||||
offset += valueBytes.length;
|
|
||||||
// 插入 bloom filter
|
// 插入 bloom filter
|
||||||
bloomFilter.insert(key);
|
bloomFilter.insert(e.key);
|
||||||
}
|
}
|
||||||
return offset;
|
return offset;
|
||||||
}
|
}
|
||||||
estimateIndexBlockSize(entries) {
|
estimateIndexBlockSize(entries) {
|
||||||
// entryCount(u32) + each: keyLen(u16)+key+blockOffset(u32)+blockSize(u32)
|
// entryCount(u32) + each: keyLen(u32)+key+blockOffset(u32)+blockSize(u32)
|
||||||
let size = 4;
|
let size = 4;
|
||||||
|
const encoder = new TextEncoder();
|
||||||
for (const entry of entries) {
|
for (const entry of entries) {
|
||||||
size += 2 + entry.key.length + 8;
|
size += 4 + encoder.encode(entry.key).byteLength + 8;
|
||||||
}
|
}
|
||||||
return size;
|
return size;
|
||||||
}
|
}
|
||||||
@@ -2895,8 +2890,8 @@ class SSTableBuilder {
|
|||||||
for (const entry of entries) {
|
for (const entry of entries) {
|
||||||
const encoder = new TextEncoder();
|
const encoder = new TextEncoder();
|
||||||
const keyBytes = encoder.encode(entry.key);
|
const keyBytes = encoder.encode(entry.key);
|
||||||
view.setUint16(offset, keyBytes.length, false);
|
view.setUint32(offset, keyBytes.length, false);
|
||||||
offset += 2;
|
offset += 4;
|
||||||
new Uint8Array(view.buffer).set(keyBytes, offset);
|
new Uint8Array(view.buffer).set(keyBytes, offset);
|
||||||
offset += keyBytes.length;
|
offset += keyBytes.length;
|
||||||
view.setUint32(offset, entry.blockOffset, false);
|
view.setUint32(offset, entry.blockOffset, false);
|
||||||
@@ -2911,8 +2906,10 @@ class SSTableBuilder {
|
|||||||
/**
|
/**
|
||||||
* AriaEngine SSTable Reader — 从 SSTable 二进制数据中读取
|
* AriaEngine SSTable Reader — 从 SSTable 二进制数据中读取
|
||||||
* @module engine/aria/index/sstable
|
* @module engine/aria/index/sstable
|
||||||
|
*
|
||||||
|
* v0.4.4: 支持 v1("SSTB",u16 长度字段)与 v2("SSTC",u32 长度字段)双格式,
|
||||||
|
* 旧库 v1 文件仍可读(小 value 场景无缺陷),新写入使用 v2。
|
||||||
*/
|
*/
|
||||||
const SSTABLE_MAGIC = 0x53535442; // "SSTB"
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// SSTableReader
|
// SSTableReader
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -2921,11 +2918,20 @@ class SSTableReader {
|
|||||||
this.indexEntries = [];
|
this.indexEntries = [];
|
||||||
this.entryCount = 0;
|
this.entryCount = 0;
|
||||||
this.bloomFilter = null;
|
this.bloomFilter = null;
|
||||||
|
/** 格式版本:1 = u16 长度字段(旧),2 = u32 长度字段(v0.4.4) */
|
||||||
|
this.format = 2;
|
||||||
this.data = data;
|
this.data = data;
|
||||||
this.view = new DataView(data.buffer, data.byteOffset, data.byteLength);
|
this.view = new DataView(data.buffer, data.byteOffset, data.byteLength);
|
||||||
this.meta = meta;
|
this.meta = meta;
|
||||||
this.parseFooter();
|
this.parseFooter();
|
||||||
}
|
}
|
||||||
|
/** 长度字段宽度:v2 = 4 字节 u32,v1 = 2 字节 u16 */
|
||||||
|
lenFieldSize() {
|
||||||
|
return this.format === 2 ? 4 : 2;
|
||||||
|
}
|
||||||
|
readLen(offset) {
|
||||||
|
return this.format === 2 ? this.view.getUint32(offset, false) : this.view.getUint16(offset, false);
|
||||||
|
}
|
||||||
// -----------------------------------------------------------------------
|
// -----------------------------------------------------------------------
|
||||||
// 查询
|
// 查询
|
||||||
// -----------------------------------------------------------------------
|
// -----------------------------------------------------------------------
|
||||||
@@ -2939,24 +2945,25 @@ class SSTableReader {
|
|||||||
return null;
|
return null;
|
||||||
const entry = this.indexEntries[blockIdx];
|
const entry = this.indexEntries[blockIdx];
|
||||||
const blockData = this.getBlockData(entry);
|
const blockData = this.getBlockData(entry);
|
||||||
// v0.4.2-fix: 残缺文件(meta 偏移超出实际长度)跳过该块,而非抛 RangeError
|
// v0.4.1-fix: 残缺文件(meta 偏移超出实际长度)跳过该块,而非抛 RangeError
|
||||||
if (!blockData)
|
if (!blockData)
|
||||||
return null;
|
return null;
|
||||||
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
||||||
|
const lenSize = this.lenFieldSize();
|
||||||
const entryCount = blockView.getUint32(0, false);
|
const entryCount = blockView.getUint32(0, false);
|
||||||
let offset = 4;
|
let offset = 4;
|
||||||
// 顺序扫描 block 内的条目(生产中应二分查找)
|
// 顺序扫描 block 内的条目(生产中应二分查找)
|
||||||
for (let i = 0; i < entryCount; i++) {
|
for (let i = 0; i < entryCount; i++) {
|
||||||
if (offset + 2 > blockData.byteLength)
|
if (offset + lenSize > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const keyLen = blockView.getUint16(offset, false);
|
const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + keyLen + 2 > blockData.byteLength)
|
if (offset + keyLen + lenSize > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
||||||
offset += keyLen;
|
offset += keyLen;
|
||||||
const valLen = blockView.getUint16(offset, false);
|
const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + valLen > blockData.byteLength)
|
if (offset + valLen > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const valBytes = blockData.slice(offset, offset + valLen);
|
const valBytes = blockData.slice(offset, offset + valLen);
|
||||||
@@ -2980,26 +2987,27 @@ class SSTableReader {
|
|||||||
const endBlockIdx = Math.min(this.indexEntries.length - 1, this.locateBlockLE(endKey));
|
const endBlockIdx = Math.min(this.indexEntries.length - 1, this.locateBlockLE(endKey));
|
||||||
if (startBlockIdx < 0 || endBlockIdx < 0 || startBlockIdx > endBlockIdx)
|
if (startBlockIdx < 0 || endBlockIdx < 0 || startBlockIdx > endBlockIdx)
|
||||||
return;
|
return;
|
||||||
|
const lenSize = this.lenFieldSize();
|
||||||
for (let bi = startBlockIdx; bi <= endBlockIdx && bi >= 0; bi++) {
|
for (let bi = startBlockIdx; bi <= endBlockIdx && bi >= 0; bi++) {
|
||||||
const entry = this.indexEntries[bi];
|
const entry = this.indexEntries[bi];
|
||||||
const blockData = this.getBlockData(entry);
|
const blockData = this.getBlockData(entry);
|
||||||
// v0.4.2-fix: 残缺块跳过(rangeScan 继续后续块,不抛异常)
|
// v0.4.1-fix: 残缺块跳过(rangeScan 继续后续块,不抛异常)
|
||||||
if (!blockData)
|
if (!blockData)
|
||||||
continue;
|
continue;
|
||||||
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
||||||
const blockEntryCount = blockView.getUint32(0, false);
|
const blockEntryCount = blockView.getUint32(0, false);
|
||||||
let offset = 4;
|
let offset = 4;
|
||||||
for (let i = 0; i < blockEntryCount; i++) {
|
for (let i = 0; i < blockEntryCount; i++) {
|
||||||
if (offset + 2 > blockData.byteLength)
|
if (offset + lenSize > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const keyLen = blockView.getUint16(offset, false);
|
const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + keyLen + 2 > blockData.byteLength)
|
if (offset + keyLen + lenSize > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
||||||
offset += keyLen;
|
offset += keyLen;
|
||||||
const valLen = blockView.getUint16(offset, false);
|
const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + valLen > blockData.byteLength)
|
if (offset + valLen > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const valBytes = blockData.slice(offset, offset + valLen);
|
const valBytes = blockData.slice(offset, offset + valLen);
|
||||||
@@ -3018,25 +3026,26 @@ class SSTableReader {
|
|||||||
}
|
}
|
||||||
/** 扫描所有条目 */
|
/** 扫描所有条目 */
|
||||||
scanAll(callback) {
|
scanAll(callback) {
|
||||||
|
const lenSize = this.lenFieldSize();
|
||||||
for (const entry of this.indexEntries) {
|
for (const entry of this.indexEntries) {
|
||||||
const blockData = this.getBlockData(entry);
|
const blockData = this.getBlockData(entry);
|
||||||
// v0.4.2-fix: 残缺块跳过(scanAll 继续后续块,不抛异常)
|
// v0.4.1-fix: 残缺块跳过(scanAll 继续后续块,不抛异常)
|
||||||
if (!blockData)
|
if (!blockData)
|
||||||
continue;
|
continue;
|
||||||
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
||||||
const blockEntryCount = blockView.getUint32(0, false);
|
const blockEntryCount = blockView.getUint32(0, false);
|
||||||
let offset = 4;
|
let offset = 4;
|
||||||
for (let i = 0; i < blockEntryCount; i++) {
|
for (let i = 0; i < blockEntryCount; i++) {
|
||||||
if (offset + 2 > blockData.byteLength)
|
if (offset + lenSize > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const keyLen = blockView.getUint16(offset, false);
|
const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + keyLen + 2 > blockData.byteLength)
|
if (offset + keyLen + lenSize > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
||||||
offset += keyLen;
|
offset += keyLen;
|
||||||
const valLen = blockView.getUint16(offset, false);
|
const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + valLen > blockData.byteLength)
|
if (offset + valLen > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const valBytes = blockData.slice(offset, offset + valLen);
|
const valBytes = blockData.slice(offset, offset + valLen);
|
||||||
@@ -3067,10 +3076,16 @@ class SSTableReader {
|
|||||||
throw new Error('SSTable too small: missing footer');
|
throw new Error('SSTable too small: missing footer');
|
||||||
}
|
}
|
||||||
const footerOffset = this.data.byteLength - 32;
|
const footerOffset = this.data.byteLength - 32;
|
||||||
// 验证魔数
|
// 验证魔数(v1 "SSTB" / v2 "SSTC")
|
||||||
const magic = this.view.getUint32(footerOffset + 24, false);
|
const magic = this.view.getUint32(footerOffset + 24, false);
|
||||||
if (magic !== SSTABLE_MAGIC) {
|
if (magic === SSTABLE_MAGIC_V1) {
|
||||||
throw new Error(`Invalid SSTable magic: expected ${SSTABLE_MAGIC}, got ${magic}`);
|
this.format = 1;
|
||||||
|
}
|
||||||
|
else if (magic === SSTABLE_MAGIC_V2) {
|
||||||
|
this.format = 2;
|
||||||
|
}
|
||||||
|
else {
|
||||||
|
throw new Error(`Invalid SSTable magic: expected ${SSTABLE_MAGIC_V1} or ${SSTABLE_MAGIC_V2}, got ${magic}`);
|
||||||
}
|
}
|
||||||
const indexOffset = this.view.getUint32(footerOffset, false);
|
const indexOffset = this.view.getUint32(footerOffset, false);
|
||||||
const indexSize = this.view.getUint32(footerOffset + 4, false);
|
const indexSize = this.view.getUint32(footerOffset + 4, false);
|
||||||
@@ -3078,7 +3093,7 @@ class SSTableReader {
|
|||||||
const bloomSize = this.view.getUint32(footerOffset + 12, false);
|
const bloomSize = this.view.getUint32(footerOffset + 12, false);
|
||||||
const bloomHashCount = this.view.getUint32(footerOffset + 16, false);
|
const bloomHashCount = this.view.getUint32(footerOffset + 16, false);
|
||||||
this.entryCount = this.view.getUint32(footerOffset + 20, false);
|
this.entryCount = this.view.getUint32(footerOffset + 20, false);
|
||||||
// v0.4.2-fix: 完整性校验 — 索引块必须完全落在文件内,否则视为残缺文件跳过
|
// v0.4.1-fix: 完整性校验 — 索引块必须完全落在文件内,否则视为残缺文件跳过
|
||||||
if (indexOffset + 4 > this.data.byteLength || indexOffset + indexSize > this.data.byteLength) {
|
if (indexOffset + 4 > this.data.byteLength || indexOffset + indexSize > this.data.byteLength) {
|
||||||
return; // 残缺文件:无索引块可读,get/rangeScan 均返回空
|
return; // 残缺文件:无索引块可读,get/rangeScan 均返回空
|
||||||
}
|
}
|
||||||
@@ -3098,13 +3113,14 @@ class SSTableReader {
|
|||||||
parseIndexBlock(offset, _size) {
|
parseIndexBlock(offset, _size) {
|
||||||
const entryCount = this.view.getUint32(offset, false);
|
const entryCount = this.view.getUint32(offset, false);
|
||||||
offset += 4;
|
offset += 4;
|
||||||
|
const lenSize = this.lenFieldSize();
|
||||||
for (let i = 0; i < entryCount; i++) {
|
for (let i = 0; i < entryCount; i++) {
|
||||||
// v0.4.2-fix: 索引条目越界(keyLen/blockOffset/blockSize 超过文件长度)时中止解析,
|
// v0.4.1-fix: 索引条目越界(keyLen/blockOffset/blockSize 超过文件长度)时中止解析,
|
||||||
// 已解析的有效条目仍可用于查询
|
// 已解析的有效条目仍可用于查询
|
||||||
if (offset + 2 > this.data.byteLength)
|
if (offset + lenSize > this.data.byteLength)
|
||||||
break;
|
break;
|
||||||
const keyLen = this.view.getUint16(offset, false);
|
const keyLen = this.readLen(offset);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + keyLen + 8 > this.data.byteLength)
|
if (offset + keyLen + 8 > this.data.byteLength)
|
||||||
break;
|
break;
|
||||||
const key = new TextDecoder().decode(this.data.slice(offset, offset + keyLen));
|
const key = new TextDecoder().decode(this.data.slice(offset, offset + keyLen));
|
||||||
@@ -3120,7 +3136,7 @@ class SSTableReader {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
/**
|
/**
|
||||||
* v0.4.2-fix: 获取索引条目对应的数据块。
|
* v0.4.1-fix: 获取索引条目对应的数据块。
|
||||||
* 块偏移/大小越界(残缺 SSTable)时返回 null,由调用方跳过而非抛 RangeError。
|
* 块偏移/大小越界(残缺 SSTable)时返回 null,由调用方跳过而非抛 RangeError。
|
||||||
*/
|
*/
|
||||||
getBlockData(entry) {
|
getBlockData(entry) {
|
||||||
|
|||||||
Vendored
+1
-1
File diff suppressed because one or more lines are too long
Vendored
+1
-1
@@ -118,7 +118,7 @@ interface MetonaPlugin {
|
|||||||
/** 销毁 */
|
/** 销毁 */
|
||||||
destroy(): void;
|
destroy(): void;
|
||||||
}
|
}
|
||||||
declare const VERSION = "0.4.3";
|
declare const VERSION = "0.4.4";
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* metona-sqlark Plugin — 插件系统
|
* metona-sqlark Plugin — 插件系统
|
||||||
|
|||||||
Vendored
+113
-97
@@ -30,7 +30,7 @@ class DatabaseError extends Error {
|
|||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// 版本
|
// 版本
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
const VERSION = '0.4.3';
|
const VERSION = '0.4.4';
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* metona-sqlark Shared WHERE Matcher — 统一的条件匹配逻辑
|
* metona-sqlark Shared WHERE Matcher — 统一的条件匹配逻辑
|
||||||
@@ -2709,7 +2709,15 @@ class BloomFilter {
|
|||||||
*
|
*
|
||||||
* 将排序后的 key-value 数据写入 SSTable 格式。
|
* 将排序后的 key-value 数据写入 SSTable 格式。
|
||||||
*
|
*
|
||||||
* SSTable 文件布局:
|
* v0.4.4 格式 v2(magic "SSTC")修复:
|
||||||
|
* - 块大小估算改用 UTF-8 字节长度(TextEncoder 预编码),
|
||||||
|
* 此前用字符串 .length(UTF-16 码元)估算而实际写入 UTF-8 字节,
|
||||||
|
* 中文内容(1 字 3 字节)导致缓冲区低估 → 写入越界崩溃
|
||||||
|
* - keyLen/valueLen 从 u16 升级为 u32(此前 >64KB 的 value 长度被截断,
|
||||||
|
* 线性格式整体错乱)
|
||||||
|
* - 大 value 单条独立成块(切分逻辑基于字节估算)
|
||||||
|
*
|
||||||
|
* SSTable 文件布局 (v2):
|
||||||
* ┌──────────────────────────────────────────────┐
|
* ┌──────────────────────────────────────────────┐
|
||||||
* │ Data Block 0 │
|
* │ Data Block 0 │
|
||||||
* │ Data Block 1 │
|
* │ Data Block 1 │
|
||||||
@@ -2723,58 +2731,56 @@ class BloomFilter {
|
|||||||
* │ - bloom_size (u32) │
|
* │ - bloom_size (u32) │
|
||||||
* │ - bloom_hash_count (u32) │
|
* │ - bloom_hash_count (u32) │
|
||||||
* │ - entry_count (u32) │
|
* │ - entry_count (u32) │
|
||||||
* │ - magic_number (u32, 0x53535442 ="SSTB")│
|
* │ - magic_number (u32, 0x53535443 ="SSTC")│
|
||||||
* │ - checksum (u32) │
|
* │ - checksum (u32) │
|
||||||
* └──────────────────────────────────────────────┘
|
* └──────────────────────────────────────────────┘
|
||||||
|
* 数据块条目: entryCount(u32) + [keyLen(u32) + key + valueLen(u32) + value]
|
||||||
|
* 索引块条目: [keyLen(u32) + key + blockOffset(u32) + blockSize(u32)]
|
||||||
*/
|
*/
|
||||||
const SSTABLE_MAGIC$1 = 0x53535442; // "SSTB"
|
/** v1 格式魔数("SSTB",u16 长度字段,兼容旧文件读取) */
|
||||||
|
const SSTABLE_MAGIC_V1 = 0x53535442;
|
||||||
|
/** v2 格式魔数("SSTC",u32 长度字段 + 字节精确估算,v0.4.4) */
|
||||||
|
const SSTABLE_MAGIC_V2 = 0x53535443;
|
||||||
const SSTABLE_FOOTER_SIZE = 32;
|
const SSTABLE_FOOTER_SIZE = 32;
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// SSTableBuilder
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
class SSTableBuilder {
|
class SSTableBuilder {
|
||||||
constructor(blockSizeLimit = 4096) {
|
constructor(blockSizeLimit = 4096) {
|
||||||
this.entries = [];
|
this.entries = [];
|
||||||
this.currentBlock = [];
|
|
||||||
this.currentBlockStartKey = '';
|
|
||||||
this.blockSizeLimit = blockSizeLimit;
|
this.blockSizeLimit = blockSizeLimit;
|
||||||
}
|
}
|
||||||
/** 添加一个 key-value 条目(必须按键排序添加) */
|
/** 添加一个 key-value 条目(必须按键排序添加) */
|
||||||
add(key, value) {
|
add(key, value) {
|
||||||
if (this.currentBlock.length === 0) {
|
|
||||||
this.currentBlockStartKey = key;
|
|
||||||
}
|
|
||||||
this.currentBlock.push([key, value]);
|
|
||||||
this.entries.push([key, value]);
|
this.entries.push([key, value]);
|
||||||
// 如果当前 Block 达到大小限制,切割
|
|
||||||
const estimated = this.estimateBlockSize();
|
|
||||||
if (estimated >= this.blockSizeLimit && this.currentBlock.length > 1) ;
|
|
||||||
}
|
}
|
||||||
/**
|
/**
|
||||||
* 构建 SSTable 文件的二进制数据。
|
* 构建 SSTable 文件的二进制数据(v2 格式)。
|
||||||
* 返回 { data: Uint8Array, indexEntries: IndexEntry[], bloomFilter: BloomFilter }
|
* 返回 { data: Uint8Array, indexEntries: IndexEntry[] }
|
||||||
*/
|
*/
|
||||||
build() {
|
build() {
|
||||||
const blocks = this.splitIntoBlocks();
|
// v0.4.4-fix: 预编码全部条目 — 块大小估算必须基于 UTF-8 字节长度,
|
||||||
|
// 字符串 .length 是 UTF-16 码元(中文 1 字 3 字节 vs 1 码元)→ 缓冲区低估越界
|
||||||
|
const encoder = new TextEncoder();
|
||||||
|
const encoded = this.entries.map(([key, value]) => ({
|
||||||
|
key,
|
||||||
|
keyBytes: encoder.encode(key),
|
||||||
|
valueBytes: encoder.encode(JSON.stringify(value)),
|
||||||
|
}));
|
||||||
|
const blocks = this.splitIntoBlocks(encoded);
|
||||||
const bloomFilter = new BloomFilter(this.entries.length);
|
const bloomFilter = new BloomFilter(this.entries.length);
|
||||||
// 预计算总大小
|
// 预计算总大小(字节)
|
||||||
let totalSize = 0;
|
let totalSize = 0;
|
||||||
const blockOffsets = [];
|
const blockOffsets = [];
|
||||||
for (const block of blocks) {
|
for (const block of blocks) {
|
||||||
blockOffsets.push(totalSize);
|
blockOffsets.push(totalSize);
|
||||||
const blockSize = this.computeBlockSize(block);
|
totalSize += this.computeBlockSize(block);
|
||||||
totalSize += blockSize;
|
|
||||||
}
|
}
|
||||||
// 索引块
|
// 索引块(块内最后一个 key 作为索引键)
|
||||||
const indexEntries = [];
|
const indexEntries = [];
|
||||||
for (let i = 0; i < blocks.length; i++) {
|
for (let i = 0; i < blocks.length; i++) {
|
||||||
const block = blocks[i];
|
const block = blocks[i];
|
||||||
const lastKey = block[block.length - 1][0];
|
|
||||||
const blockSize = this.computeBlockSize(block);
|
|
||||||
indexEntries.push({
|
indexEntries.push({
|
||||||
key: lastKey,
|
key: block[block.length - 1].key,
|
||||||
blockOffset: blockOffsets[i],
|
blockOffset: blockOffsets[i],
|
||||||
blockSize,
|
blockSize: this.computeBlockSize(block),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
const indexBlockSize = this.estimateIndexBlockSize(indexEntries);
|
const indexBlockSize = this.estimateIndexBlockSize(indexEntries);
|
||||||
@@ -2805,7 +2811,7 @@ class SSTableBuilder {
|
|||||||
view.setUint32(footerOffset + 12, bloomSize, false); // bloom_size
|
view.setUint32(footerOffset + 12, bloomSize, false); // bloom_size
|
||||||
view.setUint32(footerOffset + 16, bloomFilter.getHashCount(), false);
|
view.setUint32(footerOffset + 16, bloomFilter.getHashCount(), false);
|
||||||
view.setUint32(footerOffset + 20, this.entries.length, false);
|
view.setUint32(footerOffset + 20, this.entries.length, false);
|
||||||
view.setUint32(footerOffset + 24, SSTABLE_MAGIC$1, false);
|
view.setUint32(footerOffset + 24, SSTABLE_MAGIC_V2, false);
|
||||||
view.setUint32(footerOffset + 28, 0, false); // checksum (simplified: 0)
|
view.setUint32(footerOffset + 28, 0, false); // checksum (simplified: 0)
|
||||||
return {
|
return {
|
||||||
sstableData: new Uint8Array(buf),
|
sstableData: new Uint8Array(buf),
|
||||||
@@ -2819,12 +2825,14 @@ class SSTableBuilder {
|
|||||||
// -----------------------------------------------------------------------
|
// -----------------------------------------------------------------------
|
||||||
// 内部
|
// 内部
|
||||||
// -----------------------------------------------------------------------
|
// -----------------------------------------------------------------------
|
||||||
splitIntoBlocks() {
|
/** 按 UTF-8 字节大小切分数据块;大 value 单条独立成块 */
|
||||||
|
splitIntoBlocks(encoded) {
|
||||||
const blocks = [];
|
const blocks = [];
|
||||||
let current = [];
|
let current = [];
|
||||||
for (const entry of this.entries) {
|
for (const entry of encoded) {
|
||||||
current.push(entry);
|
current.push(entry);
|
||||||
if (this.estimateBlockSizeFromEntries(current) >= this.blockSizeLimit && current.length > 1) {
|
// v0.4.4-fix: 基于字节估算;单条超大条目(length===1)独立成块不强行切分
|
||||||
|
if (this.computeBlockSize(current) >= this.blockSizeLimit && current.length > 1) {
|
||||||
blocks.push(current.slice(0, -1));
|
blocks.push(current.slice(0, -1));
|
||||||
current = [entry];
|
current = [entry];
|
||||||
}
|
}
|
||||||
@@ -2833,22 +2841,11 @@ class SSTableBuilder {
|
|||||||
blocks.push(current);
|
blocks.push(current);
|
||||||
return blocks;
|
return blocks;
|
||||||
}
|
}
|
||||||
estimateBlockSize() {
|
/** 块字节大小:entryCount(u32) + 每对 [keyLen(u32) + key + valueLen(u32) + value] */
|
||||||
return this.estimateBlockSizeFromEntries(this.currentBlock);
|
|
||||||
}
|
|
||||||
estimateBlockSizeFromEntries(entries) {
|
|
||||||
let size = 0;
|
|
||||||
for (const [key, value] of entries) {
|
|
||||||
size += 4 + key.length + JSON.stringify(value).length;
|
|
||||||
}
|
|
||||||
return size;
|
|
||||||
}
|
|
||||||
computeBlockSize(block) {
|
computeBlockSize(block) {
|
||||||
// entryCount (u32) + 每对: keyLen(u16) + key + valueLen(u16) + value json
|
|
||||||
let size = 4;
|
let size = 4;
|
||||||
for (const [key, value] of block) {
|
for (const e of block) {
|
||||||
const json = JSON.stringify(value);
|
size += 4 + e.keyBytes.length + 4 + e.valueBytes.length;
|
||||||
size += 2 + key.length + 2 + json.length;
|
|
||||||
}
|
}
|
||||||
return size;
|
return size;
|
||||||
}
|
}
|
||||||
@@ -2856,32 +2853,30 @@ class SSTableBuilder {
|
|||||||
// entry count
|
// entry count
|
||||||
view.setUint32(offset, block.length, false);
|
view.setUint32(offset, block.length, false);
|
||||||
offset += 4;
|
offset += 4;
|
||||||
for (const [key, value] of block) {
|
for (const e of block) {
|
||||||
const encoder = new TextEncoder();
|
// v0.4.4-fix: 长度字段 u32(此前 u16 截断 >64KB 的 value)
|
||||||
const keyBytes = encoder.encode(key);
|
if (e.keyBytes.length > 0xFFFFFFFF || e.valueBytes.length > 0xFFFFFFFF) {
|
||||||
const valueBytes = encoder.encode(JSON.stringify(value));
|
throw new Error('SSTable entry too large (exceeds u32 length field)');
|
||||||
// key length
|
}
|
||||||
view.setUint16(offset, keyBytes.length, false);
|
view.setUint32(offset, e.keyBytes.length, false);
|
||||||
offset += 2;
|
offset += 4;
|
||||||
// key
|
new Uint8Array(view.buffer).set(e.keyBytes, offset);
|
||||||
new Uint8Array(view.buffer).set(keyBytes, offset);
|
offset += e.keyBytes.length;
|
||||||
offset += keyBytes.length;
|
view.setUint32(offset, e.valueBytes.length, false);
|
||||||
// value length
|
offset += 4;
|
||||||
view.setUint16(offset, valueBytes.length, false);
|
new Uint8Array(view.buffer).set(e.valueBytes, offset);
|
||||||
offset += 2;
|
offset += e.valueBytes.length;
|
||||||
// value
|
|
||||||
new Uint8Array(view.buffer).set(valueBytes, offset);
|
|
||||||
offset += valueBytes.length;
|
|
||||||
// 插入 bloom filter
|
// 插入 bloom filter
|
||||||
bloomFilter.insert(key);
|
bloomFilter.insert(e.key);
|
||||||
}
|
}
|
||||||
return offset;
|
return offset;
|
||||||
}
|
}
|
||||||
estimateIndexBlockSize(entries) {
|
estimateIndexBlockSize(entries) {
|
||||||
// entryCount(u32) + each: keyLen(u16)+key+blockOffset(u32)+blockSize(u32)
|
// entryCount(u32) + each: keyLen(u32)+key+blockOffset(u32)+blockSize(u32)
|
||||||
let size = 4;
|
let size = 4;
|
||||||
|
const encoder = new TextEncoder();
|
||||||
for (const entry of entries) {
|
for (const entry of entries) {
|
||||||
size += 2 + entry.key.length + 8;
|
size += 4 + encoder.encode(entry.key).byteLength + 8;
|
||||||
}
|
}
|
||||||
return size;
|
return size;
|
||||||
}
|
}
|
||||||
@@ -2891,8 +2886,8 @@ class SSTableBuilder {
|
|||||||
for (const entry of entries) {
|
for (const entry of entries) {
|
||||||
const encoder = new TextEncoder();
|
const encoder = new TextEncoder();
|
||||||
const keyBytes = encoder.encode(entry.key);
|
const keyBytes = encoder.encode(entry.key);
|
||||||
view.setUint16(offset, keyBytes.length, false);
|
view.setUint32(offset, keyBytes.length, false);
|
||||||
offset += 2;
|
offset += 4;
|
||||||
new Uint8Array(view.buffer).set(keyBytes, offset);
|
new Uint8Array(view.buffer).set(keyBytes, offset);
|
||||||
offset += keyBytes.length;
|
offset += keyBytes.length;
|
||||||
view.setUint32(offset, entry.blockOffset, false);
|
view.setUint32(offset, entry.blockOffset, false);
|
||||||
@@ -2907,8 +2902,10 @@ class SSTableBuilder {
|
|||||||
/**
|
/**
|
||||||
* AriaEngine SSTable Reader — 从 SSTable 二进制数据中读取
|
* AriaEngine SSTable Reader — 从 SSTable 二进制数据中读取
|
||||||
* @module engine/aria/index/sstable
|
* @module engine/aria/index/sstable
|
||||||
|
*
|
||||||
|
* v0.4.4: 支持 v1("SSTB",u16 长度字段)与 v2("SSTC",u32 长度字段)双格式,
|
||||||
|
* 旧库 v1 文件仍可读(小 value 场景无缺陷),新写入使用 v2。
|
||||||
*/
|
*/
|
||||||
const SSTABLE_MAGIC = 0x53535442; // "SSTB"
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// SSTableReader
|
// SSTableReader
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -2917,11 +2914,20 @@ class SSTableReader {
|
|||||||
this.indexEntries = [];
|
this.indexEntries = [];
|
||||||
this.entryCount = 0;
|
this.entryCount = 0;
|
||||||
this.bloomFilter = null;
|
this.bloomFilter = null;
|
||||||
|
/** 格式版本:1 = u16 长度字段(旧),2 = u32 长度字段(v0.4.4) */
|
||||||
|
this.format = 2;
|
||||||
this.data = data;
|
this.data = data;
|
||||||
this.view = new DataView(data.buffer, data.byteOffset, data.byteLength);
|
this.view = new DataView(data.buffer, data.byteOffset, data.byteLength);
|
||||||
this.meta = meta;
|
this.meta = meta;
|
||||||
this.parseFooter();
|
this.parseFooter();
|
||||||
}
|
}
|
||||||
|
/** 长度字段宽度:v2 = 4 字节 u32,v1 = 2 字节 u16 */
|
||||||
|
lenFieldSize() {
|
||||||
|
return this.format === 2 ? 4 : 2;
|
||||||
|
}
|
||||||
|
readLen(offset) {
|
||||||
|
return this.format === 2 ? this.view.getUint32(offset, false) : this.view.getUint16(offset, false);
|
||||||
|
}
|
||||||
// -----------------------------------------------------------------------
|
// -----------------------------------------------------------------------
|
||||||
// 查询
|
// 查询
|
||||||
// -----------------------------------------------------------------------
|
// -----------------------------------------------------------------------
|
||||||
@@ -2935,24 +2941,25 @@ class SSTableReader {
|
|||||||
return null;
|
return null;
|
||||||
const entry = this.indexEntries[blockIdx];
|
const entry = this.indexEntries[blockIdx];
|
||||||
const blockData = this.getBlockData(entry);
|
const blockData = this.getBlockData(entry);
|
||||||
// v0.4.2-fix: 残缺文件(meta 偏移超出实际长度)跳过该块,而非抛 RangeError
|
// v0.4.1-fix: 残缺文件(meta 偏移超出实际长度)跳过该块,而非抛 RangeError
|
||||||
if (!blockData)
|
if (!blockData)
|
||||||
return null;
|
return null;
|
||||||
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
||||||
|
const lenSize = this.lenFieldSize();
|
||||||
const entryCount = blockView.getUint32(0, false);
|
const entryCount = blockView.getUint32(0, false);
|
||||||
let offset = 4;
|
let offset = 4;
|
||||||
// 顺序扫描 block 内的条目(生产中应二分查找)
|
// 顺序扫描 block 内的条目(生产中应二分查找)
|
||||||
for (let i = 0; i < entryCount; i++) {
|
for (let i = 0; i < entryCount; i++) {
|
||||||
if (offset + 2 > blockData.byteLength)
|
if (offset + lenSize > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const keyLen = blockView.getUint16(offset, false);
|
const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + keyLen + 2 > blockData.byteLength)
|
if (offset + keyLen + lenSize > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
||||||
offset += keyLen;
|
offset += keyLen;
|
||||||
const valLen = blockView.getUint16(offset, false);
|
const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + valLen > blockData.byteLength)
|
if (offset + valLen > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const valBytes = blockData.slice(offset, offset + valLen);
|
const valBytes = blockData.slice(offset, offset + valLen);
|
||||||
@@ -2976,26 +2983,27 @@ class SSTableReader {
|
|||||||
const endBlockIdx = Math.min(this.indexEntries.length - 1, this.locateBlockLE(endKey));
|
const endBlockIdx = Math.min(this.indexEntries.length - 1, this.locateBlockLE(endKey));
|
||||||
if (startBlockIdx < 0 || endBlockIdx < 0 || startBlockIdx > endBlockIdx)
|
if (startBlockIdx < 0 || endBlockIdx < 0 || startBlockIdx > endBlockIdx)
|
||||||
return;
|
return;
|
||||||
|
const lenSize = this.lenFieldSize();
|
||||||
for (let bi = startBlockIdx; bi <= endBlockIdx && bi >= 0; bi++) {
|
for (let bi = startBlockIdx; bi <= endBlockIdx && bi >= 0; bi++) {
|
||||||
const entry = this.indexEntries[bi];
|
const entry = this.indexEntries[bi];
|
||||||
const blockData = this.getBlockData(entry);
|
const blockData = this.getBlockData(entry);
|
||||||
// v0.4.2-fix: 残缺块跳过(rangeScan 继续后续块,不抛异常)
|
// v0.4.1-fix: 残缺块跳过(rangeScan 继续后续块,不抛异常)
|
||||||
if (!blockData)
|
if (!blockData)
|
||||||
continue;
|
continue;
|
||||||
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
||||||
const blockEntryCount = blockView.getUint32(0, false);
|
const blockEntryCount = blockView.getUint32(0, false);
|
||||||
let offset = 4;
|
let offset = 4;
|
||||||
for (let i = 0; i < blockEntryCount; i++) {
|
for (let i = 0; i < blockEntryCount; i++) {
|
||||||
if (offset + 2 > blockData.byteLength)
|
if (offset + lenSize > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const keyLen = blockView.getUint16(offset, false);
|
const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + keyLen + 2 > blockData.byteLength)
|
if (offset + keyLen + lenSize > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
||||||
offset += keyLen;
|
offset += keyLen;
|
||||||
const valLen = blockView.getUint16(offset, false);
|
const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + valLen > blockData.byteLength)
|
if (offset + valLen > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const valBytes = blockData.slice(offset, offset + valLen);
|
const valBytes = blockData.slice(offset, offset + valLen);
|
||||||
@@ -3014,25 +3022,26 @@ class SSTableReader {
|
|||||||
}
|
}
|
||||||
/** 扫描所有条目 */
|
/** 扫描所有条目 */
|
||||||
scanAll(callback) {
|
scanAll(callback) {
|
||||||
|
const lenSize = this.lenFieldSize();
|
||||||
for (const entry of this.indexEntries) {
|
for (const entry of this.indexEntries) {
|
||||||
const blockData = this.getBlockData(entry);
|
const blockData = this.getBlockData(entry);
|
||||||
// v0.4.2-fix: 残缺块跳过(scanAll 继续后续块,不抛异常)
|
// v0.4.1-fix: 残缺块跳过(scanAll 继续后续块,不抛异常)
|
||||||
if (!blockData)
|
if (!blockData)
|
||||||
continue;
|
continue;
|
||||||
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
||||||
const blockEntryCount = blockView.getUint32(0, false);
|
const blockEntryCount = blockView.getUint32(0, false);
|
||||||
let offset = 4;
|
let offset = 4;
|
||||||
for (let i = 0; i < blockEntryCount; i++) {
|
for (let i = 0; i < blockEntryCount; i++) {
|
||||||
if (offset + 2 > blockData.byteLength)
|
if (offset + lenSize > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const keyLen = blockView.getUint16(offset, false);
|
const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + keyLen + 2 > blockData.byteLength)
|
if (offset + keyLen + lenSize > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
||||||
offset += keyLen;
|
offset += keyLen;
|
||||||
const valLen = blockView.getUint16(offset, false);
|
const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + valLen > blockData.byteLength)
|
if (offset + valLen > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const valBytes = blockData.slice(offset, offset + valLen);
|
const valBytes = blockData.slice(offset, offset + valLen);
|
||||||
@@ -3063,10 +3072,16 @@ class SSTableReader {
|
|||||||
throw new Error('SSTable too small: missing footer');
|
throw new Error('SSTable too small: missing footer');
|
||||||
}
|
}
|
||||||
const footerOffset = this.data.byteLength - 32;
|
const footerOffset = this.data.byteLength - 32;
|
||||||
// 验证魔数
|
// 验证魔数(v1 "SSTB" / v2 "SSTC")
|
||||||
const magic = this.view.getUint32(footerOffset + 24, false);
|
const magic = this.view.getUint32(footerOffset + 24, false);
|
||||||
if (magic !== SSTABLE_MAGIC) {
|
if (magic === SSTABLE_MAGIC_V1) {
|
||||||
throw new Error(`Invalid SSTable magic: expected ${SSTABLE_MAGIC}, got ${magic}`);
|
this.format = 1;
|
||||||
|
}
|
||||||
|
else if (magic === SSTABLE_MAGIC_V2) {
|
||||||
|
this.format = 2;
|
||||||
|
}
|
||||||
|
else {
|
||||||
|
throw new Error(`Invalid SSTable magic: expected ${SSTABLE_MAGIC_V1} or ${SSTABLE_MAGIC_V2}, got ${magic}`);
|
||||||
}
|
}
|
||||||
const indexOffset = this.view.getUint32(footerOffset, false);
|
const indexOffset = this.view.getUint32(footerOffset, false);
|
||||||
const indexSize = this.view.getUint32(footerOffset + 4, false);
|
const indexSize = this.view.getUint32(footerOffset + 4, false);
|
||||||
@@ -3074,7 +3089,7 @@ class SSTableReader {
|
|||||||
const bloomSize = this.view.getUint32(footerOffset + 12, false);
|
const bloomSize = this.view.getUint32(footerOffset + 12, false);
|
||||||
const bloomHashCount = this.view.getUint32(footerOffset + 16, false);
|
const bloomHashCount = this.view.getUint32(footerOffset + 16, false);
|
||||||
this.entryCount = this.view.getUint32(footerOffset + 20, false);
|
this.entryCount = this.view.getUint32(footerOffset + 20, false);
|
||||||
// v0.4.2-fix: 完整性校验 — 索引块必须完全落在文件内,否则视为残缺文件跳过
|
// v0.4.1-fix: 完整性校验 — 索引块必须完全落在文件内,否则视为残缺文件跳过
|
||||||
if (indexOffset + 4 > this.data.byteLength || indexOffset + indexSize > this.data.byteLength) {
|
if (indexOffset + 4 > this.data.byteLength || indexOffset + indexSize > this.data.byteLength) {
|
||||||
return; // 残缺文件:无索引块可读,get/rangeScan 均返回空
|
return; // 残缺文件:无索引块可读,get/rangeScan 均返回空
|
||||||
}
|
}
|
||||||
@@ -3094,13 +3109,14 @@ class SSTableReader {
|
|||||||
parseIndexBlock(offset, _size) {
|
parseIndexBlock(offset, _size) {
|
||||||
const entryCount = this.view.getUint32(offset, false);
|
const entryCount = this.view.getUint32(offset, false);
|
||||||
offset += 4;
|
offset += 4;
|
||||||
|
const lenSize = this.lenFieldSize();
|
||||||
for (let i = 0; i < entryCount; i++) {
|
for (let i = 0; i < entryCount; i++) {
|
||||||
// v0.4.2-fix: 索引条目越界(keyLen/blockOffset/blockSize 超过文件长度)时中止解析,
|
// v0.4.1-fix: 索引条目越界(keyLen/blockOffset/blockSize 超过文件长度)时中止解析,
|
||||||
// 已解析的有效条目仍可用于查询
|
// 已解析的有效条目仍可用于查询
|
||||||
if (offset + 2 > this.data.byteLength)
|
if (offset + lenSize > this.data.byteLength)
|
||||||
break;
|
break;
|
||||||
const keyLen = this.view.getUint16(offset, false);
|
const keyLen = this.readLen(offset);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + keyLen + 8 > this.data.byteLength)
|
if (offset + keyLen + 8 > this.data.byteLength)
|
||||||
break;
|
break;
|
||||||
const key = new TextDecoder().decode(this.data.slice(offset, offset + keyLen));
|
const key = new TextDecoder().decode(this.data.slice(offset, offset + keyLen));
|
||||||
@@ -3116,7 +3132,7 @@ class SSTableReader {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
/**
|
/**
|
||||||
* v0.4.2-fix: 获取索引条目对应的数据块。
|
* v0.4.1-fix: 获取索引条目对应的数据块。
|
||||||
* 块偏移/大小越界(残缺 SSTable)时返回 null,由调用方跳过而非抛 RangeError。
|
* 块偏移/大小越界(残缺 SSTable)时返回 null,由调用方跳过而非抛 RangeError。
|
||||||
*/
|
*/
|
||||||
getBlockData(entry) {
|
getBlockData(entry) {
|
||||||
|
|||||||
Vendored
+1
-1
File diff suppressed because one or more lines are too long
Vendored
+113
-97
@@ -36,7 +36,7 @@
|
|||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// 版本
|
// 版本
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
const VERSION = '0.4.3';
|
const VERSION = '0.4.4';
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* metona-sqlark Shared WHERE Matcher — 统一的条件匹配逻辑
|
* metona-sqlark Shared WHERE Matcher — 统一的条件匹配逻辑
|
||||||
@@ -2715,7 +2715,15 @@
|
|||||||
*
|
*
|
||||||
* 将排序后的 key-value 数据写入 SSTable 格式。
|
* 将排序后的 key-value 数据写入 SSTable 格式。
|
||||||
*
|
*
|
||||||
* SSTable 文件布局:
|
* v0.4.4 格式 v2(magic "SSTC")修复:
|
||||||
|
* - 块大小估算改用 UTF-8 字节长度(TextEncoder 预编码),
|
||||||
|
* 此前用字符串 .length(UTF-16 码元)估算而实际写入 UTF-8 字节,
|
||||||
|
* 中文内容(1 字 3 字节)导致缓冲区低估 → 写入越界崩溃
|
||||||
|
* - keyLen/valueLen 从 u16 升级为 u32(此前 >64KB 的 value 长度被截断,
|
||||||
|
* 线性格式整体错乱)
|
||||||
|
* - 大 value 单条独立成块(切分逻辑基于字节估算)
|
||||||
|
*
|
||||||
|
* SSTable 文件布局 (v2):
|
||||||
* ┌──────────────────────────────────────────────┐
|
* ┌──────────────────────────────────────────────┐
|
||||||
* │ Data Block 0 │
|
* │ Data Block 0 │
|
||||||
* │ Data Block 1 │
|
* │ Data Block 1 │
|
||||||
@@ -2729,58 +2737,56 @@
|
|||||||
* │ - bloom_size (u32) │
|
* │ - bloom_size (u32) │
|
||||||
* │ - bloom_hash_count (u32) │
|
* │ - bloom_hash_count (u32) │
|
||||||
* │ - entry_count (u32) │
|
* │ - entry_count (u32) │
|
||||||
* │ - magic_number (u32, 0x53535442 ="SSTB")│
|
* │ - magic_number (u32, 0x53535443 ="SSTC")│
|
||||||
* │ - checksum (u32) │
|
* │ - checksum (u32) │
|
||||||
* └──────────────────────────────────────────────┘
|
* └──────────────────────────────────────────────┘
|
||||||
|
* 数据块条目: entryCount(u32) + [keyLen(u32) + key + valueLen(u32) + value]
|
||||||
|
* 索引块条目: [keyLen(u32) + key + blockOffset(u32) + blockSize(u32)]
|
||||||
*/
|
*/
|
||||||
const SSTABLE_MAGIC$1 = 0x53535442; // "SSTB"
|
/** v1 格式魔数("SSTB",u16 长度字段,兼容旧文件读取) */
|
||||||
|
const SSTABLE_MAGIC_V1 = 0x53535442;
|
||||||
|
/** v2 格式魔数("SSTC",u32 长度字段 + 字节精确估算,v0.4.4) */
|
||||||
|
const SSTABLE_MAGIC_V2 = 0x53535443;
|
||||||
const SSTABLE_FOOTER_SIZE = 32;
|
const SSTABLE_FOOTER_SIZE = 32;
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// SSTableBuilder
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
class SSTableBuilder {
|
class SSTableBuilder {
|
||||||
constructor(blockSizeLimit = 4096) {
|
constructor(blockSizeLimit = 4096) {
|
||||||
this.entries = [];
|
this.entries = [];
|
||||||
this.currentBlock = [];
|
|
||||||
this.currentBlockStartKey = '';
|
|
||||||
this.blockSizeLimit = blockSizeLimit;
|
this.blockSizeLimit = blockSizeLimit;
|
||||||
}
|
}
|
||||||
/** 添加一个 key-value 条目(必须按键排序添加) */
|
/** 添加一个 key-value 条目(必须按键排序添加) */
|
||||||
add(key, value) {
|
add(key, value) {
|
||||||
if (this.currentBlock.length === 0) {
|
|
||||||
this.currentBlockStartKey = key;
|
|
||||||
}
|
|
||||||
this.currentBlock.push([key, value]);
|
|
||||||
this.entries.push([key, value]);
|
this.entries.push([key, value]);
|
||||||
// 如果当前 Block 达到大小限制,切割
|
|
||||||
const estimated = this.estimateBlockSize();
|
|
||||||
if (estimated >= this.blockSizeLimit && this.currentBlock.length > 1) ;
|
|
||||||
}
|
}
|
||||||
/**
|
/**
|
||||||
* 构建 SSTable 文件的二进制数据。
|
* 构建 SSTable 文件的二进制数据(v2 格式)。
|
||||||
* 返回 { data: Uint8Array, indexEntries: IndexEntry[], bloomFilter: BloomFilter }
|
* 返回 { data: Uint8Array, indexEntries: IndexEntry[] }
|
||||||
*/
|
*/
|
||||||
build() {
|
build() {
|
||||||
const blocks = this.splitIntoBlocks();
|
// v0.4.4-fix: 预编码全部条目 — 块大小估算必须基于 UTF-8 字节长度,
|
||||||
|
// 字符串 .length 是 UTF-16 码元(中文 1 字 3 字节 vs 1 码元)→ 缓冲区低估越界
|
||||||
|
const encoder = new TextEncoder();
|
||||||
|
const encoded = this.entries.map(([key, value]) => ({
|
||||||
|
key,
|
||||||
|
keyBytes: encoder.encode(key),
|
||||||
|
valueBytes: encoder.encode(JSON.stringify(value)),
|
||||||
|
}));
|
||||||
|
const blocks = this.splitIntoBlocks(encoded);
|
||||||
const bloomFilter = new BloomFilter(this.entries.length);
|
const bloomFilter = new BloomFilter(this.entries.length);
|
||||||
// 预计算总大小
|
// 预计算总大小(字节)
|
||||||
let totalSize = 0;
|
let totalSize = 0;
|
||||||
const blockOffsets = [];
|
const blockOffsets = [];
|
||||||
for (const block of blocks) {
|
for (const block of blocks) {
|
||||||
blockOffsets.push(totalSize);
|
blockOffsets.push(totalSize);
|
||||||
const blockSize = this.computeBlockSize(block);
|
totalSize += this.computeBlockSize(block);
|
||||||
totalSize += blockSize;
|
|
||||||
}
|
}
|
||||||
// 索引块
|
// 索引块(块内最后一个 key 作为索引键)
|
||||||
const indexEntries = [];
|
const indexEntries = [];
|
||||||
for (let i = 0; i < blocks.length; i++) {
|
for (let i = 0; i < blocks.length; i++) {
|
||||||
const block = blocks[i];
|
const block = blocks[i];
|
||||||
const lastKey = block[block.length - 1][0];
|
|
||||||
const blockSize = this.computeBlockSize(block);
|
|
||||||
indexEntries.push({
|
indexEntries.push({
|
||||||
key: lastKey,
|
key: block[block.length - 1].key,
|
||||||
blockOffset: blockOffsets[i],
|
blockOffset: blockOffsets[i],
|
||||||
blockSize,
|
blockSize: this.computeBlockSize(block),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
const indexBlockSize = this.estimateIndexBlockSize(indexEntries);
|
const indexBlockSize = this.estimateIndexBlockSize(indexEntries);
|
||||||
@@ -2811,7 +2817,7 @@
|
|||||||
view.setUint32(footerOffset + 12, bloomSize, false); // bloom_size
|
view.setUint32(footerOffset + 12, bloomSize, false); // bloom_size
|
||||||
view.setUint32(footerOffset + 16, bloomFilter.getHashCount(), false);
|
view.setUint32(footerOffset + 16, bloomFilter.getHashCount(), false);
|
||||||
view.setUint32(footerOffset + 20, this.entries.length, false);
|
view.setUint32(footerOffset + 20, this.entries.length, false);
|
||||||
view.setUint32(footerOffset + 24, SSTABLE_MAGIC$1, false);
|
view.setUint32(footerOffset + 24, SSTABLE_MAGIC_V2, false);
|
||||||
view.setUint32(footerOffset + 28, 0, false); // checksum (simplified: 0)
|
view.setUint32(footerOffset + 28, 0, false); // checksum (simplified: 0)
|
||||||
return {
|
return {
|
||||||
sstableData: new Uint8Array(buf),
|
sstableData: new Uint8Array(buf),
|
||||||
@@ -2825,12 +2831,14 @@
|
|||||||
// -----------------------------------------------------------------------
|
// -----------------------------------------------------------------------
|
||||||
// 内部
|
// 内部
|
||||||
// -----------------------------------------------------------------------
|
// -----------------------------------------------------------------------
|
||||||
splitIntoBlocks() {
|
/** 按 UTF-8 字节大小切分数据块;大 value 单条独立成块 */
|
||||||
|
splitIntoBlocks(encoded) {
|
||||||
const blocks = [];
|
const blocks = [];
|
||||||
let current = [];
|
let current = [];
|
||||||
for (const entry of this.entries) {
|
for (const entry of encoded) {
|
||||||
current.push(entry);
|
current.push(entry);
|
||||||
if (this.estimateBlockSizeFromEntries(current) >= this.blockSizeLimit && current.length > 1) {
|
// v0.4.4-fix: 基于字节估算;单条超大条目(length===1)独立成块不强行切分
|
||||||
|
if (this.computeBlockSize(current) >= this.blockSizeLimit && current.length > 1) {
|
||||||
blocks.push(current.slice(0, -1));
|
blocks.push(current.slice(0, -1));
|
||||||
current = [entry];
|
current = [entry];
|
||||||
}
|
}
|
||||||
@@ -2839,22 +2847,11 @@
|
|||||||
blocks.push(current);
|
blocks.push(current);
|
||||||
return blocks;
|
return blocks;
|
||||||
}
|
}
|
||||||
estimateBlockSize() {
|
/** 块字节大小:entryCount(u32) + 每对 [keyLen(u32) + key + valueLen(u32) + value] */
|
||||||
return this.estimateBlockSizeFromEntries(this.currentBlock);
|
|
||||||
}
|
|
||||||
estimateBlockSizeFromEntries(entries) {
|
|
||||||
let size = 0;
|
|
||||||
for (const [key, value] of entries) {
|
|
||||||
size += 4 + key.length + JSON.stringify(value).length;
|
|
||||||
}
|
|
||||||
return size;
|
|
||||||
}
|
|
||||||
computeBlockSize(block) {
|
computeBlockSize(block) {
|
||||||
// entryCount (u32) + 每对: keyLen(u16) + key + valueLen(u16) + value json
|
|
||||||
let size = 4;
|
let size = 4;
|
||||||
for (const [key, value] of block) {
|
for (const e of block) {
|
||||||
const json = JSON.stringify(value);
|
size += 4 + e.keyBytes.length + 4 + e.valueBytes.length;
|
||||||
size += 2 + key.length + 2 + json.length;
|
|
||||||
}
|
}
|
||||||
return size;
|
return size;
|
||||||
}
|
}
|
||||||
@@ -2862,32 +2859,30 @@
|
|||||||
// entry count
|
// entry count
|
||||||
view.setUint32(offset, block.length, false);
|
view.setUint32(offset, block.length, false);
|
||||||
offset += 4;
|
offset += 4;
|
||||||
for (const [key, value] of block) {
|
for (const e of block) {
|
||||||
const encoder = new TextEncoder();
|
// v0.4.4-fix: 长度字段 u32(此前 u16 截断 >64KB 的 value)
|
||||||
const keyBytes = encoder.encode(key);
|
if (e.keyBytes.length > 0xFFFFFFFF || e.valueBytes.length > 0xFFFFFFFF) {
|
||||||
const valueBytes = encoder.encode(JSON.stringify(value));
|
throw new Error('SSTable entry too large (exceeds u32 length field)');
|
||||||
// key length
|
}
|
||||||
view.setUint16(offset, keyBytes.length, false);
|
view.setUint32(offset, e.keyBytes.length, false);
|
||||||
offset += 2;
|
offset += 4;
|
||||||
// key
|
new Uint8Array(view.buffer).set(e.keyBytes, offset);
|
||||||
new Uint8Array(view.buffer).set(keyBytes, offset);
|
offset += e.keyBytes.length;
|
||||||
offset += keyBytes.length;
|
view.setUint32(offset, e.valueBytes.length, false);
|
||||||
// value length
|
offset += 4;
|
||||||
view.setUint16(offset, valueBytes.length, false);
|
new Uint8Array(view.buffer).set(e.valueBytes, offset);
|
||||||
offset += 2;
|
offset += e.valueBytes.length;
|
||||||
// value
|
|
||||||
new Uint8Array(view.buffer).set(valueBytes, offset);
|
|
||||||
offset += valueBytes.length;
|
|
||||||
// 插入 bloom filter
|
// 插入 bloom filter
|
||||||
bloomFilter.insert(key);
|
bloomFilter.insert(e.key);
|
||||||
}
|
}
|
||||||
return offset;
|
return offset;
|
||||||
}
|
}
|
||||||
estimateIndexBlockSize(entries) {
|
estimateIndexBlockSize(entries) {
|
||||||
// entryCount(u32) + each: keyLen(u16)+key+blockOffset(u32)+blockSize(u32)
|
// entryCount(u32) + each: keyLen(u32)+key+blockOffset(u32)+blockSize(u32)
|
||||||
let size = 4;
|
let size = 4;
|
||||||
|
const encoder = new TextEncoder();
|
||||||
for (const entry of entries) {
|
for (const entry of entries) {
|
||||||
size += 2 + entry.key.length + 8;
|
size += 4 + encoder.encode(entry.key).byteLength + 8;
|
||||||
}
|
}
|
||||||
return size;
|
return size;
|
||||||
}
|
}
|
||||||
@@ -2897,8 +2892,8 @@
|
|||||||
for (const entry of entries) {
|
for (const entry of entries) {
|
||||||
const encoder = new TextEncoder();
|
const encoder = new TextEncoder();
|
||||||
const keyBytes = encoder.encode(entry.key);
|
const keyBytes = encoder.encode(entry.key);
|
||||||
view.setUint16(offset, keyBytes.length, false);
|
view.setUint32(offset, keyBytes.length, false);
|
||||||
offset += 2;
|
offset += 4;
|
||||||
new Uint8Array(view.buffer).set(keyBytes, offset);
|
new Uint8Array(view.buffer).set(keyBytes, offset);
|
||||||
offset += keyBytes.length;
|
offset += keyBytes.length;
|
||||||
view.setUint32(offset, entry.blockOffset, false);
|
view.setUint32(offset, entry.blockOffset, false);
|
||||||
@@ -2913,8 +2908,10 @@
|
|||||||
/**
|
/**
|
||||||
* AriaEngine SSTable Reader — 从 SSTable 二进制数据中读取
|
* AriaEngine SSTable Reader — 从 SSTable 二进制数据中读取
|
||||||
* @module engine/aria/index/sstable
|
* @module engine/aria/index/sstable
|
||||||
|
*
|
||||||
|
* v0.4.4: 支持 v1("SSTB",u16 长度字段)与 v2("SSTC",u32 长度字段)双格式,
|
||||||
|
* 旧库 v1 文件仍可读(小 value 场景无缺陷),新写入使用 v2。
|
||||||
*/
|
*/
|
||||||
const SSTABLE_MAGIC = 0x53535442; // "SSTB"
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// SSTableReader
|
// SSTableReader
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -2923,11 +2920,20 @@
|
|||||||
this.indexEntries = [];
|
this.indexEntries = [];
|
||||||
this.entryCount = 0;
|
this.entryCount = 0;
|
||||||
this.bloomFilter = null;
|
this.bloomFilter = null;
|
||||||
|
/** 格式版本:1 = u16 长度字段(旧),2 = u32 长度字段(v0.4.4) */
|
||||||
|
this.format = 2;
|
||||||
this.data = data;
|
this.data = data;
|
||||||
this.view = new DataView(data.buffer, data.byteOffset, data.byteLength);
|
this.view = new DataView(data.buffer, data.byteOffset, data.byteLength);
|
||||||
this.meta = meta;
|
this.meta = meta;
|
||||||
this.parseFooter();
|
this.parseFooter();
|
||||||
}
|
}
|
||||||
|
/** 长度字段宽度:v2 = 4 字节 u32,v1 = 2 字节 u16 */
|
||||||
|
lenFieldSize() {
|
||||||
|
return this.format === 2 ? 4 : 2;
|
||||||
|
}
|
||||||
|
readLen(offset) {
|
||||||
|
return this.format === 2 ? this.view.getUint32(offset, false) : this.view.getUint16(offset, false);
|
||||||
|
}
|
||||||
// -----------------------------------------------------------------------
|
// -----------------------------------------------------------------------
|
||||||
// 查询
|
// 查询
|
||||||
// -----------------------------------------------------------------------
|
// -----------------------------------------------------------------------
|
||||||
@@ -2941,24 +2947,25 @@
|
|||||||
return null;
|
return null;
|
||||||
const entry = this.indexEntries[blockIdx];
|
const entry = this.indexEntries[blockIdx];
|
||||||
const blockData = this.getBlockData(entry);
|
const blockData = this.getBlockData(entry);
|
||||||
// v0.4.2-fix: 残缺文件(meta 偏移超出实际长度)跳过该块,而非抛 RangeError
|
// v0.4.1-fix: 残缺文件(meta 偏移超出实际长度)跳过该块,而非抛 RangeError
|
||||||
if (!blockData)
|
if (!blockData)
|
||||||
return null;
|
return null;
|
||||||
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
||||||
|
const lenSize = this.lenFieldSize();
|
||||||
const entryCount = blockView.getUint32(0, false);
|
const entryCount = blockView.getUint32(0, false);
|
||||||
let offset = 4;
|
let offset = 4;
|
||||||
// 顺序扫描 block 内的条目(生产中应二分查找)
|
// 顺序扫描 block 内的条目(生产中应二分查找)
|
||||||
for (let i = 0; i < entryCount; i++) {
|
for (let i = 0; i < entryCount; i++) {
|
||||||
if (offset + 2 > blockData.byteLength)
|
if (offset + lenSize > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const keyLen = blockView.getUint16(offset, false);
|
const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + keyLen + 2 > blockData.byteLength)
|
if (offset + keyLen + lenSize > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
||||||
offset += keyLen;
|
offset += keyLen;
|
||||||
const valLen = blockView.getUint16(offset, false);
|
const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + valLen > blockData.byteLength)
|
if (offset + valLen > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const valBytes = blockData.slice(offset, offset + valLen);
|
const valBytes = blockData.slice(offset, offset + valLen);
|
||||||
@@ -2982,26 +2989,27 @@
|
|||||||
const endBlockIdx = Math.min(this.indexEntries.length - 1, this.locateBlockLE(endKey));
|
const endBlockIdx = Math.min(this.indexEntries.length - 1, this.locateBlockLE(endKey));
|
||||||
if (startBlockIdx < 0 || endBlockIdx < 0 || startBlockIdx > endBlockIdx)
|
if (startBlockIdx < 0 || endBlockIdx < 0 || startBlockIdx > endBlockIdx)
|
||||||
return;
|
return;
|
||||||
|
const lenSize = this.lenFieldSize();
|
||||||
for (let bi = startBlockIdx; bi <= endBlockIdx && bi >= 0; bi++) {
|
for (let bi = startBlockIdx; bi <= endBlockIdx && bi >= 0; bi++) {
|
||||||
const entry = this.indexEntries[bi];
|
const entry = this.indexEntries[bi];
|
||||||
const blockData = this.getBlockData(entry);
|
const blockData = this.getBlockData(entry);
|
||||||
// v0.4.2-fix: 残缺块跳过(rangeScan 继续后续块,不抛异常)
|
// v0.4.1-fix: 残缺块跳过(rangeScan 继续后续块,不抛异常)
|
||||||
if (!blockData)
|
if (!blockData)
|
||||||
continue;
|
continue;
|
||||||
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
||||||
const blockEntryCount = blockView.getUint32(0, false);
|
const blockEntryCount = blockView.getUint32(0, false);
|
||||||
let offset = 4;
|
let offset = 4;
|
||||||
for (let i = 0; i < blockEntryCount; i++) {
|
for (let i = 0; i < blockEntryCount; i++) {
|
||||||
if (offset + 2 > blockData.byteLength)
|
if (offset + lenSize > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const keyLen = blockView.getUint16(offset, false);
|
const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + keyLen + 2 > blockData.byteLength)
|
if (offset + keyLen + lenSize > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
||||||
offset += keyLen;
|
offset += keyLen;
|
||||||
const valLen = blockView.getUint16(offset, false);
|
const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + valLen > blockData.byteLength)
|
if (offset + valLen > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const valBytes = blockData.slice(offset, offset + valLen);
|
const valBytes = blockData.slice(offset, offset + valLen);
|
||||||
@@ -3020,25 +3028,26 @@
|
|||||||
}
|
}
|
||||||
/** 扫描所有条目 */
|
/** 扫描所有条目 */
|
||||||
scanAll(callback) {
|
scanAll(callback) {
|
||||||
|
const lenSize = this.lenFieldSize();
|
||||||
for (const entry of this.indexEntries) {
|
for (const entry of this.indexEntries) {
|
||||||
const blockData = this.getBlockData(entry);
|
const blockData = this.getBlockData(entry);
|
||||||
// v0.4.2-fix: 残缺块跳过(scanAll 继续后续块,不抛异常)
|
// v0.4.1-fix: 残缺块跳过(scanAll 继续后续块,不抛异常)
|
||||||
if (!blockData)
|
if (!blockData)
|
||||||
continue;
|
continue;
|
||||||
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
||||||
const blockEntryCount = blockView.getUint32(0, false);
|
const blockEntryCount = blockView.getUint32(0, false);
|
||||||
let offset = 4;
|
let offset = 4;
|
||||||
for (let i = 0; i < blockEntryCount; i++) {
|
for (let i = 0; i < blockEntryCount; i++) {
|
||||||
if (offset + 2 > blockData.byteLength)
|
if (offset + lenSize > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const keyLen = blockView.getUint16(offset, false);
|
const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + keyLen + 2 > blockData.byteLength)
|
if (offset + keyLen + lenSize > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
||||||
offset += keyLen;
|
offset += keyLen;
|
||||||
const valLen = blockView.getUint16(offset, false);
|
const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + valLen > blockData.byteLength)
|
if (offset + valLen > blockData.byteLength)
|
||||||
break;
|
break;
|
||||||
const valBytes = blockData.slice(offset, offset + valLen);
|
const valBytes = blockData.slice(offset, offset + valLen);
|
||||||
@@ -3069,10 +3078,16 @@
|
|||||||
throw new Error('SSTable too small: missing footer');
|
throw new Error('SSTable too small: missing footer');
|
||||||
}
|
}
|
||||||
const footerOffset = this.data.byteLength - 32;
|
const footerOffset = this.data.byteLength - 32;
|
||||||
// 验证魔数
|
// 验证魔数(v1 "SSTB" / v2 "SSTC")
|
||||||
const magic = this.view.getUint32(footerOffset + 24, false);
|
const magic = this.view.getUint32(footerOffset + 24, false);
|
||||||
if (magic !== SSTABLE_MAGIC) {
|
if (magic === SSTABLE_MAGIC_V1) {
|
||||||
throw new Error(`Invalid SSTable magic: expected ${SSTABLE_MAGIC}, got ${magic}`);
|
this.format = 1;
|
||||||
|
}
|
||||||
|
else if (magic === SSTABLE_MAGIC_V2) {
|
||||||
|
this.format = 2;
|
||||||
|
}
|
||||||
|
else {
|
||||||
|
throw new Error(`Invalid SSTable magic: expected ${SSTABLE_MAGIC_V1} or ${SSTABLE_MAGIC_V2}, got ${magic}`);
|
||||||
}
|
}
|
||||||
const indexOffset = this.view.getUint32(footerOffset, false);
|
const indexOffset = this.view.getUint32(footerOffset, false);
|
||||||
const indexSize = this.view.getUint32(footerOffset + 4, false);
|
const indexSize = this.view.getUint32(footerOffset + 4, false);
|
||||||
@@ -3080,7 +3095,7 @@
|
|||||||
const bloomSize = this.view.getUint32(footerOffset + 12, false);
|
const bloomSize = this.view.getUint32(footerOffset + 12, false);
|
||||||
const bloomHashCount = this.view.getUint32(footerOffset + 16, false);
|
const bloomHashCount = this.view.getUint32(footerOffset + 16, false);
|
||||||
this.entryCount = this.view.getUint32(footerOffset + 20, false);
|
this.entryCount = this.view.getUint32(footerOffset + 20, false);
|
||||||
// v0.4.2-fix: 完整性校验 — 索引块必须完全落在文件内,否则视为残缺文件跳过
|
// v0.4.1-fix: 完整性校验 — 索引块必须完全落在文件内,否则视为残缺文件跳过
|
||||||
if (indexOffset + 4 > this.data.byteLength || indexOffset + indexSize > this.data.byteLength) {
|
if (indexOffset + 4 > this.data.byteLength || indexOffset + indexSize > this.data.byteLength) {
|
||||||
return; // 残缺文件:无索引块可读,get/rangeScan 均返回空
|
return; // 残缺文件:无索引块可读,get/rangeScan 均返回空
|
||||||
}
|
}
|
||||||
@@ -3100,13 +3115,14 @@
|
|||||||
parseIndexBlock(offset, _size) {
|
parseIndexBlock(offset, _size) {
|
||||||
const entryCount = this.view.getUint32(offset, false);
|
const entryCount = this.view.getUint32(offset, false);
|
||||||
offset += 4;
|
offset += 4;
|
||||||
|
const lenSize = this.lenFieldSize();
|
||||||
for (let i = 0; i < entryCount; i++) {
|
for (let i = 0; i < entryCount; i++) {
|
||||||
// v0.4.2-fix: 索引条目越界(keyLen/blockOffset/blockSize 超过文件长度)时中止解析,
|
// v0.4.1-fix: 索引条目越界(keyLen/blockOffset/blockSize 超过文件长度)时中止解析,
|
||||||
// 已解析的有效条目仍可用于查询
|
// 已解析的有效条目仍可用于查询
|
||||||
if (offset + 2 > this.data.byteLength)
|
if (offset + lenSize > this.data.byteLength)
|
||||||
break;
|
break;
|
||||||
const keyLen = this.view.getUint16(offset, false);
|
const keyLen = this.readLen(offset);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + keyLen + 8 > this.data.byteLength)
|
if (offset + keyLen + 8 > this.data.byteLength)
|
||||||
break;
|
break;
|
||||||
const key = new TextDecoder().decode(this.data.slice(offset, offset + keyLen));
|
const key = new TextDecoder().decode(this.data.slice(offset, offset + keyLen));
|
||||||
@@ -3122,7 +3138,7 @@
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
/**
|
/**
|
||||||
* v0.4.2-fix: 获取索引条目对应的数据块。
|
* v0.4.1-fix: 获取索引条目对应的数据块。
|
||||||
* 块偏移/大小越界(残缺 SSTable)时返回 null,由调用方跳过而非抛 RangeError。
|
* 块偏移/大小越界(残缺 SSTable)时返回 null,由调用方跳过而非抛 RangeError。
|
||||||
*/
|
*/
|
||||||
getBlockData(entry) {
|
getBlockData(entry) {
|
||||||
|
|||||||
Vendored
+1
-1
File diff suppressed because one or more lines are too long
Vendored
+1
-1
File diff suppressed because one or more lines are too long
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@metona-team/metona-sqlark",
|
"name": "@metona-team/metona-sqlark",
|
||||||
"version": "0.4.3",
|
"version": "0.4.4",
|
||||||
"description": "Frontend SQL database with in-memory and disk dual-mode storage",
|
"description": "Frontend SQL database with in-memory and disk dual-mode storage",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"main": "dist/metona-sqlark.cjs",
|
"main": "dist/metona-sqlark.cjs",
|
||||||
|
|||||||
+1
-1
@@ -3,7 +3,7 @@
|
|||||||
<head>
|
<head>
|
||||||
<meta charset="UTF-8">
|
<meta charset="UTF-8">
|
||||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
<title>⚡ 性能基准 — MetonaSqlark v0.4.3</title>
|
<title>⚡ 性能基准 — MetonaSqlark v0.4.4</title>
|
||||||
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><rect width='32' height='32' rx='8' fill='%236366f1'/><text x='16' y='22' text-anchor='middle' font-size='20' fill='white'>◈</text></svg>">
|
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><rect width='32' height='32' rx='8' fill='%236366f1'/><text x='16' y='22' text-anchor='middle' font-size='20' fill='white'>◈</text></svg>">
|
||||||
<style>
|
<style>
|
||||||
:root {
|
:root {
|
||||||
|
|||||||
+4
-4
@@ -3,7 +3,7 @@
|
|||||||
<head>
|
<head>
|
||||||
<meta charset="UTF-8">
|
<meta charset="UTF-8">
|
||||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
<title>🧪 在线演示 — MetonaSqlark v0.4.3</title>
|
<title>🧪 在线演示 — MetonaSqlark v0.4.4</title>
|
||||||
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><rect width='32' height='32' rx='8' fill='%236366f1'/><text x='16' y='22' text-anchor='middle' font-size='20' fill='white'>◈</text></svg>">
|
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><rect width='32' height='32' rx='8' fill='%236366f1'/><text x='16' y='22' text-anchor='middle' font-size='20' fill='white'>◈</text></svg>">
|
||||||
<style>
|
<style>
|
||||||
:root {
|
:root {
|
||||||
@@ -84,7 +84,7 @@
|
|||||||
<a href="demo.html" class="nav-active">演示</a>
|
<a href="demo.html" class="nav-active">演示</a>
|
||||||
<a href="benchmark.html">基准</a>
|
<a href="benchmark.html">基准</a>
|
||||||
</nav>
|
</nav>
|
||||||
<div class="status"><span class="dot" id="engine-dot"></span> <span id="engine-status">Memory</span> 模式 — v0.4.3</div>
|
<div class="status"><span class="dot" id="engine-dot"></span> <span id="engine-status">Memory</span> 模式 — v0.4.4</div>
|
||||||
<button class="btn btn-preset" onclick="switchEngine('memory')" id="btn-memory" style="margin:6px 4px 6px 0;padding:6px 12px;">⚡ Memory</button>
|
<button class="btn btn-preset" onclick="switchEngine('memory')" id="btn-memory" style="margin:6px 4px 6px 0;padding:6px 12px;">⚡ Memory</button>
|
||||||
<button class="btn btn-preset" onclick="switchEngine('aria')" id="btn-aria" style="margin:6px 0;padding:6px 12px;color:#ec4899;border-color:#ec4899;">🌲 Aria</button>
|
<button class="btn btn-preset" onclick="switchEngine('aria')" id="btn-aria" style="margin:6px 0;padding:6px 12px;color:#ec4899;border-color:#ec4899;">🌲 Aria</button>
|
||||||
</header>
|
</header>
|
||||||
@@ -92,7 +92,7 @@
|
|||||||
<div class="main">
|
<div class="main">
|
||||||
<div class="editor-panel">
|
<div class="editor-panel">
|
||||||
<div class="editor-area">
|
<div class="editor-area">
|
||||||
<textarea id="sql-input" placeholder="输入 SQL 语句... SELECT * FROM users; INSERT INTO users VALUES ('4', 'Diana', 'diana@test.com', 28); SELECT u.name, o.amount FROM users u INNER JOIN orders o ON u.id = o.user_id;">-- 🚀 MetonaSqlark v0.4.3 在线演示
|
<textarea id="sql-input" placeholder="输入 SQL 语句... SELECT * FROM users; INSERT INTO users VALUES ('4', 'Diana', 'diana@test.com', 28); SELECT u.name, o.amount FROM users u INNER JOIN orders o ON u.id = o.user_id;">-- 🚀 MetonaSqlark v0.4.4 在线演示
|
||||||
-- 已预置 users / orders / products 表数据
|
-- 已预置 users / orders / products 表数据
|
||||||
-- 新特性: ALTER TABLE · TRUNCATE TABLE · WAL同步 · MVCC · SQL注入防护
|
-- 新特性: ALTER TABLE · TRUNCATE TABLE · WAL同步 · MVCC · SQL注入防护
|
||||||
|
|
||||||
@@ -554,7 +554,7 @@ SELECT COUNT(*) as total FROM temp_logs;
|
|||||||
|
|
||||||
-- 清理
|
-- 清理
|
||||||
DROP TABLE temp_logs;`,
|
DROP TABLE temp_logs;`,
|
||||||
aria: `-- 🌲 AriaEngine 演示 (v0.4.3)
|
aria: `-- 🌲 AriaEngine 演示 (v0.4.4)
|
||||||
-- 点击右上角「🌲 Aria」按钮切换数据库引擎到 AriaEngine
|
-- 点击右上角「🌲 Aria」按钮切换数据库引擎到 AriaEngine
|
||||||
-- 当前数据库即运行在 Aria 引擎上(LSM-Tree · WAL 崩溃恢复 · MVCC · BloomFilter)
|
-- 当前数据库即运行在 Aria 引擎上(LSM-Tree · WAL 崩溃恢复 · MVCC · BloomFilter)
|
||||||
-- 基础 CRUD 与 Memory 引擎完全兼容
|
-- 基础 CRUD 与 Memory 引擎完全兼容
|
||||||
|
|||||||
+3
-2
@@ -3,7 +3,7 @@
|
|||||||
<head>
|
<head>
|
||||||
<meta charset="UTF-8">
|
<meta charset="UTF-8">
|
||||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||||
<title>📖 API 文档 — MetonaSqlark v0.4.3</title>
|
<title>📖 API 文档 — MetonaSqlark v0.4.4</title>
|
||||||
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><rect width='32' height='32' rx='8' fill='%236366f1'/><text x='16' y='22' text-anchor='middle' font-size='20' fill='white'>◈</text></svg>">
|
<link rel="icon" href="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 32 32'><rect width='32' height='32' rx='8' fill='%236366f1'/><text x='16' y='22' text-anchor='middle' font-size='20' fill='white'>◈</text></svg>">
|
||||||
<style>
|
<style>
|
||||||
:root {
|
:root {
|
||||||
@@ -762,7 +762,8 @@ db.<span class="f">broadcastChange</span>(<span class="s">'users'</span>);</pre>
|
|||||||
<strong>v0.2.4 生产级</strong> — 二级索引 · MVCC · BloomFilter · WAL CRC全同步 · AES-GCM加密 · Savepoint · EXPLAIN · ANALYZE · REINDEX · VACUUM · BufferPool · 零死代码。<br>
|
<strong>v0.2.4 生产级</strong> — 二级索引 · MVCC · BloomFilter · WAL CRC全同步 · AES-GCM加密 · Savepoint · EXPLAIN · ANALYZE · REINDEX · VACUUM · BufferPool · 零死代码。<br>
|
||||||
<strong>v0.3.2 表达式与并发</strong> — WAL full模式真正同步 · MVCC接入读写路径 · SSTableReader二分查找统一 · crypto实例化 · IndexedDB索引利用 · compactLevel public接口 · WAL大小阈值自动checkpoint · SQL注入防护 · ALTER TABLE · TRUNCATE TABLE · 多标签页同步 · IDB schema持久化。<br>
|
<strong>v0.3.2 表达式与并发</strong> — WAL full模式真正同步 · MVCC接入读写路径 · SSTableReader二分查找统一 · crypto实例化 · IndexedDB索引利用 · compactLevel public接口 · WAL大小阈值自动checkpoint · SQL注入防护 · ALTER TABLE · TRUNCATE TABLE · 多标签页同步 · IDB schema持久化。<br>
|
||||||
<strong>v0.4.2 生产就绪与崩溃自愈</strong> — 残缺 SSTable 打开自动跳过(不删库)· WAL 记录与计数原子写入 + 按 key 扫描恢复 · 事务进行中 checkpoint 不截断 WAL · 二级索引跨重启自动恢复 · ALTER TABLE / 事务内 DDL 全引擎持久化 · ON UPDATE 外键级联(含更新主键)· OPFS schema 持久化(空表/索引完整保留)· `repair()` / `clearAll()` 统一自愈接口 · 迁移版本持久化到库内。<br>
|
<strong>v0.4.2 生产就绪与崩溃自愈</strong> — 残缺 SSTable 打开自动跳过(不删库)· WAL 记录与计数原子写入 + 按 key 扫描恢复 · 事务进行中 checkpoint 不截断 WAL · 二级索引跨重启自动恢复 · ALTER TABLE / 事务内 DDL 全引擎持久化 · ON UPDATE 外键级联(含更新主键)· OPFS schema 持久化(空表/索引完整保留)· `repair()` / `clearAll()` 统一自愈接口 · 迁移版本持久化到库内。<br>
|
||||||
<strong>v0.4.3 关闭时序与后台任务加固</strong> — 后台 flush/compaction 不再使用 setTimeout 延迟(close 排空全部任务后才关闭存储,杜绝"backend 关闭后写存储/重开污染")· 后台失败在 `flush()`/`close()` 显式报告(`ARIA_BACKGROUND_ERROR`,不静默吞错)· 预加载等待链稳定(修复 compaction 竞态跳块丢数据)· 事务提交先落 WAL 再合并快照(崩溃一致)· OPFS 写操作串行队列 + close 等待 · 950 测试 51 套件。</p>
|
<strong>v0.4.3 关闭时序与后台任务加固</strong> — 后台 flush/compaction 不再使用 setTimeout 延迟(close 排空全部任务后才关闭存储,杜绝"backend 关闭后写存储/重开污染")· 后台失败在 `flush()`/`close()` 显式报告(`ARIA_BACKGROUND_ERROR`,不静默吞错)· 预加载等待链稳定(修复 compaction 竞态跳块丢数据)· 事务提交先落 WAL 再合并快照(崩溃一致)· OPFS 写操作串行队列 + close 等待。<br>
|
||||||
|
<strong>v0.4.4 SSTable 编码修复</strong> — 大段中文内容(如 300KB 笔记)写入 AriaEngine 不再崩溃:块大小估算改 UTF-8 字节精确计算(修复中文 3 字节 vs 1 码元导致的缓冲区低估越界)· 长度字段 u16 → u32(修复 >64KB value 截断)· 大 value 独立成块 · 格式 v2("SSTC")与 v1("SSTB")双格式兼容(旧库数据不丢)· 中文主键 / 大内容索引列同步支持 · 958 测试 52 套件。</p>
|
||||||
|
|
||||||
<h3>存储模式对比</h3>
|
<h3>存储模式对比</h3>
|
||||||
<table>
|
<table>
|
||||||
|
|||||||
+3
-3
@@ -153,7 +153,7 @@
|
|||||||
<!-- Hero -->
|
<!-- Hero -->
|
||||||
<section class="hero">
|
<section class="hero">
|
||||||
<div class="container">
|
<div class="container">
|
||||||
<div class="badge" style="margin-bottom:24px;"><span class="dot"></span> v0.4.3 关闭时序与后台任务加固 — 950测试 51套件 · close排空后台任务 · 失败不吞错 · compaction竞态修复 · 提交先WAL · 零回归</div>
|
<div class="badge" style="margin-bottom:24px;"><span class="dot"></span> v0.4.4 SSTable 编码修复 — 958测试 52套件 · 大内容(300KB中文)不再崩溃 · u32长度字段 · 双格式兼容 · 零回归</div>
|
||||||
<h1>前端的 <span class="gradient-text">SQL 数据库</span></h1>
|
<h1>前端的 <span class="gradient-text">SQL 数据库</span></h1>
|
||||||
<p>TypeScript 原生构建,5 种存储引擎,支持完整 SQL 查询。<br>零运行时依赖,开箱即用。AriaEngine 自研引擎:LSM-Tree + WAL 同步 + MVCC。</p>
|
<p>TypeScript 原生构建,5 种存储引擎,支持完整 SQL 查询。<br>零运行时依赖,开箱即用。AriaEngine 自研引擎:LSM-Tree + WAL 同步 + MVCC。</p>
|
||||||
<div class="actions">
|
<div class="actions">
|
||||||
@@ -243,7 +243,7 @@ npm install @metona-team/metona-sqlark
|
|||||||
</div>
|
</div>
|
||||||
<div class="feature-card">
|
<div class="feature-card">
|
||||||
<div class="icon">🌲</div>
|
<div class="icon">🌲</div>
|
||||||
<h3>AriaEngine <span style="font-size:0.65rem;color:var(--accent);vertical-align:super;">v0.4.3</span></h3>
|
<h3>AriaEngine <span style="font-size:0.65rem;color:var(--accent);vertical-align:super;">v0.4.4</span></h3>
|
||||||
<p>自研 LSM-Tree 页面式存储引擎。MemTable 红黑树 + 多级 SSTable、Bloom Filter 快速判存、WAL 原子写入 full模式真正同步、MVCC 快照隔离、崩溃恢复自动跳过残缺 SSTable、二级索引跨重启恢复、ON UPDATE/DELETE 外键级联。</p>
|
<p>自研 LSM-Tree 页面式存储引擎。MemTable 红黑树 + 多级 SSTable、Bloom Filter 快速判存、WAL 原子写入 full模式真正同步、MVCC 快照隔离、崩溃恢复自动跳过残缺 SSTable、二级索引跨重启恢复、ON UPDATE/DELETE 外键级联。</p>
|
||||||
</div>
|
</div>
|
||||||
<div class="feature-card">
|
<div class="feature-card">
|
||||||
@@ -404,7 +404,7 @@ npm install @metona-team/metona-sqlark
|
|||||||
<p>MetonaSqlark 的核心指标</p>
|
<p>MetonaSqlark 的核心指标</p>
|
||||||
</div>
|
</div>
|
||||||
<div class="stats">
|
<div class="stats">
|
||||||
<div class="stat-card"><div class="num">950</div><div class="label">测试用例</div></div>
|
<div class="stat-card"><div class="num">958</div><div class="label">测试用例</div></div>
|
||||||
<div class="stat-card"><div class="num">84.2%</div><div class="label">行覆盖率</div></div>
|
<div class="stat-card"><div class="num">84.2%</div><div class="label">行覆盖率</div></div>
|
||||||
<div class="stat-card"><div class="num">~27KB</div><div class="label">gzip 体积</div></div>
|
<div class="stat-card"><div class="num">~27KB</div><div class="label">gzip 体积</div></div>
|
||||||
<div class="stat-card"><div class="num">5</div><div class="label">存储引擎</div></div>
|
<div class="stat-card"><div class="num">5</div><div class="label">存储引擎</div></div>
|
||||||
|
|||||||
+1
-1
@@ -214,4 +214,4 @@ export class DatabaseError extends Error {
|
|||||||
// 版本
|
// 版本
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
export const VERSION = '0.4.3';
|
export const VERSION = '0.4.4';
|
||||||
|
|||||||
@@ -1,12 +1,14 @@
|
|||||||
/**
|
/**
|
||||||
* AriaEngine SSTable Reader — 从 SSTable 二进制数据中读取
|
* AriaEngine SSTable Reader — 从 SSTable 二进制数据中读取
|
||||||
* @module engine/aria/index/sstable
|
* @module engine/aria/index/sstable
|
||||||
|
*
|
||||||
|
* v0.4.4: 支持 v1("SSTB",u16 长度字段)与 v2("SSTC",u32 长度字段)双格式,
|
||||||
|
* 旧库 v1 文件仍可读(小 value 场景无缺陷),新写入使用 v2。
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import type { IndexEntry, SSTableMeta } from '../types';
|
import type { IndexEntry, SSTableMeta } from '../types';
|
||||||
import { BloomFilter } from './bloom';
|
import { BloomFilter } from './bloom';
|
||||||
|
import { SSTABLE_MAGIC_V1, SSTABLE_MAGIC_V2 } from './sstable_builder';
|
||||||
const SSTABLE_MAGIC = 0x53535442; // "SSTB"
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// SSTableReader
|
// SSTableReader
|
||||||
@@ -19,6 +21,8 @@ export class SSTableReader {
|
|||||||
private entryCount = 0;
|
private entryCount = 0;
|
||||||
private meta: SSTableMeta;
|
private meta: SSTableMeta;
|
||||||
private bloomFilter: BloomFilter | null = null;
|
private bloomFilter: BloomFilter | null = null;
|
||||||
|
/** 格式版本:1 = u16 长度字段(旧),2 = u32 长度字段(v0.4.4) */
|
||||||
|
private format: 1 | 2 = 2;
|
||||||
|
|
||||||
constructor(data: Uint8Array, meta: SSTableMeta) {
|
constructor(data: Uint8Array, meta: SSTableMeta) {
|
||||||
this.data = data;
|
this.data = data;
|
||||||
@@ -27,6 +31,15 @@ export class SSTableReader {
|
|||||||
this.parseFooter();
|
this.parseFooter();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** 长度字段宽度:v2 = 4 字节 u32,v1 = 2 字节 u16 */
|
||||||
|
private lenFieldSize(): number {
|
||||||
|
return this.format === 2 ? 4 : 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
private readLen(offset: number): number {
|
||||||
|
return this.format === 2 ? this.view.getUint32(offset, false) : this.view.getUint16(offset, false);
|
||||||
|
}
|
||||||
|
|
||||||
// -----------------------------------------------------------------------
|
// -----------------------------------------------------------------------
|
||||||
// 查询
|
// 查询
|
||||||
// -----------------------------------------------------------------------
|
// -----------------------------------------------------------------------
|
||||||
@@ -41,23 +54,24 @@ export class SSTableReader {
|
|||||||
|
|
||||||
const entry = this.indexEntries[blockIdx];
|
const entry = this.indexEntries[blockIdx];
|
||||||
const blockData = this.getBlockData(entry);
|
const blockData = this.getBlockData(entry);
|
||||||
// v0.4.2-fix: 残缺文件(meta 偏移超出实际长度)跳过该块,而非抛 RangeError
|
// v0.4.1-fix: 残缺文件(meta 偏移超出实际长度)跳过该块,而非抛 RangeError
|
||||||
if (!blockData) return null;
|
if (!blockData) return null;
|
||||||
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
||||||
|
const lenSize = this.lenFieldSize();
|
||||||
|
|
||||||
const entryCount = blockView.getUint32(0, false);
|
const entryCount = blockView.getUint32(0, false);
|
||||||
let offset = 4;
|
let offset = 4;
|
||||||
|
|
||||||
// 顺序扫描 block 内的条目(生产中应二分查找)
|
// 顺序扫描 block 内的条目(生产中应二分查找)
|
||||||
for (let i = 0; i < entryCount; i++) {
|
for (let i = 0; i < entryCount; i++) {
|
||||||
if (offset + 2 > blockData.byteLength) break;
|
if (offset + lenSize > blockData.byteLength) break;
|
||||||
const keyLen = blockView.getUint16(offset, false);
|
const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + keyLen + 2 > blockData.byteLength) break;
|
if (offset + keyLen + lenSize > blockData.byteLength) break;
|
||||||
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
||||||
offset += keyLen;
|
offset += keyLen;
|
||||||
const valLen = blockView.getUint16(offset, false);
|
const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + valLen > blockData.byteLength) break;
|
if (offset + valLen > blockData.byteLength) break;
|
||||||
const valBytes = blockData.slice(offset, offset + valLen);
|
const valBytes = blockData.slice(offset, offset + valLen);
|
||||||
offset += valLen;
|
offset += valLen;
|
||||||
@@ -85,10 +99,11 @@ export class SSTableReader {
|
|||||||
const endBlockIdx = Math.min(this.indexEntries.length - 1, this.locateBlockLE(endKey));
|
const endBlockIdx = Math.min(this.indexEntries.length - 1, this.locateBlockLE(endKey));
|
||||||
if (startBlockIdx < 0 || endBlockIdx < 0 || startBlockIdx > endBlockIdx) return;
|
if (startBlockIdx < 0 || endBlockIdx < 0 || startBlockIdx > endBlockIdx) return;
|
||||||
|
|
||||||
|
const lenSize = this.lenFieldSize();
|
||||||
for (let bi = startBlockIdx; bi <= endBlockIdx && bi >= 0; bi++) {
|
for (let bi = startBlockIdx; bi <= endBlockIdx && bi >= 0; bi++) {
|
||||||
const entry = this.indexEntries[bi];
|
const entry = this.indexEntries[bi];
|
||||||
const blockData = this.getBlockData(entry);
|
const blockData = this.getBlockData(entry);
|
||||||
// v0.4.2-fix: 残缺块跳过(rangeScan 继续后续块,不抛异常)
|
// v0.4.1-fix: 残缺块跳过(rangeScan 继续后续块,不抛异常)
|
||||||
if (!blockData) continue;
|
if (!blockData) continue;
|
||||||
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
||||||
|
|
||||||
@@ -96,14 +111,14 @@ export class SSTableReader {
|
|||||||
let offset = 4;
|
let offset = 4;
|
||||||
|
|
||||||
for (let i = 0; i < blockEntryCount; i++) {
|
for (let i = 0; i < blockEntryCount; i++) {
|
||||||
if (offset + 2 > blockData.byteLength) break;
|
if (offset + lenSize > blockData.byteLength) break;
|
||||||
const keyLen = blockView.getUint16(offset, false);
|
const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + keyLen + 2 > blockData.byteLength) break;
|
if (offset + keyLen + lenSize > blockData.byteLength) break;
|
||||||
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
||||||
offset += keyLen;
|
offset += keyLen;
|
||||||
const valLen = blockView.getUint16(offset, false);
|
const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + valLen > blockData.byteLength) break;
|
if (offset + valLen > blockData.byteLength) break;
|
||||||
const valBytes = blockData.slice(offset, offset + valLen);
|
const valBytes = blockData.slice(offset, offset + valLen);
|
||||||
offset += valLen;
|
offset += valLen;
|
||||||
@@ -122,9 +137,10 @@ export class SSTableReader {
|
|||||||
|
|
||||||
/** 扫描所有条目 */
|
/** 扫描所有条目 */
|
||||||
scanAll(callback: (key: string, value: Record<string, unknown>) => void): void {
|
scanAll(callback: (key: string, value: Record<string, unknown>) => void): void {
|
||||||
|
const lenSize = this.lenFieldSize();
|
||||||
for (const entry of this.indexEntries) {
|
for (const entry of this.indexEntries) {
|
||||||
const blockData = this.getBlockData(entry);
|
const blockData = this.getBlockData(entry);
|
||||||
// v0.4.2-fix: 残缺块跳过(scanAll 继续后续块,不抛异常)
|
// v0.4.1-fix: 残缺块跳过(scanAll 继续后续块,不抛异常)
|
||||||
if (!blockData) continue;
|
if (!blockData) continue;
|
||||||
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
const blockView = new DataView(blockData.buffer, blockData.byteOffset, blockData.byteLength);
|
||||||
|
|
||||||
@@ -132,14 +148,14 @@ export class SSTableReader {
|
|||||||
let offset = 4;
|
let offset = 4;
|
||||||
|
|
||||||
for (let i = 0; i < blockEntryCount; i++) {
|
for (let i = 0; i < blockEntryCount; i++) {
|
||||||
if (offset + 2 > blockData.byteLength) break;
|
if (offset + lenSize > blockData.byteLength) break;
|
||||||
const keyLen = blockView.getUint16(offset, false);
|
const keyLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + keyLen + 2 > blockData.byteLength) break;
|
if (offset + keyLen + lenSize > blockData.byteLength) break;
|
||||||
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
const key = new TextDecoder().decode(blockData.slice(offset, offset + keyLen));
|
||||||
offset += keyLen;
|
offset += keyLen;
|
||||||
const valLen = blockView.getUint16(offset, false);
|
const valLen = this.format === 2 ? blockView.getUint32(offset, false) : blockView.getUint16(offset, false);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + valLen > blockData.byteLength) break;
|
if (offset + valLen > blockData.byteLength) break;
|
||||||
const valBytes = blockData.slice(offset, offset + valLen);
|
const valBytes = blockData.slice(offset, offset + valLen);
|
||||||
offset += valLen;
|
offset += valLen;
|
||||||
@@ -175,10 +191,14 @@ export class SSTableReader {
|
|||||||
|
|
||||||
const footerOffset = this.data.byteLength - 32;
|
const footerOffset = this.data.byteLength - 32;
|
||||||
|
|
||||||
// 验证魔数
|
// 验证魔数(v1 "SSTB" / v2 "SSTC")
|
||||||
const magic = this.view.getUint32(footerOffset + 24, false);
|
const magic = this.view.getUint32(footerOffset + 24, false);
|
||||||
if (magic !== SSTABLE_MAGIC) {
|
if (magic === SSTABLE_MAGIC_V1) {
|
||||||
throw new Error(`Invalid SSTable magic: expected ${SSTABLE_MAGIC}, got ${magic}`);
|
this.format = 1;
|
||||||
|
} else if (magic === SSTABLE_MAGIC_V2) {
|
||||||
|
this.format = 2;
|
||||||
|
} else {
|
||||||
|
throw new Error(`Invalid SSTable magic: expected ${SSTABLE_MAGIC_V1} or ${SSTABLE_MAGIC_V2}, got ${magic}`);
|
||||||
}
|
}
|
||||||
|
|
||||||
const indexOffset = this.view.getUint32(footerOffset, false);
|
const indexOffset = this.view.getUint32(footerOffset, false);
|
||||||
@@ -188,7 +208,7 @@ export class SSTableReader {
|
|||||||
const bloomHashCount = this.view.getUint32(footerOffset + 16, false);
|
const bloomHashCount = this.view.getUint32(footerOffset + 16, false);
|
||||||
this.entryCount = this.view.getUint32(footerOffset + 20, false);
|
this.entryCount = this.view.getUint32(footerOffset + 20, false);
|
||||||
|
|
||||||
// v0.4.2-fix: 完整性校验 — 索引块必须完全落在文件内,否则视为残缺文件跳过
|
// v0.4.1-fix: 完整性校验 — 索引块必须完全落在文件内,否则视为残缺文件跳过
|
||||||
if (indexOffset + 4 > this.data.byteLength || indexOffset + indexSize > this.data.byteLength) {
|
if (indexOffset + 4 > this.data.byteLength || indexOffset + indexSize > this.data.byteLength) {
|
||||||
return; // 残缺文件:无索引块可读,get/rangeScan 均返回空
|
return; // 残缺文件:无索引块可读,get/rangeScan 均返回空
|
||||||
}
|
}
|
||||||
@@ -210,13 +230,14 @@ export class SSTableReader {
|
|||||||
private parseIndexBlock(offset: number, _size: number): void {
|
private parseIndexBlock(offset: number, _size: number): void {
|
||||||
const entryCount = this.view.getUint32(offset, false);
|
const entryCount = this.view.getUint32(offset, false);
|
||||||
offset += 4;
|
offset += 4;
|
||||||
|
const lenSize = this.lenFieldSize();
|
||||||
|
|
||||||
for (let i = 0; i < entryCount; i++) {
|
for (let i = 0; i < entryCount; i++) {
|
||||||
// v0.4.2-fix: 索引条目越界(keyLen/blockOffset/blockSize 超过文件长度)时中止解析,
|
// v0.4.1-fix: 索引条目越界(keyLen/blockOffset/blockSize 超过文件长度)时中止解析,
|
||||||
// 已解析的有效条目仍可用于查询
|
// 已解析的有效条目仍可用于查询
|
||||||
if (offset + 2 > this.data.byteLength) break;
|
if (offset + lenSize > this.data.byteLength) break;
|
||||||
const keyLen = this.view.getUint16(offset, false);
|
const keyLen = this.readLen(offset);
|
||||||
offset += 2;
|
offset += lenSize;
|
||||||
if (offset + keyLen + 8 > this.data.byteLength) break;
|
if (offset + keyLen + 8 > this.data.byteLength) break;
|
||||||
const key = new TextDecoder().decode(this.data.slice(offset, offset + keyLen));
|
const key = new TextDecoder().decode(this.data.slice(offset, offset + keyLen));
|
||||||
offset += keyLen;
|
offset += keyLen;
|
||||||
@@ -233,7 +254,7 @@ export class SSTableReader {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* v0.4.2-fix: 获取索引条目对应的数据块。
|
* v0.4.1-fix: 获取索引条目对应的数据块。
|
||||||
* 块偏移/大小越界(残缺 SSTable)时返回 null,由调用方跳过而非抛 RangeError。
|
* 块偏移/大小越界(残缺 SSTable)时返回 null,由调用方跳过而非抛 RangeError。
|
||||||
*/
|
*/
|
||||||
private getBlockData(entry: IndexEntry): Uint8Array | null {
|
private getBlockData(entry: IndexEntry): Uint8Array | null {
|
||||||
|
|||||||
@@ -4,7 +4,15 @@
|
|||||||
*
|
*
|
||||||
* 将排序后的 key-value 数据写入 SSTable 格式。
|
* 将排序后的 key-value 数据写入 SSTable 格式。
|
||||||
*
|
*
|
||||||
* SSTable 文件布局:
|
* v0.4.4 格式 v2(magic "SSTC")修复:
|
||||||
|
* - 块大小估算改用 UTF-8 字节长度(TextEncoder 预编码),
|
||||||
|
* 此前用字符串 .length(UTF-16 码元)估算而实际写入 UTF-8 字节,
|
||||||
|
* 中文内容(1 字 3 字节)导致缓冲区低估 → 写入越界崩溃
|
||||||
|
* - keyLen/valueLen 从 u16 升级为 u32(此前 >64KB 的 value 长度被截断,
|
||||||
|
* 线性格式整体错乱)
|
||||||
|
* - 大 value 单条独立成块(切分逻辑基于字节估算)
|
||||||
|
*
|
||||||
|
* SSTable 文件布局 (v2):
|
||||||
* ┌──────────────────────────────────────────────┐
|
* ┌──────────────────────────────────────────────┐
|
||||||
* │ Data Block 0 │
|
* │ Data Block 0 │
|
||||||
* │ Data Block 1 │
|
* │ Data Block 1 │
|
||||||
@@ -18,25 +26,34 @@
|
|||||||
* │ - bloom_size (u32) │
|
* │ - bloom_size (u32) │
|
||||||
* │ - bloom_hash_count (u32) │
|
* │ - bloom_hash_count (u32) │
|
||||||
* │ - entry_count (u32) │
|
* │ - entry_count (u32) │
|
||||||
* │ - magic_number (u32, 0x53535442 ="SSTB")│
|
* │ - magic_number (u32, 0x53535443 ="SSTC")│
|
||||||
* │ - checksum (u32) │
|
* │ - checksum (u32) │
|
||||||
* └──────────────────────────────────────────────┘
|
* └──────────────────────────────────────────────┘
|
||||||
|
* 数据块条目: entryCount(u32) + [keyLen(u32) + key + valueLen(u32) + value]
|
||||||
|
* 索引块条目: [keyLen(u32) + key + blockOffset(u32) + blockSize(u32)]
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { BloomFilter } from './bloom';
|
import { BloomFilter } from './bloom';
|
||||||
import type { IndexEntry } from '../types';
|
import type { IndexEntry } from '../types';
|
||||||
|
|
||||||
const SSTABLE_MAGIC = 0x53535442; // "SSTB"
|
/** v1 格式魔数("SSTB",u16 长度字段,兼容旧文件读取) */
|
||||||
|
export const SSTABLE_MAGIC_V1 = 0x53535442;
|
||||||
|
/** v2 格式魔数("SSTC",u32 长度字段 + 字节精确估算,v0.4.4) */
|
||||||
|
export const SSTABLE_MAGIC_V2 = 0x53535443;
|
||||||
const SSTABLE_FOOTER_SIZE = 32;
|
const SSTABLE_FOOTER_SIZE = 32;
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// SSTableBuilder
|
// SSTableBuilder
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
interface EncodedEntry {
|
||||||
|
key: string;
|
||||||
|
keyBytes: Uint8Array;
|
||||||
|
valueBytes: Uint8Array;
|
||||||
|
}
|
||||||
|
|
||||||
export class SSTableBuilder {
|
export class SSTableBuilder {
|
||||||
private entries: [string, Record<string, unknown>][] = [];
|
private entries: [string, Record<string, unknown>][] = [];
|
||||||
private currentBlock: [string, Record<string, unknown>][] = [];
|
|
||||||
private currentBlockStartKey = '';
|
|
||||||
private blockSizeLimit: number;
|
private blockSizeLimit: number;
|
||||||
|
|
||||||
constructor(blockSizeLimit: number = 4096) {
|
constructor(blockSizeLimit: number = 4096) {
|
||||||
@@ -45,48 +62,42 @@ export class SSTableBuilder {
|
|||||||
|
|
||||||
/** 添加一个 key-value 条目(必须按键排序添加) */
|
/** 添加一个 key-value 条目(必须按键排序添加) */
|
||||||
add(key: string, value: Record<string, unknown>): void {
|
add(key: string, value: Record<string, unknown>): void {
|
||||||
if (this.currentBlock.length === 0) {
|
|
||||||
this.currentBlockStartKey = key;
|
|
||||||
}
|
|
||||||
|
|
||||||
this.currentBlock.push([key, value]);
|
|
||||||
this.entries.push([key, value]);
|
this.entries.push([key, value]);
|
||||||
|
|
||||||
// 如果当前 Block 达到大小限制,切割
|
|
||||||
const estimated = this.estimateBlockSize();
|
|
||||||
if (estimated >= this.blockSizeLimit && this.currentBlock.length > 1) {
|
|
||||||
// 当前 block 结束(不在这里切割,在 build 时统一处理)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* 构建 SSTable 文件的二进制数据。
|
* 构建 SSTable 文件的二进制数据(v2 格式)。
|
||||||
* 返回 { data: Uint8Array, indexEntries: IndexEntry[], bloomFilter: BloomFilter }
|
* 返回 { data: Uint8Array, indexEntries: IndexEntry[] }
|
||||||
*/
|
*/
|
||||||
build(): { sstableData: Uint8Array; indexEntries: IndexEntry[] } {
|
build(): { sstableData: Uint8Array; indexEntries: IndexEntry[] } {
|
||||||
const blocks = this.splitIntoBlocks();
|
// v0.4.4-fix: 预编码全部条目 — 块大小估算必须基于 UTF-8 字节长度,
|
||||||
|
// 字符串 .length 是 UTF-16 码元(中文 1 字 3 字节 vs 1 码元)→ 缓冲区低估越界
|
||||||
|
const encoder = new TextEncoder();
|
||||||
|
const encoded: EncodedEntry[] = this.entries.map(([key, value]) => ({
|
||||||
|
key,
|
||||||
|
keyBytes: encoder.encode(key),
|
||||||
|
valueBytes: encoder.encode(JSON.stringify(value)),
|
||||||
|
}));
|
||||||
|
|
||||||
|
const blocks = this.splitIntoBlocks(encoded);
|
||||||
const bloomFilter = new BloomFilter(this.entries.length);
|
const bloomFilter = new BloomFilter(this.entries.length);
|
||||||
|
|
||||||
// 预计算总大小
|
// 预计算总大小(字节)
|
||||||
let totalSize = 0;
|
let totalSize = 0;
|
||||||
const blockOffsets: number[] = [];
|
const blockOffsets: number[] = [];
|
||||||
|
|
||||||
for (const block of blocks) {
|
for (const block of blocks) {
|
||||||
blockOffsets.push(totalSize);
|
blockOffsets.push(totalSize);
|
||||||
const blockSize = this.computeBlockSize(block);
|
totalSize += this.computeBlockSize(block);
|
||||||
totalSize += blockSize;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// 索引块
|
// 索引块(块内最后一个 key 作为索引键)
|
||||||
const indexEntries: IndexEntry[] = [];
|
const indexEntries: IndexEntry[] = [];
|
||||||
for (let i = 0; i < blocks.length; i++) {
|
for (let i = 0; i < blocks.length; i++) {
|
||||||
const block = blocks[i];
|
const block = blocks[i];
|
||||||
const lastKey = block[block.length - 1][0];
|
|
||||||
const blockSize = this.computeBlockSize(block);
|
|
||||||
indexEntries.push({
|
indexEntries.push({
|
||||||
key: lastKey,
|
key: block[block.length - 1].key,
|
||||||
blockOffset: blockOffsets[i],
|
blockOffset: blockOffsets[i],
|
||||||
blockSize,
|
blockSize: this.computeBlockSize(block),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -119,14 +130,14 @@ export class SSTableBuilder {
|
|||||||
|
|
||||||
// ---- Footer ----
|
// ---- Footer ----
|
||||||
const footerOffset = offset;
|
const footerOffset = offset;
|
||||||
view.setUint32(footerOffset, indexOffset, false); // index_offset
|
view.setUint32(footerOffset, indexOffset, false); // index_offset
|
||||||
view.setUint32(footerOffset + 4, indexBlockSize, false); // index_size
|
view.setUint32(footerOffset + 4, indexBlockSize, false); // index_size
|
||||||
view.setUint32(footerOffset + 8, bloomOffset, false); // bloom_offset
|
view.setUint32(footerOffset + 8, bloomOffset, false); // bloom_offset
|
||||||
view.setUint32(footerOffset + 12, bloomSize, false); // bloom_size
|
view.setUint32(footerOffset + 12, bloomSize, false); // bloom_size
|
||||||
view.setUint32(footerOffset + 16, bloomFilter.getHashCount(), false);
|
view.setUint32(footerOffset + 16, bloomFilter.getHashCount(), false);
|
||||||
view.setUint32(footerOffset + 20, this.entries.length, false);
|
view.setUint32(footerOffset + 20, this.entries.length, false);
|
||||||
view.setUint32(footerOffset + 24, SSTABLE_MAGIC, false);
|
view.setUint32(footerOffset + 24, SSTABLE_MAGIC_V2, false);
|
||||||
view.setUint32(footerOffset + 28, 0, false); // checksum (simplified: 0)
|
view.setUint32(footerOffset + 28, 0, false); // checksum (simplified: 0)
|
||||||
|
|
||||||
return {
|
return {
|
||||||
sstableData: new Uint8Array(buf),
|
sstableData: new Uint8Array(buf),
|
||||||
@@ -143,13 +154,15 @@ export class SSTableBuilder {
|
|||||||
// 内部
|
// 内部
|
||||||
// -----------------------------------------------------------------------
|
// -----------------------------------------------------------------------
|
||||||
|
|
||||||
private splitIntoBlocks(): [string, Record<string, unknown>][][] {
|
/** 按 UTF-8 字节大小切分数据块;大 value 单条独立成块 */
|
||||||
const blocks: [string, Record<string, unknown>][][] = [];
|
private splitIntoBlocks(encoded: EncodedEntry[]): EncodedEntry[][] {
|
||||||
let current: [string, Record<string, unknown>][] = [];
|
const blocks: EncodedEntry[][] = [];
|
||||||
|
let current: EncodedEntry[] = [];
|
||||||
|
|
||||||
for (const entry of this.entries) {
|
for (const entry of encoded) {
|
||||||
current.push(entry);
|
current.push(entry);
|
||||||
if (this.estimateBlockSizeFromEntries(current) >= this.blockSizeLimit && current.length > 1) {
|
// v0.4.4-fix: 基于字节估算;单条超大条目(length===1)独立成块不强行切分
|
||||||
|
if (this.computeBlockSize(current) >= this.blockSizeLimit && current.length > 1) {
|
||||||
blocks.push(current.slice(0, -1));
|
blocks.push(current.slice(0, -1));
|
||||||
current = [entry];
|
current = [entry];
|
||||||
}
|
}
|
||||||
@@ -159,24 +172,11 @@ export class SSTableBuilder {
|
|||||||
return blocks;
|
return blocks;
|
||||||
}
|
}
|
||||||
|
|
||||||
private estimateBlockSize(): number {
|
/** 块字节大小:entryCount(u32) + 每对 [keyLen(u32) + key + valueLen(u32) + value] */
|
||||||
return this.estimateBlockSizeFromEntries(this.currentBlock);
|
private computeBlockSize(block: EncodedEntry[]): number {
|
||||||
}
|
|
||||||
|
|
||||||
private estimateBlockSizeFromEntries(entries: [string, unknown][]): number {
|
|
||||||
let size = 0;
|
|
||||||
for (const [key, value] of entries) {
|
|
||||||
size += 4 + key.length + JSON.stringify(value).length;
|
|
||||||
}
|
|
||||||
return size;
|
|
||||||
}
|
|
||||||
|
|
||||||
private computeBlockSize(block: [string, unknown][]): number {
|
|
||||||
// entryCount (u32) + 每对: keyLen(u16) + key + valueLen(u16) + value json
|
|
||||||
let size = 4;
|
let size = 4;
|
||||||
for (const [key, value] of block) {
|
for (const e of block) {
|
||||||
const json = JSON.stringify(value);
|
size += 4 + e.keyBytes.length + 4 + e.valueBytes.length;
|
||||||
size += 2 + key.length + 2 + json.length;
|
|
||||||
}
|
}
|
||||||
return size;
|
return size;
|
||||||
}
|
}
|
||||||
@@ -184,43 +184,40 @@ export class SSTableBuilder {
|
|||||||
private writeDataBlock(
|
private writeDataBlock(
|
||||||
view: DataView,
|
view: DataView,
|
||||||
offset: number,
|
offset: number,
|
||||||
block: [string, Record<string, unknown>][],
|
block: EncodedEntry[],
|
||||||
bloomFilter: BloomFilter,
|
bloomFilter: BloomFilter,
|
||||||
): number {
|
): number {
|
||||||
// entry count
|
// entry count
|
||||||
view.setUint32(offset, block.length, false);
|
view.setUint32(offset, block.length, false);
|
||||||
offset += 4;
|
offset += 4;
|
||||||
|
|
||||||
for (const [key, value] of block) {
|
for (const e of block) {
|
||||||
const encoder = new TextEncoder();
|
// v0.4.4-fix: 长度字段 u32(此前 u16 截断 >64KB 的 value)
|
||||||
const keyBytes = encoder.encode(key);
|
if (e.keyBytes.length > 0xFFFFFFFF || e.valueBytes.length > 0xFFFFFFFF) {
|
||||||
const valueBytes = encoder.encode(JSON.stringify(value));
|
throw new Error('SSTable entry too large (exceeds u32 length field)');
|
||||||
|
}
|
||||||
// key length
|
view.setUint32(offset, e.keyBytes.length, false);
|
||||||
view.setUint16(offset, keyBytes.length, false);
|
offset += 4;
|
||||||
offset += 2;
|
new Uint8Array(view.buffer).set(e.keyBytes, offset);
|
||||||
// key
|
offset += e.keyBytes.length;
|
||||||
new Uint8Array(view.buffer).set(keyBytes, offset);
|
view.setUint32(offset, e.valueBytes.length, false);
|
||||||
offset += keyBytes.length;
|
offset += 4;
|
||||||
// value length
|
new Uint8Array(view.buffer).set(e.valueBytes, offset);
|
||||||
view.setUint16(offset, valueBytes.length, false);
|
offset += e.valueBytes.length;
|
||||||
offset += 2;
|
|
||||||
// value
|
|
||||||
new Uint8Array(view.buffer).set(valueBytes, offset);
|
|
||||||
offset += valueBytes.length;
|
|
||||||
|
|
||||||
// 插入 bloom filter
|
// 插入 bloom filter
|
||||||
bloomFilter.insert(key);
|
bloomFilter.insert(e.key);
|
||||||
}
|
}
|
||||||
|
|
||||||
return offset;
|
return offset;
|
||||||
}
|
}
|
||||||
|
|
||||||
private estimateIndexBlockSize(entries: IndexEntry[]): number {
|
private estimateIndexBlockSize(entries: IndexEntry[]): number {
|
||||||
// entryCount(u32) + each: keyLen(u16)+key+blockOffset(u32)+blockSize(u32)
|
// entryCount(u32) + each: keyLen(u32)+key+blockOffset(u32)+blockSize(u32)
|
||||||
let size = 4;
|
let size = 4;
|
||||||
|
const encoder = new TextEncoder();
|
||||||
for (const entry of entries) {
|
for (const entry of entries) {
|
||||||
size += 2 + entry.key.length + 8;
|
size += 4 + encoder.encode(entry.key).byteLength + 8;
|
||||||
}
|
}
|
||||||
return size;
|
return size;
|
||||||
}
|
}
|
||||||
@@ -232,8 +229,8 @@ export class SSTableBuilder {
|
|||||||
for (const entry of entries) {
|
for (const entry of entries) {
|
||||||
const encoder = new TextEncoder();
|
const encoder = new TextEncoder();
|
||||||
const keyBytes = encoder.encode(entry.key);
|
const keyBytes = encoder.encode(entry.key);
|
||||||
view.setUint16(offset, keyBytes.length, false);
|
view.setUint32(offset, keyBytes.length, false);
|
||||||
offset += 2;
|
offset += 4;
|
||||||
new Uint8Array(view.buffer).set(keyBytes, offset);
|
new Uint8Array(view.buffer).set(keyBytes, offset);
|
||||||
offset += keyBytes.length;
|
offset += keyBytes.length;
|
||||||
view.setUint32(offset, entry.blockOffset, false);
|
view.setUint32(offset, entry.blockOffset, false);
|
||||||
|
|||||||
@@ -23,8 +23,8 @@ import { createSchema } from '../src/table/schema';
|
|||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
describe('[v0.2.5] P0-1: 版本号统一', () => {
|
describe('[v0.2.5] P0-1: 版本号统一', () => {
|
||||||
test('VERSION 常量为当前版本(0.4.3)', () => {
|
test('VERSION 常量为当前版本(0.4.4)', () => {
|
||||||
expect(VERSION).toBe('0.4.3');
|
expect(VERSION).toBe('0.4.4');
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
@@ -400,7 +400,7 @@ describe('[v0.3.3] P1-9: Savepoint + MVCC 一致性', () => {
|
|||||||
|
|
||||||
describe('[v0.3.3] 端到端', () => {
|
describe('[v0.3.3] 端到端', () => {
|
||||||
test('全部修复点可共存于 MetonaSqlark API', async () => {
|
test('全部修复点可共存于 MetonaSqlark API', async () => {
|
||||||
expect(VERSION).toBe('0.4.3');
|
expect(VERSION).toBe('0.4.4');
|
||||||
const db = new MetonaSqlark({ name: `e2e-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`, mode: 'memory' });
|
const db = new MetonaSqlark({ name: `e2e-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`, mode: 'memory' });
|
||||||
await db.init();
|
await db.init();
|
||||||
await db.defineTable('users', {
|
await db.defineTable('users', {
|
||||||
|
|||||||
@@ -0,0 +1,228 @@
|
|||||||
|
/**
|
||||||
|
* v0.4.4 回归测试 — SSTable 编码缺陷修复
|
||||||
|
* 覆盖:
|
||||||
|
* - P0: 大段中文内容(300KB)写入 aria → flush → close → reopen 数据完整(缓冲区低估越界)
|
||||||
|
* - P0: >64KB value(u16 长度截断)
|
||||||
|
* - P0: 大 value 超过块大小(4096)→ 单条独立成块读取正确
|
||||||
|
* - P1: v1 旧格式文件兼容读取(旧库数据不丢)
|
||||||
|
* - P1: 中文 key / 混合条目
|
||||||
|
*/
|
||||||
|
|
||||||
|
import { AriaEngine } from '../src/engine/aria/index';
|
||||||
|
import { createSchema } from '../src/table/schema';
|
||||||
|
import { SSTableBuilder } from '../src/engine/aria/index/sstable_builder';
|
||||||
|
import { SSTableReader } from '../src/engine/aria/index/sstable';
|
||||||
|
import type { SSTableMeta } from '../src/engine/aria/types';
|
||||||
|
import 'fake-indexeddb/auto';
|
||||||
|
|
||||||
|
let idbCounter = 0;
|
||||||
|
function uniqueDB(): string {
|
||||||
|
return `r45-${Date.now()}-${++idbCounter}-${Math.random().toString(36).slice(2, 8)}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
const makeMeta = (data: Uint8Array): SSTableMeta => ({
|
||||||
|
id: 1, level: 0, minKey: '', maxKey: '\uffff',
|
||||||
|
blockCount: 1, totalSize: data.byteLength, bloomData: null,
|
||||||
|
});
|
||||||
|
|
||||||
|
/** 300KB 中文内容(模拟真实笔记/文档数据) */
|
||||||
|
function makeBigChineseContent(size = 300 * 1024): string {
|
||||||
|
const chunk = '这是一段用于测试的中文内容。数据持久化必须正确处理 UTF-8 多字节编码!';
|
||||||
|
return chunk.repeat(Math.ceil(size / chunk.length)).slice(0, size);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ===================================================================
|
||||||
|
// P0: 大内容编码(builder 字节估算)
|
||||||
|
// ===================================================================
|
||||||
|
|
||||||
|
describe('P0 — SSTable 编码:UTF-8 字节估算', () => {
|
||||||
|
it('构建含 300KB 中文内容的 SSTable 不越界,可完整读回', () => {
|
||||||
|
const builder = new SSTableBuilder(4096);
|
||||||
|
const content = makeBigChineseContent();
|
||||||
|
builder.add('note:1', { id: '1', title: '长文笔记', body: content });
|
||||||
|
builder.add('note:2', { id: '2', body: '小内容' });
|
||||||
|
const { sstableData } = builder.build();
|
||||||
|
// 修复前:缓冲区低估 → build 抛 RangeError
|
||||||
|
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
|
||||||
|
const note1 = reader.get('note:1');
|
||||||
|
expect(note1).not.toBeNull();
|
||||||
|
expect((note1 as any).body).toBe(content);
|
||||||
|
expect((note1 as any).body.length).toBe(content.length);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('构建含 >64KB value 的 SSTable(u32 长度字段,修复前 u16 截断)', () => {
|
||||||
|
const builder = new SSTableBuilder(4096);
|
||||||
|
const content = makeBigChineseContent(70 * 1024); // 70KB > 64KB
|
||||||
|
builder.add('doc:big', { id: 'big', data: content });
|
||||||
|
builder.add('doc:small', { id: 'small', data: 'x' });
|
||||||
|
const { sstableData } = builder.build();
|
||||||
|
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
|
||||||
|
const big = reader.get('doc:big');
|
||||||
|
expect((big as any).data).toBe(content);
|
||||||
|
expect((reader.get('doc:small') as any).data).toBe('x');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('大 value(>4096 块大小)单条独立成块,scanAll/get 均正确', () => {
|
||||||
|
const builder = new SSTableBuilder(64); // 极小块阈值
|
||||||
|
const big = makeBigChineseContent(8 * 1024);
|
||||||
|
for (let i = 0; i < 5; i++) {
|
||||||
|
builder.add(`k-${i}`, { v: i, body: i === 2 ? big : 'small' });
|
||||||
|
}
|
||||||
|
const { sstableData, indexEntries } = builder.build();
|
||||||
|
// 大条目独立成块(块数 > 1)
|
||||||
|
expect(indexEntries.length).toBeGreaterThan(1);
|
||||||
|
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
|
||||||
|
const collected: string[] = [];
|
||||||
|
reader.scanAll((k) => collected.push(k));
|
||||||
|
expect(collected).toEqual(['k-0', 'k-1', 'k-2', 'k-3', 'k-4']);
|
||||||
|
expect((reader.get('k-2') as any).body).toBe(big);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('中文 key(多字节)与混合条目', () => {
|
||||||
|
const builder = new SSTableBuilder(4096);
|
||||||
|
builder.add('表:笔记一', { 标题: '中文标题', 内容: '内容' });
|
||||||
|
builder.add('表:笔记二', { 标题: '第二篇' });
|
||||||
|
const { sstableData } = builder.build();
|
||||||
|
const reader = new SSTableReader(sstableData, makeMeta(sstableData));
|
||||||
|
expect((reader.get('表:笔记一') as any)['标题']).toBe('中文标题');
|
||||||
|
expect((reader.get('表:笔记二') as any)['标题']).toBe('第二篇');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('v2 魔数写入正确(SSTC)', () => {
|
||||||
|
const builder = new SSTableBuilder(4096);
|
||||||
|
builder.add('a', { v: 1 });
|
||||||
|
const { sstableData } = builder.build();
|
||||||
|
const magic = new DataView(sstableData.buffer, sstableData.byteOffset, sstableData.byteLength)
|
||||||
|
.getUint32(sstableData.byteLength - 8, false);
|
||||||
|
expect(magic).toBe(0x53535443); // "SSTC"
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ===================================================================
|
||||||
|
// P1: v1 旧格式兼容
|
||||||
|
// ===================================================================
|
||||||
|
|
||||||
|
describe('P1 — v1 旧格式兼容', () => {
|
||||||
|
/** 手工构造 v1 格式(keyLen/valueLen u16 + magic SSTB) */
|
||||||
|
function buildV1SSTable(entries: [string, Record<string, unknown>][]): Uint8Array {
|
||||||
|
const enc = new TextEncoder();
|
||||||
|
const encoded = entries.map(([k, v]) => ({ key: k, keyBytes: enc.encode(k), valueBytes: enc.encode(JSON.stringify(v)) }));
|
||||||
|
// 块(全部放一个块)
|
||||||
|
const entryCount = entries.length;
|
||||||
|
let blockSize = 4;
|
||||||
|
for (const e of encoded) blockSize += 2 + e.keyBytes.length + 2 + e.valueBytes.length;
|
||||||
|
const indexSize = 4 + 2 + encoded[encoded.length - 1].keyBytes.length + 8;
|
||||||
|
const bloomSize = 0;
|
||||||
|
const total = blockSize + indexSize + 32;
|
||||||
|
const buf = new ArrayBuffer(total);
|
||||||
|
const view = new DataView(buf);
|
||||||
|
let offset = 0;
|
||||||
|
view.setUint32(offset, entryCount, false); offset += 4;
|
||||||
|
for (const e of encoded) {
|
||||||
|
view.setUint16(offset, e.keyBytes.length, false); offset += 2;
|
||||||
|
new Uint8Array(buf).set(e.keyBytes, offset); offset += e.keyBytes.length;
|
||||||
|
view.setUint16(offset, e.valueBytes.length, false); offset += 2;
|
||||||
|
new Uint8Array(buf).set(e.valueBytes, offset); offset += e.valueBytes.length;
|
||||||
|
}
|
||||||
|
const indexOffset = offset;
|
||||||
|
view.setUint32(offset, 1, false); offset += 4;
|
||||||
|
const last = encoded[encoded.length - 1];
|
||||||
|
view.setUint16(offset, last.keyBytes.length, false); offset += 2;
|
||||||
|
new Uint8Array(buf).set(last.keyBytes, offset); offset += last.keyBytes.length;
|
||||||
|
view.setUint32(offset, 0, false); offset += 4;
|
||||||
|
view.setUint32(offset, blockSize, false); offset += 4;
|
||||||
|
const footer = offset;
|
||||||
|
view.setUint32(footer, indexOffset, false);
|
||||||
|
view.setUint32(footer + 4, indexSize, false);
|
||||||
|
view.setUint32(footer + 8, 0, false); // bloom_offset
|
||||||
|
view.setUint32(footer + 12, 0, false); // bloom_size
|
||||||
|
view.setUint32(footer + 16, 0, false); // hash_count
|
||||||
|
view.setUint32(footer + 20, entryCount, false);
|
||||||
|
view.setUint32(footer + 24, 0x53535442, false); // SSTB
|
||||||
|
view.setUint32(footer + 28, 0, false);
|
||||||
|
return new Uint8Array(buf);
|
||||||
|
}
|
||||||
|
|
||||||
|
it('v1 格式文件可读(旧库数据不丢)', () => {
|
||||||
|
const v1 = buildV1SSTable([
|
||||||
|
['u:1', { name: 'Alice', age: 30 }],
|
||||||
|
['u:2', { name: 'Bob', age: 25 }],
|
||||||
|
]);
|
||||||
|
const reader = new SSTableReader(v1, makeMeta(v1));
|
||||||
|
expect((reader.get('u:1') as any).name).toBe('Alice');
|
||||||
|
expect((reader.get('u:2') as any).name).toBe('Bob');
|
||||||
|
const keys: string[] = [];
|
||||||
|
reader.scanAll((k) => keys.push(k));
|
||||||
|
expect(keys).toEqual(['u:1', 'u:2']);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// ===================================================================
|
||||||
|
// P0: 端到端 — 大内容跨重启(爸爸 Electron 冒烟场景)
|
||||||
|
// ===================================================================
|
||||||
|
|
||||||
|
describe('P0 — 大内容端到端(aria 模式)', () => {
|
||||||
|
it('300KB 中文写入 → close → reopen 数据完整(修复前打开必崩)', async () => {
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
const engine = new AriaEngine({ storageBackend: 'indexeddb', checkpointInterval: 100000 });
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(createSchema('notes', {
|
||||||
|
id: { type: 'string', primaryKey: true },
|
||||||
|
title: { type: 'string' },
|
||||||
|
body: { type: 'string' },
|
||||||
|
}));
|
||||||
|
|
||||||
|
const content = makeBigChineseContent();
|
||||||
|
await engine.insert('notes', [
|
||||||
|
{ id: 'n1', title: '长文', body: content },
|
||||||
|
{ id: 'n2', title: '短文', body: 'hello' },
|
||||||
|
]);
|
||||||
|
// 强制落盘(修复前:flush 编码越界崩溃)
|
||||||
|
await (engine as any).lsm.flush();
|
||||||
|
await engine.close();
|
||||||
|
|
||||||
|
// 修复前:打开解析半写文件 → ARIA_OPEN_ERROR
|
||||||
|
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', checkpointInterval: 100000 });
|
||||||
|
await expect(engine2.open(dbName, 1)).resolves.toBeUndefined();
|
||||||
|
const rows = await engine2.find('notes', { table: 'notes' });
|
||||||
|
expect(rows).toHaveLength(2);
|
||||||
|
const byId = Object.fromEntries(rows.map((r) => [r.id, r]));
|
||||||
|
expect(byId['n1'].body).toBe(content);
|
||||||
|
expect(byId['n1'].body.length).toBe(content.length);
|
||||||
|
expect(byId['n2'].body).toBe('hello');
|
||||||
|
await engine2.close();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('多行 >64KB 内容批量写入重开完整(含索引列)', async () => {
|
||||||
|
const dbName = uniqueDB();
|
||||||
|
const engine = new AriaEngine({
|
||||||
|
storageBackend: 'indexeddb',
|
||||||
|
memtableSizeThreshold: 32 * 1024,
|
||||||
|
checkpointInterval: 100000,
|
||||||
|
});
|
||||||
|
await engine.open(dbName, 1);
|
||||||
|
await engine.createTable(createSchema('docs', {
|
||||||
|
id: { type: 'string', primaryKey: true },
|
||||||
|
tag: { type: 'string', index: true },
|
||||||
|
body: { type: 'string' },
|
||||||
|
}));
|
||||||
|
const contents: string[] = [];
|
||||||
|
for (let i = 0; i < 10; i++) {
|
||||||
|
const c = makeBigChineseContent(70 * 1024 + i * 100);
|
||||||
|
contents.push(c);
|
||||||
|
await engine.insert('docs', [{ id: `d${i}`, tag: i % 2 === 0 ? 'even' : 'odd', body: c }]);
|
||||||
|
}
|
||||||
|
await engine.close();
|
||||||
|
|
||||||
|
const engine2 = new AriaEngine({ storageBackend: 'indexeddb', checkpointInterval: 100000 });
|
||||||
|
await engine2.open(dbName, 1);
|
||||||
|
expect(await engine2.count('docs')).toBe(10);
|
||||||
|
const evens = await engine2.find('docs', { table: 'docs', where: { tag: 'even' } });
|
||||||
|
expect(evens).toHaveLength(5);
|
||||||
|
for (let i = 0; i < 10; i++) {
|
||||||
|
const rows = await engine2.find('docs', { table: 'docs', where: { id: `d${i}` } });
|
||||||
|
expect(rows[0].body).toBe(contents[i]);
|
||||||
|
}
|
||||||
|
await engine2.close();
|
||||||
|
});
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user