thzxx
0b44620721
方法：四个对抗性子代理分头审查（数据正确性 / 文档宣称 vs 实现 / 公共 API 契约 / 测试质量），每条结论要求可复现证据；逐条复核 + 探针确认 + 变异验证（40 项全部 被对应用例拦住）。 P0：事务活跃期间 repair()/close()/周期 checkpoint 推进 WAL 水位 → 已 COMMIT 的 事务整批消失且恢复报告"干净"。根因 hasPendingFlushData()/computeDurableLsn() 不看 txnSnapshot；守卫此前只在 CheckpointManager 两个回调里。修复：守卫下沉到 computeDurableLsn() 与 advanceWalCheckpoint() 入口（唯一实现）。 P1： - WAL 前缀缺失丢弃整段活分片（回退上一代 manifest 时 kept 为空）→ 前缀缺失单独 记录，后缀照常重放；仅 fromLsn === 0 时才算真异常 - 孤儿回收门槛只看引擎层 dataLossSuspected，漏掉 LSM 层被丢的 SSTable → 统一 describeRecoveryDamage() 聚合判定（损坏时绝不删"引用不到"的文件） - vacuum() 逐层压缩绕过维护链 → vacuumLevels() 每层作为维护链任务执行 - reclaimRetiredNow() 无视在途读者（读者把"已退休"读成"文件损坏"）→ 有读者时 退化为延迟回收 P2：WAL 记录级 CRC 损坏不计数不上报；旧格式表结构记录形状损坏静默当空库； bloomFilterBitsPerKey 配置被接受却完全不生效（构建器写死默认值，实现缺陷）； 幽灵 meta；介质读故障等于文件损坏的语义无用例；manifest 回读校验两条守卫无用例； 文件名≠载荷世代判定无用例；pageIdWatermark 单调性无用例；分片号两条真实不变量 无用例。 覆盖率口径（第二处漏洞）：interface.ts 混着三个运行时函数（cloneRow 等）却被 描述为"纯类型、不纳入统计" → 实现搬到 src/engine/row_clone.ts；搬完门禁真的 失败（functions 93.84% < 94%），补测退化路径后通过。 测试质量：3 条空壳用例改值级断言；1 条"全损坏"用例实际只走缓存 → 拆成两条真 用例；5 秒墙钟 race 改门控 + 失败上限；setTimeout 改 whenIdle()；<= 收紧为 <。 变异脚本加固：正控（干净基线必须全绿）、编译失败/0 用例单独归类、300s 超时、 逐字节 sha256 恢复校验、O_EXCL 进程锁、锚点唯一性；变异 22 → 40 项。 文档两轮订正（16 + 11 条不成立宣称）：MVCC 快照隔离、backup 一致性快照、 "空洞检测截断"、体积（251,109 B / gzip 63,145 B）、测试与覆盖率数字、 "5 种存储引擎"、Tree-shakable、错误码表补 16 个码、恢复报告字段、已知限制 （回退单向 / 多实例依赖 Web Locks / manifest 体积 / 尾部 WAL 分片不可识别）。 验证：常规套件 92 套件 / 1980 用例全绿；覆盖率 90.59 / 82.59 / 94.14 / 93.50 （阈值 90/82/94/93）；e2e 14/14（真实 Chromium + OPFS + CDP 崩溃）； 重型套件 4 套件 / 27 用例；变异 40/40；lint + 两份 tsc 干净；dist 已重建。
264 lines
11 KiB
Markdown
264 lines
11 KiB
Markdown
# Contributing to MetonaSqlark
|
||
|
||
Thanks for your interest in contributing! This document outlines the development workflow and conventions.
|
||
|
||
## Prerequisites
|
||
|
||
- Node.js >= 16.0.0
|
||
- npm >= 8.0.0
|
||
|
||
## Setup
|
||
|
||
```bash
|
||
git clone https://git.metona.cn/MetonaTeam/MetonaSqlark.git
|
||
cd MetonaSqlark
|
||
npm install
|
||
```
|
||
|
||
## Development
|
||
|
||
```bash
|
||
# Start dev server with hot reload (port 3001)
|
||
npm run dev
|
||
|
||
# Run tests in watch mode
|
||
npm run test:watch
|
||
|
||
# Type check
|
||
npm run typecheck
|
||
|
||
# Lint
|
||
npm run lint
|
||
npm run lint:fix
|
||
|
||
# Format
|
||
npm run format
|
||
```
|
||
|
||
## Project Structure
|
||
|
||
```
|
||
src/
|
||
├── index.ts # Entry point, global API (MetonaSqlark + MeSqlark)
|
||
├── core.ts # MetonaSqlark main class
|
||
├── constants.ts # Types, defaults, enums, errors
|
||
├── connection-manager.ts # Connection pool (connect/disconnect)
|
||
├── engine/ # Storage engines
|
||
│ ├── interface.ts # IStorageEngine interface
|
||
│ ├── memory.ts # MemoryEngine (Map-based)
|
||
│ ├── kvstore_engine.ts # KVStoreEngine (disk mode, self-built KV store)
|
||
│ ├── kvstore/ # KVStore (log + snapshot + atomic multi-key write)
|
||
│ └── aria/ # AriaEngine (LSM-Tree page storage engine)
|
||
│ ├── index/ # LSM / MemTable / SSTable / Bloom / MergeIterator
|
||
│ ├── page/ # 4KB slotted page format
|
||
│ ├── buffer/ # Buffer Pool (LRU eviction)
|
||
│ ├── wal/ # Write-Ahead Log + Checkpoint
|
||
│ ├── transaction/ # MVCC manager
|
||
│ ├── store/ # Backends (OPFS / KVStore / Memory / Encrypted)
|
||
│ ├── locks.ts # Web Locks multi-tab exclusive lock
|
||
│ └── compression/ # LZ4
|
||
├── hybrid/ # HybridEngine (write-through)
|
||
├── migration/ # Legacy IndexedDB migration tool (one-shot)
|
||
├── table/ # Table management & Schema validation
|
||
├── query/ # Query system
|
||
│ ├── ast.ts # SQL AST type definitions
|
||
│ ├── builder.ts # Chainable QueryBuilder API
|
||
│ ├── compiler.ts # AST → QueryPlan compiler
|
||
│ ├── executor.ts # QueryExecutor with JOIN/GROUP BY support
|
||
│ └── where-matcher.ts # Unified WHERE matching logic
|
||
├── sql/ # SQL parser
|
||
│ ├── tokens.ts # Token types & keywords
|
||
│ ├── lexer.ts # Tokenizer
|
||
│ ├── parser.ts # Recursive descent parser
|
||
│ └── params.ts # Parameter binding (? placeholders)
|
||
├── transaction/ # Transaction manager
|
||
├── plugin/ # Plugin system (14 lifecycle hooks)
|
||
└── integrations/ # React & Vue hooks
|
||
|
||
tests/ # Test suite (1980 test cases, 92 suites + 14 e2e)
|
||
tests/helpers/ # 共享测试工具（OPFS mock 等）
|
||
tests/e2e/ # Playwright e2e（真实 Chromium + OPFS）
|
||
site/ # Documentation site (index / docs / demo)
|
||
```
|
||
|
||
## Code Conventions
|
||
|
||
### TypeScript
|
||
- **Strict mode** is enabled — all code must pass `tsc --noEmit`.
|
||
- Export types explicitly. Avoid `any` where possible.
|
||
- Use `interface` for object shapes, `type` for unions/primitives.
|
||
|
||
### Style
|
||
- Run `npm run format` before committing (uses Prettier).
|
||
- Follow existing comment patterns: JSDoc `/** */` for public APIs, `//` for inline notes.
|
||
- Keep functions focused and under ~60 lines where practical.
|
||
|
||
### Testing
|
||
- **Every new feature must include tests.**
|
||
- Test files mirror source structure: `src/foo.ts` → `tests/foo.test.ts`.
|
||
- Use descriptive test names: `('does X when Y')`.
|
||
- Run the full suite before submitting: `npm test`.
|
||
|
||
### Commits
|
||
- Use conventional commit messages:
|
||
- `feat: add SQL GROUP BY support`
|
||
- `fix: JOIN with multiple ON conditions`
|
||
- `docs: update API reference`
|
||
- `test: add edge case coverage`
|
||
- `chore: optimize rollup dev build`
|
||
|
||
## Architecture
|
||
|
||
MetonaSqlark follows a layered architecture:
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────┐
|
||
│ MetonaSqlark / MeSqlark (Public API) │
|
||
├──────────────────────────────────────────────────┤
|
||
│ SQL Parser │ QueryBuilder │ Transaction │ Plugin │
|
||
│ (Lexer→AST) │ (.select…) │ Manager │ System │
|
||
├──────────────────────────────────────────────────┤
|
||
│ QueryExecutor (AST → Results) │
|
||
│ JOIN · GROUP BY · HAVING · DISTINCT · Agg │
|
||
├──────────────────────────────────────────────────┤
|
||
│ IStorageEngine (Interface) │
|
||
├─────────┬──────────┬──────────┬──────────────────┤
|
||
│ Memory │ IndexedDB │ OPFS │ Hybrid │
|
||
│ Engine │ Engine │ Engine │ Engine │
|
||
└─────────┴──────────┴──────────┴──────────────────┘
|
||
```
|
||
|
||
### Key Design Decisions
|
||
|
||
1. **AST as Universal Intermediate Representation**: Both SQL strings and Query Builder produce the same AST, ensuring consistent behavior regardless of API used.
|
||
|
||
2. **Write-Through Hybrid Strategy**: When using `mode: 'hybrid'`, all writes go to both memory and disk simultaneously. Reads always hit memory for maximum speed.
|
||
|
||
3. **Plugin Hook Pipeline**: 14 lifecycle hooks let plugins observe and adjust database
|
||
operations without modifying core code. **Hook contract (v0.8.0)**:
|
||
- return values are **ignored** (you cannot cancel an operation by returning `false`,
|
||
nor rewrite the SQL by returning a string);
|
||
- **mutate the argument object in place** to change it — this works on the Table API
|
||
path (`db.table(...).insert(rows)`);
|
||
- **throw** to abort the operation (the error propagates to the caller);
|
||
- the SQL path passes a **copy** for `beforeInsert`, so mutating it there has no effect.
|
||
Use the Table API when you need to transform rows.
|
||
These rules are covered by tests; changing them requires updating this section.
|
||
|
||
4. **Hand-Written SQL Parser**: No dependencies on parser generators — a recursive-descent parser keeps the bundle size minimal.
|
||
|
||
## Building
|
||
|
||
```bash
|
||
# Production build (all formats)
|
||
npm run build
|
||
|
||
# Output in dist/
|
||
# ├── metona-sqlark.js UMD
|
||
# ├── metona-sqlark.min.js UMD minified (251,109 B / gzip 63,145 B)
|
||
# ├── metona-sqlark.esm.js ES Module
|
||
# ├── metona-sqlark.cjs CommonJS
|
||
# └── metona-sqlark.d.ts TypeScript declarations
|
||
```
|
||
|
||
## Plugin Development
|
||
|
||
Plugins follow a simple convention:
|
||
|
||
```typescript
|
||
import type { MetonaPlugin } from '@metona-team/metona-sqlark';
|
||
|
||
const myPlugin: MetonaPlugin = {
|
||
name: 'myPlugin',
|
||
version: '1.0.0',
|
||
description: 'Description of my plugin',
|
||
// higher = executed first（v0.8.0 起真正生效：install 与钩子都按优先级降序；
|
||
// 同优先级保持 plugins 数组顺序）
|
||
priority: 50,
|
||
|
||
install(db) {
|
||
// Use db.on() to subscribe to hooks
|
||
db.on('beforeInsert', async (rows) => {
|
||
// Validate or transform data
|
||
});
|
||
db.on('afterQuery', async (sql, result) => {
|
||
// Log or cache query results
|
||
});
|
||
},
|
||
|
||
destroy() {
|
||
// Clean up event listeners, timers, etc.
|
||
},
|
||
};
|
||
|
||
// Register in config
|
||
const db = await MetonaSqlark.create({
|
||
name: 'my-app',
|
||
plugins: [myPlugin],
|
||
});
|
||
```
|
||
|
||
### Available Lifecycle Hooks
|
||
|
||
| Hook | Trigger | Parameters |
|
||
|------|---------|------------|
|
||
| `beforeCreateTable` | Before table creation | schema |
|
||
| `afterCreateTable` | After table creation | schema |
|
||
| `beforeDropTable` | Before table drop | tableName |
|
||
| `afterDropTable` | After table drop | tableName |
|
||
| `beforeInsert` | Before row insert | rows[] |
|
||
| `afterInsert` | After row insert | rows[] |
|
||
| `beforeUpdate` | Before row update | query, updates |
|
||
| `afterUpdate` | After row update | query, updates, count |
|
||
| `beforeDelete` | Before row delete | query |
|
||
| `afterDelete` | After row delete | query, count |
|
||
| `beforeQuery` | Before SQL query | sql |
|
||
| `afterQuery` | After SQL query | sql, result |
|
||
| `beforeTransaction` | Before transaction | - |
|
||
| `afterTransaction` | After transaction | - |
|
||
|
||
## Releasing
|
||
|
||
1. Update version in `package.json` and `src/constants.ts` (`VERSION` constant).
|
||
2. Update `CHANGELOG.md`.
|
||
3. Run full test suite: `npm test`.
|
||
4. Build: `npm run build`.
|
||
5. Publish: `npm publish`.
|
||
|
||
## Questions?
|
||
|
||
Open an issue at [git.metona.cn/MetonaTeam/MetonaSqlark/issues](https://git.metona.cn/MetonaTeam/MetonaSqlark/issues).
|
||
|
||
---
|
||
|
||
## 变异验证（回归套件的"是否只是陪跑"检查）
|
||
|
||
修复类提交必须能回答一个问题：**把修复回退到修复前的行为，对应用例会不会失败？**
|
||
不会失败的用例等于没有保护。
|
||
|
||
```bash
|
||
# 全部 40 项变异（存储单一提交点 / LSM 结构根治 / v0.8.0 全量审查修复）
|
||
python3 scripts/mutation-b6.py
|
||
|
||
# 只跑其中一条（按名字子串匹配）
|
||
python3 scripts/mutation-b6.py "WAL 分片号复用"
|
||
```
|
||
|
||
脚本会临时改写 `src/`、运行对应用例、再恢复源码（收到 SIGINT/SIGTERM 也会恢复），
|
||
最后打印每一条是"被拦住"还是"仍然通过"。**出现任何一条"仍然通过"，本次提交不算完成。**
|
||
|
||
### 脚本自身的保证（v0.8.0 审查后加固）
|
||
|
||
| 机制 | 为什么需要 |
|
||
| --- | --- |
|
||
| **正控** | 先跑一遍未变异的干净基线（B-6 套件 + 分片 WAL 套件必须全绿）。没有正控就无法区分"变异被测试拦住"与"这套件本来就是红的" |
|
||
| **编译失败 ≠ 被拦住** | `Test suite failed to run` / `error TS####` 单独归类为 `BAD(变异破坏编译)`：一个把源码改成语法错误的"变异"什么都没验证 |
|
||
| **用例未匹配 ≠ 被拦住** | `Tests: 0 total` / `No tests found` 归类为 `BAD(用例未匹配)`（模式写错时最容易被误读成"测试通过了"） |
|
||
| **锚点唯一性** | 锚点在文件里出现多次时拒绝变异（否则可能改到另一处、验错位置） |
|
||
| **超时** | 单条变异 300s 上限，超时归类为 `BAD(超时)` 而不是放过 |
|
||
| **逐字节恢复校验** | 每条变异跑完用 sha256 比对恢复结果；不一致立即 `FATAL` 退出 |
|
||
| **进程锁** | `.mutation-b6.lock`（`O_EXCL`）：并发运行会互相踩掉对方的源码，直接拒绝启动 |
|
||
|
||
判定顺序固定为：编译失败 → 用例未匹配 → `✕`/`Tests: N failed`（= 被拦住）→ 其余为"仍然通过"。
|
||
只有 `OK(变异被拦住)` 计入成功，其余任何状态都会让脚本以非零退出码结束。
|