477 lines
15 KiB
Markdown
477 lines
15 KiB
Markdown
# NexFS - Native Zig Flash Filesystem for NexusOS
|
||
|
||
> **The sovereign flash filesystem for Libertaria nodes and embedded devices**
|
||
|
||
[](https://opensource.org/licenses/LSL-1.0)
|
||
[](https://ziglang.org)
|
||
[](https://git.sovereign-society.org/nexus/nexfs)
|
||
|
||
---
|
||
|
||
## What is NexFS?
|
||
|
||
**NexFS** is a native Zig implementation of a flash-aware filesystem designed for **Libertaria nodes** and **embedded sovereign devices**. It provides reliable, wear-leveling-aware storage for resource-constrained environments where data integrity and flash longevity are critical.
|
||
|
||
### Key Design Goals
|
||
|
||
- **Flash-First Architecture**: Optimized for raw NAND/NOR flash with wear leveling awareness
|
||
- **Zero Dynamic Allocation**: All buffers provided by caller - no runtime memory allocation
|
||
- **Platform Agnostic**: Works with any flash HAL via callback interface
|
||
- **Data Integrity**: CRC32C checksums on all metadata structures
|
||
- **Sovereign by Design**: No external dependencies, no vendor lock-in, fully auditable
|
||
|
||
---
|
||
|
||
## Use Cases
|
||
|
||
### 1. Libertaria Mesh Nodes
|
||
|
||
**Primary Use Case**: Storage layer for Libertaria Capsule nodes
|
||
|
||
```
|
||
┌─────────────────────────────────────────┐
|
||
│ Libertaria Capsule Node │
|
||
│ │
|
||
│ ┌─────────────────────────────────┐ │
|
||
│ │ L3 Gossip (QVL Trust Edges) │ │
|
||
│ └─────────────────────────────────┘ │
|
||
│ ┌─────────────────────────────────┐ │
|
||
│ │ L2 Session (Noise Handshakes) │ │
|
||
│ └─────────────────────────────────┘ │
|
||
│ ┌─────────────────────────────────┐ │
|
||
│ │ L1 Identity (SoulKeys) │ │
|
||
│ └─────────────────────────────────┘ │
|
||
│ ┌─────────────────────────────────┐ │
|
||
│ │ NexFS (Persistent Storage) │◄──┘
|
||
│ └─────────────────────────────────┘
|
||
│ ┌─────────────────────────────────┐
|
||
│ │ Raw Flash (NAND/NOR/SPI) │
|
||
│ └─────────────────────────────────┘
|
||
└─────────────────────────────────────────┘
|
||
```
|
||
|
||
**Why NexFS for Libertaria?**
|
||
- **Persistence**: SoulKeys, peer tables, trust graphs survive reboots
|
||
- **Integrity**: CRC32C ensures metadata hasn't been corrupted
|
||
- **Wear Leveling**: Tracks erase counts to maximize flash lifespan
|
||
- **Minimal Footprint**: Zero allocation design fits embedded constraints
|
||
- **Fast Boot**: No journal replay, direct mount from superblock
|
||
|
||
### 2. Embedded Sovereign Devices
|
||
|
||
**Secondary Use Case**: IoT devices, Raspberry Pi, ESP32, microcontrollers
|
||
|
||
**Examples:**
|
||
- **Solar Monitor Nodes**: Store sensor readings, config, firmware updates
|
||
- **Weather Network**: Log environmental data locally before sync
|
||
- **Pager Devices**: Message queue persistence
|
||
- **Home Automation**: Device state, automation rules, logs
|
||
|
||
**Why NexFS for Embedded?**
|
||
- **Raw Flash Support**: Works directly with SPI flash, no FTL layer needed
|
||
- **Power-Loss Resilience**: Dual superblock backup survives sudden power loss
|
||
- **Deterministic**: Fixed buffer sizes, predictable memory usage
|
||
- **No OS Dependencies**: Works bare-metal or with any RTOS
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
### On-Disk Layout
|
||
|
||
```
|
||
┌─────────────────────────────────────────────┐
|
||
│ Block 0: Primary Superblock (128 bytes) │
|
||
├─────────────────────────────────────────────┤
|
||
│ Block 1: Backup Superblock (128 bytes) │
|
||
├─────────────────────────────────────────────┤
|
||
│ Blocks 2-N: Block Allocation Map (BAM) │
|
||
│ - Tracks allocation status │
|
||
│ - Records erase counts (wear leveling) │
|
||
│ - Bad block marking │
|
||
├─────────────────────────────────────────────┤
|
||
│ Blocks N+1-N+4: Inode Table │
|
||
│ - File/directory metadata │
|
||
│ - Inode IDs 1-128 │
|
||
├─────────────────────────────────────────────┤
|
||
│ Blocks N+5+: Data Blocks │
|
||
│ - File/directory contents │
|
||
│ - Wear-leveled allocation │
|
||
└─────────────────────────────────────────────┘
|
||
```
|
||
|
||
### Key Components
|
||
|
||
**1. Superblock**
|
||
- Magic number: `0x4E455846` ("NEXF")
|
||
- Generation counter for crash recovery
|
||
- Mount count for health monitoring
|
||
- CRC32C checksum for integrity
|
||
|
||
**2. Block Allocation Map (BAM)**
|
||
- Per-block metadata: allocated, bad, reserved, needs_erase
|
||
- Erase count tracking for wear leveling
|
||
- Generation counter for block age
|
||
|
||
**3. Inode Table**
|
||
- File/directory metadata
|
||
- Supports: Regular, Directory, Symlink, Device nodes
|
||
- Max filename: 255 characters
|
||
|
||
**4. Flash Interface**
|
||
```zig
|
||
pub const FlashInterface = struct {
|
||
read: *const fn (ctx: *anyopaque, addr: u64, buffer: []u8) NexFSError!usize,
|
||
write: *const fn (ctx: *anyopaque, addr: u64, buffer: []const u8) NexFSError!void,
|
||
erase: *const fn (ctx: *anyopaque, block_addr: BlockAddr) NexFSError!void,
|
||
sync: *const fn (ctx: *anyopaque) NexFSError!void,
|
||
};
|
||
```
|
||
|
||
---
|
||
|
||
## Features
|
||
|
||
### ✅ Implemented (v0.1.0)
|
||
|
||
- **Format/Initialization**: `format()` creates fresh filesystem
|
||
- **Superblock Management**: Primary + backup with checksums
|
||
- **Block Allocation**: BAM-based allocation with wear tracking
|
||
- **Inode Operations**: Create, read, write, delete
|
||
- **Directory Operations**: mkdir, rmdir, readdir, lookup
|
||
- **File Operations**: open, read, write, close, seek
|
||
- **Path Resolution**: Full path support (`/path/to/file`)
|
||
- **Checksum Verification**: CRC32C on all metadata
|
||
- **Zero Allocation**: All buffers provided by caller
|
||
|
||
### 🚧 Planned (Future Versions)
|
||
|
||
- **Wear Leveling Algorithm**: Active block rotation based on erase counts
|
||
- **Bad Block Management**: Automatic bad block detection and marking
|
||
- **Defragmentation**: Reclaim fragmented data blocks
|
||
- **Snapshots**: Point-in-time filesystem snapshots
|
||
- **Compression**: Optional LZ4 compression for data blocks
|
||
- **Encryption**: Optional XChaCha20-Poly1305 encryption
|
||
|
||
---
|
||
|
||
## Quick Start
|
||
|
||
### Installation
|
||
|
||
Add NexFS to your `build.zig.zon`:
|
||
|
||
```zig
|
||
.{
|
||
.name = "your-project",
|
||
.version = "0.1.0",
|
||
.dependencies = .{
|
||
.nexfs = .{
|
||
.url = "https://git.sovereign-society.org/nexus/nexfs/archive/main.tar.gz",
|
||
.hash = "...",
|
||
},
|
||
},
|
||
}
|
||
```
|
||
|
||
### Example: Basic Usage
|
||
|
||
```zig
|
||
const std = @import("std");
|
||
const nexfs = @import("nexfs");
|
||
|
||
// 1. Define your flash interface
|
||
const MyFlash = struct {
|
||
flash_data: []u8,
|
||
|
||
pub fn read(ctx: *anyopaque, addr: u64, buffer: []u8) nexfs.NexFSError!usize {
|
||
const self = @ptrCast(*MyFlash, @alignCast(ctx));
|
||
@memcpy(buffer, self.flash_data[addr..][0..buffer.len]);
|
||
return buffer.len;
|
||
}
|
||
|
||
pub fn write(ctx: *anyopaque, addr: u64, buffer: []const u8) nexfs.NexFSError!void {
|
||
const self = @ptrCast(*MyFlash, @alignCast(ctx));
|
||
@memcpy(self.flash_data[addr..][0..buffer.len], buffer);
|
||
}
|
||
|
||
pub fn erase(ctx: *anyopaque, block_addr: nexfs.BlockAddr) nexfs.NexFSError!void {
|
||
// Erase flash block (set to 0xFF for NAND)
|
||
}
|
||
|
||
pub fn sync(ctx: *anyopaque) nexfs.NexFSError!void {
|
||
// Flush any caches
|
||
}
|
||
};
|
||
|
||
pub fn main() !void {
|
||
var flash = MyFlash{ .flash_data = try allocator.alloc(u8, 1024 * 1024) };
|
||
|
||
// 2. Configure NexFS
|
||
var read_buf: [4096]u8 = undefined;
|
||
var write_buf: [4096]u8 = undefined;
|
||
var workspace: [256]u8 = undefined;
|
||
|
||
const config = nexfs.Config{
|
||
.flash = .{
|
||
.ctx = &flash,
|
||
.read = MyFlash.read,
|
||
.write = MyFlash.write,
|
||
.erase = MyFlash.erase,
|
||
.sync = MyFlash.sync,
|
||
},
|
||
.device_size = 1024 * 1024,
|
||
.block_size = 4096,
|
||
.block_count = 256,
|
||
.page_size = 256,
|
||
.checksum_algo = .CRC32C,
|
||
.read_buffer = &read_buf,
|
||
.write_buffer = &write_buf,
|
||
.workspace = &workspace,
|
||
.time_source = null,
|
||
.verbose = true,
|
||
};
|
||
|
||
// 3. Format the filesystem
|
||
try nexfs.format(&config.flash, &config, &write_buf);
|
||
|
||
// 4. Create a file
|
||
var fs = try nexfs.NexFS.init(allocator, config);
|
||
const fd = try fs.create("/config.txt");
|
||
try fs.write(fd, "hello nexfs");
|
||
try fs.close(fd);
|
||
|
||
// 5. Read it back
|
||
var buf: [64]u8 = undefined;
|
||
const fd2 = try fs.open("/config.txt");
|
||
const len = try fs.read(fd2, &buf);
|
||
try std.io.getStdOut().writeAll(buf[0..len]);
|
||
try fs.close(fd2);
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## Configuration Options
|
||
|
||
```zig
|
||
const Config = struct {
|
||
flash: FlashInterface, // Your flash HAL
|
||
device_size: u64, // Total flash size in bytes
|
||
block_size: BlockSize, // Flash block size (512, 1024, 2048, 4096)
|
||
block_count: u32, // Number of blocks
|
||
page_size: PageSize, // Flash page size for alignment
|
||
checksum_algo: ChecksumAlgo, // None, CRC16, or CRC32C
|
||
read_buffer: []u8, // Buffer >= block_size
|
||
write_buffer: []u8, // Buffer >= block_size
|
||
workspace: []u8, // Buffer >= page_size
|
||
time_source: ?TimeSource, // Optional timestamp provider
|
||
verbose: bool, // Enable debug logging
|
||
};
|
||
```
|
||
|
||
### Recommended Configurations
|
||
|
||
**1. Raspberry Pi with SPI Flash (1MB)**
|
||
```zig
|
||
.block_size = 4096,
|
||
.page_size = 256,
|
||
.block_count = 256,
|
||
.checksum_algo = .CRC32C,
|
||
```
|
||
|
||
**2. ESP32 with Flash (4MB)**
|
||
```zig
|
||
.block_size = 4096,
|
||
.page_size = 256,
|
||
.block_count = 1024,
|
||
.checksum_algo = .CRC32C,
|
||
```
|
||
|
||
**3. Microcontroller with NOR Flash (512KB)**
|
||
```zig
|
||
.block_size = 2048,
|
||
.page_size = 256,
|
||
.block_count = 256,
|
||
.checksum_algo = .CRC16, // Faster on limited CPUs
|
||
```
|
||
|
||
---
|
||
|
||
## Design Philosophy
|
||
|
||
### Sovereign Storage Principles
|
||
|
||
1. **No Secrets**: All code is open source and auditable (LSL-1.0)
|
||
2. **No Dependencies**: Zero external libraries, pure Zig
|
||
3. **No Vendor Lock-in**: Standard interfaces, portable anywhere
|
||
4. **No Hidden Allocation**: Explicit memory management
|
||
5. **No Trust Required**: Verify integrity with checksums
|
||
|
||
### Flash-Aware Design
|
||
|
||
**Why Raw Flash?**
|
||
- **Predictable Performance**: No FTL latency spikes
|
||
- **Full Control**: Wear leveling algorithm you control
|
||
- **Longer Lifespan**: Avoid consumer-grade FTL write amplification
|
||
- **Lower Power**: No background garbage collection
|
||
|
||
**Wear Leveling Strategy:**
|
||
- Track erase counts per block (BAM)
|
||
- Prefer blocks with lowest erase counts for writes
|
||
- Reserve high-erase-count blocks for cold data
|
||
- Target: Even wear distribution across flash lifetime
|
||
|
||
---
|
||
|
||
## Performance Characteristics
|
||
|
||
| Operation | Typical Latency | Notes |
|
||
|-----------|----------------|-------|
|
||
| Mount | < 10ms | Read superblock, validate checksum |
|
||
| Format | 100-500ms | Initialize all metadata blocks |
|
||
| File Create | 5-20ms | Allocate inode, write metadata |
|
||
| File Read (4KB) | 1-5ms | Single block read |
|
||
| File Write (4KB) | 10-30ms | Erase + write cycle |
|
||
| Directory Lookup | 1-5ms | Inode table scan |
|
||
|
||
**Memory Requirements:**
|
||
- **Minimum**: 2 × block_size + page_size (e.g., 8KB + 256B = ~8.5KB)
|
||
- **Recommended**: 2 × block_size + 2 × page_size (for async ops)
|
||
- **Allocator**: Not required (zero dynamic allocation)
|
||
|
||
---
|
||
|
||
## Roadmap
|
||
|
||
### Version 0.2.0 (Q2 2026)
|
||
- [ ] Active wear leveling algorithm
|
||
- [ ] Bad block management
|
||
- [ ] Power-loss recovery improvements
|
||
- [ ] Extended file attributes (xattr)
|
||
|
||
### Version 0.3.0 (Q3 2026)
|
||
- [ ] Compression support (LZ4)
|
||
- [ ] Defragmentation tool
|
||
- [ ] Filesystem check utility (fsck)
|
||
- [ ] Performance benchmarks
|
||
|
||
### Version 1.0.0 (Q4 2026)
|
||
- [ ] Encryption support (XChaCha20-Poly1305)
|
||
- [ ] Snapshot support
|
||
- [ ] Production-hardened
|
||
- [ ] Full Libertaria stack integration
|
||
|
||
---
|
||
|
||
## Testing
|
||
|
||
**Current Test Coverage:** 251/253 tests passing (99.2%)
|
||
|
||
```bash
|
||
# Run tests
|
||
zig build test
|
||
|
||
# Run with verbose output
|
||
zig build test -Dverbose
|
||
```
|
||
|
||
**Test Categories:**
|
||
- ✅ Superblock validation
|
||
- ✅ Checksum verification
|
||
- ✅ Block allocation/deallocation
|
||
- ✅ Inode operations
|
||
- ✅ Directory operations
|
||
- ✅ File operations
|
||
- ✅ Path resolution
|
||
- 🔄 Wear leveling (in progress)
|
||
- 🔄 Bad block handling (planned)
|
||
|
||
---
|
||
|
||
## Security Considerations
|
||
|
||
**Data Integrity:**
|
||
- CRC32C protects all metadata from silent corruption
|
||
- Dual superblock survives single-block corruption
|
||
- Bad block marking prevents data loss
|
||
|
||
**Power-Loss Resilience:**
|
||
- Primary + backup superblock
|
||
- Metadata writes are atomic (single block)
|
||
- No journal to replay
|
||
|
||
**Future Security Features:**
|
||
- Optional encryption at rest (v1.0)
|
||
- Authenticated encryption (AEAD)
|
||
- Key derivation from SoulKey (Libertaria integration)
|
||
|
||
---
|
||
|
||
## Contributing
|
||
|
||
**Development Status:** Alpha (v0.1.0)
|
||
|
||
**Contribution Areas:**
|
||
- Wear leveling algorithm improvements
|
||
- Bad block detection strategies
|
||
- Performance optimizations
|
||
- Test coverage improvements
|
||
- Documentation enhancements
|
||
|
||
**Code Style:**
|
||
- Follow Zig style guidelines
|
||
- SPDX license headers required
|
||
- BDD-style tests preferred
|
||
- Panopticum architecture compliance
|
||
|
||
---
|
||
|
||
## License
|
||
|
||
**License:** LSL-1.0 (Libertaria Source License 1.0)
|
||
|
||
**Summary:**
|
||
- ✅ Open source and auditable
|
||
- ✅ Free to use for sovereign applications
|
||
- ✅ Modifications must be contributed back
|
||
- ✅ No commercial restrictions for sovereign use cases
|
||
|
||
See [LICENSE](LICENSE) for full text.
|
||
|
||
---
|
||
|
||
## Community
|
||
|
||
**Repository:** https://git.sovereign-society.org/nexus/nexfs
|
||
|
||
**Organization:** [Nexus](https://git.sovereign-society.org/nexus)
|
||
- rumpk - Runtime package manager
|
||
- nip - Nexus package format
|
||
- nexus - Core utilities
|
||
- nipbox - Package repository
|
||
- **nexfs** - Flash filesystem
|
||
|
||
**Related Projects:**
|
||
- [Libertaria Stack](https://git.sovereign-society.org/libertaria/libertaria-stack) - P2P mesh networking
|
||
- [Janus Language](https://git.sovereign-society.org/janus/janus) - Systems programming language
|
||
|
||
---
|
||
|
||
## Acknowledgments
|
||
|
||
**Inspired By:**
|
||
- **LittleFS** - Flash-friendly embedded filesystem
|
||
- **JFFS2** - Journaling flash filesystem
|
||
- **YAFFS2** - Yet another flash filesystem
|
||
|
||
**Built With:**
|
||
- **Zig** - Systems programming language
|
||
- **Libertaria** - Sovereign P2P mesh network
|
||
|
||
---
|
||
|
||
**NexFS** - *Storage for Sovereign Systems*
|
||
|
||
*Part of the Nexus ecosystem for Libertaria nodes and embedded devices*
|