15 KiB

Raw Blame History

NexFS - Native Zig Flash Filesystem for NexusOS

The sovereign flash filesystem for Libertaria nodes and embedded devices

What is NexFS?

NexFS is a native Zig implementation of a flash-aware filesystem designed for Libertaria nodes and embedded sovereign devices. It provides reliable, wear-leveling-aware storage for resource-constrained environments where data integrity and flash longevity are critical.

Key Design Goals

Flash-First Architecture: Optimized for raw NAND/NOR flash with wear leveling awareness
Zero Dynamic Allocation: All buffers provided by caller - no runtime memory allocation
Platform Agnostic: Works with any flash HAL via callback interface
Data Integrity: CRC32C checksums on all metadata structures
Sovereign by Design: No external dependencies, no vendor lock-in, fully auditable

Use Cases

1. Libertaria Mesh Nodes

Primary Use Case: Storage layer for Libertaria Capsule nodes

┌─────────────────────────────────────────┐
│        Libertaria Capsule Node          │
│                                         │
│  ┌─────────────────────────────────┐   │
│  │   L3 Gossip (QVL Trust Edges)   │   │
│  └─────────────────────────────────┘   │
│  ┌─────────────────────────────────┐   │
│  │   L2 Session (Noise Handshakes) │   │
│  └─────────────────────────────────┘   │
│  ┌─────────────────────────────────┐   │
│  │   L1 Identity (SoulKeys)        │   │
│  └─────────────────────────────────┘   │
│  ┌─────────────────────────────────┐   │
│  │   NexFS (Persistent Storage)    │◄──┘
│  └─────────────────────────────────┘
│  ┌─────────────────────────────────┐
│  │   Raw Flash (NAND/NOR/SPI)      │
│  └─────────────────────────────────┘
└─────────────────────────────────────────┘

Why NexFS for Libertaria?

Persistence: SoulKeys, peer tables, trust graphs survive reboots
Integrity: CRC32C ensures metadata hasn't been corrupted
Wear Leveling: Tracks erase counts to maximize flash lifespan
Minimal Footprint: Zero allocation design fits embedded constraints
Fast Boot: No journal replay, direct mount from superblock

2. Embedded Sovereign Devices

Secondary Use Case: IoT devices, Raspberry Pi, ESP32, microcontrollers

Examples:

Solar Monitor Nodes: Store sensor readings, config, firmware updates
Weather Network: Log environmental data locally before sync
Pager Devices: Message queue persistence
Home Automation: Device state, automation rules, logs

Why NexFS for Embedded?

Raw Flash Support: Works directly with SPI flash, no FTL layer needed
Power-Loss Resilience: Dual superblock backup survives sudden power loss
Deterministic: Fixed buffer sizes, predictable memory usage
No OS Dependencies: Works bare-metal or with any RTOS

Architecture

On-Disk Layout

┌─────────────────────────────────────────────┐
│  Block 0: Primary Superblock (128 bytes)   │
├─────────────────────────────────────────────┤
│  Block 1: Backup Superblock (128 bytes)    │
├─────────────────────────────────────────────┤
│  Blocks 2-N: Block Allocation Map (BAM)    │
│  - Tracks allocation status                 │
│  - Records erase counts (wear leveling)    │
│  - Bad block marking                        │
├─────────────────────────────────────────────┤
│  Blocks N+1-N+4: Inode Table               │
│  - File/directory metadata                  │
│  - Inode IDs 1-128                          │
├─────────────────────────────────────────────┤
│  Blocks N+5+: Data Blocks                  │
│  - File/directory contents                  │
│  - Wear-leveled allocation                  │
└─────────────────────────────────────────────┘

Key Components

1. Superblock

Magic number: 0x4E455846 ("NEXF")
Generation counter for crash recovery
Mount count for health monitoring
CRC32C checksum for integrity

2. Block Allocation Map (BAM)

Per-block metadata: allocated, bad, reserved, needs_erase
Erase count tracking for wear leveling
Generation counter for block age

3. Inode Table

File/directory metadata
Supports: Regular, Directory, Symlink, Device nodes
Max filename: 255 characters

4. Flash Interface

pub const FlashInterface = struct {
    read: *const fn (ctx: *anyopaque, addr: u64, buffer: []u8) NexFSError!usize,
    write: *const fn (ctx: *anyopaque, addr: u64, buffer: []const u8) NexFSError!void,
    erase: *const fn (ctx: *anyopaque, block_addr: BlockAddr) NexFSError!void,
    sync: *const fn (ctx: *anyopaque) NexFSError!void,
};

Features

✅ Implemented (v0.1.0)

Format/Initialization: format() creates fresh filesystem
Superblock Management: Primary + backup with checksums
Block Allocation: BAM-based allocation with wear tracking
Inode Operations: Create, read, write, delete
Directory Operations: mkdir, rmdir, readdir, lookup
File Operations: open, read, write, close, seek
Path Resolution: Full path support (/path/to/file)
Checksum Verification: CRC32C on all metadata
Zero Allocation: All buffers provided by caller

🚧 Planned (Future Versions)

Wear Leveling Algorithm: Active block rotation based on erase counts
Bad Block Management: Automatic bad block detection and marking
Defragmentation: Reclaim fragmented data blocks
Snapshots: Point-in-time filesystem snapshots
Compression: Optional LZ4 compression for data blocks
Encryption: Optional XChaCha20-Poly1305 encryption

Quick Start

Installation

Add NexFS to your build.zig.zon:

.{
    .name = "your-project",
    .version = "0.1.0",
    .dependencies = .{
        .nexfs = .{
            .url = "https://git.sovereign-society.org/nexus/nexfs/archive/main.tar.gz",
            .hash = "...",
        },
    },
}

Example: Basic Usage

const std = @import("std");
const nexfs = @import("nexfs");

// 1. Define your flash interface
const MyFlash = struct {
    flash_data: []u8,

    pub fn read(ctx: *anyopaque, addr: u64, buffer: []u8) nexfs.NexFSError!usize {
        const self = @ptrCast(*MyFlash, @alignCast(ctx));
        @memcpy(buffer, self.flash_data[addr..][0..buffer.len]);
        return buffer.len;
    }

    pub fn write(ctx: *anyopaque, addr: u64, buffer: []const u8) nexfs.NexFSError!void {
        const self = @ptrCast(*MyFlash, @alignCast(ctx));
        @memcpy(self.flash_data[addr..][0..buffer.len], buffer);
    }

    pub fn erase(ctx: *anyopaque, block_addr: nexfs.BlockAddr) nexfs.NexFSError!void {
        // Erase flash block (set to 0xFF for NAND)
    }

    pub fn sync(ctx: *anyopaque) nexfs.NexFSError!void {
        // Flush any caches
    }
};

pub fn main() !void {
    var flash = MyFlash{ .flash_data = try allocator.alloc(u8, 1024 * 1024) };

    // 2. Configure NexFS
    var read_buf: [4096]u8 = undefined;
    var write_buf: [4096]u8 = undefined;
    var workspace: [256]u8 = undefined;

    const config = nexfs.Config{
        .flash = .{
            .ctx = &flash,
            .read = MyFlash.read,
            .write = MyFlash.write,
            .erase = MyFlash.erase,
            .sync = MyFlash.sync,
        },
        .device_size = 1024 * 1024,
        .block_size = 4096,
        .block_count = 256,
        .page_size = 256,
        .checksum_algo = .CRC32C,
        .read_buffer = &read_buf,
        .write_buffer = &write_buf,
        .workspace = &workspace,
        .time_source = null,
        .verbose = true,
    };

    // 3. Format the filesystem
    try nexfs.format(&config.flash, &config, &write_buf);

    // 4. Create a file
    var fs = try nexfs.NexFS.init(allocator, config);
    const fd = try fs.create("/config.txt");
    try fs.write(fd, "hello nexfs");
    try fs.close(fd);

    // 5. Read it back
    var buf: [64]u8 = undefined;
    const fd2 = try fs.open("/config.txt");
    const len = try fs.read(fd2, &buf);
    try std.io.getStdOut().writeAll(buf[0..len]);
    try fs.close(fd2);
}

Configuration Options

const Config = struct {
    flash: FlashInterface,      // Your flash HAL
    device_size: u64,            // Total flash size in bytes
    block_size: BlockSize,       // Flash block size (512, 1024, 2048, 4096)
    block_count: u32,            // Number of blocks
    page_size: PageSize,         // Flash page size for alignment
    checksum_algo: ChecksumAlgo, // None, CRC16, or CRC32C
    read_buffer: []u8,           // Buffer >= block_size
    write_buffer: []u8,          // Buffer >= block_size
    workspace: []u8,             // Buffer >= page_size
    time_source: ?TimeSource,    // Optional timestamp provider
    verbose: bool,               // Enable debug logging
};

Recommended Configurations

1. Raspberry Pi with SPI Flash (1MB)

.block_size = 4096,
.page_size = 256,
.block_count = 256,
.checksum_algo = .CRC32C,

2. ESP32 with Flash (4MB)

.block_size = 4096,
.page_size = 256,
.block_count = 1024,
.checksum_algo = .CRC32C,

3. Microcontroller with NOR Flash (512KB)

.block_size = 2048,
.page_size = 256,
.block_count = 256,
.checksum_algo = .CRC16,  // Faster on limited CPUs

Design Philosophy

Sovereign Storage Principles

No Secrets: All code is open source and auditable (LSL-1.0)
No Dependencies: Zero external libraries, pure Zig
No Vendor Lock-in: Standard interfaces, portable anywhere
No Hidden Allocation: Explicit memory management
No Trust Required: Verify integrity with checksums

Flash-Aware Design

Why Raw Flash?

Predictable Performance: No FTL latency spikes
Full Control: Wear leveling algorithm you control
Longer Lifespan: Avoid consumer-grade FTL write amplification
Lower Power: No background garbage collection

Wear Leveling Strategy:

Track erase counts per block (BAM)
Prefer blocks with lowest erase counts for writes
Reserve high-erase-count blocks for cold data
Target: Even wear distribution across flash lifetime

Performance Characteristics

Operation	Typical Latency	Notes
Mount	< 10ms	Read superblock, validate checksum
Format	100-500ms	Initialize all metadata blocks
File Create	5-20ms	Allocate inode, write metadata
File Read (4KB)	1-5ms	Single block read
File Write (4KB)	10-30ms	Erase + write cycle
Directory Lookup	1-5ms	Inode table scan

Memory Requirements:

Minimum: 2 × block_size + page_size (e.g., 8KB + 256B = ~8.5KB)
Recommended: 2 × block_size + 2 × page_size (for async ops)
Allocator: Not required (zero dynamic allocation)

Roadmap

Version 0.2.0 (Q2 2026)

Active wear leveling algorithm
Bad block management
Power-loss recovery improvements
Extended file attributes (xattr)

Version 0.3.0 (Q3 2026)

Compression support (LZ4)
Defragmentation tool
Filesystem check utility (fsck)
Performance benchmarks

Version 1.0.0 (Q4 2026)

Encryption support (XChaCha20-Poly1305)
Snapshot support
Production-hardened
Full Libertaria stack integration

Testing

Current Test Coverage: 251/253 tests passing (99.2%)

# Run tests
zig build test

# Run with verbose output
zig build test -Dverbose

Test Categories:

✅ Superblock validation
✅ Checksum verification
✅ Block allocation/deallocation
✅ Inode operations
✅ Directory operations
✅ File operations
✅ Path resolution
🔄 Wear leveling (in progress)
🔄 Bad block handling (planned)

Security Considerations

Data Integrity:

CRC32C protects all metadata from silent corruption
Dual superblock survives single-block corruption
Bad block marking prevents data loss

Power-Loss Resilience:

Primary + backup superblock
Metadata writes are atomic (single block)
No journal to replay

Future Security Features:

Optional encryption at rest (v1.0)
Authenticated encryption (AEAD)
Key derivation from SoulKey (Libertaria integration)

Contributing

Development Status: Alpha (v0.1.0)

Contribution Areas:

Wear leveling algorithm improvements
Bad block detection strategies
Performance optimizations
Test coverage improvements
Documentation enhancements

Code Style:

Follow Zig style guidelines
SPDX license headers required
BDD-style tests preferred
Panopticum architecture compliance

License

License: LSL-1.0 (Libertaria Source License 1.0)

Summary:

✅ Open source and auditable
✅ Free to use for sovereign applications
✅ Modifications must be contributed back
✅ No commercial restrictions for sovereign use cases

See LICENSE for full text.

Community

Repository: https://git.sovereign-society.org/nexus/nexfs

Organization: Nexus

rumpk - Runtime package manager
nip - Nexus package format
nexus - Core utilities
nipbox - Package repository
nexfs - Flash filesystem

Related Projects:

Libertaria Stack - P2P mesh networking
Janus Language - Systems programming language

Acknowledgments

Inspired By:

LittleFS - Flash-friendly embedded filesystem
JFFS2 - Journaling flash filesystem
YAFFS2 - Yet another flash filesystem

Built With:

Zig - Systems programming language
Libertaria - Sovereign P2P mesh network

NexFS - Storage for Sovereign Systems

Part of the Nexus ecosystem for Libertaria nodes and embedded devices

15 KiB Raw Blame History Unescape Escape