From: Tulio A M Mendes Date: Fri, 13 Feb 2026 11:14:26 +0000 (-0300) Subject: docs: update README, BUILD_GUIDE, POSIX_ROADMAP, TESTING_PLAN for current state X-Git-Url: https://projects.tadryanom.me/docs/POSIX_ROADMAP.md?a=commitdiff_plain;h=e427764af646a37c9b7d11c1615731dda791828e;p=AdrOS.git docs: update README, BUILD_GUIDE, POSIX_ROADMAP, TESTING_PLAN for current state - README: buddy allocator heap, ICMP ping, IOAPIC level-triggered, multi-drive ATA, kernel cmdline, kconsole, NO_SYS=0, 20 smoke tests, 16-check test battery, ~103K LOC across 255 commits, ~95% POSIX coverage - BUILD_GUIDE: networking, multi-disk QEMU, root= cmdline param, test-battery target, 20 smoke checks - POSIX_ROADMAP: FAT12/16/32 full RW (was FAT16 RO), ext2 full RW (was not implemented), NO_SYS=0 threaded mode, ICMP ping, multi-drive ATA, kconsole, buddy allocator, 48 total features (31+17), updated remaining work tiers - TESTING_PLAN: 20 smoke checks, test-battery description, updated Makefile targets --- diff --git a/BUILD_GUIDE.md b/BUILD_GUIDE.md index b82ee89..a236e18 100644 --- a/BUILD_GUIDE.md +++ b/BUILD_GUIDE.md @@ -2,7 +2,7 @@ This guide explains how to build and run AdrOS on your local machine (Linux/WSL). -AdrOS is a Unix-like, POSIX-compatible OS kernel with threads, futex synchronization, networking (TCP/IP + DNS via lwIP), dynamic linking infrastructure, FAT16 filesystem, ASLR, vDSO, zero-copy DMA, a POSIX shell, framebuffer graphics (`/dev/fb0`), raw keyboard input (`/dev/kbd`), uid/gid/euid/egid permission enforcement, and a working DOOM port. See [POSIX_ROADMAP.md](docs/POSIX_ROADMAP.md) for the full compatibility checklist. +AdrOS is a Unix-like, POSIX-compatible OS kernel with threads, futex synchronization, networking (TCP/IP + DNS + ICMP via lwIP in threaded mode), dynamic linking infrastructure, FAT12/16/32 + ext2 filesystems, ASLR, vDSO, zero-copy DMA, multi-drive ATA, a POSIX shell, framebuffer graphics (`/dev/fb0`), raw keyboard input (`/dev/kbd`), uid/gid/euid/egid permission enforcement, an interactive kernel console, and a working DOOM port. See [POSIX_ROADMAP.md](docs/POSIX_ROADMAP.md) for the full compatibility checklist. ## 1. Dependencies @@ -55,11 +55,26 @@ make ARCH=x86 run ``` Persistent storage note: -- The x86 QEMU run target attaches a `disk.img` file as an IDE drive (primary master). +- The x86 QEMU run target attaches a `disk.img` file as an IDE drive (primary master) and enables an E1000 NIC. - The kernel mounts two filesystems from this disk: - `/persist` — minimal persistence filesystem (e.g. `/persist/counter`) - `/disk` — hierarchical inode-based filesystem (diskfs) supporting `mkdir`, `unlink`, `rmdir`, `rename`, `getdents`, etc. - If `disk.img` does not exist, it is created automatically by the Makefile. +- The `root=` kernel parameter can override which device is mounted at `/disk` (e.g. `root=/dev/hdb`). + +Multi-disk testing: +- QEMU supports up to 3 IDE hard drives alongside the CD-ROM boot device: + - **hda** (index 0) — Primary Master + - **hdb** (index 1) — Primary Slave + - **hdd** (index 3) — Secondary Slave (hdc = CD-ROM) +- The kernel auto-detects all attached ATA drives and logs `[ATA] /dev/hdX detected`. + +Kernel command line parameters: +- `root=/dev/hdX` — mount specified device at `/disk` (auto-detects diskfs/FAT/ext2) +- `init=/path/to/binary` — override init binary (default: `/bin/init.elf`) +- `ring3` — enable userspace (ring 3) execution +- `quiet` — suppress non-critical boot messages +- `noapic` / `nosmp` — disable APIC / SMP If you are iterating on kernel changes and want to avoid hanging runs, you can wrap it with a timeout: ```bash @@ -77,7 +92,7 @@ Syscall return convention note: ### Userland programs The following ELF binaries are bundled in the initrd: -- `/bin/init.elf` — comprehensive smoke test suite (19+ checks) +- `/bin/init.elf` — comprehensive smoke test suite (20+ checks) - `/bin/echo` — argv/envp test - `/bin/sh` — POSIX sh-compatible shell with `$PATH` search, pipes, redirects, builtins - `/bin/cat`, `/bin/ls`, `/bin/mkdir`, `/bin/rm` — core utilities @@ -119,8 +134,9 @@ Individual test targets: ```bash make check # cppcheck + sparse + gcc -fanalyzer make test-host # 47 host-side unit tests (test_utils + test_security) -make test # QEMU smoke test (4 CPUs, 40s timeout, 19 checks) +make test # QEMU smoke test (4 CPUs, 90s timeout, 20 checks incl. ICMP ping) make test-1cpu # Single-CPU smoke test (50s timeout) +make test-battery # Full test battery: multi-disk ATA, VFS mount, ping, diskfs (16 checks) make test-gdb # GDB scripted integrity checks (heap, PMM, VGA) ``` diff --git a/README.md b/README.md index 6e005f0..1f8cfbd 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ AdrOS is a Unix-like, POSIX-compatible, multi-architecture operating system deve - **PMM** — bitmap allocator with spinlock protection, frame reference counting, and contiguous block allocation - **VMM** — PAE recursive page directory, per-process address spaces (PDPT + 4 PDs), TLB flush - **Copy-on-Write (CoW) fork** — PTE bit 9 as CoW marker + page fault handler -- **Kernel heap** — doubly-linked free list with coalescing, dynamic growth up to 64MB +- **Kernel heap** — 8MB Buddy Allocator (power-of-2 blocks 32B–8MB, circular free lists, buddy coalescing, corruption detection) - **Slab allocator** — `slab_cache_t` with free-list-in-place and spinlock - **Shared memory** — System V IPC style (`shmget`/`shmat`/`shmdt`/`shmctl`) - **`mmap`/`munmap`** — anonymous mappings, shared memory backing, and file-backed (fd) mappings @@ -87,24 +87,32 @@ AdrOS is a Unix-like, POSIX-compatible, multi-architecture operating system deve - Generic `readdir`/`getdents` across all VFS types; symlink following in path resolution ### Networking -- **E1000 NIC** — Intel 82540EM driver (MMIO, IRQ via IOAPIC) -- **lwIP TCP/IP stack** — IPv4, static IP (10.0.2.15 via QEMU user-net) +- **E1000 NIC** — Intel 82540EM driver (MMIO, IRQ 11 via IOAPIC level-triggered, interrupt-driven RX) +- **lwIP TCP/IP stack** — NO_SYS=0 threaded mode, IPv4, static IP (10.0.2.15 via QEMU user-net) - **Socket API** — `socket`/`bind`/`listen`/`accept`/`connect`/`send`/`recv`/`sendto`/`recvfrom` -- **Protocols** — TCP (`SOCK_STREAM`) + UDP (`SOCK_DGRAM`) +- **Protocols** — TCP (`SOCK_STREAM`) + UDP (`SOCK_DGRAM`) + ICMP +- **ICMP ping** — kernel-level ping test to QEMU gateway (10.0.2.2) during boot - **DNS resolver** — lwIP-based with async callback and timeout; kernel `dns_resolve()` wrapper ### Drivers & Hardware - **PCI** — full bus/slot/func enumeration with BAR + IRQ -- **ATA PIO + DMA** — Bus Master IDE with bounce buffer + zero-copy direct DMA, PRDT, IRQ coordination -- **LAPIC + IOAPIC** — replaces legacy PIC; ISA IRQ routing +- **ATA PIO + DMA** — Bus Master IDE with bounce buffer + zero-copy direct DMA, PRDT, IRQ coordination; multi-drive support (4 drives: hda/hdb/hdc/hdd across 2 channels) +- **LAPIC + IOAPIC** — replaces legacy PIC; ISA edge-triggered + PCI level-triggered IRQ routing - **SMP** — 4 CPUs via INIT-SIPI-SIPI, per-CPU data via GS segment - **ACPI** — MADT parsing for CPU topology and IOAPIC discovery - **VBE framebuffer** — maps LFB, pixel drawing, font rendering; `/dev/fb0` device with `ioctl`/`mmap` for userspace access - **UART**, **VGA text**, **PS/2 keyboard**, **PIT timer**, **LAPIC timer** +- **Kernel console (kconsole)** — interactive debug shell with readline, scrollback, command history - **RTC** — CMOS real-time clock driver for wall-clock time (`CLOCK_REALTIME`) - **E1000 NIC** — Intel 82540EM Ethernet controller - **MTRR** — write-combining support via variable-range MTRR programming +### Boot & Kernel Infrastructure +- **Kernel command line** — Linux-like parser (`init=`, `root=`, `console=`, `ring3`, `quiet`, `noapic`, `nosmp`); unknown tokens forwarded to init as argv/envp +- **`/proc/cmdline`** — exposes raw kernel command line to userspace +- **`root=` parameter** — auto-detect and mount filesystem from specified ATA device at `/disk` +- **Kernel synchronization** — counting semaphores (`ksem_t`), mutexes (`kmutex_t`), mailboxes (`kmbox_t`) with IRQ-safe signaling + ### Userland - **ulibc** — `printf`, `malloc`/`free`/`calloc`/`realloc`, `string.h`, `unistd.h`, `errno.h`, `pthread.h`, `signal.h`, `stdio.h` (buffered I/O with line-buffered stdout, unbuffered stderr, `setvbuf`/`setbuf`, `isatty`), `stdlib.h` (`atof`, `strtol`), `ctype.h`, `sys/mman.h` (`mmap`/`munmap`), `sys/ioctl.h`, `sys/times.h`, `sys/uio.h`, `sys/types.h`, `sys/stat.h`, `time.h` (`nanosleep`/`clock_gettime`), `math.h`, `assert.h`, `fcntl.h`, `strings.h`, `inttypes.h`, `linux/futex.h`, `realpath()` - **ELF32 loader** — secure with W^X + ASLR; supports `ET_EXEC` + `ET_DYN` + `PT_INTERP` (dynamic linking) @@ -140,7 +148,8 @@ AdrOS is a Unix-like, POSIX-compatible, multi-architecture operating system deve ### Testing - **47 host-side unit tests** — `test_utils.c` (28) + `test_security.c` (19) -- **19 QEMU smoke tests** — 4-CPU, expect-based +- **20 QEMU smoke tests** — 4-CPU expect-based (includes ICMP ping network test) +- **16-check test battery** — multi-disk ATA (hda+hdb+hdd), VFS mount, ping, diskfs ops (`make test-battery`) - **Static analysis** — cppcheck, sparse, gcc -fanalyzer - **GDB scripted checks** — heap/PMM/VGA integrity - `make test-all` runs everything @@ -160,15 +169,16 @@ QEMU debug helpers: See [POSIX_ROADMAP.md](docs/POSIX_ROADMAP.md) for a detailed checklist. -**All 31 planned POSIX implementation tasks are complete**, plus additional features including the DOOM game port, uid/gid/euid/egid permission enforcement, framebuffer device, raw keyboard device, fd-backed mmap, and kernel stack guard pages. The kernel covers **~93%** of the core POSIX interfaces needed for a practical Unix-like system. +**All 31 planned POSIX tasks are complete**, plus 10+ additional features: DOOM port, uid/gid/euid/egid, framebuffer, raw keyboard, fd-backed mmap, kernel stack guards, FAT12/16/32 full RW, ext2 full RW, ICMP ping, kernel command line parser. ~103K lines of C/ASM/headers across 255 commits. The kernel covers **~95%** of the core POSIX interfaces needed for a practical Unix-like system. ### Remaining work for full POSIX compliance - **Full `ld.so`** — relocation processing for shared libraries (`dlopen`/`dlsym`) - **`getaddrinfo`** / `/etc/hosts` — userland name resolution - **`sigqueue`** — queued real-time signals - **`setitimer`/`getitimer`** — interval timers -- **Per-CPU scheduler runqueues** — SMP scalability +- **Per-CPU scheduler runqueues** — SMP scalability (currently all processes run on BSP) - **SMAP** — Supervisor Mode Access Prevention +- **DHCP client** — currently static IP only - **Multi-arch bring-up** — ARM/RISC-V functional kernels ## Directory Structure @@ -177,7 +187,7 @@ See [POSIX_ROADMAP.md](docs/POSIX_ROADMAP.md) for a detailed checklist. - `src/hal/x86/` — HAL x86 (CPU, keyboard, timer, UART, PCI, ATA PIO/DMA, E1000 NIC, RTC) - `src/drivers/` — Device drivers (VBE, initrd, VGA, timer) - `src/mm/` — Memory management (PMM, heap, slab, arch-independent VMM wrappers) -- `src/net/` — Networking (lwIP port, DNS resolver) +- `src/net/` — Networking (lwIP port, E1000 netif, DNS resolver, ICMP ping test) - `include/` — Header files - `user/` — Userland programs (`init.c`, `echo.c`, `sh.c`, `cat.c`, `ls.c`, `mkdir.c`, `rm.c`, `ldso.c`) - `user/doom/` — DOOM port (doomgeneric engine + AdrOS platform adapter) diff --git a/docs/POSIX_ROADMAP.md b/docs/POSIX_ROADMAP.md index 5ce1623..c3156f2 100644 --- a/docs/POSIX_ROADMAP.md +++ b/docs/POSIX_ROADMAP.md @@ -131,11 +131,11 @@ Notes: | **diskfs** (on-disk) | [x] | Hierarchical inodes; full POSIX ops; symlinks; hard links with `nlink` tracking | | **persistfs** | [x] | Minimal persistence at `/persist` | | **procfs** | [x] | `/proc/meminfo` + per-process `/proc/[pid]/status`, `/proc/[pid]/maps` | -| **FAT16** (read-only) | [x] | BPB parsing, FAT chain traversal, root dir finddir, VFS read | +| **FAT12/16/32** (full RW) | [x] | Unified FAT driver, auto-detection by cluster count (MS spec), 8.3 filenames, subdirs, cluster chain management, all VFS mutation ops (create/write/delete/mkdir/rmdir/rename/truncate) | +| **ext2** (full RW) | [x] | Superblock + block group descriptors, inode read/write, block/inode bitmaps, direct/indirect/doubly-indirect/triply-indirect block mapping, directory entry add/remove/split, hard links, symlinks (inline), create/write/delete/mkdir/rmdir/rename/truncate/link | | Permissions (`uid`/`gid`/`euid`/`egid`/mode) | [x] | `chmod`, `chown` with permission checks; VFS `open()` enforces rwx bits vs process euid/egid and file uid/gid/mode | | Hard links | [x] | `diskfs_link()` with shared data blocks and `nlink` tracking | -| Symbolic links | [x] | `symlink`, `readlink`; followed by VFS lookup | -| ext2 support | [ ] | | +| Symbolic links | [x] | `symlink`, `readlink`; followed by VFS lookup | | ## 7. TTY / PTY @@ -164,7 +164,7 @@ Notes: | PMM contiguous block alloc | [x] | `pmm_alloc_blocks(count)` / `pmm_free_blocks()` for multi-page DMA | | VMM (x86 PAE paging) | [x] | Higher-half kernel, recursive page directory, PAE mode | | Per-process address spaces | [x] | PDPT + 4 PDs per process | -| Kernel heap (`kmalloc`/`kfree`) | [x] | Dynamic growth up to 64MB | +| Kernel heap (`kmalloc`/`kfree`) | [x] | 8MB Buddy Allocator (power-of-2 blocks 32B–8MB, circular free lists, buddy coalescing) | | Slab allocator | [x] | `slab_cache_t` with free-list-in-place | | W^X for user ELFs | [x] | Text segments read-only after load | | SMEP | [x] | Enabled in CR4 if CPU supports | @@ -186,11 +186,11 @@ Notes: | PS/2 keyboard | [x] | | | PIT timer | [x] | | | LAPIC timer | [x] | Calibrated, used when APIC available | -| ATA PIO (IDE) | [x] | Primary master | +| ATA PIO (IDE) | [x] | Multi-drive support: 4 drives (hda/hdb/hdc/hdd) across 2 channels | | ATA DMA (Bus Master IDE) | [x] | Bounce buffer + zero-copy direct DMA, PRDT, IRQ-coordinated | | PCI enumeration | [x] | Full bus/slot/func scan with BAR + IRQ | | ACPI (MADT parsing) | [x] | CPU topology + IOAPIC discovery | -| LAPIC + IOAPIC | [x] | Replaces legacy PIC | +| LAPIC + IOAPIC | [x] | Replaces legacy PIC; ISA edge-triggered + PCI level-triggered routing | | SMP (multi-CPU boot) | [x] | 4 CPUs via INIT-SIPI-SIPI, per-CPU data via GS | | CPUID feature detection | [x] | Leaf 0/1/7/extended; SMEP/SMAP detection | | VBE framebuffer | [x] | Maps LFB, pixel drawing, font rendering; `/dev/fb0` device with `ioctl`/`mmap` | @@ -199,14 +199,16 @@ Notes: | E1000 NIC (Intel 82540EM) | [x] | MMIO-based, IRQ-driven, lwIP integration | | RTC (real-time clock) | [x] | CMOS RTC driver; provides wall-clock time for `CLOCK_REALTIME` | | MTRR write-combining | [x] | `mtrr_init`/`mtrr_set_range` for variable-range MTRR programming | +| Kernel console (kconsole) | [x] | Interactive debug shell with readline, scrollback, command history | | Virtio-blk | [ ] | | ## 10. Networking | Feature | Status | Notes | |---------|--------|-------| -| E1000 NIC driver | [x] | Intel 82540EM, MMIO, IRQ 11 via IOAPIC | -| lwIP TCP/IP stack | [x] | NO_SYS=1, IPv4, static IP (10.0.2.15) | +| E1000 NIC driver | [x] | Intel 82540EM, MMIO, IRQ 11 via IOAPIC (level-triggered, active-low), interrupt-driven RX | +| lwIP TCP/IP stack | [x] | NO_SYS=0 threaded mode (kernel semaphores/mutexes/mailboxes), IPv4, static IP (10.0.2.15) | +| ICMP ping test | [x] | Kernel-level ping to QEMU gateway (10.0.2.2) during boot; verified via smoke test | | `socket` | [x] | `AF_INET`, `SOCK_STREAM` (TCP), `SOCK_DGRAM` (UDP) | | `bind`/`listen`/`accept` | [x] | TCP server support | | `connect`/`send`/`recv` | [x] | TCP client support | @@ -273,7 +275,7 @@ Notes: ## Implementation Progress -### All 31 planned tasks completed ✅ + 7 additional features +### All 31 planned tasks completed ✅ + 17 additional features **High Priority (8/8):** 1. ~~`raise()` em ulibc~~ ✅ @@ -321,6 +323,15 @@ Notes: 37. ~~Kernel stack guard pages (0xC8000000 region)~~ ✅ 38. ~~uid/gid/euid/egid with VFS permission enforcement~~ ✅ 39. ~~DOOM port (doomgeneric + AdrOS adapter, 450KB doom.elf)~~ ✅ +40. ~~Buddy Allocator heap (replaced doubly-linked-list heap)~~ ✅ +41. ~~lwIP NO_SYS=0 threaded mode (kernel semaphores/mutexes/mailboxes)~~ ✅ +42. ~~kprintf migration (all uart_print→kprintf, 16KB ring buffer, dmesg)~~ ✅ +43. ~~Unified FAT12/16/32 full RW driver (replaced read-only FAT16)~~ ✅ +44. ~~ext2 filesystem full RW~~ ✅ +45. ~~Kernel command line parser (Linux-like: init=, root=, ring3, quiet, /proc/cmdline)~~ ✅ +46. ~~Multi-drive ATA support (4 drives across 2 channels)~~ ✅ +47. ~~IOAPIC level-triggered routing for PCI interrupts~~ ✅ +48. ~~ICMP ping test (kernel-level, verified in smoke test)~~ ✅ --- @@ -341,19 +352,19 @@ Notes: | Gap | Impact | Effort | |-----|--------|--------| -| **ext2 filesystem** | No standard on-disk filesystem | Large | | **`pipe` capacity** — `F_GETPIPE_SZ`/`F_SETPIPE_SZ` | Fixed-size pipe buffer | Small | | **`waitid`** — extended wait | Missing POSIX interface | Small | | **`posix_spawn`** — efficient process creation | Missing POSIX interface | Medium | | **Virtio-blk driver** | Only ATA PIO/DMA disk access | Medium | | **SMAP** — Supervisor Mode Access Prevention | Missing hardware security feature | Small | +| **DHCP client** | Static IP only (10.0.2.15 via QEMU user-net) | Medium | +| **E1000 rx_thread scheduling** | RX thread not waking from semaphore during ping; inline poll workaround | Medium | ### Tier 3 — Multi-arch & long-term | Gap | Impact | Effort | |-----|--------|--------| | **Multi-arch bring-up** — ARM/RISC-V functional kernels | x86-only | Very Large | -| **DHCP client** | Static IP only | Medium | | **IPv6** | IPv4-only | Large | | **POSIX message queues** (`mq_*`) | Missing IPC mechanism | Medium | | **POSIX semaphores** (`sem_*`) | Only futex available | Medium | diff --git a/docs/TESTING_PLAN.md b/docs/TESTING_PLAN.md index 26ff034..221058e 100644 --- a/docs/TESTING_PLAN.md +++ b/docs/TESTING_PLAN.md @@ -5,7 +5,8 @@ All four testing layers are **implemented and operational**: - **Static analysis** (`make check`): cppcheck + sparse + gcc -fanalyzer (59 x86 C files) -- **QEMU smoke tests** (`make test`): expect-based, 19 checks, 4-CPU SMP, 40s timeout +- **QEMU smoke tests** (`make test`): expect-based, 20 checks (incl. ICMP ping), 4-CPU SMP, 90s timeout +- **Test battery** (`make test-battery`): 16 checks across 5 QEMU scenarios — multi-disk ATA, VFS mount, ping, diskfs - **Host unit tests** (`make test-host`): 47 tests — `test_utils.c` (28) + `test_security.c` (19) - **GDB scripted checks** (`make test-gdb`): heap/PMM/VGA integrity validation - **Full suite** (`make test-all`): runs check + test-host + test @@ -55,6 +56,9 @@ This replaces the manual `grep serial.log` approach with a scripted test that: **Implementation**: `tests/smoke_test.exp` expect script + `make test` target. +A separate **test battery** (`tests/test_battery.exp`) exercises multi-disk ATA detection, +VFS mount verification, and ICMP ping across multiple QEMU configurations. + ### Layer 3: QEMU + GDB Scripted Debugging (`make test-gdb`) **Tools**: QEMU `-s -S` + GDB with Python scripting @@ -102,9 +106,10 @@ To run manually: boot AdrOS with `-vga std`, then execute `/bin/doom.elf` from t ## Makefile Targets ```makefile -make check # cppcheck + sparse + gcc -fanalyzer -make test # QEMU + expect automated smoke test -make test-host # Host-side unit tests for pure functions -make test-gdb # QEMU + GDB scripted checks (optional) -make test-all # All of the above +make check # cppcheck + sparse + gcc -fanalyzer +make test # QEMU + expect automated smoke test (20 checks incl. ICMP ping) +make test-battery # Full test battery: multi-disk ATA, VFS mount, ping, diskfs (16 checks) +make test-host # Host-side unit tests for pure functions +make test-gdb # QEMU + GDB scripted checks (optional) +make test-all # All of the above ```