From: Tulio A M Mendes Date: Sat, 14 Mar 2026 11:59:58 +0000 (-0300) Subject: docs: audit & update all documentation to reflect current implementation X-Git-Url: https://projects.tadryanom.me/docs/static/gitweb.css?a=commitdiff_plain;h=cf7f105a13f723c296b4cec204da54aa3f16ac96;p=AdrOS.git docs: audit & update all documentation to reflect current implementation - POSIX_ROADMAP: 75→90 features (15 new: gettimeofday, mprotect, getrlimit/setrlimit, uname, getrusage, socket ops, LZ4, EPOLLET, lazy PLT, MIPS, SysV init, 50+ utilities, host test harness, native toolchain, mount syscall) - README: 80→102 smoke tests, 47→115 host tests, 75→90 features, 52 userland programs listed - BUILD_GUIDE: fixed init path (/sbin/fulltest), expanded utility list, updated test counts - TESTING_PLAN: 80→102 smoke, 19→115 host tests - SUPPLEMENTARY_ANALYSIS: LZ4 ❌→✅, per-CPU runqueues ❌→✅, test counts aligned - AUDIT_REPORT: 3 additional fixes marked (sc_* macros, SMAP, schedule race) - FULL_POSIX_AUDIT: 7/12 syscalls now ✅, Newlib/GCC/Binutils ports ✅ COMPLETE, kernel ~85%→~98% POSIX --- diff --git a/BUILD_GUIDE.md b/BUILD_GUIDE.md index c97d139..f613445 100644 --- a/BUILD_GUIDE.md +++ b/BUILD_GUIDE.md @@ -92,17 +92,32 @@ Syscall return convention note: ### Userland programs The following ELF binaries are bundled in the initrd: -- `/bin/init.elf` — comprehensive smoke test suite (44 checks) -- `/bin/echo` — argv/envp test +- `/sbin/fulltest` — comprehensive smoke test suite (102 checks) +- `/sbin/init` — SysV-like init process (inittab, runlevels, respawn) - `/bin/sh` — POSIX sh-compatible shell with `$PATH` search, pipes, redirects, builtins -- `/bin/cat`, `/bin/ls`, `/bin/mkdir`, `/bin/rm` — core utilities +- `/bin/echo`, `/bin/cat`, `/bin/ls`, `/bin/mkdir`, `/bin/rm` — core utilities +- `/bin/cp`, `/bin/mv`, `/bin/touch`, `/bin/ln` — file operations +- `/bin/head`, `/bin/tail`, `/bin/wc`, `/bin/sort`, `/bin/uniq`, `/bin/cut` — text processing +- `/bin/grep`, `/bin/sed`, `/bin/awk`, `/bin/tr` — pattern matching and text transformation +- `/bin/find`, `/bin/which` — file search and command lookup +- `/bin/chmod`, `/bin/chown`, `/bin/chgrp` — permission management +- `/bin/mount`, `/bin/umount` — filesystem mount/unmount +- `/bin/ps`, `/bin/top`, `/bin/kill` — process management +- `/bin/df`, `/bin/du`, `/bin/free` — disk and memory usage +- `/bin/date`, `/bin/hostname`, `/bin/uptime`, `/bin/uname` — system information +- `/bin/env`, `/bin/printenv`, `/bin/id` — environment and identity +- `/bin/tee`, `/bin/dd`, `/bin/pwd`, `/bin/stat` — I/O and file info +- `/bin/basename`, `/bin/dirname`, `/bin/sleep`, `/bin/clear`, `/bin/rmdir`, `/bin/dmesg`, `/bin/who` — misc utilities +- `/bin/pie_test` — PIE/shared library test binary - `/bin/doom.elf` — DOOM (doomgeneric port) — included in initrd if built (see below) -- `/lib/ld.so` — dynamic linker with auxv parsing, PLT/GOT eager relocation +- `/lib/ld.so` — dynamic linker with auxv parsing, PLT/GOT lazy relocation +- `/lib/libc.so` — shared C library (ulibc) +- `/lib/libpietest.so` — test shared library The ulibc provides: `printf`, `malloc`/`free`/`calloc`/`realloc`, `string.h`, `unistd.h`, `errno.h`, `pthread.h`, `signal.h` (with `raise`, `sigaltstack`, `sigpending`, `sigsuspend`), `stdio.h` (buffered I/O with `fopen`/`fread`/`fwrite`/`fclose`), `stdlib.h` (`atof`, `strtol`, `getenv` stub, `system` stub), `ctype.h`, `sys/mman.h` (`mmap`/`munmap`), `sys/ioctl.h` (`ioctl`), `time.h` (`nanosleep`/`clock_gettime`), `sys/times.h`, `sys/uio.h`, `sys/types.h`, `sys/stat.h`, `math.h` (`fabs`), `assert.h`, `fcntl.h`, `strings.h`, `inttypes.h`, `linux/futex.h`, and `realpath()`. ### Smoke tests -The init program (`/bin/init.elf`) runs a comprehensive suite of smoke tests on boot, covering: +The fulltest binary (`/sbin/fulltest`) runs a comprehensive suite of 102 smoke tests on boot, covering: - File I/O (`open`, `read`, `write`, `close`, `lseek`, `stat`, `fstat`) - Overlay copy-up, `dup2`, `pipe`, `select`, `poll` - TTY/ioctl, job control (`SIGTTIN`/`SIGTTOU`) @@ -122,7 +137,10 @@ The init program (`/bin/init.elf`) runs a comprehensive suite of smoke tests on - `/dev/tty` write test - Memory: `brk`, `mmap`/`munmap`, `clock_gettime`, shared memory (`shmget`/`shmat`/`shmdt`) - Advanced: `pread`/`pwrite`, `ftruncate`, `symlink`/`readlink`, `access`, `sigprocmask`/`sigpending`, `alarm`/`SIGALRM`, `O_APPEND`, `umask`, pipe capacity (`F_GETPIPE_SZ`/`F_SETPIPE_SZ`), `waitid`, `setitimer`/`getitimer`, `select`/`poll` on regular files, hard links -- New: `epoll` (create/ctl/wait on pipe), `inotify` (init/add_watch/rm_watch), `aio_*` (read/write/error/return) +- Advanced I/O: `epoll` (create/ctl/wait on pipe), `epollet` (edge-triggered), `inotify` (init/add_watch/rm_watch), `aio_*` (read/write/error/return) +- System: `gettimeofday`, `mprotect`, `getrlimit`/`setrlimit`, `uname` +- Dynamic linking: lazy PLT resolution, PLT caching +- LZ4 initrd decompression All tests print `[init] ... OK` on success. Any failure calls `sys_exit(1)`. @@ -136,8 +154,8 @@ make test-all Individual test targets: ```bash make check # cppcheck + sparse + gcc -fanalyzer -make test-host # 47 host-side unit tests (test_utils + test_security) -make test # QEMU smoke test (4 CPUs, 30s timeout, 44 checks incl. ICMP ping, epoll, inotify, aio) +make test-host # 115 host-side tests (test_utils + test_security + test_host_utils.sh) +make test # QEMU smoke test (4 CPUs, 120s timeout, 102 checks incl. ICMP ping, epoll, epollet, inotify, aio, LZ4, lazy PLT) make test-1cpu # Single-CPU smoke test (50s timeout) make test-battery # Full test battery: multi-disk ATA, VFS mount, ping, diskfs (16 checks) make test-gdb # GDB scripted integrity checks (heap, PMM, VGA) diff --git a/README.md b/README.md index 50c0f52..081f0c4 100644 --- a/README.md +++ b/README.md @@ -54,10 +54,10 @@ AdrOS is a Unix-like, POSIX-compatible, multi-architecture operating system deve - **Time** — `nanosleep`, `clock_gettime` (`CLOCK_REALTIME` via RTC, `CLOCK_MONOTONIC` with TSC nanosecond precision), `alarm`/`SIGALRM`, `times` (CPU accounting) ### Syscalls (x86, `int 0x80` + SYSENTER) -- **File I/O:** `open`, `openat`, `read`, `write`, `close`, `lseek`, `stat`, `fstat`, `fstatat`, `dup`, `dup2`, `dup3`, `pipe`, `pipe2`, `select`, `poll`, `ioctl`, `fcntl`, `getdents`, `pread`, `pwrite`, `readv`, `writev`, `truncate`, `ftruncate`, `fsync`, `fdatasync` +- **File I/O:** `open`, `openat`, `read`, `write`, `close`, `lseek`, `stat`, `fstat`, `fstatat`, `dup`, `dup2`, `dup3`, `pipe`, `pipe2`, `select`, `poll`, `ioctl`, `fcntl`, `getdents`, `pread`, `pwrite`, `readv`, `writev`, `truncate`, `ftruncate`, `fsync`, `fdatasync`, `flock` - **Directory ops:** `mkdir`, `rmdir`, `unlink`, `unlinkat`, `rename`, `chdir`, `getcwd`, `link`, `symlink`, `readlink`, `chmod`, `chown`, `access`, `umask` - **Signals:** `sigaction` (`SA_SIGINFO`), `sigprocmask`, `kill`, `sigreturn` (full trampoline), `sigpending`, `sigsuspend`, `sigaltstack`, `sigqueue` -- **Process:** `setuid`, `setgid`, `seteuid`, `setegid`, `getuid`, `getgid`, `geteuid`, `getegid`, `alarm`, `times`, `futex`, `waitid`, `posix_spawn`, `setitimer`, `getitimer`, `pivot_root` +- **Process:** `setuid`, `setgid`, `seteuid`, `setegid`, `getuid`, `getgid`, `geteuid`, `getegid`, `alarm`, `times`, `futex`, `waitid`, `posix_spawn`, `setitimer`, `getitimer`, `pivot_root`, `gettimeofday`, `mprotect`, `getrlimit`, `setrlimit`, `uname`, `getrusage`, `mount` - **IPC:** `mq_open`, `mq_close`, `mq_unlink`, `mq_send`, `mq_receive`, `mq_getattr`, `mq_setattr`, `sem_open`, `sem_close`, `sem_unlink`, `sem_wait`, `sem_post`, `sem_getvalue` - **I/O multiplexing (advanced):** `epoll_create`, `epoll_ctl`, `epoll_wait`, `inotify_init`, `inotify_add_watch`, `inotify_rm_watch` - **Async I/O:** `aio_read`, `aio_write`, `aio_error`, `aio_return`, `aio_suspend` @@ -65,7 +65,7 @@ AdrOS is a Unix-like, POSIX-compatible, multi-architecture operating system deve - **File locking:** `flock` (advisory locking with per-inode lock table) - **Shared memory:** `shmget`, `shmat`, `shmdt`, `shmctl` - **Threads:** `clone`, `gettid`, `set_thread_area`, `futex` -- **Networking:** `socket`, `bind`, `listen`, `accept`, `connect`, `send`, `recv`, `sendto`, `recvfrom`, `sendmsg`, `recvmsg` +- **Networking:** `socket`, `bind`, `listen`, `accept`, `connect`, `send`, `recv`, `sendto`, `recvfrom`, `sendmsg`, `recvmsg`, `setsockopt`, `getsockopt`, `shutdown`, `getpeername`, `getsockname` - Per-process fd table with atomic refcounted file objects - Centralized user-pointer access API (`user_range_ok`, `copy_from_user`, `copy_to_user`) - Error returns use negative errno codes (Linux-style) @@ -132,10 +132,11 @@ AdrOS is a Unix-like, POSIX-compatible, multi-architecture operating system deve - **ulibc** — `printf`, `malloc`/`free`/`calloc`/`realloc`, `string.h`, `unistd.h`, `errno.h`, `pthread.h`, `signal.h`, `stdio.h` (buffered I/O with line-buffered stdout, unbuffered stderr, `setvbuf`/`setbuf`, `isatty`), `stdlib.h` (`atof`, `strtol`), `ctype.h`, `sys/mman.h` (`mmap`/`munmap`), `sys/ioctl.h`, `sys/times.h`, `sys/uio.h`, `sys/types.h`, `sys/stat.h`, `time.h` (`nanosleep`/`clock_gettime`), `math.h`, `assert.h`, `fcntl.h`, `strings.h`, `inttypes.h`, `linux/futex.h`, `realpath()` - **ELF32 loader** — secure with W^X + ASLR; supports `ET_EXEC` + `ET_DYN` + `PT_INTERP` (dynamic linking) - **Shell** — `/bin/sh` (POSIX sh-compatible with builtins, pipes, redirects, `$PATH` search) -- **Core utilities** — `/bin/cat`, `/bin/ls`, `/bin/mkdir`, `/bin/rm`, `/bin/echo` -- `/bin/init.elf` — comprehensive smoke test suite +- **52 userland programs** — `/bin/cat`, `/bin/ls`, `/bin/mkdir`, `/bin/rm`, `/bin/echo`, `/bin/cp`, `/bin/mv`, `/bin/touch`, `/bin/ln`, `/bin/head`, `/bin/tail`, `/bin/wc`, `/bin/sort`, `/bin/uniq`, `/bin/cut`, `/bin/grep`, `/bin/sed`, `/bin/awk`, `/bin/find`, `/bin/which`, `/bin/chmod`, `/bin/chown`, `/bin/chgrp`, `/bin/mount`, `/bin/umount`, `/bin/ps`, `/bin/top`, `/bin/kill`, `/bin/df`, `/bin/du`, `/bin/free`, `/bin/date`, `/bin/hostname`, `/bin/uptime`, `/bin/uname`, `/bin/env`, `/bin/printenv`, `/bin/id`, `/bin/tee`, `/bin/dd`, `/bin/tr`, `/bin/basename`, `/bin/dirname`, `/bin/pwd`, `/bin/stat`, `/bin/sleep`, `/bin/clear`, `/bin/rmdir`, `/bin/dmesg`, `/bin/who`, `/bin/pie_test` +- `/sbin/init` — SysV-like init process (inittab, runlevels, respawn) +- `/sbin/fulltest` — comprehensive smoke test suite (102 checks) - `/bin/doom.elf` — DOOM (doomgeneric port) — runs on `/dev/fb0` + `/dev/kbd` -- `/lib/ld.so` — dynamic linker with auxv parsing, PLT/GOT eager relocation +- `/lib/ld.so` — dynamic linker with auxv parsing, PLT/GOT lazy relocation ### Dynamic Linking - **Full `ld.so`** — kernel-side relocation processing for `R_386_RELATIVE`, `R_386_32`, `R_386_GLOB_DAT`, `R_386_JMP_SLOT`, `R_386_COPY`, `R_386_PC32` @@ -162,8 +163,8 @@ AdrOS is a Unix-like, POSIX-compatible, multi-architecture operating system deve - **PMM spinlock** for SMP safety ### Testing -- **47 host-side unit tests** — `test_utils.c` (28) + `test_security.c` (19) -- **80 QEMU smoke tests** — 4-CPU expect-based (file I/O, signals, memory mgmt, IPC, devices, procfs, networking, epoll, inotify, aio, nanosleep, CoW fork, readv/writev, fsync, flock, posix_spawn, TSC precision, execve) +- **115 host-side tests** — `test_utils.c` (28) + `test_security.c` (19) + `test_host_utils.sh` (68 cross-compiled utility tests) +- **102 QEMU smoke tests** — 4-CPU expect-based (file I/O, signals, memory mgmt, IPC, devices, procfs, networking, epoll, epollet, inotify, aio, nanosleep, CoW fork, readv/writev, fsync, flock, posix_spawn, TSC precision, gettimeofday, mprotect, getrlimit/setrlimit, uname, LZ4, lazy PLT, execve) - **16-check test battery** — multi-disk ATA (hda+hdb+hdd), VFS mount, ping, diskfs ops (`make test-battery`) - **Static analysis** — cppcheck, sparse, gcc -fanalyzer - **GDB scripted checks** — heap/PMM/VGA integrity @@ -203,7 +204,7 @@ QEMU debug helpers: See [POSIX_ROADMAP.md](docs/POSIX_ROADMAP.md) for a detailed checklist. -**All 31 planned POSIX tasks are complete**, plus 44 additional features (75 total). The kernel covers **~98%** of the core POSIX interfaces needed for a practical Unix-like system. All 80 smoke tests, 16 battery checks, and 47 host unit tests pass clean. ARM64, RISC-V 64, and MIPS32 boot on QEMU. +**All 31 planned POSIX tasks are complete**, plus 59 additional features (90 total). The kernel covers **~98%** of the core POSIX interfaces needed for a practical Unix-like system. All 102 smoke tests, 16 battery checks, and 115 host tests pass clean. ARM64, RISC-V 64, and MIPS32 boot on QEMU. Rump Kernel integration is in progress — prerequisites (condition variables, TSC nanosecond clock, IRQ chaining) are implemented and the `rumpuser` hypercall scaffold is in place. @@ -219,7 +220,7 @@ Rump Kernel integration is in progress — prerequisites (condition variables, T - `src/net/` — Networking (lwIP port, E1000 netif, DNS resolver, ICMP ping test) - `src/rump/` — Rump Kernel hypercall scaffold (`rumpuser_adros.c`) - `include/` — Header files -- `user/` — Userland programs (`init.c`, `echo.c`, `sh.c`, `cat.c`, `ls.c`, `mkdir.c`, `rm.c`, `ldso.c`) +- `user/` — Userland programs (52 commands: `init.c`, `sh.c`, `cat.c`, `ls.c`, `echo.c`, `cp.c`, `mv.c`, `grep.c`, `sed.c`, `awk.c`, `find.c`, `which.c`, `ps.c`, `top.c`, `kill.c`, `mount.c`, etc. + `ldso.c`, `fulltest.c`, `pie_main.c`) - `user/doom/` — DOOM port (doomgeneric engine + AdrOS platform adapter) - `user/ulibc/` — Minimal C library (`printf`, `malloc`, `string.h`, `errno.h`, `pthread.h`, `signal.h`, `stdio.h`, `stdlib.h`, `ctype.h`, `math.h`, `sys/mman.h`, `sys/ioctl.h`, `sys/uio.h`, `time.h`, `linux/futex.h`) - `tests/` — Host unit tests, smoke tests, GDB scripted checks diff --git a/docs/AUDIT_REPORT.md b/docs/AUDIT_REPORT.md index 4b22904..a0fc4e6 100644 --- a/docs/AUDIT_REPORT.md +++ b/docs/AUDIT_REPORT.md @@ -296,15 +296,15 @@ If `name` exceeds 128 bytes (the size of `fs_node.name`), this overflows. | 3.2 | CRITICAL | Security | syscall.c | sigreturn allows IOPL escalation | **FIXED** | | 2.1 | CRITICAL | Race | pmm.c | No locking on PMM bitmap | **FIXED** | | 2.2 | CRITICAL | Race | syscall.c | file refcount not atomic | **FIXED** | -| 1.1 | CRITICAL | Layer | syscall.c | x86 registers in generic code | Open | +| 1.1 | CRITICAL | Layer | syscall.c | x86 registers in generic code | **FIXED** (sc_* macros) | | 2.3 | HIGH | Memory | slab.c | phys_to_virt can hit heap VA | **FIXED** | | 3.3 | HIGH | Security | syscall.c | execve bypasses copy_to_user | **FIXED** | -| 3.4 | HIGH | Security | - | No SMEP/SMAP | **SMEP FIXED** | +| 3.4 | HIGH | Security | - | No SMEP/SMAP | **FIXED** (SMEP+SMAP) | | 4.1 | HIGH | Memory | heap.c | Heap never grows | **FIXED** | | 2.4 | HIGH | Logic | scheduler.c | waitpid NULL deref risk | **FIXED** | | 1.2 | MODERATE | Layer | heap.c | Hardcoded heap VA | Open | | 1.3 | MODERATE | Layer | interrupts.h | x86 registers leak | Open | -| 2.5 | MODERATE | Race | scheduler.c | Unlock before context_switch | Open | +| 2.5 | MODERATE | Race | scheduler.c | Unlock before context_switch | **FIXED** | | 2.6 | MODERATE | Logic | utils.c | itoa no buffer size | Open | | 2.7 | MODERATE | Logic | utils.c | itoa UB for INT_MIN | Open | | 3.5 | MODERATE | Security | syscall.c | fd bounds not always checked | Open | @@ -321,5 +321,10 @@ SMEP enabled via CR4, heap grows dynamically to 64MB, waitpid NULL guard. **2 MODERATE fixed**: User stack guard pages (unmapped page below 32KB stack) and kernel stack guard pages (`kstack_alloc()` at `0xC8000000` with unmapped guard page per stack). -**Remaining**: 1 CRITICAL (layer violation — arch refactor), 6 MODERATE (open). -SMAP not yet enabled (SMEP is active). +**1 additional CRITICAL fixed**: syscall.c arch refactor — all x86 register accesses replaced with `sc_*` macros (`sc_num`, `sc_arg0`–`sc_arg4`, `sc_ret`, `sc_ip`, `sc_usp`). To port to ARM: only need `arch/arm/arch_syscall.h`. + +**1 additional HIGH fixed**: SMAP enabled in CR4 bit 21 (in addition to SMEP). + +**1 additional MODERATE fixed**: `schedule()` now calls `context_switch` BEFORE `spin_unlock_irqrestore`, preventing the timer-fired race window. + +**Remaining**: 4 MODERATE (open): hardcoded heap VA (1.2), x86 registers leak in interrupts.h (1.3), itoa no buffer size (2.6), itoa UB for INT_MIN (2.7), fd bounds (3.5), kfree doesn't zero (4.2). diff --git a/docs/FULL_POSIX_AUDIT.md b/docs/FULL_POSIX_AUDIT.md index fa0e517..918bccc 100644 --- a/docs/FULL_POSIX_AUDIT.md +++ b/docs/FULL_POSIX_AUDIT.md @@ -1,72 +1,53 @@ # AdrOS — Full POSIX/Unix Compatibility Audit & Porting Analysis -**Date:** 2026-03-13 -**Commit:** 2deaf85 (master) +**Date:** 2026-03-14 (updated) +**Original commit:** 2deaf85 — **Current state reflects latest master** --- ## Part 1: Build System — `git clone` Breakage Analysis -### CRITICAL: Third-Party Dependencies Are NOT Tracked +### Third-Party Dependencies — **RESOLVED ✅** -After `git clone https://github.com/.../AdrOS.git`, the following directories will be **EMPTY**: +lwIP is tracked as a git submodule (`.gitmodules` exists). DOOM is optional and documented. | Directory | Source | Status | |---|---|---| -| `third_party/lwip/` | https://github.com/lwip-tcpip/lwip.git | Nested git repo, NOT a submodule | -| `user/doom/doomgeneric/` | https://github.com/ozkl/doomgeneric.git | Nested git repo, NOT a submodule | +| `third_party/lwip/` | https://github.com/lwip-tcpip/lwip.git | ✅ Git submodule | +| `user/doom/doomgeneric/` | https://github.com/ozkl/doomgeneric.git | Optional, documented in BUILD_GUIDE.md | -**Result:** `make` fails immediately — lwIP sources referenced in `LWIP_SOURCES` don't exist. -`make iso` also fails because DOOM (optional) and lwIP (required) are missing. - -**No `.gitmodules` file exists.** No `.gitignore` exists at root level. - -### Recommended Fix (Priority: CRITICAL) - -**Option A — Git Submodules (recommended):** -```bash -git submodule add https://github.com/lwip-tcpip/lwip.git third_party/lwip -git submodule add https://github.com/ozkl/doomgeneric.git user/doom/doomgeneric -``` -Then users do `git clone --recursive` or `git submodule update --init`. - -**Option B — Setup script + documentation:** -Add a `scripts/setup-deps.sh`: -```bash -#!/bin/bash -git clone https://github.com/lwip-tcpip/lwip.git third_party/lwip -git clone https://github.com/ozkl/doomgeneric.git user/doom/doomgeneric -``` +**`.gitmodules`** and **`.gitignore`** both exist at root level. ### Other Build Issues -| Issue | Severity | Details | +| Issue | Severity | Status | |---|---|---| -| No `.gitignore` | Medium | `*.o`, `*.elf`, `*.iso`, `build/`, `disk.img`, `serial.log` etc. are committed or litter the workspace | -| `tools/mkinitrd` uses host `gcc` | Low | Correct behavior (host tool), but should be explicit: `HOST_CC ?= gcc` | -| `user/ulibc/src/*.o` tracked by git | Medium | Build artifacts in the repo — need `.gitignore` | -| Missing `README.md` build instructions | High | No documentation on prerequisites (cross-compiler, grub-mkrescue, expect, qemu, cppcheck) | +| No `.gitignore` | Medium | **FIXED** — `.gitignore` exists | +| `tools/mkinitrd` uses host `gcc` | Low | Correct behavior (host tool) | +| Missing `README.md` build instructions | High | **FIXED** — comprehensive `BUILD_GUIDE.md` exists | --- ## Part 2: POSIX/Unix Compatibility Gap Analysis -### 2A. Missing Syscalls (kernel does NOT implement) +### 2A. Previously Missing Syscalls — Status Update -| Syscall | POSIX | Needed By | Priority | -|---|---|---|---| -| `mprotect` | Required | Newlib, GCC runtime, any JIT | **Critical** | -| `getrlimit` / `setrlimit` | Required | Bash, GCC, Busybox | **Critical** | -| `gettimeofday` | Required | Many programs (fallback for clock_gettime) | **High** | -| `getrusage` | Required | Bash (time builtin), make | High | -| `setsockopt` / `getsockopt` | Required | Any network program | High | -| `shutdown` (socket) | Required | Network programs | Medium | -| `getpeername` / `getsockname` | Required | Network programs | Medium | -| `madvise` | Optional | GCC, large programs | Low | -| `mremap` | Linux ext | realloc with mmap | Low | -| `execveat` | Linux ext | Nice to have | Low | -| `umount2` | Required | Full mount/umount | Medium | -| `ioctl FIONREAD` | Required | Many programs, Bash | High | +| Syscall | POSIX | Status | +|---|---|---| +| `mprotect` | Required | ✅ **IMPLEMENTED** (syscall 128) | +| `getrlimit` / `setrlimit` | Required | ✅ **IMPLEMENTED** (syscalls 129/130) | +| `gettimeofday` | Required | ✅ **IMPLEMENTED** (syscall 127) | +| `getrusage` | Required | ✅ **IMPLEMENTED** (syscall 137) | +| `setsockopt` / `getsockopt` | Required | ✅ **IMPLEMENTED** (syscalls 131/132) | +| `shutdown` (socket) | Required | ✅ **IMPLEMENTED** (syscall 133) | +| `getpeername` / `getsockname` | Required | ✅ **IMPLEMENTED** (syscalls 134/135) | +| `madvise` | Optional | ❌ Not implemented (low priority) | +| `mremap` | Linux ext | ❌ Not implemented (low priority) | +| `execveat` | Linux ext | ❌ Not implemented (low priority) | +| `umount2` | Required | ❌ Not implemented | +| `ioctl FIONREAD` | Required | ❌ Not implemented | + +**7 of 12 previously missing syscalls are now implemented.** The kernel now has **137 syscalls** total. ### 2B. Missing ulibc Headers (completely absent) @@ -186,12 +167,12 @@ These POSIX headers do NOT exist at all in `user/ulibc/include/`: | Feature | Status | Impact | |---|---|---| | **`/etc/passwd`** and **`/etc/group`** | Not implemented | No user/group name resolution | -| **`/etc/hosts`** | Not implemented | No local hostname resolution | -| **Process groups / sessions** | Partial (basic) | Job control works but incomplete for Bash | -| **`/dev/tty`** | Exists as devfs entry | Need to verify controlling terminal semantics | -| **Proper `mode_t` permissions** | Stored but not enforced on open/exec | Need real permission checks | -| **`free()` in malloc** | No-op (bump allocator) | Programs that allocate/free heavily will OOM | -| **`wait4()`** | Missing | Some programs use this instead of waitpid | +| **`/etc/hosts`** | ✅ **IMPLEMENTED** | Kernel-level hosts file parsing and lookup | +| **Process groups / sessions** | ✅ **IMPLEMENTED** | Full job control: `setsid`, `setpgid`, `getpgrp`, `SIGTTIN`/`SIGTTOU` | +| **`/dev/tty`** | ✅ **IMPLEMENTED** | Controlling terminal with proper semantics | +| **Proper `mode_t` permissions** | ✅ **IMPLEMENTED** | VFS `open()` enforces rwx bits vs process euid/egid | +| **`free()` in malloc** | ✅ **FIXED** | ulibc uses proper `malloc`/`free`/`calloc`/`realloc` | +| **`wait4()`** | Not implemented | Some programs use this instead of waitpid | | **`time_t` as 32-bit** | `int32_t` — Y2038 issue | May cause issues with some programs | | **`off_t` as 32-bit** | `uint32_t` — 4GB file limit | Limits large file support | | **`ssize_t` return types** | `read()`/`write()` return `int` | POSIX requires `ssize_t` | @@ -227,26 +208,17 @@ Newlib is the most important prerequisite — GCC, Binutils, Bash, and Busybox a | `_execve()` | `SYSCALL_EXECVE` | ✅ Ready | | `_wait()` | `SYSCALL_WAITPID` | ✅ Ready | | `_times()` | `SYSCALL_TIMES` | ✅ Ready | -| `_gettimeofday()` | **MISSING** — need new syscall | ❌ **TODO** | +| `_gettimeofday()` | `SYSCALL_GETTIMEOFDAY` | ✅ Ready | | `_rename()` | `SYSCALL_RENAME` | ✅ Ready | | `_mkdir()` | `SYSCALL_MKDIR` | ✅ Ready | -**Steps to port Newlib:** - -1. **Add `gettimeofday` syscall** (wrapper over `clock_gettime` CLOCK_REALTIME) -2. **Add `mprotect` syscall** (needed by Newlib's `mmap`-based malloc) -3. **Create `libgloss/adros/` directory** with AdrOS-specific stubs: - - `syscalls.c` — maps `_read`, `_write`, `_open`, etc. to AdrOS `int 0x80`/`sysenter` - - `crt0.S` — C runtime startup (similar to existing ulibc `crt0.S`) -4. **Add AdrOS target to Newlib's configure** — `newlib/configure.host` entry for `i686-*-adros*` -5. **Build cross-Newlib:** - ```bash - mkdir build-newlib && cd build-newlib - ../newlib/configure --target=i686-adros --prefix=/opt/adros-toolchain - make && make install - ``` +**Newlib Port Status: ✅ COMPLETE** -**Estimated effort: Medium (2-3 days)** +- `newlib/libgloss/adros/` directory exists with syscall stubs and `posix_compat.c` +- `newlib/sysroot_headers/` provides compatibility headers +- `newlib/patches/` contains the Newlib AdrOS target patch +- All 21 required stubs are ready (including `gettimeofday` and `mprotect`) +- `toolchain/build.sh` automates the full cross-Newlib build ### 3B. GCC Port (cross-compiler targeting AdrOS) @@ -291,55 +263,45 @@ make && make install make && make install ``` -**Estimated effort: Large (1 week)** +**GCC Port Status: ✅ COMPLETE** -### 3C. Binutils Port +- Native GCC 13.2.0 (`xgcc`, `cc1`, `cpp`, `gcov`) built as ELF32 i686 static binaries +- Canadian cross build support implemented in `toolchain/build.sh` +- `toolchain/patches/gcc-adros.patch` contains all necessary target configuration -**What's needed:** - -1. **`bfd/config.bfd`** — add `i686-*-adros*` entry mapping to `bfd_elf32_i386_vec` -2. **`ld/emulparams/elf_i386_adros.sh`** — linker emulation parameters -3. **`ld/configure.tgt`** — add `i686-*-adros*` case -4. **`config.sub`** — add `adros` OS recognition +### 3C. Binutils Port -Binutils is simpler than GCC — the i386 ELF support already exists, just needs OS target wiring. +**Status: ✅ COMPLETE** -**Estimated effort: Small (1 day)** +- Native Binutils 2.42 (`ar`, `as`, `ld`, `objdump`) built as ELF32 i686 static binaries +- `toolchain/patches/binutils-adros.patch` contains target configuration ### 3D. Bash Port -**Critical missing kernel/libc features for Bash:** +**Status: Feasible.** Newlib port and native toolchain are complete. Key kernel syscalls (`getrlimit`, `gettimeofday`, `mprotect`) are now implemented. + +**Remaining blockers for Bash (via Newlib, not ulibc):** | Feature | Status | Blocking? | |---|---|---| -| `setjmp` / `longjmp` | ❌ Missing from ulibc | **YES** — used for error recovery, command abort | -| `execvp()` (PATH search) | ❌ Missing from ulibc | **YES** — core functionality | -| `getopt_long()` | ❌ Missing | **YES** — option parsing | -| `glob()` / `fnmatch()` | ❌ Missing | **YES** — wildcard expansion | -| `regex` (`regcomp/regexec`) | ❌ Missing | **YES** — `[[ =~ ]]` operator | -| `getpwnam()` / `getpwuid()` | ❌ Missing | **YES** — `~user` expansion, `$HOME` | -| `getrlimit()` / `setrlimit()` | ❌ Missing syscall | YES — `ulimit` builtin | -| `select()` with `FD_SET` macros | Syscall exists, macros missing | YES | -| `signal()` (simple handler) | ❌ Missing wrapper | YES | -| `strerror()` | ❌ Missing | YES | -| `sleep()` | ❌ Missing wrapper | YES | -| `setenv()` / `unsetenv()` | ❌ Missing | YES — environment modification | -| `strtoul()` | ❌ Missing | YES — arithmetic expansion | -| `atexit()` | ❌ Missing | YES — cleanup handlers | -| `locale` support | ❌ Missing | Partial — can stub | -| Proper `free()` in malloc | ❌ Bump allocator | **YES** — Bash allocates/frees constantly | -| `fork()` + `execve()` | ✅ Ready | — | -| `pipe()` + `dup2()` | ✅ Ready | — | -| `waitpid()` | ✅ Ready | — | -| `sigaction()` | ✅ Ready | — | -| `termios` (tcgetattr/tcsetattr) | ✅ Ready | — | -| `getcwd()` / `chdir()` | ✅ Ready | — | -| `stat()` / `fstat()` | ✅ Ready | — | -| `getenv()` | ✅ Ready | — | - -**Path to Bash:** Port Newlib first, then Bash becomes feasible. With ulibc alone, Bash is NOT portable. - -**Estimated effort: Large (1-2 weeks, after Newlib)** +| `setjmp` / `longjmp` | Provided by Newlib | ✅ | +| `execvp()` (PATH search) | Provided by Newlib | ✅ | +| `getopt_long()` | Provided by Newlib | ✅ | +| `glob()` / `fnmatch()` | Provided by Newlib (with sysroot patches) | ✅ | +| `regex` (`regcomp/regexec`) | Provided by Newlib | ✅ | +| `getpwnam()` / `getpwuid()` | ❌ Needs `/etc/passwd` + stubs | YES | +| `getrlimit()` / `setrlimit()` | ✅ Kernel syscalls implemented | ✅ | +| `select()` with `FD_SET` macros | ✅ Kernel syscall + Newlib macros | ✅ | +| `signal()` (simple handler) | Provided by Newlib | ✅ | +| `strerror()` | Provided by Newlib | ✅ | +| `sleep()` | Provided by Newlib | ✅ | +| `setenv()` / `unsetenv()` | Provided by Newlib | ✅ | +| `strtoul()` | Provided by Newlib | ✅ | +| `atexit()` | Provided by Newlib | ✅ | +| `locale` support | Provided by Newlib (C locale) | ✅ | +| Proper `free()` in malloc | ✅ ulibc fixed; Newlib has full malloc | ✅ | + +**Remaining effort: Medium (1 week) — cross-compile with Newlib, fix `/etc/passwd` stubs** ### 3E. Busybox Port @@ -351,72 +313,51 @@ Busybox uses a similar but even broader set of POSIX APIs. It requires everythin | `mntent` functions (mount table) | ❌ Missing | | `syslog()` | ❌ Missing | | `utmp` / `wtmp` (login records) | ❌ Missing | -| `getaddrinfo()` / `getnameinfo()` | Syscall exists, libc wrapper missing | -| `setsockopt()` / `getsockopt()` | ❌ Missing syscall | -| `sendmsg()` / `recvmsg()` | Syscall exists | +| `getaddrinfo()` / `getnameinfo()` | ✅ **Kernel syscall implemented** | +| `setsockopt()` / `getsockopt()` | ✅ **Kernel syscalls implemented** | +| `sendmsg()` / `recvmsg()` | ✅ **Kernel syscalls implemented** | | Full `ioctl` for network interfaces | Partial | -**Path to Busybox:** Port Newlib → port Bash → then Busybox is the next logical step. Busybox has `CONFIG_` options to disable features it can't use. +**Path to Busybox:** Newlib and toolchain are done. Cross-compile Busybox with minimal config, enable applets iteratively. -**Estimated effort: Very Large (2-3 weeks, after Newlib+Bash)** +**Estimated effort: Large (1-2 weeks)** --- ## Part 4: Prioritized Action Plan -### Phase 1: Build System Fix (immediate) -1. Add `.gitmodules` for lwIP and doomgeneric -2. Create root `.gitignore` -3. Add README.md with build prerequisites and instructions -4. Clean tracked build artifacts (`*.o`, `*.elf` in ulibc/src/) - -### Phase 2: Critical Syscalls (1-2 days) -1. `mprotect(addr, len, prot)` — map to VMM page protection change -2. `gettimeofday(tv, tz)` — wrapper over RTC + clock_gettime -3. `getrlimit` / `setrlimit` — per-process resource limits (RLIMIT_NOFILE, RLIMIT_STACK) -4. `setsockopt` / `getsockopt` — wire to lwIP - -### Phase 3: Critical ulibc Functions (2-3 days) -1. `setjmp` / `longjmp` (i386 assembly — save/restore ESP, EBP, EBX, ESI, EDI, EIP) -2. `sleep()` / `usleep()` (wrappers over nanosleep) -3. `execvp()` / `execlp()` (PATH search + execve) -4. `getopt()` / `getopt_long()` -5. `strerror()` / `perror()` -6. `strtoul()` / `strtoll()` / `strtoull()` -7. `setenv()` / `unsetenv()` / `putenv()` -8. `signal()` (simple wrapper over sigaction) -9. `abort()` / `atexit()` -10. `rand()` / `srand()` - -### Phase 4: Critical Headers (2-3 days) -1. `` — with i386 asm implementation -2. `` — stubs (C locale only) -3. `` / `` — parse /etc/passwd and /etc/group -4. `` — minimal regex engine or port TRE/PCRE -5. `` / `` -6. `` -7. `` — FD_SET/FD_CLR/FD_ISSET/FD_ZERO macros -8. Network headers (``, ``, ``, ``) - -### Phase 5: Proper malloc (1-2 days) -Replace bump allocator with a proper free-list or dlmalloc/K&R allocator in ulibc. - -### Phase 6: Newlib Port (2-3 days) -1. Create `libgloss/adros/` with syscall stubs -2. Add AdrOS target to Newlib configure -3. Build and validate - -### Phase 7: Binutils + GCC Port (1 week) -1. Add `i686-adros` target to Binutils -2. Add `i686-adros` target to GCC -3. Bootstrap cross-compiler - -### Phase 8: Bash Port (1-2 weeks) +### Phase 1: Build System Fix — **DONE ✅** +1. ✅ `.gitmodules` for lwIP +2. ✅ Root `.gitignore` +3. ✅ `README.md` + `BUILD_GUIDE.md` with full instructions + +### Phase 2: Critical Syscalls — **DONE ✅** +1. ✅ `mprotect` (syscall 128) +2. ✅ `gettimeofday` (syscall 127) +3. ✅ `getrlimit` / `setrlimit` (syscalls 129/130) +4. ✅ `setsockopt` / `getsockopt` (syscalls 131/132) + +### Phase 3: Critical ulibc Functions — **Partially done (Newlib provides the rest)** +Most of these are now provided by Newlib rather than ulibc. The ulibc `malloc`/`free` is fixed (proper buddy allocator). Newlib provides `setjmp`, `strerror`, `getopt`, `setenv`, etc. + +### Phase 4: Critical Headers — **Provided by Newlib sysroot** +Newlib sysroot headers installed at `newlib/sysroot_headers/` cover network, select, locale, etc. + +### Phase 5: Proper malloc — **DONE ✅** +ulibc now has proper `malloc`/`free`/`calloc`/`realloc`. + +### Phase 6: Newlib Port — **DONE ✅** +`newlib/libgloss/adros/` with all stubs, `toolchain/build.sh` automates build. + +### Phase 7: Binutils + GCC Port — **DONE ✅** +Native Binutils 2.42 + GCC 13.2.0 built as ELF32 i686 static binaries. + +### Phase 8: Bash Port (next step) 1. Cross-compile Bash with `i686-adros-gcc` + Newlib -2. Fix missing stubs iteratively +2. Add `/etc/passwd` stub for `getpwnam` 3. Package in initrd -### Phase 9: Busybox Port (2-3 weeks) +### Phase 9: Busybox Port (after Bash) 1. Cross-compile with minimal config 2. Enable applets iteratively 3. Replace individual `/bin/*` utilities @@ -427,13 +368,13 @@ Replace bump allocator with a proper free-list or dlmalloc/K&R allocator in ulib | Component | Current State | Ready to Port? | |---|---|---| -| **Kernel syscalls** | 126 syscalls, ~85% POSIX | Missing: mprotect, getrlimit, gettimeofday | -| **ulibc** | 27 headers, basic functions | **NOT sufficient** for Bash/Busybox | -| **Build system** | Works locally, breaks on git clone | Needs submodules + .gitignore | -| **Newlib** | Not started | **Feasible** — 19/21 required stubs ready | -| **Binutils** | Not started | Easy — just target config | -| **GCC** | Not started | Feasible after Newlib | -| **Bash** | Not started | Needs Newlib + setjmp + glob + regex | -| **Busybox** | Not started | Needs Newlib + extensive libc | - -**Bottom line:** The kernel is ~85% POSIX-ready. The main blocker is **ulibc** — it lacks too many functions for real-world programs. The fastest path to Bash/Busybox is: **fix critical syscalls → port Newlib → build cross-toolchain → cross-compile Bash**. +| **Kernel syscalls** | 137 syscalls, ~98% POSIX | ✅ All critical syscalls implemented | +| **ulibc** | Full libc for AdrOS userspace | ✅ Sufficient for 52 utilities | +| **Build system** | Works with `git clone --recursive` | ✅ Submodules + .gitignore | +| **Newlib** | ✅ **DONE** | `newlib/libgloss/adros/` with all stubs | +| **Binutils** | ✅ **DONE** | Native 2.42 (ar, as, ld, objdump) | +| **GCC** | ✅ **DONE** | Native 13.2.0 (xgcc, cc1, cpp) | +| **Bash** | Not started | **Feasible** — all kernel blockers resolved, Newlib provides libc | +| **Busybox** | Not started | **Feasible** — after Bash | + +**Bottom line:** The kernel is **~98% POSIX-ready** with 137 syscalls. The Newlib port and native toolchain (GCC 13.2 + Binutils 2.42) are **complete**. The next step is cross-compiling Bash, which is now feasible since all kernel-level blockers have been resolved. AdrOS ships with 52 native POSIX utilities, 102 smoke tests, and 115 host tests. diff --git a/docs/POSIX_ROADMAP.md b/docs/POSIX_ROADMAP.md index 161ee83..b5dea81 100644 --- a/docs/POSIX_ROADMAP.md +++ b/docs/POSIX_ROADMAP.md @@ -88,6 +88,15 @@ Notes: | `alarm` | [x] | Per-process alarm timer; delivers `SIGALRM` on expiry | | `times` | [x] | Returns `struct tms` with per-process `utime`/`stime` accounting | | `futex` | [x] | `FUTEX_WAIT`/`FUTEX_WAKE` with global waiter table | +| `waitid` | [x] | Extended wait with `P_PID`/`P_ALL`/`P_PGID` | +| `posix_spawn` | [x] | Efficient fork+exec in single syscall with file actions and attributes | +| `setitimer`/`getitimer` | [x] | `ITIMER_REAL`, `ITIMER_VIRTUAL`, `ITIMER_PROF` | +| `gettimeofday` | [x] | Returns wall-clock time (backed by RTC) | +| `mprotect` | [x] | Change memory protection flags on existing mappings | +| `getrlimit`/`setrlimit` | [x] | Get/set resource limits (`RLIMIT_NOFILE`, etc.) | +| `uname` | [x] | Returns system identification (sysname, nodename, release, version, machine) | +| `getrusage` | [x] | Returns resource usage statistics | +| `mount` | [x] | Runtime filesystem mounting (tmpfs, disk-based via `init_mount_fs`) | ## 4. Syscalls — Signals @@ -111,6 +120,7 @@ Notes: | `epoll_create` | [x] | Creates epoll instance; returns fd | | `epoll_ctl` | [x] | Add/modify/delete fd interest; `EPOLL_CTL_ADD`/`MOD`/`DEL` | | `epoll_wait` | [x] | Wait for events on epoll fd; timeout support | +| `EPOLLET` (edge-triggered) | [x] | Edge-triggered epoll mode with level-to-edge semantics | | `inotify_init` | [x] | Creates inotify instance; returns fd | | `inotify_add_watch` | [x] | Add watch on path with event mask | | `inotify_rm_watch` | [x] | Remove watch by descriptor | @@ -228,6 +238,10 @@ Notes: | DNS resolver | [x] | lwIP DNS enabled; kernel `dns_resolve()` wrapper with async callback + timeout | | `/etc/hosts` | [x] | Kernel-level hosts file parsing and lookup | | `getaddrinfo` | [x] | Kernel-level hostname resolution with hosts file + DNS fallback | +| `setsockopt`/`getsockopt` | [x] | Set/get socket options | +| `shutdown` | [x] | Shutdown socket send/receive | +| `getpeername` | [x] | Get remote address of connected socket | +| `getsockname` | [x] | Get local address of socket | ## 11. Threads & Synchronization @@ -259,6 +273,7 @@ Notes: | Feature | Status | Notes | |---------|--------|-------| | `pivot_root` | [x] | Swap root filesystem; mounts old root at specified path | +| LZ4 initrd decompression | [x] | LZ4 frame decompression for compressed initrd images (`src/kernel/lz4.c`) | ## 12. Dynamic Linking @@ -279,13 +294,31 @@ Notes: | Feature | Status | Notes | |---------|--------|-------| | ELF32 loader | [x] | Secure with W^X + ASLR; supports `ET_EXEC` + `ET_DYN` + `PT_INTERP` | -| `/bin/init.elf` (smoke tests) | [x] | Comprehensive test suite (80 checks: file I/O, signals, memory, IPC, devices, procfs, networking, epoll, inotify, aio, nanosleep, CoW fork, readv/writev, fsync, flock, posix_spawn, TSC precision, execve) | +| `/sbin/fulltest` (smoke tests) | [x] | Comprehensive test suite (102 checks: file I/O, signals, memory, IPC, devices, procfs, networking, epoll, epollet, inotify, aio, nanosleep, CoW fork, readv/writev, fsync, flock, posix_spawn, TSC precision, gettimeofday, mprotect, getrlimit/setrlimit, uname, LZ4, lazy PLT, execve) | | `/bin/echo` | [x] | argv/envp test | | `/bin/sh` | [x] | POSIX sh-compatible shell; builtins, pipes, redirects, `$PATH` search | | `/bin/cat` | [x] | | | `/bin/ls` | [x] | Uses `getdents` | | `/bin/mkdir` | [x] | | | `/bin/rm` | [x] | | +| `/bin/cp`, `/bin/mv` | [x] | File copy and move | +| `/bin/touch`, `/bin/ln` | [x] | Create files, hard/symbolic links | +| `/bin/head`, `/bin/tail` | [x] | Display first/last lines | +| `/bin/wc`, `/bin/sort`, `/bin/uniq`, `/bin/cut` | [x] | Text processing utilities | +| `/bin/grep` | [x] | Pattern matching with `-v`/`-c`/`-n` flags | +| `/bin/sed`, `/bin/awk` | [x] | Stream editor and pattern processing | +| `/bin/find`, `/bin/which` | [x] | File search and command lookup | +| `/bin/chmod`, `/bin/chown`, `/bin/chgrp` | [x] | Permission management | +| `/bin/mount`, `/bin/umount` | [x] | Filesystem mount/unmount | +| `/bin/ps`, `/bin/top`, `/bin/kill` | [x] | Process management | +| `/bin/df`, `/bin/du`, `/bin/free` | [x] | Disk and memory usage | +| `/bin/date`, `/bin/hostname`, `/bin/uptime`, `/bin/uname` | [x] | System information | +| `/bin/env`, `/bin/printenv`, `/bin/id` | [x] | Environment and identity | +| `/bin/tee`, `/bin/dd`, `/bin/tr` | [x] | I/O utilities | +| `/bin/basename`, `/bin/dirname`, `/bin/pwd`, `/bin/stat` | [x] | Path and file info | +| `/bin/sleep`, `/bin/clear`, `/bin/rmdir`, `/bin/dmesg`, `/bin/who` | [x] | Misc utilities | +| `/sbin/init` | [x] | SysV-like init process (inittab, runlevels, respawn) | +| `/bin/pie_test` | [x] | PIE/shared library test binary | | `/bin/doom.elf` | [x] | DOOM (doomgeneric port) running on `/dev/fb0` + `/dev/kbd` | | `/lib/ld.so` | [x] | Dynamic linker with auxv parsing, PLT/GOT eager relocation, `dlopen`/`dlsym`/`dlclose` | | Minimal libc (ulibc) | [x] | `printf`, `malloc`, `string.h`, `unistd.h`, `errno.h`, `pthread.h`, `signal.h`, `stdio.h`, `stdlib.h`, `ctype.h`, `sys/mman.h`, `sys/ioctl.h`, `time.h`, `math.h`, `assert.h`, `fcntl.h`, `strings.h`, `inttypes.h`, `sys/types.h`, `sys/stat.h`, `sys/times.h`, `sys/uio.h`, `linux/futex.h` | @@ -312,7 +345,7 @@ Notes: ## Implementation Progress -### All 31 planned tasks completed ✅ + 44 additional features (75 total) +### All 31 planned tasks completed ✅ + 59 additional features (90 total) **High Priority (8/8):** 1. ~~`raise()` em ulibc~~ ✅ @@ -396,6 +429,21 @@ Notes: 73. ~~`pivot_root` — root filesystem swap syscall~~ ✅ 74. ~~Shared library lazy binding — functional ld.so with auxv, PLT/GOT~~ ✅ 75. ~~`aio_*` — POSIX asynchronous I/O~~ ✅ +76. ~~`gettimeofday` syscall~~ ✅ +77. ~~`mprotect` memory protection syscall~~ ✅ +78. ~~`getrlimit`/`setrlimit` resource limits~~ ✅ +79. ~~`uname` system info syscall~~ ✅ +80. ~~`setsockopt`/`getsockopt`/`shutdown`/`getpeername`/`getsockname` socket ops~~ ✅ +81. ~~`getrusage` resource usage~~ ✅ +82. ~~LZ4 initrd decompression~~ ✅ +83. ~~`EPOLLET` edge-triggered epoll~~ ✅ +84. ~~PLT/GOT lazy binding in userspace ld.so~~ ✅ +85. ~~MIPS32 bring-up (QEMU Malta boot + UART)~~ ✅ +86. ~~SysV `/sbin/init` (inittab, runlevels, respawn)~~ ✅ +87. ~~50+ userland POSIX utilities (cp, mv, sed, awk, grep, find, etc.)~~ ✅ +88. ~~Host utility test harness (68 cross-platform tests)~~ ✅ +89. ~~Native toolchain (GCC 13.2 + Binutils 2.42, Canadian cross for i686-adros)~~ ✅ +90. ~~`mount` syscall — runtime filesystem mounting~~ ✅ --- @@ -409,9 +457,7 @@ Potential future enhancements: |------|-------------| | **Rump Kernel Phase 2** | Thread/sync hypercalls (`rumpuser_thread_create`, `rumpuser_mutex_*`, `rumpuser_cv_*`) | | **Rump Kernel Phase 4** | File/block I/O hypercalls for rump filesystem drivers | -| **Full SMP scheduling** | Move processes to AP runqueues (infrastructure in place) | +| **Full SMP scheduling** | Move processes to AP runqueues; per-CPU dispatching (infrastructure in place) | | **ARM64/RISC-V/MIPS subsystems** | PMM, VMM, scheduler, syscalls for non-x86 | | **Intel HDA audio** | DMA ring buffer audio driver | -| **USTAR+LZ4 InitRD** | Alternative initrd format (current: custom binary) | -| **PLT/GOT lazy binding** | Userspace resolver trampoline in ld.so (currently eager) | -| **epoll edge-triggered** | `EPOLLET` full implementation with level-to-edge semantics | +| **USTAR initrd format** | Alternative to custom binary format (LZ4 decompression already implemented) | diff --git a/docs/SUPPLEMENTARY_ANALYSIS.md b/docs/SUPPLEMENTARY_ANALYSIS.md index ff6a16c..453816f 100644 --- a/docs/SUPPLEMENTARY_ANALYSIS.md +++ b/docs/SUPPLEMENTARY_ANALYSIS.md @@ -70,14 +70,14 @@ Unix-like, POSIX-compatible operating system. | O(1) scheduler (bitmap + active/expired) | ✅ Full implementation | ✅ Bitmap + active/expired swap, 32 priority levels | None | | Priority queues (MLFQ) | ✅ 32 priority levels | ✅ 32 priority levels via `SCHED_NUM_PRIOS` | None | | Unix decay-based priority | ✅ `p_cpu` decay + `nice` | ✅ Priority decay on time slice exhaustion; boost on sleep wake | None | -| Per-CPU runqueues | ✅ `cpu_runqueue_t` per CPU | ❌ Single global queue | Needed for SMP | +| Per-CPU runqueues | ✅ `cpu_runqueue_t` per CPU | ✅ Per-CPU load counters with atomics, least-loaded CPU query | Infrastructure in place; full dispatch pending | | Sleep/wakeup (wait queues) | ✅ `sleep(chan, lock)` / `wakeup(chan)` | ✅ Generic `waitqueue_t` abstraction + `nanosleep` syscall | None | | Context switch (assembly) | ✅ Save/restore callee-saved + CR3 | ✅ `context_switch.S` saves/restores regs + CR3 | None | | `fork()` | ✅ Slab + CoW + enqueue | ✅ `vmm_as_clone_user_cow()` + page fault handler | None | | `execve()` | ✅ Load ELF, reset stack | ✅ `syscall_execve_impl()` — loads ELF, handles argv/envp, `O_CLOEXEC` | None | | Spinlock protection | ✅ `sched_lock` | ✅ `sched_lock` present | None | -**Summary:** AdrOS scheduler is O(1) with bitmap + active/expired arrays, 32 priority levels, and decay-based priority adjustment. Only missing per-CPU runqueues for SMP. +**Summary:** AdrOS scheduler is O(1) with bitmap + active/expired arrays, 32 priority levels, and decay-based priority adjustment. Per-CPU runqueue infrastructure is in place (load counters with atomics); full per-CPU dispatch is pending. --- @@ -106,7 +106,7 @@ Unix-like, POSIX-compatible operating system. | File descriptor table | ✅ Per-process `fd_table[16]` | ✅ Per-process `files[PROCESS_MAX_FILES]` with refcount | None | | File cursor (offset) | ✅ `cursor` field | ✅ `offset` in `struct file` | None | | USTAR InitRD parser | ✅ Full implementation | ❌ Custom binary format (`mkinitrd`) | Different approach, both work | -| LZ4 decompression | ✅ Decompress initrd.tar.lz4 | ❌ Not implemented | Enhancement | +| LZ4 decompression | ✅ Decompress initrd.tar.lz4 | ✅ LZ4 frame decompression (`src/kernel/lz4.c`) | None | | `pivot_root` | ✅ `sys_pivot_root()` | ✅ Swaps root filesystem, mounts old root at specified path | None | | Multiple FS types | ✅ USTAR + FAT | ✅ tmpfs + devfs + overlayfs + diskfs + persistfs + procfs + FAT12/16/32 + ext2 + initrd | **AdrOS is ahead** | | `readdir` generic | Mentioned | ✅ All FS types implement `readdir` callback | None | @@ -231,7 +231,7 @@ for the full list. All previously identified Tier 1/2/3 gaps have been resolved. 8. **Hardware** — PCI, ATA PIO+DMA (bounce + zero-copy), Virtio-blk, LAPIC/IOAPIC, SMP (4 CPUs), ACPI, VBE framebuffer, SYSENTER, CPUID, RTC, MTRR write-combining 9. **Networking** — E1000 NIC, lwIP TCP/IP (IPv4+IPv6 dual-stack), socket API (TCP+UDP), DNS resolver, DHCP client 10. **Userland** — ulibc (full libc), ELF loader with W^X + ASLR, functional `ld.so` (auxv + PLT/GOT + `dlopen`/`dlsym`/`dlclose`), POSIX shell, core utilities, DOOM port -11. **Testing** — 44 smoke tests, 16 battery checks, 19 host unit tests, cppcheck, sparse, gcc -fanalyzer, GDB scripted checks +11. **Testing** — 102 smoke tests, 16 battery checks, 115 host tests (28 unit + 19 security + 68 utility), cppcheck, sparse, gcc -fanalyzer, GDB scripted checks 12. **Security** — SMEP, PAE+NX, ASLR, guard pages (user + kernel), user_range_ok hardened, sigreturn eflags sanitized, atomic file refcounts, VFS permission enforcement (uid/gid/euid/egid vs file mode) 13. **Scheduler** — O(1) with bitmap + active/expired, 32 priority levels, decay-based priority, CPU time accounting 14. **Threads** — `clone`, `gettid`, TLS via GDT, pthread in ulibc, futex synchronization @@ -348,16 +348,15 @@ It is approximately **98% of the way** to a practical POSIX-compatible system. The supplementary material's architectural blueprints have been **fully realized and exceeded**: CoW memory, O(1) scheduling, slab allocator, PCI enumeration, CPUID detection, -zero-copy DMA, vDSO, E1000 networking, and PAE+NX are all implemented. AdrOS is -**significantly ahead** of the supplementary material in VFS diversity, signal handling, -TTY/PTY, driver support, networking, userland tooling, and security hardening (ASLR, -guard pages, SMEP/SMAP). +zero-copy DMA, vDSO, E1000 networking, PAE+NX, and LZ4 decompression are all implemented. +AdrOS is **significantly ahead** of the supplementary material in VFS diversity, signal +handling, TTY/PTY, driver support, networking, userland tooling (52 POSIX utilities), +and security hardening (ASLR, guard pages, SMEP/SMAP). The remaining enhancements are: **Rump Kernel integration** (Phase 2 thread/sync hypercalls and Phase 4 file/block I/O — prerequisites including condition variables, TSC nanosecond clock, and IRQ chaining are already implemented), **full SMP scheduling** -(moving processes to AP runqueues), **non-x86 subsystems** (PMM/VMM/scheduler for -ARM64/RISC-V/MIPS), Intel HDA audio, USTAR+LZ4 initrd, PLT/GOT lazy binding -(currently eager), and `EPOLLET` edge-triggered mode. +(moving processes to AP runqueues — per-CPU infrastructure in place), **non-x86 subsystems** +(PMM/VMM/scheduler for ARM64/RISC-V/MIPS), and Intel HDA audio. -80 QEMU smoke tests, 16 battery checks, and 47 host unit tests pass clean. +102 QEMU smoke tests, 16 battery checks, and 115 host tests pass clean. diff --git a/docs/TESTING_PLAN.md b/docs/TESTING_PLAN.md index 8e97f0a..98453c0 100644 --- a/docs/TESTING_PLAN.md +++ b/docs/TESTING_PLAN.md @@ -5,9 +5,9 @@ All testing layers are **implemented and operational**: - **Static analysis** (`make check`): cppcheck + sparse + gcc -fanalyzer -- **QEMU smoke tests** (`make test`): expect-based, 80 checks (file I/O, signals, memory, IPC, devices, procfs, networking, epoll, inotify, aio, nanosleep, CLOCK_REALTIME, /dev/urandom, /proc/cmdline, CoW fork, readv/writev, fsync, truncate, getuid/getgid, chmod, flock, times, gettid, posix_spawn, TSC ns precision, SIGSEGV, execve), 4-CPU SMP, 120s timeout +- **QEMU smoke tests** (`make test`): expect-based, 102 checks (file I/O, signals, memory, IPC, devices, procfs, networking, epoll, epollet, inotify, aio, nanosleep, CLOCK_REALTIME, /dev/urandom, /proc/cmdline, CoW fork, readv/writev, fsync, truncate, getuid/getgid, chmod, flock, times, gettid, posix_spawn, TSC ns precision, SIGSEGV, gettimeofday, mprotect, getrlimit/setrlimit, uname, LZ4 initrd decomp, lazy PLT, execve), 4-CPU SMP, 120s timeout - **Test battery** (`make test-battery`): 16 checks across 5 QEMU scenarios — multi-disk ATA, VFS mount, ping, diskfs -- **Host unit tests** (`make test-host`): 19 tests — `test_utils.c` + `test_security.c` +- **Host unit tests** (`make test-host`): 115 tests — `test_utils.c` (28) + `test_security.c` (19) + `test_host_utils.sh` (68 cross-compiled utility tests) - **GDB scripted checks** (`make test-gdb`): heap/PMM/VGA integrity validation - **Full suite** (`make test-all`): runs check + test-host + test - **Multi-arch build verification**: `make ARCH=arm`, `make ARCH=riscv`, and `make ARCH=mips` compile and boot on QEMU @@ -111,7 +111,7 @@ To run manually: boot AdrOS with `-vga std`, then execute `/bin/doom.elf` from t ```makefile make check # cppcheck + sparse + gcc -fanalyzer -make test # QEMU + expect automated smoke test (80 checks incl. ICMP ping, epoll, inotify, aio, CoW fork, flock, posix_spawn) +make test # QEMU + expect automated smoke test (102 checks incl. ICMP ping, epoll, epollet, inotify, aio, CoW fork, flock, posix_spawn, gettimeofday, mprotect, uname, LZ4, lazy PLT) make test-battery # Full test battery: multi-disk ATA, VFS mount, ping, diskfs (16 checks) make test-host # Host-side unit tests for pure functions make test-gdb # QEMU + GDB scripted checks (optional)