Explore enabling SH4 MMU for debug builds to catch use-after-free

[drpaneas]

Context

The Dreamcast's SH4 has an MMU, but KallistiOS disables it by default for performance. This means use-after-free bugs silently corrupt memory instead of crashing. Currently we rely on the caller doing slice = nil after freeExternal (see #2), which is easy to forget.

Idea

Enable the MMU in debug builds only, so freed memory pages are marked inaccessible. A use-after-free would trigger a hardware fault instead of silent corruption. This would not affect release builds.

What KOS provides

KOS has full MMU infrastructure in arch/mmu.h:

• mmu_init() / mmu_shutdown()
• mmu_page_map() with protection values (MMU_KERNEL_RDONLY, MMU_ALL_RDWR, etc.)
• TLB management

Challenges

• TLB pressure: The SH4 UTLB has only 64 entries. 16MB of RAM in 4KB pages = 4096 pages. Frequent TLB misses would add overhead on a 200MHz CPU.
• Hook into allocator: gc_external_alloc / gc_external_free would need to update page permissions on alloc and free. Every large-object free would require an mmu_page_map call to mark the page(s) as inaccessible.
• KOS compatibility: KOS drivers and libraries assume flat memory access. The MMU could interfere with DMA, hardware register access, or other subsystems.
• Granularity: MMU protection works at page granularity (4KB). A free() of a 128KB buffer spans 32 pages. Marking them all inaccessible is feasible but adds latency to every free.

Investigation steps

• Test basic mmu_init() on real hardware and emulator (lxdream/flycast) - does KOS still work?
• Measure TLB miss overhead with a simple benchmark (frame time with/without MMU)
• Prototype: hook gc_external_free to mark freed pages as no-access, verify that accessing freed memory triggers a fault
• Check if KOS DMA and PVR still work with MMU enabled (texture transfers, sound streaming)
• If viable, gate behind a GC_DEBUG_MMU compile flag so release builds are unaffected