Skip to content

UsingGaussianSystem.md

Latest commit

 

History

History
485 lines (382 loc) · 21.2 KB

UsingGaussianSystem.md

File metadata and controls

485 lines (382 loc) · 21.2 KB
Raw

Enabling Gaussian System in Untold Engine

The Gaussian System in the Untold Engine is responsible for rendering Gaussian Splatting models. It enables you to visualize high-quality 3D reconstructions created from photogrammetry or neural rendering techniques, providing a modern approach to displaying complex 3D scenes.

How to Enable the Gaussian System

Step 1: Create an Entity

Start by creating an entity that represents your Gaussian Splat object.

let myEntity = createEntity()

Step 2: Link a Gaussian Splat to the Entity

To display a Gaussian Splat model, load its .ply file and link it to the entity using setEntityGaussian.

setEntityGaussian(entityId: myEntity, filename: "splat", withExtension: "ply")

You can also use the source-based API:

setEntityGaussian(
    entityId: myEntity,
    source: .single(filename: "splat", withExtension: "ply")
)

Parameters:

  • entityId: The ID of the entity created earlier.
  • filename: The name of the .ply file (without the extension).
  • withExtension: The file extension, typically "ply".

Note: The Gaussian System renders point cloud data stored in the .ply format. Ensure your Gaussian Splat file is properly formatted and contains the necessary attributes (position, color, opacity, scale, rotation).

A baked .untoldgs file (see Exporting Assets) loads the same way and is the faster path: its chunks are read by byte range and decoded on the GPU, so nothing is parsed on the CPU.

setEntityGaussian(entityId: myEntity, filename: "splat", withExtension: "untoldgs")

Both forms load synchronously and keep the splat resident for the entity's lifetime, and both compute the entity's LocalTransformComponent.boundingBox automatically from the loaded splat positions — no bounding box parameter is needed for this path.

Step 3: Loading Without Blocking the Main Thread

setEntityGaussianAsync does the same immediate/resident load as setEntityGaussian, but parsing, per-splat encoding, and spherical-harmonics packing all run off the main thread — only the final component registration touches the world. Use it for a one-off splat load where you don't want a frame hitch but don't need distance-based streaming.

Task {
    let ok = await setEntityGaussianAsync(
        entityId: myEntity,
        filename: "splat",
        withExtension: "ply"
    )
    if !ok {
        print("Failed to load splat")
    }
}

completion is an optional alternative to checking the returned Bool:

await setEntityGaussianAsync(
    entityId: myEntity,
    filename: "splat",
    withExtension: "ply"
) { success in
    print(success ? "Loaded" : "Failed to load splat")
}

Running the Gaussian System

Once everything is set up:

  1. Run the project.
  2. Your Gaussian Splat model will appear in the game window.
  3. If the model is not visible or appears incorrect, revisit the file path and format to ensure everything is loaded correctly.

Several splat entities in one scene

Every frame the engine compacts the visible splats of all Gaussian entities into one shared working set, sorts it once by depth and draws it with one instanced draw. Two captures that overlap on screen — a chair partly in front of a table, a prop on a splat floor — therefore blend in true depth order; the order the entities were created in does not matter. The shared set is sized to the resident splat total, so every loaded splat fits; should the entities ever append more than it holds, the excess is dropped for that frame and reported through handleError and the Gaussian profile line as overflow. Up to 256 splat entities can be drawn in one frame.

Per-entity splat limit

Every loaded splat keeps about 320 bytes resident on the GPU (its 48-byte encoded record, a visible index per frame in flight, its 72-byte share of the shared working set per frame in flight, and its spherical harmonics), so the runtime caps one entity at GaussianRuntimeLimits.maxSplatsPerEntity: 5,242,880 splats on Apple Vision Pro, iPhone, iPad and Apple TV, 16,777,216 on the Mac. A .untoldgs or .ply above the cap fails to load with an "exceeds maximum" error. Cook large captures with a splat budget (UntoldGSCookOptions.maxSplatCount, untoldengine export --splat-max-count) that fits every platform the asset ships on, or split the scene into streamed tiles.

A splat standing in for a mesh: shells, fades and scene links

A captured object looks best as a splat up close and costs least as a mesh far away. The engine gives an application the pieces to swap between the two on one entity without popping; the policy that drives them (when to load, from what distance, how fast to fade) belongs to an application-side system built on these pieces (see the proposal's §4.5).

  • A splat on a mesh entity. setEntityGaussianAsync(entityId:url:opacityScale:) attaches a .untoldgs (or .ply) to an entity that already draws a mesh. The load is two phases a caller can also drive itself: loadGaussianSplatPayload(url:) reads and encodes off the main thread, setEntityGaussian(entityId:payload:opacityScale:) attaches the result under the world-mutation gate, so a system can check under its own gate whether the load is still wanted before applying it. The mesh stays the primary representation: the splat's bytes ride beside the mesh's MemoryBudgetManager entry (auxiliaryMeshBytes, so mesh streaming in and out leaves them intact) and the entity keeps the mesh's bounding box, with the splat's own box on GaussianComponent.localBoundingBox. removeEntityGaussian drops the splat (and a progressive splat's tiers) and only its share of the ledger. Starting with opacityScale: 0 keeps it resident but hidden.
  • GaussianComponent.opacityScale weighs every splat's opacity: 0 hides the entity and skips its cull, values between cross-fade.
  • MeshFadeComponent dithers the mesh's colour with the LOD screen-door: .fadeOut discards more pixels as progress rises, .fadeIn keeps more. Applied after the LOD and tile fades.
  • MeshOccluderComponent draws the mesh a second time depth-only, pushed shrinkMeters along its normals away from the camera (the meshOccluderShell render pass, after the opaque colour and before the HZB copy and the splat pass). The splat then passes the depth test on and just outside the surface, and is hidden behind the object's far side. With drawsColor off the mesh contributes nothing but that depth: shadows, physics and picking keep using it because RenderComponent.isVisible is untouched. Soft objects differ from their mesh by centimetres, so raise the margin until the front of the capture stops clipping. Blend-mode submeshes are left out of the shell and stop drawing with the colour.
  • GaussianAssetLinkComponent carries a .untold scene's gaussianAsset record (UntoldGaussianAssetRecordV1: payload path resolved next to the scene file, flags such as meshTwin, occluder margin, exposure offset, swap distance) onto the entity as data. setEntityMesh/setEntityMeshAsync attach it; nothing is loaded.
  • A mesh carrying a MeshOccluderComponent or MeshFadeComponent is excluded from static batching when the batcher next evaluates it, and re-admitted once they are gone. The system that adds or removes them tells the batcher with BatchingSystem.shared.notifyEntityMaterialChanged(entityId:); the group is rebuilt over a few frames, during which the batch still draws the mesh.
  • GaussianDebugOptions.shared.disableOccluderShell turns the shells off for bisecting.

A typical swap: load the payload with opacityScale: 0 when the camera is near; add a MeshOccluderComponent; add a MeshFadeComponent with direction = .fadeOut and raise its progress and the splat's opacityScale together to 1 over 250 ms; then set drawsColor = false and remove the fade. Reverse the steps when the camera leaves.

Calibrating the capture

Splats are unlit emissive surfaces composited in linear light before the look and output transforms, so a splat is tone-mapped once, like an emissive mesh next to it. The preprocess applies one linear gain per entity, GaussianComponent.colorGain: the capture white balance from the .untoldgs header, times 2^(exposureOffsetEV − captureExposureEV). A capture recorded at +1 EV therefore comes back to the scene's neutral exposure by itself, and the per-asset offset (the editor slider, exposureOffsetEV in the scene record) pushes it either way. With useRealWorldTint the colour is also multiplied by the XR lighting estimate's tint whenever RuntimeEnvironmentLightingStore is in .realWorldEstimate mode with a valid estimate, so a capture made under neutral light takes on the colour of the room.

Progressive Gaussian Splats

Progressive Gaussian loading is available without a tile-streamed scene. Use it when you want a Gaussian to appear quickly at a coarse tier, then refine toward full resolution as the camera gets closer.

Progressive assets use .untoldgs tier files:

<baseFilename>_lod0.untoldgs
<baseFilename>_lod1.untoldgs
<baseFilename>_lod2.untoldgs
...

lod0 is the finest/full-resolution tier. Higher LOD numbers are progressively coarser. The engine loads the coarsest tier first, then GaussianLODSystem requests finer tiers based on camera distance (see Overdraw-aware LOD selection below for a second, distance-independent signal that can also hold an entity on a coarser tier).

Generate tiers from a .ply source with the exporter:

untoldengine export --input "chair.ply" --output "chair.untoldgs" --lod-levels 4

The exporter prints a diagnostic meanSquaredSplatExtent per tier and a boundingBoxHalfExtent line computed from the full source asset:

✅ Exported: chair_lod0.untoldgs (meanSquaredSplatExtent: 0.0021)
✅ Exported: chair_lod1.untoldgs (meanSquaredSplatExtent: 0.0087)
✅ Exported: chair_lod2.untoldgs (meanSquaredSplatExtent: 0.0341)
✅ Exported: chair_lod3.untoldgs (meanSquaredSplatExtent: 0.1250)
ℹ️ boundingBoxHalfExtent: (0.42, 0.55, 0.38)

Both values are baked directly into each tier's .untoldgs file (its header carries the asset-level bounding box alongside meanSquaredSplatExtent) and read back automatically when the engine loads it — nothing here needs to be copied into your code. The console lines are diagnostics only (e.g. to sanity-check density/size across source captures).

Then register the entity with the source-based API:

let chair = createEntity()
translateTo(entityId: chair, position: simd_float3(0.0, 0.0, -3.0))

setEntityGaussian(
    entityId: chair,
    source: .progressive(
        baseFilename: "chair",
        levelCount: 4,
        maxDistances: [5.0, 15.0, 25.0, .greatestFiniteMagnitude]
    )
)

maxDistances must have one entry per LOD. Each value is the farthest camera distance at which that LOD is allowed to be selected:

  • lod0 can be used inside 5.0 units.
  • lod1 can be used from 5.0 to 15.0 units.
  • lod2 can be used from 15.0 to 25.0 units.
  • lod3 is used beyond 25.0 units, or while finer tiers are still loading.

The system always falls back to the best tier already resident in memory, so the entity can become visible quickly with the coarsest tier and refine toward lod0.

There's no bounding-box parameter to pass here: the engine reads the box baked into the coarsest tier's header synchronously at registration time, so the entity has a correct, exact bounding box from frame one.

Overdraw-aware LOD selection

Distance alone is a proxy for how expensive a Gaussian entity is to render — two assets at the same distance can have very different overdraw depending on splat density. On top of the distance/maxDistances selection above, GaussianLODSystem also estimates the entity's mean overdraw (blended fragments per pixel across its screen footprint) each LOD update, using meanSquaredSplatExtent values baked into each .untoldgs tier by the exporter. If a distance-selected tier would exceed LODConfig.shared.gaussianOverdrawBudget (default 12.0, see LODConfig.swift), the system walks to a coarser tier instead — it never picks a finer tier than distance already allows.

This is fully automatic for any asset baked with the current exporter — there is nothing to wire up. It only has an effect on tiers that carry a real meanSquaredSplatExtent; .untoldgs files baked before this feature existed fall back to pure distance-based selection.

Tune LODConfig.shared.gaussianOverdrawBudget on-device by watching GPU frame time while varying it — the default is a starting guess, not a derived constant.

Debugging progressive LOD

Gaussian progressive LODs participate in the same LOD debug visualization used by mesh LODs:

setSpatialDebug(.lodLevels(true))

When enabled, the renderer tints Gaussian splats by their currently selected progressive LOD. This is useful for confirming that the engine is switching tiers as the camera moves, including tiers the overdraw budget forces early.

Streaming Gaussian Splats in Large Scenes

setEntityGaussian loads a splat immediately and keeps it resident for the lifetime of the entity — fine for a small number of always-visible splats, but not what you want for props scattered across a large tile-streamed scene (chairs, tables, decor inside a streamed building). Loading every one of those up front defeats the point of streaming, and the engine has no way to unload them again on its own.

For that case, register the entity with GeometryStreamingSystem instead, via setEntityGaussianStreaming, which loads and unloads it automatically based on camera distance — the same way it already handles the surrounding streamed tile geometry. It can stream either one whole Gaussian file or a progressive .untoldgs tier set.

API overview

setEntityGaussianStreaming(
    entityId: EntityID,
    source: GaussianSource,
    options: GaussianStreamingOptions
)

GaussianSource selects what kind of Gaussian asset the streaming system should load:

.single(filename: String, withExtension: String)

.progressive(
    baseFilename: String,
    withExtension: String = "untoldgs",
    levelCount: Int,
    maxDistances: [Float]
)

GaussianStreamingOptions controls the entity's streaming behavior:

GaussianStreamingOptions(
    streamingRadius: Float = 100.0,
    unloadRadius: Float = 150.0,
    boundingBoxHalfExtent: simd_float3? = nil,
    priority: Int = 0
)

Prerequisites

This only makes sense in a scene that is already using tile-based streaming — i.e. one loaded with setEntityStreamScene (see Using the Geometry Streaming System). setEntityGaussianStreaming attaches the splat to whichever tile stub's bounds contain the entity's position, so it needs those tile stubs to already exist. Call it after setEntityStreamScene's completion handler has fired — tile stubs are guaranteed to be registered by then.

Step 1: Create and position the entity

Position and orient the entity before registering it for streaming — the position at the time you call setEntityGaussianStreaming is what determines which tile it gets attached to.

let streamSplat = createEntity()
translateTo(entityId: streamSplat, position: simd_float3(2.0, 0.0, -4.0))
rotateTo(entityId: streamSplat, angle: 180.0, axis: simd_float3(1.0, 0.0, 0.0))

Step 2: Register it for streaming

setEntityGaussianStreaming(
    entityId: streamSplat,
    source: .single(filename: "chair", withExtension: "untoldgs"),
    options: GaussianStreamingOptions(
        streamingRadius: 30.0,
        unloadRadius: 45.0
    )
)

Parameters:

  • entityId: The entity created and positioned in Step 1.
  • source: Use .single(filename:withExtension:) for a whole .ply/.untoldgs asset, or .progressive(baseFilename:levelCount:maxDistances:) for progressive tiers named <baseFilename>_lod0.untoldgs, <baseFilename>_lod1.untoldgs, etc.
  • streamingRadius: Distance from the camera at which the splat starts loading.
  • unloadRadius: Distance beyond which the splat unloads. Should be larger than streamingRadius to avoid load/unload thrashing at the boundary.
  • boundingBoxHalfExtent: Optional local-space half-extent for the entity, roughly matching the splat's real-world size. GeometryStreamingSystem's frustum gate needs a real local-space volume on the entity before it ever loads, so a .untoldgs source's box is read from its baked header synchronously at registration time when this is left nil — no value needed for that case. A raw .ply source has no baked header, so this must be supplied explicitly there — omitting it leaves the entity non-streaming (logged as a warning) rather than registering a zero-size placeholder, which would collapse the frustum gate to a single exact point and make re-streaming unreliable once the camera moves away and back. When you do need one, the exporter's printed boundingBoxHalfExtent diagnostic (see above) is a good starting value.
  • priority: Optional. Higher-priority entities load first when multiple candidates are in range at once. Defaults to 0.

Note: If no tile is found containing the entity's position, setEntityGaussianStreaming logs a warning and leaves the entity as a plain, non-streaming entity (no StreamingComponent is attached) — it will not crash, but it also will not load. Double-check the position against the streamed scene's tile bounds if this happens.

Progressive Gaussian splat streaming

Use .progressive(...) with setEntityGaussianStreaming when you want tile-driven load/unload behavior plus the same coarse-to-fine refinement (including the overdraw-aware LOD clamp) described above.

Progressive tier filenames must follow this pattern:

<baseFilename>_lod0.untoldgs
<baseFilename>_lod1.untoldgs
<baseFilename>_lod2.untoldgs
...

For example, if baseFilename is "chair" and levelCount is 4, the engine expects:

chair_lod0.untoldgs
chair_lod1.untoldgs
chair_lod2.untoldgs
chair_lod3.untoldgs

setEntityGaussianStreaming(
    entityId: streamSplat,
    source: .progressive(
        baseFilename: "chair",
        levelCount: 4,
        maxDistances: [5.0, 15.0, 25.0, .greatestFiniteMagnitude]
    ),
    options: GaussianStreamingOptions(
        streamingRadius: 30.0,
        unloadRadius: 45.0
    )
)

.untoldgs progressive tiers always have a baked header, so boundingBoxHalfExtent can be omitted here the same way it can for .single(...) with a .untoldgs file.

Putting it together: stream scene + streaming splat

setEntityGaussianStreaming needs the tile stubs setEntityStreamScene creates (see Prerequisites above), so the natural place to register streaming splat props is inside the same completion handler that loads the streamed tile scene:

let sceneRoot = createEntity()
setEntityStreamScene(entityId: sceneRoot, manifest: "dungeon", withExtension: "json") { success in
    guard success else {
        setSceneReady(false)
        return
    }

    let splat = createEntity()
    translateTo(entityId: splat, position: simd_float3(2.0, 0.0, -4.0))
    rotateBy(entityId: splat, angle: 180.0, axis: simd_float3(1.0, 0.0, 0.0))

    setEntityGaussianStreaming(
        entityId: splat,
        source: .progressive(
            baseFilename: "pooltable",
            levelCount: 4,
            maxDistances: [15.0, 25.0, 35.0, .greatestFiniteMagnitude]
        ),
        options: GaussianStreamingOptions(
            streamingRadius: 100.0,
            unloadRadius: 140.0
        )
    )

    setSceneReady(true)
}

boundingBoxHalfExtent is omitted from GaussianStreamingOptions here since pooltable is a .untoldgs progressive asset — its box comes from the baked header automatically (see API overview above). Guarding on success before registering the splat and calling setSceneReady matters: without it, a failed scene load would still try to attach a streaming prop to tile stubs that were never created, and would report the scene ready when it isn't.

Which function should I use?

Function Resident/Streamed LOD Use when
setEntityGaussian(entityId:filename:withExtension:) Resident, loads immediately (blocks) None A small number of splats that should always be visible (a hero object, a standalone demo scene).
setEntityGaussian(entityId:source:) Resident None (.single) or progressive (.progressive) Same as above, plus a single call site that can also take .progressive(...) for coarse-to-fine refinement without a tile-streamed scene.
setEntityGaussianAsync Resident, loads off-thread None Same as setEntityGaussian, but avoids a frame hitch on a large .ply.
setEntityGaussianStreaming(source:options:) Streamed via GeometryStreamingSystem None (.single) or progressive (.progressive) Props scattered across a tile-streamed scene that should load/unload with camera distance.

All progressive paths (setEntityGaussian(source: .progressive(...)) and setEntityGaussianStreaming(source: .progressive(...), options:)) share the same overdraw-aware LOD selection behavior automatically.