Galin progress
Index

The WorkshopNo. 05

Year Status: On the bench

Worldmaker

Describe a place, paste a scene from a book or drop in a photo, and it builds a 3D world you can walk around in.

My part: The idea, the design and the build.

Stack
  • Python
  • ComfyUI
  • Gaussian splatting
  • Three.js
  • Unreal Engine 5
Fields

ΔΟΣ ΜΟΙ ΠΑ ΣΤΩ ΚΑΙ ΤΑΝ ΓΑΝ ΚΙΝΑΣΩ

“Give me a place to stand, and I will move the earth.”Archimedes, as told by Pappus
Render · in preparation
Fig. β · Render
Fig. α · Sketch

Schematic

How it works

Fig. γ · Tav. V · How it works

How Worldmaker works, step by step:

  • Words, a photo or a 360° (input) to 8K panorama
  • 8K panorama to You check it
  • You check it to Drone flights: looks good
  • Drone flights to Gaussian splat: fill the unseen
  • Gaussian splat to Walk around (output)
  • Gaussian splat to VR (output)
  • Gaussian splat to Unreal Engine (output)

Note on Drone flights: Runs offline, on one graphics card

The idea

Text in, world out. The main use is fantasy worlds from books: paste a passage, and Worldmaker picks out the place and writes the scene for you. A photo or a 360° panorama works too.

The build

A build starts with an 8K 360° panorama of the place, and pauses there so you can look around it before the long work begins. From the panorama’s depth, Worldmaker plans drone flights along the most open directions, and a video model fills in what the panorama couldn’t see. The flights become training pictures for a Gaussian splat: the world, as millions of small soft points.

Then you walk around it in the app, in VR, or in Unreal Engine 5 with an invisible floor to walk on. Everything runs locally, on one NVIDIA graphics card.