Diffusion 3D Studio

1 Generate (text → 3D)

Prompt

Seed Steps (500 = best) CFG scale

Guidance (score distillation)

Static build: generation needs the local GPU server (python server.py). Pre-generated runs below are real outputs of that server.

Gallery – click any model to render it

no runs yet

2 Real-time novel-view renderer

no model loaded

drag: orbit · wheel: zoom

auto-rotate splat scale 1.00 background

training snapshot strip — CUDA rasterizer snapshots during SDS optimisation (front · side · back · side)

3 Optimisation monitor

SDS loss ½‖w(t)(ε̂−ε)‖² per step

Gradient norm reaching each Gaussian parameter (log)

Number of Gaussians (densify / prune)

4 Stress test

CUDA Gaussian rasterizer (DreamGaussian renderer, A100)

Forward render along a fixed 360° orbit, 10 warm-up frames excluded, synced timing; peak VRAM via torch.cuda.max_memory_allocated. Gaussian axis = k jittered clones per Gaussian at 512².

FPS vs output resolution

FPS vs Gaussian count (512²)

axis	res	Gaussians	FPS	VRAM MB	notes

Browser WebGL2 viewer (this page)

Same model replicated k× with jitter; FPS is requestAnimationFrame rate (display-capped), GPU ms from EXT_disjoint_timer_query when available, sort ms = worker depth sort.

resolution scale

k	Gaussians	canvas	FPS	GPU ms	sort ms

5 Improving DreamGaussian: guidance comparison (measured)

Same 5 prompts, seed 0, 500 steps, one free A100 per run. CLIP = ViT-L/14 similarity between prompt and 8 views (×100). CSD / NFSD / ISM are added on the same frozen SD 2.1 (scripts/guidance_plus.py); MVDream is upstream's multi-view guidance.

guidance	CLIP	SDS time	total	VRAM	verdict

MVDream step count

steps	CLIP	total	verdict

MVDream at 300, 500, 800 and 1200 steps — MVDream at 300 / 500 / 800 / 1200 steps: 500 is sharpest, more steps over-saturate.

front and back views for each guidance method — Front | back view per method. SDS gives the corgi a second face (Janus); MVDream is coherent and sharp.

6 What is running

Text prompt yencoded once by the frozen SD-2.1-base text encoder
3D Gaussians θ5k random blob → positions, scales, rotations, opacity, colour
Random camera celevation ±30°, full azimuth, radius 2.5
Differentiable rasterizerCUDA tile splatting, x = g(θ, c), 128→512 px
SDSadd noise at annealed t, UNet predicts ε̂ (CFG 100), ∇θ = w(t)(ε̂ − ε)·∂x/∂θ
Adam + densify/prune500 steps; then save PLY, extract textured mesh

Plan vs. reality: Assignment 2 planned an SDS-guided NeRF (volume rendering). This MVP keeps the same diffusion-guided loop but optimises explicit 3D Gaussians with the official DreamGaussian code (Tang et al., ICLR 2024) for speed and feasibility. VSD and TRELLIS are not implemented.