ReLightGS

1. Gaussian G-buffer construction. We extend 3DGS rasterisation so each Gaussian accumulates colour, coverage, depth/normal cues, blended material parameters, and a dominant Gaussian ID into screen-space buffers.
2. Surface reconstruction. Per-pixel depth and normals are reconstructed from the splats, defaulting to iso-opacity depth and max-density-plane normals.
3. Material estimation. We segment rendered views with Dense Material Segmentation (DMS), lift the labels to Gaussians via the Gaussian-ID buffer, and map each class to approximate glTF metallic-roughness PBR parameters.
4. Visibility & occlusion. Shadows are approximated with shadow-depth maps (PCF), and indirect attenuation with screen-space ambient occlusion.
5. Deferred lighting. Each pixel is shaded from the G-buffer with a Cook–Torrance GGX BRDF under inserted point, spot, directional, and environment lights.

ReLightGS is implemented in Vulkan by extending the 3DGS.cpp renderer, and runs interactively on a laptop NVIDIA RTX 3060.

A 3DGS scene is a soft volumetric density, not an explicit surface, so the exported .ply has no per-pixel depth or normals. For depth we compare three estimates:

• Weighted depth. Averages the depths of all visible Gaussians based on how much each one contributes to the pixel.
• Projected-plane depth. Treats each Gaussian footprint like a small local plane, so depth can change across the splat (following RaDe-GS [Zhang 2026]).
• Iso-opacity depth. Chooses the first depth where accumulated opacity reaches a threshold (τ = 0.7), following NVIDIA's method [NVIDIA].

We use iso-opacity depth because it produces sharper surfaces and reduces floating artifacts.

Normals are estimated from the depth buffer or the Gaussian geometry, comparing four estimators:

• Cross-product normals. Compute normals from neighbouring depth pixels, but can be noisy near edges.
• Plane-fit normals. Fit a small local plane to nearby depth points, making them smoother.
• Projected-plane normals. Use the local depth plane of each Gaussian, giving detailed variation but can have artifacts.
• Max-density plane normals. Compute normals directly from each Gaussian's covariance and camera direction [Kheradmand 2025][NVIDIA].

We use max-density plane normals because they are more coherent and less affected by depth-buffer noise, making them more stable.

We use Dense Material Segmentation (DMS) [Upchurch 2022], an AI-based material-segmentation model, to predict 2D material labels from rendered views (auxiliary views on a hemisphere around the scene: three elevations and eight azimuths, plus a top-down view). These labels are assigned to individual Gaussians using the Gaussian-ID buffer, then mapped to approximate Physically Based Rendering (PBR) parameters: diffuse, specular, roughness, and metallic.

Direct-light visibility is approximated with shadow-depth maps, and local indirect attenuation with screen-space ambient occlusion (SSAO) [Mittring 2007] computed from the reconstructed depth and normal buffers.

The final pass combines all estimated cues from the Gaussian G-buffer: depth, normals, colour, materials, shadows, and ambient occlusion. Using these buffers, ReLightGS applies point, spot, directional, and environment lighting (with a Cook–Torrance GGX BRDF) to produce the final relit result.

Qualitative comparison of reconstructed normals on the ficus and lego scenes. TensoIR stays closest to the ground truth and GS-IR shows visible artifacts and normal-direction errors, while ReLightGS preserves surface orientation more consistently than GS-IR, though its normals appear thicker around fine detail.

We relight the fixed Gaussian scenes under HDR environment maps. ReLightGS produces reasonable relighting directly from the fixed representation, though illumination baked into the original Gaussian colours still influences the final appearance. The overview below shows all four scenes under a city environment; the per-method comparison further down focuses on the lego scene against inverse-rendering baselines and the ground truth.

We evaluate on the four object-centric TensoIR Synthetic scenes (armadillo, ficus, hotdog, lego), with relighting metrics averaged over five HDR environment maps (bridge, city, fireplace, forest, night). The scenes are vanilla 3DGS models trained for 30,000 iterations, and baseline numbers are adapted from the GS-IR evaluation. For this environment-map benchmark the map intensity is set to 5.0, and screen-space ambient occlusion and shadow mapping are disabled, since it uses only environment lighting. Despite operating post-hoc on a fixed vanilla 3DGS representation, with no access to the original images and no retraining, ReLightGS improves over the Gaussian-based GS-IR baseline on every metric.

Assigning approximate PBR parameters from lifted material labels (vs. a single default material) improves environment-map relighting while leaving geometry untouched (Normal MAE is unchanged, since labels do not affect normal estimation).

We also test the effect qualitatively on the real bicycle scene from Mip-NeRF 360. Even when the non-labeled variant is given a stronger default material, the labeled variant produces more material-dependent variation, for example a more appropriate response on the bicycle tires.

Although our method can produce plausible relighting from fixed Gaussian scenes, several limitations remain. First, the reconstructed normals can become overly smooth. The armadillo normal map preserves the overall surface orientation, but fine geometric details are partly smoothed out compared with the ground truth and inverse-rendering baselines. Second, the original Gaussian colours can still contain baked illumination, since we use the spherical-harmonics appearance as albedo. In the ficus example, a yellowish tint remains in parts of the plant even though the target ground truth has a different lighting balance. Finally, material-label assignment is approximate, so some Gaussians may receive incorrect material classes, introducing variations that may not match the true scene.

• Relighting without retraining is feasible. Useful geometry, material, and visibility cues can be recovered from a fixed exported .ply Gaussian model and used for plausible, interactive relighting.
• Competitive with inverse rendering. ReLightGS improves over the Gaussian-based GS-IR baseline on Normal MAE, PSNR, SSIM, and LPIPS, despite never seeing the original multi-view images.
• Material labels help. Lifting DMS labels to Gaussians and mapping them to PBR parameters gives a clear quality gain over a single default material.
• Limitations. Reconstructed normals can be overly smooth, baked illumination remains in the spherical-harmonic appearance used as albedo, and approximate label assignment can introduce local material errors.