Gaussian Splatting with Localized Points Management

Visualization of points behavior. 3DGS produces ill-conditioned Gaussians (red box) that occlude other valid points, resulting in noticeably incorrect depth estimation. LPM handles these ill-conditioned points to reduce negative impacts and further calibrate the geometry.

Point management is a critical component in optimizing 3D Gaussian Splatting (3DGS) models, as the point initiation (e.g., via structure from motion) is distributionally inappropriate. Typically, the Adaptive Density Control (ADC) algorithm is applied, leveraging view-averaged gradient magnitude thresholding for point densification, opacity thresholding for pruning, and regular all-points opacity reset. However, we reveal that this strategy is limited in tackling intricate/special image regions (e.g., transparent) as it is unable to identify all the 3D zones that require point densification, and lacking an appropriate mechanism to handle the ill-conditioned points with negative impacts (occlusion due to false high opacity). To address these limitations, we propose a Localized Point Management (LPM) strategy, capable of identifying those error-contributing zones in the highest demand for both point addition and geometry calibration. Zone identification is achieved by leveraging the underlying multiview geometry constraints, with the guidance of image rendering errors. We apply point densification in the identified zone, whilst resetting the opacity of those points residing in front of these regions so that a new opportunity is created to correct ill-conditioned points. Serving as a versatile plugin, LPM can be seamlessly integrated into existing 3D Gaussian Splatting models. Experimental evaluation across both static 3D and dynamic 4D scenes validate the efficacy of our LPM strategy in boosting a variety of existing 3DGS models both quantitatively and qualitatively. Notably, LPM improves both vanilla 3DGS and SpaceTimeGS to achieve state-of-the-art rendering quality while retaining real-time speeds, outperforming on challenging datasets such as Tanks & Temples and the Neural 3D Video Dataset.

Overview of the Localized Point Management (LPM). (a) We start with an image rendering error map against the current view (the ground-truth). Concurrently, matching points are identified between the current view and a refereed view sampled as an adjacent view via off-the-shelf feature mapping. (b)Subsequently, cross-view region mapping is then employed to locate the correspondence region in the refereed view. (c)For each pair of corresponded regions, we cast the rays through them at their respective camera views in the cone shape, and consider their intersection as the error source zone. The final step involves identifying under-optimized or ill-conditioned points within these zones, where under-optimized/empty places are densified, ill-conditioned points are reset.

Comparison of various methods across different scenes on the Mip-NeRF 360 dataset, Tanks&Temples and Deep Blending. 3D* indicates the retrained model from the official implementation. Bold represents best, underline indicates second best.

 

Quantitative comparisons on the Neural 3D Video dataset. “FPS” is measured at a resolution of 1352 × 1014. Some methods only report results for a subset of scenes. For a fair comparison, we report LPM’s results under two pre-existing settings. Only includes the Flame Salmon scene. Bold represents best, underline indicates second best.