FPS drop when close (especially in VR)

[getrasa]

Hi all,

First, I want to express my appreciation to Ares-p and everyone involved in this fantastic project! It's incredibly useful and easy to integrate.

I'm encountering a significant FPS drop when viewing Gaussian splats up close, similar to the issue described in #10. This issue is particularly pronounced in VR, where users naturally move closer to objects to get a better view. While not always noticeable in desktop applications, in VR the frame rate can plummet from 72fps to as low as 10fps, severely impacting the experience.

Thanks to @ninjamode and their excellent VR-compatible fork (https://github.com/ninjamode/Unity-VR-Gaussian-Splatting), we can now leverage Gaussian splatting in VR environments, including the Quest 3. I initially raised this performance issue in their fork (#5: ninjamode/Unity-VR-Gaussian-Splatting#5), but after investigation, it appears the root cause likely lies within the original implementation.

Therefore, I'm creating this issue here to bring it to your attention. Any insights or potential solutions would be greatly appreciated. Thanks!

[aras-p]

I don't think there's an easy solution to this. Gaussian Splats are very similar to a particle system, conceptually, and if you get really close to a particle system so that there's a ton of particles covering most of the whole screen, it is going to be slow.

I think these days Unity has some features to do "dynamic resolution adjustment", i.e. when things get slow it could drop the internal rendering resolution, until things get to acceptable speed again. But I don't know how exactly to use it, or whether it would work with gaussian splats. You'd have to find someone who knows Unity in depth and is willing to investigate this.

[SaadMirzaBaig]

Using the latest update on the project URP in VR. I don't see my splat in VR at all. Also nothing in Quest Link. I'm using Quest 3S.

[aras-p]

@SaadMirzaBaig please comment on the appropriate issue, which is #17

[yannoid]

@SaadMirzaBaig the last update works with URP on Quest 3 (at least with the editor version i'm using : 6000.0.25f1)

You just have to use OpenXR with Multi-pass and convert a main camera to an XR Rig to make it work, nothing else to change.
I've noticed extending the splat scale helps with the frame rate (but that's not usable since It won't reflect the real model).

So far, the only thing that helped me achieving a good framerate with my scene is changing the scale of a model (like 10 times bigger) and place it further (like 18 meters away from the camera). It's "okish" with a 3D model on a distant turntable, but won't be very usefull in other cases.

[SaadMirzaBaig]

@yannoid thanks. It helped.

[getrasa]

I'm not that knowledgeable about computer graphics but I've observed that not only the number but also size of splats significantly impact performance. My splat models are trained with a cap of 150,000-175,000 splats, resulting in a relatively small number of large, overlapping splats. Since each splat is rendered as a quad, this translates to roughly 300,000-350,000 triangles. While this triangle count isn't excessively high, the large size and significant overlap of the quads put a substantial strain on the fragment shader.

To confirm this, I experimented with splat scaling. Decreasing the scale resulted in a massive performance improvement, jumping from 120 FPS to 350 FPS. Conversely, increasing the scale dropped performance to 25 FPS.

To mitigate this issue, I've tried the following:

• Adjusting the alpha discard threshold: Changing the fragment shader's alpha discard threshold from 1/255 to 2/255 yielded a roughly 10% performance gain with minimal visual impact.
  if (alpha < 2.0/255.0) discard;

• Frame upscaling: Frame upscaling techniques like FSR also provided performance benefits.

I believe the bottleneck for small dense splat models is the fragment shader, not sorting. If anyone has ideas on how to offload fragment shader that'd be great.

[aras-p]

@getrasa yeah, so this is again the typical "same optimization techniques as you'd do for a particle system with very large particles". You mention splat count, but I don't think it is often (or ever?) an issue. A quick test in any scenario: does reducing the render resolution make things run faster? If yes, then the splat count is not an issue (and out of that: sorting is not an issue, and neither is triangle/quad count). The issue in that case is "number of pixels to process", which is a factor of overall render resolution, and of splat size on screen. Upscaling effectively reduces the rendering resolution, so yes that's a very valid optimization approach.

[getrasa]

That makes sense and I guess I've said something rather obvious. Thank you for your input.