[BUG] Splat positions rendering differently after compression

[vincentwoo]

I have a splat that, when compressed by supersplat, mostly looks great except for (as far as I can tell) one specific area of interest, wherein the means of the splats themselves seem to have moved. Here's a GIF comparison:

Here I'm toggling between the compressed/uncompressed plys in a scene in SuperSplat with both loaded. The tips of the towers seem to "detach" in the compressed ply.

Here are the files if you want to try to repro/inspect more closely:

https://sutro-tower.netlify.app/scene.ply
https://sutro-tower.netlify.app/scene.compressed.ply
and you can check out the viewer I rigged up at https://sutro-tower.netlify.app itself.

(I erroneously reported previously that this issue only happened in the viewer, please ignore that version of the issue)

[marklundin]

Love this scan btw 😍

[vincentwoo]

Thank you! I hope to build more of an interactive experience around it using playcanvas! Let me know if you have any bandwidth to help, I am an extreme novice in this kind of programming (just barely getting acquainted with the playcanvas editor platform).

[simonbethke]

This is really weird. I looked into the compressed splat a bit but couldn't find a clue. Maybe it is related to some precision loss/rounding. Have you tried rescaling the scene a bit and compressing it? Will it create the same artifacts?

[vincentwoo]

Yes, I tried rescaling from supersplat and exporting the compressed ply both at 1/10th and 10x scale. At 1/10th there was much worse artifacting near the top (all geometry appeared weirdly thin), and at 10x it was the same.

[vincentwoo]

Also maybe interesting - if I cut just that part of the model and save it separately it does not seem to exhibit the problem. It seems like it arises in situ with the rest of the splats too. Maybe to do with the way the compression chunks the scene? I do also notice some issues in other less prominent areas, like trees.

[slimbuck]

Hi @vincentwoo ,

I think I know what the issue might be, but will need some time to investigate.

The compression works by:

• ordering all splats into morton order (so splats close together in world space are ordered together)
• for every chunk of 256 splats:
  • calculate position/scale scale bounds
  • normalise the position and scale based on bounds and store quantised value per splat

There are a few possibilities that might explain the issue you're seeing:

• we calculate morton order based on a (1024, 1024, 1024) integer space. Its possible that on this huge scene too many splats are falling into a single morton order bin.
  • the solution is to detect this case and reorder the binned splats again to get good bounds
• since we just blindly group 256 splats, it's possible you're seeing an unfortunate case where splats far away are being grouped together.
  • solution is to introduce some threshold and pad chunks if they contain points too far away from the rest.

Sorry I hope some of that makes sense. Basically the compress stage needs some love.

Thanks!

[vincentwoo]

No, that makes perfect sense. Basically for the morton order issue you would try a different hilbert curve through the ordering space, with the aim of minimizing the bounds for each cube defined by 256 splats? Sounds like a fun puzzle, if you point me at the code for that bit I'd love to take a look.

[slimbuck]

Awesome! Here's the sorter https://github.com/playcanvas/supersplat/blob/main/src/splat-serialize.ts#L569

[vincentwoo]

        const Part1By2 = (x: number) => {
            x &= 0x000003ff;
            x = (x ^ (x << 16)) & 0xff0000ff;
            x = (x ^ (x <<  8)) & 0x0300f00f;
            x = (x ^ (x <<  4)) & 0x030c30c3;
            x = (x ^ (x <<  2)) & 0x09249249;
            return x;
        };

hahaha, should be a good time

[vincentwoo]

Playing with it a bit:

• I tried enhancing the precision of the morton order (using 17 bits of the keys instead of 8) but didn't seem to help
• Poking around at the chunks, I removed the top 1% of chunks just by hypotenuse of the cube extents, and found that the troublesome splats were indeed in one of those chunks. Bit tricky because sometimes imprecision in a large chunk is okay (background object) and sometimes its not.

Splitting (padding) those chunks makes sense to alleviate this but I think it would require an engine change too, no? You'd need variable length chunks or a way to communicate that the chunk has dead space. I think that one might be a bit beyond me.

[slimbuck]

I wrote a node CLI tool for creating compressed.ply yesterday and noticed a bug which might explain this.

The very last chunk in the scene is almost always part-full (since chunks are 256 in size) and I realised that the remaining out-of-bounds gaussians in the last chunk will keep the values from the preceding chunk.

The out-of-bounds gaussians do not get rendered or used at runtime, but their presence in the last chunk will impact the calculated bounds, which may result in lower-than-necessary precision for the remaining gaussians.

I think it's a long shot, but I will fix this bug today and retest.

[vincentwoo]

Interesting. I just now tried to try writing out the compressed ply leaving off the last chunk, but still saw my oddly placed gaussians there.

[slimbuck]

Yep didn't fix anything.

The largest buckets in the scene are bigger than I thought:

  96047, 57649, 39528, 27071, 22609, 21627, 15920, 11865, 11340,
  11293, 11120, 10662, 10303, 10276,  9984,  9983,  9604,  9470,
   9254,  8922,  8351,  8158,  8027,  7871,  7602,  7238,  7232,
   7067,  6900,  6899,  6889,  6810,  6714,  6504,  6443,  6035,
   6032,  5882,  5787,  5674,  5623,  5609,  5594,  5503,  5461,
   5414,  5273,  5244,  5205,  5129,  5042,  5002,  4832,  4752,
   4697,  4684,  4634,  4467,  4393,  4244,  4206,  4166,  4163,
   4156,  4153,  4121,  4121,  4041,  4031,  3982,  3978,  3978,
   3964,  3948,  3940,  3926,  3903,  3852,  3737,  3718,  3717,
   3549,  3545,  3532,  3531,  3519,  3517,  3433,  3416,  3369,
   3280,  3261,  3172,  3150,  3137,  3136,  3105,  3052,  3038,

So I'm going to try recursively performing morton order on buckets larger than n next.

[slimbuck]

Recursively ordering buckets does nothing good...

[vincentwoo]

I wonder if it would be helpful to like color the splats by the chunk they came in with? There must be something strange going on with that particular bucket

[vincentwoo]

Yeah, looks like the tips are sort of in-plane with some very far away splats that just happen to be on the morton curve coming in, stretching the XZ quantization for that chunk:

I suppose this is one downside of having a splat that is too free of aerial floaters - the morton curve may have to jump over a lot of empty space

[simonbethke]

But I think especially the green splats are all very close together and still off. Maybe you can create something like a percentile ~80 bounding box and look for splats that are magnitudes outside that box to eliminate them (compress them separately).

Edit: Or implement some logarithmic position encoding starting from the median position of a bounding box.

[slimbuck]

BTW we could be padding chunks with alpha 0 gaussians in these situations.

[vincentwoo]

I tried throwing a hilbert curve at it after banging my head on typescript for a half hour but it turns out pretty much exactly the same fwiw.