[FIX] can_bus input of quaternion mistakenly get all 4 number as same number

[yjmade]

Issue

A subtle bug was discovered in the NuScenes dataset loader where the canbus input for the BEV head incorrectly assigns quaternion values for ego vehicle rotation. Specifically, when assigning the quaternion (positions 4-8), all components receive the same value as rotation.w.

Reproduction Example

In [1]: import pyquaternion

In [2]: import numpy as np
   ...: 

In [3]: q=pyquaternion.Quaternion([1,0,0,0])

In [4]: b=np.zeros([10])

In [5]: b
Out[5]: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])

In [6]: b[3:7]=q

In [7]: b
Out[7]: array([0., 0., 0., 1., 1., 1., 1., 0., 0., 0.])

Fix

The solution is to use the quaternion's underlying numpy array for assignment:

In [8]: b[3:7]=q.q

In [9]: b
Out[9]: array([0., 0., 0., 1., 0., 0., 0., 0., 0., 0.])

Why the Bug Remained Undetected

Two main factors contributed to the delayed discovery:

1. The canbus data includes yaw angles (in degrees and radians) as the last two values, which are the most critical values for planning within rotation
2. Since all dataset items contained the same error, models overfit to this incorrect data pattern

This allowed the model to still function in open-loop evaluation scenarios where the same dataloader was used.

Discovery Process

We tried to integrate the UniAD model into our simulation pipeline but faced an issue: no matter how we supplied the rotation data from the simulator, the model couldn't generate the correct turning trajectory. To troubleshoot, we inspected the training pipeline line by line and discovered the problem. We confirmed that the UniAD model behaves correctly only when we input 'w, w, w, w' as the rotation in the canbus during simulation.

Impact

We conclude that this issue will not affact the open-loop evaluation. But if people try to use the model in the close-loop environments, like simualtion or real vehicle, it will show up.

We traced the source of this issue, and it's first appearance is well-adopted BEVFormer code base. And this issue still existed there. Which means all work derived from BEVFormer share the same problem.

Required Actions

1. Existing checkpoints will not work correctly
2. Models need to be retrained with the fixed dataloader
3. Systems using this in closed-loop environments need to be updated

[justeWang]

Issue

A subtle bug was discovered in the NuScenes dataset loader where the canbus input for the BEV head incorrectly assigns quaternion values for ego vehicle rotation. Specifically, when assigning the quaternion (positions 4-8), all components receive the same value as rotation.w.

Reproduction Example

In [1]: import pyquaternion

In [2]: import numpy as np
   ...: 

In [3]: q=pyquaternion.Quaternion([1,0,0,0])

In [4]: b=np.zeros([10])

In [5]: b
Out[5]: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])

In [6]: b[3:7]=q

In [7]: b
Out[7]: array([0., 0., 0., 1., 1., 1., 1., 0., 0., 0.])

Fix

The solution is to use the quaternion's underlying numpy array for assignment:

In [8]: b[3:7]=q.q

In [9]: b
Out[9]: array([0., 0., 0., 1., 0., 0., 0., 0., 0., 0.])

Why the Bug Remained Undetected

Two main factors contributed to the delayed discovery:

1. The canbus data includes yaw angles (in degrees and radians) as the last two values, which are the most critical values for planning within rotation
2. Since all dataset items contained the same error, models overfit to this incorrect data pattern

This allowed the model to still function in open-loop evaluation scenarios where the same dataloader was used.

Discovery Process

We tried to integrate the UniAD model into our simulation pipeline but faced an issue: no matter how we supplied the rotation data from the simulator, the model couldn't generate the correct turning trajectory. To troubleshoot, we inspected the training pipeline line by line and discovered the problem. We confirmed that the UniAD model behaves correctly only when we input 'w, w, w, w' as the rotation in the canbus during simulation.

Impact

We conclude that this issue will not affact the open-loop evaluation. But if people try to use the model in the close-loop environments, like simualtion or real vehicle, it will show up.

We traced the source of this issue, and it's first appearance is well-adopted BEVFormer code base. And this issue still existed there. Which means all work derived from BEVFormer share the same problem.

Required Actions

1. Existing checkpoints will not work correctly
2. Models need to be retrained with the fixed dataloader
3. Systems using this in closed-loop environments need to be updated

您好，我正在尝试将uniad接入到carla或其他仿真器中，但uniad和nuscenes高度耦合，一直没找到方法，请问您是怎么接入到仿真器的？uniad的输入数据都需要什么？
Hello, I am trying to connect uniad to carla or other emulators, but uniad and nuscenes are highly coupled, and I have not found a method. May I ask how you connect to the emulator? What is required for uniad's input data?

[justeWang]

Issue

A subtle bug was discovered in the NuScenes dataset loader where the canbus input for the BEV head incorrectly assigns quaternion values for ego vehicle rotation. Specifically, when assigning the quaternion (positions 4-8), all components receive the same value as rotation.w.

Reproduction Example

In [1]: import pyquaternion

In [2]: import numpy as np
   ...: 

In [3]: q=pyquaternion.Quaternion([1,0,0,0])

In [4]: b=np.zeros([10])

In [5]: b
Out[5]: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])

In [6]: b[3:7]=q

In [7]: b
Out[7]: array([0., 0., 0., 1., 1., 1., 1., 0., 0., 0.])

Fix

The solution is to use the quaternion's underlying numpy array for assignment:

In [8]: b[3:7]=q.q

In [9]: b
Out[9]: array([0., 0., 0., 1., 0., 0., 0., 0., 0., 0.])

Why the Bug Remained Undetected

Two main factors contributed to the delayed discovery:

1. The canbus data includes yaw angles (in degrees and radians) as the last two values, which are the most critical values for planning within rotation
2. Since all dataset items contained the same error, models overfit to this incorrect data pattern

This allowed the model to still function in open-loop evaluation scenarios where the same dataloader was used.

Discovery Process

We tried to integrate the UniAD model into our simulation pipeline but faced an issue: no matter how we supplied the rotation data from the simulator, the model couldn't generate the correct turning trajectory. To troubleshoot, we inspected the training pipeline line by line and discovered the problem. We confirmed that the UniAD model behaves correctly only when we input 'w, w, w, w' as the rotation in the canbus during simulation.

Impact

We conclude that this issue will not affact the open-loop evaluation. But if people try to use the model in the close-loop environments, like simualtion or real vehicle, it will show up.

We traced the source of this issue, and it's first appearance is well-adopted BEVFormer code base. And this issue still existed there. Which means all work derived from BEVFormer share the same problem.

Required Actions

1. Existing checkpoints will not work correctly
2. Models need to be retrained with the fixed dataloader
3. Systems using this in closed-loop environments need to be updated

可以给一些建议吗？
Can you give me some advice?

[yjmade]

您好，我正在尝试将uniad接入到carla或其他仿真器中，但uniad和nuscenes高度耦合，一直没找到方法，请问您是怎么接入到仿真器的？uniad的输入数据都需要什么？ Hello, I am trying to connect uniad to carla or other emulators, but uniad and nuscenes are highly coupled, and I have not found a method. May I ask how you connect to the emulator? What is required for uniad's input data?

We rewrote the model's forward function, remove all elements from nusense and also all GT related to loss/metrics.