Sparse3DTrack: Monocular 3D Object Tracking Using Sparse Supervision

Monocular 3D object tracking aims to estimate temporally consistent 3D object poses across video frames, enabling autonomous agents to reason about scene dynamics. However, existing state-of-the-art approaches are fully supervised and rely on dense 3D annotations over long video sequences, which are expensive to obtain and difficult to scale. In this work, we address this fundamental limitation by proposing the first sparsely supervised framework for monocular 3D object tracking. Our approach decomposes the task into two sequential sub-problems: 2D query matching and 3D geometry estimation. Both components leverage the spatio-temporal consistency of image sequences to augment a sparse set of labeled samples and learn rich 2D and 3D representations of the scene. Leveraging these learned cues, our model automatically generates high-quality 3D pseudolabels across entire videos, effectively transforming sparse supervision into dense 3D track annotations. This enables existing fully-supervised trackers to effectively operate under extreme label sparsity. Extensive experiments on the KITTI and nuScenes datasets demonstrate that our method significantly improves tracking performance, achieving an improvement of up to 15.50 p.p. while using at most four ground truth annotations per track.

Find out more about our state-of-the-art sparse supervision framework in the Approach section!

The core idea of our Sparse3DTrack framework is to augment a small number of human annotations with rich spatio-temporal context from image sequences to generate accurate 3D pseudolabels, thereby transforming sparse supervision into dense annotations. Our framework decomposes the overall problem into two sequential sub-tasks, 2D query matching and 3D geometry estimation, as shown in the figure above. The 2D query matching module establishes object correspondences across frames by measuring feature similarity between a query region and target images. To mitigate label sparsity, it incorporates a data mining mechanism that exploits the spatio-temporal consistency of unlabeled neighboring frames to automatically generate additional training samples. This enables robust object localization despite significant variations in appearance, illumination, and viewpoint. Building on these localizations, the 3D geometry estimation module regresses the object's 3D position by fusing these matched features with depth priors from a self-supervised depth network. It also addresses the ill-posed 3D yaw estimation problem by decomposing orientation prediction into keypoint regression and depth reasoning, thus enabling accurate 3D pose recovery even under extreme label sparsity. By learning to independently reason about each sub-task and subsequently fusing their outputs, our framework produces accurate and temporally consistent 3D track pseudolabels from only a handful of human annotations.

Our framework encapsulates four key components, namely:

1. a pretrained DINOv2 (ViT-b) image encoder to extract features from the input image;
2. a transformer-based 2D query matching module to localize the query object in the target image using feature similarity between the template and target images;
3. a convolution-based self-supervised depth estimation network to generate intermediate depth features;
4. a transformer-based 3D geometry estimation module to fuse the target object and depth features, and subsequently regress the 3D object pose and size estimates;

We tackle extreme label sparsity by designing a data augmentation pipeline to generate rich training samples from limited labels. Specifically, we mine unique source and target pairs using four strategies: (a) self matching, (b) support matching, (c) cycle matching, and (d) step-support matching as shown in the figure above. The self-matching strategy uses a single labeled image \(I_0\) as both the source and target. Matching a query object to itself provides an optimal training signal to the similarity estimation module, enabling the network to learn perfect feature correspondences. In contrast, support matching uses two distinct labeled images, \(I_0\) and \(I_k\) as source and target, respectively. This strategy trains the similarity module to generalize object matching across varying views and appearances. Together, these supervised strategies constrain the network parameters, preventing them from learning incorrect relationships. Cycle matching and step-support matching leverage unlabeled image neighbors to augment network training. These approaches use an unlabeled image as an intermediate waypoint whose pseudolabel prediction is subsequently used to establish an association with a labeled image. To this end, both strategies begin by using a labeled image \(I_0\) as the source to localize the query object in an unlabeled neighbor, which then becomes the new source. For cycle matching, we match the neighbor back to the original source image \(I_0\), teaching the network to maintain appearance consistency across multiple prediction steps. In contrast, step-support matching matches the neighbor to a different labeled image \(I_k\), enhancing the network's ability to maintain long-range correspondence. By using unlabeled intermediate waypoints, we increase the number of training samples and make the network more robust to prediction noise. This skill is key for real-world inference, where tracks are propagated based on prior predictions.

The 3D geometry estimation module \(\mathcal{G}_{3d}\) estimates the 3D pose of the target object using a monocular RGB image. This module lifts the 2D object correspondence obtained from \(\mathcal{M}_{2d}\) into 3D space, thereby enabling 3D object tracking with sparse supervision. However, monocular 3D estimation is fundamentally ill-posed, whose challenge is further exacerbated by extreme label sparsity, which prevents depth reasoning via supervised learning. This module circumvents this limitation by leveraging strong depth priors from a self-supervised depth network alongside sparse annotations to promote 3D understanding.

Monocular 3D yaw estimation is also an ill-posed problem due to the ambiguity in inferring orientation from 2D projection. We address this limitation by reformulating 3D yaw estimation as a function of 3D keypoint regression on the target object. To this end, our network learns to predict the 2D projections and corresponding depth values of the object's front, center, and back points. We also predict a binary direction variable \(d \in \{\text{towards}, \text{away}\}\) which determines whether the object faces towards or away from the ego vehicle, thus resolving the direction ambiguity. This formulation decomposes 3D yaw estimation into 2D projection regression and depth estimation, where the former is well-defined and the latter can be estimated alongside 3D center estimation using self-supervised depth features.