IROS 2024 · Panoptic Mapping

DHP-Mapping

A Dense Panoptic Mapping System with Hierarchical World Representation and Label Optimization Techniques

Tianshuai Hu¹, Jianhao Jiao³, Yucheng Xu⁴, Hongji Liu¹, Sheng Wang¹, Ming Liu²*

¹ HKUST · ² HKUST (Guangzhou) · ³ University College London · ⁴ University of Edinburgh * Corresponding author

✦ arXiv ▶ Video ⌥ Code

Maps provide robots with crucial environmental knowledge, thereby enabling them to perform interactive tasks effectively. Easily accessing accurate abstract-to-detailed geometric and semantic concepts from maps is crucial for robots to make informed and efficient decisions.

We propose DHP-Mapping, a dense mapping system that represents the environment as a collection of TSDF submaps, with each submap representing a unique object. The map structure enables hierarchical modeling. It stores the voxel-level geometry and label information into the TSDF and label layer of each submap and maintains the instance-level information via the submap collection.

Voxel-hash structure: position to TSDF voxel and label voxel, with struct definitions

Our system converts sensor data into point segments. (A) A data-association process assigns each segment a submap ID. (B) Segment information is integrated into the assigned submap's TSDF and label layers through ray-tracing. Two modules are proposed to enhance mapping quality. (C) Fusing information among voxels sharing identical spatial information to avoid submap overlapping. (D) A CRF algorithm encourages label consistency among voxels exhibiting similar color and nearby position.

DHP-Mapping pipeline: sensor inputs, spatial submap ID tracker, ray-tracing integration, inter-submap label management, CRF label optimization

Qualitative results

Experiments on indoor simulation (flat) and outdoor real-world data (SemanticKITTI). DHP-Mapping categorizes semantic classes and tracks object IDs more accurately, producing denser, higher-precision maps than prior work.

Live panoptic reconstruction

Extracted mesh (marching cubes)

Fig. 1 — Meshes extracted from the TSDF map. Row 1: reconstruction using TSDF color values; rows 2–5: label maps (colors = object IDs). Compared with Panmap, DHP-Mapping yields more consistent labels and denser, more accurate geometry (columns d–e–f); refinement reduces submap overlap (black circles).

Quantitative results

Higher accuracy across panoptic, semantic, instance and geometry metrics on Semantic-KITTI and flat. Bold = best, underline = 2nd-best.

Dataset	Method	PQ↑	RQ↑	SQ↑	mIoU↑	Acc.↑	mAP.3↑	mAP.4↑	mAP.5↑
Semantic-KITTI	Panmap	0.534	0.804	0.246	0.271	0.679	0.546	0.541	0.532
	DHP w/o refine	0.533	0.678	0.297	0.240	0.678	0.765	0.761	0.752
	DHP	0.568	0.701	0.340	0.263	0.713	0.769	0.765	0.758
flat	Panmap	0.436	0.506	0.274	0.378	0.659	0.512	0.512	0.464
	DHP w/o refine	0.660	0.660	0.357	0.382	0.631	0.690	0.667	0.667
	DHP	0.708	0.787	0.702	0.629	0.854	0.690	0.690	0.690

Tab. 1 — Panoptic, semantic & instance label accuracy vs. SOTA, and the impact of label refinement.

Dataset	Method	mIoU↑	Acc.↑
S-KITTI	Kimera	0.338	0.770
S-KITTI	DHP	0.263	0.713
flat	Kimera	0.670	0.852
flat	DHP	0.629	0.854

Tab. 2 — Semantic label accuracy vs. Kimera.

Dataset	Method	Acc.↓	Comp.↓	C-L1↓	F-Score↑
S-KITTI	Kimera	0.0430	0.1871	0.1151	55.54
	Panmap	0.0688	0.3119	0.1904	9.16
	DHP	0.0506	0.0584	0.0549	56.34
flat	Kimera	0.0076	0.0648	0.0362	89.99
	Panmap	0.0086	0.0763	0.0424	88.11
	DHP	0.0073	0.0658	0.0365	89.75

Tab. 3 — Geometry reconstruction quality vs. SOTA metric-semantic mapping.

Conclusion

We design a dense volumetric mapping system that uses multiple TSDF submaps and panoptic labels to represent the scene hierarchically and holistically, maintaining both voxel-level and submap-level metric and label information. The inter-submap label management module ensures disjoint spatial information per submap, while the label refinement module improves panoptic-label accuracy by exploiting object cohesion and whole-scene context. This hierarchical TSDF-submaps-with-panoptic-labels structure enables high-level interactive tasks and dynamic-environment modeling. In future work, more abstract representations — topological connections between entities and language-based expressions beyond metric and symbolic representations — can be built on top of this structure.

BibTeX

@misc{hu2024dhpmapping,
  title         = {DHP-Mapping: A Dense Panoptic Mapping System with Hierarchical
                   World Representation and Label Optimization Techniques},
  author        = {Tianshuai Hu and Jianhao Jiao and Yucheng Xu and Hongji Liu
                   and Sheng Wang and Ming Liu},
  year          = {2024},
  eprint        = {2403.16880},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO}
}