Projects

Autonomous Exploration meets Modern Vision-Language Models (VLMs)

The Goal: To build a robot that doesn't just see a "map", but understands its environment. By combining ROS2 Humble, 2D LiDAR SLAM, and the VILA 2.7B Vision-Language Model, this robot autonomously maps indoor spaces and labels them (e.g., "Kitchen", "Office") for high-level semantic navigation.

Key Capabilities:

• Zero-Intervention Exploration: Using a frontier-based explorer and the Nav2 stack, the robot autonomously drives toward unknown areas to build a complete 2D occupancy grid of any indoor environment.
• Real-Time Semantic Understanding: While mapping, the robot captures image frames and processes them through a quantized VILA 2.7B VLM running on NVIDIA Jetson hardware.
• Spatial Consensus Mapping: Rather than relying on a single capture, the system aggregates multiple AI inferences at specific (X, Y) coordinates to ensure robust room identification.
• Natural Language Navigation: Once mapped, the robot can be sent to specific rooms using human-readable commands (e.g., "Go to the Bedroom"), translating semantic labels into precise navigation waypoints.

Performance Metrics:

• Semantic Accuracy: 80% consistency in room/zone detection across varying lighting conditions.
• Inference Latency: ~700ms per frame - optimized for real-time background processing on embedded edge hardware.
• Exploration Reliability: High-stability mapping with zero-intervention coverage of complex apartment layouts.

How it Works (Technical Deep-Dive):

• Mecanum Drivetrain: Precise 3-DOF movement allows the robot to strafe and pivot in tight spaces, optimized via a custom mecanum_odometry_node.
• Containerized Stack: The entire system - from LiDAR drivers to the VILA model - is managed via Docker Compose, ensuring consistent performance and easy updates on edge hardware.
• Sensor Fusion: Synchronizing LiDAR (for geometry) and Camera (for semantics) requires precise TF2 coordinate transforms to accurately tag the 2D map with AI-generated labels.