We propose a method to train an autonomous agent to learn to accumulate a 3D scene graph representation of its […]
Collaborative human robot autonomy publications
SASRA: Semantically-aware Spatio-temporal Reasoning Agent for Vision-and-Language Navigation in Continuous Environments
This paper presents a novel approach for the Vision-and-Language Navigation (VLN) task in continuous 3D environments.
By using our novel attention schema and auxiliary rewards to better utilize scene semantics, we outperform multiple baselines trained with only raw inputs or implicit semantic information while operating with an 80% decrease in the agent’s experience.
This paper proposes a real-time navigation approach that is able to integrate many sensor types while fulfilling performance needs and system constraints. Our approach uses a plug-and-play factor graph framework, which extends factor graph formulation to encode sensor measurements with different frequencies, latencies, and noise distributions. It provides a flexible foundation for plug-and-play sensing, and can incorporate new evolving sensors. A novel constrained optimal selection mechanism is presented to identify the optimal subset of active sensors to use, during initialization and when any sensor condition changes. This mechanism constructs candidate subsets of sensors based on heuristic rules and a ternary tree expansion algorithm. It quickly decides the optimal subset among candidates by maximizing observability coverage on state variables, while satisfying resource constraints and accuracy demands. Experimental results demonstrate that our approach selects subsets of sensors to provide satisfactory navigation solutions under various conditions, on large-scale real data sets using many sensors.
We present an innovative path following system based upon multi-camera visual odometry and visual landmark matching. This technology enables reliable mobile robot navigation in real world scenarios including GPS-denied environments both indoors and outdoors. We recover paths in full 3D, making it applicable to both on and off-road ground vehicles. Our controller relies on pose updates from visual odometry, allowing us to achieve path following even when only a joystick drive interface to the base robot platform is available. We experimentally investigate two specific applications of our technology to autonomous navigation on ground vehicles – non line-of-sight leader-following (between heterogeneous platforms) and retro-traverse to home base. For safety and reliability we add dynamic short range obstacle detection and reactive avoidance capabilities to our controller. We show the results for end-to-end real time implementation of this technology using current off-the-shelf computing and network resources in challenging environments.