• Skip to primary navigation
  • Skip to main content
SRI logo
  • About
    • Press room
    • Our history
  • Expertise
    • Advanced imaging systems
    • Artificial intelligence
    • Biomedical R&D services
    • Biomedical sciences
    • Computer vision
    • Cyber & formal methods
    • Education and learning
    • Innovation strategy and policy
    • National security
    • Ocean & space
    • Quantum
    • Robotics, sensors & devices
    • Speech & natural language
    • Video test & measurement
  • Ventures
  • NSIC
  • Careers
  • Contact
  • 日本支社
Search
Close

2D-3D reasoning and augmented reality

CONTACT US

SRI has a strong portfolio of 2D-3D reasoning. This includes navigation and mapping using 2D and 3D sensors such as video and LIDAR. In recent years, machine learning has significantly improved the semantic understanding of the 2D and 3D data. Incorporating semantics enables a new class of algorithms for navigation, Simultaneous Localization and Mapping (SLAM), geo-registration, wide-area search, augmented reality, data compression, 3D modeling, and surveillance. 

Semantic and GPS-denied navigation

CVT has developed highly efficient low-drift localization and mapping methods that exploit visual and inertial sensors. SRI has supported a large portfolio of programs and spin-offs using this technology. CVT has also incorporated high-level learning-based semantic information (recognition of objects and scene layouts) into dynamic maps and scene graphs, improving accuracy, efficiency, and robustness in our state-of-the-art navigation systems.

Map with color coded route overlays

WEATHER-INVARIANT EMBEDDINGS AND WIDE-AREA IMAGE-BASED SEARCH

Changes in lighting and weather significantly effects vision algorithms. CVT has developed a novel deep-embedding approach to project image data into a high-dimensional feature space with geo-spatial coherence to learn features that are invariant to weather and time of day. CVT processed two million images from thousands of webcams worldwide to learn how a scene changes over time (i.e., across day, night, and seasonal changes) via this approach. These learned embeddings incorporate scene semantics for contextual reasoning, which enables highly reliable image retrieval across extremely large reference image databases.

GEO-REGISTRATION AND AIR-GROUND MATCHING

Geo-registration is the process of matching video to previous geo-reference data sources such as satellite imagery or LIDAR. CVT has worked across multiple government programs to perform high-precision geo-registration with and without GPS for aerial and ground platforms. CVT has also leveraged recent advances in machine learning to extract semantic features that can be matched across large viewpoint variations and changes in sensing modalities.

Long-range, wide-area, augmented reality

CVT has combined the localization and geo-registration methods described above with low-powered, compact, ruggedized hardware to create wide-area augmented reality applications. CVT has extended its augmented reality capabilities to work over multiple square kilometers while in GPS-challenged environments. This also includes long-range 3D occlusion-reasoning for augmented reality applications.

3D scene classification and modeling

CVT has developed extremely robust 3D scene classification methods over the last decade. These methods have now transitioned to Department of Defense (DoD) programs of record and commercially available software packages. Working with the Office of Naval Research (ONR), the U.S. Army and the National Geospatial-Intelligence Agency (NGA), CVT is now developing the next-generation 3D scene-understanding methods using machine learning. These methods incorporate top-down and bottom-up contextual reasoning and human-specified geographic rules within the learning process.

3d-scene-class

3D COMPRESSION

The robust scene-understanding methods have enabled us to re-visit 3D compression methods that are widely available today. By incorporating knowledge about different feature classes (such as ground, building and foliage), we can achieve significantly better bit rates in the compression of 3D data.

Surveillance

CHANGE DETECTION

CVT’s work in change detection supports deployed improvised explosive devices (IEDs). These algorithms look at multiple passes of video data to detect change signatures of buried roadside IEDs. The recent integration of machine learning-based road-detection methods has significantly improved our change detection performance. SRI is developing novel anomaly detection and anomaly-guided change detection methods for next-generation systems. Specifically, CVT is developing a transformer-based joint spatiotemporal model encompassing multiple space and time resolutions. Transformer networks enable retention of properties of various data modalities, namely geography, weather, seasonal variations, knowledge of typical events and activities of interest, thus providing modularity that enables multimodality, scalability, and explainability.

Multi-modal target detection

CVT has developed an end-to-end pipeline that fuses multi-modal data in deep embedding space for specific tasks—such as target detection and recognition—by directly optimizing target metrics and learning the optimal contribution/control of each mode to the results. This pipeline has been applied to different modalities, including electro-optic/infrared images, hyperspectral imaging, and LIDAR/RADAR data. CVT has also incorporated scene-contextual information to further improve performance of the target task.

CONTINUAL LEARNING TO TARGET CLASSIFICATION

CVT created a new approach to continually acquire, fine-tune and transfer knowledge to optimize tasks such as target classification. Our approach advances state-of-the-art transfer learning and continual learning methods to create an in-situ algorithm-training environment to streamline the training of classifiers to new, unknown sensor data in real time.

Our work

  • A new augmented reality system delivers a smoother, more immersive experience
    July 5, 2023

    A new augmented reality system delivers a smoother, more immersive experience

    By combining ground and aerial views with computer-generated elements, users on the ground view a more accurate augmented reality experience.

  • A modern approach to building inspections
    November 22, 2021

    A modern approach to building inspections

    Using augmented reality and mobile technology to reduce construction overhead.

  • 75 Years of Innovation: augmented reality binoculars
    July 28, 2021

    75 Years of Innovation: augmented reality binoculars

    The first mobile, precision, non-jitter, augmented reality binoculars

Recent publications

more +
  • March 27, 2023

    Vision based Navigation using Cross-View Geo-registration for Outdoor Augmented Reality and Navigation Applications

    Rakesh Kumar, Supun Samarasekera, Han-Pang Chiu
  • March 25, 2023

    Cross-View Visual Geo-Localization for Outdoor Augmented Reality

    Han-Pang Chiu, Supun Samarasekera, Rakesh Kumar
  • December 12, 2022

    On auxiliary latitudes

    Charles Karney

Featured publications

August 27, 2021

Long-Range Augmented Reality with Dynamic Occlusion Rendering

SRI authors: Han-Pang Chiu; Supun Samarasekera; Rakesh Kumar

June 28, 2021

Striking the Right Balance: Recall Loss for Semantic Segmentation

SRI author: Han-Pang Chiu

September 12, 2020

RGB2LIDAR: Towards Solving Large-Scale Cross-Modal Visual Localization

SRI authors: Han-Pang Chiu, Supun Samarasekera, Rakesh Kumar

SEPTEMBER 9, 2019
Conference paper

Semantically-Aware Attentive Neural Embeddings for 2D Long-Term Visual Localization

SRI authors: Han-Pang Chiu, Supun Samarasekera, Rakesh “Teddy” Kumar

SEPTEMBER 2, 2019

Multi-Sensor Fusion for Motion Estimation in Visually-Degraded Environments

SRI authors: Han-Pang Chiu, Supun Samarasekera

MARCH 18, 2018

Augmented Reality Driving Using Semantic Geo-Registration

SRI authors: Han-Pang Chiu, Supun Samarasekera, Rakesh “Teddy” Kumar

OCTOBER 16, 2017

Utilizing Semantic Visual Landmarks for Precise Vehicle Navigation

SRI Authors: Han-Pang Chiu, Supun Samarasekera, Rakesh “Teddy” Kumar

NOVEMBER 1, 2016

Sub-Meter Vehicle Navigation Using Efficient Pre-Mapped Visual Landmarks

SRI Authors: Han-Pang Chiu, Supun Samarasekera, Rakesh “Teddy” Kumar

Career call to action image

Work with us

Search jobs

How can we help?

Once you hit send…

We’ll match your inquiry to the person who can best help you.

Expect a response within 48 hours.

Our work

Case studies

Publications

Timeline of innovation

Areas of expertise

Institute

Leadership

Press room

Media inquiries

Compliance

Careers

Job listings

Contact

SRI Ventures

Our locations

Headquarters

333 Ravenswood Ave
Menlo Park, CA 94025 USA

+1 (650) 859-2000

Subscribe to our newsletter


日本支社
SRI International
  • Contact us
  • Privacy Policy
  • Cookies
  • DMCA
  • Copyright © 2023 SRI International
Manage Cookie Consent
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes. The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
Manage options Manage services Manage {vendor_count} vendors Read more about these purposes
View preferences
{title} {title} {title}