The PRINCE software will be able to create photorealistic 3D renderings from satellite and aerial photos, with a limited number of ground-level images.
For most of us, getting a preview of a place that we’re headed to is as simple as opening the “street view” option in a mapping app. We can rotate the perspective on our screen and see what the area looks like at ground level. But for aid workers, the military, and law enforcement responding to disasters or conducting missions in remote locations, there are often very limited images of the places they’re headed and no three-dimensional walk-throughs to help them prepare.
Researchers at SRI’s Center for Vision Technologies (CVT) and their collaborators are designing software that will be able to quickly build photorealistic, three-dimensional renderings of environments with limited imagery. The system, known as PRINCE (Photorealistic Rendering from Neural Column Embeddings), will enable first responders, aid workers, the military, and law enforcement personnel to familiarize themselves with a mission site long before they actually set foot there.
“In the event of a disaster, like the fires in Hawaii, you want to be able to build an accurate 3D model very quickly, so humanitarian relief or the military can plan their response,” said Rakesh “Teddy” Kumar, Vice President of Information and Computing Sciences at SRI and director of CVT. “They can use the model to rehearse for a mission and reduce the risks for their team.”
The project is part of the Walk-though Rendering Images of Varying Altitudes program, which was created by Intelligence Advanced Research Projects Activity (IARPA). Over the next 18 months, SRI researchers and their collaborators will build, refine, and test their software against three different scenarios with decreasing amounts of available ground-level imagery. The goal of this first stage is to be able to create a high-quality rendering of a 200-square-meter area in just 12 hours.
“In the event of a disaster, like the fires in Hawaii, you want to be able to build an accurate 3D model very quickly, so humanitarian relief or the military can plan their response,” said Rakesh “Teddy” Kumar, Vice President of Information and Computing Sciences at SRI and director of CVT. “They can use the model to rehearse for a mission and reduce the risks for their team.”
The SRI team led by Kumar and Supun Samarasekera, Senior Technical Director for CVT’s Vision and Robotics laboratory, includes Vincent Sitzmann from MIT, Rongjun Qin from Ohio State University, Yajie Zhao from the University of Southern California, and Stephan Mandt from UC Irvine.
SRI brings decades of 3D modeling, image alignment, and immersive visualization expertise to the table, and their fellow team members bring additional knowledge and world-leading expertise in neural scene modeling, satellite image analysis, building synthetic 3D photorealistic environments with relighting models, uncertainty reasoning and visual image enhancement, and other important areas.
“We have assembled a team of experts that is ideally suited to solve this problem,” Samarasekera said.
The researchers will build on recent advances in machine learning to pull together images from a variety of sources—including satellite imagery, drone footage, and ground level photography—into a single three-dimensional rendering. The PRINCE program will identify from where and at what angle these images were taken, incorporate all available data into one rendering, and then be able to fill in the gaps between them by inferring what should be there.
“It takes all these images and builds a neural network model of the scene,” Rakesh said. “And then we keep training and testing and adjusting the model to make it as accurate as possible.”
The results won’t always be perfect, especially in cases with only a few available images, but perfection isn’t the goal. When a mission is in an unfamiliar location, or a location that has recently changed, any additional information can make a difference. The researchers believe that PRINCE’s 3D renderings will give aid workers, military personnel, and law enforcement the extra data and preparation they need to make their missions more likely to succeed, and safer for everyone involved.
