The emerging field of edge computing could rein in AI’s energy splurge. The SRI NeuroEdge framework aims to further increase edge computing’s efficiency and responsiveness.
With artificial intelligence (AI) applications continuing to skyrocket, many observers are concerned about the technology’s exorbitant power consumption. Various estimates point to global AI power needs rivaling the generating capacity of the entire state of California within just two years or even exceeding 10% of total power demand in the United States by 2030.
One promising way to slash energy use — while also improving AI performance in the process — is known as edge computing. Its essential strategy: moving power-hungry data processing as close as physically possible to where the data is generated or needed, cutting down on conventional data transmission and centralized processing costs. Examples of this “edge” network space include apps and sensors in places such as factory floors, self-driving cars, retail stores, and smartphones.
“NeuroEdge allows AI-based systems to efficiently operate in rapidly changing environments and adapt in real-time.” — David Zhang
To advance sustainable AI innovation, SRI is pursuing an ambitious computing project called SRI NeuroEdge. The software framework brings a “neuromorphic” approach to edge computing, drawing conceptually from how the human brain works. Already, the framework is demonstrating that it can make AI significantly more data- and power-efficient without meaningfully compromising performance.
“NeuroEdge allows AI-based systems to efficiently operate in rapidly changing environments and adapt in real-time,” says David Zhang, senior technical manager in the Vision Systems Lab at SRI’s Center for Vision Technologies. “There is a great need for smaller devices that can run with the same functionality as today’s microprocessors and GPUs [graphic processing units] but have lower power consumption and higher compute efficiency. We’re addressing that need with the suite of technologies that we’ve folded into NeuroEdge.”
The edge in NeuroEdge
NeuroEdge formally kicked off in late 2021, supported by internal funding from SRI and external funding from the governmental Defense Advanced Research Projects Agency (DARPA) and Intelligence Advanced Research Projects Activity (IARPA). Though the awards each address specific challenges, they share the common theme of ushering in advances in software and hardware that reduce or obviate reliance on centralized cloud computing resources.
Most current work on applying AI to sensor data confronts a single fundamental challenge: Some sensor data is relevant to making decisions, but much of it is not, and every additional byte of data requires additional time and electricity to process. So how can hardware and software be orchestrated to focus only on relevant data, thereby achieving results more quickly and cost-effectively?
There is no single silver bullet that can radically increase the speed and efficiency of AI at the edge. Instead, it’s a matter of carefully layering the interventions that can make the biggest difference. NeuroEdge aims to do so by orchestrating multiple approaches to improve edge computing, many of them neuromorphic in nature.
One of these approaches built into NeuroEdge is hyperdimensional computing. Unlike conventional computing, which uses the familiarly simple binary representations of 0s and 1s for instructions and data, hyperdimensional computing involves high-dimensional vectors. A vector is an array of 0s and 1s representing data — for instance, the pixel intensities and locations in an image. “Hyperdimensional,” meanwhile, means the array can be thousands of bits long, representing more than can be conveyed in just the x, y, and z dimensions in a typical 3D coordinate plane. These long arrays are redundant but “noise”-resilient, meaning they are robust against errors, and the information in them is distributed yet computationally efficient. The human brain has been shown to represent information in this same hyperdimensional mode, which can be more effective at precisely describing data attributes than standard, low-dimensional computing.
This hyperdimensional computing tactic contributes to a major feature of NeuroEdge, called domain adaptation. The software is configured to focus on detecting and collecting objects not already represented in the AI model’s training dataset. Rather than retraining the entire model away from the edge to accommodate these novel items, however — which is costly in memory, computation, and energy — NeuroEdge can have adaptors added to its model that represent mere fractions of the network’s total size. Only these small adaptors need to be retrained. Helpfully, by preserving the original model, NeuroEdge never “forgets” what it already knows, a common problem with today’s AI models. Accordingly, NeuroEdge keeps adapting as the environment changes — enabling what Zhang and colleagues dub “continual learning,” akin to the flexible incorporation of new knowledge that biological brains nimbly handle.
All told, NeuroEdge’s features contribute to a process that is 100 or even 1,000 times more efficient than a standard AI edge system configuration.
Another neuromorphic aspect of NeuroEdge is in-pixel processing, which involves performing data operations right inside a camera’s pixel array, rather than processing data elsewhere within the device or at some distant server farm. The human visual system executes a version of this in-pixel processing, known as early vision. The retina and “lower” brain regions, chiefly the thalamus, initially filter and sort what we see, instead of flooding the brain’s cognitive regions with signals from every perceived photon of light. “Our eyes don’t send the raw signal to the brain,” says Zhang. “Our brain is not able to handle it.”
Likewise, NeuroEdge does front-end processing before sending data along for more intensive back-end processing. The front end borrows another tactic from the brain playbook by avoiding redundant processing, courtesy of the so-called saccadic mechanism. When humans look at something, our eyes jerk about in their sockets, fixating on new parts of the scene while largely ignoring objects that have already been kept and categorized in visual memory.
For NeuroEdge, this saccadic approach involves using AI in componentry installed right amongst the pixels to select initial areas of highest interest. For self-driving cars, for instance, salient objects would be other vehicles and pedestrians instead of, say, clouds in the sky or roadside trees. The NeuroEdge-enabled device can then build a scene dynamically from these patches, rather than processing the entire visual field multiple times. The upshot: a ten-fold reduction in data bandwidth going from the sensor to the back end and similar drops in operations execution time.
A final and critical edge-performance-boosting concept in NeuroEdge is the innovative employment of “heterogenous federated learning,” where multiple devices share what they’ve individually learned at the edge. While this local computing method is far thriftier than sending all sensor data back to a central server, it must deal with challenges such as devices collecting highly divergent sets of data and having different hardware capabilities. NeuroEdge’s solution: scaling the AI models running and updating at each device to match available resources and to again rely on local adaptors.
All told, NeuroEdge’s features contribute to a process that is 100 or even 1,000 times more efficient than a standard AI edge system configuration.
Coming soon to an edge near you
As NeuroEdge matures, early adopter clients have already come calling and SRI is exploring the opportunities. Edge applications are vast across both national defense and industries like transportation, logistics, and consumer electronics.
One of the things that makes NeuroEdge particularly applicable across all these areas is that it’s completely agnostic to the kind of sensor data it processes. Inputs might come from cameras, hyperspectral imagers, Raman spectrometers, pressure sensors, microphones, or numerous other sources. In an indication of NeuroEdge’s widespread utility, governmental funding agencies selected NeuroEdge to be the sole software developer for certain non-conventional hardware devices — such as photonic circuits — being developed by other government contractors.
“We’re very pleased about our progress already and the interest that NeuroEdge is generating,” says Zhang. “Edge computing is indeed at the cutting edge of current compute capabilities and could lead to great advances, and we look forward to being part of the solution.”
To learn more about how SRI is charting the future of AI, contact us.
This research was, in part, funded by the U.S. Government. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government.
This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via 2022-21100600001. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
