A semiconductor chip system involves multiple levels of design abstractions and implementations – application algorithms, hardware architectures, circuits and devices. In the past, each level was handled separately; however, three professors at Texas A&M University are collaborating across algorithm and circuit levels to bridge the gap for more cost-effective, high-performance solutions.
Drs. Jiang Hu, Dileep Kalathil and Sunil Khatri in the Department of Electrical and Computer Engineering received a National Science Foundation (NSF) grant for their three-year project ERL: Edge Reinforcement Learning with Algorithm, Architecture and Circuit Co-Design. This initiative is a part of NSF’s Future of Semiconductors program.
The team’s goal is to enable energy efficient computing for reinforcement learning with applications in robotics and biomedical devices.
“Think of health monitors in nursing homes or hospitals. These are all prescribed solutions, but each person behaves differently; reinforcement learning will allow those devices to adapt to individual cases,” Hu said.
The team aims to develop cross-level core optimization techniques that allow reinforcement learning – when a computer learns behavior through trial and error – to be performed on edge devices such as cell phones, laptops and robots with limited computational resources by using flash memory technology and approximate arithmetic circuits.
Flash memory is mostly used for storing data on computers and other devices. For example, USB thumb drives and memory sticks are all based on flash. Khatri has been working on new ways to use flash for logical computing, not just for storage, which can improve energy efficiency compared to conventional circuits.
Think of health monitors in nursing homes or hospitals. These are all prescribed solutions, but each person behaves differently; reinforcement learning will allow those devices to adapt to individual cases.
“For about 10 years now, we have been demonstrating how the use of flash can result in superior circuits for both analog and Digital Integrated Circuits. In this project, we are excited to apply this knowledge to edge reinforcement learning, which is a problem that is well suited for flash-based realization,” Khatri said.
Together, Khatri and Hu are working on approximate arithmetic or stochastic arithmetic circuits. A conventional arithmetic circuit can perform multiplication, division, addition and subtraction. These circuits are supposed to be accurate when subjected to their limited bit width. The team found that if they allowed occasional small errors, they could greatly simplify the circuit and make it smaller and consume much less power.
“When we combine the approximate arithmetic circuit with flash, we are able to make the implementation much more power efficient that will potentially allow reinforcement learning to fit into resource limited edge devices,” Hu said.
While flash has advantages, it also has drawbacks. Imagine flash as a sheet of paper that you can read and write information on. The more times you write, the weaker the paper becomes. You can read as much as you want, but there is a limit to how much you can write.
To curb this limitation, Kalathil developed an algorithm that requires less write operation. This allows for more robust, long-lasting and cost-effective flash memory. It also improves energy efficiency since writing information draws more energy than reading. In small edge devices, this is especially useful since they are most often battery-powered or have limited power budgets.
“Reinforcement learning offers an attractive framework for developing learning-based algorithms for edge devices,” Kalathil said. “However, most off-the-shelf reinforcement learning algorithms require a lot of data and compute to make even small adaptation according to the edge users’ preferences, and edge devices such as mobile phones often do not have this capability."
This problem cannot be solved as a software problem or hardware problem individually, and that is where our project team’s joint expertise come in. We plan to develop novel cross-layer (architectures-circuits-algorithms) approaches to solve this challenging problem. It is truly interdisciplinary and super exciting!”