Dynamic Gradient Sparse Update for Edge Training

I. Hsuan Li*, Tian Sheuan Chang

*此作品的通信作者

研究成果: Conference contribution同行評審

摘要

Training on edge devices enables personalized model fine-tuning to enhance real-world performance and maintain data privacy. However, the gradient computation for backpropagation in the training requires significant memory buffers to store intermediate features and compute losses. This is unacceptable for memory-constrained edge devices such as microcontrollers. To tackle this issue, we propose a training acceleration method using dynamic gradient sparse updates. This method updates the important channels and layers only and skips gradient computation for the less important channels and layers to reduce memory usage for each update iteration. In addition, the channel selection is dynamic for different iterations to traverse most of the parameters in the update layers along the time dimension for better performance. The experimental result shows that the proposed method enables an ImageNet pre-trained MobileNetV2 trained on CIFAR-10 to achieve an accuracy of 85.77% while updating only 2% of convolution weights within 256KB on-chip memory. This results in a remarkable 98% reduction in feature memory usage compared to dense model training.

原文English
主出版物標題ISCAS 2024 - IEEE International Symposium on Circuits and Systems
發行者Institute of Electrical and Electronics Engineers Inc.
ISBN(電子)9798350330991
DOIs
出版狀態Published - 2024
事件2024 IEEE International Symposium on Circuits and Systems, ISCAS 2024 - Singapore, Singapore
持續時間: 19 5月 202422 5月 2024

出版系列

名字Proceedings - IEEE International Symposium on Circuits and Systems
ISSN(列印)0271-4310

Conference

Conference2024 IEEE International Symposium on Circuits and Systems, ISCAS 2024
國家/地區Singapore
城市Singapore
期間19/05/2422/05/24

指紋

深入研究「Dynamic Gradient Sparse Update for Edge Training」主題。共同形成了獨特的指紋。

引用此