Hardware-Friendly Activation Function Designs and Its Efficient VLSI Implementations for Transformer-Based Applications

Yu Hsiang Huang*, Pei Hsuan Kuo, Juinn Dar Huang

*此作品的通信作者

研究成果: Conference contribution同行評審

1 引文 斯高帕斯(Scopus)

摘要

The activation function is one of key elements in modern machine learning algorithms. However, some broadly-used activation functions are exceptionally complex, e.g., GELU in Transformer-based algorithms, which makes their precise yet efficient VLSI implementations extremely hard. In this paper, two series of hardware-friendly activation function designs, DNR and PWL, and their VLSI implementations are proposed. Both are specifically designed to replace GELU, which is widely used in Transformer-related applications. Instead of utilizing traditional lookup-table (LUT)-based approximation methods, this paper introduces new activation functions that are not only hardware-friendly but successfully alleviate the dying neuron issue. Besides, each series includes a number of members, which can be freely selected through programming to best fit a given application. Experimental results indicate that the proposed new activation functions achieve comparable or even better model accuracy as compared to GELU. Moreover, the highly efficient and flexible VLSI implementations support 16 different Q-formats to maximize the output precision under various input scales. Compared with approximation-based implementation strategies, the proposed activation function designs and the corresponding LUT-free hardware implementations do achieve a significant improvement in speed, area, and power.

原文English
主出版物標題AICAS 2023 - IEEE International Conference on Artificial Intelligence Circuits and Systems, Proceeding
發行者Institute of Electrical and Electronics Engineers Inc.
ISBN(電子)9798350332674
DOIs
出版狀態Published - 2023
事件5th IEEE International Conference on Artificial Intelligence Circuits and Systems, AICAS 2023 - Hangzhou, China
持續時間: 11 6月 202313 6月 2023

出版系列

名字AICAS 2023 - IEEE International Conference on Artificial Intelligence Circuits and Systems, Proceeding

Conference

Conference5th IEEE International Conference on Artificial Intelligence Circuits and Systems, AICAS 2023
國家/地區China
城市Hangzhou
期間11/06/2313/06/23

指紋

深入研究「Hardware-Friendly Activation Function Designs and Its Efficient VLSI Implementations for Transformer-Based Applications」主題。共同形成了獨特的指紋。

引用此