TY - GEN
T1 - Meta Learning for Domain Agnostic Soft Prompt
AU - Chen, Ming Yen
AU - Rohmatillah, Mahdin
AU - Lee, Ching Hsien
AU - Chien, Jen Tzung
N1 - Publisher Copyright:
© 2023 IEEE.
PY - 2023
Y1 - 2023
N2 - The prompt-based learning, as used in GPT-3, has become a popular approach to extract knowledge from a powerful pre-trained language model (PLM) for natural language understanding tasks. However, either applying the hard prompt for sentences by defining a collection of human-engineering prompt templates or directly optimizing the soft or continuous prompt with labeled data may not really generalize well for unseen domain data. To cope with this issue, this paper presents a new prompt-based unsupervised domain adaptation where the learned soft prompt is able to boost the frozen pre-trained language model to deal with the input tokens from unseen domains. Importantly, the meta learning and optimization is developed to carry out the domain agnostic soft prompt where the loss for masked language model is minimized. The experiments on multi-domain natural language understanding tasks show the merits of the proposed method.
AB - The prompt-based learning, as used in GPT-3, has become a popular approach to extract knowledge from a powerful pre-trained language model (PLM) for natural language understanding tasks. However, either applying the hard prompt for sentences by defining a collection of human-engineering prompt templates or directly optimizing the soft or continuous prompt with labeled data may not really generalize well for unseen domain data. To cope with this issue, this paper presents a new prompt-based unsupervised domain adaptation where the learned soft prompt is able to boost the frozen pre-trained language model to deal with the input tokens from unseen domains. Importantly, the meta learning and optimization is developed to carry out the domain agnostic soft prompt where the loss for masked language model is minimized. The experiments on multi-domain natural language understanding tasks show the merits of the proposed method.
KW - Domain adaptation
KW - meta-learning
KW - natural language understanding
KW - prompt-based learning
UR - https://www.scopus.com/pages/publications/85169613747
U2 - 10.1109/ICASSP49357.2023.10094860
DO - 10.1109/ICASSP49357.2023.10094860
M3 - Conference contribution
AN - SCOPUS:85169613747
T3 - ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
BT - ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing, Proceedings
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 48th IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2023
Y2 - 4 June 2023 through 10 June 2023
ER -